- nicer error in Registrar builder instead of an NPE
- nicer error if hosts were deleted in GenerateZoneFilesAction
- extra permissions checks in ConsoleUsersAction
- using "replace" (text) instead of "replaceAll" (regex) in
BulkDomainAction
Previously, if there were no XML bytes somehow, it'd increment a bunch
of counts for requirements that shouldn't be incremented. We should only
increment these values if we have the data to do so.
Note: this doesn't happen and shouldn't happen, so there's no issues
currently occurring, but this code is more correct anyway.
During our implementation of Expiry Access Period (XAP), adding a field to the
Registrar entity altered database heap scan order when loadAllOf() was
called without an explicit ORDER BY clause. This caused non-deterministic
row shifting in the Angular console registrar table (/console/registrars),
breaking golden image comparisons in ConsoleScreenshotTest.
To permanently fix this root cause and guarantee deterministic UI rendering
and test stability across schema evolutions, this commit introduces sorted
database loading across our persistence layer.
Specifically, this commit:
- Adds loadAllOfSorted and loadAllOfSortedStream to TransactionManager,
JpaTransactionManagerImpl, and DelegatingReplicaJpaTransactionManager.
- Enforces strict allowlist regex validation (^[a-zA-Z0-9_.]+$) on property
names in JpaTransactionManagerImpl before appending dynamic ORDER BY
clauses, preventing JPQL/SQL injection since bind parameters cannot be
used for schema identifiers.
- Adds Registrar.loadAllSorted and updates RegistrarsAction to explicitly
sort registrars alphabetically by registrarName and registrarId.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to
reflect the newly enforced deterministic alphabetical registrar ordering.
- Adds comprehensive unit test coverage in JpaTransactionManagerImplTest
verifying exact sorted entity retrieval across multiple fields and regex
rejection of invalid field names.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/531889856
This hasn't been an issue, but it seems like a good idea. Note that I'm
allowing prices of 0 here just in case there's some shenanigans at some
point in time where we want a domain to technically be premium without
having a cost.
Implement programmatic verification of Public Access Prevention (PAP) on all GCS buckets before performing write operations via GcsUtils.
- Add verifyPublicAccessPrevention(String bucketName) to GcsUtils.java, querying the bucket's IAM configuration and asserting that PAP is ENFORCED.
- Call verifyPublicAccessPrevention in all output stream and byte creation write paths in GcsUtils.
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.
Specifically, it:
- Adds Flyway migration
V224__add_registrar_expiry_access_period_enabled.sql, which adds the
expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
consult db/README.md, follow the mandatory Two-PR deployment split when
altering database schemas, and run both :db:test and
:core:sqlIntegrationTest whenever database schema or ORM mappings are
touched.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
the new default database ordering after adding the column. (Note: A
separate pull request, PR #3136 / http://go/r3pr/3136, permanently fixes
this underlying sort non-determinism across our persistence layer by
enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
This is a small edge case where a cancellation for year 2 could cancel
out events in year 1 if we re-ran billing for year 1 for some reason. I
don't think we would ever re-run one year later so this is kind of moot,
but it's probably still a good thing to do.
Configure a throwing XMLResolver on XMLInputFactory in XmlTransformer
to act as an additional defense-in-depth security layer against XML
External Entity (XXE) and entity expansion/resolution attacks.
This prevents resolution of external and internal entities in StAX
parsing pipelines, ensuring that malformed payloads containing entity
definitions are safely blocked and rejected.
Also add a unit test to EppXmlSanitizerTest verifying that EPP
messages containing DTD and external entities fail parsing and fall
back safely to their Base64 representation.
TAG=agy
CONV=610c2358-a99f-4605-94cd-ff0d4ee08176
BUG= http://b/529387728
Add validation to RefreshDnsForAllDomainsAction to ensure the tlds parameter is not empty when invoked. Previously, invoking the action without specifying tlds would result in a no-op that logged an empty list and exited without error. This change makes missing tlds an explicit error, throwing an IllegalArgumentException so users running the command via nomulus curl receive clear feedback that the parameter is required.
In commit 17b851de42 (#2979), <element name="contact"> and related types were incorrectly removed from core/src/main/java/google/registry/xml/xsd/domain.xsd and rde-domain.xsd. Because those XSD schemas are published standards (RFC 5731 and RFC 5831), altering them caused JAXB unmarshaling to reject historical database records (HistoryEntry) containing <domain:contact> elements during OT&E status checks.
Restore <element name="contact" minOccurs="0" maxOccurs="unbounded"/> to domain.xsd and rde-domain.xsd so all XSD schemas conform strictly to published EPP RFCs and historical contact elements unmarshal cleanly without throwing cvc-complex-type.2.4.a schema validation errors.
Remove deprecated <domain:contact> elements from prober creation payload (prober/.../create.xml) to match current flow rules where contact associations are forbidden.
Remove trailing slash from ote-status route parameter in RegistrarDetailsComponent to prevent double-slash (/ote-status//<id>) URL generation when checking OT&E status.
BUG= http://b/529391412
Domains in the redemption grace period can still be restored, so hosts
referencing them cannot be deleted. Fortunately this is already the
case. This just adds a test.
Reject loopback, link-local, site-local, wildcard, and multicast IP
addresses during host creation and update flows.
Glue records (A/AAAA records published in the parent zone for subordinate
name servers) must point to globally routable, public IP addresses to
ensure that recursive DNS resolvers on the public internet can reach the
authoritative name servers.
Using non-public or non-routable IP addresses in glue records is invalid
for the following reasons:
- Loopback (127.0.0.1, ::1) and Any-Local (0.0.0.0, ::) addresses point
back to the client or are unspecified, causing resolvers to query
themselves and fail.
- Private/Site-Local (e.g., 10.0.0.0/8, 192.168.0.0/16) and Link-Local
(169.254.0.0/16) addresses are not routable on the public internet,
rendering the delegated domain completely unreachable to external clients.
- Multicast addresses are designed for one-to-many delivery and cannot
be used for standard unicast DNS queries to a specific name server.
Rename LoopbackIpNotValidForHostException to IpAddressNotRoutableException
to reflect the broader set of forbidden non-routable IP addresses.
This commit addresses the following security vulnerabilities identified in the recent audit of the Console App and Backend APIs:
1. Angular XSS: Removed unsafe [innerHTML] bindings across all console-webapp templates (Contact, Registrars, Registrar Details, Users List) in favor of standard Angular interpolation.
2. Broken Access Control (IDOR): PasswordResetRequestAction and PasswordResetVerifyAction now explicitly verify that the target user's email belongs to the authorized registrarId.
3. Missing Permission Check: ConsoleEppPasswordAction now explicitly checks for CONFIGURE_EPP_CONNECTION permission before updating the EPP password.
4. Denial of Service (DoS): ConsoleBulkDomainAction now strictly limits the size of bulk domain lists (configurable, default 500) to prevent thread exhaustion.
5. Denial of Service (OOM): ConsoleHistoryDataAction now uses .setMaxResults() (configurable, default 500) on JPA native queries to prevent eager loading of the entire database into memory.
Makes the history query limit and bulk domain action limit configurable via RegistryConfig, allowing smaller limits to be used in tests to avoid heavy resource persistence.
Also removes an outdated Joda-Time migration reference from GEMINI.md.
This commit improves the database documentation in db/README.md by adding comprehensive guidelines for refactoring column types and managing two-PR schema deployments.
Key additions:
- Added a section on the "Expand and Contract" pattern for refactoring column types, explaining when it is safe to drop columns immediately vs. when a three-step release process is required.
- Added a section on writing safe NOT NULL migrations for timed transition properties, explaining the "Temporary Database Default" pattern to maintain backward compatibility with running servers during Two-PR deployments, and demonstrating the required explicit PostgreSQL `::hstore` casting syntax.
- Added a step-by-step "Recommended Git Workflow" section to help developers cleanly split their database and Java changes into chained PRs using Git.
TAG=agy
CONV=88271e71-e272-40e0-85f8-a075a423b7c2
This commit implements the database schema changes for the Expiry Access Period (XAP) launch configuration on TLDs. It represents the first step of a Two-PR deployment strategy, deploying the database schema changes in advance of the server logic.
Specifically, it replaces the `expiry_access_period_enabled` boolean column (originally introduced in PR #2804) with a new `expiry_access_period_transitions` hstore column.
Why we are making this change:
A basic boolean flag only allows an immediate, manual on/off toggle. To launch XAP on a TLD, registry operators would have to manually flip the flag at the exact launch time, which is operationally fragile and cannot be planned in advance. Refactoring this to an hstore-backed timed transition map (mapping Instant to ExpiryAccessPeriodMode) allows operators to schedule the XAP launch in advance via TLD YAML configurations. The registry will automatically transition the TLD to the ENABLED mode at the scheduled timestamp, aligning with how other scheduled TLD changes (like TLD states and EAP fee schedules) are managed.
Since the original boolean column was never mapped in Java (PR #2804 only added the database column), it is completely safe to drop it immediately in this migration.
To ensure backward compatibility with running servers (which are still executing the old Java code during the deployment transition), the new column is added as `NOT NULL` with a temporary `DEFAULT` constraint. This prevents constraint violations on inserts from old servers. A TODO has been left in the SQL migration to drop this default in a subsequent schema release once the Java changes have been deployed.
TAG=agy
CONV=88271e71-e272-40e0-85f8-a075a423b7c2
Refactor RdapDomainSearchActionTest to dynamically resolve all domain
and host Repository IDs (ROIDs) instead of asserting on hardcoded,
sequence-generated strings (like "2E-LOL" or "6-LOL").
When tests are executed in parallel (as is common in CI environments like
Kokoro), multiple test threads concurrently reset and allocate from the
shared database sequence 'project_wide_unique_id_seq'. This interleaves
ID allocations non-deterministically, causing any test asserting on
exact, hardcoded sequence values to flake.
To fix this, createManyDomainsAndHosts was updated to return the list of
persisted domains, allowing tests to dynamically resolve their ROIDs.
All other test cases were refactored to dynamically fetch the ROIDs of
pre-created domains and hosts (stored in fields or in hostNameToHostMap,
using punycode keys for IDN hosts) for their JSON assertions, rendering
the entire suite robust against sequence shifts.
we shouldn't have to rebuild it each time we get a request to a
different service or really ever at all -- we might get a tiny bit of
cache benefit here
Summarize all documentation updates across the repository to align with modern GKE, Cloud SQL Proxy v2, standard EPP fee v1.0, and Postgres database environments.
Key Updates:
- Prerequisites: Bump Java requirement to Java 25.
- Architecture & Scaling: Document GKE workloads, Cloud Tasks queues, and scheduled tasks. Replace App Engine references with GKE deployment restart commands (kubectl rollout restart).
- Configuration: Update Cloud SQL Proxy instructions to v2, fix keyring verification commands, and document IAP configuration.
- Escrow (RDE/BRDA): Fix manual generation and download procedures to match the Dataflow job ID folder structure, and correct deposit encryption/verification command parameters.
- Monitoring: Correct metric names and expand the documented metrics list with caching, locking, and reserved list metrics.
- Fixes: Standardize lists formatting across markdown files, fix broken webdriver links, and resolve various typos.
- Cleanup: Remove leftover cloud scheduler configurations for the deleted wipeOutContactHistoryPii task, and update ICANN reporting documentation to reflect open-sourced DNS query coordinator.
TAG=agy
CONV=88271e71-e272-40e0-85f8-a075a423b7c2
Normal comparison exits out once a difference is found so theoretically
a time variance can leak information. This isn't really a huge deal but
is probably still worth doing.
Introduced overloaded helper methods `persistFeatureFlag` in `DatabaseHelper` to simplify the creation and persistence of `FeatureFlag` entities in test setups.
1. `persistFeatureFlag(FeatureName, FeatureStatus)`: Persists a feature flag with a single status starting from the Unix Epoch (`START_INSTANT`).
2. `persistFeatureFlag(FeatureName, FeatureStatus, Instant, FeatureStatus)`: Persists a feature flag with an initial status at `START_INSTANT` and a subsequent transition at a specified time.
Refactored 11 occurrences of manual 1-transition flag creation and 5 occurrences of 2-transition flag creation across the test suite to use these new helpers, significantly reducing boilerplate and improving test readability.
TAG=agy
CONV=583b8a23-9fe5-476d-ac35-aeba7b218eb0
This is similar to PR #3069 but for the algorithms themselves rather
than the digest data. This forbids algorithms, that, according to RFC
9904, should not be used.
This commit reverts changes from PR #3068 that swapped 'npm run build' for 'npx ng build' while attempting to dynamically set the '--output-path' via the CLI.
Passing '--output-path' on the command line overrides the entire 'outputPath' configuration object in angular.json. Because the new Angular 18 Application Builder (esbuild) nests outputs inside a 'browser/' directory by default, overriding the configuration bypassed the 'browser: ""' flattening property, causing all client assets to be nested deeper than expected.
This resulted in empty deployments because downstream tasks (like Jetty's copyConsole and the deployment tar scripts) expected the assets to be completely flat. By removing the '--output-path' override from the 'npx ng build' calls, the Angular CLI once again respects angular.json, flattens the output into 'staged/dist/', and the restored 'doLast' block successfully copies the artifacts where they belong.
This commit reverts changes from 5599a0eb3d and most of 5286b1a0dc (PR #3068) that stripped essential dependencies (buildConsoleForAll, buildNomulusImage, buildToolImage, fragileTest) from the default './gradlew build' target, which broke downstream deployment pipelines. It restores the default build to correctly generate all necessary production artifacts and Docker images.
It introduces a new 'fastBuild' target designed explicitly for local developers and CI checks. This lightweight target disables the execution of heavy Docker image builds, Angular compilations, and fragile tests to provide rapid feedback. Sequential execution constraints for parallel Angular builds are maintained to prevent cache corruption.
It updates the ':core:generateSqlSchema' task to execute using the 'unittest' environment instead of 'alpha'. The 'alpha' configuration is a private, internal environment config that is not distributed in the open-source repository, which caused the task to fail for public contributors. By switching to 'unittest', the generator can successfully run using the public test configuration. With this fixed, it also includes the newly generated 'db-schema.sql.generated' file, which now correctly tracks the 'FORBID_INSECURE_ALGORITHMS_RFC_9904' feature flag that was recently added.
Finally, it implements a split-runner execution strategy for the 'sqlIntegrationTest' task to permanently resolve 'failed to discover tests' and 'NoSuchMethodError' exceptions on Kokoro. Because Kokoro tests cross-version compatibility against both legacy deployed artifacts (compiled with JUnit 4 @RunWith wrappers) and modern artifacts (compiled with JUnit 5 @Suite annotations), we cannot statically configure a single test runner. We now dynamically run both the legacy 'useJUnit()' and modern 'useJUnitPlatform()' runners sequentially with 'failOnNoDiscoveredTests' disabled, allowing the appropriate engine to discover and execute the suite without causing classpath collisions.
This commit adds the buildAll task to restore the existence of a target that builds everything, which was unintentionally removed when the default build was stripped down in PR #3068. It also introduces necessary sequential constraints to the console-webapp build tasks to prevent parallel execution from corrupting the Angular CLI cache. Finally, it addresses paths for the newer Angular esbuild output and hardens the style injection in ConsoleScreenshotTest to prevent fragile test failures.
gbrodman
Ben McIlwain
Weimin Yu
Juan Celhay