This commit implements the second stage (Java ORM mappings and EPP flow
enforcement) of the Expiry Access Period (XAP) launch and opt-in mechanism,
completing the Two-PR deployment split mandated by db/README.md after the
Stage 1 database migrations (PR #3134) are live.
During XAP, a TLD charges a fee for domain registration during a timed period
after deletion. To prevent accidental charges, registrars must explicitly
opt in to participate in XAP registrations.
Specifically, this commit:
- Adds expiryAccessPeriodTransitions (a TimedTransitionProperty of
ExpiryAccessPeriodMode) to Tld.java and ExpiryAccessPeriodModeTransitionUserType
for mapping XAP mode schedules to PostgreSQL hstore columns.
- Adds expiryAccessPeriodEnabled to Registrar.java with getter, builder
setter, and JSON map serialization, and regenerates db-schema.sql.generated.
- Enforces registrar opt-in in DomainCheckFlow: when an unallocated domain is
in XAP and the querying registrar has not opted in, domain:check returns
avail="0" with reason "Reserved".
- Enforces registrar opt-in in DomainCreateFlow: when attempting to register
a domain in XAP without registrar opt-in, throws DomainReservedException
(EPP error 2304 "Object status prohibits operation").
- Enforces explicit fee acknowledgment during XAP domain creation when the XAP
fee is non-zero, throwing FeesRequiredDuringExpiryAccessPeriodException in
DomainFlowUtils if omitted.
- Creates a one-time BillingEvent with Reason.FEE_EXPIRY_ACCESS when a domain
is created during XAP with a non-zero fee, and adds getXapCost() to
FeesAndCredits.
- Updates all test TLD YAML configurations and adds comprehensive unit and
integration tests across DomainCheckFlowTest, DomainCreateFlowTest,
DomainPricingLogicTest, and RegistrarTest.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
Lazy-initialize the terminal and LineReader in ShellCommand to prevent JLine from eagerly probing terminal capabilities during JCommander command-line startup.
During CLI startup, JCommander instantiates every command (including ShellCommand) to build the CLI's command map. Eager instantiation of the LineReader inside the ShellCommand constructor causes JLine to query the terminal cursor via Device Status Report (DSR) escape sequences. Standard commands (e.g. list_tlds) do not consume stdin, leaving these probing responses (such as ^[[71;1R and 1;1R) in the buffer to be leaked as garbage characters to standard output.
This change defers terminal and LineReader initialization until the shell's run() method is actually executed, while preserving the original constructor for unit tests.
BUG=http://b/534855218
this reduces prod bundle size a bit and cleans things up
by including testcontainers in the nonprod deps of core, we can still
run things like the schema-generating commands
When checking in a brand new premium list (production/*.txt) or reserved list (reserved/production/*.txt), presubmit testing commands in gradle-runner.sh fall back from update_premium_list / update_reserved_list (which require existing database entities) to create_premium_list / create_reserved_list.
However, because --dry_run (-d) and --build_environment flags were previously only defined in UpdatePremiumListCommand and UpdateReservedListCommand, passing them during non-interactive presubmit creation validation resulted in ParameterException (Was passed main parameter '-d' but no main parameter was defined...) or NullPointerException when ConfirmingCommand attempted to prompt on a null system console.
This change moves --dry_run and --build_environment parameters and dontRunCommand() behavior up into CreateOrUpdatePremiumListCommand and CreateOrUpdateReservedListCommand, enabling safe dry-run presubmit validation for newly created lists in CI environments.
BUG= http://b/534511633, http://b/529397845
Implement a hybrid Content Security Policy (CSP) for the Registrar Console
to protect against XSS
- The CspFilter injects the proper headers on the Java backend endpoints
- Uinsg Jetty's HeaderFilter to inject the header for statically-served
frontend assets
We need to add the ee10-servlets.ini file for Jetty to have acess to the
HeaderFilter class
It's not perfect because we're running outside of a transaction, but we
should make a best-effort attempt to project domains/hosts to the
current time.
- nicer error in Registrar builder instead of an NPE
- nicer error if hosts were deleted in GenerateZoneFilesAction
- extra permissions checks in ConsoleUsersAction
- using "replace" (text) instead of "replaceAll" (regex) in
BulkDomainAction
Previously, if there were no XML bytes somehow, it'd increment a bunch
of counts for requirements that shouldn't be incremented. We should only
increment these values if we have the data to do so.
Note: this doesn't happen and shouldn't happen, so there's no issues
currently occurring, but this code is more correct anyway.
During our implementation of Expiry Access Period (XAP), adding a field to the
Registrar entity altered database heap scan order when loadAllOf() was
called without an explicit ORDER BY clause. This caused non-deterministic
row shifting in the Angular console registrar table (/console/registrars),
breaking golden image comparisons in ConsoleScreenshotTest.
To permanently fix this root cause and guarantee deterministic UI rendering
and test stability across schema evolutions, this commit introduces sorted
database loading across our persistence layer.
Specifically, this commit:
- Adds loadAllOfSorted and loadAllOfSortedStream to TransactionManager,
JpaTransactionManagerImpl, and DelegatingReplicaJpaTransactionManager.
- Enforces strict allowlist regex validation (^[a-zA-Z0-9_.]+$) on property
names in JpaTransactionManagerImpl before appending dynamic ORDER BY
clauses, preventing JPQL/SQL injection since bind parameters cannot be
used for schema identifiers.
- Adds Registrar.loadAllSorted and updates RegistrarsAction to explicitly
sort registrars alphabetically by registrarName and registrarId.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to
reflect the newly enforced deterministic alphabetical registrar ordering.
- Adds comprehensive unit test coverage in JpaTransactionManagerImplTest
verifying exact sorted entity retrieval across multiple fields and regex
rejection of invalid field names.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/531889856
This hasn't been an issue, but it seems like a good idea. Note that I'm
allowing prices of 0 here just in case there's some shenanigans at some
point in time where we want a domain to technically be premium without
having a cost.
Implement programmatic verification of Public Access Prevention (PAP) on all GCS buckets before performing write operations via GcsUtils.
- Add verifyPublicAccessPrevention(String bucketName) to GcsUtils.java, querying the bucket's IAM configuration and asserting that PAP is ENFORCED.
- Call verifyPublicAccessPrevention in all output stream and byte creation write paths in GcsUtils.
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.
Specifically, it:
- Adds Flyway migration
V224__add_registrar_expiry_access_period_enabled.sql, which adds the
expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
consult db/README.md, follow the mandatory Two-PR deployment split when
altering database schemas, and run both :db:test and
:core:sqlIntegrationTest whenever database schema or ORM mappings are
touched.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
the new default database ordering after adding the column. (Note: A
separate pull request, PR #3136 / http://go/r3pr/3136, permanently fixes
this underlying sort non-determinism across our persistence layer by
enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
This is a small edge case where a cancellation for year 2 could cancel
out events in year 1 if we re-ran billing for year 1 for some reason. I
don't think we would ever re-run one year later so this is kind of moot,
but it's probably still a good thing to do.
Configure a throwing XMLResolver on XMLInputFactory in XmlTransformer
to act as an additional defense-in-depth security layer against XML
External Entity (XXE) and entity expansion/resolution attacks.
This prevents resolution of external and internal entities in StAX
parsing pipelines, ensuring that malformed payloads containing entity
definitions are safely blocked and rejected.
Also add a unit test to EppXmlSanitizerTest verifying that EPP
messages containing DTD and external entities fail parsing and fall
back safely to their Base64 representation.
TAG=agy
CONV=610c2358-a99f-4605-94cd-ff0d4ee08176
BUG= http://b/529387728
Add validation to RefreshDnsForAllDomainsAction to ensure the tlds parameter is not empty when invoked. Previously, invoking the action without specifying tlds would result in a no-op that logged an empty list and exited without error. This change makes missing tlds an explicit error, throwing an IllegalArgumentException so users running the command via nomulus curl receive clear feedback that the parameter is required.
In commit 17b851de42 (#2979), <element name="contact"> and related types were incorrectly removed from core/src/main/java/google/registry/xml/xsd/domain.xsd and rde-domain.xsd. Because those XSD schemas are published standards (RFC 5731 and RFC 5831), altering them caused JAXB unmarshaling to reject historical database records (HistoryEntry) containing <domain:contact> elements during OT&E status checks.
Restore <element name="contact" minOccurs="0" maxOccurs="unbounded"/> to domain.xsd and rde-domain.xsd so all XSD schemas conform strictly to published EPP RFCs and historical contact elements unmarshal cleanly without throwing cvc-complex-type.2.4.a schema validation errors.
Remove deprecated <domain:contact> elements from prober creation payload (prober/.../create.xml) to match current flow rules where contact associations are forbidden.
Remove trailing slash from ote-status route parameter in RegistrarDetailsComponent to prevent double-slash (/ote-status//<id>) URL generation when checking OT&E status.
BUG= http://b/529391412
Domains in the redemption grace period can still be restored, so hosts
referencing them cannot be deleted. Fortunately this is already the
case. This just adds a test.
Reject loopback, link-local, site-local, wildcard, and multicast IP
addresses during host creation and update flows.
Glue records (A/AAAA records published in the parent zone for subordinate
name servers) must point to globally routable, public IP addresses to
ensure that recursive DNS resolvers on the public internet can reach the
authoritative name servers.
Using non-public or non-routable IP addresses in glue records is invalid
for the following reasons:
- Loopback (127.0.0.1, ::1) and Any-Local (0.0.0.0, ::) addresses point
back to the client or are unspecified, causing resolvers to query
themselves and fail.
- Private/Site-Local (e.g., 10.0.0.0/8, 192.168.0.0/16) and Link-Local
(169.254.0.0/16) addresses are not routable on the public internet,
rendering the delegated domain completely unreachable to external clients.
- Multicast addresses are designed for one-to-many delivery and cannot
be used for standard unicast DNS queries to a specific name server.
Rename LoopbackIpNotValidForHostException to IpAddressNotRoutableException
to reflect the broader set of forbidden non-routable IP addresses.
This commit addresses the following security vulnerabilities identified in the recent audit of the Console App and Backend APIs:
1. Angular XSS: Removed unsafe [innerHTML] bindings across all console-webapp templates (Contact, Registrars, Registrar Details, Users List) in favor of standard Angular interpolation.
2. Broken Access Control (IDOR): PasswordResetRequestAction and PasswordResetVerifyAction now explicitly verify that the target user's email belongs to the authorized registrarId.
3. Missing Permission Check: ConsoleEppPasswordAction now explicitly checks for CONFIGURE_EPP_CONNECTION permission before updating the EPP password.
4. Denial of Service (DoS): ConsoleBulkDomainAction now strictly limits the size of bulk domain lists (configurable, default 500) to prevent thread exhaustion.
5. Denial of Service (OOM): ConsoleHistoryDataAction now uses .setMaxResults() (configurable, default 500) on JPA native queries to prevent eager loading of the entire database into memory.
Makes the history query limit and bulk domain action limit configurable via RegistryConfig, allowing smaller limits to be used in tests to avoid heavy resource persistence.
Also removes an outdated Joda-Time migration reference from GEMINI.md.
Refactor RdapDomainSearchActionTest to dynamically resolve all domain
and host Repository IDs (ROIDs) instead of asserting on hardcoded,
sequence-generated strings (like "2E-LOL" or "6-LOL").
When tests are executed in parallel (as is common in CI environments like
Kokoro), multiple test threads concurrently reset and allocate from the
shared database sequence 'project_wide_unique_id_seq'. This interleaves
ID allocations non-deterministically, causing any test asserting on
exact, hardcoded sequence values to flake.
To fix this, createManyDomainsAndHosts was updated to return the list of
persisted domains, allowing tests to dynamically resolve their ROIDs.
All other test cases were refactored to dynamically fetch the ROIDs of
pre-created domains and hosts (stored in fields or in hostNameToHostMap,
using punycode keys for IDN hosts) for their JSON assertions, rendering
the entire suite robust against sequence shifts.
we shouldn't have to rebuild it each time we get a request to a
different service or really ever at all -- we might get a tiny bit of
cache benefit here
Summarize all documentation updates across the repository to align with modern GKE, Cloud SQL Proxy v2, standard EPP fee v1.0, and Postgres database environments.
Key Updates:
- Prerequisites: Bump Java requirement to Java 25.
- Architecture & Scaling: Document GKE workloads, Cloud Tasks queues, and scheduled tasks. Replace App Engine references with GKE deployment restart commands (kubectl rollout restart).
- Configuration: Update Cloud SQL Proxy instructions to v2, fix keyring verification commands, and document IAP configuration.
- Escrow (RDE/BRDA): Fix manual generation and download procedures to match the Dataflow job ID folder structure, and correct deposit encryption/verification command parameters.
- Monitoring: Correct metric names and expand the documented metrics list with caching, locking, and reserved list metrics.
- Fixes: Standardize lists formatting across markdown files, fix broken webdriver links, and resolve various typos.
- Cleanup: Remove leftover cloud scheduler configurations for the deleted wipeOutContactHistoryPii task, and update ICANN reporting documentation to reflect open-sourced DNS query coordinator.
TAG=agy
CONV=88271e71-e272-40e0-85f8-a075a423b7c2
Normal comparison exits out once a difference is found so theoretically
a time variance can leak information. This isn't really a huge deal but
is probably still worth doing.
Introduced overloaded helper methods `persistFeatureFlag` in `DatabaseHelper` to simplify the creation and persistence of `FeatureFlag` entities in test setups.
1. `persistFeatureFlag(FeatureName, FeatureStatus)`: Persists a feature flag with a single status starting from the Unix Epoch (`START_INSTANT`).
2. `persistFeatureFlag(FeatureName, FeatureStatus, Instant, FeatureStatus)`: Persists a feature flag with an initial status at `START_INSTANT` and a subsequent transition at a specified time.
Refactored 11 occurrences of manual 1-transition flag creation and 5 occurrences of 2-transition flag creation across the test suite to use these new helpers, significantly reducing boilerplate and improving test readability.
TAG=agy
CONV=583b8a23-9fe5-476d-ac35-aeba7b218eb0
This is similar to PR #3069 but for the algorithms themselves rather
than the digest data. This forbids algorithms, that, according to RFC
9904, should not be used.
This commit reverts changes from 5599a0eb3d and most of 5286b1a0dc (PR #3068) that stripped essential dependencies (buildConsoleForAll, buildNomulusImage, buildToolImage, fragileTest) from the default './gradlew build' target, which broke downstream deployment pipelines. It restores the default build to correctly generate all necessary production artifacts and Docker images.
It introduces a new 'fastBuild' target designed explicitly for local developers and CI checks. This lightweight target disables the execution of heavy Docker image builds, Angular compilations, and fragile tests to provide rapid feedback. Sequential execution constraints for parallel Angular builds are maintained to prevent cache corruption.
It updates the ':core:generateSqlSchema' task to execute using the 'unittest' environment instead of 'alpha'. The 'alpha' configuration is a private, internal environment config that is not distributed in the open-source repository, which caused the task to fail for public contributors. By switching to 'unittest', the generator can successfully run using the public test configuration. With this fixed, it also includes the newly generated 'db-schema.sql.generated' file, which now correctly tracks the 'FORBID_INSECURE_ALGORITHMS_RFC_9904' feature flag that was recently added.
Finally, it implements a split-runner execution strategy for the 'sqlIntegrationTest' task to permanently resolve 'failed to discover tests' and 'NoSuchMethodError' exceptions on Kokoro. Because Kokoro tests cross-version compatibility against both legacy deployed artifacts (compiled with JUnit 4 @RunWith wrappers) and modern artifacts (compiled with JUnit 5 @Suite annotations), we cannot statically configure a single test runner. We now dynamically run both the legacy 'useJUnit()' and modern 'useJUnitPlatform()' runners sequentially with 'failOnNoDiscoveredTests' disabled, allowing the appropriate engine to discover and execute the suite without causing classpath collisions.
Ben McIlwain
Weimin Yu
gbrodman