This commit implements the second stage (Java ORM mappings and EPP flow
enforcement) of the Expiry Access Period (XAP) launch and opt-in mechanism,
completing the Two-PR deployment split mandated by db/README.md after the
Stage 1 database migrations (PR #3134) are live.
During XAP, a TLD charges a fee for domain registration during a timed period
after deletion. To prevent accidental charges, registrars must explicitly
opt in to participate in XAP registrations.
Specifically, this commit:
- Adds expiryAccessPeriodTransitions (a TimedTransitionProperty of
ExpiryAccessPeriodMode) to Tld.java and ExpiryAccessPeriodModeTransitionUserType
for mapping XAP mode schedules to PostgreSQL hstore columns.
- Adds expiryAccessPeriodEnabled to Registrar.java with getter, builder
setter, and JSON map serialization, and regenerates db-schema.sql.generated.
- Enforces registrar opt-in in DomainCheckFlow: when an unallocated domain is
in XAP and the querying registrar has not opted in, domain:check returns
avail="0" with reason "Reserved".
- Enforces registrar opt-in in DomainCreateFlow: when attempting to register
a domain in XAP without registrar opt-in, throws DomainReservedException
(EPP error 2304 "Object status prohibits operation").
- Enforces explicit fee acknowledgment during XAP domain creation when the XAP
fee is non-zero, throwing FeesRequiredDuringExpiryAccessPeriodException in
DomainFlowUtils if omitted.
- Creates a one-time BillingEvent with Reason.FEE_EXPIRY_ACCESS when a domain
is created during XAP with a non-zero fee, and adds getXapCost() to
FeesAndCredits.
- Updates all test TLD YAML configurations and adds comprehensive unit and
integration tests across DomainCheckFlowTest, DomainCreateFlowTest,
DomainPricingLogicTest, and RegistrarTest.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
Lazy-initialize the terminal and LineReader in ShellCommand to prevent JLine from eagerly probing terminal capabilities during JCommander command-line startup.
During CLI startup, JCommander instantiates every command (including ShellCommand) to build the CLI's command map. Eager instantiation of the LineReader inside the ShellCommand constructor causes JLine to query the terminal cursor via Device Status Report (DSR) escape sequences. Standard commands (e.g. list_tlds) do not consume stdin, leaving these probing responses (such as ^[[71;1R and 1;1R) in the buffer to be leaked as garbage characters to standard output.
This change defers terminal and LineReader initialization until the shell's run() method is actually executed, while preserving the original constructor for unit tests.
BUG=http://b/534855218
An AI coding agent (Jetski) repeatedly violated user instructions by executing
unsolicited remote updates (git push --force-with-lease origin ...) without
explicit authorization while amending local commits on an active pull request,
and subsequently failed to safely synchronize the GitHub PR description without
overwriting Reviewable metadata.
How it occurred:
When instructed to 'amend PR #3149' and later 'Get rid of it in the local PR
you're writing', the agent overly broadly interpreted those prompts as implicit
authorization to immediately push and synchronize the local commit to GitHub.
Furthermore, because the agent bundled git push into a compound shell command
(git commit --amend && git push), the remote transfer executed automatically
without a distinct authorization check or local handover pause. Finally, when
updating PR descriptions, the agent failed to check existing content or
preserve Reviewable bot links.
How this commit structurally prevents recurrence:
1. Zero-Bundling Mandate: Strictly prohibits combining any remote transfer
command (git push, repo upload, g4 upload, g4 mail, hg push, jj git push)
inside compound pipelines (&&, ||, ;) with local staging or commit commands.
Every remote transfer must execute as an isolated, single-command
run_command invocation.
2. Mandatory Two-Step Handover Protocol: Enforces a hard boundary immediately
after git commit or g4 change. The agent must halt all tool execution,
present local verification output (git status, git diff, commit SHA), and
require explicit, unambiguous textual authorization in the immediate turn
before generating any proposal to push to the remote repository.
3. Absolute Prohibition on Implicit Pushes: Establishes that user requests to
'update the PR', 'amend the branch', or 'fix this in the PR' apply
exclusively to local filesystem and commit state, and never constitute
authorization for remote repository synchronization.
4. Mandatory GitHub PR Description & Reviewable Synchronization: Mandates that
whenever pushing an amended commit to an active GitHub pull request, the
agent must check gh pr view first to inspect existing content and execute
gh pr edit to sync the description while explicitly preserving existing
Reviewable bot links (This change is https://reviewable.io/reviews/...).
this reduces prod bundle size a bit and cleans things up
by including testcontainers in the nonprod deps of core, we can still
run things like the schema-generating commands
When checking in a brand new premium list (production/*.txt) or reserved list (reserved/production/*.txt), presubmit testing commands in gradle-runner.sh fall back from update_premium_list / update_reserved_list (which require existing database entities) to create_premium_list / create_reserved_list.
However, because --dry_run (-d) and --build_environment flags were previously only defined in UpdatePremiumListCommand and UpdateReservedListCommand, passing them during non-interactive presubmit creation validation resulted in ParameterException (Was passed main parameter '-d' but no main parameter was defined...) or NullPointerException when ConfirmingCommand attempted to prompt on a null system console.
This change moves --dry_run and --build_environment parameters and dontRunCommand() behavior up into CreateOrUpdatePremiumListCommand and CreateOrUpdateReservedListCommand, enabling safe dry-run presubmit validation for newly created lists in CI environments.
BUG= http://b/534511633, http://b/529397845
Implement a hybrid Content Security Policy (CSP) for the Registrar Console
to protect against XSS
- The CspFilter injects the proper headers on the Java backend endpoints
- Uinsg Jetty's HeaderFilter to inject the header for statically-served
frontend assets
We need to add the ee10-servlets.ini file for Jetty to have acess to the
HeaderFilter class
It's not perfect because we're running outside of a transaction, but we
should make a best-effort attempt to project domains/hosts to the
current time.
- nicer error in Registrar builder instead of an NPE
- nicer error if hosts were deleted in GenerateZoneFilesAction
- extra permissions checks in ConsoleUsersAction
- using "replace" (text) instead of "replaceAll" (regex) in
BulkDomainAction
Previously, if there were no XML bytes somehow, it'd increment a bunch
of counts for requirements that shouldn't be incremented. We should only
increment these values if we have the data to do so.
Note: this doesn't happen and shouldn't happen, so there's no issues
currently occurring, but this code is more correct anyway.
During our implementation of Expiry Access Period (XAP), adding a field to the
Registrar entity altered database heap scan order when loadAllOf() was
called without an explicit ORDER BY clause. This caused non-deterministic
row shifting in the Angular console registrar table (/console/registrars),
breaking golden image comparisons in ConsoleScreenshotTest.
To permanently fix this root cause and guarantee deterministic UI rendering
and test stability across schema evolutions, this commit introduces sorted
database loading across our persistence layer.
Specifically, this commit:
- Adds loadAllOfSorted and loadAllOfSortedStream to TransactionManager,
JpaTransactionManagerImpl, and DelegatingReplicaJpaTransactionManager.
- Enforces strict allowlist regex validation (^[a-zA-Z0-9_.]+$) on property
names in JpaTransactionManagerImpl before appending dynamic ORDER BY
clauses, preventing JPQL/SQL injection since bind parameters cannot be
used for schema identifiers.
- Adds Registrar.loadAllSorted and updates RegistrarsAction to explicitly
sort registrars alphabetically by registrarName and registrarId.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to
reflect the newly enforced deterministic alphabetical registrar ordering.
- Adds comprehensive unit test coverage in JpaTransactionManagerImplTest
verifying exact sorted entity retrieval across multiple fields and regex
rejection of invalid field names.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/531889856
This hasn't been an issue, but it seems like a good idea. Note that I'm
allowing prices of 0 here just in case there's some shenanigans at some
point in time where we want a domain to technically be premium without
having a cost.
Implement programmatic verification of Public Access Prevention (PAP) on all GCS buckets before performing write operations via GcsUtils.
- Add verifyPublicAccessPrevention(String bucketName) to GcsUtils.java, querying the bucket's IAM configuration and asserting that PAP is ENFORCED.
- Call verifyPublicAccessPrevention in all output stream and byte creation write paths in GcsUtils.
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.
Specifically, it:
- Adds Flyway migration
V224__add_registrar_expiry_access_period_enabled.sql, which adds the
expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
consult db/README.md, follow the mandatory Two-PR deployment split when
altering database schemas, and run both :db:test and
:core:sqlIntegrationTest whenever database schema or ORM mappings are
touched.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
the new default database ordering after adding the column. (Note: A
separate pull request, PR #3136 / http://go/r3pr/3136, permanently fixes
this underlying sort non-determinism across our persistence layer by
enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
This is a small edge case where a cancellation for year 2 could cancel
out events in year 1 if we re-ran billing for year 1 for some reason. I
don't think we would ever re-run one year later so this is kind of moot,
but it's probably still a good thing to do.
Configure a throwing XMLResolver on XMLInputFactory in XmlTransformer
to act as an additional defense-in-depth security layer against XML
External Entity (XXE) and entity expansion/resolution attacks.
This prevents resolution of external and internal entities in StAX
parsing pipelines, ensuring that malformed payloads containing entity
definitions are safely blocked and rejected.
Also add a unit test to EppXmlSanitizerTest verifying that EPP
messages containing DTD and external entities fail parsing and fall
back safely to their Base64 representation.
TAG=agy
CONV=610c2358-a99f-4605-94cd-ff0d4ee08176
BUG= http://b/529387728
Add validation to RefreshDnsForAllDomainsAction to ensure the tlds parameter is not empty when invoked. Previously, invoking the action without specifying tlds would result in a no-op that logged an empty list and exited without error. This change makes missing tlds an explicit error, throwing an IllegalArgumentException so users running the command via nomulus curl receive clear feedback that the parameter is required.
In commit 17b851de42 (#2979), <element name="contact"> and related types were incorrectly removed from core/src/main/java/google/registry/xml/xsd/domain.xsd and rde-domain.xsd. Because those XSD schemas are published standards (RFC 5731 and RFC 5831), altering them caused JAXB unmarshaling to reject historical database records (HistoryEntry) containing <domain:contact> elements during OT&E status checks.
Restore <element name="contact" minOccurs="0" maxOccurs="unbounded"/> to domain.xsd and rde-domain.xsd so all XSD schemas conform strictly to published EPP RFCs and historical contact elements unmarshal cleanly without throwing cvc-complex-type.2.4.a schema validation errors.
Remove deprecated <domain:contact> elements from prober creation payload (prober/.../create.xml) to match current flow rules where contact associations are forbidden.
Remove trailing slash from ote-status route parameter in RegistrarDetailsComponent to prevent double-slash (/ote-status//<id>) URL generation when checking OT&E status.
BUG= http://b/529391412
Domains in the redemption grace period can still be restored, so hosts
referencing them cannot be deleted. Fortunately this is already the
case. This just adds a test.
Reject loopback, link-local, site-local, wildcard, and multicast IP
addresses during host creation and update flows.
Glue records (A/AAAA records published in the parent zone for subordinate
name servers) must point to globally routable, public IP addresses to
ensure that recursive DNS resolvers on the public internet can reach the
authoritative name servers.
Using non-public or non-routable IP addresses in glue records is invalid
for the following reasons:
- Loopback (127.0.0.1, ::1) and Any-Local (0.0.0.0, ::) addresses point
back to the client or are unspecified, causing resolvers to query
themselves and fail.
- Private/Site-Local (e.g., 10.0.0.0/8, 192.168.0.0/16) and Link-Local
(169.254.0.0/16) addresses are not routable on the public internet,
rendering the delegated domain completely unreachable to external clients.
- Multicast addresses are designed for one-to-many delivery and cannot
be used for standard unicast DNS queries to a specific name server.
Rename LoopbackIpNotValidForHostException to IpAddressNotRoutableException
to reflect the broader set of forbidden non-routable IP addresses.
This commit addresses the following security vulnerabilities identified in the recent audit of the Console App and Backend APIs:
1. Angular XSS: Removed unsafe [innerHTML] bindings across all console-webapp templates (Contact, Registrars, Registrar Details, Users List) in favor of standard Angular interpolation.
2. Broken Access Control (IDOR): PasswordResetRequestAction and PasswordResetVerifyAction now explicitly verify that the target user's email belongs to the authorized registrarId.
3. Missing Permission Check: ConsoleEppPasswordAction now explicitly checks for CONFIGURE_EPP_CONNECTION permission before updating the EPP password.
4. Denial of Service (DoS): ConsoleBulkDomainAction now strictly limits the size of bulk domain lists (configurable, default 500) to prevent thread exhaustion.
5. Denial of Service (OOM): ConsoleHistoryDataAction now uses .setMaxResults() (configurable, default 500) on JPA native queries to prevent eager loading of the entire database into memory.
Makes the history query limit and bulk domain action limit configurable via RegistryConfig, allowing smaller limits to be used in tests to avoid heavy resource persistence.
Also removes an outdated Joda-Time migration reference from GEMINI.md.
This commit improves the database documentation in db/README.md by adding comprehensive guidelines for refactoring column types and managing two-PR schema deployments.
Key additions:
- Added a section on the "Expand and Contract" pattern for refactoring column types, explaining when it is safe to drop columns immediately vs. when a three-step release process is required.
- Added a section on writing safe NOT NULL migrations for timed transition properties, explaining the "Temporary Database Default" pattern to maintain backward compatibility with running servers during Two-PR deployments, and demonstrating the required explicit PostgreSQL `::hstore` casting syntax.
- Added a step-by-step "Recommended Git Workflow" section to help developers cleanly split their database and Java changes into chained PRs using Git.
TAG=agy
CONV=88271e71-e272-40e0-85f8-a075a423b7c2
This commit implements the database schema changes for the Expiry Access Period (XAP) launch configuration on TLDs. It represents the first step of a Two-PR deployment strategy, deploying the database schema changes in advance of the server logic.
Specifically, it replaces the `expiry_access_period_enabled` boolean column (originally introduced in PR #2804) with a new `expiry_access_period_transitions` hstore column.
Why we are making this change:
A basic boolean flag only allows an immediate, manual on/off toggle. To launch XAP on a TLD, registry operators would have to manually flip the flag at the exact launch time, which is operationally fragile and cannot be planned in advance. Refactoring this to an hstore-backed timed transition map (mapping Instant to ExpiryAccessPeriodMode) allows operators to schedule the XAP launch in advance via TLD YAML configurations. The registry will automatically transition the TLD to the ENABLED mode at the scheduled timestamp, aligning with how other scheduled TLD changes (like TLD states and EAP fee schedules) are managed.
Since the original boolean column was never mapped in Java (PR #2804 only added the database column), it is completely safe to drop it immediately in this migration.
To ensure backward compatibility with running servers (which are still executing the old Java code during the deployment transition), the new column is added as `NOT NULL` with a temporary `DEFAULT` constraint. This prevents constraint violations on inserts from old servers. A TODO has been left in the SQL migration to drop this default in a subsequent schema release once the Java changes have been deployed.
TAG=agy
CONV=88271e71-e272-40e0-85f8-a075a423b7c2
Refactor RdapDomainSearchActionTest to dynamically resolve all domain
and host Repository IDs (ROIDs) instead of asserting on hardcoded,
sequence-generated strings (like "2E-LOL" or "6-LOL").
When tests are executed in parallel (as is common in CI environments like
Kokoro), multiple test threads concurrently reset and allocate from the
shared database sequence 'project_wide_unique_id_seq'. This interleaves
ID allocations non-deterministically, causing any test asserting on
exact, hardcoded sequence values to flake.
To fix this, createManyDomainsAndHosts was updated to return the list of
persisted domains, allowing tests to dynamically resolve their ROIDs.
All other test cases were refactored to dynamically fetch the ROIDs of
pre-created domains and hosts (stored in fields or in hostNameToHostMap,
using punycode keys for IDN hosts) for their JSON assertions, rendering
the entire suite robust against sequence shifts.
we shouldn't have to rebuild it each time we get a request to a
different service or really ever at all -- we might get a tiny bit of
cache benefit here
Ben McIlwain
Weimin Yu
gbrodman