In FlowReporter, we extract TLDs from EPP domain commands and log them under 'tld' and 'tlds' metadata fields to generate ICANN activity reports. Previously, invalid or extremely long TLD names (such as email addresses or long domain labels in non-validated XML payloads) could break downstream log parsing.
To prevent this issue, this change does the following:
1. Java Sanitization:
Introduces `toLogSafeLabel(...)` in `DomainFlowUtils`, which converts ASCII to lowercase, replaces any character outside of `[a-z0-9.-]` with `-`, and limits the length to 63 characters (appending '...' if truncated). FlowReporter now uses this method on guessed TLDs before logging them.
2. SQL Modernization:
Upgrades `epp_metrics.sql` and `epp_metrics_test.sql` to use BigQuery's native `JSON_EXTRACT_STRING_ARRAY(json, '$.tlds')` instead of the legacy `SPLIT(REGEXP_EXTRACT(JSON_EXTRACT(...)))` workaround. Because the native function extracts clean string elements, it removes the need to strip quotation marks with additional regexes and prevents parsing failures on commas, spaces, or nested structures.
SQL change tested in BigQuery.
BUG=http://b/535230985
We allow admin locks to overwrite already-applied locks. In the case
where that happens in between an unlock request and an unlock
completion, we shouldn't allow the unlock completion to go through.
1. make it a POST instead of a GET
2. pass the user into DomainLockUtils so we can check permissions to
make sure they have permissions on the registrar
3. update all the tests
This commit implements the second stage (Java ORM mappings and EPP flow
enforcement) of the Expiry Access Period (XAP) launch and opt-in mechanism,
completing the Two-PR deployment split mandated by db/README.md after the
Stage 1 database migrations (PR #3134) are live.
During XAP, a TLD charges a fee for domain registration during a timed period
after deletion. To prevent accidental charges, registrars must explicitly
opt in to participate in XAP registrations.
Specifically, this commit:
- Adds expiryAccessPeriodTransitions (a TimedTransitionProperty of
ExpiryAccessPeriodMode) to Tld.java and ExpiryAccessPeriodModeTransitionUserType
for mapping XAP mode schedules to PostgreSQL hstore columns.
- Adds expiryAccessPeriodEnabled to Registrar.java with getter, builder
setter, and JSON map serialization, and regenerates db-schema.sql.generated.
- Enforces registrar opt-in in DomainCheckFlow: when an unallocated domain is
in XAP and the querying registrar has not opted in, domain:check returns
avail="0" with reason "Reserved".
- Enforces registrar opt-in in DomainCreateFlow: when attempting to register
a domain in XAP without registrar opt-in, throws DomainReservedException
(EPP error 2304 "Object status prohibits operation").
- Enforces explicit fee acknowledgment during XAP domain creation when the XAP
fee is non-zero, throwing FeesRequiredDuringExpiryAccessPeriodException in
DomainFlowUtils if omitted.
- Creates a one-time BillingEvent with Reason.FEE_EXPIRY_ACCESS when a domain
is created during XAP with a non-zero fee, and adds getXapCost() to
FeesAndCredits.
- Updates all test TLD YAML configurations and adds comprehensive unit and
integration tests across DomainCheckFlowTest, DomainCreateFlowTest,
DomainPricingLogicTest, and RegistrarTest.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
Lazy-initialize the terminal and LineReader in ShellCommand to prevent JLine from eagerly probing terminal capabilities during JCommander command-line startup.
During CLI startup, JCommander instantiates every command (including ShellCommand) to build the CLI's command map. Eager instantiation of the LineReader inside the ShellCommand constructor causes JLine to query the terminal cursor via Device Status Report (DSR) escape sequences. Standard commands (e.g. list_tlds) do not consume stdin, leaving these probing responses (such as ^[[71;1R and 1;1R) in the buffer to be leaked as garbage characters to standard output.
This change defers terminal and LineReader initialization until the shell's run() method is actually executed, while preserving the original constructor for unit tests.
BUG=http://b/534855218
An AI coding agent (Jetski) repeatedly violated user instructions by executing
unsolicited remote updates (git push --force-with-lease origin ...) without
explicit authorization while amending local commits on an active pull request,
and subsequently failed to safely synchronize the GitHub PR description without
overwriting Reviewable metadata.
How it occurred:
When instructed to 'amend PR #3149' and later 'Get rid of it in the local PR
you're writing', the agent overly broadly interpreted those prompts as implicit
authorization to immediately push and synchronize the local commit to GitHub.
Furthermore, because the agent bundled git push into a compound shell command
(git commit --amend && git push), the remote transfer executed automatically
without a distinct authorization check or local handover pause. Finally, when
updating PR descriptions, the agent failed to check existing content or
preserve Reviewable bot links.
How this commit structurally prevents recurrence:
1. Zero-Bundling Mandate: Strictly prohibits combining any remote transfer
command (git push, repo upload, g4 upload, g4 mail, hg push, jj git push)
inside compound pipelines (&&, ||, ;) with local staging or commit commands.
Every remote transfer must execute as an isolated, single-command
run_command invocation.
2. Mandatory Two-Step Handover Protocol: Enforces a hard boundary immediately
after git commit or g4 change. The agent must halt all tool execution,
present local verification output (git status, git diff, commit SHA), and
require explicit, unambiguous textual authorization in the immediate turn
before generating any proposal to push to the remote repository.
3. Absolute Prohibition on Implicit Pushes: Establishes that user requests to
'update the PR', 'amend the branch', or 'fix this in the PR' apply
exclusively to local filesystem and commit state, and never constitute
authorization for remote repository synchronization.
4. Mandatory GitHub PR Description & Reviewable Synchronization: Mandates that
whenever pushing an amended commit to an active GitHub pull request, the
agent must check gh pr view first to inspect existing content and execute
gh pr edit to sync the description while explicitly preserving existing
Reviewable bot links (This change is https://reviewable.io/reviews/...).
this reduces prod bundle size a bit and cleans things up
by including testcontainers in the nonprod deps of core, we can still
run things like the schema-generating commands
When checking in a brand new premium list (production/*.txt) or reserved list (reserved/production/*.txt), presubmit testing commands in gradle-runner.sh fall back from update_premium_list / update_reserved_list (which require existing database entities) to create_premium_list / create_reserved_list.
However, because --dry_run (-d) and --build_environment flags were previously only defined in UpdatePremiumListCommand and UpdateReservedListCommand, passing them during non-interactive presubmit creation validation resulted in ParameterException (Was passed main parameter '-d' but no main parameter was defined...) or NullPointerException when ConfirmingCommand attempted to prompt on a null system console.
This change moves --dry_run and --build_environment parameters and dontRunCommand() behavior up into CreateOrUpdatePremiumListCommand and CreateOrUpdateReservedListCommand, enabling safe dry-run presubmit validation for newly created lists in CI environments.
BUG= http://b/534511633, http://b/529397845
Implement a hybrid Content Security Policy (CSP) for the Registrar Console
to protect against XSS
- The CspFilter injects the proper headers on the Java backend endpoints
- Uinsg Jetty's HeaderFilter to inject the header for statically-served
frontend assets
We need to add the ee10-servlets.ini file for Jetty to have acess to the
HeaderFilter class
It's not perfect because we're running outside of a transaction, but we
should make a best-effort attempt to project domains/hosts to the
current time.
- nicer error in Registrar builder instead of an NPE
- nicer error if hosts were deleted in GenerateZoneFilesAction
- extra permissions checks in ConsoleUsersAction
- using "replace" (text) instead of "replaceAll" (regex) in
BulkDomainAction
Previously, if there were no XML bytes somehow, it'd increment a bunch
of counts for requirements that shouldn't be incremented. We should only
increment these values if we have the data to do so.
Note: this doesn't happen and shouldn't happen, so there's no issues
currently occurring, but this code is more correct anyway.
During our implementation of Expiry Access Period (XAP), adding a field to the
Registrar entity altered database heap scan order when loadAllOf() was
called without an explicit ORDER BY clause. This caused non-deterministic
row shifting in the Angular console registrar table (/console/registrars),
breaking golden image comparisons in ConsoleScreenshotTest.
To permanently fix this root cause and guarantee deterministic UI rendering
and test stability across schema evolutions, this commit introduces sorted
database loading across our persistence layer.
Specifically, this commit:
- Adds loadAllOfSorted and loadAllOfSortedStream to TransactionManager,
JpaTransactionManagerImpl, and DelegatingReplicaJpaTransactionManager.
- Enforces strict allowlist regex validation (^[a-zA-Z0-9_.]+$) on property
names in JpaTransactionManagerImpl before appending dynamic ORDER BY
clauses, preventing JPQL/SQL injection since bind parameters cannot be
used for schema identifiers.
- Adds Registrar.loadAllSorted and updates RegistrarsAction to explicitly
sort registrars alphabetically by registrarName and registrarId.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to
reflect the newly enforced deterministic alphabetical registrar ordering.
- Adds comprehensive unit test coverage in JpaTransactionManagerImplTest
verifying exact sorted entity retrieval across multiple fields and regex
rejection of invalid field names.
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/531889856
This hasn't been an issue, but it seems like a good idea. Note that I'm
allowing prices of 0 here just in case there's some shenanigans at some
point in time where we want a domain to technically be premium without
having a cost.
Implement programmatic verification of Public Access Prevention (PAP) on all GCS buckets before performing write operations via GcsUtils.
- Add verifyPublicAccessPrevention(String bucketName) to GcsUtils.java, querying the bucket's IAM configuration and asserting that PAP is ENFORCED.
- Call verifyPublicAccessPrevention in all output stream and byte creation write paths in GcsUtils.
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.
Specifically, it:
- Adds Flyway migration
V224__add_registrar_expiry_access_period_enabled.sql, which adds the
expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
consult db/README.md, follow the mandatory Two-PR deployment split when
altering database schemas, and run both :db:test and
:core:sqlIntegrationTest whenever database schema or ORM mappings are
touched.
- Updates golden screenshot
ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
the new default database ordering after adding the column. (Note: A
separate pull request, PR #3136 / http://go/r3pr/3136, permanently fixes
this underlying sort non-determinism across our persistence layer by
enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).
TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
This is a small edge case where a cancellation for year 2 could cancel
out events in year 1 if we re-ran billing for year 1 for some reason. I
don't think we would ever re-run one year later so this is kind of moot,
but it's probably still a good thing to do.
Configure a throwing XMLResolver on XMLInputFactory in XmlTransformer
to act as an additional defense-in-depth security layer against XML
External Entity (XXE) and entity expansion/resolution attacks.
This prevents resolution of external and internal entities in StAX
parsing pipelines, ensuring that malformed payloads containing entity
definitions are safely blocked and rejected.
Also add a unit test to EppXmlSanitizerTest verifying that EPP
messages containing DTD and external entities fail parsing and fall
back safely to their Base64 representation.
TAG=agy
CONV=610c2358-a99f-4605-94cd-ff0d4ee08176
BUG= http://b/529387728
Add validation to RefreshDnsForAllDomainsAction to ensure the tlds parameter is not empty when invoked. Previously, invoking the action without specifying tlds would result in a no-op that logged an empty list and exited without error. This change makes missing tlds an explicit error, throwing an IllegalArgumentException so users running the command via nomulus curl receive clear feedback that the parameter is required.
In commit 17b851de42 (#2979), <element name="contact"> and related types were incorrectly removed from core/src/main/java/google/registry/xml/xsd/domain.xsd and rde-domain.xsd. Because those XSD schemas are published standards (RFC 5731 and RFC 5831), altering them caused JAXB unmarshaling to reject historical database records (HistoryEntry) containing <domain:contact> elements during OT&E status checks.
Restore <element name="contact" minOccurs="0" maxOccurs="unbounded"/> to domain.xsd and rde-domain.xsd so all XSD schemas conform strictly to published EPP RFCs and historical contact elements unmarshal cleanly without throwing cvc-complex-type.2.4.a schema validation errors.
Remove deprecated <domain:contact> elements from prober creation payload (prober/.../create.xml) to match current flow rules where contact associations are forbidden.
Remove trailing slash from ote-status route parameter in RegistrarDetailsComponent to prevent double-slash (/ote-status//<id>) URL generation when checking OT&E status.
BUG= http://b/529391412
Domains in the redemption grace period can still be restored, so hosts
referencing them cannot be deleted. Fortunately this is already the
case. This just adds a test.
Weimin Yu
gbrodman
Ben McIlwain