This is more complicated than the bulk load unfortunately.
Previously, for update/put or their plural equivalents, we'd have to run
one or two SELECT statements per entity to check if it exists already
(and for Hibernate to load the object into the context to merge it with
the input).
Both updateAll and putAll now run, per input class, one large SELECT statement
to find out which entities already exist, which also loads the entities
into the persistence context (which is basically a cache). For updateAll,
we require that all of the entities already exist. For putAll, we have a
split path
- for entities that already exist, we just call a normal merge()
- for entities that don't already exist, we run merge() while instructing
Hibernate that the object is transient (i.e. new). This means that we
get the "put" behavior (not modifying the original object) while avoid a
superfluous SELECT statement that'd check again to see if the object exists in
the DB already.
For new objects in putAll, we also use EntityManager::merge (rather than
insert/persist) so that Hibernate creates a deep copy of the entity
instead of mutating the caller's instance in place. To prevent
Hibernate's DefaultMergeEventListener from firing a redundant SELECT
statement when merging a known-new entity with a non-null ID,
TransactionInfo implements Interceptor and overrides
isTransient(Object) to return Boolean.TRUE while merging a
known-transient entity.
We add comments clarifying that "insert" will modify an entity in place
(i.e. UpdateAutoTimestamp or generated IDs) whereas "put" will not. This
has always been the case; the comments are just to clarify.
SafeBrowsing provides an API endpoint that returns, for each threat type
provided, a base64-encoded string that represents the concatenation of
4-byte prefixes of sha256 hashes of harmful domains. That means we can
do the following
- decode the base64 string into a byte[]
- interpret the byte[] as an int[], because each int is four bytes
- for each domain:
- hash the domain and take the hash's first four bytes as an int
- if that int is in the prefix hash array, query SafeBrowsing directly
with that domain name
- if the int is not in the prefix hash array, we know the domain is
not harmful (according to SafeBrowsing at least)
The prefix list provided by SafeBrowsing contains about 2.5 million
entries which corresponds to only about 10 MB of heap memory and only
about 0.1% of the space of all integers. Thus, we can assume there won't
be too many hash collisions.
This will reduce our number of API calls from ~5100 calls (each
containing 490 domains) to 6 API calls + (# actually-harmful domains /
490).
Add --upsert (-u) flag to UpdatePremiumListCommand and
UpdateReservedListCommand so that missing premium and reserved lists are
created automatically when --upsert is specified, while still failing
when --upsert is omitted and the list does not exist.
Allow CurrencyUnit to be nullable in PremiumListUtils.parseToPremiumList()
so that PremiumList.createFromLine() infers the currency unit from the
input lines when currency is unset, and use this in
UpdatePremiumListCommand when creating a new list via --upsert. Also fix
the mixed-currency example in config/files/premium/example.txt.
Also propagate exceptions from CreateOrUpdatePremiumListCommand.execute()
and CreateOrUpdateReservedListCommand.execute() instead of catching and
swallowing Throwable, and validate non-empty and parseable input data in
CreatePremiumListCommand.prompt(). Add comprehensive unit tests across
PremiumListUtilsTest and the create/update list command test suites.
BUG=http://b/569977604
The generated XML / XJC code pretty much never changes but because
everything is lumped together in one compile task, we had to recompile
this every time anything else in core/ changes. It's technically
possible to use a separate source set for the XJC code while having it
remain in core/, but splitting it into a separate subproject gives
better separation and makes the caching / dependency graph clearer.
Pretty much all, except for the caching configuration, of the new
xjc/build.gradle is taken from the previous core/build.gradle.
This is another instance of a grouping of code that is frequently used and
referred to, but infrequently changed. If it's part of the same
compilation unit as core, then it'll need to get recompiled every time
but if we move it out into a different project we won't need to
Also moves some utility classes (required by testing) from util to
common, and removes the unnecessary core:testJar task
In non-production environments (e.g. sandbox), allow TEST TLDs to be
included in the domain drop list export so XAP can be verified with
test TLDs.
Also remove the isInvoicingEnabled filter globally, as the
tld.getExpiryAccessPeriodModeAt(now) check is sufficient.
TAG=agy
CONV=67020217-639f-40a2-9d7c-54e592c91a53
we implemented this in https://github.com/google/nomulus/pull/3178/ as
part of the security changes but as it turns out, we'll still want to
use @registry.google email addresses in sandbox as well
- Mount the test Postgres data directory on an in-memory tmpfs and disable fsync, synchronous_commit, and full_page_writes for ephemeral test containers.
- Override the Nomulus SQL dialect to emit 'deferrable initially deferred' on foreign keys exported by HibernateSchemaExporter. This matches prod.
- Replace per-test TRUNCATES (etc) with DELETEs -- the vast majority of
tests only change a couple tables and DELETEing a few rows is much
faster than having to reallocate all the disk space for each and every
table every time
- Cache the connection to Postgres that we use for resetting data,
instead of having to re-create it each time
- Add a test for child entity foreign key generation
This drastically speeds up tests, cutting out like 30-40% of total test
time (not including compilation)
(we also update a couple queries to be deterministically ordered to
avoid test flakes)
Previously we just looped over and loaded each entity one by one. It's
better to try to limit it to fewer queries (ideally one). Fortunately,
Hibernate supports bulk load of entities with both compound and simple
IDs.
This makes tracking whether or not an entity has been persisted in the
database much simpler, since we can use null as a pure sentinel value to
say "this object has not been persisted yet". This makes things like
"updateAll" much easier and simpler to implement efficiently later
because that will rely on knowing what entities are inserts vs merges
When inserting multiple entities, if sorting isn't enabled, then
Hibernate will just add them in the order received. This is suboptimal
for batch query execution. Hibernate flushes the batch as soon as it
"encounters" a new table, so if we have inserts into tables A, B, A, B
in that order, it'll execute each one separately, rather than batching
(A,A), (B,B).
The downside to enabling ordering is that you need to either have
deferred foreign keys (which we have) or Hibernate-style entity
references, e.g. `@OneToOne Entity entity` rather than `VKey<String>
entityId`.
While any future potential migration to Spanner would require that we
remove the deferred foreign keys, this change is orthogonal to that --
we'll need to change the references to direct Hibernate references
anyway which Hibernate will know how to order properly.
This basically controls the number of child entities that will be
batch-loaded when we bulk-load multiple root entities. It will be useful
for situations like RDE where we load a bunch of domains, because it
means we'll batch-load the child entities like nsHosts and gracePeriods.
This uses the already-existing quota manager to acquire quota, if
necessary, before running a flow. If we can't acquire quota, we throw a
command use EPP exception.
This is a version of the quota manager that essentially special-cases
DomainCreateFlow, assuming that's the only flow that we wish to
throttle. If we wish to throttle more flows in the future, we can
re-architect this a bit.
Redo the exported drop list mechanism by replacing the legacy per-TLD
mode in ExportDomainListsAction with a dedicated, once-daily
ExportDropListAction.
Key changes:
1. Reverted ExportDomainListsAction to unconditionally export
single-column active registered domains and deprecated the
INCLUDE_PENDING_DELETE_DATE_FOR_DOMAINS feature flag.
2. Implemented ExportDropListAction at /_dr/task/exportDropList to query
the read replica for upcoming deletions on all open TLDs (where
invoicing is enabled) and output an alphabetically sorted CSV file
(domain_name,tld,deletion_time) to a designated Google Drive folder.
3. Added domainDropListDriveFolderId configuration setting and provider.
4. Registered the action in RequestComponent, routing.txt, and Cloud
Scheduler tasks for production and sandbox.
5. Added comprehensive test coverage in ExportDropListActionTest and
cleaned up legacy test cases in ExportDomainListsActionTest.
BUG=b/553658111
Expose the --expiry_access_period_enabled CLI flag on registrar mutation
commands and drop temporary database-level default constraints for XAP.
Specifically:
- Expose the --expiry_access_period_enabled parameter in
CreateOrUpdateRegistrarCommand and pass it to
Registrar.Builder.setExpiryAccessPeriodEnabled.
- Add expiryAccessPeriodTransitions with default DISABLED to example.yaml.
- Add unit tests for --expiry_access_period_enabled in
CreateRegistrarCommandTest and UpdateRegistrarCommandTest.
- Add Flyway migrations V229 and V230 to drop the temporary database-level
DEFAULT constraints on Tld.expiry_access_period_transitions and
Registrar.expiry_access_period_enabled per db/README.md.
- Regenerate flyway.txt, nomulus.golden.sql, and ER diagrams.
TAG=agy
BUG=http://b/437398822
We're still getting 429 errors after increasing the number of retries to
four. Using 9 means that the maximum number of seconds we wait will be
255 -- 8 wait intervals starting at 1 second means
1+2+4+8+16+32+64+128=255 seconds.
Also log to see if they give us a useful retry-after header
We don't want to have to pass the entire script to Valkey every single
time. Instead, we compute the SHA-1 hash of the script and refer to it
by hash, significantly reducing the amount of bytes we need to send
to the server. If the script is not loaded in Valkey (NOSCRIPT error),
we reload it and retry.
This separates the logic of "how many tokens should a particular
user/group have" from "manage the quota given whatever limits,
contacting Valkey".
This is in preparation for allowing other types of quota management and
throttling besides just on the EPP server (e.g. domain-create
throttling).
We also convert the expirations from seconds to milliseconds (and use a
Duration so that this is masked from users). It's better to have the API
use a full-fledged Duration object rather than an int, and this allows
for finer control over expiration times.
Note that we'll probably want to use a sliding window in the future
instead of a fixed window, but that's a problem for future us.
Currently, we process (repoId, revisionId) pairs for DomainHistory and
HostHistory individually -- they may be farmed out to worker nodes in
parallel, but each EppResource uses a separate transaction and a separate read,
which doesn't scale well when there are lots of domains/hosts. So as a
result, we should batch them up so we can load (by default) 500 per
transaction at a time.
We don't want to batch-load the domains/hosts at the same time that we
retrieve the most recent history entry for each type -- this would mean
passing relatively large objects across pipeline steps. Instead, we keep
passing the KV<String, Long> and batch retrievals.
This isn't necessarily much faster (due to having to wait on batching)
but there'll be less load on the DB.
Self-scan D.2 number 5
We're getting some 429 (too many requests) responses from the
SafeBrowsing API which is causing the Spec11 pipeline to fail. Currently
the retrier is set to have a backoff starting at 100ms and only have a
couple retries before failing (the latter is already configurable). For
429s, we want to wait significantly longer. Let's start at one second of
waiting and allow four doublings.
Change the rotatePrimaryCert field in CreateOrUpdateRegistrarCommand
from object Boolean to primitive boolean. Since the field is initialized
to default false and used as a presence-based switch without tri-state
semantics, primitive boolean accurately reflects its behavior and avoids
unnecessary object wrapper overhead.
BUG= http://b/537308816
For reserved/premium lists:
Use double-check locking so that subsequent calls to get the entire map
of entries don't need to even check the locking object. This makes
things quicker and removes lock-tracking overhead.
For TMCH CA:
we can just remove the synchronization block entirely. Everything inside
of it is either constants (e.g. ROOT_CERTS) or a Guava loading cache
(CRL_CACHE) which takes care of synchronization for us anyway.
Calculates the delay seconds automatically. This value helps ensure that
all EPP requests are enqueued before the scheduled test start time.
Since queue insertion is much slower than dispatch, this is essential to
maintain a stable QPS rate.
Also parallelizes queue insertion using a thread pool. This reduces the
delay for enqueuing the requests.
BUG=http://b/533414332
This is configured to run every 5 minutes. We need to make sure that the
cache doesn't get too out of date, otherwise we'll be serving stale
data. We'll add an alert that fires if SUCCESS or NOT_CONFIGURED hasn't
happened recently.
Per ICANN's Expired Registration Recovery Policy, all gTLD registries must
offer a Redemption Grace Period (RGP) of 30 days during which deleted
domains may be restored. Registry reservation lists should not block
domain restore commands during the RGP.
This change removes the reserved list check in DomainRestoreRequestFlow,
reverting the behavior originally added in CL 72341125 (July 2014) that
explicitly disallowed restoring reserved domains. Unit tests have been
updated to confirm restoring reserved domains succeeds for standard
registrar accounts.
BUG=b/539548743
TAG=agy
CONV=d5dff534-f924-4bba-a58e-74091dc5f496
This means we don't have to load all users and filter them out later. In
practice this doesn't matter because the user table is relatively small
(a few hundred) but 1. who knows what can happen in the future? 2. this
makes the code analysis tools happier
When executing bulk domain transfers with an explicit list of domain names or
a domain names file, enforcing by losing registrar ID is often unnecessary
and redundant (b/537294004).
This commit makes --losing_registrar_id an optional command-line parameter
and updates BulkDomainTransferAction and BatchModule to handle an optional
losing sponsor ID. Existing behavior is preserved when the parameter is
explicitly supplied.
BUG= http://b/537294004
This change hardens the EPP GKE entry point against a connection hoarding Denial of Service (DoS) vulnerability (b/534930905).
We resolve this by restricting pre-login connections to a short idle timeout and enforcing pod-local connection caps:
1. Removed certificate-based connection quota tracking. IP limits are now enforced pre-login, and authenticated Registrar ID limits are enforced post-login.
2. Implemented a 10-second scheduled timeout task during the pre-login phase. If the client does not successfully authenticate within 10 seconds of TLS handshake completion, they are disconnected.
3. Added a new response header 'Nomulus-Logged-In-Registrar' set by the backend EppRequestHandler upon successful login. EppServiceHandler monitors this header inline to perform registrar quota upgrades and cancel the pre-login timeout task.
4. Hardened EppProxyProtocolHandler to validate incoming IPs from the PROXY protocol header to prevent IP spoofing and smuggling, falling back to the TCP source IP on validation failures.
This is non-production so it's not a huge deal but in general, we should
restrict the OT&E users so that they only exist within the workspace
that we control. Other users that are created using the console already
follow this format.
b/534932209 for more info
This is not explicitly called out that I could find in the RFCs, however
it is very clear from the RFCs that hosts are subordinate to their
domains and it stands to reason that updates to a subordinate host are
updates to the superordinate domain.
Section 1.1 of RFC 5732 specifies:
```
host name "ns1.example.com" has a subordinate relationship to domain
name "example.com". EPP actions (such as object transfers) that do not
preserve this relationship MUST be explicitly disallowed.
```
Allowing host updates (e.g. renames) opens up situations where this
relationship could be severed. We add analogous prohibitions on deletion
and creation of subordinate hosts as well.
b/534930957
nowhere in the RFCs does it say that only one contact is allowed, and
indeed there are many situations where it's actually an array of
contacts. We have a bunch of registrars that already have multiple abuse
contacts. I'm not sure why the comment originally claimed this, but it's
from years and years ago.
b/534931561
Verified in support docs 2.19 and 2.33 that agents should have the
ability to edit allowed TLDs but should not have the ability to edit
whether registry lock is allowed. This reflects that.
b/534931470
this is a potential security issue if we give users who only have
EDIT_REGISTRAR_DETAILS access to become admin PoCs, which can have a
different set of permissions.
When inspecting domain entities with tools like GetDomainCommand, displaying
plaintext authentication codes is unnecessary and presents a security risk
for credential exposure in logs and bug ticket comments (b/537293980).
This commit modifies GetDomainCommand to redact sensitive authcodes by
default using an in-memory entity builder snapshot while preserving database
records. An optional --show_authcode flag with arity 1 is introduced to
explicitly display authcodes when required for transfer authorization.
BUG= http://b/537293980
Define arity 1 on requestedByRegistrar options in BulkDomainTransferCommand
and DeleteDomainCommand to accept explicit false arguments without parsing
failures. Up-to-date tests verify space-separated and equals argument syntax.
BUG= http://b/537308816
Currently we grant users IAP_SECURED_WEB_APP_USER_ROLE access to the
entire project upon creation if there is no preexisting group set up for
them to use. These users only need to be able to access the console, so
we should restrict it to the console.
If there is a group (which we do have in our prod + sandbox
configuration) then we just add them to that group instead and that
group controls all the privileges. So this is a no-op for our setup.
We prioritize the group -- if both the group and the IAP service ID are
specified, we'll still just add them to the group
b/534931170
we should run these checks when we deserialize from json objects, not
just when we use the builder to construct it. We did something similar
for TLD recently.
gbrodmanandGitHub
Ben McIlwainandGitHub
Weimin YuandGitHub
Pavlo TkachandGitHub
Juan CelhayandGitHub