1
0
mirror of https://github.com/google/nomulus synced 2026-05-25 09:10:51 +00:00

Compare commits

...

84 Commits

Author SHA1 Message Date
Michael Muller
296440b277 Remove labels from output of list_premium_lists (#1182)
* Remove labels from output of list_premium_lists

Remove the ability to show all of the labels associated with a premium list in
the list_premium_lists command.  Supporting this requires loading the entire
contents of all premium lists from the database as opposed to just the list
records, and the information can be obtained using get_premium_list.
2021-05-27 10:39:15 -04:00
Lai Jiang
50f80744d8 Change BillingEvent parent to Key<DomainHistory> (#1178) 2021-05-25 18:48:47 -04:00
Michael Muller
826320c7fd Always detach entities during load (#1116)
* Always detach entities during load

The mutations on non-transient fields that we do in some of the PostLoad
methods have been causing the objects to be marked as "dirty", and hibernate
has been quietly persisting them during transaction commit.

By detaching the entities on load, we avoid any possibility of this, which
works in our case because we treat all of our model objects as immutable
during normal use.

There is another mixed blessing to this: lazy loading won't work on these
objects once they are detached from a session, meaning that all fields must be
lazy loaded up front.  This is unfortunate in that we don't always need those
lazy-loaded fields and there is a performance cost to loading them, but it is
also useful in that objects will now be complete when used outseide of the
transaction that loaded them (prior to this, an attempt to access a
lazy-loaded field after its transaction closed would have caused an error at
runtime).

* Changes requested in review

* A few improvements to test logic

* Deal with premature detachment of mutated objects

* Add unit tests, use a more specific exception

* Changes for review

- Deal with DomainDeleteFlow, which appears to be the only case in the
  codebase where we're doing a load-after-save.
- Display the object that is being loaded after save in the exception message.
- Add a TODO for figuring out why Eager loads aren't working as expected.

* Move the recurring billing event into a parameter

* Changes for review and rebase error fix

* Remove initialization of list entries

Remove initialization of list entries that we want to be lazy loaded (premium,
reserved, and claims lists).

* Post-rebase cleanups
2021-05-25 14:34:24 -04:00
Michael Muller
8099789012 Safely lazy load claims and reserved lists (#1177)
* Safely lazy load claims and reserved lists

This moves the entries of all of these lists into "insignificant" fields and
manages them explicitly.

* Additional fixes

Fix a few problems that came up in the merge or weren't caught in earlier
local test runs.

* Changes for review

- removed debug code
- added comments
- improved some methods that were loading the entire claims list
  unnecessarily.

* Fixed javadoc links

* Reformatted

* Minor fix for review
2021-05-25 11:28:30 -04:00
gbrodman
20a0e4ce3f Remove a couple additional ofy() calls (#1171)
* Remove a couple additional ofy() calls
2021-05-24 13:12:40 -04:00
Lai Jiang
2f2e9dd49f Add methods to return subtypes of HistoryEntry when querying (#1172)
This is useful when we expect a specific subtype in the return value so
that we can set the parent resource (e. g. DomainContent for
DomainHistory) on it, or when a specific subtype is needed from the call
site.

This PR also fixes some use of generic return values. It is always better to
return <HistoryEntry> than a wildcard <? extends HistoryEntry>, because for
immutable collections, <? extends HistoryEntry> is no different than
<HistoryEntry> as return value -- you can only get a HistoryEntry from it.
The wildcard return value means that even if you are indeed getting a
<DomainHistory> from the query, the call site has no compile time knowledge of
it and can only assume it is a <HistoryEntry>.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1172)
<!-- Reviewable:end -->
2021-05-24 11:36:11 -04:00
gbrodman
5e28694053 Add an object to store database migration stages (#1170)
* Add an object to store database migration stages

go/registry-3.0-stage-management for more details

This basically boils down to storing an enum in the database so that we
can tell what stage of the migration we're in.

We use a cross-TLD parent so that we can have strong transactional
consistency on retrieval.
2021-05-21 11:49:35 -04:00
sarahcaseybot
642405375b Stop writing ClaimsList to Datastore (#1169)
* Stop writing ClaimsList to Datastore

* Fix some failing tests

* Rename ClaimsListShard to ClaimsList
2021-05-20 15:44:40 -04:00
Lai Jiang
02eb7cfcc3 Switch from using raw HistoryEntries to typed subclasses thereof (#1150)
HistoryEntry is used to record all histories (contact, domain, host) in
Datastore. In SQL it is now split into three subclasses (and thus
tables): ContactHistory, DomainHistory and HostHistory. Its builder is
genericized as a result which led to a lot of compiler warnings for the
use of a raw HistoryEntry in the existing code base.

This PR cleans things up by replacing all the explicit use of
raw HistoryEntry with the corresponding subclass and also adds some
guardrails to prevent the use of raw HistoryEntry accidentally.

Note that because DomainHistory includes nsHosts and gracePeriodHistory,
both of which are assigned a roid from ofy when built, the assigned roids for
resources after history entries are built are incremented compared to
when only HistoryEntrys are built (before this PR) in
RdapDomainSearchActionTest.

Also added a convenient tm().updateAll() varargs method.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1150)
<!-- Reviewable:end -->
2021-05-20 11:58:41 -04:00
Michael Muller
f7dca7fa96 Make PremiumList.labelsToPrices "insignificant" (#1167)
* Make PremiumList.labelsToPrices "insignificant"

Add the ImmutableObject.Insignificant annotation to labelsToPrices and also
mark it as Transient.  In order to do lazy-loads on this field, we need to do
so explicitly: doing otherwise breaks the immutability contract and prevents
detaching the object upon load.

Note that this is an expedient solution to this problem, but not the optimal
one.  Ideally, the disassociation between PremiumList and its PremiumEntry's
would be more explicit.  However, breaking labelsToPrices out would at minimum
require reworking the Create/UpdatePremiumList commands, which currently rely
on passing around a self-contained PremiumList object, both from the parser
interfaces and to the database.

If this approach is acceptable, we can apply it to ReservedList and ClaimsList
as well (though it may be easier to break the association in those cases).

* Fix premium list "delete" to support a test

* Fix a few more tests

* Changes for review (updated javadocs)

* Minor fixes

* Updated getLablesToPrices() comment

* Format fixes, fixed PremiumEntry interfaces

PremiumEntry can now be SQL only.
2021-05-20 11:21:37 -04:00
gbrodman
a7e8ae5a2c Add loadOnlyOf method to tm() (#1162)
* Add loadOnlyOf method to tm()

In addition there's a bit of a refator of SqlReplayCheckpoint to make it
more in line with the other singletons. This method is useful for the
singleton classes where we expect at most one entity to exist, e.g.
ServerSecret.
2021-05-20 10:59:01 -04:00
Michael Muller
dc7f21ca68 Convert most poll message queries to QueryComposer (#1151)
* Convert most poll message queries to QueryComposer

* Add unit test and a better exception for datastore

* Remove datastorePollMessageQuery from PollFlowUtils

* Reformatted.

* Improved test equality checks

* Changes for review

* Converted concatenated string to String.format()
2021-05-19 15:58:20 -04:00
Weimin Yu
e96873f2d0 Support text-based JPQL query for BEAM (#1168)
* Support text-based JPQL query for BEAM
2021-05-19 14:45:04 -04:00
Lai Jiang
b5f05405a0 Fix linter warnings (#1165) 2021-05-18 18:30:01 -04:00
gbrodman
f702f2670b Use a flatMap in StaticPremiumPricingEngine (#1166)
* Use a flatMap in StaticPremiumPricingEngine
2021-05-18 12:20:04 -04:00
sarahcaseybot
21aeedae11 Fix NullPointerException in StaticPremiumPricingEngine (#1164)
* Fix NullPointerException in StaticPremiumPricingEngine

* Make getPremiumList return optional

* add isPresent checks
2021-05-18 10:55:27 -04:00
sarahcaseybot
c1f0c29134 Stop writing ReservedList to Datastore (#1163) 2021-05-17 17:46:21 -04:00
gbrodman
16641e05a1 Update GCL dependency to avoid security alert (#1139)
* Update GCL dependency to avoid security alert

This required a few changes in addition to the dependency update.

- a few transitive / required dependency updates as well
- updating soyutils_usegoog.js and adding checks.js because they're
necessary as part of the Soy compilation process
- Using a trustedResourceUri in the buildSrc Soy compilation instead of
a string
- changing the arguments to the Soy-to-Java compiler to comply with the
new version
- Moving all Soy UI files to be in the registrar directory. This was
not the case before due to previous thinking that we'd have separate
admin and registrar consoles -- this is no longer the case so it's no
longer necessary. This necessitated various refactorings and reference
changes.
  - The new soy-to-javascript compiler requires this, as it removes the
  "deps" param that we were previously using to say "use the general UI
  utils as dependencies for the registrar-console files".
- Creating a SQL environment and loading test data in the test server
main method -- previously, the local test server did not work.
- Fix some JS code that was referencing now-deleted library functions
- Removal of the Karma tests, as the karma-closure library hasn't been
updated since 2018 and it no longer works. We never noticed any errors
from the Karma tests, we never change the JS, and we have the
Java+Selenium screenshot differ tests to test the UI anyway.
2021-05-17 13:21:26 -04:00
Ben McIlwain
bf1c34cc3b Add sanity checks to history entry construction (#1156)
* Add sanity checks to history entry construction

* Add more missing setClientId() calls and delete scrap tool

* Merge branch 'master' into synthetic-requestedby

* Set more client IDs

* Merge branch 'master' into synthetic-requestedby
2021-05-14 19:54:35 -04:00
sarahcaseybot
93dc812ea2 Stop writing PremiumList to Datastore (#1160)
* Stop writing PremiumList to Datastore

* Fix formatting

* Format fix

* Rename the DAO

* Fix merge conflicts and add comment
2021-05-14 16:13:05 -04:00
Weimin Yu
e09138645f Fix RegistryJpaIO.Read problem with large data (#1161)
* Fix RegistryJpaIO.Read problem with large data

The read connector needs to detach loaded entities. This 
is now the default behavior in QueryComposer

Also removed the 'transaction mode' property from the Read connector.
There are no obvious use cases for non-transaction query, and
implementation is not straightforward with the current code base.

Also changed the return type of QueryComposer.list() to ImmutableList.
2021-05-14 15:19:12 -04:00
gbrodman
238deb25ec Clean up some SqlEntity classes (#1158)
* Clean up some SqlEntity classes

This started as having a better check for when to run the
ReplayCommitLogsToSqlAction but that'll require a bit more thought, and
this is a fairly simple PR that can be split out.
2021-05-14 11:25:11 -04:00
Ben McIlwain
6ce2926c6d Remove final vestiges of domain applications (#1153)
* Remove final vestiges of domain applications
2021-05-14 10:39:25 -04:00
Rachel Guan
27f431b9cf Change premium list command to be based off of mutating command (#1123)
* Change premium list command to be based off of mutating command

* Modify test cases and add comments for better readability

* Fix typo
2021-05-14 08:40:03 -04:00
gbrodman
2bb0e7305d Convert even more classes to auditedOfy() (#1157)
* Convert even more classes to auditedOfy()

This covers almost all of the classes in the second round of the sheet.
There are still some classes that need conversion but this is the vast
majority of them.

https://docs.google.com/spreadsheets/d/1aFEFuyH6vVW6b-h71O9f5CuUc6Y7YjZ2kdRL3lwXcVk/edit?resourcekey=0-guwZVKfSH-pntER1tUit6w#gid=1355213322
for notes
2021-05-13 14:12:13 -04:00
Lai Jiang
10757863ce Reorder steps (#1159) 2021-05-13 13:15:46 -04:00
gbrodman
02079010c6 Add mapreduce action to create synthetic history entries (#1125)
* Add mapreduce action to create synthetic history entries

RDE and zone file generation require being able to tell what objects
looked like in the past (though not beyond 30 days, or whatever the
Datastore retention period is set to). In Datastore, to answer this we
look at commit logs, and in SQL we will look at the History objects
stored for each EPP resource. This action can be run once while in
Datastore-primary-SQL-secondary to make sure that every EPP resource has
at least one history entry for which the resource-at-this-time field is
filled out in the SQL world.
2021-05-13 11:48:19 -04:00
Lai Jiang
4246e7e4e0 Add indexes on contacts in the Domain table (#1145)
These indexes are used to find if a contact is linked to a domain in
during a contact delete.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1145)
<!-- Reviewable:end -->
2021-05-13 10:47:35 -04:00
Lai Jiang
9f21989f13 Remove the logic to add full certificate in the headers (#1143)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1143)
<!-- Reviewable:end -->
2021-05-12 20:52:16 -04:00
gbrodman
2073f5b59f Populate the host in HostHistory objects in Host flows (#1129)
* Populate the host in HostHistory objects in Host flows
2021-05-12 19:11:30 -04:00
Weimin Yu
66ac000ef4 Fix the JPA Read connector for large data (#1155)
* Fix the JPA Read connector for large data

Allow result set streaming by setting the fetchSize on JDBC statements.
Many JDBC drivers by default buffers the entire result set, causing
delays in first result and/or out of memory errors.

Also fixed a entity instantiation problem exposed in production runs.

Lastly, removed incorrect comments.
2021-05-12 19:07:38 -04:00
Rachel Guan
85bac9834f Add stageEntityChange() method to display difference when creating a reserved list (#1149)
* Add stageEntityChange() method to display difference before execution when creating a reserved list
2021-05-12 17:32:57 -04:00
Weimin Yu
484e30cd80 Restore a fix for flaky test (#1154)
* Restore a fix for flaky test

Restore a speculative fix for the flakiness in
DeleteExpiredDomainsActionTest. Although we identified a bug and fixed
it in a previous commit, it may not be the only bug. The removed fix may
still be necessary.
2021-05-12 16:03:42 -04:00
gbrodman
af67356aa0 Convert more ofy() to auditedOfy() calls (#1152)
A couple of these use the QueryComposer interface to avoid branching.

In addition, we enforce the Datastore restriction that there can be at
most 1 field with an inequality query, see https://cloud.google.com/appengine/docs/standard/go111/datastore/query-restrictions#inequality_filters_are_limited_to_at_most_one_property
2021-05-12 15:06:19 -04:00
Rachel Guan
8c9a2b5f4a Fix typo in comment of premium list example file (#1148)
* Fix typo in comment of premium list example file
2021-05-11 18:25:05 -04:00
gbrodman
0d67ea3a6e Combine the two Lock classes into one class (#1126)
* Combine the two Lock classes into one class

This allows us to remove the DAO and to just treat locks the same as we
would treat any other object -- generically grabbing them from the
transaction manager.

We do not need to be concerned about the changeover between Datastore
and SQL because we assume that any such changeover will require
sufficient downtime that any currently-valid acquired locks will expire
during the downtime. Otherwise, we could get into a situation where an
action has acquired a particular lock in Datastore but not SQL.
2021-05-11 16:37:40 -04:00
Rachel Guan
5b56e8b71b Create key based on the change type (#1147)
* Create key based on the change type
2021-05-11 15:24:35 -04:00
Weimin Yu
6eba8aa1c4 Fix timestamp inversion bug (#1144)
* Fix timestamp inversion bug

Set the number of commitLog buckets to 1 in CommitLog replay tests to
expose all timestamp inversion problems due to replay. Fixed
PollAckFlowTest which is related to this problem.

Also fixed a few tests that failed to advance the fake clock when they
should, using the following approaches:

- If DatabaseHelper used but clock is not injected, do it. This
  allows us to remove some unnecessary manual clock advances.
- Manually advance the clock where convenient.
- Enable clock autoIncrement mode when calling production classes that
  performs multiple transactions.

We should consider making 1-bucket the default setting for tests. This
is left to another PR.
2021-05-11 14:51:10 -04:00
Lai Jiang
8d18450e56 Update README.md (#1146)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1146)
<!-- Reviewable:end -->
2021-05-11 13:40:07 -04:00
sarahcaseybot
65be65fb24 Always use Cloud SQL as primary for ClaimsList (#1127)
* Always use Cloud SQL as primary for ClaimsList

* Add a test back
2021-05-10 16:47:34 -04:00
Weimin Yu
984f1118e3 Make secretmanager primary storage for keyring (#1124)
* Make secretmanager primary storage for keyring

Also removed the migrate_kms_keyring command.
2021-05-10 11:11:26 -04:00
gbrodman
0bcb142bc9 Add an auditedOfy marker method for allow-listed ofy() calls (#1138)
* Add an auditedOfy marker method for allow-listed ofy() calls

This will allow us to make sure that every usage of ofy() has been
hand-examined and specifically allowed.
2021-05-10 10:55:28 -04:00
Lai Jiang
d93a4e562a Delete hosts synchronously when using SQL (#1141)
Also put some common logic in helper funcions in ContactDeleteFlowTest
to reduce clutter.
2021-05-10 10:22:01 -04:00
Lai Jiang
420a579e01 Fix flaky Spec11PipelineTest (#1133) 2021-05-07 15:01:11 -04:00
Lai Jiang
1ec96b66e2 Perform synchronous contact delete in SQL (#1137)
In SQL the contact of a domain is an indexed field and therefore we can
find linked domains synchronously, without the need for MapReduce.

The delete logic is mostly lifted from DeleteContactsAndHostsAction, but
because everything happens in a transaction we do not need to recheck a
lot of the preconditions that were necessary to ensure that the async
delete request still meets the conditions that when the request was
enqueued.
2021-05-07 10:48:51 -04:00
gbrodman
51a7ba249e Populate the contact in ContactHistory objects created in Contact flows (#1111)
* Populate the contact in ContactHistory objects created in Contact flows

Minimal interesting changes here
- a bit of reconstruction in ContactHistory to get the repo ID from the
key
- making the History revision ID Long instead of long so that it can be
null in non-built intermediate entities
- adding a copyFrom(HistoryEntry.Builder) method in HistoryEntry.Builder
so that we don't need to allocate quite as many unnecessary IDs, i.e.
removing the .build() lines in provideContactHistory and
provideDomainHistory
2021-05-06 14:38:55 -04:00
Lai Jiang
5120397607 Upload the GCB delete job yaml file to GCS (#1135)
<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1135)
<!-- Reviewable:end -->
2021-05-05 21:43:51 -04:00
sarahcaseybot
038825f254 Always use Cloud SQL as primary for Reserved and Premium Lists (#1113)
* Always use Cloud SQL as primary for Reserved and Premium Lists

* small typos

* Add a state check

* Add test for bloom filter

* fix import
2021-05-05 17:24:06 -04:00
Weimin Yu
b38574a9fc Add a BEAM read connector for JPA entities (#1132)
* Add a BEAM read connector for JPA entities

Added a Read connector to load JPA entities from Cloud SQL.

Also attempted a fix to the null threadfactory problem.
2021-05-05 15:45:03 -04:00
Lai Jiang
3f6ec8f1b0 Re-enable tests in RC build (#1130)
There has been a case where the CI was broken on Friday and no one
noticied or fixed it and a RC build was built with broken tests.
The tests were disabled due to unknown test failures that have since
been fixed.

Also update the machine type used by GCB to be more powerful. This is
necessary for the tests to past because N1_HIGHCPU_8 is RAM constraint
and the tests crashes. I updated all jobs to use the new type which
hopefully will make the build faster as well.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1130)
<!-- Reviewable:end -->
2021-05-05 13:53:21 -04:00
gbrodman
65fb0c6cff Update Karma version to avoid security hole in dependency (#1134)
This also forces the karma test to use the Gradle-installed version of
node instead of the global version. The global version installed on the
Kokoro machines is too old to function with some of the newer libraries.
2021-05-05 13:50:45 -04:00
Lai Jiang
e63085fb6a Add a GCB job to delete stopped GAE versions (#1128) 2021-05-05 11:27:46 -04:00
gbrodman
b5363e9457 Populate the domain in DomainHistory objects created in Domain flows (#1106)
Unfortunately, much of the time there's a bit of a circular dependency
in the object creation, e.g. the Domain object stores references to the
billing events which store references to the history object which
contains the Domain object. As a result, we allocate the history
object's ID before creating it, so that it can be referenced in the
other objects that store that reference, e.g. billing events.

In addition, we add a utility copyFrom method in HistoryEntry.Builder to
avoid unnecessary ID allocations.
2021-05-04 19:09:27 -04:00
Ben McIlwain
cb16df235a Remove unnecessary MockitoExtension from Spec11PipelineTest (#1115)
* Remove unnecessary MockitoExtension from Spec11PipelineTest

This is kind of a shot in the dark here, but this is one of the obvious
differences between this test class (which frequently experiences flakes) and
the other pipeline test classes which do not.

It's also possible we were getting the wrong runner if the test framework was
incorrectly detecting an App Engine runtime environment, so I added an assert
that will make it very clear if this is the cause of any failures.
2021-05-04 18:38:24 -04:00
Lai Jiang
d285edef3d Fix a few linter warnings (#1122) 2021-05-04 13:35:31 -04:00
Weimin Yu
509c0dcd17 Handle bad production data when migrating to SQL (#1120)
* Handle bad production data when migrating to SQL

Ignore or fix bad entites when populating SQL with production data in
Datastore. These are mostly inconsistent foreign keys.

See b/185954992 for details.
2021-05-03 16:09:43 -04:00
sarahcaseybot
ce18bf0690 Use FakeClock to prevent Expired Certificate Violations (#1121)
* Use FakeClock to prevent Expired Certificate Violations

* Format fixes

* Make CertificateChecker static
2021-05-03 15:10:26 -04:00
Lai Jiang
8d63cbfca0 Remove enforcement date from the SslServerInitializer (#1117)
The enforcement data has passed and ICANN has confirmed that their web
WHOIS prober conforms to our requirements.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1117)
<!-- Reviewable:end -->
2021-04-30 15:44:03 -04:00
Lai Jiang
eb6a1fe1ed Remove Pipeline as a field in pipeline classes (#1119)
In tests we use a TestPipelineExtension which does some static
initialization that should not be repeated the same JVM. In our
XXXPipeline classes we save the pipeline as a field and usually write lambdas
that are pass to the pipeline. Because lambdas are effectively anonymous inner
classes they are bound to their enclosing instances. When they get serialized
during pipeline execution, their enclosing classes also do. This might result
in undefined behavior when multiple lambdas in the same XXXPipeline are used
on the same JVM (such as in tests) where the static initialization may be done
multiple times if different class loaders are used. This is very
unlikely to happen but as a best practice we still remove them as
fields.
2021-04-30 14:32:33 -04:00
Weimin Yu
431710c95b Improve usability of WipeOutCloudSqlAction (#1118)
* Improve usability of WipeOutCloudSqlAction

Replace the "drop owned" statement with ones that drops only tables and
sequences. The former statement also drops default grants for the
nomulus user, which must be restored before the database can be used by
the nomulus server and tools.
2021-04-29 23:09:20 -04:00
Michael Muller
1fdf9cb979 Convert GenerateLordnCommand to tm (#1091)
* Convert GenerateLordnCommand to tm

This makes use of QueryComposer and adds a `list()` method to it.

Since there was no test for GenerateLordnCommand, this also implements one.

* Changes requested in review

* Add test for list queries

* Stream domains instead of listing them

* Reformatted
2021-04-29 13:14:56 -04:00
Michael Muller
95fdd36c77 Make nom_build not check for ".git" directory (#1110)
* Make nom_build not check for ".git" directory

nom_build tries to verify that it is in the root of the tree prior to doing
anything, however checking for a .git directory doesn't work in a merged
directory.

* Minor formatting fix to attempt to force rebuild
2021-04-28 11:23:39 -04:00
Ben McIlwain
d239a4d706 Make the ReadDnsQueueAction tests retry on failures (#1114)
These tests are flaky due to some kind of contention/collision on the mock task
queue. Retrying seems to fix the vast majority of flakes, is easy to implement,
and is more performant than moving these tests into the fragileTests test suite.
2021-04-28 10:20:36 -04:00
gbrodman
d99278e723 Convert remaining read-only flow tests to dual-DB (#1107)
Note that there are many flow tests that aren't
@DualDatabaseTest-annotated yet but those will come later, as they will
require more changes to the flows (other PRs are coming or in progress).
This only includes the remaining EppResource flows that don't create a
history entry.
2021-04-27 20:37:09 -04:00
Ben McIlwain
9d4de806f5 Improve error when creating domain label lists for non-existent TLDs (#1112)
* Improve error message when creating domain label lists for non-existent TLDs
2021-04-27 19:17:23 -04:00
sarahcaseybot
2528ee05dd Remove SMDRL completely from Datastore (#1104)
* Remove SMDRL completely from Datastore

* Remove some unnecessary stuff

* Change row count to 10000

* Remove implement EntityTestCase
2021-04-26 17:15:50 -04:00
Rachel Guan
367a38c5b0 Display changes when updating reserved list (#1093)
* add stageEntityChange to show diff

* add test cases
2021-04-26 13:31:57 -04:00
Lai Jiang
8884425a05 Fix build (#1109) 2021-04-26 10:34:29 -04:00
gbrodman
2c4c0bf9f8 Convert more tests to use @DualDatabaseTest and SQL in general (#1101)
Nothing super crazy here other than persisting the entity changes in
DomainDeleteFlow at the end of the flow rather than almost at the end.
This means that when we return the results we give the results as they
were originally present, rather than the subsequently-changed values.
2021-04-23 18:26:44 -04:00
Michael Muller
9c89643367 Fix Spec11 domain check (#1105)
* Fix Spec11 domain check

We should be checking to see if there are _any_ active domains for a given
reported domain, not to see if _the_ domain for the name is active.

The last change caused an exception for domains with soft-deleted past domains
of the same name.  The original code only checked the first domain returned
from the query, which may have been soft-deleted.  This version checks all
domain records to see if any are active.

* filter().count() -> anyMatch()
2021-04-23 14:20:31 -04:00
gbrodman
9f69a0bf2e Begin saving the EppResource parent in *History objects (#1090)
* Begin saving the EppResource parent in *History objects

We use DomainCreateFlow as an example here of how this will work. There
were a few changes necessary:

- various changes around GracePeriod / GracePeriodHistory so that we can
actually store them without throwing NPEs
- Creating one injectable *History.Builder field and using in place of
the HistoryEntry.Builder injected field in DomainCreateFlow
- Saving the EppResource as the parent in the *History.Builder setParent
calls
- Converting to/from HistoryEntry/*History classes in
DatastoreTransactionManager. Basically, we'll want to return the
*History subclasses (and similar in the ofy portions of HistoryEntryDao)
- Converting a few HistoryEntry.Builder usages to DomainHistory.Builder
usages. Eventually we should convert all of them.
2021-04-22 15:03:37 -04:00
sarahcaseybot
40db04db8d Use CommandWithRemoteApi in SetDatabaseTransitionScheduleCommand (#1099)
* Use CommandWithRemoteApi in ConfirmingCommand

* Remove unnecessary extensions

* Remove from ConfirmingCommand
2021-04-22 14:50:19 -04:00
Lai Jiang
217b37b9d5 Migrate the billing pipeline to flex template (#1100)
This is similar to the migration of the spec11 pipeline in #1073. Also removed
a few Dagger providers that are no longer needed.

TESTED=tested the dataflow job on alpha.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1100)
<!-- Reviewable:end -->
2021-04-22 10:26:15 -04:00
Lai Jiang
09b6e300fc Remove unused BeamJpaExtension and related classes (#1102)
* Remove unused BeamJpaExtension and related classes

* Remove unused qualifiers
2021-04-22 10:02:18 -04:00
Lai Jiang
4d99a5dd35 Remove a linter warning (#1103)
* Remove a linter warning

* Remove duplicate
2021-04-22 09:42:05 -04:00
gbrodman
5d3e9da750 Defer all foreign keys in SQL (#1094)
* Defer all foreign keys in SQL

The main difference here is that the constraint violation exceptions
won't be thrown until the transaction is completed, rather than when the
insert is first performed within the transaction. We get the same error
message either way. The primary benefit to this is that when dealing
with large operations inside a single transaction (flows), we don't need
to worry about the order of insertions of removals with regards to
foreign keys.
2021-04-21 14:29:20 -04:00
Lai Jiang
464f9aed1f Migrate Spec11 pipeline to flex template (#1073)
* Migrate Spec11 pipeline to flex template

Unfortunately this PR has turned out to be much bigger than I initially
conceived. However this is no good way to separate it out because the
changes are intertwined. This PR includes 3 main changes:

1. Change the spec11 pipline to use Dataflow Flex Template.
2. Retire the use of the old JPA layer that relies on credential saved
   in KMS.
3. Some extensive refactoring to streamline the logic and improve test
   isolation.

* Fix job name and remove projectId from options

* Add parameter logs

* Set RegistryEnvironment

* Remove logging and modify safe browsing API key regex

* Rename a test method and rebase

* Remove unused Junit extension

* Specify job region
2021-04-21 00:09:50 -04:00
sarahcaseybot
a0995fa0eb Stop dual read and dual write of SMDRL (#1095)
* Stop dual read and dual write of SMDRL

* Remove some more stuff from SignedMarkRevocationListDaoTest

* Change some names
2021-04-20 17:08:59 -04:00
Weimin Yu
fff95b20e6 Skip undefined secrets in keyring migration (#1098)
* Skip undefined secrets in keyring migration

If a secret does not exist in datastore, log and skip it.
2021-04-20 16:26:40 -04:00
gbrodman
23896b64c7 Set default value of 1 for new not-null columns (#1097)
Use 1 since it's the constant singleton ID
2021-04-20 15:25:20 -04:00
Ben McIlwain
844b5ab713 Send an immediate poll message for superuser domain deletes (#1096)
* Send an immediate poll message for superuser domain deletes

This poll message is in addition to the normal poll message that is sent when
the domain's deletion is effective (typically 35 days later). It's needed
because, in the event of a superuser deletion, the owning registrar won't
otherwise necessarily know it's happening.

Note that, in the case of a --immediate superuser deletion, the normal poll
message is already being sent immediately, so this additional poll message is
not necessary.
2021-04-20 15:22:49 -04:00
sarahcaseybot
aac952d6a3 Return to using hash for login validation (#1084)
* Return to using hash for login validation

This PR also removes the start date for certificate enforcement.

* Inline verify certificate compliance
2021-04-20 14:07:01 -04:00
gbrodman
ee31f1fd95 Update various tests to work with SQL as well (#1078)
* Update various tests to work with SQL as well

The main weird bit here is adding a method in DatabaseHelper to
retrieve and initialize all objects in either database. The
initialization is necessary since it's used post-command-dry-run to make
sure that no changes were actually made.
2021-04-20 11:52:53 -04:00
Michael Muller
4657be21b7 Convert CountDomainsCommand to tm (#1092)
* Convert CountDomainsCommand to tm

As part of this, implement "select count(*)" queries in the QueryComposer.

* Replaced kludgy trick for objectify count
2021-04-20 10:38:38 -04:00
560 changed files with 10714 additions and 14642 deletions

View File

@@ -1,4 +1,5 @@
python/
node_modules/
**/build/
**/out/
.*/

1
.gitignore vendored
View File

@@ -4,6 +4,7 @@
######################################################################
# Java Ignores
gjf.out
*.class
# Mobile Tools for Java (J2ME)

View File

@@ -318,7 +318,7 @@ subprojects {
// expose to users.
if (project.name != 'docs') {
javadocSource << project.sourceSets.main.allJava
javadocClasspath << project.sourceSets.main.compileClasspath
javadocClasspath << project.sourceSets.main.runtimeClasspath
javadocClasspath << "${buildDir}/generated/sources/annotationProcessor/java/main"
javadocDependentTasks << project.tasks.compileJava
}
@@ -457,6 +457,8 @@ task javaIncrementalFormatApply {
task javadoc(type: Javadoc) {
source javadocSource
classpath = files(javadocClasspath)
// Exclude the misbehaving generated-by-Soy Java files
exclude "**/*SoyInfo.java"
destinationDir = file("${buildDir}/docs/javadoc")
options.encoding = "UTF-8"
// In a lot of places we don't write @return so suppress warnings about that.

View File

@@ -72,6 +72,7 @@ dependencies {
compile deps['com.google.auth:google-auth-library-credentials']
compile deps['com.google.auth:google-auth-library-oauth2-http']
compile deps['com.google.auto.value:auto-value-annotations']
compile deps['com.google.common.html.types:types']
compile deps['com.google.cloud:google-cloud-core']
compile deps['com.google.cloud:google-cloud-storage']
compile deps['com.google.guava:guava']

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
@@ -47,7 +48,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.13
@@ -55,9 +55,9 @@ org.apache.httpcomponents:httpcore:4.4.14
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:3.5.0
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:7.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
@@ -47,7 +48,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.13
@@ -55,9 +55,9 @@ org.apache.httpcomponents:httpcore:4.4.14
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:3.5.0
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:7.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
@@ -47,7 +48,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.13
@@ -55,9 +55,9 @@ org.apache.httpcomponents:httpcore:4.4.14
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:3.5.0
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:7.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
@@ -47,7 +48,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.13
@@ -55,9 +55,9 @@ org.apache.httpcomponents:httpcore:4.4.14
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:3.5.0
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:7.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
@@ -47,7 +48,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.13
@@ -55,9 +55,9 @@ org.apache.httpcomponents:httpcore:4.4.14
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:3.5.0
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:7.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.ibm.icu:icu4j:57.1
@@ -49,7 +50,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
junit:junit:4.13.1
net.bytebuddy:byte-buddy-agent:1.10.19
net.bytebuddy:byte-buddy:1.10.19
@@ -70,9 +70,9 @@ org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.7.7
org.objenesis:objenesis:3.1
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:9.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.ibm.icu:icu4j:57.1
@@ -49,7 +50,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
junit:junit:4.13.1
net.bytebuddy:byte-buddy-agent:1.10.19
net.bytebuddy:byte-buddy:1.10.19
@@ -70,9 +70,9 @@ org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.7.7
org.objenesis:objenesis:3.1
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:9.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.ibm.icu:icu4j:57.1
@@ -49,7 +50,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
junit:junit:4.13.1
net.bytebuddy:byte-buddy-agent:1.10.19
net.bytebuddy:byte-buddy:1.10.19
@@ -70,9 +70,9 @@ org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.7.7
org.objenesis:objenesis:3.1
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:9.0
org.threeten:threetenbp:1.5.0

View File

@@ -20,12 +20,12 @@ com.google.cloud:google-cloud-core:1.94.3
com.google.cloud:google-cloud-storage:1.113.12
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-apache-v2:1.39.0
com.google.http-client:google-http-client-appengine:1.39.0
com.google.http-client:google-http-client-gson:1.39.0
@@ -34,10 +34,11 @@ com.google.http-client:google-http-client:1.39.0
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.3
com.google.protobuf:protobuf-java:3.15.3
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.ibm.icu:icu4j:57.1
@@ -49,7 +50,6 @@ io.opencensus:opencensus-contrib-http-util:0.28.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
junit:junit:4.13.1
net.bytebuddy:byte-buddy-agent:1.10.19
net.bytebuddy:byte-buddy:1.10.19
@@ -70,9 +70,9 @@ org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.7.7
org.objenesis:objenesis:3.1
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:9.0
org.threeten:threetenbp:1.5.0

View File

@@ -23,6 +23,7 @@ import static google.registry.gradle.plugin.GcsPluginUtils.toByteArraySupplier;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.ImmutableSetMultimap;
import com.google.common.html.types.TrustedResourceUrls;
import com.google.template.soy.SoyFileSet;
import com.google.template.soy.tofu.SoyTofu;
import google.registry.gradle.plugin.ProjectData.TaskData;
@@ -118,7 +119,7 @@ final class CoverPageGenerator {
builder.put("projectState", state.toString());
builder.put("title", title);
builder.put("cssFiles", ImmutableSet.of("css/style.css"));
builder.put("cssFiles", ImmutableSet.of(TrustedResourceUrls.fromConstant("css/style.css")));
builder.put("invocation", getInvocation());
builder.put("tasksByState", getTasksByStateSoyData());
return builder.build();

View File

@@ -16,7 +16,7 @@
{template .coverPage}
{@param title: string}
{@param cssFiles: list<string>}
{@param cssFiles: list<trusted_resource_uri>}
{@param projectState: string}
{@param invocation: string}
{@param tasksByState: map<string, list<[uniqueName: string, description: string, log: string, reports: map<string, string>]>>}

View File

@@ -22,6 +22,7 @@ import google.registry.util.Clock;
import java.util.concurrent.atomic.AtomicLong;
import javax.annotation.concurrent.ThreadSafe;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.ReadableDuration;
import org.joda.time.ReadableInstant;
@@ -81,4 +82,14 @@ public final class FakeClock implements Clock {
public void setTo(ReadableInstant time) {
currentTimeMillis.set(time.getMillis());
}
/** Invokes {@link #setAutoIncrementStep} with one millisecond-step. */
public FakeClock setAutoIncrementByOneMilli() {
return setAutoIncrementStep(Duration.millis(1));
}
/** Disables the auto-increment mode. */
public FakeClock disableAutoIncrement() {
return setAutoIncrementStep(Duration.ZERO);
}
}

View File

@@ -256,6 +256,7 @@ GRADLE_FLAGS = [
'Specify a task to be excluded from execution.',
True),
]
def generate_gradle_properties() -> str:
"""Returns the expected contents of gradle.properties."""
out = io.StringIO()
@@ -270,7 +271,7 @@ def generate_gradle_properties() -> str:
def get_root() -> str:
"""Returns the root of the nomulus build tree."""
cur_dir = os.getcwd()
if not os.path.exists(os.path.join(cur_dir, '.git')) or \
if not os.path.exists(os.path.join(cur_dir, 'buildSrc')) or \
not os.path.exists(os.path.join(cur_dir, 'core')) or \
not os.path.exists(os.path.join(cur_dir, 'gradle.properties')):
raise Exception('You must run this script from the root directory')

View File

@@ -81,7 +81,7 @@ PRESUBMITS = {
".git", "/build/", "/generated/", "/generated_tests/",
"node_modules/", "JUnitBackports.java", "registrar_bin.",
"registrar_dbg.", "google-java-format-diff.py",
"nomulus.golden.sql", "soyutils_usegoog.js"
"nomulus.golden.sql", "soyutils_usegoog.js", "javascript/checks.js"
}, REQUIRED):
"File did not include the license header.",
@@ -213,6 +213,7 @@ PRESUBMITS = {
"RdapDomainSearchAction.java",
"RdapNameserverSearchAction.java",
"RdapSearchActionBase.java",
"RegistryQuery",
},
):
"The first String parameter to EntityManager.create(Native)Query "

View File

@@ -317,6 +317,7 @@ dependencies {
testCompile deps['com.google.monitoring-client:contrib']
testCompile deps['com.google.truth:truth']
testCompile deps['com.google.truth.extensions:truth-java8-extension']
testCompile deps['org.checkerframework:checker-qual']
testCompile deps['org.hamcrest:hamcrest']
testCompile deps['org.hamcrest:hamcrest-core']
testCompile deps['org.hamcrest:hamcrest-library']
@@ -432,12 +433,9 @@ task soyToJava {
// Relative paths of soy directories.
def spec11SoyDir = "google/registry/reporting/spec11/soy"
def toolsSoyDir = "google/registry/tools/soy"
def uiSoyDir = "google/registry/ui/soy"
def registrarSoyDir = "google/registry/ui/soy/registrar"
def soyRelativeDirs = [
spec11SoyDir, toolsSoyDir, uiSoyDir, registrarSoyDir,
]
def soyRelativeDirs = [spec11SoyDir, toolsSoyDir, registrarSoyDir]
soyRelativeDirs.each {
inputs.dir "${resourcesSourceDir}/${it}"
outputs.dir "${generatedDir}/${it}"
@@ -451,7 +449,8 @@ task soyToJava {
"--outputDirectory", "${outputDirectory}",
"--javaClassNameSource", "filename",
"--allowExternalCalls", "true",
"--srcs", "${soyFiles.join(',')}"
"--srcs", "${soyFiles.join(',')}",
"--compileTimeGlobalsFile", "${resourcesSourceDir}/google/registry/ui/globals.txt"
}
}
@@ -468,14 +467,6 @@ task soyToJava {
dir: "${resourcesSourceDir}/${registrarSoyDir}",
include: ['**/*.soy']))
soyToJava('google.registry.ui.soy',
"${generatedDir}/${uiSoyDir}",
files {
file("${resourcesSourceDir}/${uiSoyDir}").listFiles()
}.filter {
it.name.endsWith(".soy")
})
soyToJava('google.registry.reporting.spec11.soy',
"${generatedDir}/${spec11SoyDir}",
fileTree(
@@ -484,42 +475,24 @@ task soyToJava {
}
}
task soyToJS {
def rootSoyDirectory = "${resourcesSourceDir}/google/registry/ui/soy"
def outputSoyDirectory = "${generatedDir}/google/registry/ui/soy"
task soyToJS(type: JavaExec) {
def rootSoyDirectory = "${resourcesSourceDir}/google/registry/ui/soy/registrar"
def outputSoyDirectory = "${generatedDir}/google/registry/ui/soy/registrar"
inputs.dir rootSoyDirectory
outputs.dir outputSoyDirectory
ext.soyToJS = { outputDirectory, soyFiles , deps->
javaexec {
main = "com.google.template.soy.SoyToJsSrcCompiler"
classpath configurations.soy
def inputSoyFiles = files {
file("${rootSoyDirectory}").listFiles()
}.filter {
it.name.endsWith(".soy")
}
args "--outputPathFormat", "${outputDirectory}/{INPUT_FILE_NAME}.js",
classpath configurations.soy
main = "com.google.template.soy.SoyToJsSrcCompiler"
args "--outputPathFormat", "${outputSoyDirectory}/{INPUT_FILE_NAME}.js",
"--allowExternalCalls", "false",
"--srcs", "${soyFiles.join(',')}",
"--shouldProvideRequireSoyNamespaces", "true",
"--srcs", "${inputSoyFiles.join(',')}",
"--compileTimeGlobalsFile", "${resourcesSourceDir}/google/registry/ui/globals.txt"
if (deps != "") {
args "--deps", "${deps.join(',')}"
}
}
}
doLast {
def rootSoyFiles =
fileTree(
dir: "${rootSoyDirectory}",
include: ['*.soy'])
soyToJS("${outputSoyDirectory}", rootSoyFiles, "")
soyToJS("${outputSoyDirectory}/registrar",
files {
file("${rootSoyDirectory}/registrar").listFiles()
}.filter {
it.name.endsWith(".soy")
}, rootSoyFiles)
}
}
task stylesheetsToJavascript {
@@ -602,8 +575,8 @@ task compileProdJS(type: JavaExec) {
closureArgs << "--generate_exports"
// manually include all the required js files
closureArgs << "--js=${nodeModulesDir}/google-closure-library/**.js"
closureArgs << "--js=${jsDir}/soyutils_usegoog.js"
closureArgs << "--js=${nodeModulesDir}/google-closure-library/**/*.js"
closureArgs << "--js=${jsDir}/*.js"
closureArgs << "--js=${cssSourceDir}/registrar_bin.css.js"
closureArgs << "--js=${jsSourceDir}/**.js"
closureArgs << "--js=${externsDir}/json.js"
@@ -630,15 +603,6 @@ compileProdJS.dependsOn processResources
compileProdJS.dependsOn processTestResources
compileProdJS.dependsOn soyToJS
task karmaTest(type: Exec) {
dependsOn ':npmInstall'
workingDir rootProject.projectDir
executable 'node_modules/karma/bin/karma'
args('start', "${project.projectDir}/karma.conf.js")
}
test.dependsOn karmaTest
// Make testing artifacts available to be depended up on by other projects.
// TODO: factor out google.registry.testing to be a separate project.
task testJar(type: Jar) {
@@ -805,6 +769,14 @@ if (environment in ['alpha', 'crash']) {
mainClass: 'google.registry.beam.datastore.BulkDeleteDatastorePipeline',
metaData: 'google/registry/beam/bulk_delete_datastore_pipeline_metadata.json'
],
[
mainClass: 'google.registry.beam.spec11.Spec11Pipeline',
metaData: 'google/registry/beam/spec11_pipeline_metadata.json'
],
[
mainClass: 'google.registry.beam.invoicing.InvoicingPipeline',
metaData: 'google/registry/beam/invoicing_pipeline_metadata.json'
],
]
project.tasks.create("stage_beam_pipelines") {
doLast {

View File

@@ -1,15 +1,4 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
args4j:args4j:2.0.26
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.errorprone:error_prone_annotations:2.3.1
com.google.guava:guava:25.1-jre
com.google.j2objc:j2objc-annotations:1.1
com.google.javascript:closure-compiler-externs:v20190301
com.google.javascript:closure-compiler:v20190301
com.google.jsinterop:jsinterop-annotations:1.0.0
com.google.protobuf:protobuf-java:3.0.2
org.checkerframework:checker-qual:2.0.0
org.codehaus.mojo:animal-sniffer-annotations:1.14
com.google.javascript:closure-compiler:v20210505

View File

@@ -105,9 +105,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -135,7 +136,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -104,9 +104,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
com.google.guava:failureaccess:1.0.1
@@ -133,7 +134,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.14.0
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -110,9 +110,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -140,7 +141,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -110,9 +110,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -140,7 +141,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -105,9 +105,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -135,7 +136,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -104,9 +104,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
com.google.guava:failureaccess:1.0.1
@@ -133,7 +134,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.14.0
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -109,9 +109,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -139,7 +140,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -109,9 +109,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -139,7 +140,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -109,9 +109,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -139,7 +140,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -110,9 +110,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -140,7 +141,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:68.2

View File

@@ -5,25 +5,25 @@ aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.errorprone:error_prone_annotations:2.3.4
com.google.escapevelocity:escapevelocity:0.9.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:30.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:5.0.1
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.1
com.google.protobuf:protobuf-java:3.13.0
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.ibm.icu:icu4j:57.1
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
org.checkerframework:checker-qual:3.5.0
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.ow2.asm:asm-analysis:7.0
org.ow2.asm:asm-commons:7.0
org.ow2.asm:asm-tree:7.0
org.ow2.asm:asm-util:7.0
org.ow2.asm:asm:7.0

View File

@@ -106,9 +106,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -138,7 +139,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.googlecode.charts4j:charts4j:1.3

View File

@@ -105,9 +105,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
com.google.guava:failureaccess:1.0.1
@@ -136,7 +137,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.14.0
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.googlecode.charts4j:charts4j:1.3

View File

@@ -111,9 +111,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -143,7 +144,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.googlecode.charts4j:charts4j:1.3

View File

@@ -111,9 +111,10 @@ com.google.cloud:google-cloud-secretmanager:1.4.0
com.google.cloud:google-cloud-spanner:2.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.common.html.types:types:1.0.6
com.google.dagger:dagger:2.33
com.google.errorprone:error_prone_annotations:2.5.1
com.google.escapevelocity:escapevelocity:0.9.1
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
@@ -143,7 +144,7 @@ com.google.oauth-client:google-oauth-client:1.31.4
com.google.protobuf:protobuf-java-util:3.15.2
com.google.protobuf:protobuf-java:3.15.2
com.google.re2j:re2j:1.6
com.google.template:soy:2018-03-14
com.google.template:soy:2021-02-01
com.google.truth.extensions:truth-java8-extension:1.1.2
com.google.truth:truth:1.1.2
com.googlecode.charts4j:charts4j:1.3

View File

@@ -1,75 +0,0 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
process.env.CHROME_BIN = require('puppeteer').executablePath()
module.exports = function(config) {
config.set({
basePath: '..',
browsers: ['ChromeHeadlessNoSandbox'],
customLaunchers: {
ChromeHeadlessNoSandbox: {
base: 'ChromeHeadless',
flags: ['--no-sandbox']
}
},
frameworks: ['jasmine', 'closure'],
singleRun: true,
autoWatch: false,
files: [
'node_modules/google-closure-library/closure/goog/base.js',
'core/src/test/javascript/**/*_test.js',
{
pattern: 'core/src/test/javascript/**/!(*_test).js',
included: false
},
{
pattern: 'core/src/main/javascript/**/*.js',
included: false
},
{
pattern: 'core/build/generated/sources/custom/java/main/**/*.soy.js',
included: false
},
{
pattern: 'node_modules/google-closure-library/closure/goog/deps.js',
included: false,
served: false
},
{
pattern: 'node_modules/google-closure-library/closure/goog/**/*.js',
included: false
},
{
pattern: 'core/build/resources/main/google/registry/ui/assets/images/*.png',
included: false
},
{
pattern: 'core/build/resources/main/google/registry/ui/assets/images/icons/svg/*.svg',
included: false
}
],
preprocessors: {
'node_modules/google-closure-library/closure/goog/deps.js': ['closure', 'closure-deps'],
'node_modules/google-closure-library/closure/goog/base.js': ['closure'],
'node_modules/google-closure-library/closure/**/*.js': ['closure'],
'core/src/*/javascript/**/*.js': ['closure'],
'core/build/generated/sources/custom/java/main/**/*.soy.js': ['closure'],
},
proxies: {
"/assets/": "/base/core/build/resources/main/google/registry/ui/assets/"
}
});
};

View File

@@ -14,7 +14,7 @@
package google.registry.backup;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import com.google.appengine.api.datastore.EntityTranslator;
import com.google.common.collect.AbstractIterator;
@@ -45,7 +45,7 @@ public class BackupUtils {
* {@link OutputStream} in delimited protocol buffer format.
*/
static void serializeEntity(ImmutableObject entity, OutputStream stream) throws IOException {
EntityTranslator.convertToPb(ofy().save().toEntity(entity)).writeDelimitedTo(stream);
EntityTranslator.convertToPb(auditedOfy().save().toEntity(entity)).writeDelimitedTo(stream);
}
/**
@@ -61,11 +61,12 @@ public class BackupUtils {
@Override
protected ImmutableObject computeNext() {
EntityProto proto = new EntityProto();
if (proto.parseDelimitedFrom(input)) { // False means end of stream; other errors throw.
return ofy().load().fromEntity(EntityTranslator.createFromPb(proto));
if (proto.parseDelimitedFrom(input)) { // False means end of stream; other errors throw.
return auditedOfy().load().fromEntity(EntityTranslator.createFromPb(proto));
}
return endOfData();
}};
}
};
}
public static ImmutableList<ImmutableObject> deserializeEntities(byte[] bytes) {

View File

@@ -18,7 +18,7 @@ import static com.google.appengine.api.taskqueue.QueueFactory.getQueue;
import static com.google.appengine.api.taskqueue.TaskOptions.Builder.withUrl;
import static google.registry.backup.ExportCommitLogDiffAction.LOWER_CHECKPOINT_TIME_PARAM;
import static google.registry.backup.ExportCommitLogDiffAction.UPPER_CHECKPOINT_TIME_PARAM;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.DateTimeUtils.isBeforeOrAt;
@@ -64,8 +64,7 @@ public final class CommitLogCheckpointAction implements Runnable {
final CommitLogCheckpoint checkpoint = strategy.computeCheckpoint();
logger.atInfo().log(
"Generated candidate checkpoint for time: %s", checkpoint.getCheckpointTime());
tm()
.transact(
tm().transact(
() -> {
DateTime lastWrittenTime = CommitLogCheckpointRoot.loadRoot().getLastWrittenTime();
if (isBeforeOrAt(checkpoint.getCheckpointTime(), lastWrittenTime)) {
@@ -73,7 +72,7 @@ public final class CommitLogCheckpointAction implements Runnable {
"Newer checkpoint already written at time: %s", lastWrittenTime);
return;
}
ofy()
auditedOfy()
.saveWithoutBackup()
.entities(
checkpoint, CommitLogCheckpointRoot.create(checkpoint.getCheckpointTime()));

View File

@@ -17,7 +17,7 @@ package google.registry.backup;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Preconditions.checkState;
import static google.registry.mapreduce.MapreduceRunner.PARAM_DRY_RUN;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static java.lang.Boolean.FALSE;
import static java.lang.Boolean.TRUE;
@@ -75,9 +75,17 @@ public final class DeleteOldCommitLogsAction implements Runnable {
@Inject MapreduceRunner mrRunner;
@Inject Response response;
@Inject Clock clock;
@Inject @Config("commitLogDatastoreRetention") Duration maxAge;
@Inject @Parameter(PARAM_DRY_RUN) boolean isDryRun;
@Inject DeleteOldCommitLogsAction() {}
@Inject
@Config("commitLogDatastoreRetention")
Duration maxAge;
@Inject
@Parameter(PARAM_DRY_RUN)
boolean isDryRun;
@Inject
DeleteOldCommitLogsAction() {}
@Override
public void run() {
@@ -138,12 +146,12 @@ public final class DeleteOldCommitLogsAction implements Runnable {
// If it isn't a Key<CommitLogManifest> then it should be an EppResource, which we need to
// load to emit the revisions.
//
Object object = ofy().load().key(key).now();
Object object = auditedOfy().load().key(key).now();
checkNotNull(object, "Received a key to a missing object. key: %s", key);
checkState(
object instanceof EppResource,
"Received a key to an object that isn't EppResource nor CommitLogManifest."
+ " Key: %s object type: %s",
+ " Key: %s object type: %s",
key,
object.getClass().getName());
@@ -224,8 +232,7 @@ public final class DeleteOldCommitLogsAction implements Runnable {
* OK to delete this manifestKey. If even one source returns "false" (meaning "it's not OK to
* delete this manifest") then it won't be deleted.
*/
static class DeleteOldCommitLogsReducer
extends Reducer<Key<CommitLogManifest>, Boolean, Void> {
static class DeleteOldCommitLogsReducer extends Reducer<Key<CommitLogManifest>, Boolean, Void> {
private static final long serialVersionUID = -4918760187627937268L;
@@ -241,12 +248,12 @@ public final class DeleteOldCommitLogsAction implements Runnable {
}
public abstract Status status();
public abstract int numDeleted();
static DeletionResult create(Status status, int numDeleted) {
return
new AutoValue_DeleteOldCommitLogsAction_DeleteOldCommitLogsReducer_DeletionResult(
status, numDeleted);
return new AutoValue_DeleteOldCommitLogsAction_DeleteOldCommitLogsReducer_DeletionResult(
status, numDeleted);
}
}
@@ -257,8 +264,7 @@ public final class DeleteOldCommitLogsAction implements Runnable {
@Override
public void reduce(
final Key<CommitLogManifest> manifestKey,
ReducerInput<Boolean> canDeleteVerdicts) {
final Key<CommitLogManifest> manifestKey, ReducerInput<Boolean> canDeleteVerdicts) {
ImmutableMultiset<Boolean> canDeleteMultiset = ImmutableMultiset.copyOf(canDeleteVerdicts);
if (canDeleteMultiset.count(TRUE) > 1) {
getContext().incrementCounter("commit log manifests incorrectly mapped multiple times");
@@ -267,47 +273,54 @@ public final class DeleteOldCommitLogsAction implements Runnable {
getContext().incrementCounter("commit log manifests referenced multiple times");
}
if (canDeleteMultiset.contains(FALSE)) {
getContext().incrementCounter(
canDeleteMultiset.contains(TRUE)
? "old commit log manifests still referenced"
: "new (or nonexistent) commit log manifests referenced");
getContext().incrementCounter(
"EPP resource revisions handled",
canDeleteMultiset.count(FALSE));
getContext()
.incrementCounter(
canDeleteMultiset.contains(TRUE)
? "old commit log manifests still referenced"
: "new (or nonexistent) commit log manifests referenced");
getContext()
.incrementCounter("EPP resource revisions handled", canDeleteMultiset.count(FALSE));
return;
}
DeletionResult deletionResult = tm().transactNew(() -> {
CommitLogManifest manifest = ofy().load().key(manifestKey).now();
// It is possible that the same manifestKey was run twice, if a shard had to be restarted
// or some weird failure. If this happens, we want to exit immediately.
// Note that this can never happen in dryRun.
if (manifest == null) {
return DeletionResult.create(DeletionResult.Status.ALREADY_DELETED, 0);
}
// Doing a sanity check on the date. This is the only place we use the CommitLogManifest,
// so maybe removing this test will improve performance. However, unless it's proven that
// the performance boost is significant (and we've tested this enough to be sure it never
// happens)- the safty of "let's not delete stuff we need from prod" is more important.
if (manifest.getCommitTime().isAfter(deletionThreshold)) {
return DeletionResult.create(DeletionResult.Status.AFTER_THRESHOLD, 0);
}
Iterable<Key<CommitLogMutation>> commitLogMutationKeys = ofy().load()
.type(CommitLogMutation.class)
.ancestor(manifestKey)
.keys()
.iterable();
ImmutableList<Key<?>> keysToDelete = ImmutableList.<Key<?>>builder()
.addAll(commitLogMutationKeys)
.add(manifestKey)
.build();
// Normally in a dry run we would log the entities that would be deleted, but those can
// number in the millions so we skip the logging.
if (!isDryRun) {
ofy().deleteWithoutBackup().keys(keysToDelete);
}
return DeletionResult.create(DeletionResult.Status.SUCCESS, keysToDelete.size());
});
DeletionResult deletionResult =
tm().transactNew(
() -> {
CommitLogManifest manifest = auditedOfy().load().key(manifestKey).now();
// It is possible that the same manifestKey was run twice, if a shard had to be
// restarted or some weird failure. If this happens, we want to exit
// immediately. Note that this can never happen in dryRun.
if (manifest == null) {
return DeletionResult.create(DeletionResult.Status.ALREADY_DELETED, 0);
}
// Doing a sanity check on the date. This is the only place we use the
// CommitLogManifest, so maybe removing this test will improve performance.
// However, unless it's proven that the performance boost is significant (and
// we've tested this enough to be sure it never happens)- the safety of "let's
// not delete stuff we need from prod" is more important.
if (manifest.getCommitTime().isAfter(deletionThreshold)) {
return DeletionResult.create(DeletionResult.Status.AFTER_THRESHOLD, 0);
}
Iterable<Key<CommitLogMutation>> commitLogMutationKeys =
auditedOfy()
.load()
.type(CommitLogMutation.class)
.ancestor(manifestKey)
.keys()
.iterable();
ImmutableList<Key<?>> keysToDelete =
ImmutableList.<Key<?>>builder()
.addAll(commitLogMutationKeys)
.add(manifestKey)
.build();
// Normally in a dry run we would log the entities that would be deleted, but
// those can number in the millions so we skip the logging.
if (!isDryRun) {
auditedOfy().deleteWithoutBackup().keys(keysToDelete);
}
return DeletionResult.create(
DeletionResult.Status.SUCCESS, keysToDelete.size());
});
switch (deletionResult.status()) {
case SUCCESS:

View File

@@ -25,7 +25,7 @@ import static google.registry.backup.BackupUtils.GcsMetadataKeys.NUM_TRANSACTION
import static google.registry.backup.BackupUtils.GcsMetadataKeys.UPPER_BOUND_CHECKPOINT;
import static google.registry.backup.BackupUtils.serializeEntity;
import static google.registry.model.ofy.CommitLogBucket.getBucketKey;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.util.DateTimeUtils.START_OF_TIME;
import static google.registry.util.DateTimeUtils.isAtOrAfter;
import static java.nio.channels.Channels.newOutputStream;
@@ -89,11 +89,14 @@ public final class ExportCommitLogDiffAction implements Runnable {
checkArgument(lowerCheckpointTime.isBefore(upperCheckpointTime));
// Load the boundary checkpoints - lower is exclusive and may not exist (on the first export,
// when lowerCheckpointTime is START_OF_TIME), whereas the upper is inclusive and must exist.
CommitLogCheckpoint lowerCheckpoint = lowerCheckpointTime.isAfter(START_OF_TIME)
? verifyNotNull(ofy().load().key(CommitLogCheckpoint.createKey(lowerCheckpointTime)).now())
: null;
CommitLogCheckpoint lowerCheckpoint =
lowerCheckpointTime.isAfter(START_OF_TIME)
? verifyNotNull(
auditedOfy().load().key(CommitLogCheckpoint.createKey(lowerCheckpointTime)).now())
: null;
CommitLogCheckpoint upperCheckpoint =
verifyNotNull(ofy().load().key(CommitLogCheckpoint.createKey(upperCheckpointTime)).now());
verifyNotNull(
auditedOfy().load().key(CommitLogCheckpoint.createKey(upperCheckpointTime)).now());
// Load the keys of all the manifests to include in this diff.
List<Key<CommitLogManifest>> sortedKeys = loadAllDiffKeys(lowerCheckpoint, upperCheckpoint);
@@ -117,7 +120,7 @@ public final class ExportCommitLogDiffAction implements Runnable {
// asynchronously load the entities for the next one.
List<List<Key<CommitLogManifest>>> keyChunks = partition(sortedKeys, batchSize);
// Objectify's map return type is asynchronous. Calling .values() will block until it loads.
Map<?, CommitLogManifest> nextChunkToExport = ofy().load().keys(keyChunks.get(0));
Map<?, CommitLogManifest> nextChunkToExport = auditedOfy().load().keys(keyChunks.get(0));
for (int i = 0; i < keyChunks.size(); i++) {
// Force the async load to finish.
Collection<CommitLogManifest> chunkValues = nextChunkToExport.values();
@@ -125,10 +128,10 @@ public final class ExportCommitLogDiffAction implements Runnable {
// Since there is no hard bound on how much data this might be, take care not to let the
// Objectify session cache fill up and potentially run out of memory. This is the only safe
// point to do this since at this point there is no async load in progress.
ofy().clearSessionCache();
auditedOfy().clearSessionCache();
// Kick off the next async load, which can happen in parallel to the current GCS export.
if (i + 1 < keyChunks.size()) {
nextChunkToExport = ofy().load().keys(keyChunks.get(i + 1));
nextChunkToExport = auditedOfy().load().keys(keyChunks.get(i + 1));
}
exportChunk(gcsStream, chunkValues);
logger.atInfo().log("Exported %d manifests", chunkValues.size());
@@ -192,7 +195,8 @@ public final class ExportCommitLogDiffAction implements Runnable {
return ImmutableSet.of();
}
Key<CommitLogBucket> bucketKey = getBucketKey(bucketNum);
return ofy().load()
return auditedOfy()
.load()
.type(CommitLogManifest.class)
.ancestor(bucketKey)
.filterKey(">=", CommitLogManifest.createKey(bucketKey, lowerBound))
@@ -208,7 +212,7 @@ public final class ExportCommitLogDiffAction implements Runnable {
new ImmutableList.Builder<>();
for (CommitLogManifest manifest : chunk) {
entities.add(ImmutableList.of(manifest));
entities.add(ofy().load().type(CommitLogMutation.class).ancestor(manifest));
entities.add(auditedOfy().load().type(CommitLogMutation.class).ancestor(manifest));
}
for (ImmutableObject entity : concat(entities.build())) {
serializeEntity(entity, gcsStream);

View File

@@ -16,7 +16,7 @@ package google.registry.backup;
import static google.registry.backup.ExportCommitLogDiffAction.DIFF_FILE_PREFIX;
import static google.registry.model.ofy.EntityWritePriorities.getEntityPriority;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static javax.servlet.http.HttpServletResponse.SC_NO_CONTENT;
import static org.joda.time.Duration.standardHours;
@@ -151,7 +151,7 @@ public class ReplayCommitLogsToSqlAction implements Runnable {
}
private void handleEntityPut(Entity entity) {
Object ofyPojo = ofy().toPojo(entity);
Object ofyPojo = auditedOfy().toPojo(entity);
if (ofyPojo instanceof DatastoreEntity) {
DatastoreEntity datastoreEntity = (DatastoreEntity) ofyPojo;
datastoreEntity

View File

@@ -18,7 +18,7 @@ import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static com.google.common.collect.Iterators.peekingIterator;
import static google.registry.backup.BackupUtils.createDeserializingIterator;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.Entity;
@@ -146,10 +146,10 @@ public class RestoreCommitLogsAction implements Runnable {
private CommitLogManifest restoreOneTransaction(PeekingIterator<ImmutableObject> commitLogs) {
final CommitLogManifest manifest = (CommitLogManifest) commitLogs.next();
Result<?> deleteResult = deleteAsync(manifest.getDeletions());
List<Entity> entitiesToSave = Lists.newArrayList(ofy().save().toEntity(manifest));
List<Entity> entitiesToSave = Lists.newArrayList(auditedOfy().save().toEntity(manifest));
while (commitLogs.hasNext() && commitLogs.peek() instanceof CommitLogMutation) {
CommitLogMutation mutation = (CommitLogMutation) commitLogs.next();
entitiesToSave.add(ofy().save().toEntity(mutation));
entitiesToSave.add(auditedOfy().save().toEntity(mutation));
entitiesToSave.add(EntityTranslator.createFromPbBytes(mutation.getEntityProtoBytes()));
}
saveRaw(entitiesToSave);
@@ -176,7 +176,8 @@ public class RestoreCommitLogsAction implements Runnable {
return;
}
retrier.callWithRetry(
() -> ofy().saveWithoutBackup().entities(objectsToSave).now(), RuntimeException.class);
() -> auditedOfy().saveWithoutBackup().entities(objectsToSave).now(),
RuntimeException.class);
}
private Result<?> deleteAsync(Set<Key<?>> keysToDelete) {
@@ -185,7 +186,7 @@ public class RestoreCommitLogsAction implements Runnable {
}
return dryRun || keysToDelete.isEmpty()
? new ResultNow<Void>(null)
: ofy().deleteWithoutBackup().keys(keysToDelete);
: auditedOfy().deleteWithoutBackup().keys(keysToDelete);
}
}

View File

@@ -47,7 +47,7 @@ import javax.annotation.Nullable;
*
* <ul>
* <li>Convert an Objectify entity to a Datastore {@link Entity}: {@code
* ofy().save().toEntity(..)}
* auditedOfy().save().toEntity(..)}
* <li>Entity is serializable, but the more efficient approach is to convert an Entity to a
* ProtocolBuffer ({@link com.google.storage.onestore.v3.OnestoreEntity.EntityProto}) and then
* to raw bytes.

View File

@@ -34,7 +34,7 @@ import static google.registry.model.ResourceTransferUtils.denyPendingTransfer;
import static google.registry.model.ResourceTransferUtils.handlePendingTransferOnDelete;
import static google.registry.model.ResourceTransferUtils.updateForeignKeyIndexDeletionTime;
import static google.registry.model.eppcommon.StatusValue.PENDING_DELETE;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.model.reporting.HistoryEntry.Type.CONTACT_DELETE;
import static google.registry.model.reporting.HistoryEntry.Type.CONTACT_DELETE_FAILURE;
import static google.registry.model.reporting.HistoryEntry.Type.HOST_DELETE;
@@ -109,6 +109,7 @@ import org.joda.time.Duration;
* A mapreduce that processes batch asynchronous deletions of contact and host resources by mapping
* over all domains and checking for any references to the contacts/hosts in pending deletion.
*/
@Deprecated
@Action(
service = Action.Service.BACKEND,
path = "/_dr/task/deleteContactsAndHosts",
@@ -335,7 +336,7 @@ public class DeleteContactsAndHostsAction implements Runnable {
DeletionRequest deletionRequest, boolean hasNoActiveReferences) {
DateTime now = tm().getTransactionTime();
EppResource resource =
ofy().load().key(deletionRequest.key()).now().cloneProjectedAtTime(now);
auditedOfy().load().key(deletionRequest.key()).now().cloneProjectedAtTime(now);
// Double-check transactionally that the resource is still active and in PENDING_DELETE.
if (!doesResourceStateAllowDeletion(resource, now)) {
return DeletionResult.create(Type.ERRORED, "");
@@ -369,11 +370,10 @@ public class DeleteContactsAndHostsAction implements Runnable {
: "it was transferred prior to deletion");
HistoryEntry historyEntry =
new HistoryEntry.Builder()
HistoryEntry.createBuilderForResource(resource)
.setClientId(deletionRequest.requestingClientId())
.setModificationTime(now)
.setType(getHistoryEntryType(resource, deleteAllowed))
.setParent(deletionRequest.key())
.build();
PollMessage.OneTime pollMessage =
@@ -408,7 +408,9 @@ public class DeleteContactsAndHostsAction implements Runnable {
} else {
resourceToSave = resource.asBuilder().removeStatusValue(PENDING_DELETE).build();
}
ofy().save().<ImmutableObject>entities(resourceToSave, historyEntry, pollMessage);
auditedOfy()
.save()
.<ImmutableObject>entities(resourceToSave, historyEntry.asHistoryEntry(), pollMessage);
return DeletionResult.create(
deleteAllowed ? Type.DELETED : Type.NOT_DELETED, pollMessageText);
}
@@ -525,7 +527,8 @@ public class DeleteContactsAndHostsAction implements Runnable {
Key.create(
checkNotNull(params.get(PARAM_RESOURCE_KEY), "Resource to delete not specified"));
EppResource resource =
checkNotNull(ofy().load().key(resourceKey).now(), "Resource to delete doesn't exist");
checkNotNull(
auditedOfy().load().key(resourceKey).now(), "Resource to delete doesn't exist");
checkState(
resource instanceof ContactResource || resource instanceof HostResource,
"Cannot delete a %s via this action",

View File

@@ -18,7 +18,7 @@ import static com.google.common.base.Preconditions.checkState;
import static google.registry.config.RegistryEnvironment.PRODUCTION;
import static google.registry.mapreduce.MapreduceRunner.PARAM_DRY_RUN;
import static google.registry.mapreduce.inputs.EppResourceInputs.createEntityInput;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.request.Action.Method.POST;
@@ -125,12 +125,11 @@ public class DeleteLoadTestDataAction implements Runnable {
Key.create(EppResourceIndex.create(Key.create(resource)));
final Key<? extends ForeignKeyIndex<?>> fki = ForeignKeyIndex.createKey(resource);
int numEntitiesDeleted =
tm()
.transact(
tm().transact(
() -> {
// This ancestor query selects all descendant entities.
List<Key<Object>> resourceAndDependentKeys =
ofy().load().ancestor(resource).keys().list();
auditedOfy().load().ancestor(resource).keys().list();
ImmutableSet<Key<?>> allKeys =
new ImmutableSet.Builder<Key<?>>()
.add(fki)
@@ -140,7 +139,7 @@ public class DeleteLoadTestDataAction implements Runnable {
if (isDryRun) {
logger.atInfo().log("Would hard-delete the following entities: %s", allKeys);
} else {
ofy().deleteWithoutBackup().keys(allKeys);
auditedOfy().deleteWithoutBackup().keys(allKeys);
}
return allKeys.size();
});

View File

@@ -20,7 +20,7 @@ import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.config.RegistryEnvironment.PRODUCTION;
import static google.registry.mapreduce.MapreduceRunner.PARAM_DRY_RUN;
import static google.registry.model.ResourceTransferUtils.updateForeignKeyIndexDeletionTime;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.model.registry.Registries.getTldsOfType;
import static google.registry.model.reporting.HistoryEntry.Type.DOMAIN_DELETE;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
@@ -44,11 +44,11 @@ import google.registry.mapreduce.MapreduceRunner;
import google.registry.mapreduce.inputs.EppResourceInputs;
import google.registry.model.EppResourceUtils;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.index.EppResourceIndex;
import google.registry.model.index.ForeignKeyIndex;
import google.registry.model.registry.Registry;
import google.registry.model.registry.Registry.TldType;
import google.registry.model.reporting.HistoryEntry;
import google.registry.request.Action;
import google.registry.request.Parameter;
import google.registry.request.Response;
@@ -166,7 +166,7 @@ public class DeleteProberDataAction implements Runnable {
}
private void deleteDomain(final Key<DomainBase> domainKey) {
final DomainBase domain = ofy().load().key(domainKey).now();
final DomainBase domain = auditedOfy().load().key(domainKey).now();
DateTime now = DateTime.now(UTC);
@@ -220,14 +220,13 @@ public class DeleteProberDataAction implements Runnable {
final Key<? extends ForeignKeyIndex<?>> fki = ForeignKeyIndex.createKey(domain);
int entitiesDeleted =
tm()
.transact(
tm().transact(
() -> {
// This ancestor query selects all descendant HistoryEntries, BillingEvents,
// PollMessages,
// and TLD-specific entities, as well as the domain itself.
List<Key<Object>> domainAndDependentKeys =
ofy().load().ancestor(domainKey).keys().list();
auditedOfy().load().ancestor(domainKey).keys().list();
ImmutableSet<Key<?>> allKeys =
new ImmutableSet.Builder<Key<?>>()
.add(fki)
@@ -237,7 +236,7 @@ public class DeleteProberDataAction implements Runnable {
if (isDryRun) {
logger.atInfo().log("Would hard-delete the following entities: %s", allKeys);
} else {
ofy().deleteWithoutBackup().keys(allKeys);
auditedOfy().deleteWithoutBackup().keys(allKeys);
}
return allKeys.size();
});
@@ -254,9 +253,9 @@ public class DeleteProberDataAction implements Runnable {
.setDeletionTime(tm().getTransactionTime())
.setStatusValues(null)
.build();
HistoryEntry historyEntry =
new HistoryEntry.Builder()
.setParent(domain)
DomainHistory historyEntry =
new DomainHistory.Builder()
.setDomain(domain)
.setType(DOMAIN_DELETE)
.setModificationTime(tm().getTransactionTime())
.setBySuperuser(true)
@@ -264,11 +263,9 @@ public class DeleteProberDataAction implements Runnable {
.setClientId(registryAdminClientId)
.build();
// Note that we don't bother handling grace periods, billing events, pending
// transfers,
// poll messages, or auto-renews because these will all be hard-deleted the next
// time the
// mapreduce runs anyway.
ofy().save().entities(deletedDomain, historyEntry);
// transfers, poll messages, or auto-renews because these will all be hard-deleted
// the next time the mapreduce runs anyway.
tm().putAll(deletedDomain, historyEntry);
updateForeignKeyIndexDeletionTime(deletedDomain);
dnsQueue.addDomainRefreshTask(deletedDomain.getDomainName());
});

View File

@@ -49,6 +49,7 @@ import google.registry.model.billing.BillingEvent.OneTime;
import google.registry.model.billing.BillingEvent.Recurring;
import google.registry.model.common.Cursor;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.Period;
import google.registry.model.registry.Registry;
import google.registry.model.reporting.DomainTransactionRecord;
@@ -91,7 +92,8 @@ public class ExpandRecurringBillingEventsAction implements Runnable {
@Override
public void run() {
Cursor cursor = ofy().load().key(Cursor.createGlobalKey(RECURRING_BILLING)).now();
Cursor cursor =
tm().loadByKeyIfPresent(Cursor.createGlobalVKey(RECURRING_BILLING)).orElse(null);
DateTime executeTime = clock.nowUtc();
DateTime persistedCursorTime = (cursor == null ? START_OF_TIME : cursor.getCursorTime());
DateTime cursorTime = cursorTimeParam.orElse(persistedCursorTime);
@@ -187,12 +189,14 @@ public class ExpandRecurringBillingEventsAction implements Runnable {
// an event persisted.
for (DateTime billingTime : difference(billingTimes, existingBillingTimes)) {
// Construct a new HistoryEntry that parents over the OneTime
HistoryEntry historyEntry =
new HistoryEntry.Builder()
DomainHistory historyEntry =
new DomainHistory.Builder()
.setBySuperuser(false)
.setClientId(recurring.getClientId())
.setModificationTime(tm().getTransactionTime())
.setParent(domainKey)
// TODO (jianglai): modify this to use setDomain instead when
// converting this action to be SQL-aware.
.setDomainRepoId(domainKey.getName())
.setPeriod(Period.create(1, YEARS))
.setReason(
"Domain autorenewal by ExpandRecurringBillingEventsAction")

View File

@@ -15,7 +15,7 @@
package google.registry.batch;
import static google.registry.mapreduce.MapreduceRunner.PARAM_FAST;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.appengine.tools.mapreduce.Mapper;
@@ -104,13 +104,13 @@ public class ResaveAllEppResourcesAction implements Runnable {
boolean resaved =
tm().transact(
() -> {
EppResource originalResource = ofy().load().key(resourceKey).now();
EppResource originalResource = auditedOfy().load().key(resourceKey).now();
EppResource projectedResource =
originalResource.cloneProjectedAtTime(tm().getTransactionTime());
if (isFast && originalResource.equals(projectedResource)) {
return false;
} else {
ofy().save().entity(projectedResource).now();
auditedOfy().save().entity(projectedResource).now();
return true;
}
});

View File

@@ -19,6 +19,7 @@ import static javax.servlet.http.HttpServletResponse.SC_FORBIDDEN;
import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
import static javax.servlet.http.HttpServletResponse.SC_OK;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.common.flogger.FluentLogger;
import google.registry.config.RegistryConfig.Config;
@@ -28,10 +29,11 @@ import google.registry.request.Response;
import google.registry.request.auth.Auth;
import google.registry.util.Retrier;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.function.Supplier;
import javax.inject.Inject;
import org.flywaydb.core.api.FlywayException;
/**
* Wipes out all Cloud SQL data in a Nomulus GCP environment.
@@ -80,13 +82,13 @@ public class WipeOutCloudSqlAction implements Runnable {
try {
retrier.callWithRetry(
() -> {
try (Connection conn = connectionSupplier.get();
Statement statement = conn.createStatement()) {
statement.execute("drop owned by schema_deployer;");
try (Connection conn = connectionSupplier.get()) {
dropAllTables(conn, listTables(conn));
dropAllSequences(conn, listSequences(conn));
}
return null;
},
e -> !(e instanceof FlywayException));
e -> !(e instanceof SQLException));
response.setStatus(SC_OK);
response.setPayload("Wiped out Cloud SQL in " + projectId);
} catch (RuntimeException e) {
@@ -95,4 +97,69 @@ public class WipeOutCloudSqlAction implements Runnable {
response.setPayload("Failed to wipe out Cloud SQL in " + projectId);
}
}
/** Returns a list of all tables in the public schema of a Postgresql database. */
static ImmutableList<String> listTables(Connection connection) throws SQLException {
try (ResultSet resultSet =
connection.getMetaData().getTables(null, null, null, new String[] {"TABLE"})) {
ImmutableList.Builder<String> tables = new ImmutableList.Builder<>();
while (resultSet.next()) {
String schema = resultSet.getString("TABLE_SCHEM");
if (schema == null || !schema.equalsIgnoreCase("public")) {
continue;
}
String tableName = resultSet.getString("TABLE_NAME");
tables.add("public.\"" + tableName + "\"");
}
return tables.build();
}
}
static void dropAllTables(Connection conn, ImmutableList<String> tables) throws SQLException {
if (tables.isEmpty()) {
return;
}
try (Statement statement = conn.createStatement()) {
for (String table : tables) {
statement.addBatch(String.format("DROP TABLE IF EXISTS %s CASCADE;", table));
}
for (int code : statement.executeBatch()) {
if (code == Statement.EXECUTE_FAILED) {
throw new RuntimeException("Failed to drop some tables. Please check.");
}
}
}
}
/** Returns a list of all sequences in a Postgresql database. */
static ImmutableList<String> listSequences(Connection conn) throws SQLException {
try (Statement statement = conn.createStatement();
ResultSet resultSet =
statement.executeQuery("SELECT c.relname FROM pg_class c WHERE c.relkind = 'S';")) {
ImmutableList.Builder<String> sequences = new ImmutableList.Builder<>();
while (resultSet.next()) {
sequences.add('\"' + resultSet.getString(1) + '\"');
}
return sequences.build();
}
}
static void dropAllSequences(Connection conn, ImmutableList<String> sequences)
throws SQLException {
if (sequences.isEmpty()) {
return;
}
try (Statement statement = conn.createStatement()) {
for (String sequence : sequences) {
statement.addBatch(String.format("DROP SEQUENCE IF EXISTS %s CASCADE;", sequence));
}
for (int code : statement.executeBatch()) {
if (code == Statement.EXECUTE_FAILED) {
throw new RuntimeException("Failed to drop some sequences. Please check.");
}
}
}
}
}

View File

@@ -15,6 +15,7 @@
package google.registry.batch;
import static com.google.common.net.MediaType.PLAIN_TEXT_UTF_8;
import static google.registry.beam.BeamUtils.createJobName;
import static javax.servlet.http.HttpServletResponse.SC_FORBIDDEN;
import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
import static javax.servlet.http.HttpServletResponse.SC_OK;
@@ -30,9 +31,8 @@ import google.registry.config.RegistryConfig.Config;
import google.registry.request.Action;
import google.registry.request.Response;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import javax.inject.Inject;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
/**
* Wipes out all Cloud Datastore data in a Nomulus GCP environment.
@@ -58,17 +58,20 @@ public class WipeoutDatastoreAction implements Runnable {
private final Response response;
private final Dataflow dataflow;
private final String stagingBucketUrl;
private final Clock clock;
@Inject
WipeoutDatastoreAction(
@Config("projectId") String projectId,
@Config("defaultJobRegion") String jobRegion,
@Config("beamStagingBucketUrl") String stagingBucketUrl,
Clock clock,
Response response,
Dataflow dataflow) {
this.projectId = projectId;
this.jobRegion = jobRegion;
this.stagingBucketUrl = stagingBucketUrl;
this.clock = clock;
this.response = response;
this.dataflow = dataflow;
}
@@ -86,10 +89,7 @@ public class WipeoutDatastoreAction implements Runnable {
try {
LaunchFlexTemplateParameter parameters =
new LaunchFlexTemplateParameter()
// Job name must be unique and in [-a-z0-9].
.setJobName(
"bulk-delete-datastore-"
+ DateTime.now(DateTimeZone.UTC).toString("yyyy-MM-dd'T'HH-mm-ss'Z'"))
.setJobName(createJobName("bulk-delete-datastore-", clock))
.setContainerSpecGcsPath(
String.format("%s/%s_metadata.json", stagingBucketUrl, PIPELINE_NAME))
.setParameters(ImmutableMap.of("kindsToDelete", "*"));

View File

@@ -14,10 +14,14 @@
package google.registry.beam;
import static com.google.common.base.Preconditions.checkArgument;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableList;
import com.google.common.io.Resources;
import google.registry.util.Clock;
import google.registry.util.ResourceUtils;
import java.util.regex.Pattern;
import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord;
@@ -41,8 +45,7 @@ public class BeamUtils {
ImmutableList<String> fieldNames, SchemaAndRecord schemaAndRecord) {
GenericRecord record = schemaAndRecord.getRecord();
ImmutableList<String> nullFields =
fieldNames
.stream()
fieldNames.stream()
.filter(fieldName -> record.get(fieldName) == null)
.collect(ImmutableList.toImmutableList());
String missingFieldList = Joiner.on(", ").join(nullFields);
@@ -61,4 +64,19 @@ public class BeamUtils {
public static String getQueryFromFile(Class<?> clazz, String filename) {
return ResourceUtils.readResourceUtf8(Resources.getResource(clazz, "sql/" + filename));
}
/** Creates a beam job name and validates that it conforms to the requirements. */
public static String createJobName(String prefix, Clock clock) {
// Flex template job name must be unique and consists of only characters [-a-z0-9], starting
// with a letter and ending with a letter or number. So we replace the "T" and "Z" in ISO 8601
// with lowercase letters.
String jobName =
String.format("%s-%s", prefix, clock.nowUtc().toString("yyyy-MM-dd't'HH-mm-ss'z'"));
checkArgument(
Pattern.compile("^[a-z][-a-z0-9]*[a-z0-9]*").matcher(jobName).matches(),
"The job name %s is illegal, it consists of only characters [-a-z0-9], "
+ "starting with a letter and ending with a letter or number,",
jobName);
return jobName;
}
}

View File

@@ -21,21 +21,30 @@ import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Streams;
import google.registry.backup.AppEngineEnvironment;
import google.registry.beam.common.RegistryQuery.QueryComposerFactory;
import google.registry.beam.common.RegistryQuery.RegistryQueryFactory;
import google.registry.model.ofy.ObjectifyService;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.persistence.transaction.TransactionManagerFactory;
import java.io.Serializable;
import java.util.Objects;
import java.util.concurrent.ThreadLocalRandom;
import javax.persistence.criteria.CriteriaQuery;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.GroupIntoBatches;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Reshuffle;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.util.ShardedKey;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
/**
@@ -51,10 +60,127 @@ public final class RegistryJpaIO {
private RegistryJpaIO() {}
public static <R> Read<R, R> read(QueryComposerFactory<R> queryFactory) {
return Read.<R, R>builder().queryFactory(queryFactory).build();
}
public static <R, T> Read<R, T> read(
QueryComposerFactory<R> queryFactory, SerializableFunction<R, T> resultMapper) {
return Read.<R, T>builder().queryFactory(queryFactory).resultMapper(resultMapper).build();
}
/**
* Returns a {@link Read} connector based on the given {@code jpql} query string.
*
* <p>User should take care to prevent sql-injection attacks.
*/
public static <R, T> Read<R, T> read(String jpql, SerializableFunction<R, T> resultMapper) {
return Read.<R, T>builder().jpqlQueryFactory(jpql).resultMapper(resultMapper).build();
}
public static <T> Write<T> write() {
return Write.<T>builder().build();
}
/**
* A {@link PTransform transform} that transactionally executes a JPA {@link CriteriaQuery} and
* adds the results to the BEAM pipeline. Users have the option to transform the results before
* sending them to the next stages.
*/
@AutoValue
public abstract static class Read<R, T> extends PTransform<PBegin, PCollection<T>> {
public static final String DEFAULT_NAME = "RegistryJpaIO.Read";
abstract String name();
abstract RegistryQueryFactory<R> queryFactory();
abstract SerializableFunction<R, T> resultMapper();
abstract Coder<T> coder();
abstract Builder<R, T> toBuilder();
@Override
@SuppressWarnings("deprecation") // Reshuffle still recommended by GCP.
public PCollection<T> expand(PBegin input) {
return input
.apply("Starting " + name(), Create.of((Void) null))
.apply(
"Run query for " + name(),
ParDo.of(new QueryRunner<>(queryFactory(), resultMapper())))
.setCoder(coder())
.apply("Reshuffle", Reshuffle.viaRandomKey());
}
public Read<R, T> withName(String name) {
return toBuilder().name(name).build();
}
public Read<R, T> withResultMapper(SerializableFunction<R, T> mapper) {
return toBuilder().resultMapper(mapper).build();
}
public Read<R, T> withCoder(Coder<T> coder) {
return toBuilder().coder(coder).build();
}
static <R, T> Builder<R, T> builder() {
return new AutoValue_RegistryJpaIO_Read.Builder()
.name(DEFAULT_NAME)
.resultMapper(x -> x)
.coder(SerializableCoder.of(Serializable.class));
}
@AutoValue.Builder
public abstract static class Builder<R, T> {
abstract Builder<R, T> name(String name);
abstract Builder<R, T> queryFactory(RegistryQueryFactory<R> queryFactory);
abstract Builder<R, T> resultMapper(SerializableFunction<R, T> mapper);
abstract Builder<R, T> coder(Coder coder);
abstract Read<R, T> build();
Builder<R, T> queryFactory(QueryComposerFactory<R> queryFactory) {
return queryFactory(RegistryQuery.createQueryFactory(queryFactory));
}
Builder<R, T> jpqlQueryFactory(String jpql) {
return queryFactory(RegistryQuery.createQueryFactory(jpql));
}
}
static class QueryRunner<R, T> extends DoFn<Void, T> {
private final RegistryQueryFactory<R> queryFactory;
private final SerializableFunction<R, T> resultMapper;
QueryRunner(RegistryQueryFactory<R> queryFactory, SerializableFunction<R, T> resultMapper) {
this.queryFactory = queryFactory;
this.resultMapper = resultMapper;
}
@ProcessElement
public void processElement(OutputReceiver<T> outputReceiver) {
// AppEngineEnvironment is need for handling VKeys, which involve Ofy keys. Unlike
// SqlBatchWriter, it is unnecessary to initialize ObjectifyService in this class.
try (AppEngineEnvironment env = new AppEngineEnvironment()) {
// TODO(b/187210388): JpaTransactionManager should support non-transactional query.
jpaTm()
.transactNoRetry(
() ->
queryFactory.apply(jpaTm()).stream()
.map(resultMapper::apply)
.forEach(outputReceiver::output));
}
}
}
}
/**
* A {@link PTransform transform} that writes a PCollection of entities to the SQL database using
* the {@link JpaTransactionManager}.
@@ -182,8 +308,9 @@ public final class RegistryJpaIO {
@Setup
public void setup() {
// Below is needed as long as Objectify keys are still involved in the handling of SQL
// entities (e.g., in VKeys).
// AppEngineEnvironment is needed as long as Objectify keys are still involved in the handling
// of SQL entities (e.g., in VKeys). ObjectifyService needs to be initialized when conversion
// between Ofy entity and Datastore entity is needed.
try (AppEngineEnvironment env = new AppEngineEnvironment()) {
ObjectifyService.initOfy();
}

View File

@@ -0,0 +1,89 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.common;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.persistence.transaction.QueryComposer;
import java.util.stream.Stream;
import javax.persistence.EntityManager;
import javax.persistence.Query;
import org.apache.beam.sdk.transforms.SerializableFunction;
/** Interface for query instances used by {@link RegistryJpaIO.Read}. */
public interface RegistryQuery<T> {
Stream<T> stream();
/** Factory for {@link RegistryQuery}. */
interface RegistryQueryFactory<T>
extends SerializableFunction<JpaTransactionManager, RegistryQuery<T>> {}
// TODO(mmuller): Consider detached JpaQueryComposer that works with any JpaTransactionManager
// instance, i.e., change composer.buildQuery() to composer.buildQuery(JpaTransactionManager).
// This way QueryComposer becomes reusable and serializable (at least with Hibernate), and this
// interface would no longer be necessary.
interface QueryComposerFactory<T>
extends SerializableFunction<JpaTransactionManager, QueryComposer<T>> {}
/**
* Returns a {@link RegistryQueryFactory} that creates a JPQL query from constant text.
*
* @param <T> Type of each row in the result set, {@link Object} in single-select queries, and
* {@code Object[]} in multi-select queries.
*/
@SuppressWarnings("unchecked") // query.getResultStream: jpa api uses raw type
static <T> RegistryQueryFactory<T> createQueryFactory(String jpql) {
return (JpaTransactionManager jpa) ->
() -> {
EntityManager entityManager = jpa.getEntityManager();
Query query = entityManager.createQuery(jpql);
return query.getResultStream().map(e -> detach(entityManager, e));
};
}
static <T> RegistryQueryFactory<T> createQueryFactory(
QueryComposerFactory<T> queryComposerFactory) {
return (JpaTransactionManager jpa) ->
() -> queryComposerFactory.apply(jpa).withAutoDetachOnLoad(true).stream();
}
/**
* Removes an object from the JPA session cache if applicable.
*
* @param object An object that represents a row in the result set. It may be a JPA entity, a
* non-entity object, or an array that holds JPA entities and/or non-entities.
*/
static <T> T detach(EntityManager entityManager, T object) {
if (object.getClass().isArray()) {
for (Object arrayElement : (Object[]) object) {
detachObject(entityManager, arrayElement);
}
} else {
detachObject(entityManager, object);
}
return object;
}
static void detachObject(EntityManager entityManager, Object object) {
Class<?> objectClass = object.getClass();
if (objectClass.isPrimitive() || objectClass == String.class) {
return;
}
try {
entityManager.detach(object);
} catch (IllegalArgumentException e) {
// Not an entity. Do nothing.
}
}
}

View File

@@ -87,20 +87,18 @@ public class BulkDeleteDatastorePipeline {
private final BulkDeletePipelineOptions options;
private final Pipeline pipeline;
BulkDeleteDatastorePipeline(BulkDeletePipelineOptions options) {
this.options = options;
pipeline = Pipeline.create(options);
}
public void run() {
setupPipeline();
Pipeline pipeline = Pipeline.create(options);
setupPipeline(pipeline);
pipeline.run();
}
@SuppressWarnings("deprecation") // org.apache.beam.sdk.transforms.Reshuffle
private void setupPipeline() {
private void setupPipeline(Pipeline pipeline) {
checkState(
!FORBIDDEN_PROJECTS.contains(options.getProject()),
"Bulk delete is forbidden in %s",

View File

@@ -1,201 +0,0 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.initsql;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkState;
import static com.google.common.base.Strings.isNullOrEmpty;
import com.google.common.base.Splitter;
import dagger.Component;
import dagger.Lazy;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.keyring.kms.KmsModule;
import google.registry.persistence.PersistenceModule;
import google.registry.persistence.PersistenceModule.JdbcJpaTm;
import google.registry.persistence.PersistenceModule.SocketFactoryJpaTm;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.privileges.secretmanager.SecretManagerModule;
import google.registry.util.UtilsModule;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.channels.Channels;
import java.nio.charset.StandardCharsets;
import java.util.List;
import javax.annotation.Nullable;
import javax.inject.Singleton;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.ResourceId;
/**
* Provides bindings for {@link JpaTransactionManager} to Cloud SQL.
*
* <p>This module is intended for use in BEAM pipelines, and uses a BEAM utility to access GCS like
* a regular file system.
*/
@Module
public class BeamJpaModule {
private static final String GCS_SCHEME = "gs://";
@Nullable private final String sqlAccessInfoFile;
@Nullable private final String cloudKmsProjectId;
@Nullable private final TransactionIsolationLevel isolationOverride;
/**
* Constructs a new instance of {@link BeamJpaModule}.
*
* <p>Note: it is an unfortunately necessary antipattern to check for the validity of
* sqlAccessInfoFile in {@link #provideCloudSqlAccessInfo} rather than in the constructor.
* Unfortunately, this is a restriction imposed upon us by Dagger. Specifically, because we use
* this in at least one 1 {@link google.registry.tools.RegistryTool} command(s), it must be
* instantiated in {@code google.registry.tools.RegistryToolComponent} for all possible commands;
* Dagger doesn't permit it to ever be null. For the vast majority of commands, it will never be
* used (so a null credential file path is fine in those cases).
*
* @param sqlAccessInfoFile the path to a Cloud SQL credential file. This must refer to either a
* real encrypted file on GCS as returned by {@link
* BackupPaths#getCloudSQLCredentialFilePatterns} or an unencrypted file on local filesystem
* with credentials to a test database.
* @param cloudKmsProjectId the GCP project where the credential decryption key can be found
* @param isolationOverride the desired Transaction Isolation level for all JDBC connections
*/
public BeamJpaModule(
@Nullable String sqlAccessInfoFile,
@Nullable String cloudKmsProjectId,
@Nullable TransactionIsolationLevel isolationOverride) {
this.sqlAccessInfoFile = sqlAccessInfoFile;
this.cloudKmsProjectId = cloudKmsProjectId;
this.isolationOverride = isolationOverride;
}
public BeamJpaModule(@Nullable String sqlAccessInfoFile, @Nullable String cloudKmsProjectId) {
this(sqlAccessInfoFile, cloudKmsProjectId, null);
}
/** Returns true if the credential file is on GCS (and therefore expected to be encrypted). */
private boolean isCloudSqlCredential() {
return sqlAccessInfoFile.startsWith(GCS_SCHEME);
}
@Provides
@Singleton
SqlAccessInfo provideCloudSqlAccessInfo(Lazy<CloudSqlCredentialDecryptor> lazyDecryptor) {
checkArgument(!isNullOrEmpty(sqlAccessInfoFile), "Null or empty credentialFilePath");
String line = readOnlyLineFromCredentialFile();
if (isCloudSqlCredential()) {
line = lazyDecryptor.get().decrypt(line);
}
// See ./BackupPaths.java for explanation of the line format.
List<String> parts = Splitter.on(' ').splitToList(line.trim());
checkState(parts.size() == 3, "Expecting three phrases in %s", line);
if (isCloudSqlCredential()) {
return SqlAccessInfo.createCloudSqlAccessInfo(parts.get(0), parts.get(1), parts.get(2));
} else {
return SqlAccessInfo.createLocalSqlAccessInfo(parts.get(0), parts.get(1), parts.get(2));
}
}
String readOnlyLineFromCredentialFile() {
try {
ResourceId resourceId = FileSystems.matchSingleFileSpec(sqlAccessInfoFile).resourceId();
try (BufferedReader reader =
new BufferedReader(
new InputStreamReader(
Channels.newInputStream(FileSystems.open(resourceId)), StandardCharsets.UTF_8))) {
return reader.readLine();
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
@Provides
@Config("beamCloudSqlJdbcUrl")
String provideJdbcUrl(SqlAccessInfo sqlAccessInfo) {
return sqlAccessInfo.jdbcUrl();
}
@Provides
@Config("beamCloudSqlInstanceConnectionName")
String provideSqlInstanceName(SqlAccessInfo sqlAccessInfo) {
return sqlAccessInfo
.cloudSqlInstanceName()
.orElseThrow(() -> new IllegalStateException("Cloud SQL not provisioned."));
}
@Provides
@Config("beamCloudSqlUsername")
String provideSqlUsername(SqlAccessInfo sqlAccessInfo) {
return sqlAccessInfo.user();
}
@Provides
@Config("beamCloudSqlPassword")
String provideSqlPassword(SqlAccessInfo sqlAccessInfo) {
return sqlAccessInfo.password();
}
@Provides
@Config("beamCloudKmsProjectId")
String kmsProjectId() {
return cloudKmsProjectId;
}
@Provides
@Config("beamCloudKmsKeyRing")
static String keyRingName() {
return "nomulus-tool-keyring";
}
@Provides
@Config("beamIsolationOverride")
@Nullable
TransactionIsolationLevel providesIsolationOverride() {
return isolationOverride;
}
@Provides
@Config("beamHibernateHikariMaximumPoolSize")
static int getBeamHibernateHikariMaximumPoolSize() {
// TODO(weiminyu): make this configurable. Should be equal to number of cores.
return 4;
}
@Singleton
@Component(
modules = {
ConfigModule.class,
CredentialModule.class,
BeamJpaModule.class,
KmsModule.class,
PersistenceModule.class,
SecretManagerModule.class,
UtilsModule.class
})
public interface JpaTransactionManagerComponent {
@SocketFactoryJpaTm
JpaTransactionManager cloudSqlJpaTransactionManager();
@JdbcJpaTm
JpaTransactionManager localDbJpaTransactionManager();
}
}

View File

@@ -120,26 +120,22 @@ public class InitSqlPipeline implements Serializable {
private final InitSqlPipelineOptions options;
private final Pipeline pipeline;
InitSqlPipeline(InitSqlPipelineOptions options) {
this.options = options;
pipeline = Pipeline.create(options);
}
PipelineResult run() {
return run(Pipeline.create(options));
}
@VisibleForTesting
InitSqlPipeline(InitSqlPipelineOptions options, Pipeline pipeline) {
this.options = options;
this.pipeline = pipeline;
}
public PipelineResult run() {
setupPipeline();
PipelineResult run(Pipeline pipeline) {
setupPipeline(pipeline);
return pipeline.run();
}
@VisibleForTesting
void setupPipeline() {
void setupPipeline(Pipeline pipeline) {
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_UNCOMMITTED);
PCollectionTuple datastoreSnapshot =
pipeline.apply(

View File

@@ -1,60 +0,0 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.initsql;
import google.registry.beam.initsql.BeamJpaModule.JpaTransactionManagerComponent;
import google.registry.beam.initsql.Transforms.SerializableSupplier;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.transaction.JpaTransactionManager;
import javax.annotation.Nullable;
import org.apache.beam.sdk.transforms.SerializableFunction;
public class JpaSupplierFactory implements SerializableSupplier<JpaTransactionManager> {
private static final long serialVersionUID = 1L;
private final String credentialFileUrl;
@Nullable private final String cloudKmsProjectId;
private final SerializableFunction<JpaTransactionManagerComponent, JpaTransactionManager>
jpaGetter;
@Nullable private final TransactionIsolationLevel isolationLevelOverride;
public JpaSupplierFactory(
String credentialFileUrl,
@Nullable String cloudKmsProjectId,
SerializableFunction<JpaTransactionManagerComponent, JpaTransactionManager> jpaGetter) {
this(credentialFileUrl, cloudKmsProjectId, jpaGetter, null);
}
public JpaSupplierFactory(
String credentialFileUrl,
@Nullable String cloudKmsProjectId,
SerializableFunction<JpaTransactionManagerComponent, JpaTransactionManager> jpaGetter,
@Nullable TransactionIsolationLevel isolationLevelOverride) {
this.credentialFileUrl = credentialFileUrl;
this.cloudKmsProjectId = cloudKmsProjectId;
this.jpaGetter = jpaGetter;
this.isolationLevelOverride = isolationLevelOverride;
}
@Override
public JpaTransactionManager get() {
return jpaGetter.apply(
DaggerBeamJpaModule_JpaTransactionManagerComponent.builder()
.beamJpaModule(
new BeamJpaModule(credentialFileUrl, cloudKmsProjectId, isolationLevelOverride))
.build());
}
}

View File

@@ -19,13 +19,10 @@ import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Preconditions.checkState;
import static google.registry.beam.initsql.BackupPaths.getCommitLogTimestamp;
import static google.registry.beam.initsql.BackupPaths.getExportFilePatterns;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.setJpaTm;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.util.DateTimeUtils.START_OF_TIME;
import static google.registry.util.DateTimeUtils.isBeforeOrAt;
import static java.util.Comparator.comparing;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
import static org.apache.beam.sdk.values.TypeDescriptors.strings;
@@ -35,15 +32,14 @@ import com.google.appengine.api.datastore.EntityTranslator;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Streams;
import com.googlecode.objectify.Key;
import google.registry.backup.AppEngineEnvironment;
import google.registry.backup.CommitLogImports;
import google.registry.backup.VersionedEntity;
import google.registry.model.billing.BillingEvent.Flag;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.domain.DomainBase;
import google.registry.model.ofy.ObjectifyService;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.schema.replay.DatastoreAndSqlEntity;
import google.registry.schema.replay.SqlEntity;
import google.registry.tools.LevelDbLogReader;
@@ -53,7 +49,6 @@ import java.util.Iterator;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ThreadLocalRandom;
import java.util.function.Supplier;
import javax.annotation.Nullable;
import org.apache.beam.sdk.coders.StringUtf8Coder;
@@ -62,18 +57,14 @@ import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.io.FileIO.ReadableFile;
import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.GroupIntoBatches;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.ProcessFunction;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -268,81 +259,58 @@ public final class Transforms {
.iterator()));
}
/**
* Returns a {@link PTransform} that writes a {@link PCollection} of {@link VersionedEntity}s to a
* SQL database. and outputs an empty {@code PCollection<Void>}. This allows other operations to
* {@link org.apache.beam.sdk.transforms.Wait wait} for the completion of this transform.
*
* <p>Errors are handled according to the pipeline runner's default policy. As part of a one-time
* job, we will not add features unless proven necessary.
*
* @param transformId a unique ID for an instance of the returned transform
* @param maxWriters the max number of concurrent writes to SQL, which also determines the max
* number of connection pools created
* @param batchSize the number of entities to write in each operation
* @param jpaSupplier supplier of a {@link JpaTransactionManager}
*/
public static PTransform<PCollection<VersionedEntity>, PCollection<Void>> writeToSql(
String transformId,
int maxWriters,
int batchSize,
SerializableSupplier<JpaTransactionManager> jpaSupplier) {
return writeToSql(
transformId,
maxWriters,
batchSize,
jpaSupplier,
Transforms::convertVersionedEntityToSqlEntity,
TypeDescriptor.of(VersionedEntity.class));
}
// Production data repair configs go below. See b/185954992.
/**
* Returns a {@link PTransform} that writes a {@link PCollection} of entities to a SQL database.
* and outputs an empty {@code PCollection<Void>}. This allows other operations to {@link
* org.apache.beam.sdk.transforms.Wait wait} for the completion of this transform.
*
* <p>The converter and type descriptor are generics so that we can convert any type of entity to
* an object to be placed in SQL.
*
* <p>Errors are handled according to the pipeline runner's default policy. As part of a one-time
* job, we will not add features unless proven necessary.
*
* @param transformId a unique ID for an instance of the returned transform
* @param maxWriters the max number of concurrent writes to SQL, which also determines the max
* number of connection pools created
* @param batchSize the number of entities to write in each operation
* @param jpaSupplier supplier of a {@link JpaTransactionManager}
* @param jpaConverter the function that converts the input object to a JPA entity
* @param objectDescriptor the type descriptor of the input object
*/
public static <T> PTransform<PCollection<T>, PCollection<Void>> writeToSql(
String transformId,
int maxWriters,
int batchSize,
SerializableSupplier<JpaTransactionManager> jpaSupplier,
SerializableFunction<T, Object> jpaConverter,
TypeDescriptor<T> objectDescriptor) {
return new PTransform<PCollection<T>, PCollection<Void>>() {
@Override
public PCollection<Void> expand(PCollection<T> input) {
return input
.apply(
"Shard data for " + transformId,
MapElements.into(kvs(integers(), objectDescriptor))
.via(ve -> KV.of(ThreadLocalRandom.current().nextInt(maxWriters), ve)))
.apply("Batch output by shard " + transformId, GroupIntoBatches.ofSize(batchSize))
.apply(
"Write in batch for " + transformId,
ParDo.of(new SqlBatchWriter<T>(transformId, jpaSupplier, jpaConverter)));
}
};
}
// Prober domains in bad state, without associated contacts, hosts, billings, and history.
// They can be safely ignored.
private static final ImmutableSet<String> IGNORED_DOMAINS =
ImmutableSet.of("6AF6D2-IQCANT", "2-IQANYT");
private static Key toOfyKey(Object ofyEntity) {
return Key.create(ofyEntity);
}
// Prober hosts referencing phantom registrars. They and their associated history entries can be
// safely ignored.
private static final ImmutableSet<String> IGNORED_HOSTS =
ImmutableSet.of(
"4E21_WJ0TEST-GOOGLE",
"4E21_WJ1TEST-GOOGLE",
"4E21_WJ2TEST-GOOGLE",
"4E21_WJ3TEST-GOOGLE");
// Prober contacts referencing phantom registrars. They and their associated history entries can
// be safely ignored.
private static final ImmutableSet IGNORED_CONTACTS =
ImmutableSet.of(
"1_WJ0TEST-GOOGLE", "1_WJ1TEST-GOOGLE", "1_WJ2TEST-GOOGLE", "1_WJ3TEST-GOOGLE");
private static boolean isMigratable(Entity entity) {
// Checks specific to production data. See b/185954992 for details.
// The names of these bad entities in production do not conflict with other environments. For
// simplicities sake we apply them regardless of the source of the data.
if (entity.getKind().equals("DomainBase")
&& IGNORED_DOMAINS.contains(entity.getKey().getName())) {
return false;
}
if (entity.getKind().equals("ContactResource")) {
String roid = entity.getKey().getName();
return !IGNORED_CONTACTS.contains(roid);
}
if (entity.getKind().equals("HostResource")) {
String roid = entity.getKey().getName();
return !IGNORED_HOSTS.contains(roid);
}
if (entity.getKind().equals("HistoryEntry")) {
// Remove production bad data: History of the contacts to be ignored:
com.google.appengine.api.datastore.Key parentKey = entity.getKey().getParent();
if (parentKey.getKind().equals("ContactResource")) {
String contactRoid = parentKey.getName();
return !IGNORED_CONTACTS.contains(contactRoid);
}
if (parentKey.getKind().equals("HostResource")) {
String hostRoid = parentKey.getName();
return !IGNORED_HOSTS.contains(hostRoid);
}
}
// End of production-specific checks.
if (entity.getKind().equals("HistoryEntry")) {
// DOMAIN_APPLICATION_CREATE is deprecated type and should not be migrated.
// The Enum name DOMAIN_APPLICATION_CREATE no longer exists in Java and cannot
@@ -352,6 +320,18 @@ public final class Transforms {
return true;
}
private static Entity repairBadData(Entity entity) {
if (entity.getKind().equals("Cancellation")
&& Objects.equals(entity.getProperty("reason"), "AUTO_RENEW")) {
// AUTO_RENEW has been moved from 'reason' to flags. Change reason to RENEW and add the
// AUTO_RENEW flag. Note: all affected entities have empty flags so we can simply assign
// instead of append. See b/185954992.
entity.setUnindexedProperty("reason", Reason.RENEW.name());
entity.setUnindexedProperty("flags", ImmutableList.of(Flag.AUTO_RENEW.name()));
}
return entity;
}
private static SqlEntity toSqlEntity(Object ofyEntity) {
if (ofyEntity instanceof HistoryEntry) {
HistoryEntry ofyHistory = (HistoryEntry) ofyEntity;
@@ -372,7 +352,8 @@ public final class Transforms {
return dsEntity
.getEntity()
.filter(Transforms::isMigratable)
.map(e -> ofy().toPojo(e))
.map(Transforms::repairBadData)
.map(e -> auditedOfy().toPojo(e))
.map(Transforms::toSqlEntity)
.orElse(null);
}
@@ -458,93 +439,6 @@ public final class Transforms {
}
}
/**
* Writes a batch of entities to a SQL database.
*
* <p>Note that an arbitrary number of instances of this class may be created and freed in
* arbitrary order in a single JVM. Due to the tech debt that forced us to use a static variable
* to hold the {@code JpaTransactionManager} instance, we must ensure that JpaTransactionManager
* is not changed or torn down while being used by some instance.
*/
private static class SqlBatchWriter<T> extends DoFn<KV<Integer, Iterable<T>>, Void> {
private static int instanceCount = 0;
private static JpaTransactionManager originalJpa;
private Counter counter;
private final SerializableSupplier<JpaTransactionManager> jpaSupplier;
private final SerializableFunction<T, Object> jpaConverter;
SqlBatchWriter(
String type,
SerializableSupplier<JpaTransactionManager> jpaSupplier,
SerializableFunction<T, Object> jpaConverter) {
counter = Metrics.counter("SQL_WRITE", type);
this.jpaSupplier = jpaSupplier;
this.jpaConverter = jpaConverter;
}
@Setup
public void setup() {
try (AppEngineEnvironment env = new AppEngineEnvironment()) {
ObjectifyService.initOfy();
}
synchronized (SqlBatchWriter.class) {
if (instanceCount == 0) {
originalJpa = jpaTm();
setJpaTm(jpaSupplier);
}
instanceCount++;
}
}
@Teardown
public void teardown() {
synchronized (SqlBatchWriter.class) {
instanceCount--;
if (instanceCount == 0) {
jpaTm().teardown();
setJpaTm(() -> originalJpa);
}
}
}
@ProcessElement
public void processElement(@Element KV<Integer, Iterable<T>> kv) {
try (AppEngineEnvironment env = new AppEngineEnvironment()) {
ImmutableList<Object> ofyEntities =
Streams.stream(kv.getValue())
.map(this.jpaConverter::apply)
// TODO(b/177340730): post migration delete the line below.
.filter(Objects::nonNull)
.collect(ImmutableList.toImmutableList());
try {
jpaTm().transact(() -> jpaTm().putAll(ofyEntities));
counter.inc(ofyEntities.size());
} catch (RuntimeException e) {
processSingly(ofyEntities);
}
}
}
/**
* Writes entities in a failed batch one by one to identify the first bad entity and throws a
* {@link RuntimeException} on it.
*/
private void processSingly(ImmutableList<Object> ofyEntities) {
for (Object ofyEntity : ofyEntities) {
try {
jpaTm().transact(() -> jpaTm().put(ofyEntity));
counter.inc();
} catch (RuntimeException e) {
throw new RuntimeException(toOfyKey(ofyEntity).toString(), e);
}
}
}
}
/**
* Removes BillingEvents, {@link google.registry.model.poll.PollMessage PollMessages} and {@link
* google.registry.model.host.HostResource} from a {@link DomainBase}. These are circular foreign

View File

@@ -260,6 +260,11 @@ public abstract class BillingEvent implements Serializable {
poNumber());
}
/** Returns the grouping key for this {@code BillingEvent}, to generate the detailed report. */
String getDetailedReportGroupingKey() {
return String.format("%s_%s", registrarId(), tld());
}
/** Key for each {@code BillingEvent}, when aggregating for the overall invoice. */
@AutoValue
abstract static class InvoiceGroupingKey implements Serializable {

View File

@@ -14,28 +14,27 @@
package google.registry.beam.invoicing;
import com.google.auth.oauth2.GoogleCredentials;
import static google.registry.beam.BeamUtils.getQueryFromFile;
import static org.apache.beam.sdk.values.TypeDescriptors.strings;
import google.registry.beam.invoicing.BillingEvent.InvoiceGroupingKey;
import google.registry.beam.invoicing.BillingEvent.InvoiceGroupingKey.InvoiceGroupingKeyCoder;
import google.registry.config.CredentialModule.LocalCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.reporting.billing.BillingModule;
import google.registry.reporting.billing.GenerateInvoicesAction;
import google.registry.util.GoogleCredentialsBundle;
import google.registry.util.SqlTemplate;
import java.io.Serializable;
import javax.inject.Inject;
import org.apache.beam.runners.dataflow.DataflowRunner;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import java.time.LocalDateTime;
import java.time.LocalTime;
import java.time.YearMonth;
import java.time.format.DateTimeFormatter;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.io.DefaultFilenamePolicy.Params;
import org.apache.beam.sdk.io.FileBasedSink;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.Contextful;
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Filter;
import org.apache.beam.sdk.transforms.MapElements;
@@ -43,107 +42,48 @@ import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
/**
* Definition of a Dataflow pipeline template, which generates a given month's invoices.
* Definition of a Dataflow Flex pipeline template, which generates a given month's invoices.
*
* <p>To stage this template on GCS, run the {@link
* google.registry.tools.DeployInvoicingPipelineCommand} Nomulus command.
* <p>To stage this template locally, run the {@code stage_beam_pipeline.sh} shell script.
*
* <p>Then, you can run the staged template via the API client library, gCloud or a raw REST call.
* For an example using the API client library, see {@link GenerateInvoicesAction}.
*
* @see <a href="https://cloud.google.com/dataflow/docs/templates/overview">Dataflow Templates</a>
* @see <a href="https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates">Using
* Flex Templates</a>
*/
public class InvoicingPipeline implements Serializable {
private final String projectId;
private final String beamJobRegion;
private final String beamBucketUrl;
private final String invoiceTemplateUrl;
private final String beamStagingUrl;
private final String billingBucketUrl;
private final String invoiceFilePrefix;
private final GoogleCredentials googleCredentials;
private static final DateTimeFormatter TIMESTAMP_FORMATTER =
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSSSSS");
@Inject
public InvoicingPipeline(
@Config("projectId") String projectId,
@Config("defaultJobRegion") String beamJobRegion,
@Config("apacheBeamBucketUrl") String beamBucketUrl,
@Config("invoiceTemplateUrl") String invoiceTemplateUrl,
@Config("beamStagingUrl") String beamStagingUrl,
@Config("billingBucketUrl") String billingBucketUrl,
@Config("invoiceFilePrefix") String invoiceFilePrefix,
@LocalCredential GoogleCredentialsBundle googleCredentialsBundle) {
this.projectId = projectId;
this.beamJobRegion = beamJobRegion;
this.beamBucketUrl = beamBucketUrl;
this.invoiceTemplateUrl = invoiceTemplateUrl;
this.beamStagingUrl = beamStagingUrl;
this.billingBucketUrl = billingBucketUrl;
this.invoiceFilePrefix = invoiceFilePrefix;
this.googleCredentials = googleCredentialsBundle.getGoogleCredentials();
private final InvoicingPipelineOptions options;
InvoicingPipeline(InvoicingPipelineOptions options) {
this.options = options;
}
/** Custom options for running the invoicing pipeline. */
public interface InvoicingPipelineOptions extends DataflowPipelineOptions {
/** Returns the yearMonth we're generating invoices for, in yyyy-MM format. */
@Description("The yearMonth we generate invoices for, in yyyy-MM format.")
ValueProvider<String> getYearMonth();
/**
* Sets the yearMonth we generate invoices for.
*
* <p>This is implicitly set when executing the Dataflow template, by specifying the 'yearMonth
* parameter.
*/
void setYearMonth(ValueProvider<String> value);
PipelineResult run() {
Pipeline pipeline = Pipeline.create(options);
setupPipeline(pipeline);
return pipeline.run();
}
/** Deploys the invoicing pipeline as a template on GCS, for a given projectID and GCS bucket. */
public void deploy() {
// We can't store options as a member variable due to serialization concerns.
InvoicingPipelineOptions options = PipelineOptionsFactory.as(InvoicingPipelineOptions.class);
options.setProject(projectId);
options.setRegion(beamJobRegion);
options.setRunner(DataflowRunner.class);
// This causes p.run() to stage the pipeline as a template on GCS, as opposed to running it.
options.setTemplateLocation(invoiceTemplateUrl);
options.setStagingLocation(beamStagingUrl);
// This credential is used when Dataflow deploys the template to GCS in target GCP project.
// So, make sure the credential has write permission to GCS in that project.
options.setGcpCredential(googleCredentials);
Pipeline p = Pipeline.create(options);
void setupPipeline(Pipeline pipeline) {
PCollection<BillingEvent> billingEvents =
p.apply(
pipeline.apply(
"Read BillingEvents from Bigquery",
BigQueryIO.read(BillingEvent::parseFromRecord)
.fromQuery(InvoicingUtils.makeQueryProvider(options.getYearMonth(), projectId))
.fromQuery(makeQuery(options.getYearMonth(), options.getProject()))
.withCoder(SerializableCoder.of(BillingEvent.class))
.usingStandardSql()
.withoutValidation()
.withTemplateCompatibility());
applyTerminalTransforms(billingEvents, options.getYearMonth());
p.run();
}
/**
* Applies output transforms to the {@code BillingEvent} source collection.
*
* <p>This is factored out purely to facilitate testing.
*/
void applyTerminalTransforms(
PCollection<BillingEvent> billingEvents, ValueProvider<String> yearMonthProvider) {
billingEvents
.apply("Generate overall invoice rows", new GenerateInvoiceRows())
.apply("Write overall invoice to CSV", writeInvoice(yearMonthProvider));
saveInvoiceCsv(billingEvents, options);
billingEvents.apply(
"Write detail reports to separate CSVs keyed by registrarId_tld pair",
writeDetailReports(yearMonthProvider));
saveDetailedCsv(billingEvents, options);
}
/** Transform that converts a {@code BillingEvent} into an invoice CSV row. */
@@ -156,49 +96,85 @@ public class InvoicingPipeline implements Serializable {
"Map to invoicing key",
MapElements.into(TypeDescriptor.of(InvoiceGroupingKey.class))
.via(BillingEvent::getInvoiceGroupingKey))
.apply(Filter.by((InvoiceGroupingKey key) -> key.unitPrice() != 0))
.apply(
"Filter out free events", Filter.by((InvoiceGroupingKey key) -> key.unitPrice() != 0))
.setCoder(new InvoiceGroupingKeyCoder())
.apply("Count occurrences", Count.perElement())
.apply(
"Format as CSVs",
MapElements.into(TypeDescriptors.strings())
MapElements.into(strings())
.via((KV<InvoiceGroupingKey, Long> kv) -> kv.getKey().toCsv(kv.getValue())));
}
}
/** Returns an IO transform that writes the overall invoice to a single CSV file. */
private TextIO.Write writeInvoice(ValueProvider<String> yearMonthProvider) {
return TextIO.write()
.to(
NestedValueProvider.of(
yearMonthProvider,
yearMonth ->
/** Saves the billing events to a single overall invoice CSV file. */
static void saveInvoiceCsv(
PCollection<BillingEvent> billingEvents, InvoicingPipelineOptions options) {
billingEvents
.apply("Generate overall invoice rows", new GenerateInvoiceRows())
.apply(
"Write overall invoice to CSV",
TextIO.write()
.to(
String.format(
"%s/%s/%s/%s-%s",
billingBucketUrl,
options.getBillingBucketUrl(),
BillingModule.INVOICES_DIRECTORY,
yearMonth,
invoiceFilePrefix,
yearMonth)))
.withHeader(InvoiceGroupingKey.invoiceHeader())
.withoutSharding()
.withSuffix(".csv");
options.getYearMonth(),
options.getInvoiceFilePrefix(),
options.getYearMonth()))
.withHeader(InvoiceGroupingKey.invoiceHeader())
.withoutSharding()
.withSuffix(".csv"));
}
/** Returns an IO transform that writes detail reports to registrar-tld keyed CSV files. */
private TextIO.TypedWrite<BillingEvent, Params> writeDetailReports(
ValueProvider<String> yearMonthProvider) {
return TextIO.<BillingEvent>writeCustomType()
.to(
InvoicingUtils.makeDestinationFunction(
String.format("%s/%s", billingBucketUrl, BillingModule.INVOICES_DIRECTORY),
yearMonthProvider),
InvoicingUtils.makeEmptyDestinationParams(billingBucketUrl + "/errors"))
.withFormatFunction(BillingEvent::toCsv)
.withoutSharding()
.withTempDirectory(
FileBasedSink.convertToFileResourceIfPossible(beamBucketUrl + "/temporary"))
.withHeader(BillingEvent.getHeader())
.withSuffix(".csv");
/** Saves the billing events to detailed report CSV files keyed by registrar-tld pairs. */
static void saveDetailedCsv(
PCollection<BillingEvent> billingEvents, InvoicingPipelineOptions options) {
String yearMonth = options.getYearMonth();
billingEvents.apply(
"Write detailed report for each registrar-tld pair",
FileIO.<String, BillingEvent>writeDynamic()
.to(
String.format(
"%s/%s/%s",
options.getBillingBucketUrl(), BillingModule.INVOICES_DIRECTORY, yearMonth))
.by(BillingEvent::getDetailedReportGroupingKey)
.withNumShards(1)
.withDestinationCoder(StringUtf8Coder.of())
.withNaming(
key ->
(window, pane, numShards, shardIndex, compression) ->
String.format(
"%s_%s_%s.csv", BillingModule.DETAIL_REPORT_PREFIX, yearMonth, key))
.via(
Contextful.fn(BillingEvent::toCsv),
TextIO.sink().withHeader(BillingEvent.getHeader())));
}
/** Create the Bigquery query for a given project and yearMonth at runtime. */
static String makeQuery(String yearMonth, String projectId) {
// Get the timestamp endpoints capturing the entire month with microsecond precision
YearMonth reportingMonth = YearMonth.parse(yearMonth);
LocalDateTime firstMoment = reportingMonth.atDay(1).atTime(LocalTime.MIDNIGHT);
LocalDateTime lastMoment = reportingMonth.atEndOfMonth().atTime(LocalTime.MAX);
// Construct the month's query by filling in the billing_events.sql template
return SqlTemplate.create(getQueryFromFile(InvoicingPipeline.class, "billing_events.sql"))
.put("FIRST_TIMESTAMP_OF_MONTH", firstMoment.format(TIMESTAMP_FORMATTER))
.put("LAST_TIMESTAMP_OF_MONTH", lastMoment.format(TIMESTAMP_FORMATTER))
.put("PROJECT_ID", projectId)
.put("DATASTORE_EXPORT_DATA_SET", "latest_datastore_export")
.put("ONETIME_TABLE", "OneTime")
.put("REGISTRY_TABLE", "Registry")
.put("REGISTRAR_TABLE", "Registrar")
.put("CANCELLATION_TABLE", "Cancellation")
.build();
}
public static void main(String[] args) {
PipelineOptionsFactory.register(InvoicingPipelineOptions.class);
InvoicingPipelineOptions options =
PipelineOptionsFactory.fromArgs(args).withValidation().as(InvoicingPipelineOptions.class);
new InvoicingPipeline(options).run();
}
}

View File

@@ -0,0 +1,37 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.invoicing;
import google.registry.beam.common.RegistryPipelineOptions;
import org.apache.beam.sdk.options.Description;
/** Custom options for running the invoicing pipeline. */
public interface InvoicingPipelineOptions extends RegistryPipelineOptions {
@Description("The year and month we generate invoices for, in yyyy-MM format.")
String getYearMonth();
void setYearMonth(String value);
@Description("Filename prefix for the invoice CSV file.")
String getInvoiceFilePrefix();
void setInvoiceFilePrefix(String value);
@Description("The GCS bucket URL for invoices and detailed reports to be uploaded.")
String getBillingBucketUrl();
void setBillingBucketUrl(String value);
}

View File

@@ -1,106 +0,0 @@
// Copyright 2018 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.invoicing;
import static google.registry.beam.BeamUtils.getQueryFromFile;
import google.registry.util.SqlTemplate;
import java.time.LocalDateTime;
import java.time.LocalTime;
import java.time.YearMonth;
import java.time.format.DateTimeFormatter;
import org.apache.beam.sdk.io.DefaultFilenamePolicy.Params;
import org.apache.beam.sdk.io.FileBasedSink;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
/** Pipeline helper functions used to generate invoices from instances of {@link BillingEvent}. */
public class InvoicingUtils {
private InvoicingUtils() {}
private static final DateTimeFormatter TIMESTAMP_FORMATTER =
DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSSSSS");
/**
* Returns a function mapping from {@code BillingEvent} to filename {@code Params}.
*
* <p>Beam uses this to determine which file a given {@code BillingEvent} should get placed into.
*
* @param outputBucket the GCS bucket we're outputting reports to
* @param yearMonthProvider a runtime provider for the yyyy-MM we're generating the invoice for
*/
static SerializableFunction<BillingEvent, Params> makeDestinationFunction(
String outputBucket, ValueProvider<String> yearMonthProvider) {
return billingEvent ->
new Params()
.withShardTemplate("")
.withSuffix(".csv")
.withBaseFilename(
NestedValueProvider.of(
yearMonthProvider,
yearMonth ->
FileBasedSink.convertToFileResourceIfPossible(
String.format(
"%s/%s/%s",
outputBucket, yearMonth, billingEvent.toFilename(yearMonth)))));
}
/**
* Returns the default filename parameters for an unmappable {@code BillingEvent}.
*
* <p>The "failed" file should only be populated when an error occurs, which warrants further
* investigation.
*/
static Params makeEmptyDestinationParams(String outputBucket) {
return new Params()
.withBaseFilename(
FileBasedSink.convertToFileResourceIfPossible(
String.format("%s/%s", outputBucket, "FAILURES")));
}
/**
* Returns a provider that creates a Bigquery query for a given project and yearMonth at runtime.
*
* <p>We only know yearMonth at runtime, so this provider fills in the {@code
* sql/billing_events.sql} template at runtime.
*
* @param yearMonthProvider a runtime provider that returns which month we're invoicing for.
* @param projectId the projectId we're generating invoicing for.
*/
static ValueProvider<String> makeQueryProvider(
ValueProvider<String> yearMonthProvider, String projectId) {
return NestedValueProvider.of(
yearMonthProvider,
(yearMonth) -> {
// Get the timestamp endpoints capturing the entire month with microsecond precision
YearMonth reportingMonth = YearMonth.parse(yearMonth);
LocalDateTime firstMoment = reportingMonth.atDay(1).atTime(LocalTime.MIDNIGHT);
LocalDateTime lastMoment = reportingMonth.atEndOfMonth().atTime(LocalTime.MAX);
// Construct the month's query by filling in the billing_events.sql template
return SqlTemplate.create(getQueryFromFile(InvoicingPipeline.class, "billing_events.sql"))
.put("FIRST_TIMESTAMP_OF_MONTH", firstMoment.format(TIMESTAMP_FORMATTER))
.put("LAST_TIMESTAMP_OF_MONTH", lastMoment.format(TIMESTAMP_FORMATTER))
.put("PROJECT_ID", projectId)
.put("DATASTORE_EXPORT_DATA_SET", "latest_datastore_export")
.put("ONETIME_TABLE", "OneTime")
.put("REGISTRY_TABLE", "Registry")
.put("REGISTRAR_TABLE", "Registrar")
.put("CANCELLATION_TABLE", "Cancellation")
.build();
});
}
}

View File

@@ -14,7 +14,6 @@
package google.registry.beam.spec11;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.http.HttpStatus.SC_OK;
@@ -30,7 +29,6 @@ import java.net.URISyntaxException;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.function.Supplier;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.KV;
@@ -73,7 +71,7 @@ public class SafeBrowsingTransforms {
private static final int BATCH_SIZE = 490;
/** Provides the SafeBrowsing API key at runtime. */
private final ValueProvider<String> apiKeyProvider;
private final String apiKey;
/**
* Maps a subdomain's {@code fullyQualifiedDomainName} to its corresponding {@link Subdomain} to
@@ -93,20 +91,18 @@ public class SafeBrowsingTransforms {
private final Retrier retrier;
/**
* Constructs a {@link EvaluateSafeBrowsingFn} that gets its API key from the given provider.
* Constructs a {@link EvaluateSafeBrowsingFn} with a given API key.
*
* <p>We need to dual-cast the closeableHttpClientSupplier lambda because all {@code DoFn}
* member variables need to be serializable. The (Supplier & Serializable) dual cast is safe
* because class methods are generally serializable, especially a static function such as {@link
* HttpClients#createDefault()}.
*
* @param apiKeyProvider provides the SafeBrowsing API key from {@code KMS} at runtime
*/
@SuppressWarnings("unchecked")
EvaluateSafeBrowsingFn(ValueProvider<String> apiKeyProvider, Retrier retrier) {
this.apiKeyProvider = apiKeyProvider;
EvaluateSafeBrowsingFn(String apiKey, Retrier retrier) {
this.apiKey = apiKey;
this.retrier = retrier;
this.closeableHttpClientSupplier = (Supplier & Serializable) HttpClients::createDefault;
closeableHttpClientSupplier = (Supplier & Serializable) HttpClients::createDefault;
}
/**
@@ -117,12 +113,10 @@ public class SafeBrowsingTransforms {
*/
@VisibleForTesting
EvaluateSafeBrowsingFn(
ValueProvider<String> apiKeyProvider,
Retrier retrier,
Supplier<CloseableHttpClient> clientSupplier) {
this.apiKeyProvider = apiKeyProvider;
String apiKey, Retrier retrier, Supplier<CloseableHttpClient> clientSupplier) {
this.apiKey = apiKey;
this.retrier = retrier;
this.closeableHttpClientSupplier = clientSupplier;
closeableHttpClientSupplier = clientSupplier;
}
/** Evaluates any buffered {@link Subdomain} objects upon completing the bundle. */
@@ -159,7 +153,7 @@ public class SafeBrowsingTransforms {
try {
URIBuilder uriBuilder = new URIBuilder(SAFE_BROWSING_URL);
// Add the API key param
uriBuilder.addParameter("key", apiKeyProvider.get());
uriBuilder.addParameter("key", apiKey);
HttpPost httpPost = new HttpPost(uriBuilder.build());
httpPost.addHeader(HTTP.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString());
@@ -175,7 +169,7 @@ public class SafeBrowsingTransforms {
}
},
IOException.class);
} catch (URISyntaxException | JSONException e) {
} catch (URISyntaxException | JSONException e) {
// Fail the pipeline on a parsing exception- this indicates the API likely changed.
throw new RuntimeException("Caught parsing exception, failing pipeline.", e);
} finally {
@@ -239,7 +233,9 @@ public class SafeBrowsingTransforms {
String url = match.getJSONObject("threat").getString("url");
Subdomain subdomain = subdomainBuffer.get(url);
resultBuilder.add(
KV.of(subdomain, ThreatMatch.create(match, subdomain.domainName())));
KV.of(
subdomain,
ThreatMatch.create(match.getString("threatType"), subdomain.domainName())));
}
}
}

View File

@@ -17,32 +17,27 @@ package google.registry.beam.spec11;
import static com.google.common.base.Preconditions.checkArgument;
import static google.registry.beam.BeamUtils.getQueryFromFile;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableSet;
import google.registry.beam.initsql.Transforms;
import google.registry.beam.initsql.Transforms.SerializableSupplier;
import dagger.Component;
import dagger.Module;
import dagger.Provides;
import google.registry.beam.common.RegistryJpaIO;
import google.registry.beam.spec11.SafeBrowsingTransforms.EvaluateSafeBrowsingFn;
import google.registry.config.CredentialModule.LocalCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.model.reporting.Spec11ThreatMatch;
import google.registry.model.reporting.Spec11ThreatMatch.ThreatType;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.util.GoogleCredentialsBundle;
import google.registry.util.Retrier;
import google.registry.util.SqlTemplate;
import google.registry.util.UtilsModule;
import java.io.Serializable;
import javax.inject.Inject;
import org.apache.beam.runners.dataflow.DataflowRunner;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import javax.inject.Singleton;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.ParDo;
@@ -58,21 +53,20 @@ import org.json.JSONException;
import org.json.JSONObject;
/**
* Definition of a Dataflow pipeline template, which generates a given month's spec11 report.
* Definition of a Dataflow Flex template, which generates a given month's spec11 report.
*
* <p>To stage this template on GCS, run the {@link
* google.registry.tools.DeploySpec11PipelineCommand} Nomulus command.
* <p>To stage this template locally, run the {@code stage_beam_pipeline.sh} shell script.
*
* <p>Then, you can run the staged template via the API client library, gCloud or a raw REST call.
*
* @see <a href="https://cloud.google.com/dataflow/docs/templates/overview">Dataflow Templates</a>
* @see <a href="https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates">Using
* Flex Templates</a>
*/
public class Spec11Pipeline implements Serializable {
/**
* Returns the subdirectory spec11 reports reside in for a given local date in yyyy-MM-dd format.
*
* @see google.registry.beam.spec11.Spec11Pipeline
* @see google.registry.reporting.spec11.Spec11EmailUtils
*/
public static String getSpec11ReportFilePath(LocalDate localDate) {
@@ -87,84 +81,28 @@ public class Spec11Pipeline implements Serializable {
/** The JSON object field into which we put the threat match array for Spec11 reports. */
public static final String THREAT_MATCHES_FIELD = "threatMatches";
private final String projectId;
private final String beamJobRegion;
private final String beamStagingUrl;
private final String spec11TemplateUrl;
private final String reportingBucketUrl;
private final GoogleCredentials googleCredentials;
private final Retrier retrier;
private final SerializableSupplier<JpaTransactionManager> jpaSupplierFactory;
private final Spec11PipelineOptions options;
private final EvaluateSafeBrowsingFn safeBrowsingFn;
@Inject
public Spec11Pipeline(
@Config("projectId") String projectId,
@Config("defaultJobRegion") String beamJobRegion,
@Config("beamStagingUrl") String beamStagingUrl,
@Config("spec11TemplateUrl") String spec11TemplateUrl,
@Config("reportingBucketUrl") String reportingBucketUrl,
SerializableSupplier<JpaTransactionManager> jpaSupplierFactory,
@LocalCredential GoogleCredentialsBundle googleCredentialsBundle,
Retrier retrier) {
this.projectId = projectId;
this.beamJobRegion = beamJobRegion;
this.beamStagingUrl = beamStagingUrl;
this.spec11TemplateUrl = spec11TemplateUrl;
this.reportingBucketUrl = reportingBucketUrl;
this.jpaSupplierFactory = jpaSupplierFactory;
this.googleCredentials = googleCredentialsBundle.getGoogleCredentials();
this.retrier = retrier;
Spec11Pipeline(Spec11PipelineOptions options, EvaluateSafeBrowsingFn safeBrowsingFn) {
this.options = options;
this.safeBrowsingFn = safeBrowsingFn;
}
/** Custom options for running the spec11 pipeline. */
public interface Spec11PipelineOptions extends DataflowPipelineOptions {
/** Returns the local date we're generating the report for, in yyyy-MM-dd format. */
@Description("The local date we generate the report for, in yyyy-MM-dd format.")
ValueProvider<String> getDate();
/**
* Sets the local date we generate invoices for.
*
* <p>This is implicitly set when executing the Dataflow template, by specifying the "date"
* parameter.
*/
void setDate(ValueProvider<String> value);
/** Returns the SafeBrowsing API key we use to evaluate subdomain health. */
@Description("The API key we use to access the SafeBrowsing API.")
ValueProvider<String> getSafeBrowsingApiKey();
/**
* Sets the SafeBrowsing API key we use.
*
* <p>This is implicitly set when executing the Dataflow template, by specifying the
* "safeBrowsingApiKey" parameter.
*/
void setSafeBrowsingApiKey(ValueProvider<String> value);
PipelineResult run() {
Pipeline pipeline = Pipeline.create(options);
setupPipeline(pipeline);
return pipeline.run();
}
/** Deploys the spec11 pipeline as a template on GCS. */
public void deploy() {
// We can't store options as a member variable due to serialization concerns.
Spec11PipelineOptions options = PipelineOptionsFactory.as(Spec11PipelineOptions.class);
options.setProject(projectId);
options.setRegion(beamJobRegion);
options.setRunner(DataflowRunner.class);
// This causes p.run() to stage the pipeline as a template on GCS, as opposed to running it.
options.setTemplateLocation(spec11TemplateUrl);
options.setStagingLocation(beamStagingUrl);
// This credential is used when Dataflow deploys the template to GCS in target GCP project.
// So, make sure the credential has write permission to GCS in that project.
options.setGcpCredential(googleCredentials);
Pipeline p = Pipeline.create(options);
void setupPipeline(Pipeline pipeline) {
PCollection<Subdomain> domains =
p.apply(
pipeline.apply(
"Read active domains from BigQuery",
BigQueryIO.read(Subdomain::parseFromRecord)
.fromQuery(
SqlTemplate.create(getQueryFromFile(Spec11Pipeline.class, "subdomains.sql"))
.put("PROJECT_ID", projectId)
.put("PROJECT_ID", options.getProject())
.put("DATASTORE_EXPORT_DATASET", "latest_datastore_export")
.put("REGISTRAR_TABLE", "Registrar")
.put("DOMAIN_BASE_TABLE", "DomainBase")
@@ -174,48 +112,40 @@ public class Spec11Pipeline implements Serializable {
.withoutValidation()
.withTemplateCompatibility());
evaluateUrlHealth(
domains,
new EvaluateSafeBrowsingFn(options.getSafeBrowsingApiKey(), retrier),
options.getDate());
p.run();
PCollection<KV<Subdomain, ThreatMatch>> threatMatches =
domains.apply("Run through SafeBrowsing API", ParDo.of(safeBrowsingFn));
saveToSql(threatMatches, options);
saveToGcs(threatMatches, options);
}
/**
* Evaluate each {@link Subdomain} URL via the SafeBrowsing API.
*
* <p>This is factored out to facilitate testing.
*/
void evaluateUrlHealth(
PCollection<Subdomain> domains,
EvaluateSafeBrowsingFn evaluateSafeBrowsingFn,
ValueProvider<String> dateProvider) {
PCollection<KV<Subdomain, ThreatMatch>> subdomainsSql =
domains.apply("Run through SafeBrowsing API", ParDo.of(evaluateSafeBrowsingFn));
TypeDescriptor<KV<Subdomain, ThreatMatch>> descriptor =
new TypeDescriptor<KV<Subdomain, ThreatMatch>>() {};
subdomainsSql.apply(
Transforms.writeToSql(
"Spec11ThreatMatch",
4,
4,
jpaSupplierFactory,
(kv) -> {
Subdomain subdomain = kv.getKey();
return new Spec11ThreatMatch.Builder()
.setThreatTypes(ImmutableSet.of(ThreatType.valueOf(kv.getValue().threatType())))
.setCheckDate(LocalDate.parse(dateProvider.get(), ISODateTimeFormat.date()))
.setDomainName(subdomain.domainName())
.setDomainRepoId(subdomain.domainRepoId())
.setRegistrarId(subdomain.registrarId())
.build();
},
descriptor));
static void saveToSql(
PCollection<KV<Subdomain, ThreatMatch>> threatMatches, Spec11PipelineOptions options) {
String transformId = "Spec11 Threat Matches";
LocalDate date = LocalDate.parse(options.getDate(), ISODateTimeFormat.date());
threatMatches.apply(
"Write to Sql: " + transformId,
RegistryJpaIO.<KV<Subdomain, ThreatMatch>>write()
.withName(transformId)
.withBatchSize(options.getSqlWriteBatchSize())
.withShards(options.getSqlWriteShards())
.withJpaConverter(
(kv) -> {
Subdomain subdomain = kv.getKey();
return new Spec11ThreatMatch.Builder()
.setThreatTypes(
ImmutableSet.of(ThreatType.valueOf(kv.getValue().threatType())))
.setCheckDate(date)
.setDomainName(subdomain.domainName())
.setDomainRepoId(subdomain.domainRepoId())
.setRegistrarId(subdomain.registrarId())
.build();
}));
}
/* Store ThreatMatch objects in JSON. */
PCollection<KV<Subdomain, ThreatMatch>> subdomainsJson =
domains.apply("Run through SafeBrowsingAPI", ParDo.of(evaluateSafeBrowsingFn));
subdomainsJson
static void saveToGcs(
PCollection<KV<Subdomain, ThreatMatch>> threatMatches, Spec11PipelineOptions options) {
threatMatches
.apply(
"Map registrar ID to email/ThreatMatch pair",
MapElements.into(
@@ -260,17 +190,54 @@ public class Spec11Pipeline implements Serializable {
"Output to text file",
TextIO.write()
.to(
NestedValueProvider.of(
dateProvider,
date ->
String.format(
"%s/%s",
reportingBucketUrl,
getSpec11ReportFilePath(LocalDate.parse(date)))))
String.format(
"%s/%s",
options.getReportingBucketUrl(),
getSpec11ReportFilePath(LocalDate.parse(options.getDate()))))
.withoutSharding()
.withHeader("Map from registrar email / name to detected subdomain threats:"));
}
public static void main(String[] args) {
PipelineOptionsFactory.register(Spec11PipelineOptions.class);
DaggerSpec11Pipeline_Spec11PipelineComponent.builder()
.spec11PipelineModule(new Spec11PipelineModule(args))
.build()
.spec11Pipeline()
.run();
}
@Module
static class Spec11PipelineModule {
private final String[] args;
Spec11PipelineModule(String[] args) {
this.args = args;
}
@Provides
Spec11PipelineOptions provideOptions() {
return PipelineOptionsFactory.fromArgs(args).withValidation().as(Spec11PipelineOptions.class);
}
@Provides
EvaluateSafeBrowsingFn provideSafeBrowsingFn(Spec11PipelineOptions options, Retrier retrier) {
return new EvaluateSafeBrowsingFn(options.getSafeBrowsingApiKey(), retrier);
}
@Provides
Spec11Pipeline providePipeline(
Spec11PipelineOptions options, EvaluateSafeBrowsingFn safeBrowsingFn) {
return new Spec11Pipeline(options, safeBrowsingFn);
}
}
@Component(modules = {Spec11PipelineModule.class, UtilsModule.class, ConfigModule.class})
@Singleton
interface Spec11PipelineComponent {
Spec11Pipeline spec11Pipeline();
}
@AutoValue
abstract static class EmailAndThreatMatch implements Serializable {

View File

@@ -0,0 +1,37 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.spec11;
import google.registry.beam.common.RegistryPipelineOptions;
import org.apache.beam.sdk.options.Description;
/** Custom options for running the spec11 pipeline. */
public interface Spec11PipelineOptions extends RegistryPipelineOptions {
@Description("The local date we generate the report for, in yyyy-MM-dd format.")
String getDate();
void setDate(String value);
@Description("The API key we use to access the SafeBrowsing API.")
String getSafeBrowsingApiKey();
void setSafeBrowsingApiKey(String value);
@Description("The GCS bucket URL for Spec11 reports to be uploaded.")
String getReportingBucketUrl();
void setReportingBucketUrl(String value);
}

View File

@@ -15,6 +15,7 @@
package google.registry.beam.spec11;
import com.google.auto.value.AutoValue;
import com.google.common.annotations.VisibleForTesting;
import java.io.Serializable;
import org.json.JSONException;
import org.json.JSONObject;
@@ -31,16 +32,9 @@ public abstract class ThreatMatch implements Serializable {
/** Returns the fully qualified domain name [SLD].[TLD] of the matched threat. */
public abstract String fullyQualifiedDomainName();
/**
* Constructs a {@link ThreatMatch} by parsing a {@code SafeBrowsing API} response {@link
* JSONObject}.
*
* @throws JSONException when encountering parse errors in the response format
*/
static ThreatMatch create(JSONObject threatMatchJSON, String fullyQualifiedDomainName)
throws JSONException {
return new AutoValue_ThreatMatch(
threatMatchJSON.getString(THREAT_TYPE_FIELD), fullyQualifiedDomainName);
@VisibleForTesting
static ThreatMatch create(String threatType, String fullyQualifiedDomainName) {
return new AutoValue_ThreatMatch(threatType, fullyQualifiedDomainName);
}
/** Returns a {@link JSONObject} representing a subset of this object's data. */

View File

@@ -203,7 +203,7 @@ public final class RegistryConfig {
* Configuration for analytics services installed in the web console.
*
* @see google.registry.ui.server.registrar.ConsoleUiAction
* @see google.registry.ui.soy.AnalyticsSoyInfo
* @see google.registry.ui.soy.registrar.AnalyticsSoyInfo
*/
@Provides
@Config("analyticsConfig")
@@ -384,19 +384,6 @@ public final class RegistryConfig {
return Duration.standardHours(1);
}
/**
* Number of sharded entity group roots used for performing strongly consistent scans.
*
* <p><b>Warning:</b> This number may increase but never decrease.
*
* @see google.registry.model.index.EppResourceIndex
*/
@Provides
@Config("eppResourceIndexBucketCount")
public static int provideEppResourceIndexBucketCount(RegistryConfigSettings config) {
return config.datastore.eppResourceIndexBucketsNum;
}
@Provides
@Config("cloudSqlJdbcUrl")
public static String providesCloudSqlJdbcUrl(RegistryConfigSettings config) {
@@ -564,53 +551,6 @@ public final class RegistryConfig {
return config.gSuite.outgoingEmailDisplayName;
}
/**
* Returns the name of the GCS bucket for storing Beam templates and results.
*
* @see google.registry.reporting.billing.GenerateInvoicesAction
*/
@Provides
@Config("apacheBeamBucket")
public static String provideApacheBeamBucket(@Config("projectId") String projectId) {
return projectId + "-beam";
}
/**
* Returns the URL of the GCS location for storing Apache Beam related objects.
*
* @see google.registry.reporting.billing.GenerateInvoicesAction
*/
@Provides
@Config("apacheBeamBucketUrl")
public static String provideApacheBeamBucketUrl(@Config("apacheBeamBucket") String beamBucket) {
return "gs://" + beamBucket;
}
/**
* Returns the URL of the GCS location for storing the monthly invoicing Beam template.
*
* @see google.registry.reporting.billing.GenerateInvoicesAction
* @see google.registry.beam.invoicing.InvoicingPipeline
*/
@Provides
@Config("invoiceTemplateUrl")
public static String provideInvoiceTemplateUrl(
@Config("apacheBeamBucketUrl") String beamBucketUrl) {
return beamBucketUrl + "/templates/invoicing";
}
/**
* Returns the URL of the GCS location for storing the monthly spec11 Beam template.
*
* @see google.registry.beam.spec11.Spec11Pipeline
*/
@Provides
@Config("spec11TemplateUrl")
public static String provideSpec11TemplateUrl(
@Config("apacheBeamBucketUrl") String beamBucketUrl) {
return beamBucketUrl + "/templates/spec11";
}
/**
* Returns whether an SSL certificate hash is required to log in via EPP and run flows.
*
@@ -634,18 +574,6 @@ public final class RegistryConfig {
return config.beam.defaultJobRegion;
}
/**
* Returns the default job zone to run Apache Beam (Cloud Dataflow) jobs in.
*
* @see google.registry.reporting.billing.GenerateInvoicesAction
* @see google.registry.reporting.spec11.GenerateSpec11ReportAction
*/
@Provides
@Config("defaultJobZone")
public static String provideDefaultJobZone(RegistryConfigSettings config) {
return config.beam.defaultJobZone;
}
/** Returns the GCS bucket URL with all staged BEAM flex templates. */
@Provides
@Config("beamStagingBucketUrl")
@@ -653,19 +581,6 @@ public final class RegistryConfig {
return config.beam.stagingBucketUrl;
}
/**
* Returns the URL of the GCS location we store jar dependencies for beam pipelines.
*
* @see google.registry.beam.invoicing.InvoicingPipeline
* @see google.registry.beam.spec11.Spec11Pipeline
*/
@Provides
@Config("beamStagingUrl")
public static String provideInvoiceStagingUrl(
@Config("apacheBeamBucketUrl") String beamBucketUrl) {
return beamBucketUrl + "/staging";
}
/**
* Returns the Google Cloud Storage bucket for Spec11 and ICANN transaction and activity reports
* to be uploaded.
@@ -1227,14 +1142,6 @@ public final class RegistryConfig {
return formatComments(config.registryPolicy.reservedTermsExportDisclaimer);
}
/** Returns the clientId of the registrar used by the {@code CheckApiServlet}. */
// TODO(b/80417678): remove this once CheckApiAction no longer uses this id.
@Provides
@Config("checkApiServletRegistrarClientId")
public static String provideCheckApiServletRegistrarClientId(RegistryConfigSettings config) {
return config.registryPolicy.checkApiServletClientId;
}
/**
* Returns the clientId of the registrar that admins are automatically logged in as if they
* aren't otherwise associated with one.

View File

@@ -133,7 +133,6 @@ public class RegistryConfigSettings {
/** Configuration for Apache Beam (Cloud Dataflow). */
public static class Beam {
public String defaultJobRegion;
public String defaultJobZone;
public String stagingBucketUrl;
}

View File

@@ -420,9 +420,6 @@ misc:
beam:
# The default region to run Apache Beam (Cloud Dataflow) jobs in.
defaultJobRegion: us-east1
# The default zone to run Apache Beam (Cloud Dataflow) jobs in.
# TODO(weiminyu): consider dropping zone config. No obvious needs for this.
defaultJobZone: us-east1-c
stagingBucketUrl: gcs-bucket-with-staged-templates
keyring:

View File

@@ -1,5 +1,5 @@
# Example of a reserved list file. This is simply a CSV file with two
# columns: sub-domain name and price (specified as currency type and value).
# Example of a premium list file. This is simply a CSV file with two
# columns: sub-domain name, and price (specified as currency type and value).
#
# These are manipulated using the "nomulus" tool
# {create,update,delete,list}_premium_list commands.

View File

@@ -391,6 +391,12 @@
<url-pattern>/_dr/task/wipeOutDatastore</url-pattern>
</servlet-mapping>
<!-- Action to create synthetic history entries during async replication to SQL -->
<servlet-mapping>
<servlet-name>backend-servlet</servlet-name>
<url-pattern>/_dr/task/createSyntheticHistoryEntries</url-pattern>
</servlet-mapping>
<!-- Security config -->
<security-constraint>
<web-resource-collection>

View File

@@ -32,12 +32,12 @@ import com.google.common.net.MediaType;
import google.registry.config.RegistryConfig.Config;
import google.registry.model.registry.Registry;
import google.registry.model.registry.label.PremiumList.PremiumListEntry;
import google.registry.model.registry.label.PremiumListDualDao;
import google.registry.request.Action;
import google.registry.request.Parameter;
import google.registry.request.RequestParameters;
import google.registry.request.Response;
import google.registry.request.auth.Auth;
import google.registry.schema.tld.PremiumListDao;
import google.registry.storage.drive.DriveConnection;
import java.io.IOException;
import java.util.Optional;
@@ -113,7 +113,7 @@ public class ExportPremiumTermsAction implements Runnable {
"Skipping premium terms export for TLD %s because Drive folder isn't specified", tld);
return Optional.of("Skipping export because no Drive folder is associated with this TLD");
}
if (registry.getPremiumList() == null) {
if (!registry.getPremiumList().isPresent()) {
logger.atInfo().log("No premium terms to export for TLD %s", tld);
return Optional.of("No premium lists configured");
}
@@ -137,11 +137,13 @@ public class ExportPremiumTermsAction implements Runnable {
}
private String getFormattedPremiumTerms(Registry registry) {
String premiumListName = registry.getPremiumList().getName();
checkState(registry.getPremiumList().isPresent(), "%s does not have a premium list", tld);
String premiumListName = registry.getPremiumList().get().getName();
checkState(
PremiumListDualDao.exists(premiumListName), "Could not load premium list for " + tld);
PremiumListDao.getLatestRevision(premiumListName).isPresent(),
"Could not load premium list for " + tld);
SortedSet<String> premiumTerms =
Streams.stream(PremiumListDualDao.loadAllPremiumListEntries(premiumListName))
Streams.stream(PremiumListDao.loadAllPremiumListEntries(premiumListName))
.map(PremiumListEntry::toString)
.collect(ImmutableSortedSet.toImmutableSortedSet(String::compareTo));

View File

@@ -20,6 +20,8 @@ import com.google.common.base.Strings;
import dagger.Module;
import dagger.Provides;
import google.registry.flows.picker.FlowPicker;
import google.registry.model.contact.ContactHistory;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
import google.registry.model.eppcommon.Trid;
@@ -30,6 +32,7 @@ import google.registry.model.eppinput.ResourceCommand;
import google.registry.model.eppinput.ResourceCommand.SingleResourceCommand;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.eppoutput.Result;
import google.registry.model.host.HostHistory;
import google.registry.model.reporting.HistoryEntry;
import java.lang.annotation.Documented;
import java.util.Optional;
@@ -210,33 +213,78 @@ public class FlowModule {
return Strings.nullToEmpty(((Poll) eppInput.getCommandWrapper().getCommand()).getMessageId());
}
private static <B extends HistoryEntry.Builder<? extends HistoryEntry, ?>>
B makeHistoryEntryBuilder(
B builder,
Trid trid,
byte[] inputXmlBytes,
boolean isSuperuser,
String clientId,
EppInput eppInput) {
builder
.setTrid(trid)
.setXmlBytes(inputXmlBytes)
.setBySuperuser(isSuperuser)
.setClientId(clientId);
Optional<MetadataExtension> metadataExtension =
eppInput.getSingleExtension(MetadataExtension.class);
metadataExtension.ifPresent(
extension ->
builder
.setReason(extension.getReason())
.setRequestedByRegistrar(extension.getRequestedByRegistrar()));
return builder;
}
/**
* Provides a partially filled in {@link HistoryEntry} builder.
* Provides a partially filled in {@link ContactHistory.Builder}
*
* <p>This is not marked with {@link FlowScope} so that each retry gets a fresh one. Otherwise,
* the fact that the builder is one-use would cause NPEs.
*/
@Provides
static HistoryEntry.Builder provideHistoryEntryBuilder(
static ContactHistory.Builder provideContactHistoryBuilder(
Trid trid,
@InputXml byte[] inputXmlBytes,
@Superuser boolean isSuperuser,
@ClientId String clientId,
EppInput eppInput) {
HistoryEntry.Builder historyBuilder =
new HistoryEntry.Builder()
.setTrid(trid)
.setXmlBytes(inputXmlBytes)
.setBySuperuser(isSuperuser)
.setClientId(clientId);
Optional<MetadataExtension> metadataExtension =
eppInput.getSingleExtension(MetadataExtension.class);
if (metadataExtension.isPresent()) {
historyBuilder
.setReason(metadataExtension.get().getReason())
.setRequestedByRegistrar(metadataExtension.get().getRequestedByRegistrar());
}
return historyBuilder;
return makeHistoryEntryBuilder(
new ContactHistory.Builder(), trid, inputXmlBytes, isSuperuser, clientId, eppInput);
}
/**
* Provides a partially filled in {@link HostHistory.Builder}
*
* <p>This is not marked with {@link FlowScope} so that each retry gets a fresh one. Otherwise,
* the fact that the builder is one-use would cause NPEs.
*/
@Provides
static HostHistory.Builder provideHostHistoryBuilder(
Trid trid,
@InputXml byte[] inputXmlBytes,
@Superuser boolean isSuperuser,
@ClientId String clientId,
EppInput eppInput) {
return makeHistoryEntryBuilder(
new HostHistory.Builder(), trid, inputXmlBytes, isSuperuser, clientId, eppInput);
}
/**
* Provides a partially filled in {@link DomainHistory.Builder}
*
* <p>This is not marked with {@link FlowScope} so that each retry gets a fresh one. Otherwise,
* the fact that the builder is one-use would cause NPEs.
*/
@Provides
static DomainHistory.Builder provideDomainHistoryBuilder(
Trid trid,
@InputXml byte[] inputXmlBytes,
@Superuser boolean isSuperuser,
@ClientId String clientId,
EppInput eppInput) {
return makeHistoryEntryBuilder(
new DomainHistory.Builder(), trid, inputXmlBytes, isSuperuser, clientId, eppInput);
}
/**
@@ -249,7 +297,7 @@ public class FlowModule {
static EppResponse.Builder provideEppResponseBuilder(Trid trid) {
return new EppResponse.Builder()
.setTrid(trid)
.setResultFromCode(Result.Code.SUCCESS); // Default to success.
.setResultFromCode(Result.Code.SUCCESS); // Default to success.
}
@Provides

View File

@@ -22,15 +22,19 @@ import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.common.base.Throwables;
import com.google.common.flogger.FluentLogger;
import com.googlecode.objectify.Key;
import google.registry.flows.EppException.CommandUseErrorException;
import google.registry.flows.EppException.ParameterValueRangeErrorException;
import google.registry.flows.EppException.SyntaxErrorException;
import google.registry.flows.EppException.UnimplementedProtocolVersionException;
import google.registry.flows.custom.EntityChanges;
import google.registry.model.EppResource;
import google.registry.model.eppcommon.EppXmlTransformer;
import google.registry.model.eppinput.EppInput.WrongProtocolVersionException;
import google.registry.model.eppoutput.EppOutput;
import google.registry.model.host.InetAddressAdapter.IpVersionMismatchException;
import google.registry.model.ofy.ObjectifyService;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.translators.CurrencyUnitAdapter.UnknownCurrencyException;
import google.registry.xml.XmlException;
import java.util.List;
@@ -99,6 +103,11 @@ public final class FlowUtils {
}
}
public static <H extends HistoryEntry> Key<H> createHistoryKey(
EppResource parent, Class<H> clazz) {
return Key.create(Key.create(parent), clazz, ObjectifyService.allocateId());
}
/** Registrar is not logged in. */
public static class NotLoggedInException extends CommandUseErrorException {
public NotLoggedInException() {

View File

@@ -16,6 +16,7 @@ package google.registry.flows;
import static com.google.common.collect.Sets.intersection;
import static google.registry.model.EppResourceUtils.getLinkedDomainKeys;
import static google.registry.model.EppResourceUtils.isLinked;
import static google.registry.model.EppResourceUtils.loadByForeignKey;
import static google.registry.model.index.ForeignKeyIndex.loadAndGetKey;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
@@ -62,7 +63,10 @@ public final class ResourceFlowUtils {
private ResourceFlowUtils() {}
/** In {@link #failfastForAsyncDelete}, check this (arbitrary) number of query results. */
/**
* In {@link #checkLinkedDomains(String, DateTime, Class, Function)}, check this (arbitrary)
* number of query results.
*/
private static final int FAILFAST_CHECK_COUNT = 5;
/** Check that the given clientId corresponds to the owner of given resource. */
@@ -73,36 +77,54 @@ public final class ResourceFlowUtils {
}
}
/** Check whether an asynchronous delete would obviously fail, and throw an exception if so. */
public static <R extends EppResource> void failfastForAsyncDelete(
/**
* Check whether if there are domains linked to the resource to be deleted. Throws an exception if
* so.
*
* <p>Note that in datastore this is a smoke test as the query for linked domains is eventually
* consistent, so we only check a few domains to fail fast.
*/
public static <R extends EppResource> void checkLinkedDomains(
final String targetId,
final DateTime now,
final Class<R> resourceClass,
final Function<DomainBase, ImmutableSet<?>> getPotentialReferences) throws EppException {
// Enter a transactionless context briefly.
final Function<DomainBase, ImmutableSet<?>> getPotentialReferences)
throws EppException {
EppException failfastException =
tm().doTransactionless(
() -> {
final ForeignKeyIndex<R> fki = ForeignKeyIndex.load(resourceClass, targetId, now);
if (fki == null) {
return new ResourceDoesNotExistException(resourceClass, targetId);
}
/* Query for the first few linked domains, and if found, actually load them. The
* query is eventually consistent and so might be very stale, but the direct
* load will not be stale, just non-transactional. If we find at least one
* actual reference then we can reliably fail. If we don't find any, we can't
* trust the query and need to do the full mapreduce.
*/
Iterable<VKey<DomainBase>> keys =
getLinkedDomainKeys(fki.getResourceKey(), now, FAILFAST_CHECK_COUNT);
tm().isOfy()
? tm().doTransactionless(
() -> {
final ForeignKeyIndex<R> fki =
ForeignKeyIndex.load(resourceClass, targetId, now);
if (fki == null) {
return new ResourceDoesNotExistException(resourceClass, targetId);
}
// Query for the first few linked domains, and if found, actually load them.
// The query is eventually consistent and so might be very stale, but the
// direct load will not be stale, just non-transactional. If we find at least
// one actual reference then we can reliably fail. If we don't find any,
// we can't trust the query and need to do the full mapreduce.
Iterable<VKey<DomainBase>> keys =
getLinkedDomainKeys(fki.getResourceKey(), now, FAILFAST_CHECK_COUNT);
VKey<R> resourceVKey = fki.getResourceKey();
Predicate<DomainBase> predicate =
domain -> getPotentialReferences.apply(domain).contains(resourceVKey);
return tm().loadByKeys(keys).values().stream().anyMatch(predicate)
? new ResourceToDeleteIsReferencedException()
: null;
});
VKey<R> resourceVKey = fki.getResourceKey();
Predicate<DomainBase> predicate =
domain -> getPotentialReferences.apply(domain).contains(resourceVKey);
return tm().loadByKeys(keys).values().stream().anyMatch(predicate)
? new ResourceToDeleteIsReferencedException()
: null;
})
: tm().transact(
() -> {
final ForeignKeyIndex<R> fki =
ForeignKeyIndex.load(resourceClass, targetId, now);
if (fki == null) {
return new ResourceDoesNotExistException(resourceClass, targetId);
}
return isLinked(fki.getResourceKey(), now)
? new ResourceToDeleteIsReferencedException()
: null;
});
if (failfastException != null) {
throw failfastException;
}
@@ -123,8 +145,7 @@ public final class ResourceFlowUtils {
}
public static <R extends EppResource & ForeignKeyedEppResource> R loadAndVerifyExistence(
Class<R> clazz, String targetId, DateTime now)
throws ResourceDoesNotExistException {
Class<R> clazz, String targetId, DateTime now) throws ResourceDoesNotExistException {
return verifyExistence(clazz, targetId, loadByForeignKey(clazz, targetId, now));
}
@@ -156,16 +177,16 @@ public final class ResourceFlowUtils {
}
/** Check that the given AuthInfo is either missing or else is valid for the given resource. */
public static void verifyOptionalAuthInfo(
Optional<AuthInfo> authInfo, ContactResource contact) throws EppException {
public static void verifyOptionalAuthInfo(Optional<AuthInfo> authInfo, ContactResource contact)
throws EppException {
if (authInfo.isPresent()) {
verifyAuthInfo(authInfo.get(), contact);
}
}
/** Check that the given AuthInfo is either missing or else is valid for the given resource. */
public static void verifyOptionalAuthInfo(
Optional<AuthInfo> authInfo, DomainBase domain) throws EppException {
public static void verifyOptionalAuthInfo(Optional<AuthInfo> authInfo, DomainBase domain)
throws EppException {
if (authInfo.isPresent()) {
verifyAuthInfo(authInfo.get(), domain);
}
@@ -229,7 +250,7 @@ public final class ResourceFlowUtils {
/** Check that the same values aren't being added and removed in an update command. */
public static void checkSameValuesNotAddedAndRemoved(
ImmutableSet<?> fieldsToAdd, ImmutableSet<?> fieldsToRemove)
throws AddRemoveSameValueException {
throws AddRemoveSameValueException {
if (!intersection(fieldsToAdd, fieldsToRemove).isEmpty()) {
throw new AddRemoveSameValueException();
}

View File

@@ -16,7 +16,6 @@ package google.registry.flows;
import static com.google.common.base.MoreObjects.toStringHelper;
import static google.registry.request.RequestParameters.extractOptionalHeader;
import static google.registry.util.X509Utils.loadCertificate;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
@@ -26,24 +25,17 @@ import com.google.common.net.InetAddresses;
import dagger.Module;
import dagger.Provides;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryEnvironment;
import google.registry.flows.EppException.AuthenticationErrorException;
import google.registry.flows.certs.CertificateChecker;
import google.registry.flows.certs.CertificateChecker.InsecureCertificateException;
import google.registry.model.registrar.Registrar;
import google.registry.request.Header;
import google.registry.util.CidrAddressBlock;
import google.registry.util.Clock;
import google.registry.util.ProxyHttpHeaders;
import java.io.ByteArrayInputStream;
import java.net.InetAddress;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Base64;
import java.util.Optional;
import javax.inject.Inject;
import javax.servlet.http.HttpServletRequest;
import org.joda.time.DateTime;
/**
* Container and validation for TLS certificate and IP-allow-listing.
@@ -54,10 +46,6 @@ import org.joda.time.DateTime;
* <dt>X-SSL-Certificate
* <dd>This field should contain a base64 encoded digest of the client's TLS certificate. It is
* used only if the validation of the full certificate fails.
* <dt>X-SSL-Full-Certificate
* <dd>This field should contain a base64 encoding of the client's TLS certificate. It is
* validated during an EPP login command against a known good value that is transmitted out of
* band.
* <dt>X-Forwarded-For
* <dd>This field should contain the host and port of the connecting client. It is validated
* during an EPP login command against an IP allow list that is transmitted out of band.
@@ -66,30 +54,22 @@ import org.joda.time.DateTime;
public class TlsCredentials implements TransportCredentials {
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private static final DateTime CERT_ENFORCEMENT_START_TIME =
DateTime.parse("2021-03-01T16:00:00Z");
private final boolean requireSslCertificates;
private final Optional<String> clientCertificateHash;
private final Optional<String> clientCertificate;
private final Optional<InetAddress> clientInetAddr;
private final CertificateChecker certificateChecker;
private final Clock clock;
@Inject
public TlsCredentials(
@Config("requireSslCertificates") boolean requireSslCertificates,
@Header(ProxyHttpHeaders.CERTIFICATE_HASH) Optional<String> clientCertificateHash,
@Header(ProxyHttpHeaders.FULL_CERTIFICATE) Optional<String> clientCertificate,
@Header(ProxyHttpHeaders.IP_ADDRESS) Optional<String> clientAddress,
CertificateChecker certificateChecker,
Clock clock) {
CertificateChecker certificateChecker) {
this.requireSslCertificates = requireSslCertificates;
this.clientCertificateHash = clientCertificateHash;
this.clientCertificate = clientCertificate;
this.clientInetAddr = clientAddress.map(TlsCredentials::parseInetAddress);
this.certificateChecker = certificateChecker;
this.clock = clock;
}
static InetAddress parseInetAddress(String asciiAddr) {
@@ -103,7 +83,7 @@ public class TlsCredentials implements TransportCredentials {
@Override
public void validate(Registrar registrar, String password) throws AuthenticationErrorException {
validateIp(registrar);
validateCertificate(registrar);
validateCertificateHash(registrar);
validatePassword(registrar, password);
}
@@ -137,89 +117,8 @@ public class TlsCredentials implements TransportCredentials {
throw new BadRegistrarIpAddressException();
}
/**
* Verifies client SSL certificate is permitted to issue commands as {@code registrar}.
*
* @throws MissingRegistrarCertificateException if frontend didn't send certificate header
* @throws BadRegistrarCertificateException if registrar requires certificate and it didn't match
*/
@VisibleForTesting
void validateCertificate(Registrar registrar) throws AuthenticationErrorException {
// Check that certificate is present in registrar object
if (!registrar.getClientCertificate().isPresent()
&& !registrar.getFailoverClientCertificate().isPresent()) {
// Log an error and validate using certificate hash instead
// TODO(sarahbot): throw a RegistrarCertificateNotConfiguredException once hash is no longer
// used as failover
logger.atWarning().log(
"There is no certificate configured for registrar %s.", registrar.getClientId());
} else if (!clientCertificate.isPresent()) {
// Check that the request included the full certificate
// Log an error and validate using certificate hash instead
// TODO(sarahbot): throw a MissingRegistrarCertificateException once hash is no longer used as
// failover
logger.atWarning().log(
"Request from registrar %s did not include X-SSL-Full-Certificate.",
registrar.getClientId());
} else {
X509Certificate passedCert;
Optional<X509Certificate> storedCert;
Optional<X509Certificate> storedFailoverCert;
try {
storedCert = deserializePemCert(registrar.getClientCertificate());
storedFailoverCert = deserializePemCert(registrar.getFailoverClientCertificate());
passedCert = decodeCertString(clientCertificate.get());
} catch (Exception e) {
// TODO(Sarahbot@): remove this catch once we know it's working
logger.atWarning().log(
"Error converting certificate string to certificate for %s: %s",
registrar.getClientId(), e);
validateCertificateHash(registrar);
return;
}
// Check if the certificate is equal to the one on file for the registrar.
if (passedCert.equals(storedCert.orElse(null))
|| passedCert.equals(storedFailoverCert.orElse(null))) {
// Check certificate for any requirement violations
// TODO(Sarahbot@): Throw exceptions instead of just logging once requirement enforcement
// begins
try {
certificateChecker.validateCertificate(passedCert);
} catch (InsecureCertificateException e) {
// TODO(Sarahbot@): Remove this if statement after March 1. After March 1, exception
// should be thrown in all environments.
// throw exception in unit tests and Sandbox
if (RegistryEnvironment.get().equals(RegistryEnvironment.UNITTEST)
|| RegistryEnvironment.get().equals(RegistryEnvironment.SANDBOX)
|| clock.nowUtc().isAfter(CERT_ENFORCEMENT_START_TIME)) {
throw new CertificateContainsSecurityViolationsException(e);
}
logger.atWarning().log(
"Registrar certificate used for %s does not meet certificate requirements: %s",
registrar.getClientId(), e.getMessage());
} catch (Exception e) {
logger.atWarning().log(
"Error validating certificate for %s: %s", registrar.getClientId(), e);
}
// successfully validated, return here since hash validation is not necessary
return;
}
// Log an error and validate using certificate hash instead
// TODO(sarahbot): throw a BadRegistrarCertificateException once hash is no longer used as
// failover
logger.atWarning().log("Non-matching certificate for registrar %s.", registrar.getClientId());
}
validateCertificateHash(registrar);
}
private void validateCertificateHash(Registrar registrar) throws AuthenticationErrorException {
logger.atWarning().log(
"Error validating certificate for %s, attempting to validate using certificate hash.",
registrar.getClientId());
// Check the certificate hash as a failover
// TODO(sarahbot): Remove hash checks once certificate checks are working.
void validateCertificateHash(Registrar registrar) throws AuthenticationErrorException {
if (!registrar.getClientCertificateHash().isPresent()
&& !registrar.getFailoverClientCertificateHash().isPresent()) {
if (requireSslCertificates) {
@@ -247,6 +146,20 @@ public class TlsCredentials implements TransportCredentials {
registrar.getFailoverClientCertificateHash());
throw new BadRegistrarCertificateException();
}
if (requireSslCertificates) {
String passedCert =
clientCertificateHash.equals(registrar.getClientCertificateHash())
? registrar.getClientCertificate().get()
: registrar.getFailoverClientCertificate().get();
try {
certificateChecker.validateCertificate(passedCert);
} catch (InsecureCertificateException e) {
logger.atWarning().log(
"Registrar certificate used for %s does not meet certificate requirements: %s",
registrar.getClientId(), e.getMessage());
throw new CertificateContainsSecurityViolationsException(e);
}
}
}
private void validatePassword(Registrar registrar, String password)
@@ -256,26 +169,9 @@ public class TlsCredentials implements TransportCredentials {
}
}
// Converts a PEM formatted certificate string into an X509Certificate
private Optional<X509Certificate> deserializePemCert(Optional<String> certificateString)
throws CertificateException {
if (certificateString.isPresent()) {
return Optional.of(loadCertificate(certificateString.get()));
}
return Optional.empty();
}
// Decodes the string representation of an encoded certificate back into an X509Certificate
private X509Certificate decodeCertString(String encodedCertString) throws CertificateException {
byte decodedCert[] = Base64.getDecoder().decode(encodedCertString);
ByteArrayInputStream inputStream = new ByteArrayInputStream(decodedCert);
return loadCertificate(inputStream);
}
@Override
public String toString() {
return toStringHelper(getClass())
.add("clientCertificate", clientCertificate.orElse(null))
.add("clientCertificateHash", clientCertificateHash.orElse(null))
.add("clientAddress", clientInetAddr.orElse(null))
.toString();
@@ -336,14 +232,6 @@ public class TlsCredentials implements TransportCredentials {
return extractOptionalHeader(req, ProxyHttpHeaders.CERTIFICATE_HASH);
}
@Provides
@Header(ProxyHttpHeaders.FULL_CERTIFICATE)
static Optional<String> provideClientCertificate(HttpServletRequest req) {
// Note: This header is actually required, we just want to handle its absence explicitly
// by throwing an EPP exception rather than a generic Bad Request exception.
return extractOptionalHeader(req, ProxyHttpHeaders.FULL_CERTIFICATE);
}
@Provides
@Header(ProxyHttpHeaders.IP_ADDRESS)
static Optional<String> provideIpAddress(HttpServletRequest req) {

View File

@@ -33,6 +33,7 @@ import google.registry.flows.annotations.ReportingSpec;
import google.registry.flows.exceptions.ResourceAlreadyExistsForThisClientException;
import google.registry.flows.exceptions.ResourceCreateContentionException;
import google.registry.model.contact.ContactCommand.Create;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppinput.ResourceCommand;
@@ -61,7 +62,7 @@ public final class ContactCreateFlow implements TransactionalFlow {
@Inject ExtensionManager extensionManager;
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject @Config("contactAndHostRoidSuffix") String roidSuffix;
@Inject ContactCreateFlow() {}
@@ -93,12 +94,12 @@ public final class ContactCreateFlow implements TransactionalFlow {
historyBuilder
.setType(HistoryEntry.Type.CONTACT_CREATE)
.setModificationTime(now)
.setXmlBytes(null) // We don't want to store contact details in the history entry.
.setParent(Key.create(newContact));
.setXmlBytes(null) // We don't want to store contact details in the history entry.
.setContact(newContact);
tm().insertAll(
ImmutableSet.of(
newContact,
historyBuilder.build().toChildHistoryEntity(),
historyBuilder.build(),
ForeignKeyIndex.create(newContact, newContact.getDeletionTime()),
EppResourceIndex.create(Key.create(newContact))));
return responseBuilder

View File

@@ -15,16 +15,19 @@
package google.registry.flows.contact;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.failfastForAsyncDelete;
import static google.registry.flows.ResourceFlowUtils.checkLinkedDomains;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.ResourceFlowUtils.verifyNoDisallowedStatuses;
import static google.registry.flows.ResourceFlowUtils.verifyOptionalAuthInfo;
import static google.registry.flows.ResourceFlowUtils.verifyResourceOwnership;
import static google.registry.model.ResourceTransferUtils.denyPendingTransfer;
import static google.registry.model.ResourceTransferUtils.handlePendingTransferOnDelete;
import static google.registry.model.eppoutput.Result.Code.SUCCESS;
import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
import static google.registry.model.transfer.TransferStatus.SERVER_CANCELLED;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import google.registry.batch.AsyncTaskEnqueuer;
import google.registry.flows.EppException;
import google.registry.flows.ExtensionManager;
@@ -33,6 +36,7 @@ import google.registry.flows.FlowModule.Superuser;
import google.registry.flows.FlowModule.TargetId;
import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.metadata.MetadataExtension;
@@ -40,7 +44,8 @@ import google.registry.model.eppcommon.AuthInfo;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.eppcommon.Trid;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.eppoutput.Result.Code;
import google.registry.model.reporting.HistoryEntry.Type;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import java.util.Optional;
import javax.inject.Inject;
@@ -63,10 +68,11 @@ import org.joda.time.DateTime;
@ReportingSpec(ActivityReportField.CONTACT_DELETE)
public final class ContactDeleteFlow implements TransactionalFlow {
private static final ImmutableSet<StatusValue> DISALLOWED_STATUSES = ImmutableSet.of(
StatusValue.CLIENT_DELETE_PROHIBITED,
StatusValue.PENDING_DELETE,
StatusValue.SERVER_DELETE_PROHIBITED);
private static final ImmutableSet<StatusValue> DISALLOWED_STATUSES =
ImmutableSet.of(
StatusValue.CLIENT_DELETE_PROHIBITED,
StatusValue.PENDING_DELETE,
StatusValue.SERVER_DELETE_PROHIBITED);
@Inject ExtensionManager extensionManager;
@Inject @ClientId String clientId;
@@ -74,10 +80,12 @@ public final class ContactDeleteFlow implements TransactionalFlow {
@Inject Trid trid;
@Inject @Superuser boolean isSuperuser;
@Inject Optional<AuthInfo> authInfo;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject AsyncTaskEnqueuer asyncTaskEnqueuer;
@Inject EppResponse.Builder responseBuilder;
@Inject ContactDeleteFlow() {}
@Inject
ContactDeleteFlow() {}
@Override
public final EppResponse run() throws EppException {
@@ -85,23 +93,45 @@ public final class ContactDeleteFlow implements TransactionalFlow {
extensionManager.validate();
validateClientIsLoggedIn(clientId);
DateTime now = tm().getTransactionTime();
failfastForAsyncDelete(targetId, now, ContactResource.class, DomainBase::getReferencedContacts);
checkLinkedDomains(targetId, now, ContactResource.class, DomainBase::getReferencedContacts);
ContactResource existingContact = loadAndVerifyExistence(ContactResource.class, targetId, now);
verifyNoDisallowedStatuses(existingContact, DISALLOWED_STATUSES);
verifyOptionalAuthInfo(authInfo, existingContact);
if (!isSuperuser) {
verifyResourceOwnership(clientId, existingContact);
}
asyncTaskEnqueuer.enqueueAsyncDelete(
existingContact, tm().getTransactionTime(), clientId, trid, isSuperuser);
ContactResource newContact =
existingContact.asBuilder().addStatusValue(StatusValue.PENDING_DELETE).build();
historyBuilder
.setType(HistoryEntry.Type.CONTACT_PENDING_DELETE)
.setModificationTime(now)
.setParent(Key.create(existingContact));
tm().insert(historyBuilder.build().toChildHistoryEntity());
Type historyEntryType;
Code resultCode;
ContactResource newContact;
if (tm().isOfy()) {
asyncTaskEnqueuer.enqueueAsyncDelete(
existingContact, tm().getTransactionTime(), clientId, trid, isSuperuser);
newContact = existingContact.asBuilder().addStatusValue(StatusValue.PENDING_DELETE).build();
historyEntryType = Type.CONTACT_PENDING_DELETE;
resultCode = SUCCESS_WITH_ACTION_PENDING;
} else {
// Handle pending transfers on contact deletion.
newContact =
existingContact.getStatusValues().contains(StatusValue.PENDING_TRANSFER)
? denyPendingTransfer(existingContact, SERVER_CANCELLED, now, clientId)
: existingContact;
// Wipe out PII on contact deletion.
newContact =
newContact.asBuilder().wipeOut().setStatusValues(null).setDeletionTime(now).build();
historyEntryType = Type.CONTACT_DELETE;
resultCode = SUCCESS;
}
ContactHistory contactHistory =
historyBuilder
.setType(historyEntryType)
.setModificationTime(now)
.setContact(newContact)
.build();
if (!tm().isOfy()) {
handlePendingTransferOnDelete(existingContact, newContact, now, contactHistory);
}
tm().insert(contactHistory);
tm().update(newContact);
return responseBuilder.setResultFromCode(SUCCESS_WITH_ACTION_PENDING).build();
return responseBuilder.setResultFromCode(resultCode).build();
}
}

View File

@@ -20,19 +20,21 @@ import com.google.common.base.CharMatcher;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Sets;
import com.googlecode.objectify.Key;
import google.registry.flows.EppException;
import google.registry.flows.EppException.ParameterValuePolicyErrorException;
import google.registry.flows.EppException.ParameterValueSyntaxErrorException;
import google.registry.model.contact.ContactAddress;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.contact.PostalInfo;
import google.registry.model.poll.PendingActionNotificationResponse.ContactPendingActionNotificationResponse;
import google.registry.model.poll.PollMessage;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.transfer.TransferData;
import google.registry.model.transfer.TransferResponse.ContactTransferResponse;
import java.util.Set;
import javax.annotation.Nullable;
import org.joda.time.DateTime;
/** Static utility functions for contact flows. */
public class ContactFlowUtils {
@@ -66,31 +68,35 @@ public class ContactFlowUtils {
/** Create a poll message for the gaining client in a transfer. */
static PollMessage createGainingTransferPollMessage(
String targetId, TransferData transferData, HistoryEntry historyEntry) {
String targetId,
TransferData transferData,
DateTime now,
Key<ContactHistory> contactHistoryKey) {
return new PollMessage.OneTime.Builder()
.setClientId(transferData.getGainingClientId())
.setEventTime(transferData.getPendingTransferExpirationTime())
.setMsg(transferData.getTransferStatus().getMessage())
.setResponseData(ImmutableList.of(
createTransferResponse(targetId, transferData),
ContactPendingActionNotificationResponse.create(
targetId,
transferData.getTransferStatus().isApproved(),
transferData.getTransferRequestTrid(),
historyEntry.getModificationTime())))
.setParent(historyEntry)
.setResponseData(
ImmutableList.of(
createTransferResponse(targetId, transferData),
ContactPendingActionNotificationResponse.create(
targetId,
transferData.getTransferStatus().isApproved(),
transferData.getTransferRequestTrid(),
now)))
.setParentKey(contactHistoryKey)
.build();
}
/** Create a poll message for the losing client in a transfer. */
static PollMessage createLosingTransferPollMessage(
String targetId, TransferData transferData, HistoryEntry historyEntry) {
String targetId, TransferData transferData, Key<ContactHistory> contactHistoryKey) {
return new PollMessage.OneTime.Builder()
.setClientId(transferData.getLosingClientId())
.setEventTime(transferData.getPendingTransferExpirationTime())
.setMsg(transferData.getTransferStatus().getMessage())
.setResponseData(ImmutableList.of(createTransferResponse(targetId, transferData)))
.setParent(historyEntry)
.setParentKey(contactHistoryKey)
.build();
}

View File

@@ -32,6 +32,7 @@ import google.registry.flows.FlowModule.ClientId;
import google.registry.flows.FlowModule.TargetId;
import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
@@ -66,7 +67,7 @@ public final class ContactTransferApproveFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject Optional<AuthInfo> authInfo;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject ContactTransferApproveFlow() {}
@@ -86,15 +87,17 @@ public final class ContactTransferApproveFlow implements TransactionalFlow {
verifyResourceOwnership(clientId, existingContact);
ContactResource newContact =
approvePendingTransfer(existingContact, TransferStatus.CLIENT_APPROVED, now);
HistoryEntry historyEntry = historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_APPROVE)
.setModificationTime(now)
.setParent(Key.create(existingContact))
.build();
ContactHistory contactHistory =
historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_APPROVE)
.setModificationTime(now)
.setContact(newContact)
.build();
// Create a poll message for the gaining client.
PollMessage gainingPollMessage =
createGainingTransferPollMessage(targetId, newContact.getTransferData(), historyEntry);
tm().insertAll(ImmutableSet.of(historyEntry.toChildHistoryEntity(), gainingPollMessage));
createGainingTransferPollMessage(
targetId, newContact.getTransferData(), now, Key.create(contactHistory));
tm().insertAll(ImmutableSet.of(contactHistory, gainingPollMessage));
tm().update(newContact);
// Delete the billing event and poll messages that were written in case the transfer would have
// been implicitly server approved.

View File

@@ -32,6 +32,7 @@ import google.registry.flows.FlowModule.ClientId;
import google.registry.flows.FlowModule.TargetId;
import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
@@ -66,7 +67,7 @@ public final class ContactTransferCancelFlow implements TransactionalFlow {
@Inject Optional<AuthInfo> authInfo;
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject ContactTransferCancelFlow() {}
@@ -82,15 +83,17 @@ public final class ContactTransferCancelFlow implements TransactionalFlow {
verifyTransferInitiator(clientId, existingContact);
ContactResource newContact =
denyPendingTransfer(existingContact, TransferStatus.CLIENT_CANCELLED, now, clientId);
HistoryEntry historyEntry = historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_CANCEL)
.setModificationTime(now)
.setParent(Key.create(existingContact))
.build();
ContactHistory contactHistory =
historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_CANCEL)
.setModificationTime(now)
.setContact(newContact)
.build();
// Create a poll message for the losing client.
PollMessage losingPollMessage =
createLosingTransferPollMessage(targetId, newContact.getTransferData(), historyEntry);
tm().insertAll(ImmutableSet.of(historyEntry.toChildHistoryEntity(), losingPollMessage));
createLosingTransferPollMessage(
targetId, newContact.getTransferData(), Key.create(contactHistory));
tm().insertAll(ImmutableSet.of(contactHistory, losingPollMessage));
tm().update(newContact);
// Delete the billing event and poll messages that were written in case the transfer would have
// been implicitly server approved.

View File

@@ -32,6 +32,7 @@ import google.registry.flows.FlowModule.ClientId;
import google.registry.flows.FlowModule.TargetId;
import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
@@ -64,7 +65,7 @@ public final class ContactTransferRejectFlow implements TransactionalFlow {
@Inject Optional<AuthInfo> authInfo;
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject ContactTransferRejectFlow() {}
@@ -80,14 +81,16 @@ public final class ContactTransferRejectFlow implements TransactionalFlow {
verifyResourceOwnership(clientId, existingContact);
ContactResource newContact =
denyPendingTransfer(existingContact, TransferStatus.CLIENT_REJECTED, now, clientId);
HistoryEntry historyEntry = historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_REJECT)
.setModificationTime(now)
.setParent(Key.create(existingContact))
.build();
ContactHistory contactHistory =
historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_REJECT)
.setModificationTime(now)
.setContact(newContact)
.build();
PollMessage gainingPollMessage =
createGainingTransferPollMessage(targetId, newContact.getTransferData(), historyEntry);
tm().insertAll(ImmutableSet.of(historyEntry.toChildHistoryEntity(), gainingPollMessage));
createGainingTransferPollMessage(
targetId, newContact.getTransferData(), now, Key.create(contactHistory));
tm().insertAll(ImmutableSet.of(contactHistory, gainingPollMessage));
tm().update(newContact);
// Delete the billing event and poll messages that were written in case the transfer would have
// been implicitly server approved.

View File

@@ -14,6 +14,7 @@
package google.registry.flows.contact;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.ResourceFlowUtils.verifyAuthInfo;
@@ -36,6 +37,7 @@ import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.flows.exceptions.AlreadyPendingTransferException;
import google.registry.flows.exceptions.ObjectAlreadySponsoredException;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
@@ -81,7 +83,8 @@ public final class ContactTransferRequestFlow implements TransactionalFlow {
@Inject @ClientId String gainingClientId;
@Inject @TargetId String targetId;
@Inject @Config("contactAutomaticTransferLength") Duration automaticTransferLength;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject Trid trid;
@Inject EppResponse.Builder responseBuilder;
@Inject ContactTransferRequestFlow() {}
@@ -105,11 +108,7 @@ public final class ContactTransferRequestFlow implements TransactionalFlow {
throw new ObjectAlreadySponsoredException();
}
verifyNoDisallowedStatuses(existingContact, DISALLOWED_STATUSES);
HistoryEntry historyEntry = historyBuilder
.setType(HistoryEntry.Type.CONTACT_TRANSFER_REQUEST)
.setModificationTime(now)
.setParent(Key.create(existingContact))
.build();
DateTime transferExpirationTime = now.plus(automaticTransferLength);
ContactTransferData serverApproveTransferData =
new ContactTransferData.Builder()
@@ -120,12 +119,18 @@ public final class ContactTransferRequestFlow implements TransactionalFlow {
.setPendingTransferExpirationTime(transferExpirationTime)
.setTransferStatus(TransferStatus.SERVER_APPROVED)
.build();
Key<ContactHistory> contactHistoryKey = createHistoryKey(existingContact, ContactHistory.class);
historyBuilder
.setId(contactHistoryKey.getId())
.setType(HistoryEntry.Type.CONTACT_TRANSFER_REQUEST)
.setModificationTime(now);
// If the transfer is server approved, this message will be sent to the losing registrar. */
PollMessage serverApproveLosingPollMessage =
createLosingTransferPollMessage(targetId, serverApproveTransferData, historyEntry);
createLosingTransferPollMessage(targetId, serverApproveTransferData, contactHistoryKey);
// If the transfer is server approved, this message will be sent to the gaining registrar. */
PollMessage serverApproveGainingPollMessage =
createGainingTransferPollMessage(targetId, serverApproveTransferData, historyEntry);
createGainingTransferPollMessage(
targetId, serverApproveTransferData, now, contactHistoryKey);
ContactTransferData pendingTransferData =
serverApproveTransferData
.asBuilder()
@@ -137,8 +142,9 @@ public final class ContactTransferRequestFlow implements TransactionalFlow {
.build();
// When a transfer is requested, a poll message is created to notify the losing registrar.
PollMessage requestPollMessage =
createLosingTransferPollMessage(targetId, pendingTransferData, historyEntry).asBuilder()
.setEventTime(now) // Unlike the serverApprove messages, this applies immediately.
createLosingTransferPollMessage(targetId, pendingTransferData, contactHistoryKey)
.asBuilder()
.setEventTime(now) // Unlike the serverApprove messages, this applies immediately.
.build();
ContactResource newContact = existingContact.asBuilder()
.setTransferData(pendingTransferData)
@@ -147,7 +153,7 @@ public final class ContactTransferRequestFlow implements TransactionalFlow {
tm().update(newContact);
tm().insertAll(
ImmutableSet.of(
historyEntry.toChildHistoryEntity(),
historyBuilder.setContact(newContact).build(),
requestPollMessage,
serverApproveGainingPollMessage,
serverApproveLosingPollMessage));

View File

@@ -27,7 +27,6 @@ import static google.registry.flows.contact.ContactFlowUtils.validateContactAgai
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import google.registry.flows.EppException;
import google.registry.flows.ExtensionManager;
import google.registry.flows.FlowModule.ClientId;
@@ -38,6 +37,7 @@ import google.registry.flows.annotations.ReportingSpec;
import google.registry.flows.exceptions.ResourceHasClientUpdateProhibitedException;
import google.registry.model.contact.ContactCommand.Update;
import google.registry.model.contact.ContactCommand.Update.Change;
import google.registry.model.contact.ContactHistory;
import google.registry.model.contact.ContactResource;
import google.registry.model.contact.PostalInfo;
import google.registry.model.domain.metadata.MetadataExtension;
@@ -82,7 +82,7 @@ public final class ContactUpdateFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject ContactHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject ContactUpdateFlow() {}
@@ -102,11 +102,6 @@ public final class ContactUpdateFlow implements TransactionalFlow {
verifyAllStatusesAreClientSettable(union(statusesToAdd, statusToRemove));
}
verifyNoDisallowedStatuses(existingContact, DISALLOWED_STATUSES);
historyBuilder
.setType(HistoryEntry.Type.CONTACT_UPDATE)
.setModificationTime(now)
.setXmlBytes(null) // We don't want to store contact details in the history entry.
.setParent(Key.create(existingContact));
checkSameValuesNotAddedAndRemoved(statusesToAdd, statusToRemove);
ContactResource.Builder builder = existingContact.asBuilder();
Change change = command.getInnerChange();
@@ -150,7 +145,12 @@ public final class ContactUpdateFlow implements TransactionalFlow {
}
validateAsciiPostalInfo(newContact.getInternationalizedPostalInfo());
validateContactAgainstPolicy(newContact);
tm().insert(historyBuilder.build().toChildHistoryEntity());
historyBuilder
.setType(HistoryEntry.Type.CONTACT_UPDATE)
.setModificationTime(now)
.setXmlBytes(null) // We don't want to store contact details in the history entry.
.setContact(newContact);
tm().insert(historyBuilder.build());
tm().update(newContact);
return responseBuilder.build();
}

View File

@@ -45,7 +45,7 @@ import google.registry.model.eppinput.ResourceCommand;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.registry.Registry;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.model.tmch.ClaimsListDualDatabaseDao;
import google.registry.model.tmch.ClaimsListDao;
import google.registry.util.Clock;
import java.util.HashSet;
import java.util.Optional;
@@ -104,8 +104,7 @@ public final class DomainClaimsCheckFlow implements Flow {
verifyClaimsPeriodNotEnded(registry, now);
}
}
Optional<String> claimKey =
ClaimsListDualDatabaseDao.get().getClaimKey(parsedDomain.parts().get(0));
Optional<String> claimKey = ClaimsListDao.get().getClaimKey(parsedDomain.parts().get(0));
launchChecksBuilder.add(
LaunchCheck.create(
LaunchCheckName.create(claimKey.isPresent(), domainName), claimKey.orElse(null)));

View File

@@ -80,6 +80,7 @@ import google.registry.model.billing.BillingEvent.Recurring;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand;
import google.registry.model.domain.DomainCommand.Create;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.GracePeriod;
import google.registry.model.domain.Period;
import google.registry.model.domain.fee.FeeCreateCommandExtension;
@@ -96,7 +97,6 @@ import google.registry.model.eppinput.EppInput;
import google.registry.model.eppinput.ResourceCommand;
import google.registry.model.eppoutput.CreateData.DomainCreateData;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.host.HostResource;
import google.registry.model.index.EppResourceIndex;
import google.registry.model.index.ForeignKeyIndex;
import google.registry.model.ofy.ObjectifyService;
@@ -106,11 +106,11 @@ import google.registry.model.poll.PollMessage.Autorenew;
import google.registry.model.registry.Registry;
import google.registry.model.registry.Registry.TldState;
import google.registry.model.registry.Registry.TldType;
import google.registry.model.registry.label.ReservationType;
import google.registry.model.reporting.DomainTransactionRecord;
import google.registry.model.reporting.DomainTransactionRecord.TransactionReportField;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.persistence.VKey;
import google.registry.tmch.LordnTaskUtils;
import java.util.Optional;
import javax.inject.Inject;
@@ -206,7 +206,7 @@ public class DomainCreateFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject AllocationTokenFlowUtils allocationTokenFlowUtils;
@Inject DomainCreateFlowCustomLogic flowCustomLogic;
@@ -301,10 +301,14 @@ public class DomainCreateFlow implements TransactionalFlow {
validateFeeChallenge(targetId, now, feeCreate, feesAndCredits);
Optional<SecDnsCreateExtension> secDnsCreate =
validateSecDnsExtension(eppInput.getSingleExtension(SecDnsCreateExtension.class));
String repoId = createDomainRepoId(ObjectifyService.allocateId(), registry.getTldStr());
DateTime registrationExpirationTime = leapSafeAddYears(now, years);
HistoryEntry historyEntry = buildHistoryEntry(
repoId, registry, now, period, registry.getAddGracePeriodLength());
String repoId = createDomainRepoId(ObjectifyService.allocateId(), registry.getTldStr());
Key<DomainHistory> domainHistoryKey =
Key.create(
Key.create(DomainBase.class, repoId),
DomainHistory.class,
ObjectifyService.allocateId());
historyBuilder.setId(domainHistoryKey.getId());
// Bill for the create.
BillingEvent.OneTime createBillingEvent =
createOneTimeBillingEvent(
@@ -314,33 +318,27 @@ public class DomainCreateFlow implements TransactionalFlow {
isReserved(domainName, isSunriseCreate),
years,
feesAndCredits,
historyEntry,
domainHistoryKey,
allocationToken,
now);
// Create a new autorenew billing event and poll message starting at the expiration time.
BillingEvent.Recurring autorenewBillingEvent =
createAutorenewBillingEvent(historyEntry, registrationExpirationTime);
createAutorenewBillingEvent(domainHistoryKey, registrationExpirationTime);
PollMessage.Autorenew autorenewPollMessage =
createAutorenewPollMessage(historyEntry, registrationExpirationTime);
createAutorenewPollMessage(domainHistoryKey, registrationExpirationTime);
ImmutableSet.Builder<ImmutableObject> entitiesToSave = new ImmutableSet.Builder<>();
entitiesToSave.add(
historyEntry,
createBillingEvent,
autorenewBillingEvent,
autorenewPollMessage);
entitiesToSave.add(createBillingEvent, autorenewBillingEvent, autorenewPollMessage);
// Bill for EAP cost, if any.
if (!feesAndCredits.getEapCost().isZero()) {
entitiesToSave.add(createEapBillingEvent(feesAndCredits, createBillingEvent));
}
ImmutableSet.Builder<StatusValue> statuses = new ImmutableSet.Builder<>();
if (getReservationTypes(domainName).contains(NAME_COLLISION)) {
statuses.add(SERVER_HOLD);
entitiesToSave.add(
createNameCollisionOneTimePollMessage(targetId, historyEntry, clientId, now));
}
DomainBase newDomain =
ImmutableSet<ReservationType> reservationTypes = getReservationTypes(domainName);
ImmutableSet<StatusValue> statuses =
reservationTypes.contains(NAME_COLLISION)
? ImmutableSet.of(SERVER_HOLD)
: ImmutableSet.of();
DomainBase domain =
new DomainBase.Builder()
.setCreationClientId(clientId)
.setPersistedCurrentSponsorClientId(clientId)
@@ -351,35 +349,39 @@ public class DomainCreateFlow implements TransactionalFlow {
.setAutorenewPollMessage(autorenewPollMessage.createVKey())
.setLaunchNotice(hasClaimsNotice ? launchCreate.get().getNotice() : null)
.setSmdId(signedMarkId)
.setDsData(secDnsCreate.isPresent() ? secDnsCreate.get().getDsData() : null)
.setDsData(secDnsCreate.map(SecDnsCreateExtension::getDsData).orElse(null))
.setRegistrant(command.getRegistrant())
.setAuthInfo(command.getAuthInfo())
.setDomainName(targetId)
.setNameservers(
(ImmutableSet<VKey<HostResource>>)
command.getNameservers().stream().collect(toImmutableSet()))
.setStatusValues(statuses.build())
.setNameservers(command.getNameservers().stream().collect(toImmutableSet()))
.setStatusValues(statuses)
.setContacts(command.getContacts())
.addGracePeriod(
GracePeriod.forBillingEvent(GracePeriodStatus.ADD, repoId, createBillingEvent))
.build();
DomainHistory domainHistory =
buildDomainHistory(domain, registry, now, period, registry.getAddGracePeriodLength());
if (reservationTypes.contains(NAME_COLLISION)) {
entitiesToSave.add(
createNameCollisionOneTimePollMessage(targetId, domainHistory, clientId, now));
}
entitiesToSave.add(
newDomain,
ForeignKeyIndex.create(newDomain, newDomain.getDeletionTime()),
EppResourceIndex.create(Key.create(newDomain)));
domain,
domainHistory,
ForeignKeyIndex.create(domain, domain.getDeletionTime()),
EppResourceIndex.create(Key.create(domain)));
if (allocationToken.isPresent()
&& TokenType.SINGLE_USE.equals(allocationToken.get().getTokenType())) {
entitiesToSave.add(
allocationTokenFlowUtils.redeemToken(
allocationToken.get(), HistoryEntry.createVKey(Key.create(historyEntry))));
allocationTokenFlowUtils.redeemToken(allocationToken.get(), domainHistory.createVKey()));
}
enqueueTasks(newDomain, hasSignedMarks, hasClaimsNotice);
enqueueTasks(domain, hasSignedMarks, hasClaimsNotice);
EntityChanges entityChanges =
flowCustomLogic.beforeSave(
DomainCreateFlowCustomLogic.BeforeSaveParameters.newBuilder()
.setNewDomain(newDomain)
.setHistoryEntry(historyEntry)
.setNewDomain(domain)
.setHistoryEntry(domainHistory)
.setEntityChanges(
EntityChanges.newBuilder().setSaves(entitiesToSave.build()).build())
.setYears(years)
@@ -483,8 +485,8 @@ public class DomainCreateFlow implements TransactionalFlow {
: null);
}
private HistoryEntry buildHistoryEntry(
String repoId, Registry registry, DateTime now, Period period, Duration addGracePeriod) {
private DomainHistory buildDomainHistory(
DomainBase domain, Registry registry, DateTime now, Period period, Duration addGracePeriod) {
// We ignore prober transactions
if (registry.getTldType() == TldType.REAL) {
historyBuilder
@@ -500,7 +502,7 @@ public class DomainCreateFlow implements TransactionalFlow {
.setType(HistoryEntry.Type.DOMAIN_CREATE)
.setPeriod(period)
.setModificationTime(now)
.setParent(Key.create(DomainBase.class, repoId))
.setDomain(domain)
.build();
}
@@ -511,7 +513,7 @@ public class DomainCreateFlow implements TransactionalFlow {
boolean isReserved,
int years,
FeesAndCredits feesAndCredits,
HistoryEntry historyEntry,
Key<DomainHistory> domainHistoryKey,
Optional<AllocationToken> allocationToken,
DateTime now) {
ImmutableSet.Builder<Flag> flagsBuilder = new ImmutableSet.Builder<>();
@@ -540,12 +542,12 @@ public class DomainCreateFlow implements TransactionalFlow {
? registry.getAnchorTenantAddGracePeriodLength()
: registry.getAddGracePeriodLength()))
.setFlags(flagsBuilder.build())
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build();
}
private Recurring createAutorenewBillingEvent(
HistoryEntry historyEntry, DateTime registrationExpirationTime) {
Key<DomainHistory> domainHistoryKey, DateTime registrationExpirationTime) {
return new BillingEvent.Recurring.Builder()
.setReason(Reason.RENEW)
.setFlags(ImmutableSet.of(Flag.AUTO_RENEW))
@@ -553,18 +555,18 @@ public class DomainCreateFlow implements TransactionalFlow {
.setClientId(clientId)
.setEventTime(registrationExpirationTime)
.setRecurrenceEndTime(END_OF_TIME)
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build();
}
private Autorenew createAutorenewPollMessage(
HistoryEntry historyEntry, DateTime registrationExpirationTime) {
Key<DomainHistory> domainHistoryKey, DateTime registrationExpirationTime) {
return new PollMessage.Autorenew.Builder()
.setTargetId(targetId)
.setClientId(clientId)
.setEventTime(registrationExpirationTime)
.setMsg("Domain was auto-renewed.")
.setParent(historyEntry)
.setParentKey(domainHistoryKey)
.build();
}

View File

@@ -16,6 +16,7 @@ package google.registry.flows.domain;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Strings.isNullOrEmpty;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.persistEntityChanges;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
@@ -63,6 +64,7 @@ import google.registry.flows.custom.EntityChanges;
import google.registry.model.ImmutableObject;
import google.registry.model.billing.BillingEvent;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.GracePeriod;
import google.registry.model.domain.fee.BaseFee.FeeType;
import google.registry.model.domain.fee.Credit;
@@ -125,7 +127,7 @@ public final class DomainDeleteFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject DnsQueue dnsQueue;
@Inject Trid trid;
@Inject AsyncTaskEnqueuer asyncTaskEnqueuer;
@@ -177,8 +179,8 @@ public final class DomainDeleteFlow implements TransactionalFlow {
? Duration.ZERO
// By default, this should be 30 days of grace, and 5 days of pending delete.
: redemptionGracePeriodLength.plus(pendingDeleteLength);
HistoryEntry historyEntry =
buildHistoryEntry(existingDomain, registry, now, durationUntilDelete, inAddGracePeriod);
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder.setId(domainHistoryKey.getId());
DateTime deletionTime = now.plus(durationUntilDelete);
if (durationUntilDelete.equals(Duration.ZERO)) {
builder.setDeletionTime(now).setStatusValues(null);
@@ -208,20 +210,28 @@ public final class DomainDeleteFlow implements TransactionalFlow {
// Enqueue the deletion poll message if the delete is asynchronous or if requested by a
// superuser (i.e. the registrar didn't request this delete and thus should be notified even if
// it is synchronous).
if (!durationUntilDelete.equals(Duration.ZERO) || isSuperuser) {
if (durationUntilDelete.isLongerThan(Duration.ZERO) || isSuperuser) {
PollMessage.OneTime deletePollMessage =
createDeletePollMessage(existingDomain, historyEntry, deletionTime);
createDeletePollMessage(existingDomain, domainHistoryKey, deletionTime);
entitiesToSave.add(deletePollMessage);
builder.setDeletePollMessage(deletePollMessage.createVKey());
}
// Send a second poll message immediately if the domain is being deleted asynchronously by a
// registrar other than the sponsoring registrar (which will necessarily be a superuser).
if (durationUntilDelete.isLongerThan(Duration.ZERO)
&& !clientId.equals(existingDomain.getPersistedCurrentSponsorClientId())) {
entitiesToSave.add(
createImmediateDeletePollMessage(existingDomain, domainHistoryKey, now, deletionTime));
}
// Cancel any grace periods that were still active, and set the expiration time accordingly.
DateTime newExpirationTime = existingDomain.getRegistrationExpirationTime();
for (GracePeriod gracePeriod : existingDomain.getGracePeriods()) {
// No cancellation is written if the grace period was not for a billable event.
if (gracePeriod.hasBillingEvent()) {
entitiesToSave.add(
BillingEvent.Cancellation.forGracePeriod(gracePeriod, historyEntry, targetId));
BillingEvent.Cancellation.forGracePeriod(gracePeriod, now, domainHistoryKey, targetId));
if (gracePeriod.getOneTimeBillingEvent() != null) {
// Take the amount of amount of registration time being refunded off the expiration time.
// This can be either add grace periods or renew grace periods.
@@ -237,24 +247,29 @@ public final class DomainDeleteFlow implements TransactionalFlow {
builder.setRegistrationExpirationTime(newExpirationTime);
DomainBase newDomain = builder.build();
DomainHistory domainHistory =
buildDomainHistory(newDomain, registry, now, durationUntilDelete, inAddGracePeriod);
updateForeignKeyIndexDeletionTime(newDomain);
handlePendingTransferOnDelete(existingDomain, newDomain, now, historyEntry);
// Close the autorenew billing event and poll message. This may delete the poll message.
updateAutorenewRecurrenceEndTime(existingDomain, now);
handlePendingTransferOnDelete(existingDomain, newDomain, now, domainHistory);
// Close the autorenew billing event and poll message. This may delete the poll message. Store
// the updated recurring billing event, we'll need it later and can't reload it.
BillingEvent.Recurring recurringBillingEvent =
updateAutorenewRecurrenceEndTime(existingDomain, now);
// If there's a pending transfer, the gaining client's autorenew billing
// event and poll message will already have been deleted in
// ResourceDeleteFlow since it's listed in serverApproveEntities.
dnsQueue.addDomainRefreshTask(existingDomain.getDomainName());
entitiesToSave.add(newDomain, historyEntry);
EntityChanges entityChanges = flowCustomLogic.beforeSave(
BeforeSaveParameters.newBuilder()
.setExistingDomain(existingDomain)
.setNewDomain(newDomain)
.setHistoryEntry(historyEntry)
.setEntityChanges(EntityChanges.newBuilder().setSaves(entitiesToSave.build()).build())
.build());
persistEntityChanges(entityChanges);
entitiesToSave.add(newDomain, domainHistory);
EntityChanges entityChanges =
flowCustomLogic.beforeSave(
BeforeSaveParameters.newBuilder()
.setExistingDomain(existingDomain)
.setNewDomain(newDomain)
.setHistoryEntry(domainHistory)
.setEntityChanges(
EntityChanges.newBuilder().setSaves(entitiesToSave.build()).build())
.build());
BeforeResponseReturnData responseData =
flowCustomLogic.beforeResponse(
BeforeResponseParameters.newBuilder()
@@ -262,8 +277,10 @@ public final class DomainDeleteFlow implements TransactionalFlow {
newDomain.getDeletionTime().isAfter(now)
? SUCCESS_WITH_ACTION_PENDING
: SUCCESS)
.setResponseExtensions(getResponseExtensions(existingDomain, now))
.setResponseExtensions(
getResponseExtensions(recurringBillingEvent, existingDomain, now))
.build());
persistEntityChanges(entityChanges);
return responseBuilder
.setResultFromCode(responseData.resultCode())
.setExtensions(responseData.responseExtensions())
@@ -284,8 +301,8 @@ public final class DomainDeleteFlow implements TransactionalFlow {
}
}
private HistoryEntry buildHistoryEntry(
DomainBase existingResource,
private DomainHistory buildDomainHistory(
DomainBase domain,
Registry registry,
DateTime now,
Duration durationUntilDelete,
@@ -299,31 +316,30 @@ public final class DomainDeleteFlow implements TransactionalFlow {
registry.getRenewGracePeriodLength()));
ImmutableSet<DomainTransactionRecord> cancelledRecords =
createCancelingRecords(
existingResource,
domain,
now,
maxGracePeriod,
Sets.immutableEnumSet(Sets.union(ADD_FIELDS, RENEW_FIELDS)));
historyBuilder
.setDomainTransactionRecords(
union(
cancelledRecords,
DomainTransactionRecord.create(
existingResource.getTld(),
now.plus(durationUntilDelete),
inAddGracePeriod
? TransactionReportField.DELETED_DOMAINS_GRACE
: TransactionReportField.DELETED_DOMAINS_NOGRACE,
1)));
historyBuilder.setDomainTransactionRecords(
union(
cancelledRecords,
DomainTransactionRecord.create(
domain.getTld(),
now.plus(durationUntilDelete),
inAddGracePeriod
? TransactionReportField.DELETED_DOMAINS_GRACE
: TransactionReportField.DELETED_DOMAINS_NOGRACE,
1)));
}
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_DELETE)
.setModificationTime(now)
.setParent(Key.create(existingResource))
.setDomain(domain)
.build();
}
private PollMessage.OneTime createDeletePollMessage(
DomainBase existingDomain, HistoryEntry historyEntry, DateTime deletionTime) {
DomainBase existingDomain, Key<DomainHistory> domainHistoryKey, DateTime deletionTime) {
Optional<MetadataExtension> metadataExtension =
eppInput.getSingleExtension(MetadataExtension.class);
boolean hasMetadataMessage =
@@ -342,13 +358,29 @@ public final class DomainDeleteFlow implements TransactionalFlow {
ImmutableList.of(
DomainPendingActionNotificationResponse.create(
existingDomain.getDomainName(), true, trid, deletionTime)))
.setParent(historyEntry)
.setParentKey(domainHistoryKey)
.build();
}
private PollMessage.OneTime createImmediateDeletePollMessage(
DomainBase existingDomain,
Key<DomainHistory> domainHistoryKey,
DateTime now,
DateTime deletionTime) {
return new PollMessage.OneTime.Builder()
.setClientId(existingDomain.getPersistedCurrentSponsorClientId())
.setEventTime(now)
.setParentKey(domainHistoryKey)
.setMsg(
String.format(
"Domain %s was deleted by registry administrator with final deletion effective: %s",
existingDomain.getDomainName(), deletionTime))
.build();
}
@Nullable
private ImmutableList<FeeTransformResponseExtension> getResponseExtensions(
DomainBase existingDomain, DateTime now) {
BillingEvent.Recurring recurringBillingEvent, DomainBase existingDomain, DateTime now) {
FeeTransformResponseExtension.Builder feeResponseBuilder = getDeleteResponseBuilder();
if (feeResponseBuilder == null) {
return ImmutableList.of();
@@ -356,7 +388,7 @@ public final class DomainDeleteFlow implements TransactionalFlow {
ImmutableList.Builder<Credit> creditsBuilder = new ImmutableList.Builder<>();
for (GracePeriod gracePeriod : existingDomain.getGracePeriods()) {
if (gracePeriod.hasBillingEvent()) {
Money cost = getGracePeriodCost(gracePeriod, now);
Money cost = getGracePeriodCost(recurringBillingEvent, gracePeriod, now);
creditsBuilder.add(Credit.create(
cost.negated().getAmount(), FeeType.CREDIT, gracePeriod.getType().getXmlName()));
feeResponseBuilder.setCurrency(checkNotNull(cost.getCurrencyUnit()));
@@ -369,12 +401,12 @@ public final class DomainDeleteFlow implements TransactionalFlow {
return ImmutableList.of(feeResponseBuilder.setCredits(credits).build());
}
private Money getGracePeriodCost(GracePeriod gracePeriod, DateTime now) {
private Money getGracePeriodCost(
BillingEvent.Recurring recurringBillingEvent, GracePeriod gracePeriod, DateTime now) {
if (gracePeriod.getType() == GracePeriodStatus.AUTO_RENEW) {
// If we updated the autorenew billing event, reuse it.
DateTime autoRenewTime =
tm().loadByKey(checkNotNull(gracePeriod.getRecurringBillingEvent()))
.getRecurrenceTimeOfYear()
.getLastInstanceBeforeOrAt(now);
recurringBillingEvent.getRecurrenceTimeOfYear().getLastInstanceBeforeOrAt(now);
return getDomainRenewCost(targetId, autoRenewTime, 1);
}
return tm().loadByKey(checkNotNull(gracePeriod.getOneTimeBillingEvent())).getCost();

View File

@@ -25,11 +25,8 @@ import static com.google.common.collect.Iterables.any;
import static com.google.common.collect.Sets.difference;
import static com.google.common.collect.Sets.intersection;
import static com.google.common.collect.Sets.union;
import static google.registry.model.DatabaseMigrationUtils.getPrimaryDatabase;
import static google.registry.model.common.DatabaseTransitionSchedule.PrimaryDatabase.DATASTORE;
import static google.registry.model.common.DatabaseTransitionSchedule.TransitionId.REPLAYED_ENTITIES;
import static google.registry.model.domain.DomainBase.MAX_REGISTRATION_YEARS;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.model.registry.Registries.findTldForName;
import static google.registry.model.registry.Registries.getTlds;
import static google.registry.model.registry.Registry.TldState.GENERAL_AVAILABILITY;
@@ -129,7 +126,7 @@ import google.registry.model.registry.label.ReservedList;
import google.registry.model.reporting.DomainTransactionRecord;
import google.registry.model.reporting.DomainTransactionRecord.TransactionReportField;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.tmch.ClaimsListDualDatabaseDao;
import google.registry.model.tmch.ClaimsListDao;
import google.registry.persistence.VKey;
import google.registry.tldconfig.idn.IdnLabelValidator;
import google.registry.util.Idn;
@@ -520,8 +517,10 @@ public class DomainFlowUtils {
* <p>This may end up deleting the poll message (if closing the message interval) or recreating it
* (if opening the message interval). This may cause an autorenew billing event to have an end
* time earlier than its event time (i.e. if it's being ended before it was ever triggered).
*
* <p>Returns the new autorenew recurring billing event.
*/
public static void updateAutorenewRecurrenceEndTime(DomainBase domain, DateTime newEndTime) {
public static Recurring updateAutorenewRecurrenceEndTime(DomainBase domain, DateTime newEndTime) {
Optional<PollMessage.Autorenew> autorenewPollMessage =
tm().loadByKeyIfPresent(domain.getAutorenewPollMessage());
@@ -548,8 +547,13 @@ public class DomainFlowUtils {
tm().put(updatedAutorenewPollMessage);
}
Recurring recurring = tm().loadByKey(domain.getAutorenewBillingEvent());
tm().put(recurring.asBuilder().setRecurrenceEndTime(newEndTime).build());
Recurring recurring =
tm().loadByKey(domain.getAutorenewBillingEvent())
.asBuilder()
.setRecurrenceEndTime(newEndTime)
.build();
tm().put(recurring);
return recurring;
}
/**
@@ -993,8 +997,7 @@ public class DomainFlowUtils {
static void verifyClaimsNoticeIfAndOnlyIfNeeded(
InternetDomainName domainName, boolean hasSignedMarks, boolean hasClaimsNotice)
throws EppException {
boolean isInClaimsList =
ClaimsListDualDatabaseDao.get().getClaimKey(domainName.parts().get(0)).isPresent();
boolean isInClaimsList = ClaimsListDao.get().getClaimKey(domainName.parts().get(0)).isPresent();
if (hasClaimsNotice && !isInClaimsList) {
throw new UnexpectedClaimsNoticeException(domainName.toString());
}
@@ -1083,8 +1086,8 @@ public class DomainFlowUtils {
private static List<? extends HistoryEntry> findRecentHistoryEntries(
DomainBase domainBase, DateTime now, Duration maxSearchPeriod) {
if (getPrimaryDatabase(REPLAYED_ENTITIES).equals(DATASTORE)) {
return ofy()
if (tm().isOfy()) {
return auditedOfy()
.load()
.type(HistoryEntry.class)
.ancestor(domainBase)

View File

@@ -14,6 +14,7 @@
package google.registry.flows.domain;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.persistEntityChanges;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
@@ -33,6 +34,7 @@ import static google.registry.util.DateTimeUtils.leapSafeAddYears;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import google.registry.flows.EppException;
import google.registry.flows.EppException.ParameterValueRangeErrorException;
import google.registry.flows.ExtensionManager;
@@ -52,6 +54,7 @@ import google.registry.model.billing.BillingEvent.OneTime;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand.Renew;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.DomainRenewData;
import google.registry.model.domain.GracePeriod;
import google.registry.model.domain.Period;
@@ -123,7 +126,7 @@ public final class DomainRenewFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject DomainRenewFlowCustomLogic flowCustomLogic;
@Inject DomainPricingLogic pricingLogic;
@@ -156,22 +159,23 @@ public final class DomainRenewFlow implements TransactionalFlow {
.setNow(now)
.setYears(years)
.build());
Registry registry = Registry.get(existingDomain.getTld());
HistoryEntry historyEntry = buildHistoryEntry(
existingDomain, now, command.getPeriod(), registry.getRenewGracePeriodLength());
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder.setId(domainHistoryKey.getId());
String tld = existingDomain.getTld();
// Bill for this explicit renew itself.
BillingEvent.OneTime explicitRenewEvent =
createRenewBillingEvent(tld, feesAndCredits.getTotalCost(), years, historyEntry, now);
createRenewBillingEvent(tld, feesAndCredits.getTotalCost(), years, domainHistoryKey, now);
// Create a new autorenew billing event and poll message starting at the new expiration time.
BillingEvent.Recurring newAutorenewEvent = newAutorenewBillingEvent(existingDomain)
.setEventTime(newExpirationTime)
.setParent(historyEntry)
.build();
PollMessage.Autorenew newAutorenewPollMessage = newAutorenewPollMessage(existingDomain)
.setEventTime(newExpirationTime)
.setParent(historyEntry)
.build();
BillingEvent.Recurring newAutorenewEvent =
newAutorenewBillingEvent(existingDomain)
.setEventTime(newExpirationTime)
.setParent(domainHistoryKey)
.build();
PollMessage.Autorenew newAutorenewPollMessage =
newAutorenewPollMessage(existingDomain)
.setEventTime(newExpirationTime)
.setParentKey(domainHistoryKey)
.build();
// End the old autorenew billing event and poll message now. This may delete the poll message.
updateAutorenewRecurrenceEndTime(existingDomain, now);
DomainBase newDomain =
@@ -186,6 +190,10 @@ public final class DomainRenewFlow implements TransactionalFlow {
GracePeriod.forBillingEvent(
GracePeriodStatus.RENEW, existingDomain.getRepoId(), explicitRenewEvent))
.build();
Registry registry = Registry.get(existingDomain.getTld());
DomainHistory domainHistory =
buildDomainHistory(
newDomain, now, command.getPeriod(), registry.getRenewGracePeriodLength());
EntityChanges entityChanges =
flowCustomLogic.beforeSave(
BeforeSaveParameters.newBuilder()
@@ -193,19 +201,18 @@ public final class DomainRenewFlow implements TransactionalFlow {
.setNewDomain(newDomain)
.setNow(now)
.setYears(years)
.setHistoryEntry(historyEntry)
.setHistoryEntry(domainHistory)
.setEntityChanges(
EntityChanges.newBuilder()
.setSaves(
ImmutableSet.of(
newDomain,
historyEntry,
domainHistory,
explicitRenewEvent,
newAutorenewEvent,
newAutorenewPollMessage))
.build())
.build());
persistEntityChanges(entityChanges);
BeforeResponseReturnData responseData =
flowCustomLogic.beforeResponse(
BeforeResponseParameters.newBuilder()
@@ -213,23 +220,24 @@ public final class DomainRenewFlow implements TransactionalFlow {
.setResData(DomainRenewData.create(targetId, newExpirationTime))
.setResponseExtensions(createResponseExtensions(feesAndCredits, feeRenew))
.build());
persistEntityChanges(entityChanges);
return responseBuilder
.setResData(responseData.resData())
.setExtensions(responseData.responseExtensions())
.build();
}
private HistoryEntry buildHistoryEntry(
DomainBase existingDomain, DateTime now, Period period, Duration renewGracePeriod) {
private DomainHistory buildDomainHistory(
DomainBase newDomain, DateTime now, Period period, Duration renewGracePeriod) {
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_RENEW)
.setPeriod(period)
.setModificationTime(now)
.setParent(existingDomain)
.setDomain(newDomain)
.setDomainTransactionRecords(
ImmutableSet.of(
DomainTransactionRecord.create(
existingDomain.getTld(),
newDomain.getTld(),
now.plus(renewGracePeriod),
TransactionReportField.netRenewsFieldFromYears(period.getValue()),
1)))
@@ -255,7 +263,7 @@ public final class DomainRenewFlow implements TransactionalFlow {
}
private OneTime createRenewBillingEvent(
String tld, Money renewCost, int years, HistoryEntry historyEntry, DateTime now) {
String tld, Money renewCost, int years, Key<DomainHistory> domainHistoryKey, DateTime now) {
return new BillingEvent.OneTime.Builder()
.setReason(Reason.RENEW)
.setTargetId(targetId)
@@ -264,7 +272,7 @@ public final class DomainRenewFlow implements TransactionalFlow {
.setCost(renewCost)
.setEventTime(now)
.setBillingTime(now.plus(Registry.get(tld).getRenewGracePeriodLength()))
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build();
}

View File

@@ -14,6 +14,7 @@
package google.registry.flows.domain;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.ResourceFlowUtils.verifyOptionalAuthInfo;
@@ -49,6 +50,7 @@ import google.registry.model.billing.BillingEvent.OneTime;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand.Update;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.fee.BaseFee.FeeType;
import google.registry.model.domain.fee.Fee;
import google.registry.model.domain.fee.FeeTransformResponseExtension;
@@ -117,7 +119,7 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject DnsQueue dnsQueue;
@Inject EppResponse.Builder responseBuilder;
@Inject DomainPricingLogic pricingLogic;
@@ -142,7 +144,8 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow {
Optional<FeeUpdateCommandExtension> feeUpdate =
eppInput.getSingleExtension(FeeUpdateCommandExtension.class);
verifyRestoreAllowed(command, existingDomain, feeUpdate, feesAndCredits, now);
HistoryEntry historyEntry = buildHistoryEntry(existingDomain, now);
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder.setId(domainHistoryKey.getId());
ImmutableSet.Builder<ImmutableObject> entitiesToSave = new ImmutableSet.Builder<>();
DateTime newExpirationTime =
@@ -150,29 +153,31 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow {
// Restore the expiration time on the deleted domain, except if that's already passed, then add
// a year and bill for it immediately, with no grace period.
if (isExpired) {
entitiesToSave.add(createRenewBillingEvent(historyEntry, feesAndCredits.getRenewCost(), now));
entitiesToSave.add(
createRenewBillingEvent(domainHistoryKey, feesAndCredits.getRenewCost(), now));
}
// Always bill for the restore itself.
entitiesToSave.add(
createRestoreBillingEvent(historyEntry, feesAndCredits.getRestoreCost(), now));
createRestoreBillingEvent(domainHistoryKey, feesAndCredits.getRestoreCost(), now));
BillingEvent.Recurring autorenewEvent =
newAutorenewBillingEvent(existingDomain)
.setEventTime(newExpirationTime)
.setRecurrenceEndTime(END_OF_TIME)
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build();
PollMessage.Autorenew autorenewPollMessage =
newAutorenewPollMessage(existingDomain)
.setEventTime(newExpirationTime)
.setAutorenewEndTime(END_OF_TIME)
.setParent(historyEntry)
.setParentKey(domainHistoryKey)
.build();
DomainBase newDomain =
performRestore(
existingDomain, newExpirationTime, autorenewEvent, autorenewPollMessage, now, clientId);
updateForeignKeyIndexDeletionTime(newDomain);
entitiesToSave.add(newDomain, historyEntry, autorenewEvent, autorenewPollMessage);
DomainHistory domainHistory = buildDomainHistory(newDomain, now);
entitiesToSave.add(newDomain, domainHistory, autorenewEvent, autorenewPollMessage);
tm().putAll(entitiesToSave.build());
tm().delete(existingDomain.getDeletePollMessage());
dnsQueue.addDomainRefreshTask(existingDomain.getDomainName());
@@ -181,15 +186,15 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow {
.build();
}
private HistoryEntry buildHistoryEntry(DomainBase existingDomain, DateTime now) {
private DomainHistory buildDomainHistory(DomainBase newDomain, DateTime now) {
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_RESTORE)
.setModificationTime(now)
.setParent(Key.create(existingDomain))
.setDomain(newDomain)
.setDomainTransactionRecords(
ImmutableSet.of(
DomainTransactionRecord.create(
existingDomain.getTld(), now, TransactionReportField.RESTORED_DOMAINS, 1)))
newDomain.getTld(), now, TransactionReportField.RESTORED_DOMAINS, 1)))
.build();
}
@@ -242,20 +247,19 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow {
}
private OneTime createRenewBillingEvent(
HistoryEntry historyEntry, Money renewCost, DateTime now) {
return prepareBillingEvent(historyEntry, renewCost, now)
.setReason(Reason.RENEW)
.build();
Key<DomainHistory> domainHistoryKey, Money renewCost, DateTime now) {
return prepareBillingEvent(domainHistoryKey, renewCost, now).setReason(Reason.RENEW).build();
}
private BillingEvent.OneTime createRestoreBillingEvent(
HistoryEntry historyEntry, Money restoreCost, DateTime now) {
return prepareBillingEvent(historyEntry, restoreCost, now)
Key<DomainHistory> domainHistoryKey, Money restoreCost, DateTime now) {
return prepareBillingEvent(domainHistoryKey, restoreCost, now)
.setReason(Reason.RESTORE)
.build();
}
private OneTime.Builder prepareBillingEvent(HistoryEntry historyEntry, Money cost, DateTime now) {
private OneTime.Builder prepareBillingEvent(
Key<DomainHistory> domainHistoryKey, Money cost, DateTime now) {
return new BillingEvent.OneTime.Builder()
.setTargetId(targetId)
.setClientId(clientId)
@@ -263,7 +267,7 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow {
.setBillingTime(now)
.setPeriodYears(1)
.setCost(cost)
.setParent(historyEntry);
.setParent(domainHistoryKey);
}
private static ImmutableList<FeeTransformResponseExtension> createResponseExtensions(

View File

@@ -15,6 +15,7 @@
package google.registry.flows.domain;
import static com.google.common.collect.Iterables.getOnlyElement;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.computeExDateForApprovalTime;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
@@ -48,6 +49,7 @@ import google.registry.model.billing.BillingEvent;
import google.registry.model.billing.BillingEvent.Flag;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.GracePeriod;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.domain.rgp.GracePeriodStatus;
@@ -90,7 +92,7 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject DomainTransferApproveFlow() {}
@@ -114,10 +116,10 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
}
DomainTransferData transferData = existingDomain.getTransferData();
String gainingClientId = transferData.getGainingClientId();
Registry registry = Registry.get(existingDomain.getTld());
HistoryEntry historyEntry = buildHistoryEntry(existingDomain, registry, now, gainingClientId);
// Create a transfer billing event for 1 year, unless the superuser extension was used to set
// the transfer period to zero. There is not a transfer cost if the transfer period is zero.
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder.setId(domainHistoryKey.getId());
Optional<BillingEvent.OneTime> billingEvent =
(transferData.getTransferPeriod().getValue() == 0)
? Optional.empty()
@@ -130,10 +132,9 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
.setCost(getDomainRenewCost(targetId, transferData.getTransferRequestTime(), 1))
.setEventTime(now)
.setBillingTime(now.plus(Registry.get(tld).getTransferGracePeriodLength()))
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build());
ImmutableList.Builder<ImmutableObject> entitiesToSave = new ImmutableList.Builder<>();
entitiesToSave.add(historyEntry);
// If we are within an autorenew grace period, cancel the autorenew billing event and don't
// increase the registration time, since the transfer subsumes the autorenew's extra year.
GracePeriod autorenewGrace =
@@ -146,7 +147,8 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
// still needs to be charged for the auto-renew.
if (billingEvent.isPresent()) {
entitiesToSave.add(
BillingEvent.Cancellation.forGracePeriod(autorenewGrace, historyEntry, targetId));
BillingEvent.Cancellation.forGracePeriod(
autorenewGrace, now, domainHistoryKey, targetId));
}
}
// Close the old autorenew event and poll message at the transfer time (aka now). This may end
@@ -155,24 +157,26 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
DateTime newExpirationTime =
computeExDateForApprovalTime(existingDomain, now, transferData.getTransferPeriod());
// Create a new autorenew event starting at the expiration time.
BillingEvent.Recurring autorenewEvent = new BillingEvent.Recurring.Builder()
.setReason(Reason.RENEW)
.setFlags(ImmutableSet.of(Flag.AUTO_RENEW))
.setTargetId(targetId)
.setClientId(gainingClientId)
.setEventTime(newExpirationTime)
.setRecurrenceEndTime(END_OF_TIME)
.setParent(historyEntry)
.build();
BillingEvent.Recurring autorenewEvent =
new BillingEvent.Recurring.Builder()
.setReason(Reason.RENEW)
.setFlags(ImmutableSet.of(Flag.AUTO_RENEW))
.setTargetId(targetId)
.setClientId(gainingClientId)
.setEventTime(newExpirationTime)
.setRecurrenceEndTime(END_OF_TIME)
.setParent(domainHistoryKey)
.build();
// Create a new autorenew poll message.
PollMessage.Autorenew gainingClientAutorenewPollMessage = new PollMessage.Autorenew.Builder()
.setTargetId(targetId)
.setClientId(gainingClientId)
.setEventTime(newExpirationTime)
.setAutorenewEndTime(END_OF_TIME)
.setMsg("Domain was auto-renewed.")
.setParent(historyEntry)
.build();
PollMessage.Autorenew gainingClientAutorenewPollMessage =
new PollMessage.Autorenew.Builder()
.setTargetId(targetId)
.setClientId(gainingClientId)
.setEventTime(newExpirationTime)
.setAutorenewEndTime(END_OF_TIME)
.setMsg("Domain was auto-renewed.")
.setParentKey(domainHistoryKey)
.build();
// Construct the post-transfer domain.
DomainBase partiallyApprovedDomain =
approvePendingTransfer(existingDomain, TransferStatus.CLIENT_APPROVED, now);
@@ -204,13 +208,19 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
.setLastEppUpdateTime(now)
.setLastEppUpdateClientId(clientId)
.build();
Registry registry = Registry.get(existingDomain.getTld());
DomainHistory domainHistory = buildDomainHistory(newDomain, registry, now, gainingClientId);
// Create a poll message for the gaining client.
PollMessage gainingClientPollMessage =
createGainingTransferPollMessage(
targetId, newDomain.getTransferData(), newExpirationTime, historyEntry);
targetId, newDomain.getTransferData(), newExpirationTime, now, domainHistoryKey);
billingEvent.ifPresent(entitiesToSave::add);
entitiesToSave.add(
autorenewEvent, gainingClientPollMessage, gainingClientAutorenewPollMessage, newDomain);
autorenewEvent,
gainingClientPollMessage,
gainingClientAutorenewPollMessage,
newDomain,
domainHistory);
tm().putAll(entitiesToSave.build());
// Delete the billing event and poll messages that were written in case the transfer would have
// been implicitly server approved.
@@ -221,11 +231,11 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
.build();
}
private HistoryEntry buildHistoryEntry(
DomainBase existingDomain, Registry registry, DateTime now, String gainingClientId) {
private DomainHistory buildDomainHistory(
DomainBase newDomain, Registry registry, DateTime now, String gainingClientId) {
ImmutableSet<DomainTransactionRecord> cancelingRecords =
createCancelingRecords(
existingDomain,
newDomain,
now,
registry.getAutomaticTransferLength().plus(registry.getTransferGracePeriodLength()),
ImmutableSet.of(TRANSFER_SUCCESSFUL));
@@ -233,15 +243,15 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
.setType(HistoryEntry.Type.DOMAIN_TRANSFER_APPROVE)
.setModificationTime(now)
.setOtherClientId(gainingClientId)
.setParent(Key.create(existingDomain))
.setDomain(newDomain)
.setDomainTransactionRecords(
union(
cancelingRecords,
DomainTransactionRecord.create(
existingDomain.getTld(),
now.plus(registry.getTransferGracePeriodLength()),
TRANSFER_SUCCESSFUL,
1)))
newDomain.getTld(),
now.plus(registry.getTransferGracePeriodLength()),
TRANSFER_SUCCESSFUL,
1)))
.build();
}
}

View File

@@ -14,6 +14,7 @@
package google.registry.flows.domain;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.ResourceFlowUtils.verifyHasPendingTransfer;
@@ -39,6 +40,7 @@ import google.registry.flows.FlowModule.TargetId;
import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
import google.registry.model.eppoutput.EppResponse;
@@ -77,7 +79,7 @@ public final class DomainTransferCancelFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject DomainTransferCancelFlow() {}
@@ -95,14 +97,20 @@ public final class DomainTransferCancelFlow implements TransactionalFlow {
checkAllowedAccessToTld(clientId, existingDomain.getTld());
}
Registry registry = Registry.get(existingDomain.getTld());
HistoryEntry historyEntry = buildHistoryEntry(existingDomain, registry, now);
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder
.setId(domainHistoryKey.getId())
.setOtherClientId(existingDomain.getTransferData().getLosingClientId());
DomainBase newDomain =
denyPendingTransfer(existingDomain, TransferStatus.CLIENT_CANCELLED, now, clientId);
DomainHistory domainHistory = buildDomainHistory(newDomain, registry, now);
tm().putAll(
newDomain,
historyEntry,
domainHistory,
createLosingTransferPollMessage(
targetId, newDomain.getTransferData(), null, historyEntry));
targetId, newDomain.getTransferData(), null, domainHistoryKey));
// Reopen the autorenew event and poll message that we closed for the implicit transfer. This
// may recreate the autorenew poll message if it was deleted when the transfer request was made.
updateAutorenewRecurrenceEndTime(existingDomain, END_OF_TIME);
@@ -114,19 +122,17 @@ public final class DomainTransferCancelFlow implements TransactionalFlow {
.build();
}
private HistoryEntry buildHistoryEntry(
DomainBase existingDomain, Registry registry, DateTime now) {
private DomainHistory buildDomainHistory(DomainBase newDomain, Registry registry, DateTime now) {
ImmutableSet<DomainTransactionRecord> cancelingRecords =
createCancelingRecords(
existingDomain,
newDomain,
now,
registry.getAutomaticTransferLength().plus(registry.getTransferGracePeriodLength()),
ImmutableSet.of(TRANSFER_SUCCESSFUL));
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_TRANSFER_CANCEL)
.setOtherClientId(existingDomain.getTransferData().getLosingClientId())
.setModificationTime(now)
.setParent(Key.create(existingDomain))
.setDomain(newDomain)
.setDomainTransactionRecords(cancelingRecords)
.build();
}

View File

@@ -14,6 +14,7 @@
package google.registry.flows.domain;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.ResourceFlowUtils.verifyHasPendingTransfer;
@@ -41,6 +42,7 @@ import google.registry.flows.FlowModule.TargetId;
import google.registry.flows.TransactionalFlow;
import google.registry.flows.annotations.ReportingSpec;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.AuthInfo;
import google.registry.model.eppoutput.EppResponse;
@@ -79,7 +81,7 @@ public final class DomainTransferRejectFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject EppResponse.Builder responseBuilder;
@Inject DomainTransferRejectFlow() {}
@@ -91,7 +93,11 @@ public final class DomainTransferRejectFlow implements TransactionalFlow {
DateTime now = tm().getTransactionTime();
DomainBase existingDomain = loadAndVerifyExistence(DomainBase.class, targetId, now);
Registry registry = Registry.get(existingDomain.getTld());
HistoryEntry historyEntry = buildHistoryEntry(existingDomain, registry, now);
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder
.setId(domainHistoryKey.getId())
.setOtherClientId(existingDomain.getTransferData().getGainingClientId());
verifyOptionalAuthInfo(authInfo, existingDomain);
verifyHasPendingTransfer(existingDomain);
verifyResourceOwnership(clientId, existingDomain);
@@ -100,11 +106,12 @@ public final class DomainTransferRejectFlow implements TransactionalFlow {
}
DomainBase newDomain =
denyPendingTransfer(existingDomain, TransferStatus.CLIENT_REJECTED, now, clientId);
DomainHistory domainHistory = buildDomainHistory(newDomain, registry, now);
tm().putAll(
newDomain,
historyEntry,
domainHistory,
createGainingTransferPollMessage(
targetId, newDomain.getTransferData(), null, historyEntry));
targetId, newDomain.getTransferData(), null, now, domainHistoryKey));
// Reopen the autorenew event and poll message that we closed for the implicit transfer. This
// may end up recreating the poll message if it was deleted upon the transfer request.
updateAutorenewRecurrenceEndTime(existingDomain, END_OF_TIME);
@@ -116,24 +123,21 @@ public final class DomainTransferRejectFlow implements TransactionalFlow {
.build();
}
private HistoryEntry buildHistoryEntry(
DomainBase existingDomain, Registry registry, DateTime now) {
private DomainHistory buildDomainHistory(DomainBase newDomain, Registry registry, DateTime now) {
ImmutableSet<DomainTransactionRecord> cancelingRecords =
createCancelingRecords(
existingDomain,
newDomain,
now,
registry.getAutomaticTransferLength().plus(registry.getTransferGracePeriodLength()),
ImmutableSet.of(TRANSFER_SUCCESSFUL));
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_TRANSFER_REJECT)
.setModificationTime(now)
.setOtherClientId(existingDomain.getTransferData().getGainingClientId())
.setParent(Key.create(existingDomain))
.setDomainTransactionRecords(
union(
cancelingRecords,
DomainTransactionRecord.create(
existingDomain.getTld(), now, TRANSFER_NACKED, 1)))
DomainTransactionRecord.create(newDomain.getTld(), now, TRANSFER_NACKED, 1)))
.setDomain(newDomain)
.build();
}
}

View File

@@ -14,6 +14,7 @@
package google.registry.flows.domain;
import static google.registry.flows.FlowUtils.createHistoryKey;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.computeExDateForApprovalTime;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
@@ -51,6 +52,7 @@ import google.registry.flows.exceptions.TransferPeriodMustBeOneYearException;
import google.registry.flows.exceptions.TransferPeriodZeroAndFeeTransferExtensionException;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand.Transfer;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.Period;
import google.registry.model.domain.fee.FeeTransferCommandExtension;
import google.registry.model.domain.fee.FeeTransformResponseExtension;
@@ -126,7 +128,7 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
@Inject @ClientId String gainingClientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject Trid trid;
@Inject AsyncTaskEnqueuer asyncTaskEnqueuer;
@Inject EppResponse.Builder responseBuilder;
@@ -169,7 +171,10 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
if (feesAndCredits.isPresent()) {
validateFeeChallenge(targetId, now, feeTransfer, feesAndCredits.get());
}
HistoryEntry historyEntry = buildHistoryEntry(existingDomain, registry, now, period);
Key<DomainHistory> domainHistoryKey = createHistoryKey(existingDomain, DomainHistory.class);
historyBuilder
.setId(domainHistoryKey.getId())
.setOtherClientId(existingDomain.getCurrentSponsorClientId());
DateTime automaticTransferTime =
superuserExtension.isPresent()
? now.plusDays(superuserExtension.get().getAutomaticTransferLength())
@@ -190,7 +195,7 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
createTransferServerApproveEntities(
automaticTransferTime,
serverApproveNewExpirationTime,
historyEntry,
domainHistoryKey,
existingDomain,
trid,
gainingClientId,
@@ -209,9 +214,12 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
serverApproveEntities,
period);
// Create a poll message to notify the losing registrar that a transfer was requested.
PollMessage requestPollMessage = createLosingTransferPollMessage(
targetId, pendingTransferData, serverApproveNewExpirationTime, historyEntry)
.asBuilder().setEventTime(now).build();
PollMessage requestPollMessage =
createLosingTransferPollMessage(
targetId, pendingTransferData, serverApproveNewExpirationTime, domainHistoryKey)
.asBuilder()
.setEventTime(now)
.build();
// End the old autorenew event and poll message at the implicit transfer time. This may delete
// the poll message if it has no events left. Note that if the automatic transfer succeeds, then
// cloneProjectedAtTime() will replace these old autorenew entities with the server approve ones
@@ -225,10 +233,12 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
.setLastEppUpdateTime(now)
.setLastEppUpdateClientId(gainingClientId)
.build();
DomainHistory domainHistory = buildDomainHistory(newDomain, registry, now, period);
asyncTaskEnqueuer.enqueueAsyncResave(newDomain, now, automaticTransferTime);
tm().putAll(
new ImmutableSet.Builder<>()
.add(newDomain, historyEntry, requestPollMessage)
.add(newDomain, domainHistory, requestPollMessage)
.addAll(serverApproveEntities)
.build());
return responseBuilder
@@ -302,14 +312,13 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
}
}
private HistoryEntry buildHistoryEntry(
DomainBase existingDomain, Registry registry, DateTime now, Period period) {
private DomainHistory buildDomainHistory(
DomainBase newDomain, Registry registry, DateTime now, Period period) {
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_TRANSFER_REQUEST)
.setOtherClientId(existingDomain.getCurrentSponsorClientId())
.setPeriod(period)
.setModificationTime(now)
.setParent(Key.create(existingDomain))
.setDomain(newDomain)
.setDomainTransactionRecords(
ImmutableSet.of(
DomainTransactionRecord.create(

View File

@@ -20,10 +20,12 @@ import static google.registry.util.DateTimeUtils.END_OF_TIME;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import google.registry.model.billing.BillingEvent;
import google.registry.model.billing.BillingEvent.Flag;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.GracePeriod;
import google.registry.model.domain.Period;
import google.registry.model.domain.rgp.GracePeriodStatus;
@@ -31,7 +33,6 @@ import google.registry.model.eppcommon.Trid;
import google.registry.model.poll.PendingActionNotificationResponse.DomainPendingActionNotificationResponse;
import google.registry.model.poll.PollMessage;
import google.registry.model.registry.Registry;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.transfer.DomainTransferData;
import google.registry.model.transfer.TransferData;
import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
@@ -90,20 +91,21 @@ public final class DomainTransferUtils {
* Returns a set of entities created speculatively in anticipation of a server approval.
*
* <p>This set consists of:
*
* <ul>
* <li>The one-time billing event charging the gaining registrar for the transfer
* <li>A cancellation of an autorenew charge for the losing registrar, if the autorenew grace
* period will apply at transfer time
* <li>A new post-transfer autorenew billing event for the domain (and gaining registrar)
* <li>A new post-transfer autorenew poll message for the domain (and gaining registrar)
* <li>A poll message for the gaining registrar
* <li>A poll message for the losing registrar
* <li>The one-time billing event charging the gaining registrar for the transfer
* <li>A cancellation of an autorenew charge for the losing registrar, if the autorenew grace
* period will apply at transfer time
* <li>A new post-transfer autorenew billing event for the domain (and gaining registrar)
* <li>A new post-transfer autorenew poll message for the domain (and gaining registrar)
* <li>A poll message for the gaining registrar
* <li>A poll message for the losing registrar
* </ul>
*/
public static ImmutableSet<TransferServerApproveEntity> createTransferServerApproveEntities(
DateTime automaticTransferTime,
DateTime serverApproveNewExpirationTime,
HistoryEntry historyEntry,
Key<DomainHistory> domainHistoryKey,
DomainBase existingDomain,
Trid trid,
String gainingClientId,
@@ -123,32 +125,39 @@ public final class DomainTransferUtils {
.build();
Registry registry = Registry.get(existingDomain.getTld());
ImmutableSet.Builder<TransferServerApproveEntity> builder = new ImmutableSet.Builder<>();
if (transferCost.isPresent()) {
builder.add(
createTransferBillingEvent(
automaticTransferTime,
historyEntry,
targetId,
gainingClientId,
registry,
transferCost.get()));
}
transferCost.ifPresent(
cost ->
builder.add(
createTransferBillingEvent(
automaticTransferTime,
domainHistoryKey,
targetId,
gainingClientId,
registry,
cost)));
createOptionalAutorenewCancellation(
automaticTransferTime, historyEntry, targetId, existingDomain, transferCost)
automaticTransferTime, now, domainHistoryKey, targetId, existingDomain, transferCost)
.ifPresent(builder::add);
return builder
.add(
createGainingClientAutorenewEvent(
serverApproveNewExpirationTime, historyEntry, targetId, gainingClientId))
serverApproveNewExpirationTime, domainHistoryKey, targetId, gainingClientId))
.add(
createGainingClientAutorenewPollMessage(
serverApproveNewExpirationTime, historyEntry, targetId, gainingClientId))
serverApproveNewExpirationTime, domainHistoryKey, targetId, gainingClientId))
.add(
createGainingTransferPollMessage(
targetId, serverApproveTransferData, serverApproveNewExpirationTime, historyEntry))
targetId,
serverApproveTransferData,
serverApproveNewExpirationTime,
now,
domainHistoryKey))
.add(
createLosingTransferPollMessage(
targetId, serverApproveTransferData, serverApproveNewExpirationTime, historyEntry))
targetId,
serverApproveTransferData,
serverApproveNewExpirationTime,
domainHistoryKey))
.build();
}
@@ -157,19 +166,21 @@ public final class DomainTransferUtils {
String targetId,
TransferData transferData,
@Nullable DateTime extendedRegistrationExpirationTime,
HistoryEntry historyEntry) {
DateTime now,
Key<DomainHistory> domainHistoryKey) {
return new PollMessage.OneTime.Builder()
.setClientId(transferData.getGainingClientId())
.setEventTime(transferData.getPendingTransferExpirationTime())
.setMsg(transferData.getTransferStatus().getMessage())
.setResponseData(ImmutableList.of(
createTransferResponse(targetId, transferData, extendedRegistrationExpirationTime),
DomainPendingActionNotificationResponse.create(
targetId,
transferData.getTransferStatus().isApproved(),
transferData.getTransferRequestTrid(),
historyEntry.getModificationTime())))
.setParent(historyEntry)
.setResponseData(
ImmutableList.of(
createTransferResponse(targetId, transferData, extendedRegistrationExpirationTime),
DomainPendingActionNotificationResponse.create(
targetId,
transferData.getTransferStatus().isApproved(),
transferData.getTransferRequestTrid(),
now)))
.setParentKey(domainHistoryKey)
.build();
}
@@ -178,14 +189,15 @@ public final class DomainTransferUtils {
String targetId,
TransferData transferData,
@Nullable DateTime extendedRegistrationExpirationTime,
HistoryEntry historyEntry) {
Key<DomainHistory> domainHistoryKey) {
return new PollMessage.OneTime.Builder()
.setClientId(transferData.getLosingClientId())
.setEventTime(transferData.getPendingTransferExpirationTime())
.setMsg(transferData.getTransferStatus().getMessage())
.setResponseData(ImmutableList.of(
createTransferResponse(targetId, transferData, extendedRegistrationExpirationTime)))
.setParent(historyEntry)
.setResponseData(
ImmutableList.of(
createTransferResponse(targetId, transferData, extendedRegistrationExpirationTime)))
.setParentKey(domainHistoryKey)
.build();
}
@@ -207,7 +219,7 @@ public final class DomainTransferUtils {
private static PollMessage.Autorenew createGainingClientAutorenewPollMessage(
DateTime serverApproveNewExpirationTime,
HistoryEntry historyEntry,
Key<DomainHistory> domainHistoryKey,
String targetId,
String gainingClientId) {
return new PollMessage.Autorenew.Builder()
@@ -216,13 +228,13 @@ public final class DomainTransferUtils {
.setEventTime(serverApproveNewExpirationTime)
.setAutorenewEndTime(END_OF_TIME)
.setMsg("Domain was auto-renewed.")
.setParent(historyEntry)
.setParentKey(domainHistoryKey)
.build();
}
private static BillingEvent.Recurring createGainingClientAutorenewEvent(
DateTime serverApproveNewExpirationTime,
HistoryEntry historyEntry,
Key<DomainHistory> domainHistoryKey,
String targetId,
String gainingClientId) {
return new BillingEvent.Recurring.Builder()
@@ -232,7 +244,7 @@ public final class DomainTransferUtils {
.setClientId(gainingClientId)
.setEventTime(serverApproveNewExpirationTime)
.setRecurrenceEndTime(END_OF_TIME)
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build();
}
@@ -254,7 +266,8 @@ public final class DomainTransferUtils {
*/
private static Optional<BillingEvent.Cancellation> createOptionalAutorenewCancellation(
DateTime automaticTransferTime,
HistoryEntry historyEntry,
DateTime now,
Key<DomainHistory> domainHistoryKey,
String targetId,
DomainBase existingDomain,
Optional<Money> transferCost) {
@@ -265,7 +278,8 @@ public final class DomainTransferUtils {
domainAtTransferTime.getGracePeriodsOfType(GracePeriodStatus.AUTO_RENEW), null);
if (autorenewGracePeriod != null && transferCost.isPresent()) {
return Optional.of(
BillingEvent.Cancellation.forGracePeriod(autorenewGracePeriod, historyEntry, targetId)
BillingEvent.Cancellation.forGracePeriod(
autorenewGracePeriod, now, domainHistoryKey, targetId)
.asBuilder()
.setEventTime(automaticTransferTime)
.build());
@@ -275,7 +289,7 @@ public final class DomainTransferUtils {
private static BillingEvent.OneTime createTransferBillingEvent(
DateTime automaticTransferTime,
HistoryEntry historyEntry,
Key<DomainHistory> domainHistoryKey,
String targetId,
String gainingClientId,
Registry registry,
@@ -288,7 +302,7 @@ public final class DomainTransferUtils {
.setPeriodYears(1)
.setEventTime(automaticTransferTime)
.setBillingTime(automaticTransferTime.plus(registry.getTransferGracePeriodLength()))
.setParent(historyEntry)
.setParent(domainHistoryKey)
.build();
}

View File

@@ -43,7 +43,6 @@ import static google.registry.persistence.transaction.TransactionManagerFactory.
import com.google.common.collect.ImmutableSet;
import com.google.common.net.InternetDomainName;
import com.googlecode.objectify.Key;
import google.registry.dns.DnsQueue;
import google.registry.flows.EppException;
import google.registry.flows.ExtensionManager;
@@ -65,6 +64,7 @@ import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand.Update;
import google.registry.model.domain.DomainCommand.Update.AddRemove;
import google.registry.model.domain.DomainCommand.Update.Change;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.fee.FeeUpdateCommandExtension;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.domain.secdns.DelegationSignerData;
@@ -142,7 +142,7 @@ public final class DomainUpdateFlow implements TransactionalFlow {
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject DomainHistory.Builder historyBuilder;
@Inject DnsQueue dnsQueue;
@Inject EppResponse.Builder responseBuilder;
@Inject DomainUpdateFlowCustomLogic flowCustomLogic;
@@ -165,19 +165,19 @@ public final class DomainUpdateFlow implements TransactionalFlow {
verifyUpdateAllowed(command, existingDomain, now);
flowCustomLogic.afterValidation(
AfterValidationParameters.newBuilder().setExistingDomain(existingDomain).build());
HistoryEntry historyEntry = buildHistoryEntry(existingDomain, now);
DomainBase newDomain = performUpdate(command, existingDomain, now);
DomainHistory domainHistory = buildDomainHistory(newDomain, now);
validateNewState(newDomain);
dnsQueue.addDomainRefreshTask(targetId);
ImmutableSet.Builder<ImmutableObject> entitiesToSave = new ImmutableSet.Builder<>();
entitiesToSave.add(newDomain, historyEntry);
entitiesToSave.add(newDomain, domainHistory);
Optional<BillingEvent.OneTime> statusUpdateBillingEvent =
createBillingEventForStatusUpdates(existingDomain, newDomain, historyEntry, now);
createBillingEventForStatusUpdates(existingDomain, newDomain, domainHistory, now);
statusUpdateBillingEvent.ifPresent(entitiesToSave::add);
EntityChanges entityChanges =
flowCustomLogic.beforeSave(
BeforeSaveParameters.newBuilder()
.setHistoryEntry(historyEntry)
.setHistoryEntry(domainHistory)
.setNewDomain(newDomain)
.setExistingDomain(existingDomain)
.setEntityChanges(
@@ -217,11 +217,11 @@ public final class DomainUpdateFlow implements TransactionalFlow {
tld, add.getNameserverFullyQualifiedHostNames());
}
private HistoryEntry buildHistoryEntry(DomainBase existingDomain, DateTime now) {
private DomainHistory buildDomainHistory(DomainBase newDomain, DateTime now) {
return historyBuilder
.setType(HistoryEntry.Type.DOMAIN_UPDATE)
.setModificationTime(now)
.setParent(Key.create(existingDomain))
.setDomain(newDomain)
.build();
}
@@ -290,10 +290,7 @@ public final class DomainUpdateFlow implements TransactionalFlow {
/** Some status updates cost money. Bill only once no matter how many of them are changed. */
private Optional<BillingEvent.OneTime> createBillingEventForStatusUpdates(
DomainBase existingDomain,
DomainBase newDomain,
HistoryEntry historyEntry,
DateTime now) {
DomainBase existingDomain, DomainBase newDomain, DomainHistory historyEntry, DateTime now) {
Optional<MetadataExtension> metadataExtension =
eppInput.getSingleExtension(MetadataExtension.class);
if (metadataExtension.isPresent() && metadataExtension.get().getRequestedByRegistrar()) {

View File

@@ -16,6 +16,7 @@ package google.registry.flows.domain.token;
import static com.google.common.base.Preconditions.checkArgument;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.persistence.transaction.TransactionManagerUtil.transactIfJpaTm;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
@@ -153,7 +154,7 @@ public class AllocationTokenFlowUtils {
throw new InvalidAllocationTokenException();
}
Optional<AllocationToken> maybeTokenEntity =
tm().loadByKeyIfPresent(VKey.create(AllocationToken.class, token));
transactIfJpaTm(() -> tm().loadByKeyIfPresent(VKey.create(AllocationToken.class, token)));
if (!maybeTokenEntity.isPresent()) {
throw new InvalidAllocationTokenException();
}

View File

@@ -45,6 +45,7 @@ import google.registry.model.eppinput.ResourceCommand;
import google.registry.model.eppoutput.CreateData.HostCreateData;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.host.HostCommand.Create;
import google.registry.model.host.HostHistory;
import google.registry.model.host.HostResource;
import google.registry.model.index.EppResourceIndex;
import google.registry.model.index.ForeignKeyIndex;
@@ -85,7 +86,7 @@ public final class HostCreateFlow implements TransactionalFlow {
@Inject ExtensionManager extensionManager;
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject HistoryEntry.Builder historyBuilder;
@Inject HostHistory.Builder historyBuilder;
@Inject DnsQueue dnsQueue;
@Inject EppResponse.Builder responseBuilder;
@@ -128,14 +129,11 @@ public final class HostCreateFlow implements TransactionalFlow {
.setRepoId(createRepoId(ObjectifyService.allocateId(), roidSuffix))
.setSuperordinateDomain(superordinateDomain.map(DomainBase::createVKey).orElse(null))
.build();
historyBuilder
.setType(HistoryEntry.Type.HOST_CREATE)
.setModificationTime(now)
.setParent(Key.create(newHost));
historyBuilder.setType(HistoryEntry.Type.HOST_CREATE).setModificationTime(now).setHost(newHost);
ImmutableSet<ImmutableObject> entitiesToSave =
ImmutableSet.of(
newHost,
historyBuilder.build().toChildHistoryEntity(),
historyBuilder.build(),
ForeignKeyIndex.create(newHost, newHost.getDeletionTime()),
EppResourceIndex.create(Key.create(newHost)));
if (superordinateDomain.isPresent()) {

View File

@@ -15,17 +15,18 @@
package google.registry.flows.host;
import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.failfastForAsyncDelete;
import static google.registry.flows.ResourceFlowUtils.checkLinkedDomains;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.ResourceFlowUtils.verifyNoDisallowedStatuses;
import static google.registry.flows.ResourceFlowUtils.verifyResourceOwnership;
import static google.registry.flows.host.HostFlowUtils.validateHostName;
import static google.registry.model.eppoutput.Result.Code.SUCCESS;
import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import google.registry.batch.AsyncTaskEnqueuer;
import google.registry.dns.DnsQueue;
import google.registry.flows.EppException;
import google.registry.flows.ExtensionManager;
import google.registry.flows.FlowModule.ClientId;
@@ -39,8 +40,11 @@ import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.eppcommon.Trid;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.eppoutput.Result;
import google.registry.model.host.HostHistory;
import google.registry.model.host.HostResource;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.HistoryEntry.Type;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import javax.inject.Inject;
import org.joda.time.DateTime;
@@ -65,20 +69,25 @@ import org.joda.time.DateTime;
@ReportingSpec(ActivityReportField.HOST_DELETE)
public final class HostDeleteFlow implements TransactionalFlow {
private static final ImmutableSet<StatusValue> DISALLOWED_STATUSES = ImmutableSet.of(
StatusValue.CLIENT_DELETE_PROHIBITED,
StatusValue.PENDING_DELETE,
StatusValue.SERVER_DELETE_PROHIBITED);
private static final ImmutableSet<StatusValue> DISALLOWED_STATUSES =
ImmutableSet.of(
StatusValue.CLIENT_DELETE_PROHIBITED,
StatusValue.PENDING_DELETE,
StatusValue.SERVER_DELETE_PROHIBITED);
private static final DnsQueue dnsQueue = DnsQueue.create();
@Inject ExtensionManager extensionManager;
@Inject @ClientId String clientId;
@Inject @TargetId String targetId;
@Inject Trid trid;
@Inject @Superuser boolean isSuperuser;
@Inject HistoryEntry.Builder historyBuilder;
@Inject HostHistory.Builder historyBuilder;
@Inject AsyncTaskEnqueuer asyncTaskEnqueuer;
@Inject EppResponse.Builder responseBuilder;
@Inject HostDeleteFlow() {}
@Inject
HostDeleteFlow() {}
@Override
public final EppResponse run() throws EppException {
@@ -87,7 +96,7 @@ public final class HostDeleteFlow implements TransactionalFlow {
validateClientIsLoggedIn(clientId);
DateTime now = tm().getTransactionTime();
validateHostName(targetId);
failfastForAsyncDelete(targetId, now, HostResource.class, DomainBase::getNameservers);
checkLinkedDomains(targetId, now, HostResource.class, DomainBase::getNameservers);
HostResource existingHost = loadAndVerifyExistence(HostResource.class, targetId, now);
verifyNoDisallowedStatuses(existingHost, DISALLOWED_STATUSES);
if (!isSuperuser) {
@@ -99,16 +108,31 @@ public final class HostDeleteFlow implements TransactionalFlow {
: existingHost;
verifyResourceOwnership(clientId, owningResource);
}
asyncTaskEnqueuer.enqueueAsyncDelete(
existingHost, tm().getTransactionTime(), clientId, trid, isSuperuser);
HostResource newHost =
existingHost.asBuilder().addStatusValue(StatusValue.PENDING_DELETE).build();
historyBuilder
.setType(HistoryEntry.Type.HOST_PENDING_DELETE)
.setModificationTime(now)
.setParent(Key.create(existingHost));
tm().insert(historyBuilder.build().toChildHistoryEntity());
HistoryEntry.Type historyEntryType;
Result.Code resultCode;
HostResource newHost;
if (tm().isOfy()) {
asyncTaskEnqueuer.enqueueAsyncDelete(
existingHost, tm().getTransactionTime(), clientId, trid, isSuperuser);
newHost = existingHost.asBuilder().addStatusValue(StatusValue.PENDING_DELETE).build();
historyEntryType = Type.HOST_PENDING_DELETE;
resultCode = SUCCESS_WITH_ACTION_PENDING;
} else {
newHost = existingHost.asBuilder().setStatusValues(null).setDeletionTime(now).build();
if (existingHost.isSubordinate()) {
dnsQueue.addHostRefreshTask(existingHost.getHostName());
tm().update(
tm().loadByKey(existingHost.getSuperordinateDomain())
.asBuilder()
.removeSubordinateHost(existingHost.getHostName())
.build());
}
historyEntryType = Type.HOST_DELETE;
resultCode = SUCCESS;
}
historyBuilder.setType(historyEntryType).setModificationTime(now).setHost(newHost);
tm().insert(historyBuilder.build());
tm().update(newHost);
return responseBuilder.setResultFromCode(SUCCESS_WITH_ACTION_PENDING).build();
return responseBuilder.setResultFromCode(resultCode).build();
}
}

View File

@@ -19,6 +19,7 @@ import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.host.HostFlowUtils.validateHostName;
import static google.registry.model.EppResourceUtils.isLinked;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.persistence.transaction.TransactionManagerUtil.transactIfJpaTm;
import com.google.common.collect.ImmutableSet;
import google.registry.flows.EppException;
@@ -76,7 +77,8 @@ public final class HostInfoFlow implements Flow {
// there is no superordinate domain, the host's own values for these fields will be correct.
if (host.isSubordinate()) {
DomainBase superordinateDomain =
tm().loadByKey(host.getSuperordinateDomain()).cloneProjectedAtTime(now);
transactIfJpaTm(
() -> tm().loadByKey(host.getSuperordinateDomain()).cloneProjectedAtTime(now));
hostInfoDataBuilder
.setCurrentSponsorClientId(superordinateDomain.getCurrentSponsorClientId())
.setLastTransferTime(host.computeLastTransferTime(superordinateDomain));

Some files were not shown because too many files have changed in this diff Show More