* Partially convert RDAP ofy calls to tm calls
This converts the simple retrieval actions but does not touch the more
complicated search actions -- those use some ofy() query searching logic
and will likely end up being significantly more complicated than this
change. Here, though, we are just changing the calls that can be
converted easily to tm() lookups.
To change in future PRs:
- RdapDomainSearchAction
- RdapEntitySearchAction
- RdapNameserverSearchAction
- RdapSearchActionBase
* Refactor PremiumList storage and retrieval for dual-database setup
Previously, the storage and retrieval code was scattered across various
places haphazardly and there was no good way to set up dual database
access. This reorganizes the code so that retrieval is simpler and it
allows for dual-write and dual-read.
This includes the following changes:
- Move all static / object retrieval code out of PremiumList -- the
class should solely consist of its data and methods on its data and it
shouldn't have to worry about complicated caching or retrieval
- Split all PremiumList retrieval methods into PremiumListDatastoreDao
and PremiumListSqlDao that handle retrieval of the premium list entry
objects from the corresponding databases (since the way the actual data
itself is stored is not the same between the two
- Create a dual-DAO for PremiumList retrieval that branches between
SQL/Datastore depending on which is appropriate -- it will read from
and write to both but only log errors for the secondary DB
- Cache the mapping from name to premium list in the dual-DAO. This is a
common code path regardless of database so we can cache it at a high
level
- Cache the ways to go from premium list -> premium entries in the
Datastore and SQL DAOs. These caches are specific to the corresponding
DB and should thus be stored in the corresponding DAO.
- Moves the database-choosing code from the actions to the lower-level
dual-DAO. This is because we will often wish to access this premium list
data in flows and all accesses should use the proper DB-selecting code
* Properly set up JPA in BEAM workers
Sets up a singleton JpaTransactionManger on each worker JVM for all
pipeline nodes to share.
Also added/updated relevant dependencies. The BEAM SDK version change
caused the InitSqlPipeline's graph to change.
* Fix ContactTransferData SQL loads
ContactTransferData is currently loaded back from SQL as an unspecialized
TransferData object. Replace it with the ContactTransferData object that we
use it to reconstitute.
It's likely that this could be done more straightforwardly with a schema
change.
* Changes requested in review
* Fix obscure bug when checking restore prices of duplicate domain names
There were instances of "java.lang.IllegalArgumentException: Multiple entries
with same key" in the logs, caused by attempting to construct an ImmutableMap
containing duplicate keys. It turns out this was happening in the domain check
flow when the following conditions were all simultaneously met:
1. The older v06 fee extension is used
2. The same domain name is being queried multiple times in a single check
command (which is valid per the spec but doesn't actually make any sense)
3. Said domain exists
4. The cost of a restore (an uncommon operation) is being checked
When all of those conditions were met, an error was being thrown when the
dupe-containing list of domain names was used as the keys of a new Map. This
fixes that bug by calling .distinct() first.
Give enough registrars enough typewriters ...
BUG=179052195
* Revert BEAM pipeline back to SQL credential file
Stop using the SecretManager for SQL credential in BEAM for now. The
SecretManager cannot be injected into the code on pipeline workers
because RegistryEnvironment is not set.
See b/179839014 for details.
* Add db-compare tests to three more flows
Add database comparison to the replay tests for DomainDeleteFlowTest,
DomainRenewFlowTest and DomainUpdateFlowTest.
* Add databaseTransitionSchedule entitiy
* add UpdateDatabaseTransitionScheduleCommand
* small fixes
* change entity structure to no longer be singleton
* add get command
* fix getCache
* Change id to TransitionId enum
* more fixes
* Cleanup tests
* Add link to javadoc
* Add lastUpdateTime
* fix datatype of getCached
* Add a presubmit check to require use of templated SQL string literals
This PR proposes a coding style convention that helps prevent
SQL-injection attacks, and is easy to enforce in the presubmit check.
SQL-injections can be effectively prevented if all parameterized queries
are generated using the proper param-binding methods. In our project
which uses Hibernate exclusively, this can be achieved if we all follow
a simple convention: only use constant sql templates assigned to static
final String variables as the first parameter to creat(Native)Query
methods.
This PR adds a presubmit check to enforce the proposed rule, and
modified one class as a demo. If the team agrees with this proposal, we
will change all other use cases.
* Make BiqueryPollJobAction endpoint internal only
This endpoint makes use of java object deserialization, which allows a
malicious actor to craft a request that can initiate overly broad actions on
the server. Since this endpoint is not widely used for operational purposes,
limit its authorization to "internal only" so that no user agents (even with
admin privs) can access it.
* Add start date for cert enforcement in production
* Add TODO to remove start date check after start date
* revert changes to package-lock.json
* Make start time a constant
* Wire up DeleteExpiredDomainsAction so that it can actually be called
For now I'm just going to be calling it manually (and on sandbox for starters),
but in a few weeks, if all looks good, I'll add the cron job to regularly call
it in production, and this feature will thus be done.
* Use END_OF_TIME as sentinel value for domain's autorenewEndTime
Datastore inequality queries don't work correctly for null; null is treated as
the lowest value possible which is definitely the opposite of the intended
meaning here.
This includes an @OnLoad for backfilling purposes using the ResaveAll mapreduce.
* Convert certificate strings to certificates
* Format fixes
* Revert "Format fixes"
This reverts commit 26f88bd313.
* Revert "Convert certificate strings to certificates"
This reverts commit 6d47ed2861.
* Convert strings to certs for validation
* Add clarification comments
* Add test to verify endoded cert from proxy
* Add some helper methods
* add tests for PEM with metadata
* small changes
* replace .com with .test
* Add clientCertificate to TlsCredentials.toString()
FlowRunner.run() logs these credentials to the GAE logs by implicitly using the
toString() method, so we need to add it if we want it to appear in the logs.
* Use CertificateChecker on login
* Add actual enforcement of requirements in sandbox
* Add new Exceptions
* add validation command to RegistryToolComponent
* Fix error messages
* Add a test for production behavior
* check logs in test
* move loghandler
* Convert (most) HistoryEntry ofy calls to tm
As part of this change, it was necessary to do changes in the JPATM that
are similar (but the opposite) of the changes that we did in
DatastoreTM with regards to converting HistoryEntries to and from the
*History classes.
We leave the ofy() calls in the MapReduce ResaveAllHistoryEntriesAction
for now; that can be converted during the Beam pipeline transition.
Some other tests required registrar-name fixes as well -- because
*History objects have a foreign key on the Registrar table, we have to
use a "real" registrar name in tests.
* Add simple HistoryEntryDaoTest
* Require an override flag to allow updating pending delete domains
Needing to update pending delete domains is an uncommon situation, yet currently
we are allowing superusers to do so without any extra validation (which has led
to errors). This adds a new override flag to gate the update of pending delete
domains; without it, the update will fail.
isEmpty() is not available in the version of Java GAE uses and is throwing
runtime errors (!!). I think these got into our codebases because people don't
have the language version set correctly in IntelliJ; they show as outright
errors for me (I'm on language level 8).
* Add object comparison to replay tests
Allow optional object comparison in the replay test extension and enable it
for the DomainCreateFlow test.
To faciliate this, add two new field annotations to ImmutableObject:
DoNotCompare, to be used for fields that are not relevant to either database,
and Insignificant, to be used for fields that are mutated after they have been
accessed and therefore violate immutability (there is currently only one of
these, however we might discover more in the course of adding more comparisons
to the replay test.
* Revert commented out premium price error log
* Added static create methods for ReplayExtension
* Update the Datastore to SQL migration pipeline
The pipeline now includes all entity types to be migrated by it, and has
completed successfully using the Sandbox data set. The running time in Sandbox
is about 3 hours, extrapolating by entity count to a 12-hour run with
production data. However, actual running time is likely to be longer since
throughput is lower with domains, which accounts for a higher percentage
of the total in production. More optimization will be needed.
The migrated data has not been validated.
* Use better null-handling around registrar certificates
Now with Optional it's always very clear whether they do or do not have values.
isNullOrEmpty() shouldn't be necessary anymore (indeed it wasn't necessary prior
to this either, as the relevant setters in the Registrar builder already coerced
empty strings to null). And also the cert hash is a required HTTP header, so it
will error out in the Dagger component if null or empty long before getting to
any other code.
* Merge branch 'master' into optional-get-certs
* Allow BEAM pipeline to choose JDBC isolation levels
Some BEAM pipelines may only perform READ-ONLY (e.g., reporting) or
blind-write (datastore to sql data migration) operations, which do not
need the default TRANSACTION_SERIALIZABLE isolation level. In such
cases, a less strict level allows better performance.
* Refactor naming and behavior of bulk load methods in TransactionManager
The contract of loadByKeys(Iterable<VKey>) specifies that the method will
throw a NoSuchElementException if any of the specified keys don't exist.
We don't do that before this PR, but now we do.
Existing calls (when necessary) were converted to the new load*
methods, which have the same behavior as the previous methods.
Existing methods were also renamed to be more clear -- see b/176239831
for more details and discussion.
* Add unique constraints on domain_hosts
Add unique constraints on DomainHost (child of DomainBase) and
DomainHistoryHost (child of DomainHistory). DomainHost is non-entity
embedded object and Hibernate does not define indexes automatically.
This should improve read and write performance of the parent entities.
* Add a fast mode to the ResaveAllEppResourcesAction mapreduce
This new mode avoids writing no-op mutations for entities that don't actually
have any changes to write. The cronjobs use fast mode by default, but manual
invocations do not, as manual invocations are often used to trigger @OnLoad
migrations, and fast mode won't pick up on those changes.
* Convert AllocationToken-related classes to tm()
For the most part this is a fairly simple converstion -- changing Key
references to VKey references, using JPA transactions when necessary,
and using the TransactionManager interface. There's a bit of cleanup too
in related code
* Use PollMessageVKey to replace VKey<PollMessage> in DomainBase
* Revert changes to DomainContent
* Use BillingVKey in GracePeriodBase to restore symmetric vkey
* Rebase on HEAD
gbrodmanandGitHub
Weimin YuandGitHub
Michael MullerandGitHub
Ben McIlwainandGitHub
sarahcaseybotandGitHub
Shicong HuangandGitHub