* Fix JpaIntegrationRule in JUnit4
Made DatastoreExtension a JUnit4 Rule.
Nomulus model objects need Datastore API when manipulating Ofy keys.
As a result, JpaIntegrationTestRule must be used with AppEngineRule
or DatastoreExtension.
Also fixed WriteToSqlTest, which is the only JUnit4 test that uses
JpaIntegrationTestRule.
* Add lastUpdateTime column to epp resources
Property was inadvertently left out.
Renamed getter and setter to match the property name.
Added a test helper to compare EppResources while ignoring
lastUpdateTime, which changes every time an instance is persisted.
* Write one PCollection to SQL
Defined a transform that writes a PCollection of entities to SQL using
JPA. Allows configuring parallelism level and batch size.
* Add JUnit Params and start using it
* Convert rest of RDE tests
* Don't check headers for generated tests
* Expand visibility to fix build breakage
* Bump JUnit versions to 5.6.2
* Add a "buildFmt" gradle target
This does the same thing as the automatic Java build target, except instead of
failing if the code formatting isn't correct, it just automatically reformats as
necessary and continues on.
* Remove unnecessary mustRunAfters
* Make it run tests too, and add :taskTree task
* Rename task to coreDev and remove run afters
* Add task tree dependency
* Actually that may not be necessary
* Change Subdomain class to contain domainRepoId
* Remove jpaTm from Spec11PipelineTest and change clientId -> registrarId
* Remove 'client' from a comment
* Include changes to Spec11Pipeline
* add SafeBrowsingTransforms
* Run style
* Use an enum instead of boolean in EntityTestCase constructor
It's more clear to use an enum rather than just a simple boolean
* Add Javadoc and make the enum name more verbose
* Double the # of pubapi instances to better handle traffic spikes
We may also consider switching to an automatic scaling mode soon, on the hope
that it's working better than the last time we tried it (it would help to keep
resource costs down at least).
* Create ContactHistory class + table
This is similar to #587, but with contacts instead of hosts.
This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest)
Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory.
* Move ContactResource -> ContactBase
* Alter ContactBase and ContactResource
* Verify that the RegistryLock input has the correct registrar ID
We already verify (correctly) that the user has access to the registrar
they specify, but nowhere did we verify that the registrar ID they used
is actually the current sponsor ID for the domain in question. This is
an oversight caused by the fact that our testing framework only uses
admin accounts, which by the nature of things have access to all
registrars and domains.
In addition, rename "clientId" to "registrarId" in the RLPA object
* Change the wording on the incorrect-registrar message
* Output PO number in detailed report
The PO number header was added during the beam migration but we forgot
to print the actual data in the corresponding column. This resulted in a
misalignment of columns in the detailed report.
This PR fixes it. Note that we cannot drop PO number from the header (as is not
useful in the detailed report) because the header represents all fields
that are to be parsed from the SQL query results, and PO number *is*
needed when generating the invoice itself. By dual-purposing the header
(both as the required fields in the parser and the first line in the
detailed report) we have to include the value of PO number in the
detailed report CSV as well.
* Disambiguate injected Cloud SQL parameter names
This allows us to also inject the BeamJpaModule into RegistryTool, which
allows us to use the SocketJpaTransactionManager in Beam pipelines.
Some side effects of this include:
- duplication of KMS connections -- one standard, one Beam
- duplication of the creation of the partial Hibernate SQL configs
- removal of ambiguity between credentialFileName, credentialFilename,
and credentialFilePath -- we now use the latter.
- Performing the credential null check when instantiating the SQL
connection rather than when instantiating the module object. See the code
comments for more details on this.
I verified that this compiles and the tests run successfully when
injecting a @SocketFactoryJpaTm into a Beam pipeline.
* Remove two unnecessary config points and change the name of two params
* Use @Config instead of @Named and change the pool size
* Replace non-visible link with code
* Load Datastore snapshot from backup files
Defined a composite transform that loads from a Datastore export and
concurrent CommitLog files, identify entities that still exist at the
end of the time window, and resolve their latest states in the window.
* Exclude Test/Monitoring Registrars from escrow
Registrars used for testing and monitoring should not be included
in Data escrow. They also lack the required ianaIdentifier property
and would fail ICANN data validation.
Note that since alpha and crash environments have bad data that
break the RDE process, we need to verify this change in Sandbox.
* Make EppResource.loadCached() use batched fetch
Use a batched fetch (ofy().load().keys(...)) from datastore in
EppResource.loadCached().
To support this, convert TransactionManager.load(Iterable<VKey>) to accept the
more flexible generic parameters and return a map.
* Simplify datastore key streaming
* Changes requested in review.
* Update to generic Spec11ThreatMatch table
* Fix SQL syntax
* Make changes to the schema and add a test for null and empty threatTypes
* Fix a small typo
* Change the exception thrown with illegal arguments
Change the import for isNullOrEmpty
* Fix import for checkArgument
* Added a threat to test multiple threat types
This adds an entriesToImmutableMap() collector that can be used in place of
toImmutableMap(Map.Entry::getkey, Map.Entry::getValue()).
It also fixes up some existing calls that use toImmutableMap() when terser
alternatives exist.
* Make nomulus compatible with Java 11
This fixes the double-spacing bug with logged EPP XML on App Engine that started
appearing after App Engine switching from using Java 8 to Java 11. Java 9 made
some changes to XML Transformer classes that needed a little bit of work to
accommodate.
This also fixes the unit tests that were failing in Java 11 (all of which were
related to said XML Transformer changes).
* Make code review changes
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.
ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.
Added a mockito dependency, and upgraded mockito-core to 3.3.3.
Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
The added property will by default be null for existing entities and false for new entities. We could write a temporary MapReduce to set it to false for all existing ones, but it seems easier to just modify the query to account for the nullness of the column. The need to check for null won't exist once we migrate the Registry entity to Cloud SQL.
TESTED=deployed to alpha, set the property for tld zombo to true and triggered an export, verified that the column is null for all rows other than the row for zombo.
* Delay SQL availability check during Tool start
During Cloud SQL migration, Cloud SQL password setup in Registry
server requires a few tool commands. Such commands must be able
to run before Cloud SQL is configured.
In this change, we deplay SQL credential loading until access is
needed.
Hibernate complained that ContactResource.getTransferData should not be
final:
WARN: HHH000305: Could not create proxy factory
for:google.registry.model.contact.ContactResource
org.hibernate.HibernateException: Getter methods of lazy classes cannot
be final: google.registry.model.contact.ContactResource#getTransferData
* Set up JpaTransactionManager in BEAM pipelines
Added modules and utilities to create JpaTransactionManager in BEAM
pipelines.
Not wanting to set up AppEngine Remote API to access Keyring in the
Datastore, we instead use the credential files in GCS, which are
used by Spinnaker/Cloud Build and desktop access. Added utility
to download, decrypt, and parse the file. Also added/modified dagger
modules.
* Don't expect a renewal fee on restores when one isn't due
This is a fix on top of #632 so that domain restore commands don't require
acking an illusory renewal fee for 1 year when that isn't actually happening
(i.e. if the domain isn't yet past its original expiration).
Unfortunately, there's still a problem remaining wherein the restore fee on a
domain check will always include the additional year even if it's not
necessary. We don't have a good solution to that. Also note that in versions of
the fee extension more recent than 0.6, the fee extension cannot be passed on a
domain info command at all, so the domain check command is the only way you have
to determine what the restore fee should be. So we definitely do want to get
that right so that the apparent restore fee on a check is the same as the actual
restore fee when running the restore command. We're not quite there yet though
and it's hard to say how we will get there, since we don't load domains during a
domain check command for performance reasons yet we would need to do so in order
to know the expiration date and thus whether the additional year of renewal
should be charged.
A problem for a future PR.
* Squash everything together
Create SafeBrowsing_Threats table
Create LocalDateConverter and add indexes to SafeBrowsingThreats
Add indexes to SafeBrowsingThreats and make small style changes
Pass in DateTimeFormatter
Delete LocalDateConverterTest.java
Rebase
Make changes to ThreatType comments
Create LocalDateConverterTest
Add review changes
Add SafeBrowsingThreatTest
Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys
Change imports
Add foreign keys and rename version number
Add new generated db-schema file
Clean up null test cases
Add changes
Add foreign keys into SafeBrowsingThreatTeat and apply style checks
Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file
Make small changes to SafeBrowsingThreatTest
Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat
* Change V35 -> V36
* Add a foreign key test for a reference to Registrar
* Move some variables around
* Add testcontainers' Junit5 support dependency
Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.
Migrated PersistenceModuleTest as an example.
Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
* Fix flakiness caused by ofy entity name conflict
Overrode the 'kind' of two test entities to prevent name conflicts.
Tests are flaky because of this.
Added a check in AppEngineRuleTest for conflicting kinds.
* Show price of reserved domains when using matching allocation token
When the registrar passes the fee extension, this shows the price of the domain
on a check command for reserved domains if the provided allocation token is a
match. Of course, the price is already always displayed on non-reserved names
(regardless of whether the specific provided token is a match or not).
This affects domain checks only; the price is already always displayed on domain
creates because you already by definition have access to register the domain in
question.
* Refactor pipline for Datastore backup loading
Refactored pipeline transforms.
Added testing utilities that handles assertions better.
Investigated and documented challenges in serializing Ofy entities
without side effects.
* Remove 'fullyQualified' from host and domain names
We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.
Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API
* Add TODO to rename fields
* Implement a persistable Transaction object
Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.
* Changes requested in review
* Add a mujtation count to the persisted format
* Fix cookie processing for RDAP URL update
The existing code only does cookie processing on the _first_ Set-Cookie
header. Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.
Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
* Add proof of concept for HostBase and HostHistory classes
* Use a PROPERTY accessor for @Ids
* Add an unused setter method for Hibernate's sake
* Refactor HostHistory
* Some responses to CR
* Fix relationship and test
* Manually manage the foreign keys for HostHistory
* Protect HostBase's builder and use text for the enum type
* Add responses to CR
- Add javadocs
- Create an ID sequence for host history objects
* Don't try to set the ID
* Use a Long and remove the setter
* Add some comments and rename a couple fields
* Don't change Datastore schema
* Use Long in the Datastore schema
* Add new createVKey method
* Add comments and rename fields
* Rename v27->v31 and regenerate the golden
* Fix superordinateDomain and inetAddresses in HostHistory
* V31 -> V32
* Fix SQL files that got messed up in the merge
* Configure and use a manually-created history ID sequence
* Add three more indices to HostHistory
* Add DomainTransferData and ContactTransferData
* Refactor TransferData to remove unused fields in Contact table
* Add scope for TransferData's type parameter
* Allow class-specific creation of symmetrict VKeys
When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.
As a test case for this, implement the createVKey() static function for
DomainBase. Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
* Refactor Fee handling so that each fee knows if it's premium
This is a noop for now, as the new isPremium boolean isn't yet used by anything,
but it will be used in follow-up PRs to add additional fee information using the
fee extension (see: b/157621273). Specifically what we're trying to do here is
return <fee:command name="create" standard="1"> (using the finalized version of
the fee extension) when an entire command has no premium fee associated with
it. And in the current earlier versions of the fee extension that we support,
we'll want to display the correct fee amount and class for creates/checks on
reserved domains when a valid allocation token is passed. This also needs the
isPremium information.
There are no testing implications yet because isPremium isn't exposed anywhere,
but there will definitely be lots of test changes once it's feeding into EPP-
visible changes.
* Rename things, add method Javadoc
* Apply formatting
* Restore the original expiration time on domain restore
Except if that time is now in the past, then add a year to it.
* Apply auto-formatter changes to fix my local build
* Merge branch 'master' into restore-expiry-date
* Fix reversed comments
* Use the server host when creating the registry lock verification URL
The app doesn't know about any external configuration that may point to
this app, so there's no way of finding out that, for instance,
registry.google points to the app. Thus, we have to use what the user
gives us so that, in our case, the registry-lock verification
emails can point to https://registry.google/registry-lock-verify instead
of https://domain-registry.appspot.com/registry-lock-verify. The former
is used by clients / users to authenticate, and unfortunately
authenticating on registry.google does not give authentication to
domain-registry.apspot.com.
Tested using the RDAP code that uses getServerName() -- in that case, if
you access registry.google/rdap/<>, it uses registry.google in the URLs
but if you use domain-registry.appspot.com/rdap/<>, it uses
domain-registry.appspot.com in the URLs.
Relatedly, frontend_config_prod-appengine.asciiproto in Piper
is what configures registry.google to point to
domain-registry.appspot.com
* Log client certificate type and length
It appears that most client certificates are RSA certs, but we should
make sure that is indeed the case. Print out the strength of the cert
if it is RSA.
Also adds supports for TLS 1.3 and print out the supported cipher suites.
* Add a comment about zero length certificate
* Make length of non-RSA keys -1
* Don't use TLS 1.3 if JDK SSL provider is used
* Load CommitLog into BEAM pipeline
Created tools that can generate CommitLogs on a test
datastore.
Defined BEAM transforms that load from CommitLog files and
added a few simple tests.
This is work-in-progress. Next step is to build a consistent
backup by merging Datastore exports and CommitLogs.
We'll eventually want to shift everything over to using registrar_id and
registrarId rather than client_id and clientId but for the sake of the
Datastore schema and existing code, we won't change the Java identifier
for now. Once we're completely and only on SQL, we can rename the Java
field easily.
* Create a converter for sets of inetAddresses and use it in HostResource
This can just be a set of strings where each string represents an
address; there's no need for it to be a separate table. This allows
for simplification of the SQL schema.
* Regenerate golden SQL file after renaming v28 -> v29
* Add more tests and rename a typo in the file
* Refactor common test code and use tm methods
* Use JUnit5 API
* Rename test entity
* Add columns for TransferData in Domain and Contact
* Rename flyway file and foreign key
* Rebase on master and address comment
* Compileable commit
* Fix unit test
* Refactor TransferServerApproveEntity
* Use tm().delete(vkeys)
* Rename transfer_period fields
* Rename client_id to registrar_id
* Rebase on master
* Resolve comment
* Rebase on master
* Store the superordinateDomain reference as a VKey rather than Key
This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
Given that we currently have no way of reconstituting a symmetric key
from the asymmetric key (or at least, we don't have a 100% reliable way
of doing so) it's best to keep keys as asymmetrical, referring to the
correct database. That way, we don't get situations where we cannot
compare equality of two keys due to one being asymmetrical and one being
symmetrical.
* Remove platformType and threatEntryMetaData fields from ThreatMatch
* Run google-java-format on both files
* Add test for removal of unnecessary fields
* Removed unnecessary fields from Spec11PipelineTest.testEndToEndPipeline_generatesExpectedFiles
* Added style check
* Fix typo
* Load raw records from Datastore export
Created a tool that can export from a test instance of Datastore.
Defined Beam pipeline transforms for loading raw records back from
the export.
This is the first part of the effort to create a consistent snapshot
of Datastore. The next step is to load entity records from CommitLog
files.
* Use TransactionManager for hosts and contacts
Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts. In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.
* Minor formatting fix
* Refactor LevelDbFileBuilder to accept DS Entity
Builder now can directly work with Datastore Entity objects.
No need to wrap data in ComparableEntity.
* Generate sql schema for BillingEvent
* Change to use sequence
* Address comments
* Resolve warnings and remove duplicate cost related fields
* Increase the flayway file version to V25
* Remove extra space
* Split to 3 tables, merge VKey
* Rename talbes
* Rename repoId to domainRepoId
* Exclude VKey in schema.txt
* Rename target_id to domain_name
* Fix javadoc
* Resolve comments
If the import from Datastore to BigQuery fails, there is no point
enqueuing a job to update the snapshot view.
Also when there's an error updating the snapshot view, log it at severe
level. The HTTP exception thrown is logged at info and triggers a retry
implicitly. I'm not sure if we want this behavior though. Do we want to
retry upon snapshot updating failures? Unless the failurs are transient,
retrying doesn't help. In our case the failure (End of time out of range
in Standard SQL) is not transient.
EntityClasses.ALL_CLASSES has all of our registered entities so
we can use it to initialize VKeyTranslatorFactory.classRegistry
to avoid adding them one by one.
Also, this PR changed to use Key.getKind() to get the kind of
the entity to solve the problem that when the entity class
is an inner class, its kind should still be the class name
instead of OuterClass$InnerClass.
* Read LevelDb incrementally
Made LevelDbLogReader an iterator over a LevelDb data stream,
Reducing memory footprint which is important when used in a
Dataflow pipeline.
* Make VKey persist to datastore as a key
Convert nsHosts entirely to VKey as a proof-of-concept.
Tested as follows:
1) Deployed to crash, verified that nameservers were visible for several
domains (indicating that we are able to load a set of Keys as VKeys)
2) Updated the set of nameservers for a domain (removing some initial
hosts) and verified that the changes went through.
3) Deployed the old version to crash, verified that I was able to retrieve
the newly saved VKeys as Keys.
4) Modified the hosts for the same domain (adding back one of the hosts)
and verified that the change took effect.
5) Redeployed this change to crash, again updated the nameservers to add
another host.
6) Again restored the old version, verified that the new hosts were
visible.
* Changes in response to review
* Convert to a single VKeyTranslatorFactory instance
* Moved vkey field rename to V25
Normal HttpException logs at INFO because they usual do not indicate
anything out of the ordinary and is meant to convey to the client that
there is some expected error. However InternalServerErrorException is
something that we do care about being alerted for so we log it at SEVERE.
* Add TimedTldStateTransitionMapConverter
* Move timedTransitions to a base class and add BillingCostTransitionConverter
* Add test of TimedTransitionPropertyConverterBase
* clean up tests
* Switch tests to JUnit 5
* Make JpaUnitTestRule an extension
* Implement all DatastoreTransactionManager methods
In the course of this:
- Make assertDelete() specific to JpaTransactionManager, remove the return
value from delete()
- Converter "in transaction" assertion to IllegalStateException, which is less
JPA specific.
* Upgraded DatastoreTransactionManagerTest to junit5
* Reflect refunded billing events on deletion in expiration time
This doesn't make any change at the time of the domain deletion itself, but it
will matter if the domain is then undeleted, because we need to know what
expiration date to restore, and if there were any renew or autorenew charges
that were refunded by the deletion because they were in a grace period, they
shouldn't be coming back during the restore.
* Add tests for new expiration date behavior
* Add handling of add/renew grace period overlap
* Convert DomainBase's contacts to VKeys
Convert usage of DomainBase contacts from Key to VKey. This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
On certain machines (one of mine) the outcastTest consistently fails due
to the following error:
java.lang.NoClassDefFoundError: Could not initialize class
google.registry.persistence.transaction.JpaTestRules$JpaIntegrationTestRule
If I reduce maxparallelForks to 3 it consistently passes. This issue was
mentioned here:
https://discuss.gradle.org/t/junit-test-fails-with-noclassdeffounderror-only-when-maxparallelforks-1/6047
But this post was 8 years old and no solution was identified.
Since we rarely (if ever) want to check the result of a single element load,
make TransactionManager.load(VKey) return non-optional, non-nullable and just
throw a NoSuchElementException if the entity is not found.
Also add a maybeLoad() that does return an optional in case we ever want to do
this (exists() should suffice for an existence check).
* Implement DatastoreEntity and SqlEntity on more classes
For classes that aren't going to transition to SQL, they should just
return an empty list of SqlEntities. When reading these in from the
commit log manifests, we just won't persist anything to SQL.
By having all Datastore entity classes implement DatastoreEntity, we can
avoid potential bugs where we forget to transition some entity to SQL,
or we forget to have the capability to read back from the commit logs.
Note: the EntityTest is still @Ignore'd because there are many SQL and
Datastore classes left -- ones that we are still in the process of
converting or adding, or ones that require more complicated transitions.
Note: Locks and Cursors aren't converted (even though we could) because
they're ephemeral
* Responses to CR
Add a @EntityForTest annotation
fix null that snuck in
* Keep the test ignored for now
* Remove email-editing footgun
Email address is used as the primary key so we should be very careful
about changing it. This will have even more importance when this is the
location to which we will be sending registry lock confirmation emails.
Note: we allow addition or removal of contacts through the UI (and don't
want to disable that) and because all edits are performed by saving the
entire list of contacts, we can't explicitly prevent all possible edits
of email address in the backend. So this doesn't technically prevent
anything security-wise, but it makes it much more difficult to
accidentally edit an email when you shouldn't.
* Enforce non-deletion of registry-lock-enabled contacts
* Fix tests
* Specify contact
* Fix flaky tests due to Entity name conflicts
Objectify siliently replaces current registration of a given kind
when another class is registered for this kind. There are
several TestObject classes in the current code base, which by
default are all mapped to the same kind.
Tests have only been flaky because impacted tests need to run
in specific orders for failures to happen. Using multiple executors
in Gradle also reduced the likely hood of errors. To reproduce the
problem run the following tests in order (e.g., by putting them in
a test suite):
1. ExportCommitLogDiffActionTest
2. CreateAutoTimestampTest
3. RestoreCommitLogsActionTest
In this PR, we
- Made sure all entities have unique kinds.
- Made all test entities register with AppEngineRule instead of directly
with ObjectifyService.
- Added code in AppEngineRule to check for re-registrations.
- Added presumit check for forbidden direct registration.
* Allow the `nomulus renew_domain` command to specify the client ID
This means that a superuser can renew a domain and have the associated history
entry, one time billing event, and renewal grace period be recorded against a
specified registrar rather than the owning registrar of the domain. This is
useful to e.g. renew a domain for free by "charging" the renewal to the
registry's fake registrar. Since the grace period is written to the specified
cliend id as well, if the actual registrar deletes the domain, they don't get
back the money that they didn't pay in the first place.
* Don't NPE when nomulus tool is run without a subcommand
This occurred when an environment was specified but without a subcommand. Now,
the list of valid subcommands is outputted instead of seeing a generic NPE.
This also makes some formatting changes in other files that were causing the
incremental format check to fail.
* Try AppEngineRule
* Remove separate deployment of persistence.xml
We added a step to explicitly copy persistence.xml because for some reason it
wasn't originally getting deployed to app-engine, resulting in failures on
startup. However, this file is now included in core.jar and we are now
getting a warning about multiple persistence units with the same name as it
reads the files from both the filesystem and core.jar.
* Add SqlEntity and DatastoreEntity interfaces
These will be used when replaying transactions from either the Datastore
commit logs or the SQL Transaction objects.
When Datastore is the primary database, we will read in the
Datastore commit logs, convert each saved entity to however many SQL
entities, then save those SQL entities in SQL.
When SQL is the primary database, we will read in the SQL objects from a
yet-to-be-created SQL table, convert them to however many Datastore
entities, then save those Datastore entities in Datastore.
This PR includes a couple simple examples of how this will work for entities that are
saveable in both SQL and Datastore (the simple case).
* Add 1-1 mapping between entity annotations and interfaces
* Add JPA annotations to ContactResource and generate schema
* Resolve comments
* Resolve comments
* Manually add foreign key constraints
* Run with junit5
* Rebase on HEAD
* Fix DomainBaseSqlTest
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Hack to call setup and teardown in JUnit5 suite
JUnit 5 runner does not support @BeforeAll and @AfterAll declared
in the Suite class (as opposed to the member classes). However,
staying with the JUnit 4 suite runner would prevent any member
classes from migrating to JUnit 5.
We use a hack to invoke suite-level set up and teardown from tests.
This change is safe in that if the JUnit 5 runner implementation changes
behavior, we will only see false alarms.
* Implement VKeyConverter
Implement a SQL converter that can be used for VKey objects.
Caveats:
- This only works with string columns (there's an excellent chance that all of
our VKeys will use SQL string columns).
- Using this dpesn't establish a foreign key constraint between the referenced
type (the "T" in VKey<T>) and the entity itself: this needs to be
defined manually in the schema.
* Exclude proxy configs from the FOSS jar
No sensitve data exposed.
Added a todo to modify the release process and stop
building the foss jar on the merged repo.
* Add verification that domain labels aren't multi-level domains
In addition, I did a bit of test refactoring because previously, the
CreateOrUpdateReserveListCommandTestCase test cases weren't actually
testing the proper things -- they were failing with
IllegalArgumentExceptions, but not the right ones.
* Change test name and use IDN library
* Handle numeric labels
String like "0" or "2018" are valid labels but not valid domain names
* Use IDN validation with a dummy TLD
On Arch Linux, DumpGoldenSchemaCommandTest failed due to the follow
error:
java.lang.RuntimeException: Container.ExecResult(exitCode=1, stdout=, stderr=pg_dump: [archiver] could not open output file "/tmp/pg_dump.out": Is a directory)
However I cannot figure out why this permission error happens, as the
docker command is executed as root. Saving the pg_dump output to a
temporary file and copy it over the mapped file works, so I don't
know...
* Migrate SqlIntegrationTestSuite members to Junit5
Made InjectRule and EntityTestCase work with both JUnit4 and 5.
Note that "@RunWith(JUnit4.class)" is no longer needed on
JUnit4 test classes. Therefore, its removal from EntityTestCase
has no impact on child classes. All of them are still included in
tests.
Migrated remaining member classes in SqlIntegrationTestSuite to JUnit5.
* Add Test suite support for JUnit 5 classes
Added Gradle dependencies and updated lockfiles.
Updated SqlInegrationTestSuite to use new annotations.
Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.
Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
* Make AppEngineRule work with JUnit 5
Made AppEngineRule work with both JUnit4 and JUnit 5 and applied
it to PremiumListTest.
Next step is to convert SqlIntegrationTestSuite.java to JUnit5.
* Actually run JUnit 5 tests
In Gradle, JUnit 5 must be explicitly enabled with a call to
test.useJUnitPlatform().
The FilteringTest used in :core must also enable JUnit5 separately.
Fixed AppEngineRule to work with the few tests that have migrated
to JUnit5.
More work is needed with AppEngine before we can migrate tests
that actually use Cloud SQL.
For context, with @EnableRuleMigrationSupport, JUnit 5 runner calls
an external resource's before() and after() methods. TestRule.apply()
is not called, therefore any setup done their will be bypassed with
JUnit 5.
* Upgrade CompareDbBackup for Datastore V3
Upgrade the CompareDbBackup class to work with latest
Datastore backup directory structure.
Also fixed a few unrelated minor issues:
- Remaining cases of improper use of System.setOut
- Wrong import order in one class
* Improve error information in coverage test.
If the golden schema isn't up-to-date with the persistence model, the coverage
tests fail with an exception chain that ends in a PSQLException 'relation
"TableName" does not exist' which is kind of misleading when the problem is
that your golden schema isn't up-to-date.
Check for this error in the coverage tests and generate a more informative
error message indicating a likely root cause.
* Key to VKey conversion for Nameserver
This change illustrates the conversion of a single key in the system
(Key<HostResource> as used in the "nameservers" field of DomainBase) to a
VKey.
It currently builds, but had some curious (possibly unrelated?) test failures
that I have not fully investigated.
* Latest round of changes, all tests pass.
* Changes requested in review.
* Fix problems with null check in VKey accessors
Add maybeGet versions of getSqlKey() and getOfyKey() that return Optional
objects and make the nameserver management routines use those instead.
* Reuse JPA EntityManagerFactory in tests
Reuse EntityManagerFactory instance in tests if the requested schema
stays the same. Only truncate tables and reset sequences when reusing.
Note that the jdbc driver needs to be informed to expect out-of-band
schema changes.
* Don't rely on the password field's existence for admins
We don't have the field when it's an admin user that's logged in. A
nicer language would have caught this unfortunately.
* Drop schema instead of database in Sql tests
Speed up the database cleanup between tests by dropping the schema
instead of the database. The new approach is much faster.
Ad hoc measurement on my desktop shows that :core:sqlIntegrationTest
improves from 73 seconds to 48 seconds, and :core:standardTest
improves from 12m40 to 7m40.
* Allow backwards compatibility with JUnit 4 @Rules in JUnit 5
This allows us to defer having to re-implement all of our JUnit 4 Rules as JUnit
5 extensions for now, while continuing to in-place upgrade all existing JUnit 4
test classes to JUnit 5.
As proof of concept, this upgrades PremiumListUtils (which uses AppEngineRule,
our largest and most complicated @Rule) to use the JUnit 5 test runner.
* Apply formatter to entire file
* Flat map to avoid unclean errors in bad situations
Also properly reflect that for admins, we will use their user email
* Make MS's GAE user ID a public static field
* Add a registryLockEmailAddress field to RegistrarConctact objects
Because we need to manage the login email, it should be on an account
that we manage. However, for registry lock, we would want to send the
verification emails to a separate email address that the user can use.
As a result, we will use a second field for a user-accessible registry
lock email address. This must be set on the contact when enabling
registry lock for this contact.
* Responses to CR
* derp
* Fix the test server
This rule isn't necessary any more since we merged the SQL-starting rule
into the AppEngineRule logic. Furthermore, it actually causes the test
server to crash because we try to drop-and-create the DB twice, the
second time while the first instance is still connected.
* Make jpaTm for nomulus tool use local credential
* Remove unused methods in RegistryToolEnvironment
* Fix order of annotations
* Remove unused method in PersistenceComponent
* Move the creation of credential to the module
* Move creadential creation to AuthModule
* Add a TODO
* Add a relockDuration to the RegistryLock SQL object
This is the length of time after an unlock that we will re-lock the
domain in question.
* Sort by domain name for stability
Note: this is likely not the best solution for the UI but we can iterate
on this.
* Add nullable
* Add a converter for Duration
* Match logged-in GAE user ID with registrar POC user ID
The reasoning for this is thus:
We wish to have the users log in using Google-managed addresses--this is
so that we can manage enforcement of things like 2FA, as well as generic
account management. However, we wish for the registry-lock confirmation
emails to go to their standard non-Google email addresses--e.g.
johndoe@theregistrar.com, rather than johndoe@registry.google.
As a result, for registry lock, we will enable it on
the johndoe@registry.google account, but we will alter the email address
of the corresponding Registrar POC account to contain
johndoe@theregistrar.com. By doing this, the user will still be logging
in using the @registry.google account but we'll match to their actual
contact email.
* fix up comments and messages
* Error if >1 matching contact
* include email addresses
* set default optional
* fix tests
* Disambiguate naming of VKey.create() overloads
It was discovered in the course of trying to convert the larger codebase
to VKey.create() calls that method overloading isn't a very effective
discriminator in cases where "Object" is one of the distinguishing
argument types:-)
Convert the two specialized create() methods to createOfy() and
createSql() so that (at least in the former case) we'll get a
compile-time error if we aim to create a VKey for an Ofy key from an
object of the incorrect type.
* Add a RelockAction and reference to relocks in RegistryLocks
* Respond to CR
- refactor the request param exception logging a bit
- don't log an error if the domain was already locked, just skip
* Save a relock for all locks (if possible)
* derp
* Long -> long + remove unnecessary transact
* semantic merge conflict woo
* fix another semantic merge conflict
* Update SqlIntegrationTestSuite
Edited Javadoc to emphasize that suite members should be DAO tests.
Removed functional tests from the suite. They do not benefit much
from running against different schemas when the entities they use
are already covered by DAO tests.
Added DomainBaseSqlTest to the suite, which tests DomainBase.
* Create a nom_build wrapper script
nom_build is a wrapper around ./gradlew. It's purpose is to help us
deal with properties. The main problem that it is trying to solve is
that when properties are specified using -P, we don't get an error if
the property we specify isn't correct. As a result, a user or a build
agent can launch a build with unintended parameters.
nom_build consolidates all of the properties that we define into a
python script where the properties are translated to flags (actual
gradlew flags are also proxied). It also generates the property file
and warns the user if the current properties file is out of sync with
the script and includes documentation on each of the properties.
* Enable Cloud SQL when Datastore is enabled for unit test
* Add explanation for why add a ETA field in GenerateEscrowDepositCommand
* Fix line length
* Ignore membershipt test but bring back test suite
* Fix tiny issue
* Listen to the user hitting enter in the lock/unlock modal input fields
Listen to both, just in case one or the other is disabled
* Don't require that the element exist
* Fix broken builds when Maven Central is used
Gradle 6.2.1 apparently introduces a behavior change wrt boolean
expression: empty string used to eval to false, but now evals to
true.
Pre Gradle 6.2.1, root project's Gradle properties apparently were
not set to buildSrc. Now they are passed on to buildSrc -- mavenUrl
in buildSrc changes from null to "".
Both changes break the project when mavenUrl and/or pluginsUrl are
not set on command line.
Also added junit.jupiter-api as testCompile dependencies to projects.
This is a directly used dependency, whose absence causes a Lint
warning.
* Add BasicDao
* Refactor RegistrarDao to extend BasicDao
* Introduce VKey and rewrite BasicDao
* Move CRUD methods to TransactionManager
* Refactor code to simplify the way to get id from entity and sqlKey
* Assert in transaction
* Fix broken test
* Change methods name
* Always validate domain name on allocation token
This is in response to a client-reported error, where they accidentally sent the
wrong domain name on a domain create that included an allocation token. What
should have happened (and that now happens as of this commit) is an error being
thrown that the allocation token does not match the domain name being created.
What happened instead was that, since the incorrectly submitted domain name was
not reserved, the create succeeded (as it would for all creates of unreserved
domains in GA) and the allocation token was redeemed, which is not what you'd
expect.
* Fix tests to reflect changed check behavior
* Refactor DomainLockUtils methods to take a time rather than a clock
* Add administratively (un)lock methods
* Responses to CR
- Javadoc changes
- Method renames
- Variable renames
* Refactor lock methods to use JPA transaction time
* Remove clock, use Datastore transaction time
* Properly use Datastore transaction time, batched
* Continue to throw exceptions on invalid domains
* DAO writes should be in a transaction
* Assume in-transaction for all RLDao methods
* clean up test
* Fix more tests
* add comment
* Add daily cron job for IcannReportingUploadAction
This job checks each ICANN cursor's cursorTime. If all cursorTime values are in the future, the action exits and does nothing. For each cursor that has a cursorTime in the past, the corresponding report is uploaded if it is staged, or logs an error message if the report has not been staged yet.
* Change cron job description
* Add Registry Lock UI
* Responses to CRs, mostly
TODO:
- Figure out wording for the 'not enabled yet' message
- Include the server status change cost in the email, or in the UI?
- Should we show non-completed lock requests in the UI?
* Fix get action test
* Change the not-allowed-for-registrar msg to include support email
* Change the wording on the price
* Move TLD input into the modal, and other changes
- don't log the password
- test to make sure the password shows bullets
* Responses to CR and cleanup
* Format closer to something proper
* Use -t instead of main params for TLDs in nomulus count_domains command
This makes the command consistent with list_domains. I use both frequently and it
was annoying forgetting which one takes -t and which uses main parameters. Now
they both work the same way.
* Add a scrap command to backfill registry locks
* fix tests
* Change comments and messages
* Use URS time (best effort) if one exists
* Don't bother with root cause
* Lazily load jpaTm
This fixes an exception we were seeing in production where commands that only
implemented CommandWithRemoteApi (and that had nothing to do with Cloud SQL)
were nevertheless trying to initialize a JPA TM instance, and then failing.
That stacktrace looked like:
com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error": "invalid_grant",
"error_description": "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:423)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
at google.registry.keyring.kms.KmsConnectionImpl.attemptDecrypt(KmsConnectionImpl.java:163)
at google.registry.keyring.kms.KmsConnectionImpl.lambda$decrypt$0(KmsConnectionImpl.java:148)
at google.registry.util.Retrier.callWithRetry(Retrier.java:153)
at google.registry.util.Retrier.callWithRetry(Retrier.java:130)
at google.registry.util.Retrier.callWithRetry(Retrier.java:95)
at google.registry.keyring.kms.KmsConnectionImpl.decrypt(KmsConnectionImpl.java:147)
at google.registry.keyring.kms.KmsKeyring.getDecryptedData(KmsKeyring.java:209)
at google.registry.keyring.kms.KmsKeyring.getString(KmsKeyring.java:178)
at google.registry.keyring.kms.KmsKeyring.getToolsCloudSqlPassword(KmsKeyring.java:100)
at google.registry.persistence.PersistenceModule.providesNomulusToolJpaTm(PersistenceModule.java:124)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.proxyProvidesNomulusToolJpaTm(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:61)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:39)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:12)
at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
at google.registry.persistence.DaggerPersistenceComponent.nomulusToolJpaTransactionManager(DaggerPersistenceComponent.java:168)
at google.registry.persistence.transaction.TransactionManagerFactory.createJpaTransactionManager(TransactionManagerFactory.java:38)
at google.registry.persistence.transaction.TransactionManagerFactory.<clinit>(TransactionManagerFactory.java:29)
at google.registry.model.registry.Registries.lambda$createFreshCache$2(Registries.java:60)
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:243)
at google.registry.model.registry.Registries.getTlds(Registries.java:85)
at google.registry.model.registry.Registries.assertTldsExist(Registries.java:112)
at google.registry.tools.CountDomainsCommand.run(CountDomainsCommand.java:41)
at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:243)
at google.registry.tools.RegistryCli.run(RegistryCli.java:168)
at google.registry.tools.RegistryTool.main(RegistryTool.java:127)
The TL;DR is that RegistryCli was over-eagerly creating the jpaTm, because
there's no reason the Registries cache (which is Datastore-only) should ever
need it, but because this cache is using the Datastore transaction manager, the
other one was being created too.
* Start using JUnit 5
This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration
In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.
* Delete no longer needed lockfiles
* Merge branch 'master' into first-junit5
* Add dual write for Registrar
* Use @AlsoLoad to set street field for Cloud SQL
* Change email body to use the new streetLine field
* Refactored the logic to handle street fields
* Simplify postLoad function
* Address comments
* Add a TODO to remove street
* Add test for onLoad and postLoad
* Rebase on master
* Add the schema and DAO for Locks in CloudSQL
* Addresses some comments
* Change number on flyway file
* Small changes
* More small changes
* Use checkArgumentNotNull instead of checkNotNull
* Address comments
* fix javadocs
* update persistence
* Use JSON API for Maven Repo on GCS
The url pattern https://storage.googleapis.com/{Bucket}/{Path}
uses the legacy XML API, which seems to be less robust than
the JSON API. We have observed connection resets after a few
thousand-file download bursts over 30 minutes.
This PR changes all urls to registry's Maven repo on GCS to
gcs://{Bucket}/{Path}. Gradle uses the JSON API for such urls.
TESTED=In Cloud Build with local change
* Use dependency cache in all Gradle tasks in GCB
Make the initial test and the final publishing steps use the shared
dependency cache.
Also make the initial test use the registry's own maven repo instead
of Maven Central.
* Make Gradle dependency cache shareable in GCB
Make Gradle put its caches in the source tree so that
they can be preserved across steps. When left at their
default location, caches are lost after each step.
* Use CA cert in tests
Our tests were using a self-signed cert created by a netty utility.
That cert is not CA, and cannot be used with newer JDK versions
(starting from u242 with Ubuntu openjdk).
Now we generate a unsafe cert for tests. The new cert,
SelfSignedCaCertificate, needs to be made test-only.
To do so we need to refactor both prober and proxy.
* Always save reserved lists to Cloud SQL
Otherwise it's too easy to forget to run this additional option, and then things will
fall out of sync and start throwing errors.
* Use randomly-generated strings for RegistryLock verification codes
We were using UUIDs before which are also fine, but unnecessarily long.
The RegistryLock class itself does not enforce any particular format for
the lock verification codes.
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
* Add a RegistryLockPostAction and tests
* Response to CR and more robust email content test
* Fix typo
* Run save + email in a JPA transaction
* Use action in subject
* Allow non-CA cert as trust anchor in tests
Stay with the legacy behavior to unblock release build.
We will update the test cert generation code later.
TESTED=reproduced and fixed test failures in builder container
* Work around Spinnaker issue wrt variables
Cloud Build variable reference need to stay from the ${var} pattern
to prevent Spinnaker from trying to resolve it. In all files that
are used by Spinnaker, we change variable reference to the $var form.
We made the minimum amount of change possible, and will review this
issue after the permanent solution is available.
* Add RegistryLockVerifyAction
The action takes two parameters
- isLock is a boolean, determining whether we're trying to lock or
unlock a domain
- lockVerificationCode is the UUID by which we'll look up the lock
object in question.
The lock in question must not be expired and must be in a valid lockable
/ unlockable state
* Some responses to CR
* Add slash and move test method
* Add more data and tests
* Fix screenshot
* Add JPA annotations to class Registrar
* Use array for Java list
* Exclude parent field
* Use 3 columns for address and use text for enum
* Use EnumParameter and 3 properties in Address
* Rename columns and rebase on HEAD
* Get persistence of DomainBase actually working
Fix all of the existing problems with DomainBase persistence:
- Remove "final" keywords on getters that cause errors during startup.
- Remove Transient from creationTime (since there's a converter for
CreateAutoTimestamp)
- Fix DesignatedContext persistence so that it only creates a single table.
This is a lot more efficient given that these are many-to-one with their
domains.
- Add a flyway script, update the golden schema.
- Create a unit test, add it to the integration test suite.
* Changes request in review
* Regenerated generated schema file.
* Changes for review
* Persist status value enum set
* Changes in response to review
* Changes requested in review
* Fixes for #456
* Rename Domain "status" column to "statuses"
* Require tests on all declared JPA entities
The JPA entity classes declared in persistence.xml reflects
JPA's view of the database. The view must be compatible, though
not always consistent with the actual schema. By defining and
using a new JpaIntegrationWithCoverageRule, this change verifies
that all JPA entities in persistence.xml are included in some
tests in the SqlIntegrationTestSuite.
Also removed one test from SqlIntegrationTestSuite since it
does not use Cloud SQL.
Lastly, global resources such as clock should be provided to
a test fixture. Having one fixture creating its own makes it
harder to use in a multi-system integration test. This change
refactored JpaTransactionManagerRule to take a user provided
clock.
* Change reserved messages for domains w/ required allocation token
In domain check reasons for unavailable domains, we were previously saying that
domains were "Reserved" regardless of whether the domain was fully blocked or
reserved for an anchor tenant or other specific use (e.g. the .new Limited
Release Program). This commit changes the message for the latter situations to
be "Allocation token required", so that registrars have a hint that domain will
show as available if the correct allocation token is supplied to the domain
check command using the allocation token extension.
* Allow claims keys on domain creates during quiet periods
Since we're using TldState.QUIET_PERIOD for the .new LRP (in which only reserved
domains with allocation tokens may be registered), we also need to support
claims keys during this phase. Otherwise, domains that appear in MarksDB will
not be registrable during LRP, even if the correct claims key is provided.
This is based on an error report from a registrar, in which a correct-looking
domain create was failing with the error message "Declared launch extension
phase does not match the current registry phase".
* Add test for checking reserved domain with token
* Allow domain checks with a valid token if the domain isn't fully blocked
* Check specifically for specific-use or anchor reservations
* Add test for LRP in-time token
* Refactor lock/unlock commands to use Registry Locks
* CR responses
* Remove unnecessary XML test files
* Add tests
* Respond to CR
* Refactor further the creation and verification of locks
* remove isUnlocked
* Responses to CR
* Fix tests
* Add admin-override back to unlocking
* Add TODO
* Fix imports
* Refactor g.r.model.transaction -> g.r.persistence.transaction
The ultimate goal here is to eventually delete the model package entirely once
we're no longer using Datastore. To that end, the Cloud SQL stuff that we'll be
keeping needs to be in a different package. Here's the rough outline of the
final desired state I'm thinking of:
g.r.persistence
.transaction -- what's being moved in this commit
.converters -- to be moved in another commit
[maybe more?]
g.r.schema
[same-ish subpackages as model currently contains]
I'm also trying to think if it makes sense to move the DAO classes out of
g.r.schema, and haven't come up with a good answer yet.
* Also move test resource files
* Merge branch 'master' into mv-tx-to-persistence
* Check for post-deployment Flyway script changes
Add a script that checks for changes to scripts that
have been deployed to sandbox. This is test in to
be invoked in presubmit and ci.
Extracted common shell functions to an 'rc' file.
Also renamed existing script to be consistent with
other shell scripts.
* Implement a hibernate type for enums
Leverage GenericCollectionUserType to store enum fields as an array of
strings.
* Changes requested in review
* Remove extraneous import
Curiously, this causes checkstyle to fail now that the import was /removed
from a comment./
* Fix exDate calculation in domain transfer flows
* Link ICANN rule for zero-period transfer
* Get rid of some unnecessary method parameters
* Add domain info commands to test case
* Fix flaky SQL integration tests
As a sanity check, the text fixture verifies that all test jdbc connections
have been closed. However, the query to get the connection count is too
general, and may count background admin tasks such as autovacuum and
replication etc, that may appear at any time.
* Upgradle JUnit to 4.13
Removed third_party/junit folder and all usage of the
JunitBackPort class. As a result, third_party is no
longer a Gradle subproject.
Minor code changes were needed to work around an
error-prone pattern: multiple statement in assertThrows'
runnable lambda.
Also third_party/activation and third_party/jsch. These
dependencies are loaded from remote maven repo. The local
copies are not in use.
* Make Jacoco work with multiple test tasks
By default Jacoco only looks at execution data from
the 'test' task. This is a problem to 'core' which
has multiple test sets.
A helper script to run server/schema compatibility tests from
a CI platform. It allows the caller to choose a system under
test (sql or nomulus) and to specify the deployed version
by environment (sandbox or production).
* Implement changes suggested by testcontainer dev
See https://github.com/google/nomulus/issues/401
Specifically:
- Use getContainerIpAddress() instead of localhost to insulate us from
off-machine docker usage.
- Remove shutdown hook to close the container, as testcontainers does this for
us.
* Relax premium list existence check to allow Cloud SQL migration
We need to be able to simultaneously update premium lists that already exist in
Datastore and create them in Cloud SQL (because they haven't been migrated over
yet). This temporarily relaxes the existence check for Cloud SQL so that
"updates" will work even when the list doesn't yet exist there.
* Fix IcannReportingUploadAction to upload reports from the previous month
getFileName now sets the file name of the report to upload to use the month before cursor time.
IcannReportingUploadAction no longer uploads the MANIFEST.txt file since it is not required based on (https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-07#page-9) and the previous implementation of this action did not upload it.
Deletes the ICANN_UPLOAD_MANIFEST cursor since it is no loner needed.
* Add the ICANN_UPLOAD_MANIFEST cursor back
* Fix flakiness in JodaMoneyConverterTest
JpdaMoneyConverterTest relies on Hibernate to deploy its schema.
This introduces an extra jdbc connection in the middle of a test
suite, and may break the connection count checks between tests made
by JpaTransactionManagerRule.
This change updated HibernateSchemaExporter to include Hibernate
proprietary mappings in META-INF/orm.xml.
This change also disabled Hibernate schema push for all tests,
and enabled sql statement logging.
* Support premium list updating in Cloud SQL
This also removes the requirement to specify --also_cloud_sql in nomulus premium
list tooling, instead always persisting to Cloud SQL. It removes a non-USD
premium label in the global test premium list (the Cloud SQL schema doesn't
support a mix of currency units in a single premium list). And it adds a method
to PremiumListDao to grab the most recent version of a given list.
* Merge branch 'master' into premium-lists-always-cloud-sql
* Revert test change
* Create new PremiumListUtils class and refactor out existing method
* Fix tests and update an existing premium price
* Add unlock fields to RegistryLocks
This will make it easier to reason around inter-connected registry lock
objects (like when we add dependent roids). It will make it easier to
answer the question of "Have all locks associated with this host/contact
roid been unlocked?", as well as the question of "Was the last lock
object associated with this domain unlocked?"
* Responses to CR
* Make the DAO API more specific
* whoops, undo rename
* Consolidate certificate supplier module
Both the proxy and the proxy needs certificate suppliers. The PR
consolidates the module that providings those bindings to a shared
module and switched the proxy to use that module. The prober currently
uses P12 file to store its certificates. I am debating keeping that
supplier ro converting them to PEM files for simplicity.
* Rename mode enum values to be more descriptive
* Update annotation names to be more descriptive
* Separate JPA rules for unit and integration tests
Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.
* Separate JPA rules for unit and integration tests
Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.
* Add initial support to write reserved list to Cloud SQL
* Add support to update reserved list in Cloud SQL
* Fix wrong check when override is enabled in create command
* Add sql dependent tests to the suite
* Address comment
* Make invocation of super.execute more readable
* Add test to check upper case and non-puny code label
* Move ReservedListDao related classes to schema package
* Simplify a function name
The promise should be set outside the try block because if we want
warning only, we still want the promise to be set even if the
clientCertificate.checkValidity() throws an error.
* Remove the schema jar url override for tests
Revert pull request 374: "Allow schema-loading from arbitrary url in tests".
Pull request 403 (f48e3933f5) is a more
general solution.
This reverts commit 68887d427f.
* Create a new app to hold GenerateSqlSchemaCommand
GenerateSqlSchemaCommand starts postgresql using testcontainer.
This makes junit etc a runtime dependency, allowing them to get
into release artifacts.
By moving this command to a separate tool, we can remove junit
etc as compile/runtime dependency.
* Fix sql script name conflict
There are two V11__ files due to concurrent merge. Renamed one
to V12__
Also removed a @NotNull annotation, which is the fist in the code base.
Most of the code base use @Nullable instead. If we do want to use
@NotNull, we may want to use the javax one instead.
* Run cross-release SQL integration tests
Run SQL integration tests across arbitrary schema and server
releases.
Refer to integration/README.md in this change for more information.
TESTED=Cloud build changes tested with cloud-build-local
Used the published jars to test sqlIntegration task locally.
* Add Cloud SQL premium list caches and compare prices with Datastore
Nothing will fail if the prices can't be loaded from Cloud SQL, or if the prices
are different. All that happens is that the error is logged. Then, once this is
running in production for awhile, we'll look at the logs and see if there will
be any pricing implications from switching over to the Cloud SQL version of the
premium lists.
* Add setMaxResults(1) per code review
* Add tests and reorder public functions
* Don't statically import caches
* Improve test pass rate
* Merge branch 'master' into dual-read-premium
* Add PremiumEntry mapping
* Allow update
* Revert column order
* Alphabetize PremiumEntry columns
* Don't bother trying to enforce order
* Private constructor
* Make devProject a project property
Properties set in rootProject's ext block are not overridden by
commandline flags.
* Make flyway commands runnable from FOSS repo
rootProject.devProject, defined in projects.gradle, cannot be
overridden by commandline flags. Added a flag check in :db's
project script.
* Add schema for Cursor
* Add CursorDao and CursorDaoTest
* Fix comment on getTld
* Change tld column to scope
* Fix cursorTime to be converted to DateTime internally and other small fixes
* Add a CursorType enum and a createGlobal constructor for Cursor
* Rename flyway file
* Use cursorType from common/Cursor.java and add null checks
* Verify RegistryTool can instantiate
Add a task that instantiates all command classes in RegistryTool
with runtimeClasspath.
Also make sure that runtimeClasspath is a superset of
compileClasspath.
* Adding junit back into the runtime classpath
Unfortunately, GenerateSqlSchemaCommand depends on junit via testcontainers.
We should really move GenerateSqlSchemaCommand out of nomulus tool (we only
use it during development) but this gets nomulus tool working for the time
being.
* Removed unnnecessary trace line.
* lockfiles generated after update_dependency.sh
In some cases, we define JpaTransactionManagerRule in a TestCase
class which is extended by the concrete test class. So, we need
to check if JpaTransactionManagerRule is also defined in the super
class.
Endpoints annotated with AUTH_INTERNAL_ONLY used to be accessible
manually with an internal RPC tool that adds App Engine specific HTTP
headers to a request to make it look like it comes from App Engine
(hence internal). This tool is used by admins to hit such endpoints
during debugging, making them effectively AUTH_INTERNAL_OR_ADMIN.
This RPC tool has never been made available outside Google so the open
source admins do not have such ability. A recent change in the RPC tool
made this hack stop working internally as well. This PR replaces all
all occurances of AUTH_INTERNAL_ONLY with AUTH_INTERNAL_OR_ADMIN and
brings the open source build into feature parity with the internal
version.
Also fixed a few issues the router tests.
* Allow project dependency to use runtimeClasspath
Project dependency should use runtimeClasspath. However, if
left unspecified, it uses 'default', which is the same as
the legacy 'runtime' configuration. (runtimeOnly dependencies
are left out).
Since runtimeClasspath cannot be referenced directly, we use
a custom config (deploy_jar) as a proxy.
By excluding testjars (leaked into 'compile' by third-party
dependencies) from runtimeClasspath, we prevent them from
getting into release artifacts.
Two meaningful changes in appengine_war.gradle and java_common.gradle
TESTED=Diffed contents of services/{module}/build/exploded-*
Only three jars are removed: hamcrest-core, junit, and
mockito-core.
Client SSL handler already performs the necessary validation. Only tests are
added.
Server SSL handler does not currently check for the validity period of
the client certificate as the insecure trust manager is used. This PR
added the check but does not actually terminate the connection yet. It
will log the expired certificates so that we can contact the registrars
to update them.
Once we are certain that all certificates are updated, we can turn off
dryrun mode.
We should also consider checking if the certificate has too long a
validity period as it defeats the purpose of using regularly updated
certificates to deprecate insecure cipher suites.
* Stop publish Cloud SQL schema jar to maven repo
The original purpose of the maven publication is for
use in server/schema compatibility tests. A commandline
flag can direct a test run to use different versions of
the schema jar. However, this won't work due to dependency
locking.
The prober now uses the common SSL initializer in the networking
subproject.
Also changed both initializers to take an ImmutableList of certificates
other than an array of those, for better immutability.
I have no idea where these lockfile changes are coming from. They seem
to be pure noise as far as code review is concerned.
* Add a cursor for tracking monthly uploads of the transaction report to ICANN
* Add cursors to track activity, transaction, and manifest report uploads.
* Address comments
* Add @Nullable annotation to manifestCursor
* Add lock and batch load cursors.
* Add string formatting, autovalue CursorInfo object, and handling for null cursors
* Add some helper functions for loadCursors and restructure to require less round trips to the database
* Switch new cursors to be created with cursorTime at first of next month
* Make dev project configurable
We should not hardcode our dev project in the public config file.
* Remove the use of .ext when using external properties
They are only needed when defining properties.
* Update the Registries cache to leverage/populate the Registry cache
This is accomplished by also providing a loadAll() method on the Registry cache
that can be used to load an entire batch of Registry objects at once.
This improves efficiency, because now, any operation on Registries that loads
all the Registry entities (getTlds(), getTldsOfType(), and getTldEntities()), is
plumbed through the Registry cache, therefore loading it from that cache if it
exists and only hitting the DB if not. If not, this populates the Registry cache
upon loading, so that subsequent calls to Registry.get() will now hit the cache
instead of the DB.
To give a concrete example, the following code:
for (String tld : Registries.getTlds()) {
// ...
doSomethingWith(Registry.get(tld));
// ...
}
is now much more efficient, because the initial call to Registries.getTlds()
populates all the entities in cache, and the subsequent individual calls to
Registry.get(tld) now retrieve them from the cache. Prior to this change,
Registries.getTlds() did not populate the Registry cache, and each subsequent
Registry.get(tld) had the potential to trigger an individual round-trip to the
DB, which is obviously bad for performance.
* Break circular dependency between core and util
Created a new :common project and moved a minimum
number of classes to break the circular dependency
between the two projects. This gets rid of the
gradle lint dependency warnings.
Also separated api classes and testing helpers into
separate source sets in :common so that testing
classes may be restricted to test configurations.
* Allow schema-loading from arbitrary url in tests
Server/Schema compatibility tests must be able to load different versions
of the SQL schema. This change allows test runners to override the
schema location using a system property.
Note: due to dependency-locking, we cannot manipulate the dependencies
closure in the build script to load different schema jars. The jars
must not be on the classpath.
* Add a new method to load all Registry entities of a given type
This is useful for things that need to know more than just the TLD strings
themselves, which is all Registries currently provides.
* Require explict tag when starting psql docker
Defined a util class to return docker tag of desired PSQL version.
Class is defined in ':db' and shared by ':db' and ':core'. Used
an artifact declaration to exclude unnecesary compile dependencies.
Added a presubmit check for instantiations without explicit tag.
* Update schema deployment doc and flyway tool
Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.
Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).
Updated README file on schema push process. Also reformatted
the entire file.
* Add by-repo-id method to RegistryLockDao
When we create new registry lock objects on user request, we will want
to make sure that there are no pending lock/unlock actions on this
domain. In order to do that, we will need to know what lock actions
there have been for this domain.
* Inline the query into a single statement
* whoops
* comments
Replace deprecated bouncycastle class in SslInitializerTestUils.
Generic array as vargs: worked around it in ProbingAction and
removed unused method in CircularList.
Defined Docker image for schema deployment.
Included schema deploymer docker in the Cloud Build release process.
Defined Cloud Build config for schema deployment.
TESTED=Used cloud-build-local to test deployment flow
TESTED=Used docker to test schema deployer image in more ways
Implementation of CircularList is an overkill for its stated
purpose. Remove an unnecessary usage so that the remaining
use case may be refactored in isolation.
* Refactor UI actions to be more reliable
- Created HtmlAction to handle the nitty-gritty of login and setting up
HTML pages
- Created a test to verify that all UI actions implement JSON GET, JSON
POST, or HTML classes
- Move CSS renaming into a utility class
* Move logging of request into HtmlAction
* Comment and wording in exception
* mention JsonGetAction in the comment
* JsonGetAction extends Runnable
* Run both the AppEngineRule and JpaTransactionManagerRule in tests
As previously written, the AppEngineRule wraps the infinite loop of the
server handling requests. This changes the code so that we'll have a
chain that does both the AppEngine management and the JPA transaction
manager management so we can actually use SQL in tests and in the test
server.
* remove log line
We should communicate to the users why this command failed, that they
are not allowed to use discounted allocation tokens on premium domains.
Currently it still fails, but we don't yet tell them why.
* Add a GET action and tests for registry lock retrieval
* Create isVerified method
* Allow lock access for admins even if they're not enabled on the registrar
* Simple CR responses
* Move locks retrieval to the GET action
* add newline at eof
* Switch to using ID
* Improve FilteringTest so it works in all cases
- Disable failOnMatchingTests so we don't fail if we filter out all of the
tests for any given task.
- Add excludeTestCases flag so that we can turn this behavior off for test
tasks that need to run TestCas/TestSuite classes.
- Add includeAllTests() function for test tasks where we don't explicitly
include our required tests.
This makes all test tasks in core FilteringTest, with the exception of the
default test task which now does nothing (it just depends on the other test
tasks).
* Improve wording of excludeTestCases comment
When we add the extra test entity to the current JpaTransactionManagerRule by calling jpaRule.withEntityClass(TestEntity.class) and jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update"), the rule would override the golden database scheme with the schema derived from the current entity class(This is an expected behavior by enabling HBM2DDL_AUTO). This behavior prevents us from detecting breaking changes in ORM classes.
This PR fixed the issue by adding HibernateSchemaExporter to export the DDL script for given extra entity class, and make JpaTransactionManagerRule execute the DDL script to create the corresponding table for the extra entity when initializing the database. This PR also simplified the code when adding extra entity class for testing. For now, you don't need to(and shouldn't) call jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update").
* Move soyutils_usegoog.js out of node_modules
Everytime the npmInstall runs, it removes this file from node_modules.
Move it outside the folder to prevent this from happening.
* Move karma.conf.js and soyutils_usegooge.js
* Move karma.conf.js to be under core
* Release SQL schema in Cloud Build
Tentatively release SQL schema at the same time as the server release.
Publish schema jar to gs://domain-registry-maven-repository/nomulus
and also upload it with server artifacts.
Also removed the Gradle 'version' variable which is not used.
Tested=On cloud-build with a simplified version of
cloudbuild-nomulus.yaml.
* Don't wrap exceptions thrown inside Cloud SQL transactions
There's no reason to wrap them in PersistenceException, and it makes dealing
with thrown exceptions harder as seen in the diffs on test classes in this
commit.
* Add a converter for CurrencyUnits stored in the database
This uses the well-known String representation for currency units. It also
provides a base class for other converters that will be persisting the
toString() representation.
* Add DB and formatting changes
* Add tests, make minor fixes
* Save release tag during deployment
* Save current tag for every environment
Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
* Save current tag for every environment
Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
* Instrument proxy frontend latency metric
Backend latency only captures the time spent waiting for the GAE backend
to respond to a request, which is not representitive of what clients
experience. For instance it does not take into account the time the
proxy spends on processing the requests and the time it takes to send a
response to the client.
This PR adds a metric for frontend latency, which is the time spent on
answering a client request. This should serve as a better proxy for the
latency clients observe.
* Rename AbstractMetrics to BaseMetrics
"Public Domain" license must be reviewed case by case.
Removed blanket acceptance and named accepted dependencies
individually.
Also added a README file to warn about this license and WTFPL.
* Don't retry permanent failures when uploading ICANN monthly reports
There are two kinds of permanent failures that this checks for that we know will
never succeed, so it makes no sense to continue retrying 11 more times before
moving onto the next file to upload. These errors are:
1.
com.google.api.client.http.HttpResponseException: 403
Your IP address xx.xx.xx.xx is not allowed to connect
2.
com.google.api.client.http.HttpResponseException: 400
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><response xmlns="urn:ietf:params:xml:ns:iirdea-1.0"><result code="2002"><msg>A report for that month already exists, the cut-off date already passed.</msg><description>Date: 2019-09</description></result></response>
In order to implement this new functionality, this commit also adds a new way to
call Retriable that allows specifying the isRetryable Predicate (which is quite
useful).
* Reenable JpaTransactionManager for Alpha and Crash
* Make UploadClaimsListCommand implement CommandWithCloudSql
* Fix wrong call to get password
* Use Cloud SQL Socket library to provision TransactionManager
* Change to use dataSource configs
* Add another test contact for Registry Lock testing
Previously, we only had two contacts -- one per registrar. This PR adds
a second, registry-lock-enabled, contact to one registrar for two
reasons:
1. For registry-lock-related testing, we'd like to be able to test both
positively and negatively, making sure that the permissions work the way
they should
2. In general, the UI tests should include the case where we have
multiple contacts in the same registrar. Previously, this was never the
case in tests.
* Merge remote-tracking branch 'origin/master' into addTestContact
* Don't destroy existing registry lock passwords in contacts
The existing code assumes that the "contacts" segment of the form
contains an exact representation of the registrar contacts. This breaks
when we have a contact with an existing registry lock password because
we don't want to keep passing around that password in plain text (we
never store it in plain text)
This PR changes the code so that instead of assuming the contact is
provided in its entirety, we load the contact from storage first
(matching by email address) if it exists. We then set the required
fields from the JSON object, and set the password optionally if it was
provided.
Alternatives:
- Create a separate RegistrarContactPassword object with a
RegistrarContact parent. This increases complexity significantly since
we'd be adding a parent-child relationship and adding more objects to
Datastore during the transition to SQL. It also doesn't completely solve
the problem of "When should we set the password?" because the password
field still must be part of the same form.
- Rearrange the UI so that the password is set as part of a completely
separate form with a separate submit action. This would be possible but
is sub-optimal for two reasons. First, we are trying to not re-engineer
the web console as much as possible since we're likely starting it from
scratch before too long anyway. Second, we want the
lock-password-setting to be part of the standard contact modification
workflow.
* Responses to CR
* Actually we need to allow "removal" of fields
* Remove optional
* one-statement building the contacts
We don't want to override toDiffableFieldMap because (per the javadoc)
that is supposed to contain sensitive information. So, we should just
remove it before sending it out.
* Use base64-encoded SQL credentials
Encode Cloud SQL credential files on gcs with base64,
to be consistent with our Cloud Build practices.
Also renamed a property that specifies where to publish
the schema jar. New name is schema_publish_repo.
The segregated test targets in core break the --tests filter. Fix this by
defining a "testFilter" property and creating the FilteringTest task type that
applies it to the property set by "--tests".
* Make Flyway schema task work with prod and sandbox
Also renamed the 'superuser' role to 'admin' since
we do not own super user in Cloud SQL.
* Allow pushing schema to all env with Flyway
Desktop schema push to production is needed in the short term.
Long-termly we need to decide if this should be kept for glass
breaking
Schema push to sandbox and production requires interactiveconfirmation.
Also fixed a typo in initialize_roles.sql.
* Add Bloom filters to the Cloud SQL PremiumList schema
They are slightly different from the existing Bloom filters stored in Datastore
in that they now use an ASCII String encoding rather than the more generic
CharSequence, and there is no maximum size (whereas we previously had to live
within the 1 MB max entity size for Datastore).
* Modify Cloud SQL user management scripts
Create readonly and readwrite roles that may be granted to users.
Also configured default privileges for tables created in the future.
Made sure arbitrary users may not create database or tables.
* Modify Cloud SQL user management scripts
Create readonly and readwrite roles that may be granted to users.
Also configured default privileges for tables created in the future.
Made sure arbitrary users may not create database or tables.
* Load persistence.xml classes before adding test entities
* Also use persistence.xml in GenerateSqlSchemaCommand
* Add exception message
* remove duplicate line
* Add initial support for persisting premium lists to Cloud SQL
This adds support to the `nomulus create_premium_list` command only; support for
`nomulus update_premium_list` will be in a subsequent PR.
The design goals for this PR were:
1. Do not change the existing codepaths for premium lists at all, especially not
on the read path.
2. Write premium lists to Cloud SQL only if requested (i.e. not by default), and
write to Datastore first so as to not be blocked by errors with Cloud SQL.
3. Reuse existing codepaths to the maximum possible extent (e.g. don't yet
re-implement premium list parsing; take advantage of the existing logic), but
also ...
4. Some duplication is OK, since the existing Datastore path will be deleted
once this migration is complete, leaving only the codepaths for Cloud SQL.
* Refactor out common logic
* Add DAO test
* Add tests for parsing premium lists
* Use containsExactly
* Code review changes
* Format
* Re-generate schema
* Fix column names
* Make some tests pass
* Add SQL migration scripts
* Fix test errors
* Add an explanation to dummied-out JPA init
Add a more elaborate explanation of why actual JpaTransactionManager
initialization was removed from the factory.
* Add a DAO for RegistryLock objects
* Add an index on verification code and remove old file
* Move to v4
* Use camelCase in index names
* Javadoc fixes
* Allow alteration of RegistryLock objects in-place
* save, load-modify, read in separate transactions
* Change the creation timestamp to be a CreateAutoTimestamp
* Add persistence.xml to the war files
* Always use the DummyJpaTransactionManager
Use the DJTM until we get all of the dependencies set up for all of the
environments.
This shouldn't affect any of the unit tests, these use the
JpaTransactionManagerRule to set up a local database and connection.
This fixes the App Engine build.
* Implement CreateAutoTimestampConverter
Implement a JPA-based converter for CreateAutoTimestamp, allowing us to
persist instances of this class.
Note that converters appear to be required to convert to and from database
types that are generally known to JDBC. For example, conversion to Timestamp
works, conversion to OffsetDateTime does not (even though this works through
the JDBC interface directly).
* Improve error handling for environment property
Improve the error messages that we get for a bad or missing environment
property. Move the property processing into the main build so that we do it
only once and configure all of the appengine deployment tasks to check that a
project has been defined and print a friendly error if it hasn't.
Note that even if the check isn't configured, this change prevents deployment
because the gcpProject will be set to null. It just won't print as useful an
error message.
Tested: ran appengineDeployAll with and without the environment property, ran
"build" to verify that we don't get any complaints for non-deployment targets.
* Changes for review.
* Changes for review
* Changes for review.
* Changes for review.
* Give JpaTransactionManagerRule more parameters
Allow users of the rule to add annotated classes and properties, both useful
for testing.
* Change in response to review.
* Changes for review.
* Move test EntityManagerFactory create method
Move the test create method into the JpaTransactionManagerRuleTest.
* Remove nomulus SQL dialect from G.S.S.Command
Remove NomulusPostgreSQLDialect from GenerateSqlSchemaCommand (it has been
moved to its own top-level class).
* Add maven-publish task for SQL schema jar
Add task to publish SQL schema jar with flyway scripts and
golden schema to a maven repo. This will be used
for pre-release testing in the future.
This task is not part of build and needs to be invoked explicitly.
User needs to provide schema_jar_repo and schema_version
properties.
* Merge branch 'master' of https://github.com/google/nomulus into publish-schema-jar
* Add maven-publish task for SQL schema jar
Add task to publish SQL schema jar with flyway scripts and
golden schema to a maven repo. This will be used
for pre-release testing in the future.
This task is not part of build and needs to be invoked explicitly.
User needs to provide schema_jar_repo and schema_version
properties.
Export of cl/270900150.
To refer to a KMS key or key ring, we should use the stable `.self_link`. Using `.id` instead provides an unstable identifier which may change (and it will change in the upcoming update of the google provider to 2.9.1). A change in the identifier will cause Terraform to destroy and recreate the key. Destroying the key means all data associated with it is lost; the key cannot be recreated.
This CL replaces `.id` with `.self_link`, so all of those problems will not happen. In addition, `prevent_destroy` protects the key against delete-and-recreate in general.
* Upgrade to Truth 1.0
Refactored fail(...) to assertWithMessage().fail().
Upgraded com.google.monitoring-client family of dependencies to 1.0.6
Also fixed bad use of io.StringIO (on binary buffer) recently introduced to
google-java-format-diff.py.
For some reason tools.jar (which we don't need in the runtime in the first
place) gets exploded readonly, causing subsequent builds to fail since they
can't overwrite it. This hack makes it writable at the end of explodeWar.
* Fix dependency-locking config
Reenable dependency locking after a bug errorneouly turned it off.
Removed the guava-related workaround that forcefully resolve to
the -jre distribution.
Enabled locking for buildSrc by updating its property file.
Updated all lock files.
* Output command test output as well as consuming it
CommandTestCase currently consumes stdout & stderr for the command being
tested. Unfortunately, this results in us not being able to see the command
output. Add an output splitter so that output gets written to the original
stream in addition to being captured.
A simpler approach would be to print the captured data after command
completion. However, this won't work for tests that become hung and also
won't display results in real-time.
Tested: Ran a command test with verboseTestOutput=true, verified that standard
output was visible.
* Save and restore original stdout/err in cmd tests
We have to restore the original stdout/stderr print streams otherwise we end
up nesting them across tests which eventually causes the RDE tests to OOM.
* Add RegistryLock schema to Flyway deployment folder
Added creation script of RegistryLock to Flyway deployment folder.
Fixed previous scripts (PremiumList- and ClaimsList-related) for
FK name change (cause by table name changes: names are quoted now).
We should consider generating foreign key names by ourselves.
Since the alpha database is empty, we dropped and recreated the schema.
Added instructions on how to submit new database incremental changes
in the README file.
Updated RegistryLock.java, removing unnecessary annotations:
- For most fields, the 'name=' property is no longer necessary not that
the naming strategy is in place. The exceptions are the two used in
the unique index.
- The @Column annotation is implicit.
* Force Gradle to use jre version of Guava
Guava is often specified by version only by other
dependencies. In such cases, Gradle usually resolves
it to the '-android' version, which lacks the collection
classes.
We use custom resolution strategy to force the selection
of the '-jre' version.
* Add schema deployment tests
Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.
Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.
Upgraded testcontainers to 1.12.1.
Added a custom Truth subject for better diffing of multi-line
text blocks.
Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
* Add schema deployment tests
Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.
Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.
Upgraded testcontainers to 1.12.1.
Added a custom Truth subject for better diffing of multi-line
text blocks.
Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
* Add schema deployment tests
Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.
Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.
Upgraded testcontainers to 1.12.1.
Added a custom Truth subject for better diffing of multi-line
text blocks.
Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
* Regenerate schema using tools command
Rerun GenerateSqlSchemaCommand to pick up RegistryLock
and naming strategy change.
Also updated a new license term which seems to just pop up.
* Add RegistryLock SQL schema
* Refactor a bit
* Move registrylock -> domain
* Clearing up lock workflow
* Add more docs and remove LockStatus
* Responses to CR
* Add repoId javadoc
* Add registry lock to persistence xml file
* Quote rather than backtick
* Remove unnecessary check
* File TODO
* Remove uniqueness constraint on verification code
* Remove import
* add index
* Add to SQL generation task
* Move fields around to be the same order as Hibernate's generated sql
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Corrected the type of ClaimsEntry's revision_id column.
It should be plain int8, not bigserial.
Make GenerateSqlSchemaCommand use a custom dialect that
converts all varchar type to 'text' and timestamp to
'timestamptz'.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Re-add other schema classes
* Add Cloud SQL schema for premium lists
This won't work quite yet, pending a solution for the type translator issue
(which will be needed for the currency field, and potentially others).
* Generate basic schema for all of DomainBase
Generate a basic schema for DomainBase and everything that is part of it.
This still isn't complete, in particular it lacks:
- Correct conversions for problematic types (e.g. DateTime, Key...)
- Schema generation for history records.
- Name translation.
* Make parameter names in generate_sql_schema command consistent
The rest of the nomulus commands use underscores for delimiting words in
parameter names, so this should too.
Also fixed capitalization of some proper nouns.
* Move EntityManagerFactoryProviderTest to fragile
* Add EMF Provider Test to docker tests
Add EntityManagerFactoryProviderTest to the docker incompatible test patterns
and use the latter list to compose the fragile tests.
* Start postgresql container in generate_sql_schema
Add a --start-postgresql option to the nomulus generate_sql_schema command so
that users don't have to start their own docker container to run it.
* Made default behavior be to give guidance
* Don't write TX records for domains deleted in autorenew grace period
When the project was originally being designed, we envisioned have a purely
point-in-time architecture that would allow the system to run indefinitely
without requiring any background batch jobs. That is, you could create a domain,
and 10 years later you could infer every autorenewal billing event that should
have happened during those 10 years, without ever having to run any code that
would go through and retroactively create those events as they happened.
This ended up being very complicated, especially when it came to generating
invoices, so we gave up on it and instead wrote the
ExpandRecurringBillingEventsAction mapreduce, which would run as a cronjob and
periodically expand the recurring billing information into actual one-time
billing events. This made the invoicing scripts MUCH less complicated since they
only had to tabulate one-time billing events that had actually occurred over the
past month, rather than perform complicated logic to infer every one-time event
over an arbitrarily long period.
I bring this up because this architectural legacy explains why billing events
are more complicated than could otherwise be explained from current
requirements. This is why, for instance, when a domain is deleted during the 45
day autorenewal period, the ExpandRecurringBillingEventsAction will still write
out a history entry (and corresponding billing events) on the 45th day, because
it needs to be offset by the cancellation billing event for the autorenew grace
period that was already written out synchronously as part of the delete flow.
This no longer really makes sense, and it would be simpler to just not write out
these phantom history entries and billing events at all, but it would be a
larger modification to fix this, so I'm not touching it here.
Instead, what I have done is to simply not write out the DomainTransactionRecord
in the mapreduce if the recurring billing event has already been canceled
(i.e. because the domain was deleted or transferred). This seems inconsistent
but actually does make sense, because domain transaction records are never
written out speculatively (unlike history entries and billing events); they
correspond only to actions that have actually happen. This is because they were
architected much more recently than billing events, and don't use the
point-in-time hierarchy.
So, here's a full accounting of how DomainTransactionRecords work as of this commit:
1. When a domain is created, one is written out.
2. When a domain is explicitly renewed, one is written out.
3. When a domain is autorenewed, one is written out at the end of the grace period.
4. When a domain is deleted (in all cases), a record is written out recording the
deletion.
5. When a domain is deleted in the add grace period, an offsetting record is
written out with a negative number of years, in addition to the deletion record.
6. When a domain is deleted in the renewal grace period, an offsetting record is
likely written out in addition.
7. When a domain is deleted in the autorenew grace period, there is no record that
needs to be offset because no code ran at the exact time of the autorenew, so
NO additional record should be written out by the expand mapreduce.
*THIS IS CHANGED AS OF THIS COMMIT*.
8. When a domain is transferred, all existing grace periods are cancelled and
corresponding cancelling records are written out. Note that transfers include a
mandatory, irrevocable 1 year renewal.
9. In the rare event that a domain is restored, all recurring events are
re-created, and there is a 1 year mandatory renewal as part of the restore with
corresponding record written out.
So, in summary, billing events and history entries are often written out
speculatively, and can subsequently be canceled, but the same is not true of
domain transaction records. Domain transaction records are only written out as
part of a corresponding action (which for autorenewals is the expand recurring
cronjob).
* rm unused import
* Remove the "showAllOutput" property from the build
It doesn't work very well and has been superseded by "verboseTestOutput",
which does the same thing and more.
* Remove 'value' from RDAP link responses
* Change application type to rdap+json
* Merge remote-tracking branch 'origin/master' into removeValueRdap
* CR response
* Only apply Google Java format to changed lines
* Only apply Google Java format to changed lines
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
It is burdensome to have to maintain two sets of tools, one of which
contains a strict subset of functionalities of the other. All admins
should use the same tool and their ability to administer should be
restricted by the IAM roles they have, not the tools they use.
* Add a registry lock password to contacts
* enabled -> allowed
* Simple CR responses, still need to add tests
* Add a very simple hashing test file
* Allow setting of RL password rather than directly setting it
* Round out pw tests
* Include 'allowedToSet...' in registrar contact JSON
* Responses to CR
* fix the hardcoded tests
* Use null or empty rather than just null
* Add a generate_schema command
Add a generate_schema command to nomulus tool and add the necessary
instrumentation to EppResource and DomainBase to allow us to generate a
proof-of-concept schema for DomainBase.
* Added forgotten command description
* Revert "Added forgotten command description"
This reverts commit 09326cb8ac.
(checked in the wrong file)
* Added fixes requested during review
* Add a todo to start postgresql container
Add a todo to start a postgresql container from generate_sql_command.
* Added MetricsHandler and Clock to ProbingSequences
* Minor fixes after rebase onto master
* Added metrics gathering to ProbingSequences
* Added testing of MetricsCollector method calls in ProbingSequenceTest
* Added tests on latency recording to ProbingSequenceTest
* Added response name as label to metrics
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Minor syle fixes after succesful rebase onto master
* Initial Commit.
* Added Protocol Class and its Basic Unit Tests
* Fixed Gitignore to take out AutoValue generated code
* Final Changes in .gitignore
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Removed AutoVaue ignore clause from .gitignore
* Rebased to Master and added in modified Handlers and ProbingAction
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Modified license header dates
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Fixed files to pass all style tests
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Added circular linked list to utils
* License Header added
* Refactored probing sequence to be circular linked list iterator
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Added circular linked list to utils
* Added circular linked list to utils
* License Header added
* License Header added
* Refactored probing sequence to be circular linked list iterator
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Added missing license header to DefaultCircularLinkedListIterator
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Fixed max column length to be 100
* Rebased to Master and added in modified Handlers and ProbingAction
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Added circular linked list to utils
* Added circular linked list to utils
* License Header added
* License Header added
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* ProbingStepTest modified to have fewer unnecessary helper methods
* Updated issues in rebasing
* Fixed max column length to be 100
* Minor changes to pass style tests
* Successful rebase onto finished web-whois branch
* Removed need for TestTokens with Mockito mocks of Tokens
* Fixed style issues in DefaultCircularLinkedListIterator and AbstractCircularLinkedListIterator
* Modified CircularList according to changes suggested by jianglai.
* Added Protocol Class and its Basic Unit Tests
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Updated WebWhois tests.
* Added Basic EPP structure
* Added Basic EPP structure
* Prober Updated tests
* Prober Updated tests
* Fully functioning EPP sequences with modified WebWhois base.
* Fully functioning EPP sequences with modified WebWhois base.
* Added Modified test server infrastructure.
* Added Modified test server infrastructure.
* Allowed ActionHandler to pass status to next hanlder in pipeline (to be MetricsHandler).
* Allowed ActionHandler to pass status to next hanlder in pipeline (to be MetricsHandler).
* Javadoc on EppRequestMessage added
* Javadoc on EppRequestMessage added
* Updated EppServer to properly send successful Check responses.
* Updated EppServer to properly send successful Check responses.
* Allowed for expected failures in EPP actions.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Fully rebased branch to prober-web-whois after refactoring
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to master
* Fixed max column length to be 100
* Fixed files to pass all style tests
* Minor changes to pass style tests
* Successful rebase onto circular-list
* Epp Refactored to accomodate circular linked list PR
* Modified construction of Epp Probing Sequences to reflect CircularList change
* Renamed ProberModule provided Duration
* Removed unnecessary ServerSideException file
* Google-Java-Format run on all prober files
* Style fix on ProbingSequence and its unit tests
* Removed subclasses of EppRequestMessage and EppResponseMessage and fixed style and other minor issues
* Style changes implemented as suggested by jianglai
* Added style fixes suggested by mindhog
* Fixed changes suggested by CydeWeys
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Initial Commit.
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* fixed build issues
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Minor Style Fixes
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Rebased to master
* Updated issues in rebasing
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Fixed files to pass all style tests
* Fixed files to pass all style tests
* Minor syle fixes after succesful rebase onto master
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added circular linked list to utils
* Added circular linked list to utils
* Added circular linked list to utils
* Added circular linked list to utils
* Added circular linked list to utils
* License Header added
* License Header added
* License Header added
* License Header added
* License Header added
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* ProbingStepTest modified to have fewer unnecessary helper methods
* ProbingStepTest modified to have fewer unnecessary helper methods
* Updated issues in rebasing
* Updated issues in rebasing
* Added missing license header to DefaultCircularLinkedListIterator
* Fixed max column length to be 100
* Fixed max column length to be 100
* Minor changes to pass style tests
* Successful rebase onto finished web-whois branch
* Removed need for TestTokens with Mockito mocks of Tokens
* Fixed style issues in DefaultCircularLinkedListIterator and AbstractCircularLinkedListIterator
* Modified CircularList according to changes suggested by jianglai.
* Merge branch 'master' into prober-circular-list
* Modified ProbingSequenceTest to not expect unnecessary NullPointerException
* ProbingSequence and tests modified to reflect addition of UnrecoverableStateException and restarts on failures
* Modified ProbingSequence and its tests to reflect action generation and calling being put in the same try catch block
* Clean up token generation
- Allow tokenLength of 0
- If specifying a token length of 0, throw an error if numTokens > 1
* Allow generation of 0-length strings
* Allow for --tokens option to generate specific tokens
* Revert String generators and disallow 0 'length' param
* Add verifyInput method and batch the listed tokens
* Check the number of tokens created
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Fixed changes suggested by CydeWeys
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Minor syle fixes after succesful rebase onto master
https://github.com/google/nomulus/pull/129 migrated `GoogleCredential`
to `GoogleCredentialsBundle` and introduced a subtle bug. I don't fully
understand why but there are times when the access token is null but
`credentials.refresh()` is not called, resulting in NullPointerException
when `credentials.getAccessToken().getTokenValue()` is called.
Since the new GoogleCredentials class supports `shouldRefresh()`, we now
just rely on it to make sure that we always get a value access token.
This PR created the new interface named TransactionManager which defines
methods to manage transaction. Also, the access to all transaction related
methods of Ofy.java are restricted to package private, and they will be exposed
by DatastoreTransactionManager which is the datastore implementation of
TransactionManager.
* Generate code coverage report
Enable jacoco, the official Gradle code coverage plugin.
The 'build' task will write a code coverage report to
build/reports/jacoco for each subproject that has tests.
We should consider publish periodical reports to a well known
location.
This change also defines a minimum coverage verification task.
The task is for experiment only, and is not added to the build
process yet.
* Remove unused log argument
* Use the right accept-encoding
By default we request gzip and theoretically we'd decode it
automatically on our end but for some reason that's not working. I
tested this on Alpha and it worked
* Create a Gradle task to run the test server
As an artifact of the old build system, the test server relies on having
the built registrar_(bin|dbg)*(\.css)?.js in place (see ConsoleUiAction
among others). As a result, we create a Gradle task that puts those
files into the correct, readable, location before running the test
server.
* Depend on assemble rather than build
* refactor gitignores
The file name confused the licensee app
(https://github.com/licensee/licensee) that GitHub uses to detect
license files. It thinks this file is also a license file and was not
able to determine is type.
Login failures will happen any time that we aren't coming from a
whitelisted IP for that particular TLD. Since whitelists are out of date
(and we don't whitelist IPs for every TLD anyway) those failures aren't
interesting. Store and fully-log the interesting failures if one
happened.
Using the new GoogleCredentials to access Drive API caused 403 forbidden
exception. So, this PR brought back the old GoogleCredential to
temporarily resolve the production issue while we are figuring out the
long term fix.
TESTED=Deployed to alpha and verified exportPremiumTerms succeeded, see
https://paste.googleplex.com/6153215760400384.
* Fail gracefully when copying detailed reports
When the detailed reports are copied from GCS to registrars' Drive
folders, do not fail the entire copy operation when a single registrar
fails. Instead, send an alert email about the failure, and continue to copy the
rest of the reports.
Also, instead of creating duplicates, overwrite the existing files on
Drive.
BUG=127690361
* Merge beam and GAE configs deployment to one GCB job
Deployment of GAE configs requires that the credential used by gcloud to
have GAE admin role of the project to be managed. We do not want to
grant the GCB service account that role, because it would all *any* GCB
job to deploy anything to GAE. Instead we use a dedicated credential
originally created to deploy beam pipelines. This credential is
encrypted by KMS and stored on GCS. Since the beam pipeline deployment
GCB job already does the decryption, it make sense to add the config
deployment step there as well. The beam deployment steps are tweaked to
use the nomulus tool docker image instead of the jar file.
Also moved the content of deploy_configs_to_env.sh to the GCB yaml file
itself because the shell script is not uploaded to GC Bat the same time
as the yaml file when the job is triggered by Spinnaker.
Lastly, due to b/137891685, using GCB to deploy cron jobs does not work
as we cannot use service account credential to deploy to projects under
google.com.
* Add a Cloud Build task to update YAML configs
* CR responses
* Move config deployment to a script
* Pin builder version
* Create different beam and deploy-config files per environment
* Update comments and make a for loop
* Build docker image of nomulus tool
In the course of "gradle build", build a docker image of nomulus tool so that
users can run this to allow us to bundle the java version with the image.
* Don't extend expiration times for deleted domains
* Flip order and add a comment
* oops forgot a period
* Use END_OF_TIME
* Add tests for expiration times of domains with pending transfers
* Add test for transfer during autorenew and clean up other tests
* Clarify tests
* Add domain expiration check in EppLifecycleDomainTest
* Add a comment and format test files
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Merge remote-tracking branch 'upstream/master'
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Save for later
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Add a metric for EPP processing time regardless of ID/TLD
* Change name to request_time
* Record EPP processing time by traffic type
* grammar
* request type
* semicolon
* Check dependencies' open-source licenses
Use jk1/Gradle-License-Report to verify that all
dependencies have open-source licenses.
Note that the following dependencies are not checked:
Dependencies of official Gradle plugins
Dependencies declared in buildscript block
Dependencies of jk1/Gradle-License-Report itself
* Remove the maybeRuntime configuration
It contains dependencies present in the bazel
build but not needed for compile. We now know
they are not needed in runtime either.
Sometimes, the webdriver tests get stuck forever for no reason. It could
be some issue in the test container but it is hard to root cause it. So,
adding a 30s timeout can either trigger the retry earlier or let the
test just fail.
This PR prevents Gradle from copying the golden images
to build/resources/test, so the screenshot test would
read golden images from src/test/resources directly and
display the path in test log if the test fails. Because
the path pointing to the actual file in src/ folder,
the engineer can easily find it.
We got non-serialization object error when deploying the invoicing
pipeline. It turns out that Beam requires every field in the pipeline
object is serilizable. However, it is non-trivial to make
GoogleCredentialsBundle serilizable because almost all of its
dependency are not serilizable and not contraled by us. Also,
it is non-necessary to inject the credential as the spec11
pipeline also writes output to GCS without having injected
credential. So, removing the injected variable can solve the
problem.
TESTED=First reproduced the problem locally by deploying the invoicing pipeline with the previous code; applied this change and successfully deploy the pipeline without having any issue.
* Attempt login to MosAPI via all available TLDs
There's no reason why we should need a TLD as input here because it
doesn't actually matter which one we use (they all have the same
password).
* Refactor the TLD loop and change cron jobs
* Re-throw the last exception if one exists
* Fix tests and exception
* Remove alpha cron job
* Move test resource files into src/test/resources
* fix a test
* Remove references to javatests/ in Java files
* fix import order
* fix semantic merge conflict
* Throw a more useful error message on attempted domain restore reports
Per DomainRestoreRequestFlow's Javadoc, we automatically approve and instantly
enact all domain restore requests, thus we don't use or support restore
reports. This improves the registrar-visible error message to help make this
more clear.
Replace deprecated GoogleCredential with new lib
This PR also introduced a CredentialsBundle class to carry
HttpTransport and JsonFactory object which are needed by
most of the GCP library to instantiate client.
* Refactor the license check to be a generic presubmit check
This includes all of the old presubmits that were in our METADATA file
that are still valid.
* Responses to CR
Added docstrings, refactored the file finder, changed variable names,
and formatted the file
* More respones to CR
We found that some webdriver tests failed because RegistryEnvironment
was set to 'production' by other test and was carried over to the
webdriver test, and it was nontrivial to fix them because the instance
of RegistryEnvironment was injected into the testing web server.
Also, to eliminate this problem from potential victims, we stopped
provisioning RegistryEnvironment from Dagger. Instead, we just use
RegistryEnvironment.get() function to get the instance, which indeed
retrives the value from system property every time. If any test case
needs to run the test with other environment, it can just simply use
the existing SystemPropertyRule and RegistryEnvironment.${ENV}.setup
to change it.
* Update GCB beam deployment pipeline
Some of the texts are not really secerts because they are per-project.
Also changed the location of the credential file to `secerts` so that in
the future we may add more secerts in that folder.
The encrypted file is base64 encoded, consistent with how the proxy
certificates are encoded. Also made some changes to the other pipelines
to facilitate automation with Spinnaker
* Move terraform and kubernetes folder to be under proxy
There is no reason for them to be under proxy/src/... any more now that
we have a Gradle-idiomatic folder structure.
The check for environment cannot be done when the task is defined,
otherwise running any task will fail when the check does not pass.
Instead, check the environment when the task executes.
Our CI (Travis & Kokoro) runs "gradle build", so we need to make
sure that all tasks used in the release process are called during
the build so that breakage can be caught earlier.
In order to stage the GAE folder we need gcloud to be present.
Therefore the Travis config is changed to install gcloud.
See: https://gist.github.com/mjackson/5887963e7d8b8fb0615416c510ae8857
IntelliJ is complaining when this annotation is used on a base class that has no
actual runnable tests itself. The solution is different for different classes
depending on the existing pattern of where the @RunWith annotation is; for some,
it's simplest to move the annotation down to the few extending classes that are
missing it, whereas in others it's easiest just to annotate the base class as
abstract.
Fix or suppress deprecation warnings except those about GoogleCredential,
which is being handled separately.
The @SuppressWarnings("deprecation") annotation does not cover imports
even when it is at the class level. We removed imports of deprecated
classes and use their fully qualified names in class body.
* Consolidate testcontainer used by WebDriver test
Previously, we hard coded the version of the docker image for
provisioning Chrome browser and WebDriver server because the
version of the Chrome browser has to match the version of
the webdriver client, otherwise the screenshot test will fail.
Changing to use BrowserWebDriverContainer can delegate the match
to the library itself because it chooses the correct docker image
version based on the WebDriver version on our classpath.
* Increase maxColorDiff to 20
This is to supress the test flakness after switching to use
BrowserWebDriverContainer to provision browser and webdriver
service.
I'm seeing odd errors in versions of Karma more recent than 3.0.0. While
I investiage, I'd like to see what Github has to say about these
dependencies (we got some vulnerability messages for earlier versions)
This option causes Intellij build to fail if the
'Delegate IDE build/run actions to gradle' box is checked.
We do not know of anyway to change the Intellij behavior.
This change allows an IDE user to prevent -Werror to be passed
to javac by adding a Gradle VM option: -Dno_werror=true
* Add a link to kokoro
This allows us to click on the build badge to visit kokoro and find
out build status. The link will redirect to a status page that is
only accessible to Googlers.
Our existing uber jar task overwrote duplicate files when merging JARs,
and this would break the application using Java SPI service(Beam library
is one of them) so we changed to use ShadowJar plugin as it provides a
convenient way to merge META-INF/services/* files when generating the
uber jar.
TESTED=Deployed spec11 pipeline successfully with the new nomulus.jar;
also randomly checked a few nomulus commands to make sure they are still
working.
We aren't turning on the cron job yet - we want to make sure it works correctly first.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252726789
CharMatcher.isLetterOrDigit() is deprecated for failing to support
supplementary characters. We explicitly declare a matcher for ascii
letters and digits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252671830
With https://github.com/spinnaker/spinnaker/issues/4048 Spinnaker now natively supports GCB. We are able to start a GCB job from Spinnaker, and also there is better support to consume GCB pub/sub messages. Some changes are made to remove the workaround no longer needed.
Two new workflows are added, one to rsync a GCS folder to live/ after the deployment is done (so that the nomulus.jar file can then be fetched to x20 by a []cron job), and the other to tag the proxy image as live once it is deployed.
Lastly, the docs/ folders are needed when running tests. Remove it from .gcloudignore so that when a test run is kicked off by running "gcloud builds submit" the folder is sent to GCB. Ideally .gcloudignore should be identical to .gitignore but since they both are version controlled it is hard it make one a symlink of another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252625901
To do this we add a field of "rdapServers" in the Registrar object. Currently, we can only set this field manually, but a subsequent CL will add a cron-job to read these values from the ICANN servers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252438618
Original change by cpovirk@cpovirk:rosie251284456-0055_Rosie:31511:citc on 2019/06/04 09:48:27.
Update to Truth 0.45, and address deprecations.
Renames may include:
- containsAllOf => containsAtLeast
- containsAllIn => containsAtLeastElementsIn
- isSameAs => isSameInstanceAs
- isOrdered => isInOrder
- isStrictlyOrdered => isInStrictOrder
The other major change is to change custom subjects to extend raw Subject instead of supplying type parameters. The type parameters are being removed from Subject. This CL will temporarily produce rawtypes warnings, which will go away when I remove the type parameters (as soon as this batch of CLs is submitted).
Some CLs in this batch also migrate calls away from actualAsString(). Its literal replacement is `"<" + actual + ">"` (unless an object overrides actualCustomStringRepresentation()), but usually I've made a larger change, such as switching from an old-style "Not true that..." failure message to one generated with the Fact API. In that case, the new code usually contains a direct reference to this.actual (a field that I occasionally had to create). Another larger change I sometimes made is to switch from a manual check-and-fail approach to instead use check(...). And sometimes I just remove a withMessage() call that's no longer necessary now that the code uses check(...), or I introduce a check(...) call. (An assertion made with check(...) automatically includes the actual value from the original subject, so there's no need to set it again with withMessage().)
Finally, there's one CL in this batch in which I migrate a Correspondence subclass to instead use Correspondence.from.
END_PUBLIC
If this is too many changes at once, let me know, and I can split it up and/or explain further. In the past, I've erred on the side of sending separate CLs for each change, which has required some owners to manually reapply each one, so now I'm trying this way.
Thanks again for your patience. There is an outside chance that this will be the last CL I send you before Truth 1.0 -- but certainly no promises :)
More information:
Renames:
[]
[]
[]
Removing type parameters: []
Migration from old fail*(...) methods to new ones and to check(...): []
Changes that replace assert*(...) with introduce check(...): []
Correspondence subclass to Correspondence.from: []
Tested:
TAP --sample ran all affected tests and none failed
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251921007
We need to first populate the apt sources before installing anything. It was working on my machine because the layer was cached. After I purged all images I was able to replicate the same error message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251918035
It would be nice to be able to query them using SQL in order to debug
some potential issues registrars might be facing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251863662
We'll use this for LRP. This is safe because we must specifically reserve a
domain by including it in a reserved list, create an associated allocation
token, and distribute that token, before a create would succeed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250901144
They are useful for viewing and diagnosing problems, and are not that
expensive.
We aren't enabling all the metrics though - the whole "per batch / per domain"
performance indicators are still disabled
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250876554
This specifically includes info, transfer, and update. Note that normal
domain creates do not work in quiet periods and are not expected to.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250755940
These updates can generally ran slowly in the background (i.e. they're low
priority), and shouldn't clog up the system to the point where they slow down
live incoming customer-requested updates.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250732171
|[](https://storage.googleapis.com/domain-registry-kokoro/internal/index.html)|[](https://storage.googleapis.com/domain-registry-kokoro/foss/index.html)|[](https://lgtm.com/projects/g/google/nomulus/alerts/)|[](https://github.com/google/nomulus/blob/master/LICENSE)|[](https://cs.opensource.google/nomulus/nomulus)|
<propertyname="message"value="String.toUpper/LowerCase() can have unexpected results depending on locale. Either set the locale explicitly - e.g., toUpperCase(Locale.ENGLISH) - or use Ascii.toUpper/LowerCase(...) instead."/>
</module>
<!-- Checks that DateTime.now(...) is always passed a DateTimeZone parameter. -->
<modulename="RegexpSingleline">
<propertyname="format"value="DateTime\.now\(\)"/>
<propertyname="message"value="DateTime.now() must be called with a DateTimeZone parameter, e.g. DateTime.now(UTC)"/>
</module>
<!-- Checks that Javadoc does not include a malformed @see tag. -->
<propertyname="message"value='Your Javadocs appear to use invalid <a> tag syntax in @see tags. Please use the correct syntax: @see <a href="http(s)://your_url">url_description</a>'/>
</module>
<!-- Checks that our Ofy wrapper is used instead of the "real" ofy(). -->
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
