* Use the parent to store the history repo ID and fill in the base object
Storing the repo ID in the parent and in the base object has two primary
benefits.
First, it unifies the parent information in the HistoryEntry's `parent`
object. This simplifies the builders and the data flow.
Second, when possible (which should be always, post-migration) we fill
out the DomainContent's repo ID (similarly for the other EPP resources)
which means that when reconstituting the ofy keys we don't need to pass
the repo ID in from a separate object. This way, all the data are
encapsulated where they should be.
The primary downside here is that it further reduces the "immutability"
of the history objects (since we're using the Hibernate setter for the
parent repo ID) but we weren't immutable anyway.
* Respond to CR
- compare the entire vkeys in tests
- always return the parent for repo ID
* Simplify creation of parent VKeys
* Fix flipped isAssignableFrom check in VKey
* Merge remote-tracking branch 'origin/master' into historyRepoId
* Minor python changes
Use dataclasses instead of attrs. The former is part of the standard lib
while the latter may need to be installed separately.
Also added python3 to the list of prerequisites.
* Maintain a release-to-Version map in deployment
Keep track of the mapping between Nomulus release tags and AppEngine
version ids with a mapping file. This is necessary because AppEngine
does not support custom versioning. With this mapping, rollbacks could
be automated. Automation of rollbacks is important since there are
test-supporting metadata to be updated, but are easily forgotten.
During the last stage of deployment, current per-service version ids
are fetched using gcloud and are appended to a file on GCS. Each line
is of the format "{RELEASE_TAG},{APPENGINE_SERVICE},{APPENGINE_VERSION}.
This change has been tested in crash. The rollback script is still a
work in progress.
* Convert CertificateViolation into an enum
This ends up being nicer to deal with from callsites than class instances, while
still permitting full configurability of all parameters. There are various other
changes/fixes as well.
* Restore composite vkeys in DomainContent
PollMessage/BillingEvent vkeys in DomainContent must have their ofy keys
restored from other fields in DomainContent (namely the repo id and their
specific history event ids).
Add PostLoad methods to DomainContent and DomainHistory to do the restoration.
* Fixes for review.
* Deal with foreign-key cycles
* CertificateChecker with checks for expiration and key length
* Add validity length check
* Get rid of hard-coded constants and DSA checks
* add files that for some reason weren't included in last commit
* Rename violations and other fixes
* Add displayMessage to CertificateViolation enum
* Switch violations from an enum to a class
* small changes
* Get rid of ECDSA checks
* add checks for old validity length
* Change error message for validity length
When loading the VKeys for the BillingEvents hierarchy, it is necessary to
restore the original concrete class for the type, otherwise we end up with a
different (and incompatible) VKey.
As part of this, convert the cancellation matching billing event to
VKey<Recurring>, which seems like the only thing it actually can be.
Add a restoreOfy() instance method and a restoreOfyFrom() static method to
assist in restoring the objectify key for classes that have composite keys
that do not restore automatically.
* Create a flyway index file and verify correctness
Create an index file (flyway.txt) containing the names of all of the flyway
files and verify that it is ordered and in sync with the actual contents of
the flyway directory. Also provide a target (generateFlywayIndex) to
automatically generate it.
The purpose of flyway.txt is to cause a merge conflict in the event that two
different developers add a flyway file with the same sequence number, an event
which has occurred multiple times.
Without it we kept getting the following warning:
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
* Enhance the test for forbidden Schema changes
Current test is git-based. It is difficult to maintain and does not
catch out-of-order version numbers. It is also more aggressive than
necessary, failing on changes to submitted scripts that have not been
deployed yet.
The new test starts a database, deploys the current schema to it,
then deploys the set of Flyway scripts in this repository to the database.
* Allow explicitly for null EPP resources in History objects
* Repo IDs should always be nonnull
* Add a test to verify loading / comparison of legacy HistoryEntry objects
* Format javadoc + annotations
* More javadoc changes
* V52 -> V56
* V56 -> V57
* saveNew -> insert in new tests
* Improve naming of TransactionManager methods
Per internal discussion, convert names of methods as follows:
saveNew -> insert
saveNewOrUpdate -> put
checkExists -> exists
Likewise, convert derived names to their corresponding forms, e.g.
saveNewOrUpdateAll -> putAll.
* Add domain-specific history fields to DomainHistory objects
* Add javadoc for Hibernate-only methods
* V52 -> V54
* Use only a single DomainTransactionRecord table
* Add nullables and fix up a comment
* V54 -> V55
* Regenerate db schema
* Regen SQL file
* Create a separate per-tld registry lock/unlock cost
Currently we use the standard server status change cost for this, but
this might not be ideal at some point in the future if we wish to allow
manual forced updates outside of the standard registry lock system (we
would charge for these manual forced updates, even if we don't charge
for registry locks).
* Remove period
1. It appears that when we have a 200 response, the response content is
garbled, but we don't care since we know the request is successful.
When we have a 400 response, the response is indeed UTF-8 encoded.
Print the stack trace of the HTTP exception doesn't help anymore.
2. The result code is a complex type which includes the xml element with
all its attributes, which don't care. We only want to print the
value.
* Add TLD table
* Change reservedLists to array
* Change ReservedLists back to a set
* Rename reservedListKeyConverter to ReservedListKeySetConverter
* Add a postload method
See b/168508962 for context.
1. Changed the contact to iana-contact@google.com
2. Changed the header from "Script" to "Language" for zh-Hans and
zh-Hant.
3. Commented out the references in zh-Hans and Zh-Hant
* Fix accessing superclass fields in checkExists()
JpaTransactionManagerImpl doesn't respect @Id fields in mapped superclasses.
Replace calls to getDeclaredId() and getDeclaredField() with superclass
friendly counterparts.
* Fix the billing occurrence foreign key
Fix the Domain.billing_occurrence_id foreign key constraint to reference the
correct table (BillingRecurrence, not BillingEvent).
* Change disable invoicing flag to enable invoicing flag
This flag will be the sole determinor on if invoicing is enabled,
regardless of TLD types.
Once this PR is deployed we will need to run the nomulus command to
update this flag on all launched open TLDs.
For context on why this change is made, see b/159626744.
* Rename enableInvoicing to InvoicingEnabled
* Implement DatastoreEntity/SqlEntity for many more classes
We still have many more classes to go, but this gets us closer to
guaranteeing that we can convert from Datastore to SQL objects and back
again.
* Shift SqlEntity impl to HistoryEntry
* Make ICANN reporting not fail on success upload
According to the spec
(https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-13#page-16),
when an upload succeeds (HTTP response code 200), the result code
contained in the response message is always 1000 (success). So there is
no need to parse the response content and check the result code. Given
that we are having a problem parsing the response content due to encoding,
it is best that we don't check it so as to not get false negative
alerts when the upload is successful.
The current logic also has a bug: HttpRequest.execute() will by default
throw when the response code is non-20X. Therefore for a 400 response,
our parsing logic never runs on it. Coincidentally, this month when we
uploaded the July activity report (due to stale cursors), we get 400
responses (due to existing reports on the ICANN servers). The stack
trace printed for the thrown exceptions from the 400 responses contained
correctly parsed response contents. This lead us to believe that the issue with
encoding was transient last month. However when we tried again to upload this
month's report, our parser failed again (because the response code was 200 this
time, and our parser actually ran on the response contents).
This seems to suggest that ICANN is sending back readable response
contents, but our parser somehow failed to understand it, assuming that
ICANN is using the same encoding for 200 (which we tried and failed to
parse) and 400 response contents (which caused an exception and was printed
corrected in the stack trace).
This PR changed the transport behavior so that it doesn't throw
automatically for non-20X responses. We will print the content for both
200 and 400 responses, but only try to parse 400 response content. We
put the 400 response in an HttpResponseException and print stack trace
from it, which should display the content correctly so that we can
compare it with the result of our own parsing.
* Add tests
* Use composite primary key for DomainHistory
* Move History table's SequenceGenerator to orm.xml
* Rebase on HEAD and remove default value for key in History tables
* Use primitive type for id.
* Revert the cache change
* Fix flaky web driver tests
Identified two flaky tests in RegistrarConsoleScreenshotTest through
local testing and fixed them by waiting for specific web elements instead
of using fixed delays.
Refactored the wait methods to support different test scenarios,
and removed unnecessary delays.
Extensively tested locally. Also ran multiple presubmits on Kokoro.
* Cover more base in forbidden SQL change check
Update the forbidden SQL change detection script to include file deletion and
renaming as well as edits.
* Add success/failure notifications for the RelockDomainAction
If a relock fails for some reason, we should noisily notify both our
alerting email and also the registry lock contacts for the registrar in
question. The consequences of a silent failure could be large so it's
something we want to avoid if at all possible.
In addition, we only retry tasks up to two times (one in 5min, one in
10min).
This model of retries / notifications, as well as the language contained
in the emails, have been LGTMed by Bruno and Kirsten
* Change the wording on the success email
* Change the times in which we send emails
For transient failures:
- Retry every ten minutes for six hours
- Send an email after a half hour (three failures) saying that we'll
retry
- Send a success email if we succeed any time after that
For non-transient failures:
Send an email with the error message and don't retry
* Add a test for the max-failure-email
* Responses to CR
- retry indefinitely
- send an email to just the alert address if we can't find the lock
- refactor the task enqueuer a bit
* non-transient -> non-retryable
* Use a lenient stubber for the AESU
* Add a DS transaction around the re-lock
* Persist *History objects as HistoryEntry objects
While Datastore is the primary database, we will store *History objects
as HistoryEntry objects and convert to/from the proper objects in the
Datastore transaction manager. This means that History objects will not
properly store the copy of the EppResource until we move to SQL as
primary, but this is the way the world exists anyway so it's not a
problem.
* Format code and simplify the bulk loading
* Add comments with context
* Fix JPA setup in Nomulus tool
Hibernate unnecessarily scans third-party classes in the Nomulus tool,
hitting a bug and fails to set up.
In this change we properly configured persistence.xml to include the orm mapping file (orm.xml) and disable
auto detection, and provided a custom (NOOP) scanner
to work around Hibernate scanner bugs.
Also improved on the :core:registryIntegrationTest task to test for
JPA setup as well as dependency-packaging.
* Fix missing-driver in HibernateSchemaExporterTest
HibernateSchemaExporterTest is failing with "Driver not found" error
after Java 11 upgrade. Reason is that ServiceLoader now only checks
modules for services.
Proper fix is to define modules.
This short term fix is to declare the driver class explicitly.
* Add check to prevent creating VKeys with incorrectly null parents
Datastore entities that are not the roots of entity groups are uniquely defined
by an inheritance chain in addition to the id/name, not just by id/name.
* Fix empty Domain nameserver loads
Domains with no nameservers were being loaded from SQL as an empty set instead
of null as they should be.
Discovered this will trying to test updates, so added a test for updates in
the course of it.
* Add a SQL schema to AllocationToken
* Respond to CR
- rename field in tests
- rename allowed_registrar_ids field
- remove unnecessary db load in GATC
* Add TODO for HistoryEntry vkeys
* Run autoformat
* V48 -> V49
* Allow schema push in alpha and crash
This allows unsubmitted changes to be tested in alpha and
crash.
Also updated the README file and reformatted with the internal
mdformat tool.
* Update user-facing documentation
Give our docs a complete overhaul to account for changes in the system,
notably the requirement to configure postgresql.
* Fix dangling sentence.
* Merge branch 'master' into admin-docs
For some inexplicable reasons I have to move the javax.mail package one
spot up to avoid its classes being shadowed by those provided in the
appengine package...
* Create DAO for Spec11ThreatMatch
* Add tests
* Execute SQL for deleteEntriesByDategit status
* Remove testing line
* Rename createSpec11ThreatMatch()
* Add comments about jpaTm and use jpaTm() in test
* Fix technicality in comment
* Remove a new line
* Truth chaining for comparing ImmutableLists of matches
* Javadoc formatting
* Add some missing @Nullables and types
Also deletes two unused VKey.createOfy() methods that simply don't work, because
a kind and an id is not enough to create a Datastore key; you also need the full
entity group inheritance chain for entities that are not roots themselves
(which is most of the entities in our schema).
* Merge branch 'master' into add-missing-nullables
* Throw UnsupportedOperationException for contacts/hosts too
* Merge branch 'master' into add-missing-nullables
* Update BEAM SDK to work with Java 11
Upgraded BEAM dependencies to 2.23.0.
Updated Spec11 and invoice pipelines:
- Added the required region parameter.
- Removed the workaround code for staging.
Verified that staging is successful in alpha:
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_spec11_pipeline --project domain-registry-alpha'
and
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_invoicing_pipeline'
* Double the timeout length for WebDriver screenshot tests
My theory is that this timeout is being applied to all retries of a failing test
rather than each one, and thus flaky screenshot tests aren't being given
sufficient time to complete any attempt past the first.
This test is flaky and the going theory is that we take a screenshot
immediately after we click the "Add" button and the CSS hover effect is
somehow still playing on the button.
* Restore original jpa manager after tests
AppEngineExtensionTest fails to restore original jpa manager in some
tests. This results in flakiness in DummyJpaTransactionManagerTest.
The current setup causes the GCB job to fail validation and not run because it
uses backticks in the configuration yaml, which is not allowed -- there is no
shell to perform backtick substitution. See the error message here:
https://spinnaker.endpoints.domain-registry-dev.cloud.goog/gate/pipelines/01EF5GRMD625613H6Z033DBD3Z
In the future please make sure to test the GCB pipeline as instructed in
the comments at the beginning of each file before committing.
I tried to work around it by downloading the nomulus tool jar file
instead (running the nomulus-tool docker image inside a docker image is
not advisable). However the "nomulus deploy_spec11_pipeline" command
still fails. I'm not sure why. Has the command itself been tested
locally? The error message is shown below:
```
Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
Step #2: WARNING: --region not set; will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones
Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory tryCreateDefaultBucket
Step #2: INFO: No tempLocation specified, attempting to use default bucket: dataflow-staging-us-central1-937378958468
Step #2: Aug 09, 2020 3:11:47 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse
Step #2: WARNING: Request failed with code 409, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://www.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=domain-registry-alpha.
Step #2: Exception in thread "main"
Step #2: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
Step #2: at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
Step #2:
Step #2: at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
Step #2: at org.apache.beam.sdk.Pipeline.create(Pipeline.java:147)
Step #2:
Step #2: at google.registry.beam.spec11.Spec11Pipeline.deploy(Spec11Pipeline.java:157)
Step #2: at google.registry.tools.DeploySpec11PipelineCommand.run(DeploySpec11PipelineCommand.java:80)
Step #2: at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:257)
Step #2: at google.registry.tools.RegistryCli.run(RegistryCli.java:182)
Step #2: at google.registry.tools.RegistryTool.main(RegistryTool.java:129)
Step #2: Caused by: java.lang.reflect.InvocationTargetException
Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Step #2:
Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Step #2: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Step #2: at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
Step #2: ... 8 more
Step #2: Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@5cb0d902, only URLClassLoaders are supported.
Step #2: at org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:58)
Step #2:
Step #2: at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:285)
Step #2:
Step #2: ... 13 more
```
Lastly the "--project" flag refers to the KMS project. While I'm not
sure which project is that, I don't think we can use the PROJECT_ID
variable as this is a GCB-substituted variable which refers to the
project that the GCB job runs in, which in our cases means
domain-registry-dev. We shouldn't use that project for KMS. I've changed
it to the same project as the one we are deploying to, but please note
that we have a separate project ${project_id}-keys that is used for all
KMS purposes. This is specified in the config file so if that's what you
meant to use, there is no need to specify it in the command line. Actually
if you meant use the project to be deployed to for KMS, it also
shouldn't be necessary to specify it separately as this information is
already known when you specified "nomulus -e ENV".
https://team.git.corp.google.com/domain-registry-eng/nomulus-internal/+/refs/heads/master/core/src/main/java/google/registry/config/files/nomulus-config-production.yaml#168
Can you add more description on what the KMS project is supposed to be?
I don't think we specify a project for KMS purpose in any other
commands.
Given that there are several unresolved issues, I've commented out my
proposed solution so that deployment can proceed.
For reasons unclear at the moment the tests are not passing. Disabling
them for now so that release candidates can be built. We have CI runs
after each merge so we should be pretty confident if the build is broken
or not.
The node installed by nvm gives errors when running "npm install".
Also installs Python as it is need. Presumbly the system provided npm
version has python as a dependency so it was installed when npm was
installed.
* Add retrier to retry exceptions we've seen so far
* Check for nested exceptions
* fix formatting
* check for nested transactions
* add JDBCConnectionException to isFailedQueryRetriable
* Add retrier to methods with suppliers
- Reuse DS record format processing from the create/update domain commands
(BIND format, commonly used in URS requests)
- Remove the CLIENT_HOLD status from domains that have it (this blocks us from
serving the new nameservers and DS record)
* Replace jpaTm with a JpaSupplierFactory
* Style
* Style
* Pipeline takes in a SerializableSupplier instead
* Change the ordering of imports
* Test a good domain in addition to a bad one
* Rename and check good domain for Transact Answer
* Use standard Mockito verify
* Verify transact call and no more interactions
* Remove Answer comment
* Naming chsnges
* Deploy Spec 11 pipeline correctly
* Fix formatting of deploy file
* Use a file to persist state across Cloud Build steps
Co-authored-by: Gus Brodman <gbrodman@google.com>
* Refactor DomainBase into DomainContent and create DomainHistory
This is similar to #587 and #634, but for domains.
One caveat is that we refactor some of the Domain* instance methods to
be static so that they can be called either on DomainBase or
DomainContent, returning the appropriate type each time.
Note that we set DomainHistory to use the same revision ID sequence as
HostHistory and ContactHistory.
In addition, we refactor the tests to the History objects a bit to
reduce duplicate code and because we cannot guarantee yet that the
SQL-stored VKeys are symmetrical -- the ofy keys are not persisted at
the moment.
In addition, rename the DomainHost table to the default Domain_nsHosts so that it automatically creates two separate nsHosts tables for us -- one foreign-keyed on the domain repo ID, and one foreign-keyed on the history revision ID
* Use access hackery to allow manual names for nsHosts tables
* Clean up post merge artifacts
* Add unused setters that Hibernate requires
* Fix the tests and semantic merge conflicts
* Change ns_hosts to ns_host everywhere
* Rename ns_host to host_repo_id
* V42 -> V44
* Enable Java 11 features
As of this commit Java 11 must be used to build. The generated bytecode
is still at Java 8 due to App Engine task queue limit.
Also fixed a bug where the included google-java-format jar file is not
used, requiring the user to install it separately.
See: https://cloud.google.com/appengine/docs/standard/java/taskqueue/push
* Add use of interval data type
* Add support for Millis
* Use Java-object type
* Change column type for relock_duration
* add years and months
* Add tests for hours, minutes, and seconds
* Add javadoc describing how joda duration is stored
* Add test for lots of days
It is not clear to me why this folder is deleted in standardTest first,
presumably this is done when the webdriver tests were part of
standardTest?
Also not clear to me is why screenshots are only deleted in presubmits
but not locally. At least on my workstation standardTest runs after
fragileTest, just like in kokoro, so it should have deleted the
screenshots
folder, but it didn't.
Regardless, the correct thing to do is to delete this folder first in
fragileTest so that it doesn't interfere with the results.
Add the class paths of the source files generated by annotation processors to
the javadoc task's class path so that it doesn't complain about missing
Dagger classes.
Also remove empty <p> tags in all generated source files, because jaxb
genrerates files in multiple locations.
Lastly, for unkown reasons when the source level is set to > 8, the core
subproject throws a warning about a Gradle internal annotator processor
that only supports up to Java 8 and cause the Java compliation to fail because
we set -Werror on all java compliation tasks. I don't think there is a
strong reason that we set -Werror anyway, so this commit removes it.
* Get rid of all remaining JUnit 4 usages except in prober & proxy subprojects
Caveat: Test suites aren't yet implemented in JUnit 5 so we still use the ones
from JUnit 5 in the core subproject.
* Fix some build errors
* Migrate the documentation package to Java 11
The old Doclet API is deprected and removed in Java 12. This commit
changes the documentation package to use the new recommended API.
However it is not a drop-in replacement and there are non-idiomatic
usages all over the place. I think it is eaiser to keep the current code
logic and kind of shoehorn in the new API than starting afresh as the
return on investment of a do-over is not great.
Also note that the docs package is disabled as of this commit because we
are still using Java 8 to compile which lacks the new API. Once we
switch our toolchains to Java 11 (but still compiling Java 8 bytecode)
we can re-enable this package.
TESTED=ran `./gradlew :docs:test` locally with the documentation package
enabled.
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
* Run InitSqlPipeline
Added the main() method to InitSqlPipeline.
Added a Gradle task to run InitSqlPipeline from command line. This
task is meant for testing and experiments.
Corrected the file name prefix of Datastore export files. Should
be 'output-', defined as 'input-'.
Apparently, in domain check responses, `avail=false, reason=Allocation token
required` was not sufficiently understood by all registrars. This changes it to
`avail=false, reason=Reserved; alloc. token required` to hopefully make it
crystal clear that the domain in question is reserved, i.e. if you were supposed
to be able to register this domain you'd already know it because we'd have
already given you the requisite allocation token.
This makes it easier to later migrate the package to Java 11. If we move
and migrate in a single PR, because of the portion of the contents that
s changed, git will have trouble recognizing that some files are
renamed *and* modified and treat them as distinct files, making code
review difficult.
* Integrate transaction persistence into JpaTM
Store the serialized transaction whenever we commit from the JPA transaction
manager. This change also adds:
- The Transaction table.
- The TransactionEntity which is stored in it.
- Changes to the test infrastructure to register the TransactionEntity for
tests where we don't load the nomulus schema.
- A new configuration variable to allow us to turn the transaction
persistence functionality on and off (default is "off").
* Changes for review.
* Incremented sequence number of flyway file
* Upgrade App Engine and webserver tests from JUnit 4 to 5
* Fix most errors
* Merge branch 'master' into junit5ification
* Fix test server by extracting non-test setup/tear-down
* Merge branch 'master' into junit5ification
* Fix backup tests
* Don't createFile(); asCharSink does it
* Increase the timeout for all WebDriver tests to 60s (helps w/ flakiness)
* Set up deployment of the Spec11 pipeline with JPA TM
* Remove unnecessarily pipeline options setting
* Use enviroment name in BeamJpaModuleTest
* Fix checkstyle error
Add a "pseudo-task" in nom_build to do the three step process of generating
the golden schema. In the course of this, add support for pseudo-tasks in
general, improve the database directory readme and make nom_build not call
gradlew if there are no tasks.
* Run the (Un)lockDomainCommand in an outer JPA txn
There are a couple things going on here in this commit.
First, we add an external JPA transaction in the
LockOrUnlockDomainCommand class. This doesn't appear to do much, but it
avoids a situation similar to deadlock if an error occurs in Datastore
when saving the domain object. Specifically, DomainLockUtils relies on
the fact that any error in Datastore will be re-thrown in the JPA
transaction, meaning that any Datastore error will back out of the SQL
transaction as well. However, this is no longer true if we are already
in a Datastore transaction when calling DomainLockUtils (unless, again,
we are also in a JPA transaction). Basically, we require that the outer
transaction is the JPA one.
Secondly, this just allows for more breakglass operations in the lock or
unlock domain commands -- in a situation where things possibly go
haywire, we should allow admins to make sure with certainty that a
domain is locked or unlocked.
* Add more robustness and tests for admins locking locked domains
* Fix expected exception message in tests
* Use rearranged sql credentials in flyway task
Let the flyway tasks use the sql credential files set up for BEAM
pipelines.
Credential files have been created for each environment in GCS
at gs://${project}-beam/cloudsql/admin_credential.enc. All
project editors have access to this file, including the Dataflow
control service account.
Alpha and crash use the 'nomulus-tools-key' in their own project to
decrypt the credential file.
Sandbox and production use the 'nomulus-tools-key' in
domain-registry-dev to decrypt the credential file.
Note that this setup is temporary. It will become obsolete once
we migrate to Cloud Secret Manager for secret storage.
* End-to-end Datastore to SQL pipeline
Defined InitSqlPipeline that performs end-to-end migration from
a Datastore backup to a SQL database.
Also fixed/refined multiple tests related to this migration.
This is copied in here with the absolute minimum # of modifications required
(just a rename to JUnit 5 format and some small fixes required to enable
compilation to be successful).
This is in preparation for the next commit where I'll convert this Rule into a
JUnit 5 extension, which is the entire goal here. But I wanted to get the code
from Apache Beam in with the maximum possible fidelity so that my changes will
be in a separate commit and will thus be obvious.
Note that we do unfortunately need to modify/rewrite the Rule itself; merely
wrapping it in some manner isn't possible.
* Fix some SQL credential issues identified when deploying Beam pipelines
There are two issues fixed here.
1. Without calling `FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create()), the Nomulus tool doesn't know how to handle gs:// scheme files. Thus, if you try to deploy (for instance) the Spec11 pipeline using a GCS credential file, it fails.
2. There was a misunderstanding before about what the credential file
actually refers to -- there is a credential file in JSON format that is
used for gcloud authorization, and there is a space-delimited SQL access
info file that has the instance name, username, and password. These are
separate options and should have separate command-line params.
* Actually we don't need this for remote deployment
* Create an ImmutableObjectSubject for comparing SQL objects
Many times, when comparing objects that are loaded in from / saved to
SQL in tests, there are some fields we don't care about. Specifically,
we might not care about the last update time, revision ID, or other
things like that that are autoassigned by the DB. If we use this, we can
ignore those fields while still comparing the other ones.
* Create an ImmutableObject Correspondence for more flexible usage
* Add a 'Host' parameter to the relock action enqueuer
I believe this is why we are seeing 404s currently -- we should be
specifying the backend host as the target like we do for the
resave-entity async action.
* Fix JpaIntegrationRule in JUnit4
Made DatastoreExtension a JUnit4 Rule.
Nomulus model objects need Datastore API when manipulating Ofy keys.
As a result, JpaIntegrationTestRule must be used with AppEngineRule
or DatastoreExtension.
Also fixed WriteToSqlTest, which is the only JUnit4 test that uses
JpaIntegrationTestRule.
* Add lastUpdateTime column to epp resources
Property was inadvertently left out.
Renamed getter and setter to match the property name.
Added a test helper to compare EppResources while ignoring
lastUpdateTime, which changes every time an instance is persisted.
* Write one PCollection to SQL
Defined a transform that writes a PCollection of entities to SQL using
JPA. Allows configuring parallelism level and batch size.
* Add JUnit Params and start using it
* Convert rest of RDE tests
* Don't check headers for generated tests
* Expand visibility to fix build breakage
* Bump JUnit versions to 5.6.2
* Add a "buildFmt" gradle target
This does the same thing as the automatic Java build target, except instead of
failing if the code formatting isn't correct, it just automatically reformats as
necessary and continues on.
* Remove unnecessary mustRunAfters
* Make it run tests too, and add :taskTree task
* Rename task to coreDev and remove run afters
* Add task tree dependency
* Actually that may not be necessary
* Change Subdomain class to contain domainRepoId
* Remove jpaTm from Spec11PipelineTest and change clientId -> registrarId
* Remove 'client' from a comment
* Include changes to Spec11Pipeline
* add SafeBrowsingTransforms
* Run style
* Use an enum instead of boolean in EntityTestCase constructor
It's more clear to use an enum rather than just a simple boolean
* Add Javadoc and make the enum name more verbose
* Double the # of pubapi instances to better handle traffic spikes
We may also consider switching to an automatic scaling mode soon, on the hope
that it's working better than the last time we tried it (it would help to keep
resource costs down at least).
* Create ContactHistory class + table
This is similar to #587, but with contacts instead of hosts.
This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest)
Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory.
* Move ContactResource -> ContactBase
* Alter ContactBase and ContactResource
* Verify that the RegistryLock input has the correct registrar ID
We already verify (correctly) that the user has access to the registrar
they specify, but nowhere did we verify that the registrar ID they used
is actually the current sponsor ID for the domain in question. This is
an oversight caused by the fact that our testing framework only uses
admin accounts, which by the nature of things have access to all
registrars and domains.
In addition, rename "clientId" to "registrarId" in the RLPA object
* Change the wording on the incorrect-registrar message
* Output PO number in detailed report
The PO number header was added during the beam migration but we forgot
to print the actual data in the corresponding column. This resulted in a
misalignment of columns in the detailed report.
This PR fixes it. Note that we cannot drop PO number from the header (as is not
useful in the detailed report) because the header represents all fields
that are to be parsed from the SQL query results, and PO number *is*
needed when generating the invoice itself. By dual-purposing the header
(both as the required fields in the parser and the first line in the
detailed report) we have to include the value of PO number in the
detailed report CSV as well.
* Disambiguate injected Cloud SQL parameter names
This allows us to also inject the BeamJpaModule into RegistryTool, which
allows us to use the SocketJpaTransactionManager in Beam pipelines.
Some side effects of this include:
- duplication of KMS connections -- one standard, one Beam
- duplication of the creation of the partial Hibernate SQL configs
- removal of ambiguity between credentialFileName, credentialFilename,
and credentialFilePath -- we now use the latter.
- Performing the credential null check when instantiating the SQL
connection rather than when instantiating the module object. See the code
comments for more details on this.
I verified that this compiles and the tests run successfully when
injecting a @SocketFactoryJpaTm into a Beam pipeline.
* Remove two unnecessary config points and change the name of two params
* Use @Config instead of @Named and change the pool size
* Replace non-visible link with code
* Load Datastore snapshot from backup files
Defined a composite transform that loads from a Datastore export and
concurrent CommitLog files, identify entities that still exist at the
end of the time window, and resolve their latest states in the window.
* Exclude Test/Monitoring Registrars from escrow
Registrars used for testing and monitoring should not be included
in Data escrow. They also lack the required ianaIdentifier property
and would fail ICANN data validation.
Note that since alpha and crash environments have bad data that
break the RDE process, we need to verify this change in Sandbox.
* Make EppResource.loadCached() use batched fetch
Use a batched fetch (ofy().load().keys(...)) from datastore in
EppResource.loadCached().
To support this, convert TransactionManager.load(Iterable<VKey>) to accept the
more flexible generic parameters and return a map.
* Simplify datastore key streaming
* Changes requested in review.
* Update to generic Spec11ThreatMatch table
* Fix SQL syntax
* Make changes to the schema and add a test for null and empty threatTypes
* Fix a small typo
* Change the exception thrown with illegal arguments
Change the import for isNullOrEmpty
* Fix import for checkArgument
* Added a threat to test multiple threat types
This adds an entriesToImmutableMap() collector that can be used in place of
toImmutableMap(Map.Entry::getkey, Map.Entry::getValue()).
It also fixes up some existing calls that use toImmutableMap() when terser
alternatives exist.
* Make nomulus compatible with Java 11
This fixes the double-spacing bug with logged EPP XML on App Engine that started
appearing after App Engine switching from using Java 8 to Java 11. Java 9 made
some changes to XML Transformer classes that needed a little bit of work to
accommodate.
This also fixes the unit tests that were failing in Java 11 (all of which were
related to said XML Transformer changes).
* Make code review changes
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.
ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.
Added a mockito dependency, and upgraded mockito-core to 3.3.3.
Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
The added property will by default be null for existing entities and false for new entities. We could write a temporary MapReduce to set it to false for all existing ones, but it seems easier to just modify the query to account for the nullness of the column. The need to check for null won't exist once we migrate the Registry entity to Cloud SQL.
TESTED=deployed to alpha, set the property for tld zombo to true and triggered an export, verified that the column is null for all rows other than the row for zombo.
* Delay SQL availability check during Tool start
During Cloud SQL migration, Cloud SQL password setup in Registry
server requires a few tool commands. Such commands must be able
to run before Cloud SQL is configured.
In this change, we deplay SQL credential loading until access is
needed.
Hibernate complained that ContactResource.getTransferData should not be
final:
WARN: HHH000305: Could not create proxy factory
for:google.registry.model.contact.ContactResource
org.hibernate.HibernateException: Getter methods of lazy classes cannot
be final: google.registry.model.contact.ContactResource#getTransferData
* Set up JpaTransactionManager in BEAM pipelines
Added modules and utilities to create JpaTransactionManager in BEAM
pipelines.
Not wanting to set up AppEngine Remote API to access Keyring in the
Datastore, we instead use the credential files in GCS, which are
used by Spinnaker/Cloud Build and desktop access. Added utility
to download, decrypt, and parse the file. Also added/modified dagger
modules.
* Don't expect a renewal fee on restores when one isn't due
This is a fix on top of #632 so that domain restore commands don't require
acking an illusory renewal fee for 1 year when that isn't actually happening
(i.e. if the domain isn't yet past its original expiration).
Unfortunately, there's still a problem remaining wherein the restore fee on a
domain check will always include the additional year even if it's not
necessary. We don't have a good solution to that. Also note that in versions of
the fee extension more recent than 0.6, the fee extension cannot be passed on a
domain info command at all, so the domain check command is the only way you have
to determine what the restore fee should be. So we definitely do want to get
that right so that the apparent restore fee on a check is the same as the actual
restore fee when running the restore command. We're not quite there yet though
and it's hard to say how we will get there, since we don't load domains during a
domain check command for performance reasons yet we would need to do so in order
to know the expiration date and thus whether the additional year of renewal
should be charged.
A problem for a future PR.
* Squash everything together
Create SafeBrowsing_Threats table
Create LocalDateConverter and add indexes to SafeBrowsingThreats
Add indexes to SafeBrowsingThreats and make small style changes
Pass in DateTimeFormatter
Delete LocalDateConverterTest.java
Rebase
Make changes to ThreatType comments
Create LocalDateConverterTest
Add review changes
Add SafeBrowsingThreatTest
Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys
Change imports
Add foreign keys and rename version number
Add new generated db-schema file
Clean up null test cases
Add changes
Add foreign keys into SafeBrowsingThreatTeat and apply style checks
Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file
Make small changes to SafeBrowsingThreatTest
Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat
* Change V35 -> V36
* Add a foreign key test for a reference to Registrar
* Move some variables around
* Add testcontainers' Junit5 support dependency
Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.
Migrated PersistenceModuleTest as an example.
Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
* Fix flakiness caused by ofy entity name conflict
Overrode the 'kind' of two test entities to prevent name conflicts.
Tests are flaky because of this.
Added a check in AppEngineRuleTest for conflicting kinds.
* Show price of reserved domains when using matching allocation token
When the registrar passes the fee extension, this shows the price of the domain
on a check command for reserved domains if the provided allocation token is a
match. Of course, the price is already always displayed on non-reserved names
(regardless of whether the specific provided token is a match or not).
This affects domain checks only; the price is already always displayed on domain
creates because you already by definition have access to register the domain in
question.
* Refactor pipline for Datastore backup loading
Refactored pipeline transforms.
Added testing utilities that handles assertions better.
Investigated and documented challenges in serializing Ofy entities
without side effects.
* Remove 'fullyQualified' from host and domain names
We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.
Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API
* Add TODO to rename fields
* Implement a persistable Transaction object
Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.
* Changes requested in review
* Add a mujtation count to the persisted format
* Fix cookie processing for RDAP URL update
The existing code only does cookie processing on the _first_ Set-Cookie
header. Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.
Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
* Add proof of concept for HostBase and HostHistory classes
* Use a PROPERTY accessor for @Ids
* Add an unused setter method for Hibernate's sake
* Refactor HostHistory
* Some responses to CR
* Fix relationship and test
* Manually manage the foreign keys for HostHistory
* Protect HostBase's builder and use text for the enum type
* Add responses to CR
- Add javadocs
- Create an ID sequence for host history objects
* Don't try to set the ID
* Use a Long and remove the setter
* Add some comments and rename a couple fields
* Don't change Datastore schema
* Use Long in the Datastore schema
* Add new createVKey method
* Add comments and rename fields
* Rename v27->v31 and regenerate the golden
* Fix superordinateDomain and inetAddresses in HostHistory
* V31 -> V32
* Fix SQL files that got messed up in the merge
* Configure and use a manually-created history ID sequence
* Add three more indices to HostHistory
* Add DomainTransferData and ContactTransferData
* Refactor TransferData to remove unused fields in Contact table
* Add scope for TransferData's type parameter
* Allow class-specific creation of symmetrict VKeys
When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.
As a test case for this, implement the createVKey() static function for
DomainBase. Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
* Refactor Fee handling so that each fee knows if it's premium
This is a noop for now, as the new isPremium boolean isn't yet used by anything,
but it will be used in follow-up PRs to add additional fee information using the
fee extension (see: b/157621273). Specifically what we're trying to do here is
return <fee:command name="create" standard="1"> (using the finalized version of
the fee extension) when an entire command has no premium fee associated with
it. And in the current earlier versions of the fee extension that we support,
we'll want to display the correct fee amount and class for creates/checks on
reserved domains when a valid allocation token is passed. This also needs the
isPremium information.
There are no testing implications yet because isPremium isn't exposed anywhere,
but there will definitely be lots of test changes once it's feeding into EPP-
visible changes.
* Rename things, add method Javadoc
* Apply formatting
* Restore the original expiration time on domain restore
Except if that time is now in the past, then add a year to it.
* Apply auto-formatter changes to fix my local build
* Merge branch 'master' into restore-expiry-date
* Fix reversed comments
* Use the server host when creating the registry lock verification URL
The app doesn't know about any external configuration that may point to
this app, so there's no way of finding out that, for instance,
registry.google points to the app. Thus, we have to use what the user
gives us so that, in our case, the registry-lock verification
emails can point to https://registry.google/registry-lock-verify instead
of https://domain-registry.appspot.com/registry-lock-verify. The former
is used by clients / users to authenticate, and unfortunately
authenticating on registry.google does not give authentication to
domain-registry.apspot.com.
Tested using the RDAP code that uses getServerName() -- in that case, if
you access registry.google/rdap/<>, it uses registry.google in the URLs
but if you use domain-registry.appspot.com/rdap/<>, it uses
domain-registry.appspot.com in the URLs.
Relatedly, frontend_config_prod-appengine.asciiproto in Piper
is what configures registry.google to point to
domain-registry.appspot.com
* Log client certificate type and length
It appears that most client certificates are RSA certs, but we should
make sure that is indeed the case. Print out the strength of the cert
if it is RSA.
Also adds supports for TLS 1.3 and print out the supported cipher suites.
* Add a comment about zero length certificate
* Make length of non-RSA keys -1
* Don't use TLS 1.3 if JDK SSL provider is used
* Load CommitLog into BEAM pipeline
Created tools that can generate CommitLogs on a test
datastore.
Defined BEAM transforms that load from CommitLog files and
added a few simple tests.
This is work-in-progress. Next step is to build a consistent
backup by merging Datastore exports and CommitLogs.
We'll eventually want to shift everything over to using registrar_id and
registrarId rather than client_id and clientId but for the sake of the
Datastore schema and existing code, we won't change the Java identifier
for now. Once we're completely and only on SQL, we can rename the Java
field easily.
* Create a converter for sets of inetAddresses and use it in HostResource
This can just be a set of strings where each string represents an
address; there's no need for it to be a separate table. This allows
for simplification of the SQL schema.
* Regenerate golden SQL file after renaming v28 -> v29
* Add more tests and rename a typo in the file
* Refactor common test code and use tm methods
* Use JUnit5 API
* Rename test entity
* Add columns for TransferData in Domain and Contact
* Rename flyway file and foreign key
* Rebase on master and address comment
* Compileable commit
* Fix unit test
* Refactor TransferServerApproveEntity
* Use tm().delete(vkeys)
* Rename transfer_period fields
* Rename client_id to registrar_id
* Rebase on master
* Resolve comment
* Rebase on master
* Store the superordinateDomain reference as a VKey rather than Key
This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
Given that we currently have no way of reconstituting a symmetric key
from the asymmetric key (or at least, we don't have a 100% reliable way
of doing so) it's best to keep keys as asymmetrical, referring to the
correct database. That way, we don't get situations where we cannot
compare equality of two keys due to one being asymmetrical and one being
symmetrical.
* Remove platformType and threatEntryMetaData fields from ThreatMatch
* Run google-java-format on both files
* Add test for removal of unnecessary fields
* Removed unnecessary fields from Spec11PipelineTest.testEndToEndPipeline_generatesExpectedFiles
* Added style check
* Fix typo
* Load raw records from Datastore export
Created a tool that can export from a test instance of Datastore.
Defined Beam pipeline transforms for loading raw records back from
the export.
This is the first part of the effort to create a consistent snapshot
of Datastore. The next step is to load entity records from CommitLog
files.
* Use TransactionManager for hosts and contacts
Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts. In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.
* Minor formatting fix
* Refactor LevelDbFileBuilder to accept DS Entity
Builder now can directly work with Datastore Entity objects.
No need to wrap data in ComparableEntity.
* Generate sql schema for BillingEvent
* Change to use sequence
* Address comments
* Resolve warnings and remove duplicate cost related fields
* Increase the flayway file version to V25
* Remove extra space
* Split to 3 tables, merge VKey
* Rename talbes
* Rename repoId to domainRepoId
* Exclude VKey in schema.txt
* Rename target_id to domain_name
* Fix javadoc
* Resolve comments
If the import from Datastore to BigQuery fails, there is no point
enqueuing a job to update the snapshot view.
Also when there's an error updating the snapshot view, log it at severe
level. The HTTP exception thrown is logged at info and triggers a retry
implicitly. I'm not sure if we want this behavior though. Do we want to
retry upon snapshot updating failures? Unless the failurs are transient,
retrying doesn't help. In our case the failure (End of time out of range
in Standard SQL) is not transient.
EntityClasses.ALL_CLASSES has all of our registered entities so
we can use it to initialize VKeyTranslatorFactory.classRegistry
to avoid adding them one by one.
Also, this PR changed to use Key.getKind() to get the kind of
the entity to solve the problem that when the entity class
is an inner class, its kind should still be the class name
instead of OuterClass$InnerClass.
* Read LevelDb incrementally
Made LevelDbLogReader an iterator over a LevelDb data stream,
Reducing memory footprint which is important when used in a
Dataflow pipeline.
* Make VKey persist to datastore as a key
Convert nsHosts entirely to VKey as a proof-of-concept.
Tested as follows:
1) Deployed to crash, verified that nameservers were visible for several
domains (indicating that we are able to load a set of Keys as VKeys)
2) Updated the set of nameservers for a domain (removing some initial
hosts) and verified that the changes went through.
3) Deployed the old version to crash, verified that I was able to retrieve
the newly saved VKeys as Keys.
4) Modified the hosts for the same domain (adding back one of the hosts)
and verified that the change took effect.
5) Redeployed this change to crash, again updated the nameservers to add
another host.
6) Again restored the old version, verified that the new hosts were
visible.
* Changes in response to review
* Convert to a single VKeyTranslatorFactory instance
* Moved vkey field rename to V25
Normal HttpException logs at INFO because they usual do not indicate
anything out of the ordinary and is meant to convey to the client that
there is some expected error. However InternalServerErrorException is
something that we do care about being alerted for so we log it at SEVERE.
* Add TimedTldStateTransitionMapConverter
* Move timedTransitions to a base class and add BillingCostTransitionConverter
* Add test of TimedTransitionPropertyConverterBase
* clean up tests
* Switch tests to JUnit 5
* Make JpaUnitTestRule an extension
* Implement all DatastoreTransactionManager methods
In the course of this:
- Make assertDelete() specific to JpaTransactionManager, remove the return
value from delete()
- Converter "in transaction" assertion to IllegalStateException, which is less
JPA specific.
* Upgraded DatastoreTransactionManagerTest to junit5
* Reflect refunded billing events on deletion in expiration time
This doesn't make any change at the time of the domain deletion itself, but it
will matter if the domain is then undeleted, because we need to know what
expiration date to restore, and if there were any renew or autorenew charges
that were refunded by the deletion because they were in a grace period, they
shouldn't be coming back during the restore.
* Add tests for new expiration date behavior
* Add handling of add/renew grace period overlap
* Convert DomainBase's contacts to VKeys
Convert usage of DomainBase contacts from Key to VKey. This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
On certain machines (one of mine) the outcastTest consistently fails due
to the following error:
java.lang.NoClassDefFoundError: Could not initialize class
google.registry.persistence.transaction.JpaTestRules$JpaIntegrationTestRule
If I reduce maxparallelForks to 3 it consistently passes. This issue was
mentioned here:
https://discuss.gradle.org/t/junit-test-fails-with-noclassdeffounderror-only-when-maxparallelforks-1/6047
But this post was 8 years old and no solution was identified.
Since we rarely (if ever) want to check the result of a single element load,
make TransactionManager.load(VKey) return non-optional, non-nullable and just
throw a NoSuchElementException if the entity is not found.
Also add a maybeLoad() that does return an optional in case we ever want to do
this (exists() should suffice for an existence check).
* Implement DatastoreEntity and SqlEntity on more classes
For classes that aren't going to transition to SQL, they should just
return an empty list of SqlEntities. When reading these in from the
commit log manifests, we just won't persist anything to SQL.
By having all Datastore entity classes implement DatastoreEntity, we can
avoid potential bugs where we forget to transition some entity to SQL,
or we forget to have the capability to read back from the commit logs.
Note: the EntityTest is still @Ignore'd because there are many SQL and
Datastore classes left -- ones that we are still in the process of
converting or adding, or ones that require more complicated transitions.
Note: Locks and Cursors aren't converted (even though we could) because
they're ephemeral
* Responses to CR
Add a @EntityForTest annotation
fix null that snuck in
* Keep the test ignored for now
* Remove email-editing footgun
Email address is used as the primary key so we should be very careful
about changing it. This will have even more importance when this is the
location to which we will be sending registry lock confirmation emails.
Note: we allow addition or removal of contacts through the UI (and don't
want to disable that) and because all edits are performed by saving the
entire list of contacts, we can't explicitly prevent all possible edits
of email address in the backend. So this doesn't technically prevent
anything security-wise, but it makes it much more difficult to
accidentally edit an email when you shouldn't.
* Enforce non-deletion of registry-lock-enabled contacts
* Fix tests
* Specify contact
* Fix flaky tests due to Entity name conflicts
Objectify siliently replaces current registration of a given kind
when another class is registered for this kind. There are
several TestObject classes in the current code base, which by
default are all mapped to the same kind.
Tests have only been flaky because impacted tests need to run
in specific orders for failures to happen. Using multiple executors
in Gradle also reduced the likely hood of errors. To reproduce the
problem run the following tests in order (e.g., by putting them in
a test suite):
1. ExportCommitLogDiffActionTest
2. CreateAutoTimestampTest
3. RestoreCommitLogsActionTest
In this PR, we
- Made sure all entities have unique kinds.
- Made all test entities register with AppEngineRule instead of directly
with ObjectifyService.
- Added code in AppEngineRule to check for re-registrations.
- Added presumit check for forbidden direct registration.
* Allow the `nomulus renew_domain` command to specify the client ID
This means that a superuser can renew a domain and have the associated history
entry, one time billing event, and renewal grace period be recorded against a
specified registrar rather than the owning registrar of the domain. This is
useful to e.g. renew a domain for free by "charging" the renewal to the
registry's fake registrar. Since the grace period is written to the specified
cliend id as well, if the actual registrar deletes the domain, they don't get
back the money that they didn't pay in the first place.
* Don't NPE when nomulus tool is run without a subcommand
This occurred when an environment was specified but without a subcommand. Now,
the list of valid subcommands is outputted instead of seeing a generic NPE.
This also makes some formatting changes in other files that were causing the
incremental format check to fail.
* Try AppEngineRule
* Remove separate deployment of persistence.xml
We added a step to explicitly copy persistence.xml because for some reason it
wasn't originally getting deployed to app-engine, resulting in failures on
startup. However, this file is now included in core.jar and we are now
getting a warning about multiple persistence units with the same name as it
reads the files from both the filesystem and core.jar.
* Add SqlEntity and DatastoreEntity interfaces
These will be used when replaying transactions from either the Datastore
commit logs or the SQL Transaction objects.
When Datastore is the primary database, we will read in the
Datastore commit logs, convert each saved entity to however many SQL
entities, then save those SQL entities in SQL.
When SQL is the primary database, we will read in the SQL objects from a
yet-to-be-created SQL table, convert them to however many Datastore
entities, then save those Datastore entities in Datastore.
This PR includes a couple simple examples of how this will work for entities that are
saveable in both SQL and Datastore (the simple case).
* Add 1-1 mapping between entity annotations and interfaces
* Add JPA annotations to ContactResource and generate schema
* Resolve comments
* Resolve comments
* Manually add foreign key constraints
* Run with junit5
* Rebase on HEAD
* Fix DomainBaseSqlTest
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Allow Entity instantiation without AppEngineRule
Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
* Hack to call setup and teardown in JUnit5 suite
JUnit 5 runner does not support @BeforeAll and @AfterAll declared
in the Suite class (as opposed to the member classes). However,
staying with the JUnit 4 suite runner would prevent any member
classes from migrating to JUnit 5.
We use a hack to invoke suite-level set up and teardown from tests.
This change is safe in that if the JUnit 5 runner implementation changes
behavior, we will only see false alarms.
* Implement VKeyConverter
Implement a SQL converter that can be used for VKey objects.
Caveats:
- This only works with string columns (there's an excellent chance that all of
our VKeys will use SQL string columns).
- Using this dpesn't establish a foreign key constraint between the referenced
type (the "T" in VKey<T>) and the entity itself: this needs to be
defined manually in the schema.
* Exclude proxy configs from the FOSS jar
No sensitve data exposed.
Added a todo to modify the release process and stop
building the foss jar on the merged repo.
* Add verification that domain labels aren't multi-level domains
In addition, I did a bit of test refactoring because previously, the
CreateOrUpdateReserveListCommandTestCase test cases weren't actually
testing the proper things -- they were failing with
IllegalArgumentExceptions, but not the right ones.
* Change test name and use IDN library
* Handle numeric labels
String like "0" or "2018" are valid labels but not valid domain names
* Use IDN validation with a dummy TLD
On Arch Linux, DumpGoldenSchemaCommandTest failed due to the follow
error:
java.lang.RuntimeException: Container.ExecResult(exitCode=1, stdout=, stderr=pg_dump: [archiver] could not open output file "/tmp/pg_dump.out": Is a directory)
However I cannot figure out why this permission error happens, as the
docker command is executed as root. Saving the pg_dump output to a
temporary file and copy it over the mapped file works, so I don't
know...
* Migrate SqlIntegrationTestSuite members to Junit5
Made InjectRule and EntityTestCase work with both JUnit4 and 5.
Note that "@RunWith(JUnit4.class)" is no longer needed on
JUnit4 test classes. Therefore, its removal from EntityTestCase
has no impact on child classes. All of them are still included in
tests.
Migrated remaining member classes in SqlIntegrationTestSuite to JUnit5.
* Add Test suite support for JUnit 5 classes
Added Gradle dependencies and updated lockfiles.
Updated SqlInegrationTestSuite to use new annotations.
Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.
Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
* Make AppEngineRule work with JUnit 5
Made AppEngineRule work with both JUnit4 and JUnit 5 and applied
it to PremiumListTest.
Next step is to convert SqlIntegrationTestSuite.java to JUnit5.
* Actually run JUnit 5 tests
In Gradle, JUnit 5 must be explicitly enabled with a call to
test.useJUnitPlatform().
The FilteringTest used in :core must also enable JUnit5 separately.
Fixed AppEngineRule to work with the few tests that have migrated
to JUnit5.
More work is needed with AppEngine before we can migrate tests
that actually use Cloud SQL.
For context, with @EnableRuleMigrationSupport, JUnit 5 runner calls
an external resource's before() and after() methods. TestRule.apply()
is not called, therefore any setup done their will be bypassed with
JUnit 5.
* Upgrade CompareDbBackup for Datastore V3
Upgrade the CompareDbBackup class to work with latest
Datastore backup directory structure.
Also fixed a few unrelated minor issues:
- Remaining cases of improper use of System.setOut
- Wrong import order in one class
* Improve error information in coverage test.
If the golden schema isn't up-to-date with the persistence model, the coverage
tests fail with an exception chain that ends in a PSQLException 'relation
"TableName" does not exist' which is kind of misleading when the problem is
that your golden schema isn't up-to-date.
Check for this error in the coverage tests and generate a more informative
error message indicating a likely root cause.
* Key to VKey conversion for Nameserver
This change illustrates the conversion of a single key in the system
(Key<HostResource> as used in the "nameservers" field of DomainBase) to a
VKey.
It currently builds, but had some curious (possibly unrelated?) test failures
that I have not fully investigated.
* Latest round of changes, all tests pass.
* Changes requested in review.
* Fix problems with null check in VKey accessors
Add maybeGet versions of getSqlKey() and getOfyKey() that return Optional
objects and make the nameserver management routines use those instead.
* Reuse JPA EntityManagerFactory in tests
Reuse EntityManagerFactory instance in tests if the requested schema
stays the same. Only truncate tables and reset sequences when reusing.
Note that the jdbc driver needs to be informed to expect out-of-band
schema changes.
* Don't rely on the password field's existence for admins
We don't have the field when it's an admin user that's logged in. A
nicer language would have caught this unfortunately.
* Drop schema instead of database in Sql tests
Speed up the database cleanup between tests by dropping the schema
instead of the database. The new approach is much faster.
Ad hoc measurement on my desktop shows that :core:sqlIntegrationTest
improves from 73 seconds to 48 seconds, and :core:standardTest
improves from 12m40 to 7m40.
* Allow backwards compatibility with JUnit 4 @Rules in JUnit 5
This allows us to defer having to re-implement all of our JUnit 4 Rules as JUnit
5 extensions for now, while continuing to in-place upgrade all existing JUnit 4
test classes to JUnit 5.
As proof of concept, this upgrades PremiumListUtils (which uses AppEngineRule,
our largest and most complicated @Rule) to use the JUnit 5 test runner.
* Apply formatter to entire file
* Flat map to avoid unclean errors in bad situations
Also properly reflect that for admins, we will use their user email
* Make MS's GAE user ID a public static field
* Add a registryLockEmailAddress field to RegistrarConctact objects
Because we need to manage the login email, it should be on an account
that we manage. However, for registry lock, we would want to send the
verification emails to a separate email address that the user can use.
As a result, we will use a second field for a user-accessible registry
lock email address. This must be set on the contact when enabling
registry lock for this contact.
* Responses to CR
* derp
* Fix the test server
This rule isn't necessary any more since we merged the SQL-starting rule
into the AppEngineRule logic. Furthermore, it actually causes the test
server to crash because we try to drop-and-create the DB twice, the
second time while the first instance is still connected.
* Make jpaTm for nomulus tool use local credential
* Remove unused methods in RegistryToolEnvironment
* Fix order of annotations
* Remove unused method in PersistenceComponent
* Move the creation of credential to the module
* Move creadential creation to AuthModule
* Add a TODO
* Add a relockDuration to the RegistryLock SQL object
This is the length of time after an unlock that we will re-lock the
domain in question.
* Sort by domain name for stability
Note: this is likely not the best solution for the UI but we can iterate
on this.
* Add nullable
* Add a converter for Duration
* Match logged-in GAE user ID with registrar POC user ID
The reasoning for this is thus:
We wish to have the users log in using Google-managed addresses--this is
so that we can manage enforcement of things like 2FA, as well as generic
account management. However, we wish for the registry-lock confirmation
emails to go to their standard non-Google email addresses--e.g.
johndoe@theregistrar.com, rather than johndoe@registry.google.
As a result, for registry lock, we will enable it on
the johndoe@registry.google account, but we will alter the email address
of the corresponding Registrar POC account to contain
johndoe@theregistrar.com. By doing this, the user will still be logging
in using the @registry.google account but we'll match to their actual
contact email.
* fix up comments and messages
* Error if >1 matching contact
* include email addresses
* set default optional
* fix tests
* Disambiguate naming of VKey.create() overloads
It was discovered in the course of trying to convert the larger codebase
to VKey.create() calls that method overloading isn't a very effective
discriminator in cases where "Object" is one of the distinguishing
argument types:-)
Convert the two specialized create() methods to createOfy() and
createSql() so that (at least in the former case) we'll get a
compile-time error if we aim to create a VKey for an Ofy key from an
object of the incorrect type.
* Add a RelockAction and reference to relocks in RegistryLocks
* Respond to CR
- refactor the request param exception logging a bit
- don't log an error if the domain was already locked, just skip
* Save a relock for all locks (if possible)
* derp
* Long -> long + remove unnecessary transact
* semantic merge conflict woo
* fix another semantic merge conflict
* Update SqlIntegrationTestSuite
Edited Javadoc to emphasize that suite members should be DAO tests.
Removed functional tests from the suite. They do not benefit much
from running against different schemas when the entities they use
are already covered by DAO tests.
Added DomainBaseSqlTest to the suite, which tests DomainBase.
* Create a nom_build wrapper script
nom_build is a wrapper around ./gradlew. It's purpose is to help us
deal with properties. The main problem that it is trying to solve is
that when properties are specified using -P, we don't get an error if
the property we specify isn't correct. As a result, a user or a build
agent can launch a build with unintended parameters.
nom_build consolidates all of the properties that we define into a
python script where the properties are translated to flags (actual
gradlew flags are also proxied). It also generates the property file
and warns the user if the current properties file is out of sync with
the script and includes documentation on each of the properties.
* Enable Cloud SQL when Datastore is enabled for unit test
* Add explanation for why add a ETA field in GenerateEscrowDepositCommand
* Fix line length
* Ignore membershipt test but bring back test suite
* Fix tiny issue
* Listen to the user hitting enter in the lock/unlock modal input fields
Listen to both, just in case one or the other is disabled
* Don't require that the element exist
* Fix broken builds when Maven Central is used
Gradle 6.2.1 apparently introduces a behavior change wrt boolean
expression: empty string used to eval to false, but now evals to
true.
Pre Gradle 6.2.1, root project's Gradle properties apparently were
not set to buildSrc. Now they are passed on to buildSrc -- mavenUrl
in buildSrc changes from null to "".
Both changes break the project when mavenUrl and/or pluginsUrl are
not set on command line.
Also added junit.jupiter-api as testCompile dependencies to projects.
This is a directly used dependency, whose absence causes a Lint
warning.
* Add BasicDao
* Refactor RegistrarDao to extend BasicDao
* Introduce VKey and rewrite BasicDao
* Move CRUD methods to TransactionManager
* Refactor code to simplify the way to get id from entity and sqlKey
* Assert in transaction
* Fix broken test
* Change methods name
* Always validate domain name on allocation token
This is in response to a client-reported error, where they accidentally sent the
wrong domain name on a domain create that included an allocation token. What
should have happened (and that now happens as of this commit) is an error being
thrown that the allocation token does not match the domain name being created.
What happened instead was that, since the incorrectly submitted domain name was
not reserved, the create succeeded (as it would for all creates of unreserved
domains in GA) and the allocation token was redeemed, which is not what you'd
expect.
* Fix tests to reflect changed check behavior
* Refactor DomainLockUtils methods to take a time rather than a clock
* Add administratively (un)lock methods
* Responses to CR
- Javadoc changes
- Method renames
- Variable renames
* Refactor lock methods to use JPA transaction time
* Remove clock, use Datastore transaction time
* Properly use Datastore transaction time, batched
* Continue to throw exceptions on invalid domains
* DAO writes should be in a transaction
* Assume in-transaction for all RLDao methods
* clean up test
* Fix more tests
* add comment
* Add daily cron job for IcannReportingUploadAction
This job checks each ICANN cursor's cursorTime. If all cursorTime values are in the future, the action exits and does nothing. For each cursor that has a cursorTime in the past, the corresponding report is uploaded if it is staged, or logs an error message if the report has not been staged yet.
* Change cron job description
* Add Registry Lock UI
* Responses to CRs, mostly
TODO:
- Figure out wording for the 'not enabled yet' message
- Include the server status change cost in the email, or in the UI?
- Should we show non-completed lock requests in the UI?
* Fix get action test
* Change the not-allowed-for-registrar msg to include support email
* Change the wording on the price
* Move TLD input into the modal, and other changes
- don't log the password
- test to make sure the password shows bullets
* Responses to CR and cleanup
* Format closer to something proper
* Use -t instead of main params for TLDs in nomulus count_domains command
This makes the command consistent with list_domains. I use both frequently and it
was annoying forgetting which one takes -t and which uses main parameters. Now
they both work the same way.
* Add a scrap command to backfill registry locks
* fix tests
* Change comments and messages
* Use URS time (best effort) if one exists
* Don't bother with root cause
* Lazily load jpaTm
This fixes an exception we were seeing in production where commands that only
implemented CommandWithRemoteApi (and that had nothing to do with Cloud SQL)
were nevertheless trying to initialize a JPA TM instance, and then failing.
That stacktrace looked like:
com.google.api.client.http.HttpResponseException: 400 Bad Request
{
"error": "invalid_grant",
"error_description": "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:423)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
at google.registry.keyring.kms.KmsConnectionImpl.attemptDecrypt(KmsConnectionImpl.java:163)
at google.registry.keyring.kms.KmsConnectionImpl.lambda$decrypt$0(KmsConnectionImpl.java:148)
at google.registry.util.Retrier.callWithRetry(Retrier.java:153)
at google.registry.util.Retrier.callWithRetry(Retrier.java:130)
at google.registry.util.Retrier.callWithRetry(Retrier.java:95)
at google.registry.keyring.kms.KmsConnectionImpl.decrypt(KmsConnectionImpl.java:147)
at google.registry.keyring.kms.KmsKeyring.getDecryptedData(KmsKeyring.java:209)
at google.registry.keyring.kms.KmsKeyring.getString(KmsKeyring.java:178)
at google.registry.keyring.kms.KmsKeyring.getToolsCloudSqlPassword(KmsKeyring.java:100)
at google.registry.persistence.PersistenceModule.providesNomulusToolJpaTm(PersistenceModule.java:124)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.proxyProvidesNomulusToolJpaTm(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:61)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:39)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:12)
at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
at google.registry.persistence.DaggerPersistenceComponent.nomulusToolJpaTransactionManager(DaggerPersistenceComponent.java:168)
at google.registry.persistence.transaction.TransactionManagerFactory.createJpaTransactionManager(TransactionManagerFactory.java:38)
at google.registry.persistence.transaction.TransactionManagerFactory.<clinit>(TransactionManagerFactory.java:29)
at google.registry.model.registry.Registries.lambda$createFreshCache$2(Registries.java:60)
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:243)
at google.registry.model.registry.Registries.getTlds(Registries.java:85)
at google.registry.model.registry.Registries.assertTldsExist(Registries.java:112)
at google.registry.tools.CountDomainsCommand.run(CountDomainsCommand.java:41)
at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:243)
at google.registry.tools.RegistryCli.run(RegistryCli.java:168)
at google.registry.tools.RegistryTool.main(RegistryTool.java:127)
The TL;DR is that RegistryCli was over-eagerly creating the jpaTm, because
there's no reason the Registries cache (which is Datastore-only) should ever
need it, but because this cache is using the Datastore transaction manager, the
other one was being created too.
* Start using JUnit 5
This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration
In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.
* Delete no longer needed lockfiles
* Merge branch 'master' into first-junit5
* Add dual write for Registrar
* Use @AlsoLoad to set street field for Cloud SQL
* Change email body to use the new streetLine field
* Refactored the logic to handle street fields
* Simplify postLoad function
* Address comments
* Add a TODO to remove street
* Add test for onLoad and postLoad
* Rebase on master
* Add the schema and DAO for Locks in CloudSQL
* Addresses some comments
* Change number on flyway file
* Small changes
* More small changes
* Use checkArgumentNotNull instead of checkNotNull
* Address comments
* fix javadocs
* update persistence
* Use JSON API for Maven Repo on GCS
The url pattern https://storage.googleapis.com/{Bucket}/{Path}
uses the legacy XML API, which seems to be less robust than
the JSON API. We have observed connection resets after a few
thousand-file download bursts over 30 minutes.
This PR changes all urls to registry's Maven repo on GCS to
gcs://{Bucket}/{Path}. Gradle uses the JSON API for such urls.
TESTED=In Cloud Build with local change
* Use dependency cache in all Gradle tasks in GCB
Make the initial test and the final publishing steps use the shared
dependency cache.
Also make the initial test use the registry's own maven repo instead
of Maven Central.
* Make Gradle dependency cache shareable in GCB
Make Gradle put its caches in the source tree so that
they can be preserved across steps. When left at their
default location, caches are lost after each step.
* Use CA cert in tests
Our tests were using a self-signed cert created by a netty utility.
That cert is not CA, and cannot be used with newer JDK versions
(starting from u242 with Ubuntu openjdk).
Now we generate a unsafe cert for tests. The new cert,
SelfSignedCaCertificate, needs to be made test-only.
To do so we need to refactor both prober and proxy.
* Always save reserved lists to Cloud SQL
Otherwise it's too easy to forget to run this additional option, and then things will
fall out of sync and start throwing errors.
* Use randomly-generated strings for RegistryLock verification codes
We were using UUIDs before which are also fine, but unnecessarily long.
The RegistryLock class itself does not enforce any particular format for
the lock verification codes.
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
* Add a RegistryLockPostAction and tests
* Response to CR and more robust email content test
* Fix typo
* Run save + email in a JPA transaction
* Use action in subject
* Allow non-CA cert as trust anchor in tests
Stay with the legacy behavior to unblock release build.
We will update the test cert generation code later.
TESTED=reproduced and fixed test failures in builder container
* Work around Spinnaker issue wrt variables
Cloud Build variable reference need to stay from the ${var} pattern
to prevent Spinnaker from trying to resolve it. In all files that
are used by Spinnaker, we change variable reference to the $var form.
We made the minimum amount of change possible, and will review this
issue after the permanent solution is available.
* Add RegistryLockVerifyAction
The action takes two parameters
- isLock is a boolean, determining whether we're trying to lock or
unlock a domain
- lockVerificationCode is the UUID by which we'll look up the lock
object in question.
The lock in question must not be expired and must be in a valid lockable
/ unlockable state
* Some responses to CR
* Add slash and move test method
* Add more data and tests
* Fix screenshot
* Add JPA annotations to class Registrar
* Use array for Java list
* Exclude parent field
* Use 3 columns for address and use text for enum
* Use EnumParameter and 3 properties in Address
* Rename columns and rebase on HEAD
* Get persistence of DomainBase actually working
Fix all of the existing problems with DomainBase persistence:
- Remove "final" keywords on getters that cause errors during startup.
- Remove Transient from creationTime (since there's a converter for
CreateAutoTimestamp)
- Fix DesignatedContext persistence so that it only creates a single table.
This is a lot more efficient given that these are many-to-one with their
domains.
- Add a flyway script, update the golden schema.
- Create a unit test, add it to the integration test suite.
* Changes request in review
* Regenerated generated schema file.
* Changes for review
* Persist status value enum set
* Changes in response to review
* Changes requested in review
* Fixes for #456
* Rename Domain "status" column to "statuses"
* Require tests on all declared JPA entities
The JPA entity classes declared in persistence.xml reflects
JPA's view of the database. The view must be compatible, though
not always consistent with the actual schema. By defining and
using a new JpaIntegrationWithCoverageRule, this change verifies
that all JPA entities in persistence.xml are included in some
tests in the SqlIntegrationTestSuite.
Also removed one test from SqlIntegrationTestSuite since it
does not use Cloud SQL.
Lastly, global resources such as clock should be provided to
a test fixture. Having one fixture creating its own makes it
harder to use in a multi-system integration test. This change
refactored JpaTransactionManagerRule to take a user provided
clock.
* Change reserved messages for domains w/ required allocation token
In domain check reasons for unavailable domains, we were previously saying that
domains were "Reserved" regardless of whether the domain was fully blocked or
reserved for an anchor tenant or other specific use (e.g. the .new Limited
Release Program). This commit changes the message for the latter situations to
be "Allocation token required", so that registrars have a hint that domain will
show as available if the correct allocation token is supplied to the domain
check command using the allocation token extension.
* Allow claims keys on domain creates during quiet periods
Since we're using TldState.QUIET_PERIOD for the .new LRP (in which only reserved
domains with allocation tokens may be registered), we also need to support
claims keys during this phase. Otherwise, domains that appear in MarksDB will
not be registrable during LRP, even if the correct claims key is provided.
This is based on an error report from a registrar, in which a correct-looking
domain create was failing with the error message "Declared launch extension
phase does not match the current registry phase".
* Add test for checking reserved domain with token
* Allow domain checks with a valid token if the domain isn't fully blocked
* Check specifically for specific-use or anchor reservations
* Add test for LRP in-time token
* Refactor lock/unlock commands to use Registry Locks
* CR responses
* Remove unnecessary XML test files
* Add tests
* Respond to CR
* Refactor further the creation and verification of locks
* remove isUnlocked
* Responses to CR
* Fix tests
* Add admin-override back to unlocking
* Add TODO
* Fix imports
* Refactor g.r.model.transaction -> g.r.persistence.transaction
The ultimate goal here is to eventually delete the model package entirely once
we're no longer using Datastore. To that end, the Cloud SQL stuff that we'll be
keeping needs to be in a different package. Here's the rough outline of the
final desired state I'm thinking of:
g.r.persistence
.transaction -- what's being moved in this commit
.converters -- to be moved in another commit
[maybe more?]
g.r.schema
[same-ish subpackages as model currently contains]
I'm also trying to think if it makes sense to move the DAO classes out of
g.r.schema, and haven't come up with a good answer yet.
* Also move test resource files
* Merge branch 'master' into mv-tx-to-persistence
* Check for post-deployment Flyway script changes
Add a script that checks for changes to scripts that
have been deployed to sandbox. This is test in to
be invoked in presubmit and ci.
Extracted common shell functions to an 'rc' file.
Also renamed existing script to be consistent with
other shell scripts.
* Implement a hibernate type for enums
Leverage GenericCollectionUserType to store enum fields as an array of
strings.
* Changes requested in review
* Remove extraneous import
Curiously, this causes checkstyle to fail now that the import was /removed
from a comment./
* Fix exDate calculation in domain transfer flows
* Link ICANN rule for zero-period transfer
* Get rid of some unnecessary method parameters
* Add domain info commands to test case
* Fix flaky SQL integration tests
As a sanity check, the text fixture verifies that all test jdbc connections
have been closed. However, the query to get the connection count is too
general, and may count background admin tasks such as autovacuum and
replication etc, that may appear at any time.
* Upgradle JUnit to 4.13
Removed third_party/junit folder and all usage of the
JunitBackPort class. As a result, third_party is no
longer a Gradle subproject.
Minor code changes were needed to work around an
error-prone pattern: multiple statement in assertThrows'
runnable lambda.
Also third_party/activation and third_party/jsch. These
dependencies are loaded from remote maven repo. The local
copies are not in use.
* Make Jacoco work with multiple test tasks
By default Jacoco only looks at execution data from
the 'test' task. This is a problem to 'core' which
has multiple test sets.
A helper script to run server/schema compatibility tests from
a CI platform. It allows the caller to choose a system under
test (sql or nomulus) and to specify the deployed version
by environment (sandbox or production).
* Implement changes suggested by testcontainer dev
See https://github.com/google/nomulus/issues/401
Specifically:
- Use getContainerIpAddress() instead of localhost to insulate us from
off-machine docker usage.
- Remove shutdown hook to close the container, as testcontainers does this for
us.
* Relax premium list existence check to allow Cloud SQL migration
We need to be able to simultaneously update premium lists that already exist in
Datastore and create them in Cloud SQL (because they haven't been migrated over
yet). This temporarily relaxes the existence check for Cloud SQL so that
"updates" will work even when the list doesn't yet exist there.
* Fix IcannReportingUploadAction to upload reports from the previous month
getFileName now sets the file name of the report to upload to use the month before cursor time.
IcannReportingUploadAction no longer uploads the MANIFEST.txt file since it is not required based on (https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-07#page-9) and the previous implementation of this action did not upload it.
Deletes the ICANN_UPLOAD_MANIFEST cursor since it is no loner needed.
* Add the ICANN_UPLOAD_MANIFEST cursor back
* Fix flakiness in JodaMoneyConverterTest
JpdaMoneyConverterTest relies on Hibernate to deploy its schema.
This introduces an extra jdbc connection in the middle of a test
suite, and may break the connection count checks between tests made
by JpaTransactionManagerRule.
This change updated HibernateSchemaExporter to include Hibernate
proprietary mappings in META-INF/orm.xml.
This change also disabled Hibernate schema push for all tests,
and enabled sql statement logging.
* Support premium list updating in Cloud SQL
This also removes the requirement to specify --also_cloud_sql in nomulus premium
list tooling, instead always persisting to Cloud SQL. It removes a non-USD
premium label in the global test premium list (the Cloud SQL schema doesn't
support a mix of currency units in a single premium list). And it adds a method
to PremiumListDao to grab the most recent version of a given list.
* Merge branch 'master' into premium-lists-always-cloud-sql
* Revert test change
* Create new PremiumListUtils class and refactor out existing method
* Fix tests and update an existing premium price
* Add unlock fields to RegistryLocks
This will make it easier to reason around inter-connected registry lock
objects (like when we add dependent roids). It will make it easier to
answer the question of "Have all locks associated with this host/contact
roid been unlocked?", as well as the question of "Was the last lock
object associated with this domain unlocked?"
* Responses to CR
* Make the DAO API more specific
* whoops, undo rename
* Consolidate certificate supplier module
Both the proxy and the proxy needs certificate suppliers. The PR
consolidates the module that providings those bindings to a shared
module and switched the proxy to use that module. The prober currently
uses P12 file to store its certificates. I am debating keeping that
supplier ro converting them to PEM files for simplicity.
* Rename mode enum values to be more descriptive
* Update annotation names to be more descriptive
* Separate JPA rules for unit and integration tests
Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.
* Separate JPA rules for unit and integration tests
Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.
* Add initial support to write reserved list to Cloud SQL
* Add support to update reserved list in Cloud SQL
* Fix wrong check when override is enabled in create command
* Add sql dependent tests to the suite
* Address comment
* Make invocation of super.execute more readable
* Add test to check upper case and non-puny code label
* Move ReservedListDao related classes to schema package
* Simplify a function name
The promise should be set outside the try block because if we want
warning only, we still want the promise to be set even if the
clientCertificate.checkValidity() throws an error.
* Remove the schema jar url override for tests
Revert pull request 374: "Allow schema-loading from arbitrary url in tests".
Pull request 403 (f48e3933f5) is a more
general solution.
This reverts commit 68887d427f.
* Create a new app to hold GenerateSqlSchemaCommand
GenerateSqlSchemaCommand starts postgresql using testcontainer.
This makes junit etc a runtime dependency, allowing them to get
into release artifacts.
By moving this command to a separate tool, we can remove junit
etc as compile/runtime dependency.
* Fix sql script name conflict
There are two V11__ files due to concurrent merge. Renamed one
to V12__
Also removed a @NotNull annotation, which is the fist in the code base.
Most of the code base use @Nullable instead. If we do want to use
@NotNull, we may want to use the javax one instead.
* Run cross-release SQL integration tests
Run SQL integration tests across arbitrary schema and server
releases.
Refer to integration/README.md in this change for more information.
TESTED=Cloud build changes tested with cloud-build-local
Used the published jars to test sqlIntegration task locally.
* Add Cloud SQL premium list caches and compare prices with Datastore
Nothing will fail if the prices can't be loaded from Cloud SQL, or if the prices
are different. All that happens is that the error is logged. Then, once this is
running in production for awhile, we'll look at the logs and see if there will
be any pricing implications from switching over to the Cloud SQL version of the
premium lists.
* Add setMaxResults(1) per code review
* Add tests and reorder public functions
* Don't statically import caches
* Improve test pass rate
* Merge branch 'master' into dual-read-premium
* Add PremiumEntry mapping
* Allow update
* Revert column order
* Alphabetize PremiumEntry columns
* Don't bother trying to enforce order
* Private constructor
* Make devProject a project property
Properties set in rootProject's ext block are not overridden by
commandline flags.
* Make flyway commands runnable from FOSS repo
rootProject.devProject, defined in projects.gradle, cannot be
overridden by commandline flags. Added a flag check in :db's
project script.
* Add schema for Cursor
* Add CursorDao and CursorDaoTest
* Fix comment on getTld
* Change tld column to scope
* Fix cursorTime to be converted to DateTime internally and other small fixes
* Add a CursorType enum and a createGlobal constructor for Cursor
* Rename flyway file
* Use cursorType from common/Cursor.java and add null checks
* Verify RegistryTool can instantiate
Add a task that instantiates all command classes in RegistryTool
with runtimeClasspath.
Also make sure that runtimeClasspath is a superset of
compileClasspath.
* Adding junit back into the runtime classpath
Unfortunately, GenerateSqlSchemaCommand depends on junit via testcontainers.
We should really move GenerateSqlSchemaCommand out of nomulus tool (we only
use it during development) but this gets nomulus tool working for the time
being.
* Removed unnnecessary trace line.
* lockfiles generated after update_dependency.sh
In some cases, we define JpaTransactionManagerRule in a TestCase
class which is extended by the concrete test class. So, we need
to check if JpaTransactionManagerRule is also defined in the super
class.
Endpoints annotated with AUTH_INTERNAL_ONLY used to be accessible
manually with an internal RPC tool that adds App Engine specific HTTP
headers to a request to make it look like it comes from App Engine
(hence internal). This tool is used by admins to hit such endpoints
during debugging, making them effectively AUTH_INTERNAL_OR_ADMIN.
This RPC tool has never been made available outside Google so the open
source admins do not have such ability. A recent change in the RPC tool
made this hack stop working internally as well. This PR replaces all
all occurances of AUTH_INTERNAL_ONLY with AUTH_INTERNAL_OR_ADMIN and
brings the open source build into feature parity with the internal
version.
Also fixed a few issues the router tests.
* Allow project dependency to use runtimeClasspath
Project dependency should use runtimeClasspath. However, if
left unspecified, it uses 'default', which is the same as
the legacy 'runtime' configuration. (runtimeOnly dependencies
are left out).
Since runtimeClasspath cannot be referenced directly, we use
a custom config (deploy_jar) as a proxy.
By excluding testjars (leaked into 'compile' by third-party
dependencies) from runtimeClasspath, we prevent them from
getting into release artifacts.
Two meaningful changes in appengine_war.gradle and java_common.gradle
TESTED=Diffed contents of services/{module}/build/exploded-*
Only three jars are removed: hamcrest-core, junit, and
mockito-core.
Client SSL handler already performs the necessary validation. Only tests are
added.
Server SSL handler does not currently check for the validity period of
the client certificate as the insecure trust manager is used. This PR
added the check but does not actually terminate the connection yet. It
will log the expired certificates so that we can contact the registrars
to update them.
Once we are certain that all certificates are updated, we can turn off
dryrun mode.
We should also consider checking if the certificate has too long a
validity period as it defeats the purpose of using regularly updated
certificates to deprecate insecure cipher suites.
* Stop publish Cloud SQL schema jar to maven repo
The original purpose of the maven publication is for
use in server/schema compatibility tests. A commandline
flag can direct a test run to use different versions of
the schema jar. However, this won't work due to dependency
locking.
The prober now uses the common SSL initializer in the networking
subproject.
Also changed both initializers to take an ImmutableList of certificates
other than an array of those, for better immutability.
I have no idea where these lockfile changes are coming from. They seem
to be pure noise as far as code review is concerned.
* Add a cursor for tracking monthly uploads of the transaction report to ICANN
* Add cursors to track activity, transaction, and manifest report uploads.
* Address comments
* Add @Nullable annotation to manifestCursor
* Add lock and batch load cursors.
* Add string formatting, autovalue CursorInfo object, and handling for null cursors
* Add some helper functions for loadCursors and restructure to require less round trips to the database
* Switch new cursors to be created with cursorTime at first of next month
* Make dev project configurable
We should not hardcode our dev project in the public config file.
* Remove the use of .ext when using external properties
They are only needed when defining properties.
* Update the Registries cache to leverage/populate the Registry cache
This is accomplished by also providing a loadAll() method on the Registry cache
that can be used to load an entire batch of Registry objects at once.
This improves efficiency, because now, any operation on Registries that loads
all the Registry entities (getTlds(), getTldsOfType(), and getTldEntities()), is
plumbed through the Registry cache, therefore loading it from that cache if it
exists and only hitting the DB if not. If not, this populates the Registry cache
upon loading, so that subsequent calls to Registry.get() will now hit the cache
instead of the DB.
To give a concrete example, the following code:
for (String tld : Registries.getTlds()) {
// ...
doSomethingWith(Registry.get(tld));
// ...
}
is now much more efficient, because the initial call to Registries.getTlds()
populates all the entities in cache, and the subsequent individual calls to
Registry.get(tld) now retrieve them from the cache. Prior to this change,
Registries.getTlds() did not populate the Registry cache, and each subsequent
Registry.get(tld) had the potential to trigger an individual round-trip to the
DB, which is obviously bad for performance.
* Break circular dependency between core and util
Created a new :common project and moved a minimum
number of classes to break the circular dependency
between the two projects. This gets rid of the
gradle lint dependency warnings.
Also separated api classes and testing helpers into
separate source sets in :common so that testing
classes may be restricted to test configurations.
* Allow schema-loading from arbitrary url in tests
Server/Schema compatibility tests must be able to load different versions
of the SQL schema. This change allows test runners to override the
schema location using a system property.
Note: due to dependency-locking, we cannot manipulate the dependencies
closure in the build script to load different schema jars. The jars
must not be on the classpath.
* Add a new method to load all Registry entities of a given type
This is useful for things that need to know more than just the TLD strings
themselves, which is all Registries currently provides.
* Require explict tag when starting psql docker
Defined a util class to return docker tag of desired PSQL version.
Class is defined in ':db' and shared by ':db' and ':core'. Used
an artifact declaration to exclude unnecesary compile dependencies.
Added a presubmit check for instantiations without explicit tag.
* Update schema deployment doc and flyway tool
Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.
Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).
Updated README file on schema push process. Also reformatted
the entire file.
* Add by-repo-id method to RegistryLockDao
When we create new registry lock objects on user request, we will want
to make sure that there are no pending lock/unlock actions on this
domain. In order to do that, we will need to know what lock actions
there have been for this domain.
* Inline the query into a single statement
* whoops
* comments
Replace deprecated bouncycastle class in SslInitializerTestUils.
Generic array as vargs: worked around it in ProbingAction and
removed unused method in CircularList.
Defined Docker image for schema deployment.
Included schema deploymer docker in the Cloud Build release process.
Defined Cloud Build config for schema deployment.
TESTED=Used cloud-build-local to test deployment flow
TESTED=Used docker to test schema deployer image in more ways
Implementation of CircularList is an overkill for its stated
purpose. Remove an unnecessary usage so that the remaining
use case may be refactored in isolation.
* Refactor UI actions to be more reliable
- Created HtmlAction to handle the nitty-gritty of login and setting up
HTML pages
- Created a test to verify that all UI actions implement JSON GET, JSON
POST, or HTML classes
- Move CSS renaming into a utility class
* Move logging of request into HtmlAction
* Comment and wording in exception
* mention JsonGetAction in the comment
* JsonGetAction extends Runnable
* Run both the AppEngineRule and JpaTransactionManagerRule in tests
As previously written, the AppEngineRule wraps the infinite loop of the
server handling requests. This changes the code so that we'll have a
chain that does both the AppEngine management and the JPA transaction
manager management so we can actually use SQL in tests and in the test
server.
* remove log line
We should communicate to the users why this command failed, that they
are not allowed to use discounted allocation tokens on premium domains.
Currently it still fails, but we don't yet tell them why.
* Add a GET action and tests for registry lock retrieval
* Create isVerified method
* Allow lock access for admins even if they're not enabled on the registrar
* Simple CR responses
* Move locks retrieval to the GET action
* add newline at eof
* Switch to using ID
* Improve FilteringTest so it works in all cases
- Disable failOnMatchingTests so we don't fail if we filter out all of the
tests for any given task.
- Add excludeTestCases flag so that we can turn this behavior off for test
tasks that need to run TestCas/TestSuite classes.
- Add includeAllTests() function for test tasks where we don't explicitly
include our required tests.
This makes all test tasks in core FilteringTest, with the exception of the
default test task which now does nothing (it just depends on the other test
tasks).
* Improve wording of excludeTestCases comment
When we add the extra test entity to the current JpaTransactionManagerRule by calling jpaRule.withEntityClass(TestEntity.class) and jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update"), the rule would override the golden database scheme with the schema derived from the current entity class(This is an expected behavior by enabling HBM2DDL_AUTO). This behavior prevents us from detecting breaking changes in ORM classes.
This PR fixed the issue by adding HibernateSchemaExporter to export the DDL script for given extra entity class, and make JpaTransactionManagerRule execute the DDL script to create the corresponding table for the extra entity when initializing the database. This PR also simplified the code when adding extra entity class for testing. For now, you don't need to(and shouldn't) call jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update").
* Move soyutils_usegoog.js out of node_modules
Everytime the npmInstall runs, it removes this file from node_modules.
Move it outside the folder to prevent this from happening.
* Move karma.conf.js and soyutils_usegooge.js
* Move karma.conf.js to be under core
* Release SQL schema in Cloud Build
Tentatively release SQL schema at the same time as the server release.
Publish schema jar to gs://domain-registry-maven-repository/nomulus
and also upload it with server artifacts.
Also removed the Gradle 'version' variable which is not used.
Tested=On cloud-build with a simplified version of
cloudbuild-nomulus.yaml.
* Don't wrap exceptions thrown inside Cloud SQL transactions
There's no reason to wrap them in PersistenceException, and it makes dealing
with thrown exceptions harder as seen in the diffs on test classes in this
commit.
* Add a converter for CurrencyUnits stored in the database
This uses the well-known String representation for currency units. It also
provides a base class for other converters that will be persisting the
toString() representation.
* Add DB and formatting changes
* Add tests, make minor fixes
* Save release tag during deployment
* Save current tag for every environment
Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
* Save current tag for every environment
Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
* Instrument proxy frontend latency metric
Backend latency only captures the time spent waiting for the GAE backend
to respond to a request, which is not representitive of what clients
experience. For instance it does not take into account the time the
proxy spends on processing the requests and the time it takes to send a
response to the client.
This PR adds a metric for frontend latency, which is the time spent on
answering a client request. This should serve as a better proxy for the
latency clients observe.
* Rename AbstractMetrics to BaseMetrics
"Public Domain" license must be reviewed case by case.
Removed blanket acceptance and named accepted dependencies
individually.
Also added a README file to warn about this license and WTFPL.
* Don't retry permanent failures when uploading ICANN monthly reports
There are two kinds of permanent failures that this checks for that we know will
never succeed, so it makes no sense to continue retrying 11 more times before
moving onto the next file to upload. These errors are:
1.
com.google.api.client.http.HttpResponseException: 403
Your IP address xx.xx.xx.xx is not allowed to connect
2.
com.google.api.client.http.HttpResponseException: 400
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><response xmlns="urn:ietf:params:xml:ns:iirdea-1.0"><result code="2002"><msg>A report for that month already exists, the cut-off date already passed.</msg><description>Date: 2019-09</description></result></response>
In order to implement this new functionality, this commit also adds a new way to
call Retriable that allows specifying the isRetryable Predicate (which is quite
useful).
* Reenable JpaTransactionManager for Alpha and Crash
* Make UploadClaimsListCommand implement CommandWithCloudSql
* Fix wrong call to get password
* Use Cloud SQL Socket library to provision TransactionManager
* Change to use dataSource configs
* Add another test contact for Registry Lock testing
Previously, we only had two contacts -- one per registrar. This PR adds
a second, registry-lock-enabled, contact to one registrar for two
reasons:
1. For registry-lock-related testing, we'd like to be able to test both
positively and negatively, making sure that the permissions work the way
they should
2. In general, the UI tests should include the case where we have
multiple contacts in the same registrar. Previously, this was never the
case in tests.
* Merge remote-tracking branch 'origin/master' into addTestContact
* Don't destroy existing registry lock passwords in contacts
The existing code assumes that the "contacts" segment of the form
contains an exact representation of the registrar contacts. This breaks
when we have a contact with an existing registry lock password because
we don't want to keep passing around that password in plain text (we
never store it in plain text)
This PR changes the code so that instead of assuming the contact is
provided in its entirety, we load the contact from storage first
(matching by email address) if it exists. We then set the required
fields from the JSON object, and set the password optionally if it was
provided.
Alternatives:
- Create a separate RegistrarContactPassword object with a
RegistrarContact parent. This increases complexity significantly since
we'd be adding a parent-child relationship and adding more objects to
Datastore during the transition to SQL. It also doesn't completely solve
the problem of "When should we set the password?" because the password
field still must be part of the same form.
- Rearrange the UI so that the password is set as part of a completely
separate form with a separate submit action. This would be possible but
is sub-optimal for two reasons. First, we are trying to not re-engineer
the web console as much as possible since we're likely starting it from
scratch before too long anyway. Second, we want the
lock-password-setting to be part of the standard contact modification
workflow.
* Responses to CR
* Actually we need to allow "removal" of fields
* Remove optional
* one-statement building the contacts
We don't want to override toDiffableFieldMap because (per the javadoc)
that is supposed to contain sensitive information. So, we should just
remove it before sending it out.
* Use base64-encoded SQL credentials
Encode Cloud SQL credential files on gcs with base64,
to be consistent with our Cloud Build practices.
Also renamed a property that specifies where to publish
the schema jar. New name is schema_publish_repo.
The segregated test targets in core break the --tests filter. Fix this by
defining a "testFilter" property and creating the FilteringTest task type that
applies it to the property set by "--tests".
* Make Flyway schema task work with prod and sandbox
Also renamed the 'superuser' role to 'admin' since
we do not own super user in Cloud SQL.
* Allow pushing schema to all env with Flyway
Desktop schema push to production is needed in the short term.
Long-termly we need to decide if this should be kept for glass
breaking
Schema push to sandbox and production requires interactiveconfirmation.
Also fixed a typo in initialize_roles.sql.
* Add Bloom filters to the Cloud SQL PremiumList schema
They are slightly different from the existing Bloom filters stored in Datastore
in that they now use an ASCII String encoding rather than the more generic
CharSequence, and there is no maximum size (whereas we previously had to live
within the 1 MB max entity size for Datastore).
* Modify Cloud SQL user management scripts
Create readonly and readwrite roles that may be granted to users.
Also configured default privileges for tables created in the future.
Made sure arbitrary users may not create database or tables.
* Modify Cloud SQL user management scripts
Create readonly and readwrite roles that may be granted to users.
Also configured default privileges for tables created in the future.
Made sure arbitrary users may not create database or tables.
* Load persistence.xml classes before adding test entities
* Also use persistence.xml in GenerateSqlSchemaCommand
* Add exception message
* remove duplicate line
* Add initial support for persisting premium lists to Cloud SQL
This adds support to the `nomulus create_premium_list` command only; support for
`nomulus update_premium_list` will be in a subsequent PR.
The design goals for this PR were:
1. Do not change the existing codepaths for premium lists at all, especially not
on the read path.
2. Write premium lists to Cloud SQL only if requested (i.e. not by default), and
write to Datastore first so as to not be blocked by errors with Cloud SQL.
3. Reuse existing codepaths to the maximum possible extent (e.g. don't yet
re-implement premium list parsing; take advantage of the existing logic), but
also ...
4. Some duplication is OK, since the existing Datastore path will be deleted
once this migration is complete, leaving only the codepaths for Cloud SQL.
* Refactor out common logic
* Add DAO test
* Add tests for parsing premium lists
* Use containsExactly
* Code review changes
* Format
* Re-generate schema
* Fix column names
* Make some tests pass
* Add SQL migration scripts
* Fix test errors
* Add an explanation to dummied-out JPA init
Add a more elaborate explanation of why actual JpaTransactionManager
initialization was removed from the factory.
* Add a DAO for RegistryLock objects
* Add an index on verification code and remove old file
* Move to v4
* Use camelCase in index names
* Javadoc fixes
* Allow alteration of RegistryLock objects in-place
* save, load-modify, read in separate transactions
* Change the creation timestamp to be a CreateAutoTimestamp
* Add persistence.xml to the war files
* Always use the DummyJpaTransactionManager
Use the DJTM until we get all of the dependencies set up for all of the
environments.
This shouldn't affect any of the unit tests, these use the
JpaTransactionManagerRule to set up a local database and connection.
This fixes the App Engine build.
* Implement CreateAutoTimestampConverter
Implement a JPA-based converter for CreateAutoTimestamp, allowing us to
persist instances of this class.
Note that converters appear to be required to convert to and from database
types that are generally known to JDBC. For example, conversion to Timestamp
works, conversion to OffsetDateTime does not (even though this works through
the JDBC interface directly).
* Improve error handling for environment property
Improve the error messages that we get for a bad or missing environment
property. Move the property processing into the main build so that we do it
only once and configure all of the appengine deployment tasks to check that a
project has been defined and print a friendly error if it hasn't.
Note that even if the check isn't configured, this change prevents deployment
because the gcpProject will be set to null. It just won't print as useful an
error message.
Tested: ran appengineDeployAll with and without the environment property, ran
"build" to verify that we don't get any complaints for non-deployment targets.
* Changes for review.
* Changes for review
* Changes for review.
* Changes for review.
* Give JpaTransactionManagerRule more parameters
Allow users of the rule to add annotated classes and properties, both useful
for testing.
* Change in response to review.
* Changes for review.
* Move test EntityManagerFactory create method
Move the test create method into the JpaTransactionManagerRuleTest.
* Remove nomulus SQL dialect from G.S.S.Command
Remove NomulusPostgreSQLDialect from GenerateSqlSchemaCommand (it has been
moved to its own top-level class).
* Add maven-publish task for SQL schema jar
Add task to publish SQL schema jar with flyway scripts and
golden schema to a maven repo. This will be used
for pre-release testing in the future.
This task is not part of build and needs to be invoked explicitly.
User needs to provide schema_jar_repo and schema_version
properties.
* Merge branch 'master' of https://github.com/google/nomulus into publish-schema-jar
* Add maven-publish task for SQL schema jar
Add task to publish SQL schema jar with flyway scripts and
golden schema to a maven repo. This will be used
for pre-release testing in the future.
This task is not part of build and needs to be invoked explicitly.
User needs to provide schema_jar_repo and schema_version
properties.
Export of cl/270900150.
To refer to a KMS key or key ring, we should use the stable `.self_link`. Using `.id` instead provides an unstable identifier which may change (and it will change in the upcoming update of the google provider to 2.9.1). A change in the identifier will cause Terraform to destroy and recreate the key. Destroying the key means all data associated with it is lost; the key cannot be recreated.
This CL replaces `.id` with `.self_link`, so all of those problems will not happen. In addition, `prevent_destroy` protects the key against delete-and-recreate in general.
* Upgrade to Truth 1.0
Refactored fail(...) to assertWithMessage().fail().
Upgraded com.google.monitoring-client family of dependencies to 1.0.6
Also fixed bad use of io.StringIO (on binary buffer) recently introduced to
google-java-format-diff.py.
For some reason tools.jar (which we don't need in the runtime in the first
place) gets exploded readonly, causing subsequent builds to fail since they
can't overwrite it. This hack makes it writable at the end of explodeWar.
* Fix dependency-locking config
Reenable dependency locking after a bug errorneouly turned it off.
Removed the guava-related workaround that forcefully resolve to
the -jre distribution.
Enabled locking for buildSrc by updating its property file.
Updated all lock files.
* Output command test output as well as consuming it
CommandTestCase currently consumes stdout & stderr for the command being
tested. Unfortunately, this results in us not being able to see the command
output. Add an output splitter so that output gets written to the original
stream in addition to being captured.
A simpler approach would be to print the captured data after command
completion. However, this won't work for tests that become hung and also
won't display results in real-time.
Tested: Ran a command test with verboseTestOutput=true, verified that standard
output was visible.
* Save and restore original stdout/err in cmd tests
We have to restore the original stdout/stderr print streams otherwise we end
up nesting them across tests which eventually causes the RDE tests to OOM.
* Add RegistryLock schema to Flyway deployment folder
Added creation script of RegistryLock to Flyway deployment folder.
Fixed previous scripts (PremiumList- and ClaimsList-related) for
FK name change (cause by table name changes: names are quoted now).
We should consider generating foreign key names by ourselves.
Since the alpha database is empty, we dropped and recreated the schema.
Added instructions on how to submit new database incremental changes
in the README file.
Updated RegistryLock.java, removing unnecessary annotations:
- For most fields, the 'name=' property is no longer necessary not that
the naming strategy is in place. The exceptions are the two used in
the unique index.
- The @Column annotation is implicit.
* Force Gradle to use jre version of Guava
Guava is often specified by version only by other
dependencies. In such cases, Gradle usually resolves
it to the '-android' version, which lacks the collection
classes.
We use custom resolution strategy to force the selection
of the '-jre' version.
* Add schema deployment tests
Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.
Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.
Upgraded testcontainers to 1.12.1.
Added a custom Truth subject for better diffing of multi-line
text blocks.
Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
* Add schema deployment tests
Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.
Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.
Upgraded testcontainers to 1.12.1.
Added a custom Truth subject for better diffing of multi-line
text blocks.
Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
* Add schema deployment tests
Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.
Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.
Upgraded testcontainers to 1.12.1.
Added a custom Truth subject for better diffing of multi-line
text blocks.
Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
* Regenerate schema using tools command
Rerun GenerateSqlSchemaCommand to pick up RegistryLock
and naming strategy change.
Also updated a new license term which seems to just pop up.
* Add RegistryLock SQL schema
* Refactor a bit
* Move registrylock -> domain
* Clearing up lock workflow
* Add more docs and remove LockStatus
* Responses to CR
* Add repoId javadoc
* Add registry lock to persistence xml file
* Quote rather than backtick
* Remove unnecessary check
* File TODO
* Remove uniqueness constraint on verification code
* Remove import
* add index
* Add to SQL generation task
* Move fields around to be the same order as Hibernate's generated sql
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.
Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Corrected the type of ClaimsEntry's revision_id column.
It should be plain int8, not bigserial.
Make GenerateSqlSchemaCommand use a custom dialect that
converts all varchar type to 'text' and timestamp to
'timestamptz'.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.
Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.
Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
* Re-add other schema classes
* Add Cloud SQL schema for premium lists
This won't work quite yet, pending a solution for the type translator issue
(which will be needed for the currency field, and potentially others).
* Generate basic schema for all of DomainBase
Generate a basic schema for DomainBase and everything that is part of it.
This still isn't complete, in particular it lacks:
- Correct conversions for problematic types (e.g. DateTime, Key...)
- Schema generation for history records.
- Name translation.
* Make parameter names in generate_sql_schema command consistent
The rest of the nomulus commands use underscores for delimiting words in
parameter names, so this should too.
Also fixed capitalization of some proper nouns.
* Move EntityManagerFactoryProviderTest to fragile
* Add EMF Provider Test to docker tests
Add EntityManagerFactoryProviderTest to the docker incompatible test patterns
and use the latter list to compose the fragile tests.
* Start postgresql container in generate_sql_schema
Add a --start-postgresql option to the nomulus generate_sql_schema command so
that users don't have to start their own docker container to run it.
* Made default behavior be to give guidance
* Don't write TX records for domains deleted in autorenew grace period
When the project was originally being designed, we envisioned have a purely
point-in-time architecture that would allow the system to run indefinitely
without requiring any background batch jobs. That is, you could create a domain,
and 10 years later you could infer every autorenewal billing event that should
have happened during those 10 years, without ever having to run any code that
would go through and retroactively create those events as they happened.
This ended up being very complicated, especially when it came to generating
invoices, so we gave up on it and instead wrote the
ExpandRecurringBillingEventsAction mapreduce, which would run as a cronjob and
periodically expand the recurring billing information into actual one-time
billing events. This made the invoicing scripts MUCH less complicated since they
only had to tabulate one-time billing events that had actually occurred over the
past month, rather than perform complicated logic to infer every one-time event
over an arbitrarily long period.
I bring this up because this architectural legacy explains why billing events
are more complicated than could otherwise be explained from current
requirements. This is why, for instance, when a domain is deleted during the 45
day autorenewal period, the ExpandRecurringBillingEventsAction will still write
out a history entry (and corresponding billing events) on the 45th day, because
it needs to be offset by the cancellation billing event for the autorenew grace
period that was already written out synchronously as part of the delete flow.
This no longer really makes sense, and it would be simpler to just not write out
these phantom history entries and billing events at all, but it would be a
larger modification to fix this, so I'm not touching it here.
Instead, what I have done is to simply not write out the DomainTransactionRecord
in the mapreduce if the recurring billing event has already been canceled
(i.e. because the domain was deleted or transferred). This seems inconsistent
but actually does make sense, because domain transaction records are never
written out speculatively (unlike history entries and billing events); they
correspond only to actions that have actually happen. This is because they were
architected much more recently than billing events, and don't use the
point-in-time hierarchy.
So, here's a full accounting of how DomainTransactionRecords work as of this commit:
1. When a domain is created, one is written out.
2. When a domain is explicitly renewed, one is written out.
3. When a domain is autorenewed, one is written out at the end of the grace period.
4. When a domain is deleted (in all cases), a record is written out recording the
deletion.
5. When a domain is deleted in the add grace period, an offsetting record is
written out with a negative number of years, in addition to the deletion record.
6. When a domain is deleted in the renewal grace period, an offsetting record is
likely written out in addition.
7. When a domain is deleted in the autorenew grace period, there is no record that
needs to be offset because no code ran at the exact time of the autorenew, so
NO additional record should be written out by the expand mapreduce.
*THIS IS CHANGED AS OF THIS COMMIT*.
8. When a domain is transferred, all existing grace periods are cancelled and
corresponding cancelling records are written out. Note that transfers include a
mandatory, irrevocable 1 year renewal.
9. In the rare event that a domain is restored, all recurring events are
re-created, and there is a 1 year mandatory renewal as part of the restore with
corresponding record written out.
So, in summary, billing events and history entries are often written out
speculatively, and can subsequently be canceled, but the same is not true of
domain transaction records. Domain transaction records are only written out as
part of a corresponding action (which for autorenewals is the expand recurring
cronjob).
* rm unused import
* Remove the "showAllOutput" property from the build
It doesn't work very well and has been superseded by "verboseTestOutput",
which does the same thing and more.
* Remove 'value' from RDAP link responses
* Change application type to rdap+json
* Merge remote-tracking branch 'origin/master' into removeValueRdap
* CR response
* Only apply Google Java format to changed lines
* Only apply Google Java format to changed lines
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
* Only apply Google Java format to changed regions
Diffs are relative to origin/master.
Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
javaIncrementalFormatApply is invoked.
These tasks work from the root directory and process the buildSrc directory
too.
The Spotless Java config is removed.
It is burdensome to have to maintain two sets of tools, one of which
contains a strict subset of functionalities of the other. All admins
should use the same tool and their ability to administer should be
restricted by the IAM roles they have, not the tools they use.
* Add a registry lock password to contacts
* enabled -> allowed
* Simple CR responses, still need to add tests
* Add a very simple hashing test file
* Allow setting of RL password rather than directly setting it
* Round out pw tests
* Include 'allowedToSet...' in registrar contact JSON
* Responses to CR
* fix the hardcoded tests
* Use null or empty rather than just null
* Add a generate_schema command
Add a generate_schema command to nomulus tool and add the necessary
instrumentation to EppResource and DomainBase to allow us to generate a
proof-of-concept schema for DomainBase.
* Added forgotten command description
* Revert "Added forgotten command description"
This reverts commit 09326cb8ac.
(checked in the wrong file)
* Added fixes requested during review
* Add a todo to start postgresql container
Add a todo to start a postgresql container from generate_sql_command.
* Added MetricsHandler and Clock to ProbingSequences
* Minor fixes after rebase onto master
* Added metrics gathering to ProbingSequences
* Added testing of MetricsCollector method calls in ProbingSequenceTest
* Added tests on latency recording to ProbingSequenceTest
* Added response name as label to metrics
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Minor syle fixes after succesful rebase onto master
* Initial Commit.
* Added Protocol Class and its Basic Unit Tests
* Fixed Gitignore to take out AutoValue generated code
* Final Changes in .gitignore
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Removed AutoVaue ignore clause from .gitignore
* Rebased to Master and added in modified Handlers and ProbingAction
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Modified license header dates
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Fixed files to pass all style tests
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Added circular linked list to utils
* License Header added
* Refactored probing sequence to be circular linked list iterator
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Added circular linked list to utils
* Added circular linked list to utils
* License Header added
* License Header added
* Refactored probing sequence to be circular linked list iterator
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Added missing license header to DefaultCircularLinkedListIterator
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Fixed max column length to be 100
* Rebased to Master and added in modified Handlers and ProbingAction
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Added circular linked list to utils
* Added circular linked list to utils
* License Header added
* License Header added
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* ProbingStepTest modified to have fewer unnecessary helper methods
* Updated issues in rebasing
* Fixed max column length to be 100
* Minor changes to pass style tests
* Successful rebase onto finished web-whois branch
* Removed need for TestTokens with Mockito mocks of Tokens
* Fixed style issues in DefaultCircularLinkedListIterator and AbstractCircularLinkedListIterator
* Modified CircularList according to changes suggested by jianglai.
* Added Protocol Class and its Basic Unit Tests
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Updated build.gradle file
* Updated WebWhois tests.
* Added Basic EPP structure
* Added Basic EPP structure
* Prober Updated tests
* Prober Updated tests
* Fully functioning EPP sequences with modified WebWhois base.
* Fully functioning EPP sequences with modified WebWhois base.
* Added Modified test server infrastructure.
* Added Modified test server infrastructure.
* Allowed ActionHandler to pass status to next hanlder in pipeline (to be MetricsHandler).
* Allowed ActionHandler to pass status to next hanlder in pipeline (to be MetricsHandler).
* Javadoc on EppRequestMessage added
* Javadoc on EppRequestMessage added
* Updated EppServer to properly send successful Check responses.
* Updated EppServer to properly send successful Check responses.
* Allowed for expected failures in EPP actions.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Fully rebased branch to prober-web-whois after refactoring
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Rebased to master
* Fixed max column length to be 100
* Fixed files to pass all style tests
* Minor changes to pass style tests
* Successful rebase onto circular-list
* Epp Refactored to accomodate circular linked list PR
* Modified construction of Epp Probing Sequences to reflect CircularList change
* Renamed ProberModule provided Duration
* Removed unnecessary ServerSideException file
* Google-Java-Format run on all prober files
* Style fix on ProbingSequence and its unit tests
* Removed subclasses of EppRequestMessage and EppResponseMessage and fixed style and other minor issues
* Style changes implemented as suggested by jianglai
* Added style fixes suggested by mindhog
* Fixed changes suggested by CydeWeys
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Updated build.gradle file
* Modified license header dates
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Initial Commit.
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* fixed build issues
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Minor Style Fixes
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Rebased to master
* Updated issues in rebasing
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Fixed files to pass all style tests
* Fixed files to pass all style tests
* Minor syle fixes after succesful rebase onto master
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* Added circular linked list to utils
* Added circular linked list to utils
* Added circular linked list to utils
* Added circular linked list to utils
* Added circular linked list to utils
* License Header added
* License Header added
* License Header added
* License Header added
* License Header added
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Refactored probing sequence to be circular linked list iterator
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Modified ProbingStep tests to reflect new ProbingStep structure.
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* ProbingStepTest modified to have fewer unnecessary helper methods
* ProbingStepTest modified to have fewer unnecessary helper methods
* Updated issues in rebasing
* Updated issues in rebasing
* Added missing license header to DefaultCircularLinkedListIterator
* Fixed max column length to be 100
* Fixed max column length to be 100
* Minor changes to pass style tests
* Successful rebase onto finished web-whois branch
* Removed need for TestTokens with Mockito mocks of Tokens
* Fixed style issues in DefaultCircularLinkedListIterator and AbstractCircularLinkedListIterator
* Modified CircularList according to changes suggested by jianglai.
* Merge branch 'master' into prober-circular-list
* Modified ProbingSequenceTest to not expect unnecessary NullPointerException
* ProbingSequence and tests modified to reflect addition of UnrecoverableStateException and restarts on failures
* Modified ProbingSequence and its tests to reflect action generation and calling being put in the same try catch block
* Clean up token generation
- Allow tokenLength of 0
- If specifying a token length of 0, throw an error if numTokens > 1
* Allow generation of 0-length strings
* Allow for --tokens option to generate specific tokens
* Revert String generators and disallow 0 'length' param
* Add verifyInput method and batch the listed tokens
* Check the number of tokens created
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Fixed changes suggested by CydeWeys
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Fixed changes suggested by CydeWeys
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Full WebWhoIs Sequence Added
* fixed build issues
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Fixed changes suggested by CydeWeys
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Fixed Travis Build Issues
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Fixed changes suggested by CydeWeys
* Rebased to Master and added in modified Handlers and ProbingAction
* Added missing license headers and JavaDoc
* Added missing license headers and JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor fix in NewChannelAction JavaDoc
* Minor Style Fix
* Minor Style Fix
* Full WebWhoIs Sequence Added
* Full WebWhoIs Sequence Added
* fixed build issues
* fixed build issues
* Refactored by responses suggested by jianglai.
* Refactored by responses suggested by jianglai.
* Minor Style Fixes
* Minor Style Fixes
* Updated build.gradle file
* Updated build.gradle file
* Modified license header dates
* Modified license header dates
* Updated WebWhois tests.
* Updated WebWhois tests.
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring
* SpotlessApply run to fix style issues
* SpotlessApply run to fix style issues
* Added license header and newline where appropriate.
* Added license header and newline where appropriate.
* Javadoc style fix in tests and removed unused methods
* Javadoc style fix in tests and removed unused methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Refactored ProbingAction to minimize number of unnecessary methods
* Modified tests for WebWhois according to changes suggested by laijiang.
* Modified tests for WebWhois according to changes suggested by laijiang.
* Removed TestProvider from TestUtils.
* Removed TestProvider from TestUtils.
* Rebased to master
* Updated issues in rebasing
* Minor style change on prober/build.gradle
* Fixed warnings for java compilation
* Fixed files to pass all style tests
* Minor syle fixes after succesful rebase onto master
https://github.com/google/nomulus/pull/129 migrated `GoogleCredential`
to `GoogleCredentialsBundle` and introduced a subtle bug. I don't fully
understand why but there are times when the access token is null but
`credentials.refresh()` is not called, resulting in NullPointerException
when `credentials.getAccessToken().getTokenValue()` is called.
Since the new GoogleCredentials class supports `shouldRefresh()`, we now
just rely on it to make sure that we always get a value access token.
This PR created the new interface named TransactionManager which defines
methods to manage transaction. Also, the access to all transaction related
methods of Ofy.java are restricted to package private, and they will be exposed
by DatastoreTransactionManager which is the datastore implementation of
TransactionManager.
* Generate code coverage report
Enable jacoco, the official Gradle code coverage plugin.
The 'build' task will write a code coverage report to
build/reports/jacoco for each subproject that has tests.
We should consider publish periodical reports to a well known
location.
This change also defines a minimum coverage verification task.
The task is for experiment only, and is not added to the build
process yet.
* Remove unused log argument
* Use the right accept-encoding
By default we request gzip and theoretically we'd decode it
automatically on our end but for some reason that's not working. I
tested this on Alpha and it worked
* Create a Gradle task to run the test server
As an artifact of the old build system, the test server relies on having
the built registrar_(bin|dbg)*(\.css)?.js in place (see ConsoleUiAction
among others). As a result, we create a Gradle task that puts those
files into the correct, readable, location before running the test
server.
* Depend on assemble rather than build
* refactor gitignores
The file name confused the licensee app
(https://github.com/licensee/licensee) that GitHub uses to detect
license files. It thinks this file is also a license file and was not
able to determine is type.
Login failures will happen any time that we aren't coming from a
whitelisted IP for that particular TLD. Since whitelists are out of date
(and we don't whitelist IPs for every TLD anyway) those failures aren't
interesting. Store and fully-log the interesting failures if one
happened.
Using the new GoogleCredentials to access Drive API caused 403 forbidden
exception. So, this PR brought back the old GoogleCredential to
temporarily resolve the production issue while we are figuring out the
long term fix.
TESTED=Deployed to alpha and verified exportPremiumTerms succeeded, see
https://paste.googleplex.com/6153215760400384.
* Fail gracefully when copying detailed reports
When the detailed reports are copied from GCS to registrars' Drive
folders, do not fail the entire copy operation when a single registrar
fails. Instead, send an alert email about the failure, and continue to copy the
rest of the reports.
Also, instead of creating duplicates, overwrite the existing files on
Drive.
BUG=127690361
* Merge beam and GAE configs deployment to one GCB job
Deployment of GAE configs requires that the credential used by gcloud to
have GAE admin role of the project to be managed. We do not want to
grant the GCB service account that role, because it would all *any* GCB
job to deploy anything to GAE. Instead we use a dedicated credential
originally created to deploy beam pipelines. This credential is
encrypted by KMS and stored on GCS. Since the beam pipeline deployment
GCB job already does the decryption, it make sense to add the config
deployment step there as well. The beam deployment steps are tweaked to
use the nomulus tool docker image instead of the jar file.
Also moved the content of deploy_configs_to_env.sh to the GCB yaml file
itself because the shell script is not uploaded to GC Bat the same time
as the yaml file when the job is triggered by Spinnaker.
Lastly, due to b/137891685, using GCB to deploy cron jobs does not work
as we cannot use service account credential to deploy to projects under
google.com.
* Add a Cloud Build task to update YAML configs
* CR responses
* Move config deployment to a script
* Pin builder version
* Create different beam and deploy-config files per environment
* Update comments and make a for loop
* Build docker image of nomulus tool
In the course of "gradle build", build a docker image of nomulus tool so that
users can run this to allow us to bundle the java version with the image.
* Don't extend expiration times for deleted domains
* Flip order and add a comment
* oops forgot a period
* Use END_OF_TIME
* Add tests for expiration times of domains with pending transfers
* Add test for transfer during autorenew and clean up other tests
* Clarify tests
* Add domain expiration check in EppLifecycleDomainTest
* Add a comment and format test files
* Initial Commit.
* Deleted unfinished features. Added ActionHandler and its Unit Tests.
* Included prober subproject in settings.gradle
* Added Protocol Class and its Basic Unit Tests
* Added Changes Suggested by jianglai
* Fixed Gitignore to take out AutoValue generated code
* Removed AutoValue java files
* Added gitignore within prober
* Removed all generated java
* Final Changes in .gitignore
* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class
* Fixed build.gradle changes requested
* Removed Files irrelevant to current pull request
* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle
* Fully Updated ActionHandler (missing updated JavaDoc)
* Added changed Protocol and both Inbound and Outbound Markers
* Removed AutoVaue ignore clause from .gitignore
* removed unneccessary dependencies in build.gradle
* Fixed Javadoc and comments for ActionHandler
* Fixed comments and JavaDoc on other files
* EOL added
* Removed Unnecessary Files
* fixed .gradle files styles
* Merge remote-tracking branch 'upstream/master'
* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces
* Fixed javadoc for Marker Interfaced
* Modified Comments on ActionHandler
* Removed LocalAddress from Protocol
* Fixed Travis Build Issues
* Save for later
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Check for unused dependencies
Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
* Add a metric for EPP processing time regardless of ID/TLD
* Change name to request_time
* Record EPP processing time by traffic type
* grammar
* request type
* semicolon
* Check dependencies' open-source licenses
Use jk1/Gradle-License-Report to verify that all
dependencies have open-source licenses.
Note that the following dependencies are not checked:
Dependencies of official Gradle plugins
Dependencies declared in buildscript block
Dependencies of jk1/Gradle-License-Report itself
* Remove the maybeRuntime configuration
It contains dependencies present in the bazel
build but not needed for compile. We now know
they are not needed in runtime either.
Sometimes, the webdriver tests get stuck forever for no reason. It could
be some issue in the test container but it is hard to root cause it. So,
adding a 30s timeout can either trigger the retry earlier or let the
test just fail.
This PR prevents Gradle from copying the golden images
to build/resources/test, so the screenshot test would
read golden images from src/test/resources directly and
display the path in test log if the test fails. Because
the path pointing to the actual file in src/ folder,
the engineer can easily find it.
We got non-serialization object error when deploying the invoicing
pipeline. It turns out that Beam requires every field in the pipeline
object is serilizable. However, it is non-trivial to make
GoogleCredentialsBundle serilizable because almost all of its
dependency are not serilizable and not contraled by us. Also,
it is non-necessary to inject the credential as the spec11
pipeline also writes output to GCS without having injected
credential. So, removing the injected variable can solve the
problem.
TESTED=First reproduced the problem locally by deploying the invoicing pipeline with the previous code; applied this change and successfully deploy the pipeline without having any issue.
* Attempt login to MosAPI via all available TLDs
There's no reason why we should need a TLD as input here because it
doesn't actually matter which one we use (they all have the same
password).
* Refactor the TLD loop and change cron jobs
* Re-throw the last exception if one exists
* Fix tests and exception
* Remove alpha cron job
* Move test resource files into src/test/resources
* fix a test
* Remove references to javatests/ in Java files
* fix import order
* fix semantic merge conflict
* Throw a more useful error message on attempted domain restore reports
Per DomainRestoreRequestFlow's Javadoc, we automatically approve and instantly
enact all domain restore requests, thus we don't use or support restore
reports. This improves the registrar-visible error message to help make this
more clear.
Replace deprecated GoogleCredential with new lib
This PR also introduced a CredentialsBundle class to carry
HttpTransport and JsonFactory object which are needed by
most of the GCP library to instantiate client.
* Refactor the license check to be a generic presubmit check
This includes all of the old presubmits that were in our METADATA file
that are still valid.
* Responses to CR
Added docstrings, refactored the file finder, changed variable names,
and formatted the file
* More respones to CR
We found that some webdriver tests failed because RegistryEnvironment
was set to 'production' by other test and was carried over to the
webdriver test, and it was nontrivial to fix them because the instance
of RegistryEnvironment was injected into the testing web server.
Also, to eliminate this problem from potential victims, we stopped
provisioning RegistryEnvironment from Dagger. Instead, we just use
RegistryEnvironment.get() function to get the instance, which indeed
retrives the value from system property every time. If any test case
needs to run the test with other environment, it can just simply use
the existing SystemPropertyRule and RegistryEnvironment.${ENV}.setup
to change it.
* Update GCB beam deployment pipeline
Some of the texts are not really secerts because they are per-project.
Also changed the location of the credential file to `secerts` so that in
the future we may add more secerts in that folder.
The encrypted file is base64 encoded, consistent with how the proxy
certificates are encoded. Also made some changes to the other pipelines
to facilitate automation with Spinnaker
* Move terraform and kubernetes folder to be under proxy
There is no reason for them to be under proxy/src/... any more now that
we have a Gradle-idiomatic folder structure.
The check for environment cannot be done when the task is defined,
otherwise running any task will fail when the check does not pass.
Instead, check the environment when the task executes.
Our CI (Travis & Kokoro) runs "gradle build", so we need to make
sure that all tasks used in the release process are called during
the build so that breakage can be caught earlier.
In order to stage the GAE folder we need gcloud to be present.
Therefore the Travis config is changed to install gcloud.
See: https://gist.github.com/mjackson/5887963e7d8b8fb0615416c510ae8857
IntelliJ is complaining when this annotation is used on a base class that has no
actual runnable tests itself. The solution is different for different classes
depending on the existing pattern of where the @RunWith annotation is; for some,
it's simplest to move the annotation down to the few extending classes that are
missing it, whereas in others it's easiest just to annotate the base class as
abstract.
Fix or suppress deprecation warnings except those about GoogleCredential,
which is being handled separately.
The @SuppressWarnings("deprecation") annotation does not cover imports
even when it is at the class level. We removed imports of deprecated
classes and use their fully qualified names in class body.
* Consolidate testcontainer used by WebDriver test
Previously, we hard coded the version of the docker image for
provisioning Chrome browser and WebDriver server because the
version of the Chrome browser has to match the version of
the webdriver client, otherwise the screenshot test will fail.
Changing to use BrowserWebDriverContainer can delegate the match
to the library itself because it chooses the correct docker image
version based on the WebDriver version on our classpath.
* Increase maxColorDiff to 20
This is to supress the test flakness after switching to use
BrowserWebDriverContainer to provision browser and webdriver
service.
I'm seeing odd errors in versions of Karma more recent than 3.0.0. While
I investiage, I'd like to see what Github has to say about these
dependencies (we got some vulnerability messages for earlier versions)
This option causes Intellij build to fail if the
'Delegate IDE build/run actions to gradle' box is checked.
We do not know of anyway to change the Intellij behavior.
This change allows an IDE user to prevent -Werror to be passed
to javac by adding a Gradle VM option: -Dno_werror=true
* Add a link to kokoro
This allows us to click on the build badge to visit kokoro and find
out build status. The link will redirect to a status page that is
only accessible to Googlers.
Our existing uber jar task overwrote duplicate files when merging JARs,
and this would break the application using Java SPI service(Beam library
is one of them) so we changed to use ShadowJar plugin as it provides a
convenient way to merge META-INF/services/* files when generating the
uber jar.
TESTED=Deployed spec11 pipeline successfully with the new nomulus.jar;
also randomly checked a few nomulus commands to make sure they are still
working.
We aren't turning on the cron job yet - we want to make sure it works correctly first.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252726789
CharMatcher.isLetterOrDigit() is deprecated for failing to support
supplementary characters. We explicitly declare a matcher for ascii
letters and digits.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252671830
With https://github.com/spinnaker/spinnaker/issues/4048 Spinnaker now natively supports GCB. We are able to start a GCB job from Spinnaker, and also there is better support to consume GCB pub/sub messages. Some changes are made to remove the workaround no longer needed.
Two new workflows are added, one to rsync a GCS folder to live/ after the deployment is done (so that the nomulus.jar file can then be fetched to x20 by a []cron job), and the other to tag the proxy image as live once it is deployed.
Lastly, the docs/ folders are needed when running tests. Remove it from .gcloudignore so that when a test run is kicked off by running "gcloud builds submit" the folder is sent to GCB. Ideally .gcloudignore should be identical to .gitignore but since they both are version controlled it is hard it make one a symlink of another.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252625901
To do this we add a field of "rdapServers" in the Registrar object. Currently, we can only set this field manually, but a subsequent CL will add a cron-job to read these values from the ICANN servers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252438618
Original change by cpovirk@cpovirk:rosie251284456-0055_Rosie:31511:citc on 2019/06/04 09:48:27.
Update to Truth 0.45, and address deprecations.
Renames may include:
- containsAllOf => containsAtLeast
- containsAllIn => containsAtLeastElementsIn
- isSameAs => isSameInstanceAs
- isOrdered => isInOrder
- isStrictlyOrdered => isInStrictOrder
The other major change is to change custom subjects to extend raw Subject instead of supplying type parameters. The type parameters are being removed from Subject. This CL will temporarily produce rawtypes warnings, which will go away when I remove the type parameters (as soon as this batch of CLs is submitted).
Some CLs in this batch also migrate calls away from actualAsString(). Its literal replacement is `"<" + actual + ">"` (unless an object overrides actualCustomStringRepresentation()), but usually I've made a larger change, such as switching from an old-style "Not true that..." failure message to one generated with the Fact API. In that case, the new code usually contains a direct reference to this.actual (a field that I occasionally had to create). Another larger change I sometimes made is to switch from a manual check-and-fail approach to instead use check(...). And sometimes I just remove a withMessage() call that's no longer necessary now that the code uses check(...), or I introduce a check(...) call. (An assertion made with check(...) automatically includes the actual value from the original subject, so there's no need to set it again with withMessage().)
Finally, there's one CL in this batch in which I migrate a Correspondence subclass to instead use Correspondence.from.
END_PUBLIC
If this is too many changes at once, let me know, and I can split it up and/or explain further. In the past, I've erred on the side of sending separate CLs for each change, which has required some owners to manually reapply each one, so now I'm trying this way.
Thanks again for your patience. There is an outside chance that this will be the last CL I send you before Truth 1.0 -- but certainly no promises :)
More information:
Renames:
[]
[]
[]
Removing type parameters: []
Migration from old fail*(...) methods to new ones and to check(...): []
Changes that replace assert*(...) with introduce check(...): []
Correspondence subclass to Correspondence.from: []
Tested:
TAP --sample ran all affected tests and none failed
[]
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251921007
We need to first populate the apt sources before installing anything. It was working on my machine because the layer was cached. After I purged all images I was able to replicate the same error message.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251918035
It would be nice to be able to query them using SQL in order to debug
some potential issues registrars might be facing.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251863662
We'll use this for LRP. This is safe because we must specifically reserve a
domain by including it in a reserved list, create an associated allocation
token, and distribute that token, before a create would succeed.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250901144
They are useful for viewing and diagnosing problems, and are not that
expensive.
We aren't enabling all the metrics though - the whole "per batch / per domain"
performance indicators are still disabled
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250876554
This specifically includes info, transfer, and update. Note that normal
domain creates do not work in quiet periods and are not expected to.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250755940
These updates can generally ran slowly in the background (i.e. they're low
priority), and shouldn't clog up the system to the point where they slow down
live incoming customer-requested updates.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250732171
|[](https://storage.googleapis.com/domain-registry-kokoro/internal/index.html)|[](https://storage.googleapis.com/domain-registry-kokoro/foss/index.html)|[](https://lgtm.com/projects/g/google/nomulus/alerts/)|[](https://github.com/google/nomulus/blob/master/LICENSE)|[](https://cs.opensource.google/nomulus/nomulus)|
<propertyname="message"value="String.toUpper/LowerCase() can have unexpected results depending on locale. Either set the locale explicitly - e.g., toUpperCase(Locale.ENGLISH) - or use Ascii.toUpper/LowerCase(...) instead."/>
</module>
<!-- Checks that DateTime.now(...) is always passed a DateTimeZone parameter. -->
<modulename="RegexpSingleline">
<propertyname="format"value="DateTime\.now\(\)"/>
<propertyname="message"value="DateTime.now() must be called with a DateTimeZone parameter, e.g. DateTime.now(UTC)"/>
</module>
<!-- Checks that Javadoc does not include a malformed @see tag. -->
<propertyname="message"value='Your Javadocs appear to use invalid <a> tag syntax in @see tags. Please use the correct syntax: @see <a href="http(s)://your_url">url_description</a>'/>
</module>
<!-- Checks that our Ofy wrapper is used instead of the "real" ofy(). -->
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.