1
0
mirror of https://github.com/google/nomulus synced 2026-01-16 02:33:16 +00:00

Compare commits

...

720 Commits

Author SHA1 Message Date
gbrodman
4ec7f23e84 Use the parent to store the history repo ID and fill in the base object (#830)
* Use the parent to store the history repo ID and fill in the base object

Storing the repo ID in the parent and in the base object has two primary
benefits.

First, it unifies the parent information in the HistoryEntry's `parent`
object. This simplifies the builders and the data flow.

Second, when possible (which should be always, post-migration) we fill
out the DomainContent's repo ID (similarly for the other EPP resources)
which means that when reconstituting the ofy keys we don't need to pass
the repo ID in from a separate object. This way, all the data are
encapsulated where they should be.

The primary downside here is that it further reduces the "immutability"
of the history objects (since we're using the Hibernate setter for the
parent repo ID) but we weren't immutable anyway.

* Respond to CR

- compare the entire vkeys in tests
- always return the parent for repo ID

* Simplify creation of parent VKeys

* Fix flipped isAssignableFrom check in VKey

* Merge remote-tracking branch 'origin/master' into historyRepoId
2020-10-09 16:01:51 -04:00
Shicong Huang
7a68b1b6f0 Revert package-lock.json to version from #676 (#834)
Co-authored-by: gbrodman <gbrodman@google.com>
2020-10-09 15:57:23 -04:00
Shicong Huang
14e593d9e1 Add SchemaCrawler as dependency (#833) 2020-10-09 15:02:11 -04:00
Weimin Yu
2d5de96fbd Minor python changes (#832)
* Minor python changes

Use dataclasses instead of attrs. The former is part of the standard lib
while the latter may need to be installed separately.

Also added python3 to the list of prerequisites.
2020-10-09 14:50:21 -04:00
Weimin Yu
13d30b0bfb Maintain a release-to-Version map in deployment (#831)
* Maintain a release-to-Version map in deployment

Keep track of the mapping between Nomulus release tags and AppEngine
version ids with a mapping file. This is necessary because AppEngine
does not support custom versioning. With this mapping, rollbacks could
be automated. Automation of rollbacks is important since there are
test-supporting metadata to be updated, but are easily forgotten.

During the last stage of deployment, current per-service version ids
are fetched using gcloud and are appended to a file on GCS. Each line
is of the format "{RELEASE_TAG},{APPENGINE_SERVICE},{APPENGINE_VERSION}.

This change has been tested in crash. The rollback script is still a
work in progress.
2020-10-09 13:32:52 -04:00
Shicong Huang
b05f6b4ba3 Add SQL schema for DelegationSignerData (#713)
* Add SQL schema for DelegationSignerData

* Remove join table

* Rebased on HEAD

* Rebase on head
2020-10-09 10:22:31 -04:00
Shicong Huang
17a1387184 Disable auto-generation on id for HostHistory and ContactHistory (#827) 2020-10-08 12:30:54 -04:00
Ben McIlwain
2e230664fd Convert CertificateViolation into an enum (#829)
* Convert CertificateViolation into an enum

This ends up being nicer to deal with from callsites than class instances, while
still permitting full configurability of all parameters. There are various other
changes/fixes as well.
2020-10-07 22:19:36 -04:00
Michael Muller
299b093f78 Correctly restore composite VKeys in DomainContent (#825)
* Restore composite vkeys in DomainContent

PollMessage/BillingEvent vkeys in DomainContent must have their ofy keys
restored from other fields in DomainContent (namely the repo id and their
specific history event ids).

Add PostLoad methods to DomainContent and DomainHistory to do the restoration.

* Fixes for review.

* Deal with foreign-key cycles
2020-10-07 12:42:01 -04:00
Ben McIlwain
61e7fa89f7 Fix incorrect repackaged App Engine import (#828)
* Fix incorrect repackaged App Engine import
2020-10-07 11:52:17 -04:00
sarahcaseybot
6ab69d4226 Add a CertificateChecker class (#793)
* CertificateChecker with checks for expiration and key length

* Add validity length check

* Get rid of hard-coded constants and DSA checks

* add files that for some reason weren't included in last commit

* Rename violations and other fixes

* Add displayMessage to CertificateViolation enum

* Switch violations from an enum to a class

* small changes

* Get rid of ECDSA checks

* add checks for old validity length

* Change error message for validity length
2020-10-06 15:47:42 -04:00
gbrodman
0f09a4a0ab Add more Datastore/Sql Entity annotations (#826)
* Add more Datastore/Sql Entity annotations

* Move comments up a line
2020-10-05 13:07:53 -04:00
Shicong Huang
95f6ccc657 Fix vkey reconstruction for PollMessage (#823)
* Fix vkey reconstruction for PollMessage

* Add foreign key

* Rebase on HEAD
2020-10-05 10:35:40 -04:00
Michael Muller
77fabe4dc4 Move "WithLongVKey" to BillingEvent subclasses (#821)
When loading the VKeys for the BillingEvents hierarchy, it is necessary to
restore the original concrete class for the type, otherwise we end up with a
different (and incompatible) VKey.

As part of this, convert the cancellation matching billing event to
VKey<Recurring>, which seems like the only thing it actually can be.
2020-10-02 15:20:23 -04:00
Lai Jiang
71fa12f773 Fix invoicing SQL (#824) 2020-10-01 14:29:49 -04:00
Shicong Huang
fd40a6a2b9 Use composite primary key for HostHistory and ContactHistory (#809)
* Use composite primary key for HostHistory and ContactHistory

* Update flyway file version

* Make getters private

* Add javadoc

* Rebase on HEAD
2020-10-01 11:01:57 -04:00
Michael Muller
71f86c9970 Add VKey.restoreOfy() method for fixing ofy keys (#820)
Add a restoreOfy() instance method and a restoreOfyFrom() static method to
assist in restoring the objectify key for classes that have composite keys
that do not restore automatically.
2020-09-30 11:15:58 -04:00
Michael Muller
6f75dfd116 Create a flyway index file and verify correctness (#819)
* Create a flyway index file and verify correctness

Create an index file (flyway.txt) containing the names of all of the flyway
files and verify that it is ordered and in sync with the actual contents of
the flyway directory.  Also provide a target (generateFlywayIndex) to
automatically generate it.

The purpose of flyway.txt is to cause a merge conflict in the event that two
different developers add a flyway file with the same sequence number, an event
which has occurred multiple times.
2020-09-29 11:26:05 -04:00
Lai Jiang
ad5a74fee9 Revert "Request 101m CPU in sandbox proxy (#813)" (#818)
This reverts commit e30c0f9a11.

The proposed solution didn't work.
2020-09-25 11:55:46 -04:00
Lai Jiang
29b1ec4211 Add log4j-core as a runtime dependency (#817)
Without it we kept getting the following warning:

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
2020-09-24 19:59:39 -04:00
Weimin Yu
553d5717cb Enhance the test for forbidden Schema changes (#815)
* Enhance the test for forbidden Schema changes

Current test is git-based. It is difficult to maintain and does not
catch out-of-order version numbers. It is also more aggressive than
necessary, failing on changes to submitted scripts that have not been
deployed yet.

The new test starts a database, deploys the current schema to it,
then deploys the set of Flyway scripts in this repository to the database.
2020-09-24 12:31:08 -04:00
Shicong Huang
1056fdbb64 Fix VKey reconstruction issue in BillingEvent (#805)
* Fix VKey reconstruction issue in BillingEvent

* Rebase on head
2020-09-23 19:04:58 -04:00
Lai Jiang
4aaf31be9f Update IDN tables per ICANN's request (#812)
See b/168508962 for the request.
2020-09-21 23:08:10 -04:00
Lai Jiang
e30c0f9a11 Request 101m CPU in sandbox proxy (#813)
This is suggested as a mitigation to allow us to deploy to sandbox. The default
value is 100m.

See: https://b.corp.google.com/issues/167295064#comment36.
2020-09-21 21:22:41 -04:00
gbrodman
2a5d9c8ef5 Allow explicitly for null EPP resources in History objects (#790)
* Allow explicitly for null EPP resources in History objects

* Repo IDs should always be nonnull

* Add a test to verify loading / comparison of legacy HistoryEntry objects

* Format javadoc + annotations

* More javadoc changes

* V52 -> V56

* V56 -> V57

* saveNew -> insert in new tests
2020-09-21 15:50:15 -04:00
gbrodman
597f5746a4 Rename V54 -> V56 on host table rename (#811) 2020-09-21 13:34:37 -04:00
Michael Muller
5bff53a711 Rename HostResource table to Host (#804)
* Rename HostResource table to Host

* Convert drop/creates to renames
2020-09-21 11:43:25 -04:00
Michael Muller
933394e8c3 Improve naming of TransactionManager methods (#802)
* Improve naming of TransactionManager methods

Per internal discussion, convert names of methods as follows:

    saveNew -> insert
    saveNewOrUpdate -> put
    checkExists -> exists

Likewise, convert derived names to their corresponding forms, e.g.
saveNewOrUpdateAll -> putAll.
2020-09-21 09:10:01 -04:00
gbrodman
3b841bbb5b Add domain-specific history fields to DomainHistory objects (#794)
* Add domain-specific history fields to DomainHistory objects

* Add javadoc for Hibernate-only methods

* V52 -> V54

* Use only a single DomainTransactionRecord table

* Add nullables and fix up a comment

* V54 -> V55

* Regenerate db schema

* Regen SQL file
2020-09-18 15:55:17 -04:00
gbrodman
798879d031 Fix semantic merge conflict in Registry (#810)
* Fix semantic merge conflict in Registry
2020-09-18 14:40:11 -04:00
gbrodman
1a63d50b82 Create a separate per-tld registry lock/unlock cost (#800)
* Create a separate per-tld registry lock/unlock cost

Currently we use the standard server status change cost for this, but
this might not be ideal at some point in the future if we wish to allow
manual forced updates outside of the standard registry lock system (we
would charge for these manual forced updates, even if we don't charge
for registry locks).

* Remove period
2020-09-18 13:22:29 -04:00
Lai Jiang
054571a625 Update ICANN activity reporting logging (#808)
1. It appears that when we have a 200 response, the response content is
   garbled, but we don't care since we know the request is successful.
   When we have a 400 response, the response is indeed UTF-8 encoded.
   Print the stack trace of the HTTP exception doesn't help anymore.

2. The result code is a complex type which includes the xml element with
   all its attributes, which don't care. We only want to print the
   value.
2020-09-18 11:37:23 -04:00
sarahcaseybot
7468a9915b Migrate Registry objects to a TLD table in Cloud SQL (#803)
* Add TLD table

* Change reservedLists to array

* Change ReservedLists back to a set

* Rename reservedListKeyConverter to ReservedListKeySetConverter

* Add a postload method
2020-09-17 12:47:50 -04:00
Lai Jiang
157d9f75c1 Use the save API version for the HPA controller and the deployment (#807)
This supposedly can fix our deployment problem. Tested on alpha.

Also updated the deployment script to replace the service object as
well.
2020-09-16 09:15:35 -04:00
Lai Jiang
b2e4f07bb9 Update IDN tables (#806)
See b/168508962 for context.

1. Changed the contact to iana-contact@google.com
2. Changed the header from "Script" to "Language" for zh-Hans and
   zh-Hant.
3. Commented out the references in zh-Hans and Zh-Hant
2020-09-16 09:15:27 -04:00
Michael Muller
5488e1b323 Fix accessing superclass fields in checkExists() (#799)
* Fix accessing superclass fields in checkExists()

JpaTransactionManagerImpl doesn't respect @Id fields in mapped superclasses.
Replace calls to getDeclaredId() and getDeclaredField() with superclass
friendly counterparts.
2020-09-11 13:45:51 -04:00
Shicong Huang
5ab0f97351 Add and use temp_history_id_sequence to avoid release error (#795) 2020-09-11 12:25:08 -04:00
sarahcaseybot
f7b65327da Add type converter for Key<ReservedList> and Key<PremiumList> (#796)
* Add converter for reservedlist and premiumlist keys

* Remove public modifier from test classes
2020-09-10 17:36:22 -04:00
Michael Muller
36482ce94f Fix the billing occurrence foreign key (#797)
* Fix the billing occurrence foreign key

Fix the Domain.billing_occurrence_id foreign key constraint to reference the
correct table (BillingRecurrence, not BillingEvent).
2020-09-10 12:02:24 -04:00
Lai Jiang
125f509b46 Change disable invoicing flag to enable invoicing flag (#783)
* Change disable invoicing flag to enable invoicing flag

This flag will be the sole determinor on if invoicing is enabled,
regardless of TLD types.

Once this PR is deployed we will need to run the nomulus command to
update this flag on all launched open TLDs.

For context on why this change is made, see b/159626744.

* Rename enableInvoicing to InvoicingEnabled
2020-09-09 19:37:41 -04:00
gbrodman
fb7ba80b86 Implement DatastoreEntity/SqlEntity for many more classes (#788)
* Implement DatastoreEntity/SqlEntity for many more classes

We still have many more classes to go, but this gets us closer to
guaranteeing that we can convert from Datastore to SQL objects and back
again.

* Shift SqlEntity impl to HistoryEntry
2020-09-09 13:56:59 -04:00
Lai Jiang
a86fcf79f7 Make ICANN reporting not fail on success upload (#791)
* Make ICANN reporting not fail on success upload

According to the spec
(https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-13#page-16),
when an upload succeeds (HTTP response code 200), the result code
contained in the response message is always 1000 (success). So there is
no need to parse the response content and check the result code. Given
that we are having a problem parsing the response content due to encoding,
it is best that we don't check it so as to not get false negative
alerts when the upload is successful.

The current logic also has a bug: HttpRequest.execute() will by default
throw when the response code is non-20X. Therefore for a 400 response,
our parsing logic never runs on it. Coincidentally, this month when we
uploaded the July activity report (due to stale cursors), we get 400
responses (due to existing reports on the ICANN servers). The stack
trace printed for the thrown exceptions from the 400 responses contained
correctly parsed response contents. This lead us to believe that the issue with
encoding was transient last month. However when we tried again to upload this
month's report, our parser failed again (because the response code was 200 this
time, and our parser actually ran on the response contents).

This seems to suggest that ICANN is sending back readable response
contents, but our parser somehow failed to understand it, assuming that
ICANN is using the same encoding for 200 (which we tried and failed to
parse) and 400 response contents (which caused an exception and was printed
corrected in the stack trace).

This PR changed the transport behavior so that it doesn't throw
automatically for non-20X responses. We will print the content for both
200 and 400 responses, but only try to parse 400 response content. We
put the 400 response in an HttpResponseException and print stack trace
from it, which should display the content correctly so that we can
compare it with the result of our own parsing.

* Add tests
2020-09-03 15:57:30 -04:00
Lai Jiang
dc8e095e55 Upgrade to Gradle 6.6.1 (#792) 2020-09-03 15:56:52 -04:00
Shicong Huang
cdf2c7f7cb Merge ClaimsList into ClaimsListShard (#694)
* Merge ClaimsList into ClaimsListShard

* Add a TODO to rename the class

* Rebase on HEAD

* Improve javadoc
2020-09-03 11:18:40 -04:00
Shicong Huang
ecafebdc3d Use composite primary key for DomainHistory (#767)
* Use composite primary key for DomainHistory

* Move History table's SequenceGenerator to orm.xml

* Rebase on HEAD and remove default value for key in History tables

* Use primitive type for id.

* Revert the cache change
2020-09-03 10:21:23 -04:00
Lai Jiang
c6c8d21281 Update jackson-core to the latest version (#789)
Vomit identified a vulnerability in the current version.
2020-09-03 09:11:12 -04:00
Shicong Huang
5f6ea2cbf2 Fix cascade issue for GracePeriod (#775)
* Fix cascade issue for GracePeriod

* Rebase on HEAD

* Make GracePeriod immutable

* Add javadoc and use nullToEmptyImmutableCopy
2020-09-02 20:05:53 -04:00
Shicong Huang
393c388e0d Consolidate conversion from Duration to Period in DurationConverter (#786)
* Consolidate conversion from Duration to Period in DurationConverter

* Resolve comment
2020-09-01 11:29:28 -04:00
gbrodman
5a08ce498e Revert "Change the wording on the lock-not-enabled page (#504)" (#787)
This reverts commit 28d3af0ee9.

We are now ready to accept new Registry Lock requests so we can have the
originally-designed wording back in place
2020-08-31 15:19:42 -04:00
Weimin Yu
5db8cbc994 Fix flaky web driver tests (#784)
* Fix flaky web driver tests

Identified two flaky tests in RegistrarConsoleScreenshotTest through
local testing and fixed them by waiting for specific web elements instead
of using fixed delays.

Refactored the wait methods to support different test scenarios,
and removed unnecessary delays.

Extensively tested locally. Also ran multiple presubmits on Kokoro.
2020-08-31 15:09:54 -04:00
Weimin Yu
bbcafea98e Cover more base in forbidden SQL change check (#785)
* Cover more base in forbidden SQL change check

Update the forbidden SQL change detection script to include file deletion and
renaming as well as edits.
2020-08-31 15:08:37 -04:00
gbrodman
1bba68dd96 Add success/failure notifications for the RelockDomainAction (#733)
* Add success/failure notifications for the RelockDomainAction

If a relock fails for some reason, we should noisily notify both our
alerting email and also the registry lock contacts for the registrar in
question. The consequences of a silent failure could be large so it's
something we want to avoid if at all possible.

In addition, we only retry tasks up to two times (one in 5min, one in
10min).

This model of retries / notifications, as well as the language contained
in the emails, have been LGTMed by Bruno and Kirsten

* Change the wording on the success email

* Change the times in which we send emails

For transient failures:
- Retry every ten minutes for six hours
- Send an email after a half hour (three failures) saying that we'll
retry
- Send a success email if we succeed any time after that

For non-transient failures:
Send an email with the error message and don't retry

* Add a test for the max-failure-email

* Responses to CR

- retry indefinitely
- send an email to just the alert address if we can't find the lock
- refactor the task enqueuer a bit

* non-transient -> non-retryable

* Use a lenient stubber for the AESU

* Add a DS transaction around the re-lock
2020-08-31 14:15:47 -04:00
gbrodman
0423c7ae22 Fix semantic merge conflict (#781)
Note: the schema change isn't from anything I did, I think, but from an
unrelated semantic merge conflict
2020-08-26 12:53:39 -04:00
gbrodman
266bd43792 Persist *History objects as HistoryEntry objects (#749)
* Persist *History objects as HistoryEntry objects

While Datastore is the primary database, we will store *History objects
as HistoryEntry objects and convert to/from the proper objects in the
Datastore transaction manager. This means that History objects will not
properly store the copy of the EppResource until we move to SQL as
primary, but this is the way the world exists anyway so it's not a
problem.

* Format code and simplify the bulk loading

* Add comments with context
2020-08-26 11:45:09 -04:00
Weimin Yu
df15b38a1e Fix JPA setup in Nomulus tool (#780)
* Fix JPA setup in Nomulus tool

Hibernate unnecessarily scans third-party classes in the Nomulus tool,
hitting a bug and fails to set up.

In this change we properly configured persistence.xml to include the orm mapping file (orm.xml) and disable 
auto detection, and provided a custom (NOOP) scanner
to work around Hibernate scanner bugs.

Also improved on the :core:registryIntegrationTest task to test for
JPA setup as well as dependency-packaging.
2020-08-26 09:51:33 -04:00
Ben McIlwain
daa8bb6b2c Add --autorenews parameter to nomulus update_domain tool (#772) 2020-08-25 17:24:09 -04:00
gbrodman
ea2a6165e5 Fix AllocationTokenTest (#779) 2020-08-24 13:08:45 -04:00
Weimin Yu
c36f0c89c8 Fix missing-driver in HibernateSchemaExporterTest (#777)
* Fix missing-driver in HibernateSchemaExporterTest

HibernateSchemaExporterTest is failing with "Driver not found" error
after Java 11 upgrade. Reason is that ServiceLoader now only checks
modules for services.

Proper fix is to define modules.

This short term fix is to declare the driver class explicitly.
2020-08-24 10:20:19 -04:00
Ben McIlwain
8d38086d40 Add check to prevent creating VKeys with incorrectly null parents (#774)
* Add check to prevent creating VKeys with incorrectly null parents

Datastore entities that are not the roots of entity groups are uniquely defined
by an inheritance chain in addition to the id/name, not just by id/name.
2020-08-22 09:13:55 -04:00
Shicong Huang
bb2f35b673 Use composite primary key for RegistrarContact (#761)
* Use composite primary key for RegistrarPoc

* Increase the serial number for flyway file and resolve comments

* Rebase on HEAD

* Rebase on HEAD
2020-08-21 11:17:36 -04:00
Michael Muller
15587f03d6 Fix empty Domain nameserver loads (#769)
* Fix empty Domain nameserver loads

Domains with no nameservers were being loaded from SQL as an empty set instead
of null as they should be.

Discovered this will trying to test updates, so added a test for updates in
the course of it.
2020-08-21 11:11:01 -04:00
gbrodman
5c30ef7086 Add a SQL schema to AllocationToken (#763)
* Add a SQL schema to AllocationToken

* Respond to CR

- rename field in tests
- rename allowed_registrar_ids field
- remove unnecessary db load in GATC

* Add TODO for HistoryEntry vkeys

* Run autoformat

* V48 -> V49
2020-08-20 20:18:34 -04:00
Ben McIlwain
a28632dbe1 Reduce excessive visibility levels in test classes (#773)
* Reduce excessive visibility levels in test classes
2020-08-20 18:34:59 -04:00
Weimin Yu
70274da971 Allow schema push in alpha and crash (#771)
* Allow schema push in alpha and crash

This allows unsubmitted changes to be tested in alpha and
crash.

Also updated the README file and reformatted with the internal
mdformat tool.
2020-08-19 00:05:57 -04:00
Lai Jiang
f9e3986a4a Hex dump the ICANN report response content for debugging. (#770)
For example if the content is a byte array of "Hello", the Hex dump will
be "48656C6C6F".
2020-08-18 10:23:28 -04:00
Michael Muller
b44c7d2786 Update user-facing documentation (#662)
* Update user-facing documentation

Give our docs a complete overhaul to account for changes in the system,
notably the requirement to configure postgresql.

* Fix dangling sentence.

* Merge branch 'master' into admin-docs
2020-08-17 14:46:32 -04:00
Lai Jiang
a302cf3e04 Update dependencies to get rid of some illegal reflection warnings (#768)
For some inexplicable reasons I have to move the javax.mail package one
spot up to avoid its classes being shadowed by those provided in the
appengine package...
2020-08-14 16:53:22 -04:00
Lai Jiang
be773a9d0d Upgrade to Gradle 6.6 (#766) 2020-08-13 13:28:17 -04:00
Legina Chen
fff048d9a8 Create DAO for Spec11ThreatMatch (#750)
* Create DAO for Spec11ThreatMatch

* Add tests

* Execute SQL for deleteEntriesByDategit status

* Remove testing line

* Rename createSpec11ThreatMatch()

* Add comments about jpaTm and use jpaTm() in test

* Fix technicality in comment

* Remove a new line

* Truth chaining for comparing ImmutableLists of matches

* Javadoc formatting
2020-08-13 10:00:42 -07:00
Ben McIlwain
d873b9f69a Add autoRenewEndTime field to Domain entity (#765)
* Add autoRenewEndTime field to Domain entity

This is the first step towards allowing administrators to selectively disable autorenew.
2020-08-13 10:40:11 -04:00
Lai Jiang
98db79d3d1 Re-enable invoicing pipeline deployment (#764)
Now that beam deployment is compatible with Java 11. Re-enable this
step.
2020-08-11 17:26:17 -04:00
Ben McIlwain
8fe9cde9ff Add some missing @Nullables and types (#753)
* Add some missing @Nullables and types

Also deletes two unused VKey.createOfy() methods that simply don't work, because
a kind and an id is not enough to create a Datastore key; you also need the full
entity group inheritance chain for entities that are not roots themselves
(which is most of the entities in our schema).

* Merge branch 'master' into add-missing-nullables

* Throw UnsupportedOperationException for contacts/hosts too

* Merge branch 'master' into add-missing-nullables
2020-08-11 17:19:00 -04:00
Weimin Yu
8dbfbb0f33 Update BEAM SDK to work with Java 11 (#762)
* Update BEAM SDK to work with Java 11

Upgraded BEAM dependencies to 2.23.0.

Updated Spec11 and invoice pipelines:
- Added the required region parameter.
- Removed the workaround code for staging.

Verified that staging is successful in alpha:
./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_spec11_pipeline --project domain-registry-alpha'

and

./nom_build :core:registryTool --args='-e alpha --sql_access_info "gs://..." deploy_invoicing_pipeline'
2020-08-10 20:56:08 -04:00
Shicong Huang
a7e1bd800b Add a nomulus command to resave entity with unique id (#656)
* Add a nomulus command to resave entity with unique id

* Remove PollMessage

* Remove logic for PollMessage and resolve comments

* Resolve comments
2020-08-10 16:34:24 -04:00
Ben McIlwain
082086bde9 Double the timeout length for WebDriver screenshot tests (#758)
* Double the timeout length for WebDriver screenshot tests

My theory is that this timeout is being applied to all retries of a failing test
rather than each one, and thus flaky screenshot tests aren't being given
sufficient time to complete any attempt past the first.
2020-08-10 15:16:34 -04:00
gbrodman
4d92ba4b8e Manually set the files to stage in Beam using the classpath (#760)
See https://issues.apache.org/jira/browse/BEAM-2530 for more details on
why this is necessary, but basically Beam assumes that we are using a
URLClassLoader which is no longer the case post-Java-8. This means that
we have to manually specify the files to stage.

See https://stackoverflow.com/questions/48292491/java-dataflow-unable-to-use-classloader-to-detect-classpath-elements

Tested by building, deploying, and running the Spec11 pipeline on Alpha
using Java 11.
2020-08-10 15:15:30 -04:00
Lai Jiang
7b2f7c08e4 Comment out invoicing pipeline deployment temporarily (#759)
Currently it doesn't work with Java > 8. Fix inflight. Disable it to
unblock deployment.
2020-08-10 15:11:34 -04:00
gbrodman
44d7ad61c0 Remove spec11 domain foreign key (#757)
We'll want this eventually but until the Domain SQL table is populated,
we can't rely on domains' existence.
2020-08-10 14:55:40 -04:00
Lai Jiang
26dfdd5b71 Wait for contact add page to load before doing screenshot diff (#756)
This test is flaky and the going theory is that we take a screenshot
immediately after we click the "Add" button and the CSS hover effect is
somehow still playing on the button.
2020-08-10 13:29:47 -04:00
Weimin Yu
85970daa70 Make index on contactId not unique (#751)
* Make index on contactId not unique

This column cannot be unique due to the fundamantal design of the domain
model.
2020-08-10 13:09:37 -04:00
Weimin Yu
9701ba1254 Restore original jpa manager after tests (#754)
* Restore original jpa manager after tests

AppEngineExtensionTest fails to restore original jpa manager in some
tests. This results in flakiness in DummyJpaTransactionManagerTest.
2020-08-10 11:05:58 -04:00
Lai Jiang
ab4cecba22 Temporarily disable spec 11 pipeline deployment in GCB (#755)
The current setup causes the GCB job to fail validation and not run because it
uses backticks in the configuration yaml, which is not allowed -- there is no
shell to perform backtick substitution. See the error message here:

https://spinnaker.endpoints.domain-registry-dev.cloud.goog/gate/pipelines/01EF5GRMD625613H6Z033DBD3Z

In the future please make sure to test the GCB pipeline as instructed in
the comments at the beginning of each file before committing.

I tried to work around it by downloading the nomulus tool jar file
instead (running the nomulus-tool docker image inside a docker image is
not advisable). However the "nomulus deploy_spec11_pipeline" command
still fails. I'm not sure why. Has the command itself been tested
locally? The error message is shown below:

```
Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
Step #2: WARNING: --region not set; will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones
Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory tryCreateDefaultBucket
Step #2: INFO: No tempLocation specified, attempting to use default bucket: dataflow-staging-us-central1-937378958468
Step #2: Aug 09, 2020 3:11:47 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse
Step #2: WARNING: Request failed with code 409, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://www.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=domain-registry-alpha.
Step #2: Exception in thread "main"
Step #2: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
Step #2:        at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
Step #2:        at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
Step #2:
Step #2:        at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
Step #2:        at org.apache.beam.sdk.Pipeline.create(Pipeline.java:147)
Step #2:
Step #2:        at google.registry.beam.spec11.Spec11Pipeline.deploy(Spec11Pipeline.java:157)
Step #2:        at google.registry.tools.DeploySpec11PipelineCommand.run(DeploySpec11PipelineCommand.java:80)
Step #2:        at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:257)
Step #2:        at google.registry.tools.RegistryCli.run(RegistryCli.java:182)
Step #2:        at google.registry.tools.RegistryTool.main(RegistryTool.java:129)
Step #2: Caused by: java.lang.reflect.InvocationTargetException
Step #2:        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Step #2:
Step #2:        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Step #2:        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Step #2:        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Step #2:        at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
Step #2:        ... 8 more
Step #2: Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@5cb0d902, only URLClassLoaders are supported.
Step #2:        at org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:58)
Step #2:
Step #2:        at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:285)
Step #2:
Step #2:        ... 13 more
```

Lastly the "--project" flag refers to the KMS project. While I'm not
sure which project is that, I don't think we can use the PROJECT_ID
variable as this is a GCB-substituted variable which refers to the
project that the GCB job runs in, which in our cases means
domain-registry-dev. We shouldn't use that project for KMS. I've changed
it to the same project as the one we are deploying to, but please note
that we have a separate project ${project_id}-keys that is used for all
KMS purposes. This is specified in the config file so if that's what you
meant to use, there is no need to specify it in the command line. Actually
if you meant use the project to be deployed to for KMS, it also
shouldn't be necessary to specify it separately as this information is
already known when you specified "nomulus -e ENV".

https://team.git.corp.google.com/domain-registry-eng/nomulus-internal/+/refs/heads/master/core/src/main/java/google/registry/config/files/nomulus-config-production.yaml#168

Can you add more description on what the KMS project is supposed to be?
I don't think we specify a project for KMS purpose in any other
commands.

Given that there are several unresolved issues, I've commented out my
proposed solution so that deployment can proceed.
2020-08-09 22:41:31 -04:00
Lai Jiang
00941ddc3d Disable tests in RC builds (#752)
For reasons unclear at the moment the tests are not passing. Disabling
them for now so that release candidates can be built. We have CI runs
after each merge so we should be pretty confident if the build is broken
or not.
2020-08-07 17:51:34 -04:00
Shicong Huang
7b2b49dd53 Fix vkey construction in PollMessage and BillingEvent (#747)
* Fix vkey construction in PollMessage and BillingEvent

* Remove null check
2020-08-07 15:38:53 -04:00
Lai Jiang
bc8df8f34e Exclude a test that fails in GCB build using Java 11 (#748) 2020-08-07 10:53:46 -04:00
Ben McIlwain
74b4424509 Improve and consolidate testing of poll messages/billing events (#745)
* Improve and consolidate testing of poll messages/billing events
2020-08-06 21:13:27 -04:00
Shicong Huang
486bf32353 Add SQL schema for GracePeriod (#709)
* Add SQL schema for GracePeriod

* Remove the join table

* Add a domainRepoId in GracePeriod

* Move the clone logic to GracePeriod

* Rebase on HEAD
2020-08-06 10:26:19 -04:00
Ben McIlwain
b2a78b5d68 Allow allocation token discounts on premiums and for multiple years (#744)
* Allow allocation token discounts on premiums and for multiple years

* Add domain check flow tests

* Address code review comments

* Update schema file
2020-08-05 17:54:47 -04:00
Lai Jiang
95f4ae0e3a Use nodesource to install node (#742)
The node installed by nvm gives errors when running "npm install".

Also installs Python as it is need. Presumbly the system provided npm
version has python as a dependency so it was installed when npm was
installed.
2020-08-05 14:56:40 -04:00
Lai Jiang
915405b735 Remove extra space after bashslash (#743) 2020-08-04 20:00:46 -04:00
sarahcaseybot
d7fb6097ba Add retrier to JpaTransactionManager (#693)
* Add retrier to retry exceptions we've seen so far

* Check for nested exceptions

* fix formatting

* check for nested transactions

* add JDBCConnectionException to isFailedQueryRetriable

* Add retrier to methods with suppliers
2020-08-04 12:22:08 -04:00
Michael Muller
d224d96924 Improve the uniform_rapid_suspension command (#739)
- Reuse DS record format processing from the create/update domain commands
  (BIND format, commonly used in URS requests)
- Remove the CLIENT_HOLD status from domains that have it (this blocks us from
  serving the new nameservers and DS record)
2020-08-04 11:06:02 -04:00
Lai Jiang
6d110c77ac Use the latest version of node in the builder image (#741)
The default node version from the base image (Ubuntu 18.04) is too older
and karma is not happy about it.
2020-08-03 17:40:50 -04:00
Legina Chen
a79940e822 Persist ThreatMatches into Spec11ThreatMatch (#723)
* Replace jpaTm with a JpaSupplierFactory

* Style

* Style

* Pipeline takes in a SerializableSupplier instead

* Change the ordering of imports

* Test a good domain in addition to a bad one

* Rename and check good domain for Transact Answer

* Use standard Mockito verify

* Verify transact call and no more interactions

* Remove Answer comment

* Naming chsnges

* Deploy Spec 11 pipeline correctly

* Fix formatting of deploy file

* Use a file to persist state across Cloud Build steps

Co-authored-by: Gus Brodman <gbrodman@google.com>
2020-08-03 14:40:00 -07:00
gbrodman
d8ec6294c3 Refactor DomainBase into DomainContent and create DomainHistory (#668)
* Refactor DomainBase into DomainContent and create DomainHistory

This is similar to #587 and #634, but for domains.

One caveat is that we refactor some of the Domain* instance methods to
be static so that they can be called either on DomainBase or
DomainContent, returning the appropriate type each time.

Note that we set DomainHistory to use the same revision ID sequence as
HostHistory and ContactHistory.

In addition, we refactor the tests to the History objects a bit to
reduce duplicate code and because we cannot guarantee yet that the
SQL-stored VKeys are symmetrical -- the ofy keys are not persisted at
the moment.

In addition, rename the DomainHost table to the default Domain_nsHosts so that it automatically creates two separate nsHosts tables for us -- one foreign-keyed on the domain repo ID, and one foreign-keyed on the history revision ID

* Use access hackery to allow manual names for nsHosts tables

* Clean up post merge artifacts

* Add unused setters that Hibernate requires

* Fix the tests and semantic merge conflicts

* Change ns_hosts to ns_host everywhere

* Rename ns_host to host_repo_id

* V42 -> V44
2020-08-03 17:36:12 -04:00
Lai Jiang
9304e2f421 Enable Java 11 features (#737)
* Enable Java 11 features

As of this commit Java 11 must be used to build. The generated bytecode
is still at Java 8 due to App Engine task queue limit.

Also fixed a bug where the included google-java-format jar file is not
used, requiring the user to install it separately.

See: https://cloud.google.com/appengine/docs/standard/java/taskqueue/push
2020-08-03 16:08:29 -04:00
Lai Jiang
5f2be914a1 Use Java 11 in GCB to build release candidates (#736) 2020-08-03 13:13:08 -04:00
sarahcaseybot
4ad7f9734d Add IntervalDescriptor and change DurationConverter to use Interval datatype (#653)
* Add use of interval data type

* Add support for Millis

* Use Java-object type

* Change column type for relock_duration

* add years and months

* Add tests for hours, minutes, and seconds

* Add javadoc describing how joda duration is stored

* Add test for lots of days
2020-08-03 12:42:34 -04:00
Lai Jiang
90d080d42f Delete screenshots folder in fragile test first (#735)
It is not clear to me why this folder is deleted in standardTest first,
presumably this is done when the webdriver tests were part of
standardTest?

Also not clear to me is why screenshots are only deleted in presubmits
but not locally. At least on my workstation standardTest runs after
fragileTest, just like in kokoro, so it should have deleted the
screenshots
folder, but it didn't.

Regardless, the correct thing to do is to delete this folder first in
fragileTest so that it doesn't interfere with the results.
2020-08-03 11:46:34 -04:00
Ben McIlwain
e95ce30fa6 Upgrade prober & proxy packages from JUnit 4 to JUnit 5 (#734)
* Upgrade prober & proxy packages from JUnit 4 to JUnit 5
2020-08-01 11:34:58 -04:00
Lai Jiang
090c233592 Some more changes to make it possible to use Java 11 to compile (#732)
Add the class paths of the source files generated by annotation processors to
the javadoc task's class path so that it doesn't complain about missing
Dagger classes.

Also remove empty <p> tags in all generated source files, because jaxb
genrerates files in multiple locations.

Lastly, for unkown reasons when the source level is set to > 8, the core
subproject throws a warning about a Gradle internal annotator processor
that only supports up to Java 8 and cause the Java compliation to fail because
we set -Werror on all java compliation tasks. I don't think there is a
strong reason that we set -Werror anyway, so this commit removes it.
2020-07-31 17:21:28 -04:00
Ben McIlwain
16a31e460c Get rid of all remaining JUnit 4 usages except in prober & proxy (#731)
* Get rid of all remaining JUnit 4 usages except in prober & proxy subprojects

Caveat: Test suites aren't yet implemented in JUnit 5 so we still use the ones
from JUnit 5 in the core subproject.

* Fix some build errors
2020-07-30 20:29:00 -04:00
Lai Jiang
a02b67caf5 Migrate the documentation package to Java 11 (#729)
* Migrate the documentation package to Java 11

The old Doclet API is deprected and removed in Java 12. This commit
changes the documentation package to use the new recommended API.
However it is not a drop-in replacement and there are non-idiomatic
usages all over the place. I think it is eaiser to keep the current code
logic and kind of shoehorn in the new API than starting afresh as the
return on investment of a do-over is not great.

Also note that the docs package is disabled as of this commit because we
are still using Java 8 to compile which lacks the new API. Once we
switch our toolchains to Java 11 (but still compiling Java 8 bytecode)
we can re-enable this package.

TESTED=ran `./gradlew :docs:test` locally with the documentation package
enabled.
2020-07-30 17:12:33 -04:00
Lai Jiang
bf20a8ef96 Remove empty <p> tag from generated javadoc (#730) 2020-07-30 16:31:25 -04:00
Lai Jiang
8750c07fef Clean up generated sources configuration (#724)
1. The Gradle apt plugin is no longer needed to process annotations.
2. Without the apt plugin, Gralde puts the source files generated by
   annotation processors in build/generated/sources/annotationProcessor.
3. Change the location of custom generated files to be consistent.
4. Fix a javadoc formatting error.
2020-07-30 15:43:42 -04:00
Weimin Yu
7821de67f8 Run InitSqlPipeline (#727)
* Run InitSqlPipeline

Added the main() method to InitSqlPipeline.

Added a Gradle task to run InitSqlPipeline from command line. This
task is meant for testing and experiments.

Corrected the file name prefix of Datastore export files. Should
be 'output-', defined as 'input-'.
2020-07-30 14:28:53 -04:00
Shicong Huang
1022817384 Fix broken RegistryCli (#728) 2020-07-30 14:10:09 -04:00
Ben McIlwain
8c04bf2599 Rename InjectRule and remove some JUnit4-only dependencies (#726)
* Rename InjectRule and remove some deps

* Merge remote-tracking branch 'upstream/master' into final-core-tests
2020-07-29 20:27:47 -04:00
Ben McIlwain
34116e3811 Clarify un-avail reason on allocation-token-reserved domains (#725)
Apparently, in domain check responses, `avail=false, reason=Allocation token
required` was not sufficiently understood by all registrars. This changes it to
`avail=false, reason=Reserved; alloc. token required` to hopefully make it
crystal clear that the domain in question is reserved, i.e. if you were supposed
to be able to register this domain you'd already know it because we'd have
already given you the requisite allocation token.
2020-07-29 17:13:38 -04:00
Lai Jiang
d180ef43ac Move the documentation package to its own subproject (#722)
This makes it easier to later migrate the package to Java 11. If we move
and migrate in a single PR, because of the portion of the contents that
s changed, git will have trouble recognizing that some files are
renamed *and* modified and treat them as distinct files, making code
review difficult.
2020-07-29 13:41:02 -04:00
Michael Muller
f55270c46f Integrate transaction persistence into JpaTM (#717)
* Integrate transaction persistence into JpaTM

Store the serialized transaction whenever we commit from the JPA transaction
manager.  This change also adds:

-   The Transaction table.
-   The TransactionEntity which is stored in it.
-   Changes to the test infrastructure to register the TransactionEntity for
    tests where we don't load the nomulus schema.
-   A new configuration variable to allow us to turn the transaction
    persistence functionality on and off (default is "off").

* Changes for review.

* Incremented sequence number of flyway file
2020-07-28 19:23:44 -04:00
Ben McIlwain
d6d9874da1 Upgrade App Engine and webserver tests from JUnit 4 to 5 (#720)
* Upgrade App Engine and webserver tests from JUnit 4 to 5

* Fix most errors

* Merge branch 'master' into junit5ification

* Fix test server by extracting non-test setup/tear-down

* Merge branch 'master' into junit5ification

* Fix backup tests

* Don't createFile(); asCharSink does it

* Increase the timeout for all WebDriver tests to 60s (helps w/ flakiness)
2020-07-28 14:18:16 -04:00
gbrodman
e0d04cec4f Set up deployment of the Spec11 pipeline with JPA TM (#716)
* Set up deployment of the Spec11 pipeline with JPA TM

* Remove unnecessarily pipeline options setting

* Use enviroment name in BeamJpaModuleTest

* Fix checkstyle error
2020-07-27 21:04:52 -04:00
Michael Muller
0ce431212e Add the :nom:generate_golden_schema pseudo-task (#718)
Add a "pseudo-task" in nom_build to do the three step process of generating
the golden schema.  In the course of this, add support for pseudo-tasks in
general, improve the database directory readme and make nom_build not call
gradlew if there are no tasks.
2020-07-27 18:33:16 -04:00
gbrodman
32868b3ab8 Run the (Un)lockDomainCommand in an outer JPA txn (#688)
* Run the (Un)lockDomainCommand in an outer JPA txn

There are a couple things going on here in this commit.

First, we add an external JPA transaction in the
LockOrUnlockDomainCommand class. This doesn't appear to do much, but it
avoids a situation similar to deadlock if an error occurs in Datastore
when saving the domain object. Specifically, DomainLockUtils relies on
the fact that any error in Datastore will be re-thrown in the JPA
transaction, meaning that any Datastore error will back out of the SQL
transaction as well. However, this is no longer true if we are already
in a Datastore transaction when calling DomainLockUtils (unless, again,
we are also in a JPA transaction). Basically, we require that the outer
transaction is the JPA one.

Secondly, this just allows for more breakglass operations in the lock or
unlock domain commands -- in a situation where things possibly go
haywire, we should allow admins to make sure with certainty that a
domain is locked or unlocked.

* Add more robustness and tests for admins locking locked domains

* Fix expected exception message in tests
2020-07-27 18:16:24 -04:00
Shicong Huang
0ecc20b48c Rename a V40 flyway file to V41 to resolve conflict (#719) 2020-07-27 15:16:01 -04:00
Shicong Huang
c65af4b480 Add remaining columns to Domain's SQL schema (#702) 2020-07-27 13:32:39 -04:00
Legina Chen
3a15a8bdc7 Drop foreign key constraint for Registrar table (#715) 2020-07-27 09:05:30 -07:00
Weimin Yu
9806fab880 Use rearranged sql credentials in flyway task (#712)
* Use rearranged sql credentials in flyway task

Let the flyway tasks use the sql credential files set up for BEAM
pipelines.

Credential files have been created for each environment in GCS
at gs://${project}-beam/cloudsql/admin_credential.enc. All
project editors have access to this file, including the Dataflow
control service account.

Alpha and crash use the 'nomulus-tools-key' in their own project to
decrypt the credential file.

Sandbox and production use the 'nomulus-tools-key' in
domain-registry-dev to decrypt the credential file.

Note that this setup is temporary. It will become obsolete once
we migrate to Cloud Secret Manager for secret storage.
2020-07-24 15:32:01 -04:00
Weimin Yu
6591e0672a End-to-end Datastore to SQL pipeline (#707)
* End-to-end Datastore to SQL pipeline

Defined InitSqlPipeline that performs end-to-end migration from
a Datastore backup to a SQL database.

Also fixed/refined multiple tests related to this migration.
2020-07-24 09:57:43 -04:00
Ben McIlwain
91b7d92cf8 Upgrade TestPipeline extension from JUnit 4 to 5 2020-07-23 21:21:58 -04:00
Ben McIlwain
33910613da Get presubmits passing
This involves Guava -> Java 8 util migrations and fixing the license header.
2020-07-23 21:21:58 -04:00
Ben McIlwain
1fde678250 Copy TestPipeline rule from Apache Beam project into our codebase
This is copied in here with the absolute minimum # of modifications required
(just a rename to JUnit 5 format and some small fixes required to enable
compilation to be successful).

This is in preparation for the next commit where I'll convert this Rule into a
JUnit 5 extension, which is the entire goal here. But I wanted to get the code
from Apache Beam in with the maximum possible fidelity so that my changes will
be in a separate commit and will thus be obvious.

Note that we do unfortunately need to modify/rewrite the Rule itself; merely
wrapping it in some manner isn't possible.
2020-07-23 21:21:58 -04:00
gbrodman
8d56577653 Don't run presubmits over the .git folder (#711) 2020-07-23 18:12:34 -04:00
Ben McIlwain
3891d411de Upgrade most of remaining tests from JUnit 4 to JUnit 5 (#708) 2020-07-23 15:43:59 -04:00
gbrodman
cadecb15d8 Rename the email field in UI and include rlock email if it exists (#697)
* Rename the email field in UI and include rlock email if it exists

* Change the capitalization of fields and titles and add a description
2020-07-23 14:30:12 -04:00
gbrodman
9b7f6ce500 Fix some SQL credential issues identified when deploying Beam pipelines (#706)
* Fix some SQL credential issues identified when deploying Beam pipelines

There are two issues fixed here.
1. Without calling `FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create()), the Nomulus tool doesn't know how to handle gs:// scheme files. Thus, if you try to deploy (for instance) the Spec11 pipeline using a GCS credential file, it fails.
2. There was a misunderstanding before about what the credential file
actually refers to -- there is a credential file in JSON format that is
used for gcloud authorization, and there is a space-delimited SQL access
info file that has the instance name, username, and password. These are
separate options and should have separate command-line params.

* Actually we don't need this for remote deployment
2020-07-22 16:52:31 -04:00
Ben McIlwain
cd23748fe8 Upgrade rest of tools test classes to JUnit 5 (#705) 2020-07-22 11:09:21 -04:00
Ben McIlwain
cf41f5d354 Upgrade all remaining flows tests to JUnit 5 (#704) 2020-07-21 19:52:33 -04:00
Ben McIlwain
9a5ba249db Upgrade converters/TMCH/RDAP to JUnit 5 (#703)
Also renames some existing Rules to Extensions (and removes JUnit 4 features
from them entirely if no longer being used).
2020-07-21 18:48:41 -04:00
Shicong Huang
f5186f8476 Merge two PremiumList entities (#690) 2020-07-21 18:18:52 -04:00
Lai Jiang
4e0ca19d2e Remove IDN elements from BRDA (#670)
Also added unit tests for RdeStagingReducer.
2020-07-21 15:29:32 -04:00
Ben McIlwain
c812807ab3 Upgrade mapreduce and DNS tests from JUnit 4 to JUnit 5 (#701)
* Upgrade mapreduce and DNS tests from JUnit 4 to JUnit 5

* Merge branch 'master' into junit5-batch-and-dns
2020-07-20 21:33:24 -04:00
Ben McIlwain
9edb43f3e4 Upgrade command test classes from JUnit 4 to JUnit 5 (#700)
* Convert first batch of command tests to JUnit 5

* Upgrade rest of command tests to JUnit 5

* Migrate the last few test classes
2020-07-20 20:45:52 -04:00
gbrodman
b721533759 Create an ImmutableObjectSubject for comparing SQL objects (#695)
* Create an ImmutableObjectSubject for comparing SQL objects

Many times, when comparing objects that are loaded in from / saved to
SQL in tests, there are some fields we don't care about. Specifically,
we might not care about the last update time, revision ID, or other
things like that that are autoassigned by the DB. If we use this, we can
ignore those fields while still comparing the other ones.

* Create an ImmutableObject Correspondence for more flexible usage
2020-07-20 13:14:09 -04:00
gbrodman
ce35f6bc93 Include the user's registry lock email in the lock/unlock modal (#696)
* Include the user's registry lock email in the lock/unlock modal
2020-07-20 12:01:34 -04:00
gbrodman
f7a67b7676 Add a 'Host' parameter to the relock action enqueuer (#699)
* Add a 'Host' parameter to the relock action enqueuer

I believe this is why we are seeing 404s currently -- we should be
specifying the backend host as the target like we do for the
resave-entity async action.
2020-07-17 15:35:44 -04:00
gbrodman
4438944900 Validate potentially-invalid domain names when (un)locking domains (#698)
* Validate potentially-invalid domain names when (un)locking domains
2020-07-17 12:05:19 -04:00
Legina Chen
a22998e1bc Change clientId to registrarId to resolve the bug wrt the mismatch of the variable names (#692) 2020-07-15 11:03:52 -07:00
Weimin Yu
03d02ab299 Fix JpaIntegrationRule in JUnit4 (#687)
* Fix JpaIntegrationRule in JUnit4

Made DatastoreExtension a JUnit4 Rule.

Nomulus model objects need Datastore API when manipulating Ofy keys.
As a result, JpaIntegrationTestRule must be used with AppEngineRule
or DatastoreExtension.

Also fixed WriteToSqlTest, which is the only JUnit4 test that uses
JpaIntegrationTestRule.
2020-07-15 10:42:33 -04:00
Lai Jiang
47f65f70ab Fix a typo (#689) 2020-07-15 10:39:11 -04:00
Weimin Yu
1aa1f351bf Run rdeStaging twice daily in Sandbox (#684)
* Run rdeStaging twice daily in Sandbox

This will allow the cursor to catch up to current date if
it somehow falls behind.
2020-07-14 14:54:34 -04:00
Weimin Yu
94c8c6b9f3 Add lastUpdateTime column to epp resources (#683)
* Add lastUpdateTime column to epp resources

Property was inadvertently left out.

Renamed getter and setter to match the property name.

Added a test helper to compare EppResources while ignoring
lastUpdateTime, which changes every time an instance is persisted.
2020-07-14 14:53:05 -04:00
gbrodman
e74a9e6f02 Allow overrides of ContactBase methods (#681)
Hibernate might (will?) need to override these, so they shouldn't be
final.
2020-07-14 14:47:47 -04:00
gbrodman
37d3cc44b4 Fix small naming issue in a test (#685) 2020-07-14 13:57:44 -04:00
Lai Jiang
c844c8e9b1 Add the ability to parse PKCS#8 private key in PEM file (#682) 2020-07-14 11:20:00 -04:00
gbrodman
f747610533 Include the relock action in the web.xml routing file (#680) 2020-07-13 21:57:35 -04:00
Shicong Huang
e1db357fc3 Merge two reserved list entities (#616)
* Merge reserved list

* Replace INSTANCE with getInstance()

* Fix broken test

* Rebase on master

* Simplify class
2020-07-13 13:40:34 -04:00
Weimin Yu
ba1915e271 Write one PCollection to SQL (#664)
* Write one PCollection to SQL

Defined a transform that writes a PCollection of entities to SQL using
JPA. Allows configuring parallelism level and batch size.
2020-07-13 13:34:01 -04:00
Shicong Huang
58618a274e Add two folders of auto-generated Java classes to .gitignore (#679) 2020-07-13 10:09:56 -04:00
Lai Jiang
e4d0571125 Increase the maximum number of nodes in a nood pool to 15 (#672) 2020-07-10 21:54:18 -04:00
Ben McIlwain
4cb88ab6e7 Convert RDE tests (and some test rules) from JUnit 4 to JUnit 5 (#677)
* Add JUnit Params and start using it

* Convert rest of RDE tests

* Don't check headers for generated tests

* Expand visibility to fix build breakage

* Bump JUnit versions to 5.6.2
2020-07-10 21:32:36 -04:00
gbrodman
987f390ff7 Run 'npm audit fix' to fix low-severity vulnerabilities in packages (#676) 2020-07-10 15:57:59 -04:00
Ben McIlwain
ca756e14e6 Migrate all model tests from JUnit 4 to JUnit 5 (#675)
* Make first handful of tests JUnit 5

* Migrate rest of model package to JUnit 5
2020-07-10 14:56:28 -04:00
Ben McIlwain
caa0cd9d61 Add a "coreDev" gradle target (#667)
* Add a "buildFmt" gradle target

This does the same thing as the automatic Java build target, except instead of
failing if the code formatting isn't correct, it just automatically reformats as
necessary and continues on.

* Remove unnecessary mustRunAfters

* Make it run tests too, and add :taskTree task

* Rename task to coreDev and remove run afters

* Add task tree dependency

* Actually that may not be necessary
2020-07-10 10:03:59 -04:00
Legina Chen
7806cc7edb Add domainRepoId to Subdomain class (#674)
* Change Subdomain class to contain domainRepoId

* Remove jpaTm from Spec11PipelineTest and change clientId -> registrarId

* Remove 'client' from a comment

* Include changes to Spec11Pipeline

* add SafeBrowsingTransforms

* Run style
2020-07-09 16:26:35 -07:00
Lai Jiang
0964fdf1dc Upgrade to Gradle 6.5.1 (#673) 2020-07-09 14:04:22 -04:00
gbrodman
d17ec1fcb1 Use an enum instead of boolean in EntityTestCase constructor (#669)
* Use an enum instead of boolean in EntityTestCase constructor

It's more clear to use an enum rather than just a simple boolean

* Add Javadoc and make the enum name more verbose
2020-07-09 12:54:32 -04:00
Ben McIlwain
fac5987c13 Double the # of pubapi instances to better handle traffic spikes (#671)
* Double the # of pubapi instances to better handle traffic spikes

We may also consider switching to an automatic scaling mode soon, on the hope
that it's working better than the last time we tried it (it would help to keep
resource costs down at least).
2020-07-09 11:52:15 -04:00
Ben McIlwain
a3319e0026 Upgrade flow test classes to JUnit 5 (#666)
Most of the diffs are visibility changes.

Also deletes ShardableTestCase, which was only necessary because of Blaze (and
possible Bazel) limitations.
2020-07-08 14:08:05 -04:00
Weimin Yu
5578464e06 Make sure uncommitted txn is rolled back (#665)
* Make sure uncommit txn is rolled back

The try block around commit that catches RuntimeException should also
catch Error, which is also unchecked.
2020-07-06 17:39:13 -04:00
gbrodman
c24a61f813 Refactor ContactResource into ContactBase and create ContactHistory (#634)
* Create ContactHistory class + table

This is similar to #587, but with contacts instead of hosts.

This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest)

Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory.

* Move ContactResource -> ContactBase

* Alter ContactBase and ContactResource
2020-07-06 12:52:16 -04:00
gbrodman
806f3b2456 Verify that the RegistryLock input has the correct registrar ID (#661)
* Verify that the RegistryLock input has the correct registrar ID

We already verify (correctly) that the user has access to the registrar
they specify, but nowhere did we verify that the registrar ID they used
is actually the current sponsor ID for the domain in question. This is
an oversight caused by the fact that our testing framework only uses
admin accounts, which by the nature of things have access to all
registrars and domains.

In addition, rename "clientId" to "registrarId" in the RLPA object

* Change the wording on the incorrect-registrar message
2020-07-05 22:31:14 -04:00
gbrodman
333170a724 Allow users the option of seeing their registry lock password (#663)
* Allow users the option of seeing their registry lock password

Only when entering it for the first time, of course.
2020-07-05 20:08:22 -04:00
Lai Jiang
47eeb8c4e4 Output PO number in detailed report (#659)
* Output PO number in detailed report

The PO number header was added during the beam migration but we forgot
to print the actual data in the corresponding column. This resulted in a
misalignment of columns in the detailed report.

This PR fixes it. Note that we cannot drop PO number from the header (as is not
useful in the detailed report) because the header represents all fields
that are to be parsed from the SQL query results, and PO number *is*
needed when generating the invoice itself. By dual-purposing the header
(both as the required fields in the parser and the first line in the
detailed report) we have to include the value of PO number in the
detailed report CSV as well.
2020-07-01 19:09:05 -04:00
Shicong Huang
391929b518 Expand AckPollMessagesCommand to ack PollMessage.Autorenew (#647)
* Expand AckPollMessagesCommand to ack PollMessage.Autorenew

* Rebase on master and address comment

* Resolve comments
2020-07-01 15:06:35 -04:00
gbrodman
7f62b7a89c Include the registry lock email in the JS object as a sensitive field (#658)
* Include the registry lock email in the JS object as a sensitive field

* Change wording of exceptions to be more consistent
2020-07-01 13:05:21 -04:00
gbrodman
a1da32bfde Disambiguate injected Cloud SQL parameter names (#657)
* Disambiguate injected Cloud SQL parameter names

This allows us to also inject the BeamJpaModule into RegistryTool, which
allows us to use the SocketJpaTransactionManager in Beam pipelines.

Some side effects of this include:
- duplication of KMS connections -- one standard, one Beam
- duplication of the creation of the partial Hibernate SQL configs
- removal of ambiguity between credentialFileName, credentialFilename,
and credentialFilePath -- we now use the latter.
- Performing the credential null check when instantiating the SQL
connection rather than when instantiating the module object. See the code
comments for more details on this.

I verified that this compiles and the tests run successfully when
injecting a @SocketFactoryJpaTm into a Beam pipeline.

* Remove two unnecessary config points and change the name of two params

* Use @Config instead of @Named and change the pool size

* Replace non-visible link with code
2020-07-01 11:55:21 -04:00
Weimin Yu
1961a5759d Load Datastore snapshot from backup files (#660)
* Load Datastore snapshot from backup files

Defined a composite transform that loads from a Datastore export and
concurrent CommitLog files, identify entities that still exist at the
end of the time window, and resolve their latest states in the window.
2020-07-01 09:58:42 -04:00
Weimin Yu
d065ff63fc Exclude Test/Monitoring Registrars from escrow (#655)
* Exclude Test/Monitoring Registrars from escrow

Registrars used for testing and monitoring should not be included
in Data escrow. They also lack the required ianaIdentifier property
and would fail ICANN data validation.

Note that since alpha and crash environments have bad data that
break the RDE process, we need to verify this change in Sandbox.
2020-06-26 19:11:22 -04:00
Michael Muller
07ff6279bb Make EppResource.loadCached() use batched fetch (#652)
* Make EppResource.loadCached() use batched fetch

Use a batched fetch (ofy().load().keys(...)) from datastore in
EppResource.loadCached().

To support this, convert TransactionManager.load(Iterable<VKey>) to accept the
more flexible generic parameters and return a map.

* Simplify datastore key streaming

* Changes requested in review.
2020-06-26 13:50:02 -04:00
Legina Chen
5c5b6b20ce Allow multiple threat types in the Spec11ThreatMatch table (#650)
* Update to generic Spec11ThreatMatch table

* Fix SQL syntax

* Make changes to the schema and add a test for null and empty threatTypes

* Fix a small typo

* Change the exception thrown with illegal arguments

Change the import for isNullOrEmpty

* Fix import for checkArgument

* Added a threat to test multiple threat types
2020-06-26 10:35:00 -07:00
Ben McIlwain
74b2de5c35 Make ImmutableMap Stream collect()ion nicer (#654)
This adds an entriesToImmutableMap() collector that can be used in place of
toImmutableMap(Map.Entry::getkey, Map.Entry::getValue()).

It also fixes up some existing calls that use toImmutableMap() when terser
alternatives exist.
2020-06-26 11:57:26 -04:00
Ben McIlwain
fba6804d3b Properly handle restore fees on domain checks (#646)
* Properly handle restore command fees for domain checks

* Get tests working and handle fee classes better

* Remove unused ImmutableSet imports

* Fix code review concerns, mostly surrounding immutability

* Rename more targetIds and make them immutable too

* Merge remote-tracking branch 'upstream/master' into domain-check-restore-fees

* Fix Javadoc formatting
2020-06-26 10:59:46 -04:00
Lai Jiang
db5311075d Patch terraform changes made internally (#651)
There were several LSC that made some formatting changes to our .tf
files. Export these changes externally for consistency.
2020-06-25 13:59:37 -04:00
Ben McIlwain
6e26dacdff Make nomulus compatible with Java 11 (#649)
* Make nomulus compatible with Java 11

This fixes the double-spacing bug with logged EPP XML on App Engine that started
appearing after App Engine switching from using Java 8 to Java 11. Java 9 made
some changes to XML Transformer classes that needed a little bit of work to
accommodate.

This also fixes the unit tests that were failing in Java 11 (all of which were
related to said XML Transformer changes).

* Make code review changes
2020-06-25 13:17:22 -04:00
Weimin Yu
2e5466f32f Fix flaky tests that depends on order of rules
SystemPropertyRule in some cases should be applied last:
when multiple rules exist and and modified property is checked
in cleanups.

ConsoleOteSetupActionTest and ConsoleRegistrarCreatorActionTest
are two such classes, and can be flaky in JUnit 4. This PR
migrates them to JUnit5 and applies ordering to extensions in
them.

Added a mockito dependency, and upgraded mockito-core to 3.3.3.

Meaningful changes: SystemPropertyRule.java and
ConsoleOteSetupActionTest.java, and
ConsoleRegistrarCreatorActionTest.java
2020-06-24 23:14:47 -04:00
Lai Jiang
a65a3fd8b7 Add a property in Registry to disable invoicing (#644)
The added property will by default be null for existing entities and false for new entities. We could write a temporary MapReduce to set it to false for all existing ones, but it seems easier to just modify the query to account for the nullness of the column. The need to check for null won't exist once we migrate the Registry entity to Cloud SQL.

TESTED=deployed to alpha, set the property for tld zombo to true and triggered an export, verified that the column is null for all rows other than the row for zombo.
2020-06-24 16:21:38 -04:00
Weimin Yu
071800ae50 Delay SQL availability check during Tool start (#643)
* Delay SQL availability check during Tool start

During Cloud SQL migration, Cloud SQL password setup in Registry
server requires a few tool commands. Such commands must be able
to run before Cloud SQL is configured.

In this change, we deplay SQL credential loading until access is
needed.
2020-06-24 14:00:36 -04:00
Shicong Huang
75659671db Fix a warning from Hibernate (#645)
Hibernate complained that ContactResource.getTransferData should not be
final:
WARN: HHH000305: Could not create proxy factory
for:google.registry.model.contact.ContactResource
org.hibernate.HibernateException: Getter methods of lazy classes cannot
be final: google.registry.model.contact.ContactResource#getTransferData
2020-06-24 10:02:42 -04:00
Michael Muller
c9d4ffb233 Convert all VKeys to symmetric VKeys (#641)
* Convert hosts to symmetric VKey

* Convert ContactResource to symmetric VKeys

* Convert BillingEvents to symmetric VKeys

* Converted PollMessage to symmetric VKeys

* Convert AllocationToken to symmetric VKeys

* Remove static methods, get everything working

* Changes requested in review.

* Removed newly introduced createOfy() calls
2020-06-24 08:02:11 -04:00
Weimin Yu
11ec4d64f2 Set up JpaTransactionManager in BEAM pipelines (#639)
* Set up JpaTransactionManager in BEAM pipelines

Added modules and utilities to create JpaTransactionManager in BEAM
pipelines.

Not wanting to set up AppEngine Remote API to access Keyring in the
Datastore, we instead use the credential files in GCS, which are
used by Spinnaker/Cloud Build and desktop access. Added utility
to download, decrypt, and parse the file. Also added/modified dagger
modules.
2020-06-23 11:04:52 -04:00
Ben McIlwain
e9ad1b6f72 Don't expect a renewal fee on restores when one isn't due (#637)
* Don't expect a renewal fee on restores when one isn't due

This is a fix on top of #632 so that domain restore commands don't require
acking an illusory renewal fee for 1 year when that isn't actually happening
(i.e. if the domain isn't yet past its original expiration).

Unfortunately, there's still a problem remaining wherein the restore fee on a
domain check will always include the additional year even if it's not
necessary. We don't have a good solution to that. Also note that in versions of
the fee extension more recent than 0.6, the fee extension cannot be passed on a
domain info command at all, so the domain check command is the only way you have
to determine what the restore fee should be. So we definitely do want to get
that right so that the apparent restore fee on a check is the same as the actual
restore fee when running the restore command. We're not quite there yet though
and it's hard to say how we will get there, since we don't load domains during a
domain check command for performance reasons yet we would need to do so in order
to know the expiration date and thus whether the additional year of renewal
should be charged.

A problem for a future PR.
2020-06-22 15:24:36 -04:00
Shicong Huang
e7db9b3c1a Add a test to verify JPA supports polimorphic VKey (#642) 2020-06-22 15:13:38 -04:00
Legina Chen
ec09226baa Create a Java entity to store ThreatMatch objects in SQL (#617)
* Squash everything together
Create SafeBrowsing_Threats table

Create LocalDateConverter and add indexes to SafeBrowsingThreats

Add indexes to SafeBrowsingThreats and make small style changes

Pass in DateTimeFormatter

Delete LocalDateConverterTest.java

Rebase

Make changes to ThreatType comments

Create LocalDateConverterTest

Add review changes

Add SafeBrowsingThreatTest

Rename repoId, refactor LocalDateConverterTest/SafeBrowsingThreatTest, add foreign keys

Change imports

Add foreign keys and rename version number

Add new generated db-schema file

Clean up null test cases

Add changes

Add foreign keys into SafeBrowsingThreatTeat and apply style checks

Add SafeBrowsingThreatTest into SqlIntegrationTestSuite and change golden file

Make small changes to SafeBrowsingThreatTest

Add tests for ForeignKeyViolations and remove setId in SafeBrowsingThreat

* Change V35 -> V36

* Add a foreign key test for a reference to Registrar

* Move some variables around
2020-06-22 12:07:59 -07:00
Weimin Yu
2c243a7d5f Add testcontainers' Junit5 support dependency (#640)
* Add testcontainers' Junit5 support dependency

Also updated guava, dagger, hibernate, postgresql, and cloud socket factory
to latest version.

Migrated PersistenceModuleTest as an example.

Real changes:
- dependencies.gradle
- core/build.gradle
- PersistenceModuleTest.java
2020-06-22 13:12:58 -04:00
Ben McIlwain
df8ce38796 Rename whitelist -> allow list (#635)
* Rename whitelist -> allow list

* Merge branch 'master' into allowlist-denylist
2020-06-18 18:36:05 -04:00
Weimin Yu
382c8014de Fix flakiness caused by ofy entity name conflict (#636)
* Fix flakiness caused by ofy entity name conflict

Overrode the 'kind' of two test entities to prevent name conflicts.
Tests are flaky because of this.

Added a check in AppEngineRuleTest for conflicting kinds.
2020-06-18 13:14:10 -04:00
Ben McIlwain
57113b4746 Show price of reserved domains when using matching allocation token (#632)
* Show price of reserved domains when using matching allocation token

When the registrar passes the fee extension, this shows the price of the domain
on a check command for reserved domains if the provided allocation token is a
match. Of course, the price is already always displayed on non-reserved names
(regardless of whether the specific provided token is a match or not).

This affects domain checks only; the price is already always displayed on domain
creates because you already by definition have access to register the domain in
question.
2020-06-18 11:57:22 -04:00
Weimin Yu
d43564172f Refactor pipline for Datastore backup loading (#628)
* Refactor pipline for Datastore backup loading

Refactored pipeline transforms.

Added testing utilities that handles assertions better.

Investigated and documented challenges in serializing Ofy entities
without side effects.
2020-06-17 22:10:14 -04:00
gbrodman
69a1d04c18 Remove 'fullyQualified' from host and domain names (#631)
* Remove 'fullyQualified' from host and domain names

We don't actually enforce that these are properly fully-qualified
(there's no dot at the end) and we specifically use the term "label
name" when talking about labels.

Note: this doesn't convert FQDN -> DN (et al) in at least two types of
cases:
1. When the term is part of the XML schema
2. When the term is used by some external system, e.g. SafeBrowsing API

* Add TODO to rename fields
2020-06-17 16:19:26 -04:00
Michael Muller
19395def5b Implement a persistable Transaction object (#614)
* Implement a persistable Transaction object

Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.

* Changes requested in review

* Add a mujtation count to the persisted format
2020-06-17 14:16:48 -04:00
Shicong Huang
2f600e3e69 Implement remaining methods in JpaTransactionManager (#633) 2020-06-17 12:39:34 -04:00
Michael Muller
31841ccc55 Fix cookie processing for RDAP URL update (#630)
* Fix cookie processing for RDAP URL update

The existing code only does cookie processing on the _first_ Set-Cookie
header.  Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.

Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
2020-06-16 15:07:13 -04:00
gbrodman
4f37c65af5 Fix versioning semantic merge conflict (#629) 2020-06-16 12:51:28 -04:00
gbrodman
47178d4fb5 Add HostBase and HostHistory classes (#587)
* Add proof of concept for HostBase and HostHistory classes

* Use a PROPERTY accessor for @Ids

* Add an unused setter method for Hibernate's sake

* Refactor HostHistory

* Some responses to CR

* Fix relationship and test

* Manually manage the foreign keys for HostHistory

* Protect HostBase's builder and use text for the enum type

* Add responses to CR

- Add javadocs
- Create an ID sequence for host history objects

* Don't try to set the ID

* Use a Long and remove the setter

* Add some comments and rename a couple fields

* Don't change Datastore schema

* Use Long in the Datastore schema

* Add new createVKey method

* Add comments and rename fields

* Rename v27->v31 and regenerate the golden

* Fix superordinateDomain and inetAddresses in HostHistory

* V31 -> V32

* Fix SQL files that got messed up in the merge

* Configure and use a manually-created history ID sequence

* Add three more indices to HostHistory
2020-06-16 11:47:17 -04:00
Shicong Huang
26e2a51180 Refactor TransferData to remove unused fields in Contact table (#623)
* Add DomainTransferData and ContactTransferData

* Refactor TransferData to remove unused fields in Contact table

* Add scope for TransferData's type parameter
2020-06-16 10:42:57 -04:00
Michael Muller
21f2f38ad1 Allow class-specific creation of symmetric VKeys (#625)
* Allow class-specific creation of symmetrict VKeys

When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.

As a test case for this, implement the createVKey() static function for
DomainBase.  Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
2020-06-15 11:35:03 -04:00
Lai Jiang
0e04f6ca5b Upgrade to Gradle 6.5 (#627) 2020-06-15 10:57:29 -04:00
Ben McIlwain
4be70c8509 Refactor Fee handling so that each fee knows if it's premium (#626)
* Refactor Fee handling so that each fee knows if it's premium

This is a noop for now, as the new isPremium boolean isn't yet used by anything,
but it will be used in follow-up PRs to add additional fee information using the
fee extension (see: b/157621273).  Specifically what we're trying to do here is
return <fee:command name="create" standard="1"> (using the finalized version of
the fee extension) when an entire command has no premium fee associated with
it. And in the current earlier versions of the fee extension that we support,
we'll want to display the correct fee amount and class for creates/checks on
reserved domains when a valid allocation token is passed. This also needs the
isPremium information.

There are no testing implications yet because isPremium isn't exposed anywhere,
but there will definitely be lots of test changes once it's feeding into EPP-
visible changes.

* Rename things, add method Javadoc

* Apply formatting
2020-06-12 16:43:02 -04:00
Ben McIlwain
cf1448bca8 Restore the original expiration time on domain restore (#601)
* Restore the original expiration time on domain restore

Except if that time is now in the past, then add a year to it.

* Apply auto-formatter changes to fix my local build

* Merge branch 'master' into restore-expiry-date

* Fix reversed comments
2020-06-12 14:33:49 -04:00
gbrodman
f62473542f Use the requested server host when creating the registry lock verification URL (#624)
* Use the server host when creating the registry lock verification URL

The app doesn't know about any external configuration that may point to
this app, so there's no way of finding out that, for instance,
registry.google points to the app. Thus, we have to use what the user
gives us so that, in our case, the registry-lock verification
emails can point to https://registry.google/registry-lock-verify instead
of https://domain-registry.appspot.com/registry-lock-verify. The former
is used by clients / users to authenticate, and unfortunately
authenticating on registry.google does not give authentication to
domain-registry.apspot.com.

Tested using the RDAP code that uses getServerName() -- in that case, if
you access registry.google/rdap/<>, it uses registry.google in the URLs
but if you use domain-registry.appspot.com/rdap/<>, it uses
domain-registry.appspot.com in the URLs.

Relatedly, frontend_config_prod-appengine.asciiproto in Piper
is what configures registry.google to point to
domain-registry.appspot.com
2020-06-12 10:11:53 -04:00
Lai Jiang
484173b659 Log client certificate type and length (#622)
* Log client certificate type and length

It appears that most client certificates are RSA certs, but we should
make sure that is indeed the case. Print out the strength of the cert
if it is RSA.

Also adds supports for TLS 1.3 and print out the supported cipher suites.

* Add a comment about zero length certificate

* Make length of non-RSA keys -1

* Don't use TLS 1.3 if JDK SSL provider is used
2020-06-11 17:11:40 -04:00
Weimin Yu
d3fd826dc1 Load CommitLog into BEAM pipeline (#618)
* Load CommitLog into BEAM pipeline

Created tools that can generate CommitLogs on a test
datastore.

Defined BEAM transforms that load from CommitLog files and
added a few simple tests.

This is work-in-progress. Next step is to build a consistent
backup by merging Datastore exports and CommitLogs.
2020-06-11 11:38:53 -04:00
gbrodman
1c62728886 Rename V30 -> V31 to avoid duplicates (#621) 2020-06-10 16:08:31 -04:00
Lai Jiang
b5d3186e67 Update Netty to the latest version (#620)
* Upgrade to the latest version of Netty

* Update lock files
2020-06-10 16:08:11 -04:00
gbrodman
b4dfec5fd5 Rename client_id to registrar_id in SQL (#619)
We'll eventually want to shift everything over to using registrar_id and
registrarId rather than client_id and clientId but for the sake of the
Datastore schema and existing code, we won't change the Java identifier
for now. Once we're completely and only on SQL, we can rename the Java
field easily.
2020-06-10 15:11:27 -04:00
gbrodman
40b14fb695 Create a converter for sets of InetAddresses and use it in HostResource (#612)
* Create a converter for sets of inetAddresses and use it in HostResource

This can just be a set of strings where each string represents an
address;  there's no need for it to be a separate table. This allows
for simplification of the SQL schema.

* Regenerate golden SQL file after renaming v28 -> v29

* Add more tests and rename a typo in the file

* Refactor common test code and use tm methods

* Use JUnit5 API

* Rename test entity
2020-06-10 13:04:20 -04:00
Shicong Huang
fdac686250 Add columns for TransferData in Domain and Contact (#577)
* Add columns for TransferData in Domain and Contact

* Rename flyway file and foreign key

* Rebase on master and address comment

* Compileable commit

* Fix unit test

* Refactor TransferServerApproveEntity

* Use tm().delete(vkeys)

* Rename transfer_period fields

* Rename client_id to registrar_id

* Rebase on master

* Resolve comment

* Rebase on master
2020-06-09 16:39:55 -04:00
Shicong Huang
f0765dc893 Make JpaUnitTestRule not depend on the nomulus schema (#613) 2020-06-09 14:51:13 -04:00
Lai Jiang
2995bb03fd Update Javadoc URL (#615) 2020-06-09 10:25:56 -04:00
gbrodman
0f415f78a6 Use the correct text VKey for HostResource's superordinateDomain (#608)
* Store the superordinateDomain reference as a VKey rather than Key

This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
2020-06-08 12:21:51 -04:00
gbrodman
b324fb98d3 Only use asymmetric VKeys for EPP resources (#611)
Given that we currently have no way of reconstituting a symmetric key
from the asymmetric key (or at least, we don't have a 100% reliable way
of doing so) it's best to keep keys as asymmetrical, referring to the
correct database. That way, we don't get situations where we cannot
compare equality of two keys due to one being asymmetrical and one being
symmetrical.
2020-06-05 16:55:12 -04:00
Lai Jiang
d27fe8ead5 Fix a typo (#610) 2020-06-05 15:53:17 -04:00
Lai Jiang
da65a38782 Add a GCB job to build and publish javadoc (#609) 2020-06-05 13:00:15 -04:00
Legina Chen
5a1f3d0376 Remove platformType and threatEntryMetaData fields from ThreatMatch (#607)
* Remove platformType and threatEntryMetaData fields from ThreatMatch

* Run google-java-format on both files

* Add test for removal of unnecessary fields

* Removed unnecessary fields from Spec11PipelineTest.testEndToEndPipeline_generatesExpectedFiles

* Added style check

* Fix typo
2020-06-05 09:00:07 -07:00
Shicong Huang
b1241b98b2 Generate sql schema for PollMessage (#582)
* Generate sql schema for PollMessage

* Rework columns and resolve comments

* Fix datastore schema
2020-06-04 18:24:59 -04:00
Shicong Huang
b42ded9451 Add test to verify the behavior of @DualDatabaseTest (#606) 2020-06-03 14:55:37 -04:00
Shicong Huang
472503541b Add deleteAll method to TransactionManager (#604)
* Add deleteAll method to TransactionManager

* Rename deleteAll to delete

* Add bucket.getLastWrittenTime() before second mutation
2020-06-03 10:02:48 -04:00
Weimin Yu
ed64dd3548 Load raw records from Datastore export (#605)
* Load raw records from Datastore export

Created a tool that can export from a test instance of Datastore.

Defined Beam pipeline transforms for loading raw records back from
the export.

This is the first part of the effort to create a consistent snapshot
of Datastore. The next step is to load entity records from CommitLog
files.
2020-06-02 18:55:03 -04:00
Michael Muller
6a96b1a9cd Use TransactionManager for hosts and contacts (#603)
* Use TransactionManager for hosts and contacts

Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts.  In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.

* Minor formatting fix
2020-06-02 13:17:16 -04:00
Michael Muller
c23d4f3ba5 Add createVKey() at the EppResource level (#600)
* Add createVKey() at the EppResource level

Also convert createKey() to createVKey() to normalize with what we've settled
on.
2020-05-29 08:36:57 -04:00
Weimin Yu
2b794347e6 Refactor LevelDbFileBuilder to accept DS Entity (#599)
* Refactor LevelDbFileBuilder to accept DS Entity

Builder now can directly work with Datastore Entity objects.
No need to wrap data in ComparableEntity.
2020-05-28 13:38:00 -04:00
Shicong Huang
26fb5388a4 Generate sql schema for BillingEvent (#565)
* Generate sql schema for BillingEvent

* Change to use sequence

* Address comments

* Resolve warnings and remove duplicate cost related fields

* Increase the flayway file version to V25

* Remove extra space

* Split to 3 tables, merge VKey

* Rename talbes

* Rename repoId to domainRepoId

* Exclude VKey in schema.txt

* Rename target_id to domain_name

* Fix javadoc

* Resolve comments
2020-05-27 15:59:19 -04:00
Lai Jiang
bd443633f6 Add a task to compile javadoc across all packages (#597)
Also fixes various issues that prevent javadoc compliation.
2020-05-27 10:33:46 -04:00
Weimin Yu
d87f119b36 Add a test for SQL logging config (#598)
* Add a test for SQL logging config

Verifies that SQL statements are logged by Hibernate when
configured to do so.
2020-05-26 16:25:33 -04:00
Weimin Yu
54f1357d83 Fix show-sql which stopped working (#596)
* Fix show-sql which stopped working

Made show-sql property configurable in JpaUnitTestRules.

Added a few comments on foreign key constraint behavior.
2020-05-21 12:20:56 -04:00
Lai Jiang
c73d154084 Do not enqueue update snapshot task if import fails (#578)
If the import from Datastore to BigQuery fails, there is no point
enqueuing a job to update the snapshot view.

Also when there's an error updating the snapshot view, log it at severe
level. The HTTP exception thrown is logged at info and triggers a retry
implicitly. I'm not sure if we want this behavior though. Do we want to
retry upon snapshot updating failures? Unless the failurs are transient,
retrying doesn't help. In our case the failure (End of time out of range
in Standard SQL) is not transient.
2020-05-21 11:40:45 -04:00
gbrodman
259d2e2cdc Run "npm audit fix" to fix a vulnerability (#592) 2020-05-20 15:12:27 -04:00
Shicong Huang
0f174d9ce0 Add all existing entities to VKeyTranslatorFactory (#595)
EntityClasses.ALL_CLASSES has all of our registered entities so
we can use it to initialize VKeyTranslatorFactory.classRegistry
to avoid adding them one by one.

Also, this PR changed to use Key.getKind() to get the kind of
the entity to solve the problem that when the entity class
is an inner class, its kind should still be the class name
instead of OuterClass$InnerClass.
2020-05-20 14:24:45 -04:00
Weimin Yu
ca2edb6a17 Close input channel in LevelDbLogReader (#594)
* Close input channel in LevelDbLogReader

Input channel should be closed when all data has been read.
2020-05-20 12:54:13 -04:00
Weimin Yu
3947ac6ef7 Read LevelDb incrementally (#593)
* Read LevelDb incrementally

Made LevelDbLogReader an iterator over a LevelDb data stream,
Reducing memory footprint which is important when used in a
Dataflow pipeline.
2020-05-20 10:26:34 -04:00
Michael Muller
579a3d0ac1 Make VKey persist to datastore as a key (#591)
* Make VKey persist to datastore as a key

Convert nsHosts entirely to VKey as a proof-of-concept.

Tested as follows:
    1) Deployed to crash, verified that nameservers were visible for several
       domains (indicating that we are able to load a set of Keys as VKeys)
    2) Updated the set of nameservers for a domain (removing some initial
       hosts) and verified that the changes went through.
    3) Deployed the old version to crash, verified that I was able to retrieve
       the newly saved VKeys as Keys.
    4) Modified the hosts for the same domain (adding back one of the hosts)
       and verified that the change took effect.
    5) Redeployed this change to crash, again updated the nameservers to add
       another host.
    6) Again restored the old version, verified that the new hosts were
       visible.

* Changes in response to review

* Convert to a single VKeyTranslatorFactory instance

* Moved vkey field rename to V25
2020-05-19 14:10:28 -04:00
Lai Jiang
5fe929b027 Log InternalServerErrorException at SEVERE (#585)
Normal HttpException logs at INFO because they usual do not indicate
anything out of the ordinary and is meant to convey to the client that
there is some expected error. However InternalServerErrorException is
something that we do care about being alerted for so we log it at SEVERE.
2020-05-18 22:55:13 -04:00
Lai Jiang
fb335b7d89 Upgrade to Gradle 6.4.1 (#590) 2020-05-18 16:47:02 -04:00
Shicong Huang
a0f4013d53 Add JUnit5 extension to run test twice against different databases (#588)
* Add JUnit5 extension to run test against different databases

* Fix typos

* Add some explanation
2020-05-18 11:06:21 -04:00
Lai Jiang
5e596bb389 Upgrade to Gradle 6.4 (#589) 2020-05-14 14:57:24 -04:00
Lai Jiang
f62fd82803 Log information about SSL connection from the client (#586) 2020-05-14 09:38:33 -04:00
sarahcaseybot
b7353ef338 Add TimedTransitionProperty Converters (#561)
* Add TimedTldStateTransitionMapConverter

* Move timedTransitions to a base class and add BillingCostTransitionConverter

* Add test of TimedTransitionPropertyConverterBase

* clean up tests

* Switch tests to JUnit 5

* Make JpaUnitTestRule an extension
2020-05-12 11:46:19 -04:00
Michael Muller
832e1ce047 Implement all DatastoreTransactionManager methods (#581)
* Implement all DatastoreTransactionManager methods

In the course of this:

- Make assertDelete() specific to JpaTransactionManager, remove the return
  value from delete()
- Converter "in transaction" assertion to IllegalStateException, which is less
  JPA specific.

* Upgraded DatastoreTransactionManagerTest to junit5
2020-05-11 17:17:57 -04:00
0xflotus
8087f5bbca (docs): fixed small errors (#572)
* Update first-steps-tutorial.md

* Update proxy-setup.md (#1)

* Update registrar-faq.md (#2)

* Update first-steps-tutorial.md
2020-05-11 10:01:47 -04:00
Ben McIlwain
7f3dbfb62f Reflect refunded billing events on deletion in expiration time (#579)
* Reflect refunded billing events on deletion in expiration time

This doesn't make any change at the time of the domain deletion itself, but it
will matter if the domain is then undeleted, because we need to know what
expiration date to restore, and if there were any renew or autorenew charges
that were refunded by the deletion because they were in a grace period, they
shouldn't be coming back during the restore.

* Add tests for new expiration date behavior

* Add handling of add/renew grace period overlap
2020-05-08 21:51:20 -04:00
Michael Muller
04f429c4d6 Convert DomainBase's contacts to VKeys (#574)
* Convert DomainBase's contacts to VKeys

Convert usage of DomainBase contacts from Key to VKey.  This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
2020-05-07 11:19:15 -04:00
Lai Jiang
40a4c3101c Fix test failures on certain machines (#571)
On certain machines (one of mine) the outcastTest consistently fails due
to the following error:

java.lang.NoClassDefFoundError: Could not initialize class
google.registry.persistence.transaction.JpaTestRules$JpaIntegrationTestRule

If I reduce maxparallelForks to 3 it consistently passes. This issue was
mentioned here:

https://discuss.gradle.org/t/junit-test-fails-with-noclassdeffounderror-only-when-maxparallelforks-1/6047

But this post was 8 years old and no solution was identified.
2020-05-04 11:38:54 -04:00
Michael Muller
e2dfb6488d Improve return value semantices for tm().load() (#576)
Since we rarely (if ever) want to check the result of a single element load,
make TransactionManager.load(VKey) return non-optional, non-nullable and just
throw a NoSuchElementException if the entity is not found.

Also add a maybeLoad() that does return an optional in case we ever want to do
this (exists() should suffice for an existence check).
2020-05-04 10:49:36 -04:00
gbrodman
c5aa0125ab Implement DatastoreEntity and SqlEntity on more classes (#570)
* Implement DatastoreEntity and SqlEntity on more classes

For classes that aren't going to transition to SQL, they should just
return an empty list of SqlEntities. When reading these in from the
commit log manifests, we just won't persist anything to SQL.

By having all Datastore entity classes implement DatastoreEntity, we can
avoid potential bugs where we forget to transition some entity to SQL,
or we forget to have the capability to read back from the commit logs.

Note: the EntityTest is still @Ignore'd because there are many SQL and
Datastore classes left -- ones that we are still in the process of
converting or adding, or ones that require more complicated transitions.

Note: Locks and Cursors aren't converted (even though we could) because
they're ephemeral

* Responses to CR

Add a @EntityForTest annotation
fix null that snuck in

* Keep the test ignored for now
2020-05-01 17:04:13 -04:00
Shicong Huang
01e2d24658 Revert "Remove minimumIdle config in HikariCP (#557)" (#573)
This reverts commit d8066ca752.
2020-05-01 16:21:10 -04:00
Shicong Huang
19bc1c9c9c Add annotation processor to generate converter for VKey (#566) 2020-04-29 17:29:05 -04:00
gbrodman
c361c9e601 Remove email-editing footgun (#503)
* Remove email-editing footgun

Email address is used as the primary key so we should be very careful
about changing it. This will have even more importance when this is the
location to which we will be sending registry lock confirmation emails.

Note: we allow addition or removal of contacts through the UI (and don't
want to disable that) and because all edits are performed by saving the
entire list of contacts, we can't explicitly prevent all possible edits
of email address in the backend. So this doesn't technically prevent
anything security-wise, but it makes it much more difficult to
accidentally edit an email when you shouldn't.

* Enforce non-deletion of registry-lock-enabled contacts

* Fix tests

* Specify contact
2020-04-29 11:44:51 -04:00
Weimin Yu
aa0dcea537 Fix flaky tests due to Entity name conflicts (#569)
* Fix flaky tests due to Entity name conflicts

Objectify siliently replaces current registration of a given kind
when another class is registered for this kind. There are
several TestObject classes in the current code base, which by
default are all mapped to the same kind.

Tests have only been flaky because impacted tests need to run
in specific orders for failures to happen. Using multiple executors
in Gradle also reduced the likely hood of errors. To reproduce the
problem run the following tests in order (e.g., by putting them in
a test suite):
1. ExportCommitLogDiffActionTest
2. CreateAutoTimestampTest
3. RestoreCommitLogsActionTest

In this PR, we
- Made sure all entities have unique kinds.
- Made all test entities register with AppEngineRule instead of directly
  with ObjectifyService.
- Added code in AppEngineRule to check for re-registrations.
- Added presumit check for forbidden direct registration.
2020-04-28 15:32:42 -04:00
sarahcaseybot
e920e4d201 Remove Lock Dual Read and Dual Write (#568) 2020-04-27 17:30:51 -04:00
Ben McIlwain
cd13f6c5d3 Allow the nomulus renew_domain command to specify the client ID (#567)
* Allow the `nomulus renew_domain` command to specify the client ID

This means that a superuser can renew a domain and have the associated history
entry, one time billing event, and renewal grace period be recorded against a
specified registrar rather than the owning registrar of the domain.  This is
useful to e.g. renew a domain for free by "charging" the renewal to the
registry's fake registrar.  Since the grace period is written to the specified
cliend id as well, if the actual registrar deletes the domain, they don't get
back the money that they didn't pay in the first place.
2020-04-24 18:06:27 -04:00
Ben McIlwain
210de9340e Don't NPE when nomulus tool is run without a subcommand (#564)
* Don't NPE when nomulus tool is run without a subcommand

This occurred when an environment was specified but without a subcommand. Now,
the list of valid subcommands is outputted instead of seeing a generic NPE.

This also makes some formatting changes in other files that were causing the
incremental format check to fail.

* Try AppEngineRule
2020-04-24 17:32:58 -04:00
Michael Muller
5d58be6f0a Remove separate deployment of persistence.xml (#563)
* Remove separate deployment of persistence.xml

We added a step to explicitly copy persistence.xml because for some reason it
wasn't originally getting deployed to app-engine, resulting in failures on
startup.  However, this file is now included in core.jar and we are now
getting a warning about multiple persistence units with the same name as it
reads the files from both the filesystem and core.jar.
2020-04-23 13:35:37 -04:00
Shicong Huang
d8066ca752 Remove minimumIdle config in HikariCP (#557)
* Remove minimumIdle config for HikariCP

* Add comment

* Resolve comment
2020-04-22 19:35:02 -04:00
gbrodman
ca3ae9b0e4 Add SqlEntity and DatastoreEntity interfaces (#562)
* Add SqlEntity and DatastoreEntity interfaces

These will be used when replaying transactions from either the Datastore
commit logs or the SQL Transaction objects.

When Datastore is the primary database, we will read in the
Datastore commit logs, convert each saved entity to however many SQL
entities, then save those SQL entities in SQL.

When SQL is the primary database, we will read in the SQL objects from a
yet-to-be-created SQL table, convert them to however many Datastore
entities, then save those Datastore entities in Datastore.

This PR includes a couple simple examples of how this will work for entities that are
saveable in both SQL and Datastore (the simple case).

* Add 1-1 mapping between entity annotations and interfaces
2020-04-21 17:28:49 -04:00
Shicong Huang
295251ee78 Add JPA annotations to ContactResource and generate schema (#547)
* Add JPA annotations to ContactResource and generate schema

* Resolve comments

* Resolve comments

* Manually add foreign key constraints

* Run with junit5

* Rebase on HEAD

* Fix DomainBaseSqlTest
2020-04-21 15:40:16 -04:00
Michael Muller
7ca0e9387c Persist DomainBase.nsHosts VKeys to SQL (#541)
Persist nsHosts in Cloud SQL

Persist the VKey based nameserver hosts field of DomainBase in Cloud SQL with
foreign key constraints.
2020-04-20 13:03:12 -04:00
Weimin Yu
4f988d42c7 Allow Entity instantiation without AppEngineRule (#559)
* Allow Entity instantiation without AppEngineRule

Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.

* Allow Entity instantiation without AppEngineRule

Defined an extension that sets up a fake AppEngine environment
so that Datastore entities can be instantiated.
2020-04-16 17:03:27 -04:00
Weimin Yu
9b47a6cfee Hack to call setup and teardown in JUnit5 suite (#560)
* Hack to call setup and teardown in JUnit5 suite

JUnit 5 runner does not support @BeforeAll and @AfterAll declared
in the Suite class (as opposed to the member classes). However,
staying with the JUnit 4 suite runner would prevent any member
classes from migrating to JUnit 5.

We use a hack to invoke suite-level set up and teardown from tests.
This change is safe in that if the JUnit 5 runner implementation changes
behavior, we will only see false alarms.
2020-04-16 14:46:08 -04:00
Shicong Huang
9db4d1a082 Add a listener to invoke entity callbacks (#551)
* Add a listener to invoke entity callbacks

* Resolve comments

* Add test
2020-04-16 14:30:43 -04:00
Michael Muller
ec22d4d1a0 Implement VKeyConverter (#538)
* Implement VKeyConverter

Implement a SQL converter that can be used for VKey objects.

Caveats:

- This only works with string columns (there's an excellent chance that all of
  our VKeys will use SQL string columns).
- Using this dpesn't establish a foreign key constraint between the referenced
  type (the "T" in VKey<T>) and the entity itself: this needs to be
  defined manually in the schema.
2020-04-16 09:45:23 -04:00
Weimin Yu
0fcf26def0 Exclude proxy configs from the FOSS jar (#558)
* Exclude proxy configs from the FOSS jar

No sensitve data exposed.

Added a todo to modify the release process and stop
building the foss jar on the merged repo.
2020-04-15 12:21:41 -04:00
gbrodman
3d88ba4e1b Add verification that domain labels aren't multi-level domains (#553)
* Add verification that domain labels aren't multi-level domains

In addition, I did a bit of test refactoring because previously, the
CreateOrUpdateReserveListCommandTestCase test cases weren't actually
testing the proper things -- they were failing with
IllegalArgumentExceptions, but not the right ones.

* Change test name and use IDN library

* Handle numeric labels

String like "0" or "2018" are valid labels but not valid domain names

* Use IDN validation with a dummy TLD
2020-04-15 11:54:40 -04:00
Weimin Yu
580a3b6981 Disable JpaEntityCoverageCheck by default (#555)
* Disable JpaEntityCoverageCheck by default

Only members of SqlIntegrationTestSuite should enable the check,
which incurs per-test overhead.
2020-04-14 12:48:21 -04:00
gbrodman
6990d6058f Allow a --token option when checking a domain (#556)
* Allow a --token option when checking a domain
2020-04-14 10:20:27 -04:00
gbrodman
dfeed63c40 Run automated NPM update (#554) 2020-04-11 11:47:25 -04:00
Lai Jiang
9eac9621cb Add a Test workaround for certain Linux distro (#552)
On Arch Linux, DumpGoldenSchemaCommandTest failed due to the follow
error:

java.lang.RuntimeException: Container.ExecResult(exitCode=1, stdout=, stderr=pg_dump: [archiver] could not open output file "/tmp/pg_dump.out": Is a directory)

However I cannot figure out why this permission error happens, as the
docker command is executed as root. Saving the pg_dump output to a
temporary file and copy it over the mapped file works, so I don't
know...
2020-04-10 12:44:36 -04:00
Weimin Yu
b7efc5dd25 Migrate SqlIntegrationTestSuite members to Junit5 (#550)
* Migrate SqlIntegrationTestSuite members  to Junit5

Made InjectRule and EntityTestCase work with both JUnit4 and 5.

Note that "@RunWith(JUnit4.class)" is no longer needed on
JUnit4 test classes. Therefore, its removal from EntityTestCase
has no impact on child classes. All of them are still included in
tests.

Migrated remaining member classes in SqlIntegrationTestSuite to JUnit5.
2020-04-09 12:54:16 -04:00
Weimin Yu
1911c11623 Add Test suite support for JUnit 5 classes (#549)
* Add Test suite support for JUnit 5 classes

Added Gradle dependencies and updated lockfiles.

Updated SqlInegrationTestSuite to use new annotations.

Migrated one member class in SqlIntegrationTestSuite (CursorDaoTest)
to JUnit 5, and verified that the new Suite runner can handle a
mixture of JUnit 4 and 5 tests in one suite.

Note that Gradle tests that run TestSuites must choose JUnit 4.
Updated core/build.gradle and integration/build.gradle.
2020-04-07 21:06:49 -04:00
Weimin Yu
b8df0bac24 Make AppEngineRule work with JUnit 5 (#548)
* Make AppEngineRule work with JUnit 5

Made AppEngineRule work with both JUnit4 and JUnit 5 and applied
it to PremiumListTest.

Next step is to convert SqlIntegrationTestSuite.java to JUnit5.
2020-04-07 14:59:25 -04:00
Lai Jiang
0561c7754e Upgrade to Gradle 6.3 (#546) 2020-04-06 22:14:14 -04:00
Weimin Yu
904f16c8b5 Actually run JUnit 5 tests (#545)
* Actually run JUnit 5 tests

In Gradle, JUnit 5 must be explicitly enabled with a call to
test.useJUnitPlatform().

The FilteringTest used in :core must also enable JUnit5 separately.

Fixed AppEngineRule to work with the few tests that have migrated
to JUnit5.

More work is needed with AppEngine before we can migrate tests
that actually use Cloud SQL.

For context, with @EnableRuleMigrationSupport, JUnit 5 runner calls
an external resource's before() and after() methods. TestRule.apply()
is not called, therefore any setup done their will be bypassed with
JUnit 5.
2020-04-06 13:26:38 -04:00
Weimin Yu
3a7d71e411 Upgrade CompareDbBackup for Datastore V3 (#543)
* Upgrade CompareDbBackup for Datastore V3

Upgrade the CompareDbBackup class to work with latest
Datastore backup directory structure.

Also fixed a few unrelated minor issues:
- Remaining cases of improper use of System.setOut
- Wrong import order in one class
2020-04-06 10:50:38 -04:00
Shicong Huang
1ded33ecea Resolve warnings in the Hibernate log (#542) 2020-04-03 18:04:55 -04:00
Shicong Huang
bf4659f11c Auto-apply JPA converters for map type (#520)
* Add map converter

* Delete old map usertype

* Refactor bind

* Change to use map entry

* Use Map.Entry
2020-04-03 10:47:42 -04:00
Shicong Huang
bac1998d6a Auto-apply JPA converters for map type (#520)
* Add map converter

* Delete old map usertype

* Refactor bind

* Change to use map entry

* Use Map.Entry
2020-04-02 16:43:08 -04:00
gbrodman
4a34369ba9 Submit a task to relock domains if desired upon verification (#529)
* Submit a task to relock domains if desired upon verification

* Merge remote-tracking branch 'origin/master' into verifyRelock

* Respond to CR
2020-04-02 15:18:29 -04:00
Shicong Huang
db7d49801d Supress exccesive logging message from Cloud SQL (#540)
* Supress exccesive logging message from Cloud SQL

* Upgrade package versions that were downgraded before
2020-03-31 17:57:18 -04:00
sarahcaseybot
0c52d209e5 Fix IllegalArgumentException (#536)
* Fix IllegalArgumentException

* Add more information about global locks

* Add null checks
2020-03-30 11:36:09 -04:00
gbrodman
73b98d298b Add a set of radio buttons for relock duration (#535)
* Add a set of radio buttons for relock duration
2020-03-30 11:06:32 -04:00
Michael Muller
7880aab386 Fix optional access for VKey nested keys (#539)
* Fix optional access for VKey nested keys

We should have used ofNullable() instead of of() for key creation.  Also add a
unit test.
2020-03-30 11:01:25 -04:00
Michael Muller
fa9134328a Improve error information in coverage test. (#537)
* Improve error information in coverage test.

If the golden schema isn't up-to-date with the persistence model, the coverage
tests fail with an exception chain that ends in a PSQLException 'relation
"TableName" does not exist' which is kind of misleading when the problem is
that your golden schema isn't up-to-date.

Check for this error in the coverage tests and generate a more informative
error message indicating a likely root cause.
2020-03-27 14:58:01 -04:00
Michael Muller
5b452bf074 Key to VKey conversion for Nameserver (#476)
* Key to VKey conversion for Nameserver

This change illustrates the conversion of a single key in the system
(Key<HostResource> as used in the "nameservers" field of DomainBase) to a
VKey.

It currently builds, but had some curious (possibly unrelated?) test failures
that I have not fully investigated.

* Latest round of changes, all tests pass.

* Changes requested in review.

* Fix problems with null check in VKey accessors

Add maybeGet versions of getSqlKey() and getOfyKey() that return Optional
objects and make the nameserver management routines use those instead.
2020-03-26 17:13:30 -04:00
Weimin Yu
f749236500 Reuse JPA EntityManagerFactory in tests (#533)
* Reuse JPA EntityManagerFactory in tests

Reuse EntityManagerFactory instance in tests if the requested schema
stays the same. Only truncate tables and reset sequences when reusing.

Note that the jdbc driver needs to be informed to expect out-of-band
schema changes.
2020-03-26 16:51:47 -04:00
gbrodman
e7825fae66 Don't rely on the password field's existence for admins (#534)
* Don't rely on the password field's existence for admins

We don't have the field when it's an admin user that's logged in. A
nicer language would have caught this unfortunately.
2020-03-26 15:55:54 -04:00
gbrodman
91155d6c67 Fix up lock modal wording (#532)
* Fix up lock modal wording

When locking a domain, the "domain" variable is null so we shouldn't
display it.
2020-03-26 15:52:11 -04:00
gbrodman
2ff1026cfd Handle null GAE user IDs gracefully (for non-admins) (#531)
Unfortunately in our testing environments, we're all admins so it's easy
to miss things like this.
2020-03-26 14:29:05 -04:00
Weimin Yu
f1c46b8030 Drop postgresql schema instead of database in Sql tests (#530)
* Drop schema instead of database in Sql tests

Speed up the database cleanup between tests by dropping the schema
instead of the database. The new approach is much faster.

Ad hoc measurement on my desktop shows that :core:sqlIntegrationTest
improves from 73 seconds to 48 seconds, and :core:standardTest
improves from 12m40 to 7m40.
2020-03-25 21:03:58 -04:00
gbrodman
d663bf4db5 Add CSS spinner while loading locks content (#527)
* Add CSS spinner while loading locks content
2020-03-24 15:33:17 -04:00
gbrodman
acf0baf048 Fix semantic merge conflict (#528) 2020-03-24 12:29:11 -04:00
gbrodman
2998b56982 Add min length to password fields (#524)
* Add min length to password fields
2020-03-24 11:16:05 -04:00
gbrodman
7b602300d8 Use the relock duration if provided in RLPA (#519)
* Use the relock duration if provided in RLPA
2020-03-24 10:33:43 -04:00
Ben McIlwain
fe760d7066 Allow backwards compatibility with JUnit 4 @Rules in JUnit 5 (#526)
* Allow backwards compatibility with JUnit 4 @Rules in JUnit 5

This allows us to defer having to re-implement all of our JUnit 4 Rules as JUnit
5 extensions for now, while continuing to in-place upgrade all existing JUnit 4
test classes to JUnit 5.

As proof of concept, this upgrades PremiumListUtils (which uses AppEngineRule,
our largest and most complicated @Rule) to use the JUnit 5 test runner.

* Apply formatter to entire file
2020-03-23 14:45:54 -04:00
gbrodman
ad06f265a5 Flat-map registry lock emails to avoid unclean errors in bad situations (#525)
* Flat map to avoid unclean errors in bad situations

Also properly reflect that for admins, we will use their user email

* Make MS's GAE user ID a public static field
2020-03-23 11:45:49 -04:00
Shicong Huang
fa9400ebc5 Set postgres package back to runtime dependency (#522) 2020-03-20 15:43:30 -04:00
gbrodman
519a85af85 Add a registryLockEmailAddress field to RegistrarConctact objects (#523)
* Add a registryLockEmailAddress field to RegistrarConctact objects

Because we need to manage the login email, it should be on an account
that we manage. However, for registry lock, we would want to send the
verification emails to a separate email address that the user can use.

As a result, we will use a second field for a user-accessible registry
lock email address. This must be set on the contact when enabling
registry lock for this contact.

* Responses to CR

* derp
2020-03-20 14:12:00 -04:00
sarahcaseybot
b2df127dc4 Add lock dual read (#517)
* Add lock dual read

* small changes
2020-03-20 14:11:00 -04:00
gbrodman
b21042bda9 Fix the test server (#521)
* Fix the test server

This rule isn't necessary any more since we merged the SQL-starting rule
into the AppEngineRule logic. Furthermore, it actually causes the test
server to crash because we try to drop-and-create the DB twice, the
second time while the first instance is still connected.
2020-03-19 11:05:51 -04:00
Lai Jiang
36378f6b10 Upgrade to Gradle 6.2.2 (#518) 2020-03-18 21:38:37 -04:00
Shicong Huang
d01f1f7604 Make jpaTm for nomulus tool use local credential (#515)
* Make jpaTm for nomulus tool use local credential

* Remove unused methods in RegistryToolEnvironment

* Fix order of annotations

* Remove unused method in PersistenceComponent

* Move the creation of credential to the module

* Move creadential creation to AuthModule

* Add a TODO
2020-03-17 20:16:42 -04:00
gbrodman
e9610636e4 Add a relockDuration to the RegistryLock SQL object (#514)
* Add a relockDuration to the RegistryLock SQL object

This is the length of time after an unlock that we will re-lock the
domain in question.

* Sort by domain name for stability

Note: this is likely not the best solution for the UI but we can iterate
on this.

* Add nullable

* Add a converter for Duration
2020-03-16 17:44:25 -04:00
gbrodman
d09fc7ee05 Match logged-in GAE user ID with registrar POC user ID (#511)
* Match logged-in GAE user ID with registrar POC user ID

The reasoning for this is thus:
We wish to have the users log in using Google-managed addresses--this is
so that we can manage enforcement of things like 2FA, as well as generic
account management. However, we wish for the registry-lock confirmation
emails to go to their standard non-Google email addresses--e.g.
johndoe@theregistrar.com, rather than johndoe@registry.google.

As a result, for registry lock, we will enable it on
the johndoe@registry.google account, but we will alter the email address
of the corresponding Registrar POC account to contain
johndoe@theregistrar.com. By doing this, the user will still be logging
in using the @registry.google account but we'll match to their actual
contact email.

* fix up comments and messages

* Error if >1 matching contact

* include email addresses

* set default optional

* fix tests
2020-03-16 11:38:05 -04:00
Shicong Huang
0545375eba Change cloud sql SDK to compile level dependency (#516) 2020-03-16 10:24:19 -04:00
Michael Muller
8a045aedd0 Disambiguate naming of VKey.create() overloads (#513)
* Disambiguate naming of VKey.create() overloads

It was discovered in the course of trying to convert the larger codebase
to VKey.create() calls that method overloading isn't a very effective
discriminator in cases where "Object" is one of the distinguishing
argument types:-)

Convert the two specialized create() methods to createOfy() and
createSql() so that (at least in the former case) we'll get a
compile-time error if we aim to create a VKey for an Ofy key from an
object of the incorrect type.
2020-03-12 16:13:12 -04:00
gbrodman
560bec1e83 Add a RelockDomainAction for future auto-relocks (#485)
* Add a RelockAction and reference to relocks in RegistryLocks

* Respond to CR

- refactor the request param exception logging a bit
- don't log an error if the domain was already locked, just skip

* Save a relock for all locks (if possible)

* derp

* Long -> long + remove unnecessary transact

* semantic merge conflict woo

* fix another semantic merge conflict
2020-03-12 16:02:27 -04:00
Lai Jiang
3e7ea75b6f Use cs.opensource.google for code search (#512)
* Use cs.opensource.google for code search

* Change logo size

* Make texts in the table center-aligned
2020-03-12 14:06:04 -04:00
Weimin Yu
6ed7e00b00 Update SqlIntegrationTestSuite (#510)
* Update SqlIntegrationTestSuite

Edited Javadoc to emphasize that suite members should be DAO tests.

Removed functional tests from the suite. They do not benefit much
from running against different schemas when the entities they use
are already covered by DAO tests.

Added DomainBaseSqlTest to the suite, which tests DomainBase.
2020-03-11 14:11:53 -04:00
Michael Muller
6e1231233e Create a nom_build wrapper script (#508)
* Create a nom_build wrapper script

nom_build is a wrapper around ./gradlew.  It's purpose is to help us
deal with properties.  The main problem that it is trying to solve is
that when properties are specified using -P, we don't get an error if
the property we specify isn't correct.  As a result, a user or a build
agent can launch a build with unintended parameters.

nom_build consolidates all of the properties that we define into a
python script where the properties are translated to flags (actual
gradlew flags are also proxied).  It also generates the property file
and warns the user if the current properties file is out of sync with
the script and includes documentation on each of the properties.
2020-03-10 16:32:14 -04:00
Shicong Huang
3098048fdb Enable Cloud SQL when Datastore is enabled for unit test (#502)
* Enable Cloud SQL when Datastore is enabled for unit test

* Add explanation for why add a ETA field in GenerateEscrowDepositCommand

* Fix line length

* Ignore membershipt test but bring back test suite

* Fix tiny issue
2020-03-10 12:26:25 -04:00
gbrodman
f2846fc914 Gray out the password field for admins (#506)
* Gray out the password field for admins

We don't check it for admins since it's not necessary, so ignore it

* Remove the field entirely
2020-03-10 11:30:20 -04:00
gbrodman
499237ac57 Listen to the user hitting enter in the lock/unlock modal input fields (#505)
* Listen to the user hitting enter in the lock/unlock modal input fields

Listen to both, just in case one or the other is disabled

* Don't require that the element exist
2020-03-10 11:22:57 -04:00
Weimin Yu
6bd50421bc Fix broken builds when Maven Central is used (#509)
* Fix broken builds when Maven Central is used

Gradle 6.2.1 apparently introduces a behavior change wrt boolean
expression: empty string used to eval to false, but now evals to
true.

Pre Gradle 6.2.1, root project's Gradle properties apparently were
not set to buildSrc. Now they are passed on to buildSrc -- mavenUrl
in buildSrc changes from null to "".

Both changes break the project when mavenUrl and/or pluginsUrl are
not set on command line.

Also added junit.jupiter-api as testCompile dependencies to projects.
This is a directly used dependency, whose absence causes a Lint
warning.
2020-03-10 11:21:03 -04:00
sarahcaseybot
dbdd2b4491 Add Lock dual write (#496)
* Add Lock dual write

* wrap calls in DB transaction
2020-03-09 11:13:46 -04:00
gbrodman
f83f8f92a3 Show locks in the case where you have an expired unlock request (#507)
* Show locks in the case where you have an expired unlock request
2020-03-06 22:00:42 -05:00
gbrodman
28d3af0ee9 Change the wording on the lock-not-enabled page (#504)
* Change the wording on the lock-not-enabled page

* fix the screenshot
2020-03-06 16:15:11 -05:00
Lai Jiang
08a6a333ad Upgrade to Gradle 6.2.1 (#501) 2020-03-05 18:47:25 -05:00
Shicong Huang
adafab60c4 Add common CRUD operations to TransactionManager (#487)
* Add BasicDao

* Refactor RegistrarDao to extend BasicDao

* Introduce VKey and rewrite BasicDao

* Move CRUD methods to TransactionManager

* Refactor code to simplify the way to get id from entity and sqlKey

* Assert in transaction

* Fix broken test

* Change methods name
2020-03-05 14:03:03 -05:00
gbrodman
fec806ef8b Handle both ways the UI passes in booleans (#500)
* Handle both ways the UI passes in booleans

* fix the JS test

* Quote the value
2020-03-02 12:49:09 -05:00
gbrodman
e8ff4081a9 Show pending locks in the locked-domains table (#495)
* Show pending locks in the locked-domains table

* asdf

* fix the tests

* including pending unlocks in the table

* fix the screenshot test
2020-02-28 13:57:33 -05:00
Ben McIlwain
9283cd263f Always validate domain name on allocation token (#498)
* Always validate domain name on allocation token

This is in response to a client-reported error, where they accidentally sent the
wrong domain name on a domain create that included an allocation token. What
should have happened (and that now happens as of this commit) is an error being
thrown that the allocation token does not match the domain name being created.
What happened instead was that, since the incorrectly submitted domain name was
not reserved, the create succeeded (as it would for all creates of unreserved
domains in GA) and the allocation token was redeemed, which is not what you'd
expect.

* Fix tests to reflect changed check behavior
2020-02-27 16:48:37 -05:00
gbrodman
429bc8e6d2 Set focus correctly in lock modal (#499) 2020-02-27 13:20:10 -05:00
gbrodman
fc0a9160b2 Create methods to administratively (un)lock domains (#494)
* Refactor DomainLockUtils methods to take a time rather than a clock

* Add administratively (un)lock methods

* Responses to CR

- Javadoc changes
- Method renames
- Variable renames

* Refactor lock methods to use JPA transaction time

* Remove clock, use Datastore transaction time

* Properly use Datastore transaction time, batched

* Continue to throw exceptions on invalid domains

* DAO writes should be in a transaction

* Assume in-transaction for all RLDao methods

* clean up test

* Fix more tests

* add comment
2020-02-26 17:11:16 -05:00
Ben McIlwain
3cd0b4d5e5 Upgrade Gradle license report plugin to latest version (1.13) (#492)
* Upgrade Gradle license report plugin to latest version (1.13)
2020-02-25 15:04:56 -05:00
sarahcaseybot
cbe216ced2 Add logging to CursorDao.saveCursor (#486)
* Add log messages

* small fix

* add null check

* change to checkArgumentNotNull
2020-02-21 15:49:45 -05:00
sarahcaseybot
1be0ad885c Add daily cron job for IcannReportingUploadAction (#385)
* Add daily cron job for IcannReportingUploadAction

This job checks each ICANN cursor's cursorTime. If all cursorTime values are in the future, the action exits and does nothing. For each cursor that has a cursorTime in the past, the corresponding report is uploaded if it is staged, or logs an error message if the report has not been staged yet.

* Change cron job description
2020-02-21 12:22:17 -05:00
gbrodman
8b819a27e7 Add Registry Lock UI (#369)
* Add Registry Lock UI

* Responses to CRs, mostly

TODO:
- Figure out wording for the 'not enabled yet' message
- Include the server status change cost in the email, or in the UI?
- Should we show non-completed lock requests in the UI?

* Fix get action test

* Change the not-allowed-for-registrar msg to include support email

* Change the wording on the price

* Move TLD input into the modal, and other changes

- don't log the password
- test to make sure the password shows bullets

* Responses to CR and cleanup

* Format closer to something proper
2020-02-20 18:04:32 -05:00
Ben McIlwain
ec6157b6ac Use -t instead of main params for TLDs in nomulus count_domains command (#493)
* Use -t instead of main params for TLDs in nomulus count_domains command

This makes the command consistent with list_domains. I use both frequently and it
was annoying forgetting which one takes -t and which uses main parameters. Now
they both work the same way.
2020-02-20 16:23:38 -05:00
gbrodman
05ed4fd849 Check for an empty/null allocation token (#489)
* Check for an empty/null allocation token

In addition, we should be returning an authorization exception for an
invalid token, not a parameter syntax exception. See https://tools.ietf.org/html/draft-ietf-regext-allocation-token-04#section-2.1

* Add comment
2020-02-20 15:50:10 -05:00
gbrodman
ecf1721755 Add a scrap command to backfill registry locks (#478)
* Add a scrap command to backfill registry locks

* fix tests

* Change comments and messages

* Use URS time (best effort) if one exists

* Don't bother with root cause
2020-02-20 15:07:39 -05:00
Ben McIlwain
7bb69e50c5 Lazily instantiate jpaTm (#491)
* Lazily load jpaTm

This fixes an exception we were seeing in production where commands that only
implemented CommandWithRemoteApi (and that had nothing to do with Cloud SQL)
were nevertheless trying to initialize a JPA TM instance, and then failing.
That stacktrace looked like:

com.google.api.client.http.HttpResponseException: 400 Bad Request
{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:193)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:165)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:151)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:423)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:542)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:475)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:592)
at google.registry.keyring.kms.KmsConnectionImpl.attemptDecrypt(KmsConnectionImpl.java:163)
at google.registry.keyring.kms.KmsConnectionImpl.lambda$decrypt$0(KmsConnectionImpl.java:148)
at google.registry.util.Retrier.callWithRetry(Retrier.java:153)
at google.registry.util.Retrier.callWithRetry(Retrier.java:130)
at google.registry.util.Retrier.callWithRetry(Retrier.java:95)
at google.registry.keyring.kms.KmsConnectionImpl.decrypt(KmsConnectionImpl.java:147)
at google.registry.keyring.kms.KmsKeyring.getDecryptedData(KmsKeyring.java:209)
at google.registry.keyring.kms.KmsKeyring.getString(KmsKeyring.java:178)
at google.registry.keyring.kms.KmsKeyring.getToolsCloudSqlPassword(KmsKeyring.java:100)
at google.registry.persistence.PersistenceModule.providesNomulusToolJpaTm(PersistenceModule.java:124)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.proxyProvidesNomulusToolJpaTm(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:61)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:39)
at google.registry.persistence.PersistenceModule_ProvidesNomulusToolJpaTmFactory.get(PersistenceModule_ProvidesNomulusToolJpaTmFactory.java:12)
at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
at google.registry.persistence.DaggerPersistenceComponent.nomulusToolJpaTransactionManager(DaggerPersistenceComponent.java:168)
at google.registry.persistence.transaction.TransactionManagerFactory.createJpaTransactionManager(TransactionManagerFactory.java:38)
at google.registry.persistence.transaction.TransactionManagerFactory.<clinit>(TransactionManagerFactory.java:29)
at google.registry.model.registry.Registries.lambda$createFreshCache$2(Registries.java:60)
at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:243)
at google.registry.model.registry.Registries.getTlds(Registries.java:85)
at google.registry.model.registry.Registries.assertTldsExist(Registries.java:112)
at google.registry.tools.CountDomainsCommand.run(CountDomainsCommand.java:41)
at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:243)
at google.registry.tools.RegistryCli.run(RegistryCli.java:168)
at google.registry.tools.RegistryTool.main(RegistryTool.java:127)

The TL;DR is that RegistryCli was over-eagerly creating the jpaTm, because
there's no reason the Registries cache (which is Datastore-only) should ever
need it, but because this cache is using the Datastore transaction manager, the
other one was being created too.
2020-02-20 12:45:06 -05:00
Ben McIlwain
bbacdb9704 Start using JUnit 5 (#488)
* Start using JUnit 5

This converts a single test class over to JUnit 5 (YamlUtilsTest). The main
differences you'll notice are that @RunWith isn't needed anymore, test classes
and test methods can now be package-private, and the @Test annotation comes from
the org.junit.jupiter.api package instead of org.junit. There's a lot more
differences between 4 and 5 than this that we'll need to keep in mind when
converting more test classes; for some more details, see:
https://www.baeldung.com/junit-5-migration

In order to allow JUnit 4 and 5 test classes to coexist, I've had to add two new
dependencies, org.junit.jupiter:junit-jupiter-engine and
org.junit.vintage:junit-vintage-engine, which exist in addition to junit:junit
for now. Eventually, once we've completed migrating over all JUnit 4 test
classes, then we can remove junit and junit-vintage-engine and just be left with
junit-jupiter-engine.

* Delete no longer needed lockfiles

* Merge branch 'master' into first-junit5
2020-02-19 18:29:59 -05:00
sarahcaseybot
f53aa8d55e Add dual read for cursors (#473)
* Add dual read for cursors

* Fix loadAndCompareAll to batch load cursors

* fix javadocs
2020-02-19 16:10:19 -05:00
Shicong Huang
be395611ca Move all JPA converters to a dedicated package (#480)
* Move all JPA converters to a dedicated package

* Move tests as well
2020-02-19 13:54:27 -05:00
Shicong Huang
b9c40648d0 Add dual write for Registrar (#474)
* Add dual write for Registrar

* Use @AlsoLoad to set street field for Cloud SQL

* Change email body to use the new streetLine field

* Refactored the logic to handle street fields

* Simplify postLoad function

* Address comments

* Add a TODO to remove street

* Add test for onLoad and postLoad

* Rebase on master
2020-02-13 16:00:21 -05:00
sarahcaseybot
22a879e655 Add the schema and DAO for Locks in CloudSQL (#462)
* Add the schema and DAO for Locks in CloudSQL

* Addresses some comments

* Change number on flyway file

* Small changes

* More small changes

* Use checkArgumentNotNull instead of checkNotNull

* Address comments

* fix javadocs

* update persistence
2020-02-13 10:22:10 -05:00
Weimin Yu
62433c2238 Use JSON API for Maven Repo on GCS (#483)
* Use JSON API for Maven Repo on GCS

The url pattern https://storage.googleapis.com/{Bucket}/{Path}
uses the legacy XML API, which seems to be less robust than
the JSON API. We have observed connection resets after a few
thousand-file download bursts over 30 minutes.

This PR changes all urls to registry's Maven repo on GCS to
gcs://{Bucket}/{Path}. Gradle uses the JSON API for such urls.

TESTED=In Cloud Build with local change
2020-02-12 14:03:50 -05:00
Shicong Huang
90945bcc30 Add a test to verify persistence.xml (#482) 2020-02-12 12:39:03 -05:00
Weimin Yu
f134c4bf37 Use dependency cache in all Gradle tasks in GCB (#481)
* Use dependency cache in all Gradle tasks in GCB

Make the initial test and the final publishing steps use the shared
dependency cache.

Also make the initial test use the registry's own maven repo instead
of Maven Central.
2020-02-11 14:50:22 -05:00
gbrodman
44921c29d6 Clean up RegistryLockPostAction (#470)
* Clean up RegistryLockPostAction

* pocId -> userEmail when appropriate

* Merge remote-tracking branch 'origin/master' into lockPostImprovements

* Remove pocId
2020-02-11 08:43:44 -07:00
Weimin Yu
ce80278ab7 Make Gradle dependency cache shareable in GCB (#479)
* Make Gradle dependency cache shareable in GCB

Make Gradle put its caches in the source tree so that
they can be preserved across steps. When left at their
default location, caches are lost after each step.
2020-02-10 11:20:11 -05:00
Shicong Huang
594ce30122 Auto-apply JPA converters for collection type (#469)
* Auto-apply JPA converters for collection type

* Extract common logic to a base class

* Remove extra lines

* Rebase on master
2020-02-10 10:33:43 -05:00
Weimin Yu
736f788eea Use CA cert in tests (#475)
* Use CA cert in tests

Our tests were using a self-signed cert created by a netty utility.
That cert is not CA, and cannot be used with newer JDK versions
(starting from u242 with Ubuntu openjdk).

Now we generate a unsafe cert for tests. The new cert,
SelfSignedCaCertificate, needs to be made test-only.
To do so we need to refactor both prober and proxy.
2020-02-07 10:23:24 -05:00
Shicong Huang
d6f49f5c08 Add JPA annotations to RegistrarContact (#432) 2020-02-06 16:58:15 -05:00
Ben McIlwain
8b9139bc4c Always attempt to save the claims list to Cloud SQL (#472)
* Always attempt to save the claims list to Cloud SQL
2020-02-06 12:21:34 -05:00
Ben McIlwain
b148102716 Always save reserved lists to Cloud SQL (#471)
* Always save reserved lists to Cloud SQL

Otherwise it's too easy to forget to run this additional option, and then things will
fall out of sync and start throwing errors.
2020-02-06 12:21:26 -05:00
Shicong Huang
9ea06ebf35 Use JPA @Enumerated to convert enum to string (#466) 2020-02-04 10:50:03 -05:00
gbrodman
c0afb9aeee Use randomly-generated base-58 strings for RegistryLock verification codes (#464)
* Use randomly-generated strings for RegistryLock verification codes

We were using UUIDs before which are also fine, but unnecessarily long.
The RegistryLock class itself does not enforce any particular format for
the lock verification codes.
2020-02-03 13:50:54 -05:00
Michael Muller
76d8afe856 Implement dump_golden_schema command in devtool (#467)
* Implement dump_golden_schema command in devtool

Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.

Refactor the SQL container functionality from GenerateSqlCommand.  There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.

* Reformatted and changes in response to review

* Fix getDockerTag() usage

* Fix "leaked resource"
2020-02-03 13:25:27 -05:00
gbrodman
36787c37fb Add a RegistryLockPostAction and tests (#459)
* Add a RegistryLockPostAction and tests

* Response to CR and more robust email content test

* Fix typo

* Run save + email in a JPA transaction

* Use action in subject
2020-02-03 11:34:01 -05:00
Weimin Yu
36e46097e3 Allow non-CA cert as trust anchor in tests (#468)
* Allow non-CA cert as trust anchor in tests

Stay with the legacy behavior to unblock release build.
We will update the test cert generation code later.

TESTED=reproduced and fixed test failures in builder container
2020-02-03 11:02:52 -05:00
Weimin Yu
ce2f98f680 Work around Spinnaker issue wrt variables (#465)
* Work around Spinnaker issue wrt variables

Cloud Build variable reference need to stay from the  ${var} pattern
to prevent Spinnaker from trying to resolve it. In all files that
are used by Spinnaker, we change variable reference to the $var form.

We made the minimum amount of change possible, and will review this
issue after the permanent solution is available.
2020-01-30 13:28:36 -05:00
gbrodman
daaf231d39 Add RegistryLockVerifyAction (#461)
* Add RegistryLockVerifyAction

The action takes two parameters
- isLock is a boolean, determining whether we're trying to lock or
unlock a domain
- lockVerificationCode is the UUID by which we'll look up the lock
object in question.

The lock in question must not be expired and must be in a valid lockable
/ unlockable state

* Some responses to CR

* Add slash and move test method

* Add more data and tests

* Fix screenshot
2020-01-29 16:36:39 -05:00
Shicong Huang
955c3d9aeb Fix money conversion for JPY in PremiumListDao (#463) 2020-01-29 13:10:58 -05:00
Shicong Huang
d03cea2443 Add JPA annotations to class Registrar (#430)
* Add JPA annotations to class Registrar

* Use array for Java list

* Exclude parent field

* Use 3 columns for address and use text for enum

* Use EnumParameter and 3 properties in Address

* Rename columns and rebase on HEAD
2020-01-29 11:06:23 -05:00
Michael Muller
e386bf5bd8 Get persistence of DomainBase actually working (#438)
* Get persistence of DomainBase actually working

Fix all of the existing problems with DomainBase persistence:
- Remove "final" keywords on getters that cause errors during startup.
- Remove Transient from creationTime (since there's a converter for
  CreateAutoTimestamp)
- Fix DesignatedContext persistence so that it only creates a single table.
  This is a lot more efficient given that these are many-to-one with their
  domains.
- Add a flyway script, update the golden schema.
- Create a unit test, add it to the integration test suite.

* Changes request in review

* Regenerated generated schema file.

* Changes for review

* Persist status value enum set

* Changes in response to review

* Changes requested in review

* Fixes for #456

* Rename Domain "status" column to "statuses"
2020-01-28 11:32:26 -05:00
sarahcaseybot
1912453f4e Fix IcannReportingUploadAction to use correct year in January (#452)
* Fix IcannReportingUploadAction to use correct year in January

* small fixes
2020-01-27 17:38:59 -05:00
Shicong Huang
43682aa7f4 Enable Cloud SQL for nomulus command implementing CommandWithRemoteApi (#460) 2020-01-27 10:28:38 -05:00
Lai Jiang
cdcbe1311a Make the builder script error out when a command fails (#457) 2020-01-23 21:41:17 -05:00
Shicong Huang
7c3f685946 Relax active connection check to reduce flakiness (#458) 2020-01-23 18:09:20 -05:00
Weimin Yu
ab0410422c Require tests on all declared JPA entities (#456)
* Require tests on all declared JPA entities

The JPA entity classes declared in persistence.xml reflects
JPA's view of the database. The view must be compatible, though
not always consistent with the actual schema. By defining and
using a new JpaIntegrationWithCoverageRule, this change verifies
that all JPA entities in persistence.xml are included in some
tests in the SqlIntegrationTestSuite.

Also removed one test from SqlIntegrationTestSuite since it
does not use Cloud SQL.

Lastly, global resources such as clock should be provided to
a test fixture. Having one fixture creating its own makes it
harder to use in a multi-system integration test. This change
refactored JpaTransactionManagerRule to take a user provided
clock.
2020-01-23 16:41:27 -05:00
Michael Muller
37315eb743 Add generateSqlSchema as a build task (#454) 2020-01-23 15:07:07 -05:00
Ben McIlwain
50729c9105 Change reserved messages for domains w/ required allocation token (#455)
* Change reserved messages for domains w/ required allocation token

In domain check reasons for unavailable domains, we were previously saying that
domains were "Reserved" regardless of whether the domain was fully blocked or
reserved for an anchor tenant or other specific use (e.g. the .new Limited
Release Program). This commit changes the message for the latter situations to
be "Allocation token required", so that registrars have a hint that domain will
show as available if the correct allocation token is supplied to the domain
check command using the allocation token extension.
2020-01-23 14:48:36 -05:00
Ben McIlwain
2a29ecc2a2 Allow claims keys on domain creates during quiet periods (#453)
* Allow claims keys on domain creates during quiet periods

Since we're using TldState.QUIET_PERIOD for the .new LRP (in which only reserved
domains with allocation tokens may be registered), we also need to support
claims keys during this phase. Otherwise, domains that appear in MarksDB will
not be registrable during LRP, even if the correct claims key is provided.

This is based on an error report from a registrar, in which a correct-looking
domain create was failing with the error message "Declared launch extension
phase does not match the current registry phase".
2020-01-23 12:17:14 -05:00
gbrodman
16b4d15292 Allow checking reserved domains with a valid token (#451)
* Add test for checking reserved domain with token

* Allow domain checks with a valid token if the domain isn't fully blocked

* Check specifically for specific-use or anchor reservations

* Add test for LRP in-time token
2020-01-23 11:45:07 -05:00
gbrodman
7df3d85243 Refactor lock/unlock commands to use Registry Locks (#390)
* Refactor lock/unlock commands to use Registry Locks

* CR responses

* Remove unnecessary XML test files

* Add tests

* Respond to CR

* Refactor further the creation and verification of locks

* remove isUnlocked

* Responses to CR

* Fix tests

* Add admin-override back to unlocking

* Add TODO

* Fix imports
2020-01-23 10:22:56 -05:00
Ben McIlwain
02e7106262 Refactor g.r.model.transaction -> g.r.persistence.transaction (#450)
* Refactor g.r.model.transaction -> g.r.persistence.transaction

The ultimate goal here is to eventually delete the model package entirely once
we're no longer using Datastore. To that end, the Cloud SQL stuff that we'll be
keeping needs to be in a different package. Here's the rough outline of the
final desired state I'm thinking of:

g.r.persistence
  .transaction -- what's being moved in this commit
  .converters -- to be moved in another commit
  [maybe more?]
g.r.schema
  [same-ish subpackages as model currently contains]

I'm also trying to think if it makes sense to move the DAO classes out of
g.r.schema, and haven't come up with a good answer yet.

* Also move test resource files

* Merge branch 'master' into mv-tx-to-persistence
2020-01-22 17:19:29 -05:00
Weimin Yu
f2512bc6cc Check for post-deployment Flyway script changes (#448)
* Check for post-deployment Flyway script changes

Add a script that checks for changes to scripts that
have been deployed to sandbox. This is test in to
be invoked in presubmit and ci.

Extracted common shell functions to an 'rc' file.

Also renamed existing script to be consistent with
other shell scripts.
2020-01-21 15:20:37 -05:00
sarahcaseybot
0b717d40ff Add dual write for Cursors (#414)
* Add dual write for Cursors

* Fix UpdateCursorCommand to dual write multiple cursors

* Small fixes

* Make UpdateCursorsCommand implement CommandWithCloudSql
2020-01-21 15:04:14 -05:00
Shicong Huang
ad2cf933c2 Add dual read for ReservedList (#423)
* Add dual read for ReservedList

* Extract loading cloud sql list to a method
2020-01-17 15:49:58 -05:00
Weimin Yu
3a9e5d398e Remove MapUserTypeTest from Integration TestSuite (#449)
* Remove MapUserTypeTest from Integration TestSuite
2020-01-17 14:14:14 -05:00
Michael Muller
2009540f8b Implement a hibernate type for enums (#446)
* Implement a hibernate type for enums

Leverage GenericCollectionUserType to store enum fields as an array of
strings.

* Changes requested in review

* Remove extraneous import

Curiously, this causes checkstyle to fail now that the import was /removed
from a comment./
2020-01-16 14:56:41 -05:00
sarahcaseybot
ca0d95b066 Fix exDate calculation in domain transfer flows (#389)
* Fix exDate calculation in domain transfer flows

* Link ICANN rule for zero-period transfer

* Get rid of some unnecessary method parameters

* Add domain info commands to test case
2020-01-15 16:43:39 -05:00
Shicong Huang
851465a89d Relax reserved list existence check to allow Cloud SQL migration (#429) 2020-01-15 13:10:50 -05:00
Shicong Huang
6aa7c19344 Add MapUserType to support converstion between Map and hstore (#443) 2020-01-15 10:37:14 -05:00
Lai Jiang
cf0e8e1b26 Use monitoring client v1.0.7 (#445)
This version of the monitoring client no longer depends on backported
JUnit classes.
2020-01-14 23:11:54 -05:00
Weimin Yu
c90d5ed43b Fix flaky SQL integration tests (#441)
* Fix flaky SQL integration tests

As a sanity check, the text fixture verifies that all test jdbc connections
have been closed. However, the query to get the connection count is too
general, and may count background admin tasks such as autovacuum and
replication etc, that may appear at any time.
2020-01-14 10:59:55 -05:00
Weimin Yu
e106e25800 Upgradle JUnit to 4.13 (#442)
* Upgradle JUnit to 4.13

Removed third_party/junit folder and all usage of the
JunitBackPort class. As a result, third_party is no
longer a Gradle subproject.

Minor code changes were needed to work around an
error-prone pattern: multiple statement in assertThrows'
runnable lambda.

Also third_party/activation and third_party/jsch. These
dependencies are loaded from remote maven repo. The local
copies are not in use.
2020-01-14 10:54:09 -05:00
Michael Muller
54471e2714 Fix procedure for schema generation (#436)
* Fix procedure for schema generation
2020-01-13 07:41:20 -05:00
Weimin Yu
bd5299b2f2 Print more info during compatibility test (#440)
* Print more info during compatibility test

Prints a summary of the environments to be tested and their
corresponding version tags.
2020-01-10 15:52:27 -05:00
Shicong Huang
4eae10800d Add user types to support Array in Cloud SQL (#439)
* Add user types to support Array in Cloud SQL

* Remove duplicate null check

* Simplify class structure
2020-01-10 14:54:21 -05:00
Weimin Yu
533515817f Make Jacoco work with multiple test tasks (#437)
* Make Jacoco work with multiple test tasks

By default Jacoco only looks at execution data from
the 'test' task. This is a problem to 'core' which
has multiple test sets.
2020-01-09 16:32:33 -05:00
Weimin Yu
d4b1048363 Upload Cloud Build schema-deploy config to GCS (#435)
* Upload Cloud Build schema-deploy config to GCS

Forgot to upload cloudbuild-schema-deploy.yaml to GCS.
2020-01-09 15:04:10 -05:00
Weimin Yu
2ffcfc4c5e Fix outdated comment (#433)
* Fix outdated comment

Removed reference to a testcontainer issues that may no longer
exist.
2020-01-08 16:02:04 -05:00
Weimin Yu
b47e152ca5 Add script to run compatibility tests (#431)
A helper script to run server/schema compatibility tests from
a CI platform. It allows the caller to choose a system under
test (sql or nomulus) and to specify the deployed version
by environment (sandbox or production).
2020-01-08 11:34:09 -05:00
Michael Muller
b523b3835f Implement changes suggested by testcontainer dev (#426)
* Implement changes suggested by testcontainer dev

See https://github.com/google/nomulus/issues/401
Specifically:
- Use getContainerIpAddress() instead of localhost to insulate us from
  off-machine docker usage.
- Remove shutdown hook to close the container, as testcontainers does this for
  us.
2020-01-08 10:41:36 -05:00
Ben McIlwain
eb7d779bc8 Relax premium list existence check to allow Cloud SQL migration (#428)
* Relax premium list existence check to allow Cloud SQL migration

We need to be able to simultaneously update premium lists that already exist in
Datastore and create them in Cloud SQL (because they haven't been migrated over
yet). This temporarily relaxes the existence check for Cloud SQL so that
"updates" will work even when the list doesn't yet exist there.
2020-01-06 18:09:21 -05:00
Lai Jiang
8fb12d21a2 Use the correct certificate provider type (#427)
TESTED: tested in alpha. Previous I only  tested locally and missed this
typo.
2020-01-06 16:19:50 -05:00
sarahcaseybot
5d9bf64b96 Fix IcannReportingUploadAction to upload reports from the previous month (#425)
* Fix IcannReportingUploadAction to upload reports from the previous month

getFileName now sets the file name of the report to upload to use the month before cursor time.

IcannReportingUploadAction no longer uploads the MANIFEST.txt file since it is not required based on (https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-07#page-9) and the previous implementation of this action did not upload it.

Deletes the ICANN_UPLOAD_MANIFEST cursor since it is no loner needed.

* Add the ICANN_UPLOAD_MANIFEST cursor back
2020-01-06 15:59:01 -05:00
Lai Jiang
76603812e3 Create kzip file for Kythe cross-referencing (#424)
This is set up per b/141716384.

TESTED=Tested on alpha and successfully uploaded the merged kzip file to
GCS.
2020-01-03 16:57:39 -05:00
Shicong Huang
a28754dc5d Add dual read claims list (#413)
* Add dual read claims list

* Improve warning log and use longer duration for cache

* Extract the comparison logic to a method

* Move cache to DAO
2020-01-03 10:59:34 -05:00
Weimin Yu
d6bf6f375e Fix flakiness in JodaMoneyConverterTest (#421)
* Fix flakiness in JodaMoneyConverterTest

JpdaMoneyConverterTest relies on Hibernate to deploy its schema.
This introduces an extra jdbc connection in the middle of a test
suite, and may break the connection count checks between tests made
by JpaTransactionManagerRule.

This change updated HibernateSchemaExporter to include Hibernate
proprietary mappings in META-INF/orm.xml.

This change also disabled Hibernate schema push for all tests,
and enabled sql statement logging.
2020-01-02 11:51:20 -05:00
Ben McIlwain
8f67c4b9cf Support premium list updating in Cloud SQL (#422)
* Support premium list updating in Cloud SQL

This also removes the requirement to specify --also_cloud_sql in nomulus premium
list tooling, instead always persisting to Cloud SQL. It removes a non-USD
premium label in the global test premium list (the Cloud SQL schema doesn't
support a mix of currency units in a single premium list). And it adds a method
to PremiumListDao to grab the most recent version of a given list.

* Merge branch 'master' into premium-lists-always-cloud-sql

* Revert test change

* Create new PremiumListUtils class and refactor out existing method

* Fix tests and update an existing premium price
2020-01-02 11:30:58 -05:00
gbrodman
c17a5c489c Add unlock fields to RegistryLocks (#408)
* Add unlock fields to RegistryLocks

This will make it easier to reason around inter-connected registry lock
objects (like when we add dependent roids). It will make it easier to
answer the question of "Have all locks associated with this host/contact
roid been unlocked?", as well as the question of "Was the last lock
object associated with this domain unlocked?"

* Responses to CR

* Make the DAO API more specific

* whoops, undo rename
2019-12-30 14:34:06 -07:00
Lai Jiang
dd0e3b7c24 Consolidate certificate supplier module (#410)
* Consolidate certificate supplier module

Both the proxy and the proxy needs certificate suppliers. The PR
consolidates the module that providings those bindings to a shared
module and switched the proxy to use that module. The prober currently
uses P12 file to store its certificates. I am debating keeping that
supplier ro converting them to PEM files for simplicity.

* Rename mode enum values to be more descriptive

* Update annotation names to be more descriptive
2019-12-23 13:09:47 -05:00
Weimin Yu
24d671c070 Separate JPA rules for unit and integration tests (#420)
* Separate JPA rules for unit and integration tests

Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.

* Separate JPA rules for unit and integration tests

Define two subclasses of JpaTransactionManagerRule, one for unit
tests and the other for integration tests. The difference is that
the former does not need nomulus schema and need not be included
in server/schema compatibility tests.
2019-12-19 14:49:54 -05:00
Shicong Huang
f76030f7f0 Add initial support to write/update reserved lists to Cloud SQL (#388)
* Add initial support to write reserved list to Cloud SQL

* Add support to update reserved list in Cloud SQL

* Fix wrong check when override is enabled in create command

* Add sql dependent tests to the suite

* Address comment

* Make invocation of super.execute more readable

* Add test to check upper case and non-puny code label

* Move ReservedListDao related classes to schema package

* Simplify a function name
2019-12-19 12:51:48 -05:00
Shicong Huang
f301314d97 Move ClaimsListDao related classes to new package (#416) 2019-12-19 10:50:41 -05:00
gbrodman
25d43511f6 Add BCC capabilities to the Spec11 reports (#418)
* Add BCC capabilities to the Spec11 reports
2019-12-18 17:08:06 -05:00
Lai Jiang
bfd61ef867 Fix another bug in the proxy (#419)
The promise should be set outside the try block because if we want
warning only, we still want the promise to be set even if the
clientCertificate.checkValidity() throws an error.
2019-12-18 16:24:23 -05:00
Weimin Yu
ffe3eb1548 Remove the schema jar url override for tests (#415)
* Remove the schema jar url override for tests

Revert pull request 374: "Allow schema-loading from arbitrary url in tests".

Pull request 403 (f48e3933f5) is a more
general solution.

This reverts commit 68887d427f.
2019-12-17 11:24:20 -05:00
Weimin Yu
6aaf081489 Create a new app to hold GenerateSqlSchemaCommand (#409)
* Create a new app to hold GenerateSqlSchemaCommand

GenerateSqlSchemaCommand starts postgresql using testcontainer.
This makes junit etc a runtime dependency, allowing them to get
into release artifacts.

By moving this command to a separate tool, we can remove junit
etc as compile/runtime dependency.
2019-12-13 16:05:35 -05:00
Weimin Yu
1143b25391 Fix test on use of external schema jar (#412)
* Fix test on use of external schema jar

One test breaks if path to external schema is set and
test is run from the sqlIntegrationTest task.
2019-12-13 11:32:19 -05:00
Weimin Yu
65cf49f204 Fix sql script name conflict (#411)
* Fix sql script name conflict

There are two V11__ files due to concurrent merge. Renamed one
to V12__

Also removed a @NotNull annotation, which is the fist in the code base.
Most of the code base use @Nullable instead. If we do want to use
@NotNull, we may want to use the javax one instead.
2019-12-12 16:16:43 -05:00
Weimin Yu
f48e3933f5 Run cross-release SQL integration tests (#403)
* Run cross-release SQL integration tests

Run SQL integration tests across arbitrary schema and server
releases.

Refer to integration/README.md in this change for more information.

TESTED=Cloud build changes tested with cloud-build-local
       Used the published jars to test sqlIntegration task locally.
2019-12-12 13:47:49 -05:00
Lai Jiang
9853f23d94 Fix null pointer excpetion bug (#407)
The factory method passes a null trustedCertificates instead of an empty
list.
2019-12-12 13:06:43 -05:00
Ben McIlwain
db7fcf6c38 Add Cloud SQL premium list caches and compare prices with Datastore (#376)
* Add Cloud SQL premium list caches and compare prices with Datastore

Nothing will fail if the prices can't be loaded from Cloud SQL, or if the prices
are different. All that happens is that the error is logged. Then, once this is
running in production for awhile, we'll look at the logs and see if there will
be any pricing implications from switching over to the Cloud SQL version of the
premium lists.

* Add setMaxResults(1) per code review

* Add tests and reorder public functions

* Don't statically import caches

* Improve test pass rate

* Merge branch 'master' into dual-read-premium

* Add PremiumEntry mapping

* Allow update

* Revert column order

* Alphabetize PremiumEntry columns

* Don't bother trying to enforce order

* Private constructor
2019-12-11 16:20:19 -05:00
Weimin Yu
3aad8b6aa7 Use TextDiffSubject to compare multi-line text (#406)
* Use TextDiffSubject to compare  multi-line text

It illustrates differences better.

Moved TextDiffSubject.java to the common project for sharing.
2019-12-10 16:44:59 -05:00
Weimin Yu
f7b243e390 Make devProject a project property (#405)
* Make devProject a project property

Properties set in rootProject's ext block are not overridden by
commandline flags.

* Make flyway commands runnable from FOSS repo

rootProject.devProject, defined in projects.gradle, cannot be
overridden by commandline flags. Added a flag check in :db's
project script.
2019-12-10 11:39:56 -05:00
Michael Muller
e8745f7538 Update lockfiles (#402)
* Update lockfiles

Update lockfiles for the latest dependencies and metadata.
2019-12-10 09:28:44 -05:00
sarahcaseybot
2478a4a93b Add schema and DAO for cursors in cloudsql (#370)
* Add schema for Cursor

* Add CursorDao and CursorDaoTest

* Fix comment on getTld

* Change tld column to scope

* Fix cursorTime to be converted to DateTime internally and other small fixes

* Add a CursorType enum and a createGlobal constructor for Cursor

* Rename flyway file

* Use cursorType from common/Cursor.java and add null checks
2019-12-09 17:47:06 -05:00
Weimin Yu
bba5aff4b6 Verify RegistryTool can instantiate (#400)
* Verify RegistryTool can instantiate

Add a task that instantiates all command classes in RegistryTool
with runtimeClasspath.

Also make sure that runtimeClasspath is a superset of
compileClasspath.
2019-12-06 12:08:16 -05:00
Michael Muller
df7d272157 Adding junit back into the runtime classpath (#399)
* Adding junit back into the runtime classpath

Unfortunately, GenerateSqlSchemaCommand depends on junit via testcontainers.
We should really move GenerateSqlSchemaCommand out of nomulus tool (we only
use it during development) but this gets nomulus tool working for the time
being.

* Removed unnnecessary trace line.

* lockfiles generated after update_dependency.sh
2019-12-05 16:01:05 -05:00
Shicong Huang
988f78274e Check SQL dependency in super class for SqlIntegrationMembershipTest (#398)
In some cases, we define JpaTransactionManagerRule in a TestCase
class which is extended by the concrete test class. So, we need
to check if JpaTransactionManagerRule is also defined in the super
class.
2019-12-05 13:12:56 -05:00
Lai Jiang
369c1259fb Remove AUTH_INTERNAL_ONLY (#393)
Endpoints annotated with AUTH_INTERNAL_ONLY used to be accessible
manually with an internal RPC tool that adds App Engine specific HTTP
headers to a request to make it look like it comes from App Engine
(hence internal). This tool is used by admins to hit such endpoints
during debugging, making them effectively AUTH_INTERNAL_OR_ADMIN.

This RPC tool has never been made available outside Google so the open
source admins do not have such ability. A recent change in the RPC tool
made this hack stop working internally as well. This PR replaces all
all occurances of AUTH_INTERNAL_ONLY with AUTH_INTERNAL_OR_ADMIN and
brings the open source build into feature parity with the internal
version.

Also fixed a few issues the router tests.
2019-12-04 12:52:18 -05:00
Michael Muller
1e1c8cdd80 Replace TransactionManager.Work with Supplier (#397)
* Replace TransactionManager.Work with Supplier

Replace the "Work" class with the equivalent java.util.function.Supplier.
2019-12-04 12:00:13 -05:00
Weimin Yu
3d3d390a9f Fix bad dependency lock file (#396)
* Fix bad dependency lock file
2019-12-03 13:31:57 -05:00
Weimin Yu
f0fc53e035 Allow project dependency to use runtimeClasspath (#395)
* Allow project dependency to use runtimeClasspath

Project dependency should use runtimeClasspath. However, if
left unspecified, it uses 'default', which is the same as
the legacy 'runtime' configuration. (runtimeOnly dependencies
are left out).

Since runtimeClasspath cannot be referenced directly, we use
a custom config (deploy_jar) as a proxy.

By excluding testjars (leaked into 'compile' by third-party
dependencies) from runtimeClasspath, we prevent them from
getting into release artifacts.

Two meaningful changes in appengine_war.gradle and java_common.gradle

TESTED=Diffed contents of services/{module}/build/exploded-*
       Only three jars are removed: hamcrest-core, junit, and
       mockito-core.
2019-12-02 16:10:13 -05:00
Lai Jiang
c156842844 Upgrade to Gradle 6.0.1 (#394) 2019-12-02 15:24:18 -05:00
Lai Jiang
c584de9f72 Respect certificate validity period (#391)
Client SSL handler already performs the necessary validation. Only tests are
added.

Server SSL handler does not currently check for the validity period of
the client certificate as the insecure trust manager is used. This PR
added the check but does not actually terminate the connection yet. It
will log the expired certificates so that we can contact the registrars
to update them.

Once we are certain that all certificates are updated, we can turn off
dryrun mode.

We should also consider checking if the certificate has too long a
validity period as it defeats the purpose of using regularly updated
certificates to deprecate insecure cipher suites.
2019-11-27 16:08:38 -05:00
Shicong Huang
9be5091c84 Add entity for reserved list (#381)
This PR added the Cloud SQL entity for reserved list.
2019-11-26 16:51:41 -05:00
Michael Muller
28499d23a0 Print filenames that need to be reformatted (#386)
* Print filenames that need to be reformatted

Print the names of all java files that need reformatting during the check and
reformat operations.
2019-11-26 13:20:27 -05:00
Ben McIlwain
961d7e88f4 Use Maps.transformEntries() utility method to improve Map composition (#387)
* Use Maps.transformEntries() utility method to improve Map composition
2019-11-26 12:20:00 -05:00
Weimin Yu
215de62fa7 Stop publish Cloud SQL schema jar to maven repo (#383)
* Stop publish Cloud SQL schema jar to maven repo

The original purpose of the maven publication is for
use in server/schema compatibility tests. A commandline
flag can direct a test run to use different versions of
the schema jar. However, this won't work due to dependency
locking.
2019-11-25 18:23:02 -05:00
Lai Jiang
05d56fe1a2 Remove SSL initializer from the prober (#378)
The prober now uses the common SSL initializer in the networking
subproject.

Also changed both initializers to take an ImmutableList of certificates
other than an array of those, for better immutability.

I have no idea where these lockfile changes are coming from. They seem
to be pure noise as far as code review is concerned.
2019-11-22 17:46:06 -05:00
sarahcaseybot
e318f47fc6 Add a cursor for tracking monthly uploads of ICANN report (#343)
* Add a cursor for tracking monthly uploads of the transaction report to ICANN

* Add cursors to track activity, transaction, and manifest report uploads.

* Address comments

* Add @Nullable annotation to manifestCursor

* Add lock and batch load cursors.

* Add string formatting, autovalue CursorInfo object, and handling for null cursors

* Add some helper functions for loadCursors and restructure to require less round trips to the database

* Switch new cursors to be created with cursorTime at first of next month
2019-11-22 17:40:31 -05:00
Lai Jiang
cc5f62587e Make dev project configurable (#371)
* Make dev project configurable

We should not hardcode our dev project in the public config file.

* Remove the use of .ext when using external properties

They are only needed when defining properties.
2019-11-22 16:20:07 -05:00
Lai Jiang
02846bcbdd No-op: Use nicer HCL2 syntax. (#384)
Generated with perl -pi -e 's/\"\$\{([a-zA-Z0-9._-]*)\}\"/$1/g' $(find ./ -name '*.tf')

Copied from cl/282012376.
2019-11-22 16:08:56 -05:00
Ben McIlwain
c920f709ef Update the Registries cache to leverage/populate the Registry cache (#382)
* Update the Registries cache to leverage/populate the Registry cache

This is accomplished by also providing a loadAll() method on the Registry cache
that can be used to load an entire batch of Registry objects at once.

This improves efficiency, because now, any operation on Registries that loads
all the Registry entities (getTlds(), getTldsOfType(), and getTldEntities()), is
plumbed through the Registry cache, therefore loading it from that cache if it
exists and only hitting the DB if not. If not, this populates the Registry cache
upon loading, so that subsequent calls to Registry.get() will now hit the cache
instead of the DB.

To give a concrete example, the following code:

    for (String tld : Registries.getTlds()) {
      // ...
      doSomethingWith(Registry.get(tld));
      // ...
    }

is now much more efficient, because the initial call to Registries.getTlds()
populates all the entities in cache, and the subsequent individual calls to
Registry.get(tld) now retrieve them from the cache. Prior to this change,
Registries.getTlds() did not populate the Registry cache, and each subsequent
Registry.get(tld) had the potential to trigger an individual round-trip to the
DB, which is obviously bad for performance.
2019-11-22 14:47:09 -05:00
Ben McIlwain
ad9daac1ab Update premium and reserved list management docs (#380)
* Update premium and reserved list management docs

They were a little bit out of date.
2019-11-21 16:44:45 -05:00
Weimin Yu
9f0e24132a Break circular dependency between core and util (#379)
* Break circular dependency between core and util

Created a new :common project and moved a minimum
number of classes to break the circular dependency
between the two projects. This gets rid of the
gradle lint dependency warnings.

Also separated api classes and testing helpers into
separate source sets in :common so that testing
classes may be restricted to test configurations.
2019-11-21 15:36:55 -05:00
Shicong Huang
98414cb7cb Add a test to verify generated schema (#377) 2019-11-21 13:37:37 -05:00
Lai Jiang
6af1896362 Refactor common code used by the proxy and the prober (#375) 2019-11-20 12:42:44 -05:00
Weimin Yu
68887d427f Allow schema-loading from arbitrary url in tests (#374)
* Allow schema-loading from arbitrary url in tests

Server/Schema compatibility tests must be able to load different versions
of the SQL schema. This change allows test runners to override the
schema location using a system property.

Note: due to dependency-locking, we cannot manipulate the dependencies
closure in the build script to load different schema jars. The jars
must not be on the classpath.
2019-11-20 12:22:48 -05:00
Ben McIlwain
8a06ef09c0 Add a new method to load all Registry entities of a given type (#373)
* Add a new method to load all Registry entities of a given type

This is useful for things that need to know more than just the TLD strings
themselves, which is all Registries currently provides.
2019-11-18 17:40:42 -05:00
Shicong Huang
9e0368b77c Enable JpaTransactionManager in all environment (#352)
* Enable JpaTransactionManager in all environment

* Refactor to have a single place to create JpaTm
2019-11-18 14:53:49 -05:00
Weimin Yu
05c45da07a Use psql 11 docker image in all tests (#372)
* Use psql 11 docker image in all tests
2019-11-18 14:08:58 -05:00
Weimin Yu
365c5da942 Require explict tag when starting psql docker (#368)
* Require explict tag when starting psql docker

Defined a util class to return docker tag of desired PSQL version.
Class is defined in ':db' and shared by ':db' and ':core'. Used
an artifact declaration to exclude unnecesary compile dependencies.

Added a presubmit check for instantiations without explicit tag.
2019-11-18 11:33:26 -05:00
Weimin Yu
2cc2571375 Update schema deployment doc and flyway tool (#363)
* Update schema deployment doc and flyway tool

Disabled Flyway Gradle tasks with side effects on Cloud SQL
instances. They can still be used on local databases.

Also switched Flyway Gradle tasks to get credentials from
new locations (in domain-registry-dev).

Updated README file on schema push process. Also reformatted
the entire file.
2019-11-15 11:44:21 -05:00
Lai Jiang
d55230933b Upgrade to Gradle 6.0 (#364) 2019-11-13 13:17:29 -05:00
Shicong Huang
09aef04117 Add PersistenceXmlUtility and refactor related code (#357)
* Refactor GenerateSqlSchemaCommand

* Add and throw UncheckedClassNotFoundException
2019-11-12 15:08:54 -05:00
gbrodman
a392100852 Add by-repo-id method to RegistryLockDao (#344)
* Add by-repo-id method to RegistryLockDao

When we create new registry lock objects on user request, we will want
to make sure that there are no pending lock/unlock actions on this
domain. In order to do that, we will need to know what lock actions
there have been for this domain.

* Inline the query into a single statement

* whoops

* comments
2019-11-12 11:47:59 -05:00
Weimin Yu
b005e3aeb0 Fix a few lint errors (#361)
Replace deprecated bouncycastle class in SslInitializerTestUils.

Generic array as vargs: worked around it in ProbingAction and
removed unused method in CircularList.
2019-11-12 11:14:51 -05:00
Weimin Yu
bce09a3aa3 Deploy SQL schema from Cloud Build (#350)
Defined Docker image for schema deployment.

Included schema deploymer docker in the Cloud Build release process.

Defined Cloud Build config for schema deployment.

TESTED=Used cloud-build-local to test deployment flow
TESTED=Used docker to test schema deployer image in more ways
2019-11-12 11:06:16 -05:00
Weimin Yu
cb87658827 Remove unnecessary usage of CircularList (#358)
Implementation of CircularList is an overkill for its stated
purpose. Remove an unnecessary usage so that the remaining
use case may be refactored in isolation.
2019-11-11 18:04:46 -05:00
Michael Muller
455daae25c Add converter for DateTime (#346)
* Add converter for DateTime

* Added to sql integration test suite

* Removed obsolete "auto update schema" property
2019-11-11 17:24:20 -05:00
gbrodman
dea7dfcf28 Refactor UI actions to be more reliable (#348)
* Refactor UI actions to be more reliable

- Created HtmlAction to handle the nitty-gritty of login and setting up
HTML pages
- Created a test to verify that all UI actions implement JSON GET, JSON
POST, or HTML classes
- Move CSS renaming into a utility class

* Move logging of request into HtmlAction

* Comment and wording in exception

* mention JsonGetAction in the comment

* JsonGetAction extends Runnable
2019-11-11 16:18:48 -05:00
Shicong Huang
3a47fa2fe9 Fix beam pipeline deployment issue (#360) 2019-11-11 15:57:33 -05:00
gbrodman
0b7a0ba99d Update puppeteer (#359)
* Update puppeteer

They released 2.0.0 which fixes the security vuln in https-proxy-agent
2019-11-11 15:38:44 -05:00
gbrodman
ae89a8b76f Run both the AppEngineRule and JpaTransactionManagerRule in tests (#356)
* Run both the AppEngineRule and JpaTransactionManagerRule in tests

As previously written, the AppEngineRule wraps the infinite loop of the
server handling requests. This changes the code so that we'll have a
chain that does both the AppEngine management and the JPA transaction
manager management so we can actually use SQL in tests and in the test
server.

* remove log line
2019-11-11 11:26:20 -05:00
Shicong Huang
f0345ddf89 Resolve test flakiness caused by connection leak (#355) 2019-11-08 15:40:27 -05:00
Weimin Yu
e9f20b3401 Delete obsolet gradle config in root project (#354)
* Delete obsolet gradle config in root project
2019-11-08 14:32:08 -05:00
Lai Jiang
64e7a593ef Make Joda Money embeddable in entities (#340) 2019-11-07 17:03:00 -05:00
gbrodman
135ab66e55 Throw an EPP exception when using discount on premium domains (#351)
We should communicate to the users why this command failed, that they
are not allowed to use discounted allocation tokens on premium domains.
Currently it still fails, but we don't yet tell them why.
2019-11-07 15:30:23 -05:00
sarahcaseybot
69acb9f6de Add cursors for tracking ICANN report uploads to Cursor.java (#349) 2019-11-07 10:49:21 -05:00
gbrodman
5359f4640a Clean up Karma config file (#345)
Because we moved it into the source dirs, we don't need to reference
soyutils_usegoog.js separately
2019-11-06 10:33:58 -05:00
gbrodman
5c2856e3e2 Add RegistryLockGetActionTest to SQL testclasses (#347) 2019-11-05 15:10:13 -05:00
gbrodman
86e1fb85b6 Add a GET action and tests for registry lock retrieval (#326)
* Add a GET action and tests for registry lock retrieval

* Create isVerified method

* Allow lock access for admins even if they're not enabled on the registrar

* Simple CR responses

* Move locks retrieval to the GET action

* add newline at eof

* Switch to using ID
2019-11-05 13:19:32 -05:00
Michael Muller
301ab54fb4 Improve FilteringTest so it works in all cases (#339)
* Improve FilteringTest so it works in all cases

- Disable failOnMatchingTests so we don't fail if we filter out all of the
  tests for any given task.
- Add excludeTestCases flag so that we can turn this behavior off for test
  tasks that need to run TestCas/TestSuite classes.
- Add includeAllTests() function for test tasks where we don't explicitly
  include our required tests.

This makes all test tasks in core FilteringTest, with the exception of the
default test task which now does nothing (it just depends on the other test
tasks).

* Improve wording of excludeTestCases comment
2019-11-05 12:04:56 -05:00
Shicong Huang
2c6d71f04c Prevent test from changing golden schema (#337)
When we add the extra test entity to the current JpaTransactionManagerRule by calling jpaRule.withEntityClass(TestEntity.class) and jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update"), the rule would override the golden database scheme with the schema derived from the current entity class(This is an expected behavior by enabling HBM2DDL_AUTO). This behavior prevents us from detecting breaking changes in ORM classes.

This PR fixed the issue by adding HibernateSchemaExporter to export the DDL script for given extra entity class, and make JpaTransactionManagerRule execute the DDL script to create the corresponding table for the extra entity when initializing the database. This PR also simplified the code when adding extra entity class for testing. For now, you don't need to(and shouldn't) call jpaRule.withProperty(Environment.HBM2DDL_AUTO, "update").
2019-11-05 11:36:03 -05:00
Shicong Huang
ee4e6ace2a Remove jdbcUrl config for nomulus tool (#330) 2019-11-05 10:33:53 -05:00
Lai Jiang
c4721121eb Move soyutils_usegoog.js out of node_modules (#342)
* Move soyutils_usegoog.js out of node_modules

Everytime the npmInstall runs, it removes this file from node_modules.
Move it outside the folder to prevent this from happening.

* Move karma.conf.js and soyutils_usegooge.js

* Move karma.conf.js to be under core
2019-11-04 15:22:32 -05:00
Weimin Yu
37b9999cfc Release SQL schema in Cloud Build (#341)
* Release SQL schema in Cloud Build

Tentatively release SQL schema at the same time as the server release.
Publish schema jar to gs://domain-registry-maven-repository/nomulus
and also upload it with server artifacts.

Also removed the Gradle 'version' variable which is not used.

Tested=On cloud-build with a simplified version of
cloudbuild-nomulus.yaml.
2019-11-04 10:22:05 -05:00
Ben McIlwain
5f62f91cd5 Don't wrap exceptions thrown inside Cloud SQL transactions (#338)
* Don't wrap exceptions thrown inside Cloud SQL transactions

There's no reason to wrap them in PersistenceException, and it makes dealing
with thrown exceptions harder as seen in the diffs on test classes in this
commit.
2019-11-03 14:08:31 -05:00
sarahcaseybot
6f87fc115f Don't allow generation of allocation tokens for invalid domain names (#327)
* Address comments

* Delete client name change

* Delete skaffold.yaml
2019-11-01 12:19:10 -04:00
Lai Jiang
aaca6651c8 Make compilation work on Windows (#331)
* Make the Gradle upload plugin work on Windows

* Remove the bash script to generate package-info.java file

* Minor changes to the format
2019-10-31 17:04:35 -04:00
Ben McIlwain
03bbb2c057 Add a converter for CurrencyUnits stored in the database (#334)
* Add a converter for CurrencyUnits stored in the database

This uses the well-known String representation for currency units. It also
provides a base class for other converters that will be persisting the
toString() representation.

* Add DB and formatting changes

* Add tests, make minor fixes
2019-10-31 15:26:40 -04:00
Shicong Huang
d00ade8ae0 Enable JpaTransactionManager in sandbox (#323)
* Enable JpaTransactionManager in sandbox
2019-10-31 14:23:38 -04:00
Ben McIlwain
5e61adb396 Make the DB update instructions more comprehensive (#335)
* Make the DB update instructions more comprehensive

They were missing some initial steps.
2019-10-31 14:07:57 -04:00
Weimin Yu
6fe9cced3e Save current deployment tag for every environment (#332)
* Save release tag during deployment

* Save current tag for every environment

Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.

* Save current tag for every environment

Store tag of the current deployment in each environment.
This is used by the server-sql compatibility test.
2019-10-30 13:58:56 -04:00
Shicong Huang
6a3bd9418f Upgrade org.apache.beam related packages to latest version (#333) 2019-10-30 11:44:26 -04:00
Lai Jiang
3106958f18 Instrument proxy frontend latency metric (#320)
* Instrument proxy frontend latency metric

Backend latency only captures the time spent waiting for the GAE backend
to respond to a request, which is not representitive of what clients
experience. For instance it does not take into account the time the
proxy spends on processing the requests and the time it takes to send a
response to the client.

This PR adds a metric for frontend latency, which is the time spent on
answering a client request. This should serve as a better proxy for the
latency clients observe.

* Rename AbstractMetrics to BaseMetrics
2019-10-29 16:02:47 -04:00
Weimin Yu
7da94c90dc Restrict "Public Domain" license acceptance (#329)
"Public Domain" license must be reviewed case by case.
Removed blanket acceptance and named accepted dependencies
individually.

Also added a README file to warn about this license and WTFPL.
2019-10-28 13:32:42 -04:00
Ben McIlwain
dbb4092680 Don't retry permanent failures when uploading ICANN monthly reports (#328)
* Don't retry permanent failures when uploading ICANN monthly reports

There are two kinds of permanent failures that this checks for that we know will
never succeed, so it makes no sense to continue retrying 11 more times before
moving onto the next file to upload. These errors are:

1.
com.google.api.client.http.HttpResponseException: 403
Your IP address xx.xx.xx.xx is not allowed to connect

2.
com.google.api.client.http.HttpResponseException: 400
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><response xmlns="urn:ietf:params:xml:ns:iirdea-1.0"><result code="2002"><msg>A report for that month already exists, the cut-off date already passed.</msg><description>Date: 2019-09</description></result></response>

In order to implement this new functionality, this commit also adds a new way to
call Retriable that allows specifying the isRetryable Predicate (which is quite
useful).
2019-10-25 13:54:47 -04:00
Weimin Yu
af3223d086 Add a test task for all schema-dependent tests (#325)
Such tests are used in server-schema compatibility tests.

Also added a test that verifies that all eligible tests
are included.
2019-10-25 13:25:01 -04:00
Shicong Huang
2d7f80baaf Fix issues related to Cloud SQL connection (#321)
* Reenable JpaTransactionManager for Alpha and Crash

* Make UploadClaimsListCommand implement CommandWithCloudSql

* Fix wrong call to get password

* Use Cloud SQL Socket library to provision TransactionManager

* Change to use dataSource configs
2019-10-25 12:40:43 -04:00
gbrodman
d4d5d6a570 Add another test contact for Registry Lock testing (#324)
* Add another test contact for Registry Lock testing

Previously, we only had two contacts -- one per registrar. This PR adds
a second, registry-lock-enabled, contact to one registrar for two
reasons:

1. For registry-lock-related testing, we'd like to be able to test both
positively and negatively, making sure that the permissions work the way
they should
2. In general, the UI tests should include the case where we have
multiple contacts in the same registrar. Previously, this was never the
case in tests.

* Merge remote-tracking branch 'origin/master' into addTestContact
2019-10-25 10:24:59 -04:00
gbrodman
53c0be6537 Don't destroy existing registry lock passwords in contacts (#317)
* Don't destroy existing registry lock passwords in contacts

The existing code assumes that the "contacts" segment of the form
contains an exact representation of the registrar contacts. This breaks
when we have a contact with an existing registry lock password because
we don't want to keep passing around that password in plain text (we
never store it in plain text)

This PR changes the code so that instead of assuming the contact is
provided in its entirety, we load the contact from storage first
(matching by email address) if it exists. We then set the required
fields from the JSON object, and set the password optionally if it was
provided.

Alternatives:
- Create a separate RegistrarContactPassword object with a
RegistrarContact parent. This increases complexity significantly since
we'd be adding a parent-child relationship and adding more objects to
Datastore during the transition to SQL. It also doesn't completely solve
the problem of "When should we set the password?" because the password
field still must be part of the same form.
- Rearrange the UI so that the password is set as part of a completely
separate form with a separate submit action. This would be possible but
is sub-optimal for two reasons. First, we are trying to not re-engineer
the web console as much as possible since we're likely starting it from
scratch before too long anyway. Second, we want the
lock-password-setting to be part of the standard contact modification
workflow.

* Responses to CR

* Actually we need to allow "removal" of fields

* Remove optional

* one-statement building the contacts
2019-10-24 20:18:37 -04:00
gbrodman
63bb2dd79b Don't include password hash + salt in visible diffs (#322)
We don't want to override toDiffableFieldMap because (per the javadoc)
that is supposed to contain sensitive information. So, we should just
remove it before sending it out.
2019-10-23 10:57:46 -07:00
gbrodman
8278b5409e Add a registrarId index to RegistryLock (#312)
* Add a registrarId index to RegistryLock

* Merge remote-tracking branch 'origin/master' into getByRegistrar

* Responses to CR
2019-10-23 06:51:20 -07:00
gbrodman
f98b0f8739 Use merge instead of persist for RegistryLockDao (#310)
* Use merge instead of persist for RegistryLockDao

* CR responses
2019-10-22 12:21:32 -07:00
gbrodman
128fde16c3 Fix security alerts by upgrading (#318)
Note: this will be incomplete until Puppeteer releases a new version as
we await their https-proxy-agent version bump
2019-10-21 14:17:03 -07:00
Lai Jiang
45f7c89efd Upgrade to Gradle 5.6.3 (#315) 2019-10-18 15:13:46 -07:00
Weimin Yu
13dc758747 Use base64-encoded SQL credentials (#314)
* Use base64-encoded SQL credentials

Encode Cloud SQL credential files on gcs with base64,
to be consistent with our Cloud Build practices.

Also renamed a property that specifies where to publish
the schema jar. New name is schema_publish_repo.
2019-10-18 11:48:40 -04:00
Shicong Huang
85c11ca889 Use a single database container for all tests (#313) 2019-10-15 15:09:03 -04:00
sarahcaseybot
340bf40642 Update Spec11 email template (#308)
* Changes to Spec11 notice email.

* Fix lines that were too long
2019-10-14 12:23:28 -04:00
Michael Muller
53ece5eda4 Enable filtering across all test tasks (#311)
The segregated test targets in core break the --tests filter.  Fix this by
defining a "testFilter" property and creating the FilteringTest task type that
applies it to the property set by "--tests".
2019-10-11 14:24:01 -04:00
Shicong Huang
6c220567c8 Write ClaimsList to Cloud SQL (#223)
* Rewrite ClaimsListShard with new API

* Write ClaimsList to Cloud SQL

* Add creationTimestamp
2019-10-11 12:31:34 -04:00
Weimin Yu
c3e3a1353b Allow schema-push to all env with Flyway (#309)
* Make Flyway schema task work with prod and sandbox

Also renamed the 'superuser' role to 'admin' since
we do not own super user in Cloud SQL.

* Allow pushing schema to all env with Flyway

Desktop schema push to production is needed in the short term.
Long-termly we need to decide if this should be kept for glass
breaking

Schema push to sandbox and production requires interactiveconfirmation.

Also fixed a typo in initialize_roles.sql.
2019-10-10 16:32:21 -04:00
Ben McIlwain
ce480a5191 Add Bloom filters to the Cloud SQL PremiumList schema (#306)
* Add Bloom filters to the Cloud SQL PremiumList schema

They are slightly different from the existing Bloom filters stored in Datastore
in that they now use an ASCII String encoding rather than the more generic
CharSequence, and there is no maximum size (whereas we previously had to live
within the 1 MB max entity size for Datastore).
2019-10-09 17:06:42 -04:00
Weimin Yu
f2a2b2d2e2 Modify Cloud SQL user management scripts (#302)
* Modify Cloud SQL user management scripts

Create readonly and readwrite roles that may be granted to users.
Also configured default privileges for tables created in the future.

Made sure arbitrary users may not create database or tables.

* Modify Cloud SQL user management scripts

Create readonly and readwrite roles that may be granted to users.
Also configured default privileges for tables created in the future.

Made sure arbitrary users may not create database or tables.
2019-10-09 16:02:42 -04:00
gbrodman
906b054f4b Load persistence.xml classes before adding test entities (#307)
* Load persistence.xml classes before adding test entities

* Also use persistence.xml in GenerateSqlSchemaCommand

* Add exception message

* remove duplicate line
2019-10-09 15:15:04 -04:00
Shicong Huang
a694e247cd Add support for nomulus tool to connect to Cloud SQL (#303) 2019-10-09 10:30:35 -04:00
Shicong Huang
022f397cd9 Implement ZonedDateTimeConverter (#287)
* Implement ZonedDateTimeConverter

* Use dedicated TestEntity for ZonedDateTimeConverterTest
2019-10-08 16:46:07 -04:00
Ben McIlwain
5dc058ec99 Automatically apply JPA type converters (#305)
* Automatically apply JPA type converters

* Include converters in tests and schema generation too
2019-10-08 13:16:39 -04:00
Ben McIlwain
0fd7cf29b5 Make it clear that registered domain lists are also exported to Drive (#304)
* Make it clear that registered domain lists are also exported to Drive
2019-10-08 13:15:55 -04:00
Ben McIlwain
bc7f3546c7 Add initial support for persisting premium lists to Cloud SQL (#285)
* Add initial support for persisting premium lists to Cloud SQL

This adds support to the `nomulus create_premium_list` command only; support for
`nomulus update_premium_list` will be in a subsequent PR.

The design goals for this PR were:
1. Do not change the existing codepaths for premium lists at all, especially not
   on the read path.
2. Write premium lists to Cloud SQL only if requested (i.e. not by default), and
   write to Datastore first so as to not be blocked by errors with Cloud SQL.
3. Reuse existing codepaths to the maximum possible extent (e.g. don't yet
   re-implement premium list parsing; take advantage of the existing logic), but
   also ...
4. Some duplication is OK, since the existing Datastore path will be deleted
   once this migration is complete, leaving only the codepaths for Cloud SQL.

* Refactor out common logic

* Add DAO test

* Add tests for parsing premium lists

* Use containsExactly

* Code review changes

* Format

* Re-generate schema

* Fix column names

* Make some tests pass

* Add SQL migration scripts

* Fix test errors
2019-10-08 11:47:22 -04:00
Ben McIlwain
658f61bd8f Fix errors in DB update README file (#301)
* Fix errors in DB update README file
2019-10-07 15:55:53 -04:00
Michael Muller
8ca8fff387 Refactor environment -> project map out of build (#300)
Move the project mapping into its own file so that it can be more easily
overriden.
2019-10-07 13:34:49 -04:00
Michael Muller
8bcfb1802e Add an explanation to dummied-out JPA init (#299)
* Add an explanation to dummied-out JPA init

Add a more elaborate explanation of why actual JpaTransactionManager
initialization was removed from the factory.
2019-10-07 12:47:24 -04:00
gbrodman
a259dee986 Add a DAO for RegistryLock objects (#290)
* Add a DAO for RegistryLock objects

* Add an index on verification code and remove old file

* Move to v4

* Use camelCase in index names

* Javadoc fixes

* Allow alteration of RegistryLock objects in-place

* save, load-modify, read in separate transactions

* Change the creation timestamp to be a CreateAutoTimestamp
2019-10-07 11:24:08 -04:00
Michael Muller
ebc8d54f94 Add persistence.xml to the war files (#293)
* Add persistence.xml to the war files
* Always use the DummyJpaTransactionManager

Use the DJTM until we get all of the dependencies set up for all of the
environments.

This shouldn't affect any of the unit tests, these use the
JpaTransactionManagerRule to set up a local database and connection.

This fixes the App Engine build.
2019-10-06 21:32:27 -04:00
sarahcaseybot
e9ce389269 Implement UpdateAutoTimestampConverter (#298)
* Implement a JPA-based converter for UpdateAutoTimestamp, allowing us to persist instances of this class.
2019-10-04 16:45:32 -04:00
Michael Muller
e07be9780e Implement CreateAutoTimestampConverter (#282)
* Implement CreateAutoTimestampConverter

Implement a JPA-based converter for CreateAutoTimestamp, allowing us to
persist instances of this class.

Note that converters appear to be required to convert to and from database
types that are generally known to JDBC.  For example, conversion to Timestamp
works, conversion to OffsetDateTime does not (even though this works through
the JDBC interface directly).
2019-10-04 16:31:14 -04:00
Michael Muller
31b5abb9a1 Improve error handling for environment property (#295)
* Improve error handling for environment property

Improve the error messages that we get for a bad or missing environment
property.  Move the property processing into the main build so that we do it
only once and configure all of the appengine deployment tasks to check that a
project has been defined and print a friendly error if it hasn't.

Note that even if the check isn't configured, this change prevents deployment
because the gcpProject will be set to null.  It just won't print as useful an
error message.

Tested: ran appengineDeployAll with and without the environment property, ran
"build" to verify that we don't get any complaints for non-deployment targets.

* Changes for review.

* Changes for review

* Changes for review.

* Changes for review.
2019-10-04 13:23:36 -04:00
Michael Muller
aa4e242a34 Give JpaTransactionManagerRule more parameters (#292)
* Give JpaTransactionManagerRule more parameters

Allow users of the rule to add annotated classes and properties, both useful
for testing.

* Change in response to review.

* Changes for review.

* Move test EntityManagerFactory create method

Move the test create method into the JpaTransactionManagerRuleTest.

* Remove nomulus SQL dialect from G.S.S.Command

Remove NomulusPostgreSQLDialect from GenerateSqlSchemaCommand (it has been
moved to its own top-level class).
2019-10-03 16:14:28 -04:00
Weimin Yu
964f264c9d Add maven-publish task for SQL schema jar (#289)
* Add maven-publish task for SQL schema jar

Add task to publish SQL schema jar with flyway scripts and
golden schema to a maven repo. This will be used
for pre-release testing in the future.

This task is not part of build and needs to be invoked explicitly.

User needs to provide schema_jar_repo and schema_version
properties.

* Merge branch 'master' of https://github.com/google/nomulus into publish-schema-jar

* Add maven-publish task for SQL schema jar

Add task to publish SQL schema jar with flyway scripts and
golden schema to a maven repo. This will be used
for pre-release testing in the future.

This task is not part of build and needs to be invoked explicitly.

User needs to provide schema_jar_repo and schema_version
properties.
2019-10-02 14:27:36 -04:00
gbrodman
cc018a6dac Don't crash on a null completion timestamp (#296)
* Don't crash on a null completion timestamp

* optional
2019-10-02 12:53:22 -04:00
Weimin Yu
c25adbbd9c Restrict nomulus user access to flyway table (#297)
* Restrict nomulus user access to flyway table

The regular read-write user should not have write permissions to
the flyway metadata table.
2019-10-02 11:05:46 -04:00
gbrodman
42edf16f3f Edit a couple of nits in the db README (#291)
* Edit a couple of nits in the db README

* Update README.md
2019-09-30 15:50:19 -04:00
Shicong Huang
36ab0cb45c Add Cloud SQL configs for nomulus tool (#288)
We will use a different user for nomulus tool to connect to Cloud SQL. 
This PR added corresponding configurations for that.
2019-09-27 15:58:18 -04:00
Shicong Huang
34da498958 Add sql scripts to create other types of user (#283)
1. User with read-only permission to all tables
2. User with read-write permission to schema and all tables
2019-09-27 15:12:59 -04:00
Lai Jiang
a3317b8552 Protect KMS-secured data against destruction in upcoming google provider update. (#284)
Export of cl/270900150.

To refer to a KMS key or key ring, we should use the stable `.self_link`. Using `.id` instead provides an unstable identifier which may change (and it will change in the upcoming update of the google provider to 2.9.1). A change in the identifier will cause Terraform to destroy and recreate the key. Destroying the key means all data associated with it is lost; the key cannot be recreated.

This CL replaces `.id` with `.self_link`, so all of those problems will not happen. In addition, `prevent_destroy` protects the key against delete-and-recreate in general.
2019-09-27 12:12:58 -04:00
Shicong Huang
9c7872dbb7 Add socket factory as runtime dependency (#286)
* Add Cloud SQL socket factory as runtime dependency

* Lock update from update_dependency.sh
2019-09-27 11:41:59 -04:00
Shicong Huang
ff439f598d Add JpaTransactionManagerRule (#277)
* Add RegistryRuntime and JpaTransactionManagerRule

* Revert RegistryJavaRuntime change

* Add JpaTransactionManager interface
2019-09-24 15:38:53 -04:00
Weimin Yu
b0c13042a6 Upgrade to Truth 1.0 (#281)
* Upgrade to Truth 1.0

Refactored fail(...) to assertWithMessage().fail().

Upgraded com.google.monitoring-client family of dependencies to 1.0.6

Also fixed bad use of io.StringIO (on binary buffer) recently introduced to
google-java-format-diff.py.
2019-09-24 10:23:58 -04:00
Lai Jiang
3009f754fa Link to Sourcegraph logo a different file. (#280)
The logo will be visible on cloned repos outside GitHub. See
b/141380880.
2019-09-23 15:31:09 -04:00
Shicong Huang
cc314d4838 Make project core depend on db at test runtime (#278)
This PR also upgraded the version of Guava to 28.1-jre to
resolve missing stream package issue.
2019-09-23 14:12:24 -04:00
Michael Muller
153fc7d6bf Add work-around for our "explode war" failure (#279)
For some reason tools.jar (which we don't need in the runtime in the first
place) gets exploded readonly, causing subsequent builds to fail since they
can't overwrite it.  This hack makes it writable at the end of explodeWar.
2019-09-20 16:23:36 -04:00
Shicong Huang
b35c813ae6 Add sql files to manage nomulus user (#274) 2019-09-20 14:46:02 -04:00
Lai Jiang
3764e2b26c Update registryTool task to better handle command line arguments (#273)
* Update registryTool task to better handle command line arguments

Tokens are delimited by pipes and can be escaped. If we use -args
directly, we will not be able to escape the delimiter, if both the
double and single quotes happen to be present in the arguments.

See:

https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/JavaExec.html#setArgsString-java.lang.String-
2019-09-20 13:04:01 -04:00
gbrodman
8f5bd5da01 Fix non-Python3-compliant code in formatter script (#275)
* Fix non-Python3-compliant code in formatter script

* Add space
2019-09-19 12:49:22 -04:00
Weimin Yu
8123c53916 Fix dependency-locking config (#276)
* Fix dependency-locking config

Reenable dependency locking after a bug errorneouly turned it off.

Removed the guava-related workaround that forcefully resolve to
the -jre distribution.

Enabled locking for buildSrc by updating its property file.

Updated all lock files.
2019-09-19 11:37:20 -04:00
Shicong Huang
d031dedad6 Implement JpaTransactionManager (#268) 2019-09-19 10:01:40 -04:00
Michael Muller
d14f0fe485 Output command test output as well as consuming it (#248)
* Output command test output as well as consuming it

CommandTestCase currently consumes stdout & stderr for the command being
tested.  Unfortunately, this results in us not being able to see the command
output.  Add an output splitter so that output gets written to the original
stream in addition to being captured.

A simpler approach would be to print the captured data after command
completion.  However, this won't work for tests that become hung and also
won't display results in real-time.

Tested: Ran a command test with verboseTestOutput=true, verified that standard
output was visible.

* Save and restore original stdout/err in cmd tests

We have to restore the original stdout/stderr print streams otherwise we end
up nesting them across tests which eventually causes the RDE tests to OOM.
2019-09-17 13:23:30 -04:00
Lai Jiang
b86a35bca1 Upgrade to Gradle 5.6.2 (#272) 2019-09-16 14:07:59 -07:00
Weimin Yu
1db5fba452 Add RegistryLock schema to Flyway deployment folder (#270)
* Add RegistryLock schema to Flyway deployment folder

Added creation script of RegistryLock to Flyway deployment folder.

Fixed previous scripts (PremiumList- and ClaimsList-related) for
FK name change (cause by table name changes: names are quoted now).
We should consider generating foreign key names by ourselves.

Since the alpha database is empty, we dropped and recreated the schema.

Added instructions on how to submit new database incremental changes
in the README file.

Updated RegistryLock.java, removing unnecessary annotations:
- For most fields, the 'name=' property is no longer necessary not that
  the naming strategy is in place. The exceptions are the two used in
  the unique index.
- The @Column annotation is implicit.
2019-09-16 16:47:58 -04:00
Weimin Yu
668ed7c5f6 Force Gradle to use jre version of Guava (#271)
* Force Gradle to use jre version of Guava

Guava is often specified by version only by other
dependencies. In such cases, Gradle usually resolves
it to the '-android' version, which lacks the collection
classes.

We use custom resolution strategy to force the selection
of the '-jre' version.
2019-09-16 14:36:14 -04:00
Weimin Yu
3fb799f112 Add schema deployment tests (#265)
* Add schema deployment tests

Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.

Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.

Upgraded testcontainers to 1.12.1.

Added a custom Truth subject for better diffing of multi-line
text blocks.

Removed claims_list.sql and premium_list.sql, as we do not have use for
them.

* Add schema deployment tests

Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.

Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.

Upgraded testcontainers to 1.12.1.

Added a custom Truth subject for better diffing of multi-line
text blocks.

Removed claims_list.sql and premium_list.sql, as we do not have use for
them.

* Add schema deployment tests

Updated flyway schema script files so that they reflect what is
currently deployed in alpha: ClaimsList and PremiumList related
elements.

Put post-schema-push pg_dump output in nomulus.golden.sql as the
authoritative schema. Also added test to verify that the schema
pushed by flyway will result in exactly the golden schema.

Upgraded testcontainers to 1.12.1.

Added a custom Truth subject for better diffing of multi-line
text blocks.

Removed claims_list.sql and premium_list.sql, as we do not have use for
them.
2019-09-12 15:16:12 -04:00
Weimin Yu
40a6b788a0 Regenerate schema using tools command (#269)
* Regenerate schema using tools command

Rerun GenerateSqlSchemaCommand to pick up RegistryLock
and naming strategy change.

Also updated a new license term which seems to just pop up.
2019-09-12 14:26:22 -04:00
Lai Jiang
b87ef869a0 Add a task to run the nomulus tool (#262) 2019-09-12 08:09:32 -07:00
Shicong Huang
48d8b1274f Add configurations for Cloud SQL secrets (#266) 2019-09-11 12:20:08 -04:00
gbrodman
401653ad4a Add RegistryLock SQL schema (#243)
* Add RegistryLock SQL schema

* Refactor a bit

* Move registrylock -> domain

* Clearing up lock workflow

* Add more docs and remove LockStatus

* Responses to CR

* Add repoId javadoc

* Add registry lock to persistence xml file

* Quote rather than backtick

* Remove unnecessary check

* File TODO

* Remove uniqueness constraint on verification code

* Remove import

* add index

* Add to SQL generation task

* Move fields around to be the same order as Hibernate's generated sql
2019-09-10 18:17:36 -04:00
gbrodman
b35026b30d Add naming strategy (#256)
* Add naming strategy

* Add test for formatting in GenerateSqlSchemaCommandTest

* "domain" -> "Domain"

* Call site literals

* checkstyle

* varchar -> text

* Fix external messaging capitalization typo
2019-09-09 14:27:52 -04:00
Michael Muller
c5e4e862bd Fix command tests that set stdout/stderr (#258)
* Fix command tests that set stdout/stderr

CommandTestCase already wraps stdout stderr, just use the provided methods.
2019-09-09 09:12:48 -04:00
gbrodman
729b69550e Add DateTimeUtils conversion methods (#260)
* Add DateTimeUtils conversion methods

* Use a more reliabile method than getId

* Add some more tests
2019-09-06 16:46:51 -04:00
Weimin Yu
471ed7caa7 Use Flyway to deploy SQL schema to non-prod (#255)
* Use Flyway to deploy SQL schema to non-prod

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.

* Use Flyway to deploy SQL schema to non-prod

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.

* Use Flyway to deploy SQL schema to non-prod

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Corrected the type of ClaimsEntry's revision_id column.
It should be plain int8, not bigserial.

Make GenerateSqlSchemaCommand use a custom dialect that
converts all varchar type to 'text' and timestamp to
'timestamptz'.

* Use Flyway to deploy SQL schema to non-prod

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.

Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.

* Use Flyway to deploy SQL schema to non-prod

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.

Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.

* Use Flyway to deploy SQL schema to non-prod

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.

Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
2019-09-06 16:29:49 -04:00
Ben McIlwain
ded6d38223 Add Cloud SQL schema for premium lists (#254)
* Re-add other schema classes

* Add Cloud SQL schema for premium lists

This won't work quite yet, pending a solution for the type translator issue
(which will be needed for the currency field, and potentially others).
2019-09-06 11:25:34 -04:00
Ben McIlwain
859b70098c Fix typo of word "dependencies" in gradle build files (#259)
* Fix typo of word "dependencies" in gradle build files
2019-09-05 16:52:46 -04:00
gbrodman
1e0be188fe Serve up all the closure library files (#257)
Not necessary for the tests but it gets rid of log lines
2019-09-05 10:57:35 -04:00
Michael Muller
f83a8a221b Generate basic schema for all of DomainBase (#246)
* Generate basic schema for all of DomainBase

Generate a basic schema for DomainBase and everything that is part of it.
This still isn't complete, in particular it lacks:

- Correct conversions for problematic types (e.g. DateTime, Key...)
- Schema generation for history records.
- Name translation.
2019-09-05 10:54:29 -04:00
Shicong Huang
2967256766 Consolidate EntityManagerFactoryProviderTest (#253)
Check if EntityManagerFactory is null before calling close() to avoid NullPointerException.
2019-09-04 15:46:13 -04:00
Ben McIlwain
28700cd610 Make parameter names in generate_sql_schema command consistent (#250)
* Make parameter names in generate_sql_schema command consistent

The rest of the nomulus commands use underscores for delimiting words in
parameter names, so this should too.

Also fixed capitalization of some proper nouns.
2019-09-04 11:10:22 -04:00
Michael Muller
c4b87a1d13 Move EntityManagerFactoryProviderTest to fragile (#251)
* Move EntityManagerFactoryProviderTest to fragile

* Add EMF Provider Test to docker tests

Add EntityManagerFactoryProviderTest to the docker incompatible test patterns
and use the latter list to compose the fragile tests.
2019-09-03 19:47:58 -04:00
gbrodman
1a728e96cb Allow setting the registry lock password in the UI (#241)
* Allow setting the lock password in the UI

* Add more screenshot tests

* Responses to CR and more screenshot tests

* Formatting

* Simplify lambda
2019-09-03 16:39:02 -04:00
Michael Muller
d3ccad3aa7 Start postgresql container in generate_sql_schema (#249)
* Start postgresql container in generate_sql_schema

Add a --start-postgresql option to the nomulus generate_sql_schema command so
that users don't have to start their own docker container to run it.

* Made default behavior be to give guidance
2019-08-30 16:04:34 -04:00
Ben McIlwain
dc9d9158d8 Don't write TX records for domains deleted in autorenew grace period (#244)
* Don't write TX records for domains deleted in autorenew grace period

When the project was originally being designed, we envisioned have a purely
point-in-time architecture that would allow the system to run indefinitely
without requiring any background batch jobs. That is, you could create a domain,
and 10 years later you could infer every autorenewal billing event that should
have happened during those 10 years, without ever having to run any code that
would go through and retroactively create those events as they happened.

This ended up being very complicated, especially when it came to generating
invoices, so we gave up on it and instead wrote the
ExpandRecurringBillingEventsAction mapreduce, which would run as a cronjob and
periodically expand the recurring billing information into actual one-time
billing events. This made the invoicing scripts MUCH less complicated since they
only had to tabulate one-time billing events that had actually occurred over the
past month, rather than perform complicated logic to infer every one-time event
over an arbitrarily long period.

I bring this up because this architectural legacy explains why billing events
are more complicated than could otherwise be explained from current
requirements. This is why, for instance, when a domain is deleted during the 45
day autorenewal period, the ExpandRecurringBillingEventsAction will still write
out a history entry (and corresponding billing events) on the 45th day, because
it needs to be offset by the cancellation billing event for the autorenew grace
period that was already written out synchronously as part of the delete flow.

This no longer really makes sense, and it would be simpler to just not write out
these phantom history entries and billing events at all, but it would be a
larger modification to fix this, so I'm not touching it here.

Instead, what I have done is to simply not write out the DomainTransactionRecord
in the mapreduce if the recurring billing event has already been canceled
(i.e. because the domain was deleted or transferred). This seems inconsistent
but actually does make sense, because domain transaction records are never
written out speculatively (unlike history entries and billing events); they
correspond only to actions that have actually happen.  This is because they were
architected much more recently than billing events, and don't use the
point-in-time hierarchy.

So, here's a full accounting of how DomainTransactionRecords work as of this commit:
1. When a domain is created, one is written out.
2. When a domain is explicitly renewed, one is written out.
3. When a domain is autorenewed, one is written out at the end of the grace period.
4. When a domain is deleted (in all cases), a record is written out recording the
   deletion.
5. When a domain is deleted in the add grace period, an offsetting record is
   written out with a negative number of years, in addition to the deletion record.
6. When a domain is deleted in the renewal grace period, an offsetting record is
   likely written out in addition.
7. When a domain is deleted in the autorenew grace period, there is no record that
   needs to be offset because no code ran at the exact time of the autorenew, so
   NO additional record should be written out by the expand mapreduce.
   *THIS IS CHANGED AS OF THIS COMMIT*.
8. When a domain is transferred, all existing grace periods are cancelled and
   corresponding cancelling records are written out. Note that transfers include a
   mandatory, irrevocable 1 year renewal.
9. In the rare event that a domain is restored, all recurring events are
   re-created, and there is a 1 year mandatory renewal as part of the restore with
   corresponding record written out.

So, in summary, billing events and history entries are often written out
speculatively, and can subsequently be canceled, but the same is not true of
domain transaction records.  Domain transaction records are only written out as
part of a corresponding action (which for autorenewals is the expand recurring
cronjob).

* rm unused import
2019-08-30 12:04:35 -04:00
Michael Muller
0daa89ae25 Remove the "showAllOutput" property from the build (#247)
* Remove the "showAllOutput" property from the build

It doesn't work very well and has been superseded by "verboseTestOutput",
which does the same thing and more.
2019-08-30 10:43:07 -04:00
gbrodman
d6bcdc241e Remove 'value' from RDAP link responses (#236)
* Remove 'value' from RDAP link responses

* Change application type to rdap+json

* Merge remote-tracking branch 'origin/master' into removeValueRdap

* CR response
2019-08-30 10:21:03 -04:00
Shicong Huang
487b695a10 Set up database connection pool (#234)
Set up database connection pool
2019-08-29 16:12:28 -04:00
Weimin Yu
b5ef99a8f8 Only apply Google Java format to changed lines (#176)
* Only apply Google Java format to changed lines

* Only apply Google Java format to changed lines

* Only apply Google Java format to changed regions

Diffs are relative to origin/master.

Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
  will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
  format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
  javaIncrementalFormatApply is invoked.

These tasks work from the root directory and process the buildSrc directory
too.

The Spotless Java config is removed.

* Only apply Google Java format to changed regions

Diffs are relative to origin/master.

Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
  will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
  format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
  javaIncrementalFormatApply is invoked.

These tasks work from the root directory and process the buildSrc directory
too.

The Spotless Java config is removed.

* Only apply Google Java format to changed regions

Diffs are relative to origin/master.

Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
  will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
  format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
  javaIncrementalFormatApply is invoked.

These tasks work from the root directory and process the buildSrc directory
too.

The Spotless Java config is removed.

* Only apply Google Java format to changed regions

Diffs are relative to origin/master.

Three tasks are added:
- javaIncrementalFormatCheck is added to the build workflow, and
  will abort build if format violations are found.
- javaIncrementalFormatApply needs to be manually invoked to correct
  format violations, the same behavior as spotlessApply.
- javaIncrementalFormatDryRun shows the changes that would happen if
  javaIncrementalFormatApply is invoked.

These tasks work from the root directory and process the buildSrc directory
too.

The Spotless Java config is removed.
2019-08-29 16:04:34 -04:00
Lai Jiang
dd4300fce7 Update proxy deployment script description (#245) 2019-08-28 17:35:43 -04:00
Lai Jiang
844c47061b Use Gradle from gradle.org in Gradle wrapper (#239)
During release we substitute the link to a binary hosted on GCS. The
normal GitHub build can use the binary on gradle.org
2019-08-27 11:11:35 -04:00
Shicong Huang
dcceb0d49a Add sql schema and entity class for ClaimsList (#227) 2019-08-26 18:30:51 -04:00
Lai Jiang
f72c6c3c80 Remove GtechTool (#242)
It is burdensome to have to maintain two sets of tools, one of which
contains a strict subset of functionalities of the other. All admins
should use the same tool and their ability to administer should be
restricted by the IAM roles they have, not the tools they use.
2019-08-26 17:15:48 -04:00
Michael Muller
c645fe6766 Fix skipDockerIncompatibleTests (#240)
We have to add a check for the skipDockerIncompatibleTests property in the
fragileTest task, since that's where these tests now live.
2019-08-26 12:31:24 -04:00
Lai Jiang
bf52a78e89 Update Gradle wrapper to 5.6 (#232)
* Update Gradle wrapper to 5.6

* Upgrade shadowJar to 5.1 and increase JVM heap size

* Use Gradle binary on GCS
2019-08-23 22:10:55 -07:00
gbrodman
8ec16dca8d Add a registry lock password to contacts (#226)
* Add a registry lock password to contacts

* enabled -> allowed

* Simple CR responses, still need to add tests

* Add a very simple hashing test file

* Allow setting of RL password rather than directly setting it

* Round out pw tests

* Include 'allowedToSet...' in registrar contact JSON

* Responses to CR

* fix the hardcoded tests

* Use null or empty rather than just null
2019-08-23 22:34:43 -04:00
Michael Muller
69cb852a9c Add a "showErrorOutput" property (#237)
* Add a "showErrorOutput" property

Add a property to let us dump test output and final status in real-time to the
console.
2019-08-22 11:10:56 -04:00
Michael Muller
6dee3d526e Add a generate_sql_schema command (#230)
* Add a generate_schema  command

Add a generate_schema command to nomulus tool and add the necessary
instrumentation to EppResource and DomainBase to allow us to generate a
proof-of-concept schema for DomainBase.

* Added forgotten command description

* Revert "Added forgotten command description"

This reverts commit 09326cb8ac.
(checked in the wrong file)

* Added fixes requested during review

* Add a todo to start postgresql container

Add a todo to start a postgresql container from generate_sql_command.
2019-08-20 12:29:36 -04:00
Shicong Huang
3ba6e7bd06 Remove hardcoded Java home path (#235) 2019-08-20 10:54:45 -04:00
Lai Jiang
2e84cdfc4b Rename packages in the prober (#233)
Package names should not be plural.
2019-08-19 12:11:19 -04:00
Lai Jiang
77f998ee4b Disable Travis (#231) 2019-08-16 17:13:43 -04:00
Lai Jiang
c93248ec10 Update build badges to use Kokoro results for both FOSS and internal (#229) 2019-08-16 15:06:02 -04:00
Aman Sanger
3ac179aead Prober metrics collection (#222)
* Added MetricsHandler and Clock to ProbingSequences

* Minor fixes after rebase onto master

* Added metrics gathering to ProbingSequences

* Added testing of MetricsCollector method calls in ProbingSequenceTest

* Added tests on latency recording to ProbingSequenceTest

* Added response name as label to metrics
2019-08-16 13:49:50 -04:00
gbrodman
92f2f3274e Add a boolean for when a registrar has enabled registry lock (#228)
* Add a boolean for when a registrar has enabled registry lock

* enabled -> allowed

* get -> is
2019-08-16 10:46:07 -04:00
Aman Sanger
57975898d5 Prober EPP components added (#202)
* Updated issues in rebasing

* Minor style change on prober/build.gradle

* Fixed warnings for java compilation

* Fixed files to pass all style tests

* Initial Commit.

* Deleted unfinished features. Added ActionHandler and its Unit Tests.

* Included prober subproject in settings.gradle

* Added Protocol Class and its Basic Unit Tests

* Added Changes Suggested by jianglai

* Fixed Gitignore to take out AutoValue generated code

* Removed AutoValue java files

* Added gitignore within prober

* Removed all generated java

* Final Changes in .gitignore

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed build.gradle changes requested

* Removed Files irrelevant to current pull request

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Fully Updated ActionHandler (missing updated JavaDoc)

* Added changed Protocol and both Inbound and Outbound Markers

* Removed AutoVaue ignore clause from .gitignore

* removed unneccessary dependencies in build.gradle

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Fixed javadoc for Marker Interfaced

* Modified Comments on ActionHandler

* Removed LocalAddress from Protocol

* Fixed Travis Build Issues

* Rebased to Master and added in modified Handlers and ProbingAction

* Fixed changes suggested by CydeWeys

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Modified license header dates

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Rebased to Master and added in modified Handlers and ProbingAction

* Fixed changes suggested by CydeWeys

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Modified license header dates

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Rebased to master

* Updated issues in rebasing

* Minor style change on prober/build.gradle

* Fixed warnings for java compilation

* Fixed files to pass all style tests

* Minor syle fixes after succesful rebase onto master

* Initial Commit.

* Added Protocol Class and its Basic Unit Tests

* Fixed Gitignore to take out AutoValue generated code

* Final Changes in .gitignore

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Removed AutoVaue ignore clause from .gitignore

* Rebased to Master and added in modified Handlers and ProbingAction

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Modified license header dates

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Fixed files to pass all style tests

* Fixed changes suggested by CydeWeys

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Modified license header dates

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Rebased to Master and added in modified Handlers and ProbingAction

* Fixed changes suggested by CydeWeys

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Modified license header dates

* Updated WebWhois tests.

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Rebased to master

* Updated issues in rebasing

* Added circular linked list to utils

* License Header added

* Refactored probing sequence to be circular linked list iterator

* Modified ProbingStep tests to reflect new ProbingStep structure.

* Added circular linked list to utils

* Added circular linked list to utils

* License Header added

* License Header added

* Refactored probing sequence to be circular linked list iterator

* Modified ProbingStep tests to reflect new ProbingStep structure.

* Added missing license header to DefaultCircularLinkedListIterator

* Fixed changes suggested by CydeWeys

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Fixed max column length to be 100

* Rebased to Master and added in modified Handlers and ProbingAction

* Modified license header dates

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Fixed changes suggested by CydeWeys

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Modified license header dates

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Added circular linked list to utils

* Added circular linked list to utils

* License Header added

* License Header added

* Refactored probing sequence to be circular linked list iterator

* Refactored probing sequence to be circular linked list iterator

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* ProbingStepTest modified to have fewer unnecessary helper methods

* Updated issues in rebasing

* Fixed max column length to be 100

* Minor changes to pass style tests

* Successful rebase onto finished web-whois branch

* Removed need for TestTokens with Mockito mocks of Tokens

* Fixed style issues in DefaultCircularLinkedListIterator and AbstractCircularLinkedListIterator

* Modified CircularList according to changes suggested by jianglai.

* Added Protocol Class and its Basic Unit Tests

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Minor Style Fix

* Minor Style Fix

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Updated build.gradle file

* Updated WebWhois tests.

* Added Basic EPP structure

* Added Basic EPP structure

* Prober Updated tests

* Prober Updated tests

* Fully functioning EPP sequences with modified WebWhois base.

* Fully functioning EPP sequences with modified WebWhois base.

* Added Modified test server infrastructure.

* Added Modified test server infrastructure.

* Allowed ActionHandler to pass status to next hanlder in pipeline (to be MetricsHandler).

* Allowed ActionHandler to pass status to next hanlder in pipeline (to be MetricsHandler).

* Javadoc on EppRequestMessage added

* Javadoc on EppRequestMessage added

* Updated EppServer to properly send successful Check responses.

* Updated EppServer to properly send successful Check responses.

* Allowed for expected failures in EPP actions.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Fully rebased branch to prober-web-whois after refactoring

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Rebased to master

* Fixed max column length to be 100

* Fixed files to pass all style tests

* Minor changes to pass style tests

* Successful rebase onto circular-list

* Epp Refactored to accomodate circular linked list PR

* Modified construction of Epp Probing Sequences to reflect CircularList change

* Renamed ProberModule provided Duration

* Removed unnecessary ServerSideException file

* Google-Java-Format run on all prober files

* Style fix on ProbingSequence and its unit tests

* Removed subclasses of EppRequestMessage and EppResponseMessage and fixed style and other minor issues

* Style changes implemented as suggested by jianglai

* Added style fixes suggested by mindhog
2019-08-15 16:03:36 -04:00
Weimin Yu
86fefa9a03 Make postgres dependency runtime (#225)
* Make postgres dependency runtime

Labeling it 'compile' unnecessarily makes psql-dependency a
public contract.
2019-08-14 12:32:07 -04:00
Shicong Huang
d143cc83a0 Add Hibernate and PostgreSQL packages (#224) 2019-08-14 11:22:01 -04:00
Aman Sanger
df5f450435 Prober circular list (#218)
* Fixed changes suggested by CydeWeys

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Fixed javadoc for Marker Interfaced

* Modified Comments on ActionHandler

* Removed LocalAddress from Protocol

* Fixed Travis Build Issues

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Minor Style Fixes

* Updated build.gradle file

* Modified license header dates

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Initial Commit.

* Initial Commit.

* Deleted unfinished features. Added ActionHandler and its Unit Tests.

* Included prober subproject in settings.gradle

* Added Protocol Class and its Basic Unit Tests

* Added Protocol Class and its Basic Unit Tests

* Added Changes Suggested by jianglai

* Fixed Gitignore to take out AutoValue generated code

* Fixed Gitignore to take out AutoValue generated code

* Removed AutoValue java files

* Added gitignore within prober

* Removed all generated java

* Final Changes in .gitignore

* Final Changes in .gitignore

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed build.gradle changes requested

* Removed Files irrelevant to current pull request

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Fully Updated ActionHandler (missing updated JavaDoc)

* Added changed Protocol and both Inbound and Outbound Markers

* Removed AutoVaue ignore clause from .gitignore

* Removed AutoVaue ignore clause from .gitignore

* removed unneccessary dependencies in build.gradle

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Fixed javadoc for Marker Interfaced

* Modified Comments on ActionHandler

* Removed LocalAddress from Protocol

* Fixed Travis Build Issues

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Minor Style Fix

* Minor Style Fix

* Minor Style Fix

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* fixed build issues

* fixed build issues

* fixed build issues

* fixed build issues

* fixed build issues

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Minor Style Fixes

* Minor Style Fixes

* Minor Style Fixes

* Minor Style Fixes

* Minor Style Fixes

* Updated build.gradle file

* Updated build.gradle file

* Updated build.gradle file

* Updated build.gradle file

* Modified license header dates

* Modified license header dates

* Modified license header dates

* Modified license header dates

* Modified license header dates

* Updated WebWhois tests.

* Updated WebWhois tests.

* Updated WebWhois tests.

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Rebased to master

* Rebased to master

* Updated issues in rebasing

* Updated issues in rebasing

* Minor style change on prober/build.gradle

* Minor style change on prober/build.gradle

* Fixed warnings for java compilation

* Fixed warnings for java compilation

* Fixed files to pass all style tests

* Fixed files to pass all style tests

* Fixed files to pass all style tests

* Minor syle fixes after succesful rebase onto master

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Minor Style Fix

* Minor Style Fix

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* fixed build issues

* fixed build issues

* fixed build issues

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Minor Style Fixes

* Minor Style Fixes

* Updated build.gradle file

* Updated build.gradle file

* Updated build.gradle file

* Modified license header dates

* Modified license header dates

* Modified license header dates

* Updated WebWhois tests.

* Updated WebWhois tests.

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* Added circular linked list to utils

* Added circular linked list to utils

* Added circular linked list to utils

* Added circular linked list to utils

* Added circular linked list to utils

* License Header added

* License Header added

* License Header added

* License Header added

* License Header added

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Refactored probing sequence to be circular linked list iterator

* Refactored probing sequence to be circular linked list iterator

* Refactored probing sequence to be circular linked list iterator

* Refactored probing sequence to be circular linked list iterator

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Modified ProbingStep tests to reflect new ProbingStep structure.

* Modified ProbingStep tests to reflect new ProbingStep structure.

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Rebased to master

* ProbingStepTest modified to have fewer unnecessary helper methods

* ProbingStepTest modified to have fewer unnecessary helper methods

* Updated issues in rebasing

* Updated issues in rebasing

* Added missing license header to DefaultCircularLinkedListIterator

* Fixed max column length to be 100

* Fixed max column length to be 100

* Minor changes to pass style tests

* Successful rebase onto finished web-whois branch

* Removed need for TestTokens with Mockito mocks of Tokens

* Fixed style issues in DefaultCircularLinkedListIterator and AbstractCircularLinkedListIterator

* Modified CircularList according to changes suggested by jianglai.

* Merge branch 'master' into prober-circular-list

* Modified ProbingSequenceTest to not expect unnecessary NullPointerException

* ProbingSequence and tests modified to reflect addition of UnrecoverableStateException and restarts on failures

* Modified ProbingSequence and its tests to reflect action generation and calling being put in the same try catch block
2019-08-12 18:41:50 -04:00
gbrodman
89a44f176c Clean up token generation (#205)
* Clean up token generation

- Allow tokenLength of 0
- If specifying a token length of 0, throw an error if numTokens > 1

* Allow generation of 0-length strings

* Allow for --tokens option to generate specific tokens

* Revert String generators and disallow 0 'length' param

* Add verifyInput method and batch the listed tokens

* Check the number of tokens created
2019-08-12 17:41:29 -04:00
gbrodman
d2319b13fa Bucket RDAP metrics by type (#220)
* Bucket RDAP metrics by type

* Rename method

* Use Guava Splitter and a joining Collector

* Use Iterables.limit

* Whoops, extra paren

* Checkstyle
2019-08-12 11:11:48 -04:00
gbrodman
77259f368d Fix checkstyle in Prober (#221) 2019-08-09 14:10:36 -04:00
Aman Sanger
27b81ba898 Add full prober WebWHOIS sequence functionality (#180)
* Initial Commit.

* Deleted unfinished features. Added ActionHandler and its Unit Tests.

* Included prober subproject in settings.gradle

* Added Protocol Class and its Basic Unit Tests

* Added Changes Suggested by jianglai

* Fixed Gitignore to take out AutoValue generated code

* Removed AutoValue java files

* Added gitignore within prober

* Removed all generated java

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed build.gradle changes requested

* Removed Files irrelevant to current pull request

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Fully Updated ActionHandler (missing updated JavaDoc)

* Added changed Protocol and both Inbound and Outbound Markers

* Removed AutoVaue ignore clause from .gitignore

* removed unneccessary dependencies in build.gradle

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Fixed javadoc for Marker Interfaced

* Modified Comments on ActionHandler

* Removed LocalAddress from Protocol

* Fixed Travis Build Issues

* Rebased to Master and added in modified Handlers and ProbingAction

* Fixed changes suggested by CydeWeys

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Initial Commit.

* Deleted unfinished features. Added ActionHandler and its Unit Tests.

* Included prober subproject in settings.gradle

* Added Protocol Class and its Basic Unit Tests

* Added Changes Suggested by jianglai

* Fixed Gitignore to take out AutoValue generated code

* Removed AutoValue java files

* Added gitignore within prober

* Removed all generated java

* Final Changes in .gitignore

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed build.gradle changes requested

* Removed Files irrelevant to current pull request

* Fixed changes suggested by CydeWeys

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Fully Updated ActionHandler (missing updated JavaDoc)

* Added changed Protocol and both Inbound and Outbound Markers

* Removed AutoVaue ignore clause from .gitignore

* removed unneccessary dependencies in build.gradle

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Full WebWhoIs Sequence Added

* fixed build issues

* Refactored by responses suggested by jianglai.

* Minor Style Fixes

* Minor Style Fixes

* Updated build.gradle file

* Updated build.gradle file

* Modified license header dates

* Modified license header dates

* Updated WebWhois tests.

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Rebased to master

* Updated issues in rebasing

* Minor style change on prober/build.gradle

* Fixed warnings for java compilation

* Fixed files to pass all style tests

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Initial Commit.

* Deleted unfinished features. Added ActionHandler and its Unit Tests.

* Included prober subproject in settings.gradle

* Added Protocol Class and its Basic Unit Tests

* Added Changes Suggested by jianglai

* Fixed Gitignore to take out AutoValue generated code

* Removed AutoValue java files

* Added gitignore within prober

* Removed all generated java

* Final Changes in .gitignore

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed build.gradle changes requested

* Removed Files irrelevant to current pull request

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Fixed changes suggested by CydeWeys

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Fully Updated ActionHandler (missing updated JavaDoc)

* Added changed Protocol and both Inbound and Outbound Markers

* Removed AutoVaue ignore clause from .gitignore

* removed unneccessary dependencies in build.gradle

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Fixed javadoc for Marker Interfaced

* Fixed javadoc for Marker Interfaced

* Modified Comments on ActionHandler

* Modified Comments on ActionHandler

* Removed LocalAddress from Protocol

* Removed LocalAddress from Protocol

* Fixed Travis Build Issues

* Fixed Travis Build Issues

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Minor Style Fix

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* fixed build issues

* fixed build issues

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Minor Style Fixes

* Minor Style Fixes

* Updated build.gradle file

* Updated build.gradle file

* Modified license header dates

* Modified license header dates

* Updated WebWhois tests.

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Rebased to master

* Updated issues in rebasing

* Minor style change on prober/build.gradle

* Fixed warnings for java compilation

* Fixed files to pass all style tests

* Fixed changes suggested by CydeWeys

* Rebased to Master and added in modified Handlers and ProbingAction

* Added missing license headers and JavaDoc

* Added missing license headers and JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor fix in NewChannelAction JavaDoc

* Minor Style Fix

* Minor Style Fix

* Full WebWhoIs Sequence Added

* Full WebWhoIs Sequence Added

* fixed build issues

* fixed build issues

* Refactored by responses suggested by jianglai.

* Refactored by responses suggested by jianglai.

* Minor Style Fixes

* Minor Style Fixes

* Updated build.gradle file

* Updated build.gradle file

* Modified license header dates

* Modified license header dates

* Updated WebWhois tests.

* Updated WebWhois tests.

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* Refactored WebWhois to accomodate jianglai's suggested changes and modified tests to reflect this refactoring

* SpotlessApply run to fix style issues

* SpotlessApply run to fix style issues

* Added license header and newline where appropriate.

* Added license header and newline where appropriate.

* Javadoc style fix in tests and removed unused methods

* Javadoc style fix in tests and removed unused methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Refactored ProbingAction to minimize number of unnecessary methods

* Modified tests for WebWhois according to changes suggested by laijiang.

* Modified tests for WebWhois according to changes suggested by laijiang.

* Removed TestProvider from TestUtils.

* Removed TestProvider from TestUtils.

* Rebased to master

* Updated issues in rebasing

* Minor style change on prober/build.gradle

* Fixed warnings for java compilation

* Fixed files to pass all style tests

* Minor syle fixes after succesful rebase onto master
2019-08-08 15:17:41 -04:00
Lai Jiang
c7478fc52b Fix expired access token bug in the proxy (#217)
https://github.com/google/nomulus/pull/129 migrated `GoogleCredential`
to `GoogleCredentialsBundle` and introduced a subtle bug. I don't fully
understand why but there are times when the access token is null but
`credentials.refresh()` is not called, resulting in NullPointerException
when `credentials.getAccessToken().getTokenValue()` is called.

Since the new GoogleCredentials class supports `shouldRefresh()`, we now
just rely on it to make sure that we always get a value access token.
2019-08-08 14:44:33 -04:00
Shicong Huang
a68b1a12fd Implement TransactionManager for datastore (#207)
This PR created the new interface named TransactionManager which defines
methods to manage transaction. Also, the access to all transaction related
methods of Ofy.java are restricted to package private, and they will be exposed
by DatastoreTransactionManager which is the datastore implementation of
TransactionManager.
2019-08-07 12:59:08 -04:00
Weimin Yu
ce791e8b97 Generate code coverage report (#216)
* Generate code coverage report

Enable jacoco, the official Gradle code coverage plugin.

The 'build' task will write a code coverage report to
build/reports/jacoco for each subproject that has tests.
We should consider publish periodical reports to a well known
location.

This change also defines a minimum coverage verification task.
The task is for experiment only, and is not added to the build
process yet.
2019-08-07 11:21:04 -04:00
gbrodman
cf3f9603f7 Clean up RDAP action logs and request plain text (#213)
* Remove unused log argument

* Use the right accept-encoding

By default we request gzip and theoretically we'd decode it
automatically on our end but for some reason that's not working. I
tested this on Alpha and it worked
2019-08-06 14:00:09 -04:00
Lai Jiang
4fd32c8b5c Add a Gradle task to deploy the proxy (#214)
Also refactored some common code regarding the mapping between
environments and projects.

The script to update proxy deployments and kill all pods are lifted from
https://cs.corp.google.com/piper///depot/google3/domain/registry/tools/bashrc?l=163
2019-08-06 11:45:34 -04:00
gbrodman
3488f95990 Clean up Gradle stylings and fix issues IDed in Prober (#212) 2019-08-05 15:54:20 -04:00
Shicong Huang
8e00f85f60 Use OpenJDK8 for Gradle build (#210) 2019-08-02 17:00:35 -04:00
Shicong Huang
3cc8d83396 Hard code webdriver docker version (#208) 2019-08-01 14:02:57 -04:00
gbrodman
0a779612f6 Remove old EPP processing time metrics (#206) 2019-07-31 19:18:40 -04:00
gbrodman
545a03618b Add an underline when hovering/focusing on <a> tags (#193) 2019-07-31 17:36:53 -04:00
Ben McIlwain
bcdacc88d3 Remove vestigial path for streaming EPP metrics (#184)
* Remove vestigial path for streaming EPP metrics

The relevant action was deleted last year here: google@218c451

This removes the final hanging piece.
2019-07-29 14:37:04 -04:00
gbrodman
56b10ea136 Create a Gradle task to run the test server (#192)
* Create a Gradle task to run the test server

As an artifact of the old build system, the test server relies on having
the built registrar_(bin|dbg)*(\.css)?.js in place (see ConsoleUiAction
among others). As a result, we create a Gradle task that puts those
files into the correct, readable, location before running the test
server.

* Depend on assemble rather than build

* refactor gitignores
2019-07-29 11:03:29 -04:00
Ben McIlwain
9479e1e8b9 Rename Spec11 reply-to email and also use it as sending address (#201)
* Rename Spec11 reply-to email and also use it as sending address
2019-07-26 15:30:46 -04:00
Shicong Huang
110bd9c057 Fix deploying to alpha from workstation (#198)
appengineDeployAll requires appengine.deploy.version to be set
otherwiese the deployment would fail.
2019-07-26 14:37:49 -04:00
Lai Jiang
2ae5aba7ff Rename dependency_license.gradle file (#196)
The file name confused the licensee app
(https://github.com/licensee/licensee) that GitHub uses to detect
license files. It thinks this file is also a license file and was not
able to determine is type.
2019-07-26 14:19:08 -04:00
Lai Jiang
b878e5acc1 Remove port to protocol map (#200)
There's no need for it. We can get the port from the protocol.
2019-07-25 22:05:12 -04:00
gbrodman
4d0409c924 Add HTML lang tags where possible (#199) 2019-07-25 21:09:32 -04:00
gbrodman
0292887cb9 Store only interesting failures, not login failures (#188)
Login failures will happen any time that we aren't coming from a
whitelisted IP for that particular TLD. Since whitelists are out of date
(and we don't whitelist IPs for every TLD anyway) those failures aren't
interesting. Store and fully-log the interesting failures if one
happened.
2019-07-24 10:55:55 -04:00
Lai Jiang
464f6ba90a Add a missing space (#197) 2019-07-24 10:50:27 -04:00
Lai Jiang
0ab0a8c2e6 Use positive booleans to improve readability (#190)
This allows us to use lgtm.com to scan for vulnerabilities.
2019-07-23 22:30:46 -04:00
Lai Jiang
2cc4e5fa94 Add a badge for LGTM analysis results (#194) 2019-07-23 21:42:19 -04:00
gbrodman
47a890253e Revert "Use parallel Gradle builds by default (#189)" (#195)
This reverts commit 5dfd96d26d.
2019-07-23 18:12:04 -04:00
Lai Jiang
b452b0628d Add customized .lgtm.yml (#191)
* Add customized .lgtm.yml
2019-07-23 17:09:05 -04:00
gbrodman
5dfd96d26d Use parallel Gradle builds by default (#189) 2019-07-23 15:27:30 -04:00
Shicong Huang
e2a673d914 Bring back the old GoogleCredential for Drive API (#187)
Using the new GoogleCredentials to access Drive API caused 403 forbidden
exception. So, this PR brought back the old GoogleCredential to
temporarily resolve the production issue while we are figuring out the
long term fix.

TESTED=Deployed to alpha and verified exportPremiumTerms succeeded, see
https://paste.googleplex.com/6153215760400384.
2019-07-23 11:31:35 -04:00
gbrodman
bf29d159f9 Fix a few deprecations (#186) 2019-07-22 14:12:55 -04:00
Lai Jiang
e17cb52bf7 Fail gracefully when copying detailed reports (#181)
* Fail gracefully when copying detailed reports

When the detailed reports are copied from GCS to registrars' Drive
folders, do not fail the entire copy operation when a single registrar
fails. Instead, send an alert email about the failure, and continue to copy the
rest of the reports.

Also, instead of creating duplicates, overwrite the existing files on
Drive.

BUG=127690361
2019-07-22 14:09:49 -04:00
Lai Jiang
7352f9b4a6 Remove unused local variable (#185) 2019-07-22 10:04:16 -04:00
Lai Jiang
5da48184f9 Merge beam and GAE configs deployment to one GCB job (#182)
* Merge beam and GAE configs deployment to one GCB job

Deployment of GAE configs requires that the credential used by gcloud to
have GAE admin role of the project to be managed. We do not want to
grant the GCB service account that role, because it would all *any* GCB
job to deploy anything to GAE. Instead we use a dedicated credential
originally created to deploy beam pipelines. This credential is
encrypted by KMS and stored on GCS. Since the beam pipeline deployment
GCB job already does the decryption, it make sense to add the config
deployment step there as well. The beam deployment steps are tweaked to
use the nomulus tool docker image instead of the jar file.

Also moved the content of deploy_configs_to_env.sh to the GCB yaml file
itself because the shell script is not uploaded to GC Bat the same time
as the yaml file when the job is triggered by Spinnaker.

Lastly, due to b/137891685, using GCB to deploy cron jobs does not work
as we cannot use service account credential to deploy to projects under
google.com.
2019-07-19 16:54:56 -04:00
gbrodman
5bd2ccd210 Add a Cloud Build task to update YAML configs (#177)
* Add a Cloud Build task to update YAML configs

* CR responses

* Move config deployment to a script

* Pin builder version

* Create different beam and deploy-config files per environment

* Update comments and make a for loop
2019-07-18 12:15:15 -04:00
Lai Jiang
8fd5ab2bec Build proxy image in Gradle (#179) 2019-07-17 20:38:03 -04:00
Lai Jiang
30f6113b05 Upgrade to Gradle 5.5.1 (#178)
Also make the default wrapper type "all" instead of "binary". This is
helpful for IDEs to understand the gradle script.
2019-07-17 17:37:44 -04:00
Michael Muller
bd48041961 Build docker image of nomulus tool (#142)
* Build docker image of nomulus tool

In the course of "gradle build", build a docker image of nomulus tool so that
users can run this to allow us to bundle the java version with the image.
2019-07-16 20:18:44 -04:00
gbrodman
39ceda628c Don't extend expiration times for deleted domains (#160)
* Don't extend expiration times for deleted domains

* Flip order and add a comment

* oops forgot a period

* Use END_OF_TIME

* Add tests for expiration times of domains with pending transfers

* Add test for transfer during autorenew and clean up other tests

* Clarify tests

* Add domain expiration check in EppLifecycleDomainTest

* Add a comment and format test files
2019-07-16 18:34:21 -04:00
Aman Sanger
0e9b75e5e9 Created Prober subproject and setup basic ActionHandler and its unit tests (#133)
* Initial Commit.

* Deleted unfinished features. Added ActionHandler and its Unit Tests.

* Included prober subproject in settings.gradle

* Added Protocol Class and its Basic Unit Tests

* Added Changes Suggested by jianglai

* Fixed Gitignore to take out AutoValue generated code

* Removed AutoValue java files

* Added gitignore within prober

* Removed all generated java

* Final Changes in .gitignore

* Added Ssl and WebWhois Action Handlers and their unit tests in addition to the ProbingAction class

* Fixed build.gradle changes requested

* Removed Files irrelevant to current pull request

* Minor fixes to ActionHandler, as responded in comments, removed package-info, and updated settings.gradle

* Fully Updated ActionHandler (missing updated JavaDoc)

* Added changed Protocol and both Inbound and Outbound Markers

* Removed AutoVaue ignore clause from .gitignore

* removed unneccessary dependencies in build.gradle

* Fixed Javadoc and comments for ActionHandler

* Fixed comments and JavaDoc on other files

* EOL added

* Removed Unnecessary Files

* fixed .gradle files styles

* Merge remote-tracking branch 'upstream/master'

* Removed outbound message from ActionHandler's fields and renamed Marker Interfaces

* Fixed javadoc for Marker Interfaced

* Modified Comments on ActionHandler

* Removed LocalAddress from Protocol

* Fixed Travis Build Issues
2019-07-16 10:35:14 -04:00
Lai Jiang
c1207464d8 Fix a typo (#174) 2019-07-15 17:49:22 -04:00
Weimin Yu
62eab98921 Unused deps check (#171)
* Save for later

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.

* Check for unused dependencies

Add a task to check for unused dependencies in
dependencies.gradle. This file should only contain
dependencies explicitly added to a configuration
in root project or some subprojects.
2019-07-15 16:19:34 -04:00
Shicong Huang
633dd887f4 Fix permission issue in Beam pipeline deployment (#170) 2019-07-15 16:13:42 -04:00
Lai Jiang
650f1fdd52 Bump lodash from 4.17.11 to 4.17.14 (#173) 2019-07-15 11:26:12 -04:00
gbrodman
b0f6a498fd Bump the version of the AppEngine Gradle plugin (#169)
* Bump the version of the AppEngine Gradle plugin

* deploy should depend on deployAll
2019-07-15 11:25:23 -04:00
gbrodman
77590dcd8e Add a metric for EPP processing time regardless of ID/TLD (#163)
* Add a metric for EPP processing time regardless of ID/TLD

* Change name to request_time

* Record EPP processing time by traffic type

* grammar

* request type

* semicolon
2019-07-11 14:28:37 -04:00
Weimin Yu
29e4d8de05 Check dependencies' open-source licenses (#165)
* Check dependencies' open-source licenses

Use jk1/Gradle-License-Report to verify that all
dependencies have open-source licenses.

Note that the following dependencies are not checked:

Dependencies of official Gradle plugins
Dependencies declared in buildscript block
Dependencies of jk1/Gradle-License-Report itself
2019-07-11 12:08:08 -04:00
Weimin Yu
242a560f20 Remove the maybeRuntime configuration (#164)
* Remove the maybeRuntime configuration

It contains dependencies present in the bazel
build but not needed for compile. We now know
they are not needed in runtime either.
2019-07-10 16:56:53 -04:00
gbrodman
3903abd9de Move and/or delete a bunch of random resources (#152) 2019-07-09 17:38:24 -04:00
Shicong Huang
8371cb838c Add a 30s timeout for all webdriver tests (#161)
Sometimes, the webdriver tests get stuck forever for no reason. It could
be some issue in the test container but it is hard to root cause it. So,
adding a 30s timeout can either trigger the retry earlier or let the
test just fail.
2019-07-09 14:42:32 -04:00
Shicong Huang
8dd6797614 Read golden images from src directly (#159)
This PR prevents Gradle from copying the golden images
to build/resources/test, so the screenshot test would
read golden images from src/test/resources directly and
display the path in test log if the test fails. Because
the path pointing to the actual file in src/ folder,
the engineer can easily find it.
2019-07-09 10:49:20 -04:00
Lai Jiang
730f108e13 Upgrade to Gradle 5.5 (#158) 2019-07-08 12:56:43 -04:00
gbrodman
e5bafddd2f Move JS and CSS files to a Javascript source dir (#156) 2019-07-05 12:01:16 -04:00
Lai Jiang
82f51accbd Set beam deployment environment in GCB trigger (#157) 2019-07-03 16:28:28 -04:00
Shicong Huang
6536631857 Remove injected credentials from invoice pipeline (#155)
We got non-serialization object error when deploying the invoicing
pipeline. It turns out that Beam requires every field in the pipeline
object is serilizable. However, it is non-trivial to make
GoogleCredentialsBundle serilizable because almost all of its
dependency are not serilizable and not contraled by us. Also,
it is non-necessary to inject the credential as the spec11
pipeline also writes output to GCS without having injected
credential. So, removing the injected variable can solve the
problem.

TESTED=First reproduced the problem locally by deploying the invoicing pipeline with the previous code; applied this change and successfully deploy the pipeline without having any issue.
2019-07-03 15:12:48 -04:00
gbrodman
1be92968bf Attempt login to MosAPI via all available TLDs (#141)
* Attempt login to MosAPI via all available TLDs

There's no reason why we should need a TLD as input here because it
doesn't actually matter which one we use (they all have the same
password).

* Refactor the TLD loop and change cron jobs

* Re-throw the last exception if one exists

* Fix tests and exception

* Remove alpha cron job
2019-07-03 14:25:39 -04:00
Ben McIlwain
0564b207f2 Prevent accidentally using full Drive URL as folder ID (#144)
* Prevent accidentally using full Drive URL as folder ID
2019-07-03 14:22:15 -04:00
gbrodman
8afc4f4d3d Enforce that source files end in a newline (#153) 2019-07-03 12:16:11 -04:00
guyben13
888bc158fe Add explanation on how to use the MoSAPI endpoint (#137)
* Add explanation on how to use the MoSAPI endpoint

* Add a $ before each command so that it's clearer
2019-07-02 18:20:53 -04:00
Lai Jiang
25ee92b5d8 Re-add hello.xml file (#147)
* Re-add hello.xml file

Apparently it was lost during the great refactoring.

* Remove hello.xml from test resources
2019-07-02 17:21:40 -04:00
gbrodman
cf507dad6d Move test resource files into src/test/resources (#143)
* Move test resource files into src/test/resources

* fix a test

* Remove references to javatests/ in Java files

* fix import order

* fix semantic merge conflict
2019-07-02 16:54:49 -04:00
gbrodman
32d5940be3 Remove old Bazel files (#151) 2019-07-02 16:24:01 -04:00
Shicong Huang
82fa3d7349 Change button color to blue style (#150) 2019-07-02 15:29:17 -04:00
Ben McIlwain
07239710ef Throw a more useful error message on attempted domain restore reports (#145)
* Throw a more useful error message on attempted domain restore reports

Per DomainRestoreRequestFlow's Javadoc, we automatically approve and instantly
enact all domain restore requests, thus we don't use or support restore
reports. This improves the registrar-visible error message to help make this
more clear.
2019-07-02 14:11:37 -04:00
gbrodman
0e8724a48f Require the license in Gradle files (#149) 2019-07-02 11:47:35 -04:00
gbrodman
8332664af9 Move golden test files to src/test/resources (#146) 2019-07-02 11:34:41 -04:00
gbrodman
39abee6279 Update checkstyle.xml with many presubmit checks (#130)
* Update checkstyle.xml with many presubmit checks

* Remove a comment
2019-07-02 10:35:32 -04:00
Shicong Huang
6daf72a54e Replace deprecated GoogleCredential with new auth lib (#129)
Replace deprecated GoogleCredential with new lib

This PR also introduced a CredentialsBundle class to carry
HttpTransport and JsonFactory object which are needed by
most of the GCP library to instantiate client.
2019-07-02 10:29:51 -04:00
gbrodman
7ce37147c0 Refactor the license check to be a generic presubmit check (#131)
* Refactor the license check to be a generic presubmit check

This includes all of the old presubmits that were in our METADATA file
that are still valid.

* Responses to CR

Added docstrings, refactored the file finder, changed variable names,
and formatted the file

* More respones to CR
2019-06-27 10:08:27 -04:00
Shicong Huang
661e348fcb Stop provisioning RegistryEnvironment from Dagger (#140)
We found that some webdriver tests failed because RegistryEnvironment
was set to 'production' by other test and was carried over to the
webdriver test, and it was nontrivial to fix them because the instance
of RegistryEnvironment was injected into the testing web server.

Also, to eliminate this problem from potential victims, we stopped
provisioning RegistryEnvironment from Dagger. Instead, we just use
RegistryEnvironment.get() function to get the instance, which indeed
retrives the value from system property every time. If any test case
needs to run the test with other environment, it can just simply use
the existing SystemPropertyRule and RegistryEnvironment.${ENV}.setup
to change it.
2019-06-26 16:57:19 -04:00
Lai Jiang
33a7808e9e Disable Javadoc and tests in Java 12 (#135)
The API is deprecated and removed in Java 12. Do not build these
classes when using Java 12, till we migrate to the new API.
2019-06-25 22:19:36 -04:00
gbrodman
16b0c420a2 Use strict versions in NPM versioning (#132)
It'll reduce surprise version changes
2019-06-25 16:10:04 -04:00
Lai Jiang
b7f3154f4e Update GCB beam deployment pipeline (#134)
* Update GCB beam deployment pipeline

Some of the texts are not really secerts because they are per-project.
Also changed the location of the credential file to `secerts` so that in
the future we may add more secerts in that folder.

The encrypted file is base64 encoded, consistent with how the proxy
certificates are encoded. Also made some changes to the other pipelines
to facilitate automation with Spinnaker
2019-06-24 14:36:56 -04:00
Lai Jiang
56105105b5 Exclude Visual Studio Code project files (#136)
* Exclude Visual Studio Code project files
2019-06-24 11:39:48 -04:00
Lai Jiang
368c264e3c Update IAM binding to restrict proxy service account's access to GCS (#125)
Per
https://cloud.google.com/container-registry/docs/access-control#granting_users_and_other_projects_access_to_a_registry,
for a service account to access GCR, it does not need reader access to *all*
buckets in a project, but just the specific one.

This is duped from cl/254092941.
2019-06-21 15:59:01 -04:00
Lai Jiang
21be4b8227 Update README.md (#128) 2019-06-21 15:24:17 -04:00
gbrodman
3a485a03a6 Add a license-checking Python script (#123)
* Add a license-checking Python script

* Responses to CR
2019-06-21 12:25:35 -04:00
gbrodman
00147c1bc4 Install Chrome in the Cloud Build process (#124)
* Install Chrome in the Cloud Build process

It's not in the built-in Linux apt repos so we need to manually wget it

* Update package lock file
2019-06-20 18:19:23 -04:00
Lai Jiang
520145f641 Move terraform and kubernetes folder to be under proxy (#127)
* Move terraform and kubernetes folder to be under proxy

There is no reason for them to be under proxy/src/... any more now that
we have a Gradle-idiomatic folder structure.
2019-06-20 14:28:32 -04:00
Lai Jiang
9cd7c9402f Always clone the internal repo to nomulus-internal
Also updated .gcloudignore to not pull in unnecessary files when running
`gcloud builds submit`.
2019-06-20 14:26:36 -04:00
jianglai
6d5777559f Put environment check in its separate task
The check for environment cannot be done when the task is defined,
otherwise running any task will fail when the check does not pass.
Instead, check the environment when the task executes.
2019-06-19 18:46:46 -04:00
jianglai
2a04e1adb4 Ask the gradle wrapper to download source as well
This helps IntelliJ display documentation when editing the `.gradle`
files.
2019-06-19 18:46:46 -04:00
jianglai
e7b00dc4ee Only allow commandine deployment to alpha 2019-06-19 18:46:46 -04:00
jianglai
0b52b87d92 Add a newline 2019-06-19 18:46:46 -04:00
jianglai
06c2456f45 Set default environment to production
This way when "gradle build" runs it will test if we can successfully
stage the produciton GAE files.
2019-06-19 18:46:46 -04:00
Lai Jiang
c56432ea09 Add tasks used in release as dependencies of the build task
Our CI (Travis & Kokoro) runs "gradle build", so we need to make
sure that all tasks used in the release process are called during
the build so that breakage can be caught earlier.

In order to stage the GAE folder we need gcloud to be present.
Therefore the Travis config is changed to install gcloud.

See: https://gist.github.com/mjackson/5887963e7d8b8fb0615416c510ae8857
2019-06-19 18:46:46 -04:00
Ben McIlwain
c50f16853b Don't use @RunWith JUnit annotations on classes with no @Tests
IntelliJ is complaining when this annotation is used on a base class that has no
actual runnable tests itself. The solution is different for different classes
depending on the existing pattern of where the @RunWith annotation is; for some,
it's simplest to move the annotation down to the few extending classes that are
missing it, whereas in others it's easiest just to annotate the base class as
abstract.
2019-06-19 17:43:01 -04:00
gbrodman
d3f0d34e2b Use Gerrit / Googlesource version of nomulus-internal (#95)
* Use Gerrit / Googlesource version of nomulus-internal

* Remove tag deletion and use a variable substitution
2019-06-19 17:33:23 -04:00
gbrodman
e1db339b88 Add Spotless to the Gradle build (#88)
* Add Spotless to the Gradle build

* spotless apply

* one dot instead of two
2019-06-19 16:24:46 -04:00
gbrodman
973b29bc7a Move the Soy files to the resources folder (#120) 2019-06-19 12:10:57 -04:00
Weimin Yu
efe33fc66e Merge pull request #121 from weiminyu/deprecate
Fix or suppress some deprecation warnings
2019-06-19 11:30:02 -04:00
Shicong Huang
c6f2448d63 Merge pull request #122 from hstonec/upgrade-google-auth-lib
Upgrade google-auth-library-java to latest version
2019-06-18 17:54:24 -04:00
Shicong Huang
0630b371df Upgrade google-auth-library-java to latest version
This is a preparation for replacing the deprecated GoogleCredential
library.
2019-06-18 17:27:21 -04:00
Weimin Yu
e2a44b19ef Fix or suppress some deprecation warnings
Fix or suppress deprecation warnings except those about GoogleCredential,
which is being handled separately.

The @SuppressWarnings("deprecation") annotation does not cover imports
even when it is at the class level. We removed imports of deprecated
classes and use their fully qualified names in class body.
2019-06-18 17:23:45 -04:00
Lai Jiang
fd1c364a14 Remove unnecessary provider method (#113)
* Remove unnecessary provide method

This provider is not used. Instead, depending on if the login is performed on
[a remote machine or not](8bdc77501f/core/src/main/java/google/registry/tools/LoginCommand.java (L43)),
the instance is created directly in the command.

* Remove unused imports
2019-06-18 15:55:33 -04:00
Shicong Huang
3df11e7849 Merge pull request #105 from hstonec/deploy-beam-from-gcb
Add a CloudBuild task to deploy Beam pipeline
2019-06-18 15:34:28 -04:00
Shicong Huang
6ca5cab66f Consolidate testcontainer used by WebDriver test (#112)
* Consolidate testcontainer used by WebDriver test

Previously, we hard coded the version of the docker image for
provisioning Chrome browser and WebDriver server because the
version of the Chrome browser has to match the version of
the webdriver client, otherwise the screenshot test will fail.

Changing to use BrowserWebDriverContainer can delegate the match
to the library itself because it chooses the correct docker image
version based on the WebDriver version on our classpath.

* Increase maxColorDiff to 20

This is to supress the test flakness after switching to use
BrowserWebDriverContainer to provision browser and webdriver
service.
2019-06-18 11:51:55 -04:00
gbrodman
c7ca9e7d09 Bump karma-related dependencies (#119)
I'm seeing odd errors in versions of Karma more recent than 3.0.0. While
I investiage, I'd like to see what Github has to say about these
dependencies (we got some vulnerability messages for earlier versions)
2019-06-18 11:46:18 -04:00
gbrodman
0e3b37cca5 Merge pull request #87 from google/gb/karma-testing
Move the Javascript tests to use Jasmine/Karma
2019-06-18 11:09:29 -04:00
Shicong Huang
15622c2f84 Add a CloudBuild task to deploy Beam pipeline 2019-06-18 10:48:41 -04:00
Gus Brodman
a2236ae6f9 Default for 'served' is already true 2019-06-18 10:01:47 -04:00
Weimin Yu
a090ded769 Merge pull request #116 from weiminyu/no_werror_again
Fix merge error
2019-06-17 15:02:41 -04:00
Weimin Yu
9ee60a8fd9 Fix merge error 2019-06-17 13:28:47 -04:00
gbrodman
77e7ba9488 Merge pull request #97 from gbrodman/spec11DNS
Only inspect DNS-published domains in Spec11
2019-06-17 12:16:24 -04:00
gbrodman
4dd442a87c Merge pull request #111 from gbrodman/idea
Apply IDEA plugin
2019-06-17 12:14:27 -04:00
Gus Brodman
cf48c643e9 Respond to CR 2019-06-17 12:04:43 -04:00
Weimin Yu
fb0715e3e2 Merge pull request #115 from weiminyu/disable_werror
Allow switching off the -Werror javac option
2019-06-17 11:53:31 -04:00
Gus Brodman
b644a9f4c9 Move the Javascript tests to use Jasmine/Karma 2019-06-17 11:44:52 -04:00
Weimin Yu
24c5c2ce20 Allow switching off the -Werror javac option
This option causes Intellij build to fail if the
'Delegate IDE build/run actions to gradle' box is checked.
We do not know of anyway to change the Intellij behavior.

This change allows an IDE user to prevent -Werror to be passed
to javac by adding a Gradle VM option: -Dno_werror=true
2019-06-17 11:26:50 -04:00
Wilder Pereira
e378855093 Update source code location (#114) 2019-06-17 09:23:43 -04:00
Gus Brodman
38a47ca7d0 Apply IDEA plugin 2019-06-14 17:19:58 -04:00
Lai Jiang
8bdc77501f Disable an Error Prone check incompatible with Java 12 (#100) 2019-06-14 15:36:32 -04:00
Gus Brodman
da89fc1268 Filter out non-published domains in Spec11 emails 2019-06-14 14:34:12 -04:00
gbrodman
0f6dfed830 Merge pull request #91 from google/gb/validateEmails
Validate provided email addresses when creating a registrar
2019-06-14 11:37:24 -04:00
gbrodman
01d7ef0fd5 Merge branch 'master' into gb/validateEmails 2019-06-14 10:48:57 -04:00
Lai Jiang
9859490f5f Add a link to kokoro (#101)
* Add a link to kokoro

This allows us to click on the build badge to visit kokoro and find
out build status. The link will redirect to a status page that is
only accessible to Googlers.
2019-06-14 10:48:35 -04:00
gbrodman
0d0d048cd0 Merge branch 'master' into gb/validateEmails 2019-06-14 10:44:23 -04:00
gbrodman
af5618ac3d Merge pull request #92 from google/gb/upgradeJCommander
Upgrade the JCommander dependency to 1.60
2019-06-13 18:19:07 -04:00
gbrodman
1e8b464c67 Merge branch 'master' into gb/upgradeJCommander 2019-06-13 18:18:55 -04:00
Lai Jiang
8c5fc6410b Remove Bazel build (#94)
* Remove bazel build

Also updated the build badge to reflect internal build status.
2019-06-13 18:15:33 -04:00
gbrodman
a7fcfca354 Merge branch 'master' into gb/upgradeJCommander 2019-06-13 18:05:28 -04:00
Gus Brodman
db932fe336 Add more tests in RegistrarTest 2019-06-13 18:04:57 -04:00
Lai Jiang
aca18a1d4a Fix builds after refactor (#99)
Fixed both GAE and proxy builds after #90 refactored the code structure.

Also removed now unnecessary chmod and chown from GCB scripts.
2019-06-13 18:01:30 -04:00
Lai Jiang
b74b3dbcb3 Merge pull request #98 from google/gbrodman-patch-1
Add shicong@ and gbrodman@ to CONTRIBUTORS
2019-06-13 16:56:20 -04:00
gbrodman
392cd3a31a Add shicong@ and gbrodman@ to CONTRIBUTORS 2019-06-13 16:07:31 -04:00
Gus Brodman
7cf0dce057 Validate provided email addresses when creating a registrar 2019-06-13 13:13:06 -04:00
Gus Brodman
f06975df75 Upgrade jcommander dep 2019-06-13 13:04:48 -04:00
gbrodman
8f2a8835d7 Merge pull request #90 from google/gb/refactorGradle
Refactor to be more in line with a standard Gradle project structure
2019-06-13 13:01:45 -04:00
Gus Brodman
0a34c5ba36 More moving of hello.xml files -- they're all the same 2019-06-13 11:52:04 -04:00
Gus Brodman
b883f86933 Move hello.xml and don't use project.ext.foo 2019-06-13 10:14:50 -04:00
Gus Brodman
a7a983bfed Refactor to be more in line with a standard Gradle project structure 2019-06-13 09:41:11 -04:00
Shicong Huang
8fa45e8c76 Merge pull request #96 from google/sc/fix-uber-jar
Use ShadowJar plugin to generate uber jar
2019-06-12 23:07:28 -04:00
Shicong Huang
46a58b7141 Use ShadowJar plugin to generate uber jar
Our existing uber jar task overwrote duplicate files when merging JARs,
and this would break the application using Java SPI service(Beam library
is one of them) so we changed to use ShadowJar plugin as it provides a
convenient way to merge META-INF/services/* files when generating the
uber jar.

TESTED=Deployed spec11 pipeline successfully with the new nomulus.jar;
also randomly checked a few nomulus commands to make sure they are still
working.
2019-06-12 19:48:57 -04:00
guyben
cb7ca611a9 Add action to sync all Registrar RDAP base URLs from the ICANN endpoint
We aren't turning on the cron job yet - we want to make sure it works correctly first.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252726789
2019-06-12 13:15:12 -04:00
mcilwain
95111809bd Add a nomulus tool command to bulk ACK poll messages
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252674014
2019-06-12 13:13:33 -04:00
weiminyu
d7da7f6da4 Replace deprecated CharMatcher method
CharMatcher.isLetterOrDigit() is deprecated for failing to support
supplementary characters. We explicitly declare a matcher for ascii
letters and digits.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252671830
2019-06-12 13:11:44 -04:00
guyben
00c7c0f1af Clarify that the supplied RDAP Base URLs might get overridden
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252661230
2019-06-12 13:09:58 -04:00
jianglai
b8ffa732b2 Add GCB workflows to promote the nomulus tool command after deployment
With https://github.com/spinnaker/spinnaker/issues/4048 Spinnaker now natively supports GCB. We are able to start a GCB job from Spinnaker, and also there is better support to consume GCB pub/sub messages. Some changes are made to remove the workaround no longer needed.

Two new workflows are added, one to rsync a GCS folder to live/ after the deployment is done (so that the nomulus.jar file can then be fetched to x20 by a []cron job), and the other to tag the proxy image as live once it is deployed.

Lastly, the docs/ folders are needed when running tests. Remove it from .gcloudignore so that when a test run is kicked off by running "gcloud builds submit" the folder is sent to GCB. Ideally .gcloudignore should be identical to .gitignore but since they both are version controlled it is hard it make one a symlink of another.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252625901
2019-06-12 13:08:11 -04:00
gbrodman
deb826794d Use a 'number' input type for number fields in registrar creation
No tests for this since we don't have any to begin with

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252508756
2019-06-12 13:06:29 -04:00
guyben
4110cae814 Link the Registrar's RDAP server from RDAP domain replies
To do this we add a field of "rdapServers" in the Registrar object. Currently, we can only set this field manually, but a subsequent CL will add a cron-job to read these values from the ICANN servers.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=252438618
2019-06-12 13:04:40 -04:00
weiminyu
7c64992c73 Cloned from CL 251456914 by 'g4 patch'.
Original change by cpovirk@cpovirk:rosie251284456-0055_Rosie:31511:citc on 2019/06/04 09:48:27.

Update to Truth 0.45, and address deprecations.

Renames may include:
- containsAllOf => containsAtLeast
- containsAllIn => containsAtLeastElementsIn
- isSameAs => isSameInstanceAs
- isOrdered => isInOrder
- isStrictlyOrdered => isInStrictOrder

The other major change is to change custom subjects to extend raw Subject instead of supplying type parameters. The type parameters are being removed from Subject. This CL will temporarily produce rawtypes warnings, which will go away when I remove the type parameters (as soon as this batch of CLs is submitted).

Some CLs in this batch also migrate calls away from actualAsString(). Its literal replacement is `"<" + actual + ">"` (unless an object overrides actualCustomStringRepresentation()), but usually I've made a larger change, such as switching from an old-style "Not true that..." failure message to one generated with the Fact API. In that case, the new code usually contains a direct reference to this.actual (a field that I occasionally had to create). Another larger change I sometimes made is to switch from a manual check-and-fail approach to instead use check(...). And sometimes I just remove a withMessage() call that's no longer necessary now that the code uses check(...), or I introduce a check(...) call. (An assertion made with check(...) automatically includes the actual value from the original subject, so there's no need to set it again with withMessage().)

Finally, there's one CL in this batch in which I migrate a Correspondence subclass to instead use Correspondence.from.

END_PUBLIC

If this is too many changes at once, let me know, and I can split it up and/or explain further. In the past, I've erred on the side of sending separate CLs for each change, which has required some owners to manually reapply each one, so now I'm trying this way.

Thanks again for your patience. There is an outside chance that this will be the last CL I send you before Truth 1.0 -- but certainly no promises :)

More information:
  Renames:
    []
    []
    []
  Removing type parameters: []
  Migration from old fail*(...) methods to new ones and to check(...): []
  Changes that replace assert*(...) with introduce check(...): []
  Correspondence subclass to Correspondence.from: []

Tested:
    TAP --sample ran all affected tests and none failed
    []

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251921007
2019-06-07 11:46:44 -04:00
jianglai
e8c789a401 Fix builder error
We need to first populate the apt sources before installing anything. It was working on my machine because the layer was cached. After I purged all images I was able to replicate the same error message.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251918035
2019-06-07 11:44:57 -04:00
guyben
ca1d525e28 Conform to RDAP Technical Implementation Guide
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251864499
2019-06-07 11:43:09 -04:00
mcilwain
55dc735ba0 Export PollMessage entities to BigQuery
It would be nice to be able to query them using SQL in order to debug
some potential issues registrars might be facing.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=251863662
2019-06-07 11:41:22 -04:00
mcilwain
ff6d327183 Allow reserved domains to be created during quiet periods
We'll use this for LRP. This is safe because we must specifically reserve a
domain by including it in a reserved list, create an associated allocation
token, and distribute that token, before a create would succeed.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250901144
2019-06-06 10:53:27 -04:00
gbrodman
79bcb227be De-duplicate flow documentation error reasons on a per-code basis
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250894847
2019-06-06 10:51:43 -04:00
guyben
e489539df4 Enable more DNS update metrics on production
They are useful for viewing and diagnosing problems, and are not that
expensive.

We aren't enabling all the metrics though - the whole "per batch / per domain"
performance indicators are still disabled

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250876554
2019-06-06 10:49:55 -04:00
mcilwain
dd66664e42 Add tests that some domain operations work in quiet periods
This specifically includes info, transfer, and update. Note that normal
domain creates do not work in quiet periods and are not expected to.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250755940
2019-06-06 10:48:05 -04:00
mcilwain
7bdbd89e4d Add test that anchor tenants can be created in quiet period
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250752454
2019-06-06 10:46:14 -04:00
mcilwain
987a648296 Smear out DNS updates over time when updating DNS for all domains
These updates can generally ran slowly in the background (i.e. they're low
priority), and shouldn't clog up the system to the point where they slow down
live incoming customer-requested updates.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250732171
2019-06-06 10:44:27 -04:00
gbrodman
d2f955a488 Split ResourceAlreadyExistsException based on if this client owns the resource
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=250728711
2019-06-06 10:42:38 -04:00
3996 changed files with 109796 additions and 42834 deletions

View File

@@ -1,8 +1,9 @@
docs/**
python/**
bazel-*/**
gradle/**/build/**
gradle/**/WEB-INF/**
gradle/.*/**
python/
**/build/
**/out/
.*/
repos/**
**/.idea/**
**/.idea/
*.jar
!third_party/**/*.jar
!/gradle/wrapper/**/*.jar

34
.gitignore vendored
View File

@@ -74,6 +74,21 @@ local.properties
# Auto-generated java files
autogenerated/
# IDEA
nomulus.iml
nomulus.ipr
nomulus.iws
# Auto-generated java classes by Intellij
*/src/main/generated/
*/src/test/generated_tests/
# VScode
.vscode
# Javascript output
**/out/*
######################################################################
# Python Ignores
@@ -82,12 +97,17 @@ autogenerated/
######################################################################
# Gradle Ignores
# We don't want to ignore the gradle jar files
!/gradle/gradle/wrapper/**/*.jar
# We don't want to ignore the gradle or google-java-format jar files
!/gradle/wrapper/**/*.jar
!/java-format/*.jar
.gradle/
/gradle/.gradle
/gradle/**/WEB-INF
/gradle/**/build
/gradle/node_modules/**
!/gradle/node_modules/soyutils_usegoog.js
**/build
cloudbuild-caches/
node_modules/**
/repos/
# Compiled JS/CSS code
core/**/registrar_bin*.js
core/**/registrar_dbg*.js
core/**/registrar_bin*.css
core/**/registrar_dbg*.css

6
.lgtm.yml Normal file
View File

@@ -0,0 +1,6 @@
extraction:
java:
prepare:
packages: "npm"
index:
java_version: "11"

View File

@@ -1,58 +0,0 @@
# Copyright 2018 The Nomulus Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Collecting report files from a travis build to GCS requires setting the
# "CREDZ" and "REPORT_GCS_BUCKET" environment variables in your travis
# repository.
#
# The report destination can be any GCS path (e.g. "gcs://bucket-name/path").
# You'll want to select "Display value in build log", otherwise travis will
# hide the bucket name in the URL that is displayed.
#
# The CREDZ variable should be the contents of a json credentials file for
# a service account with write access to the bucket, escaped for bash shell
# usage (usually just wrapping it in single quotes should suffice).
language: java
install: true
jdk:
# Our builds fail against Oracle Java for reasons yet unknown.
- openjdk8
# Caching options suggested by a random article.
before_cache:
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock
- rm -f $HOME/.gradle/caches/*/plugin-resolution/
cache:
directories:
- $HOME/.gradle/caches/
- $HOME/.gradle/wrapper/
# WebDriver tests need Chrome and ChromeDriver provisioned by the docker image
services:
- docker
env:
# Disable fancy status information (looks bad on travis and exceeds logfile
# quota)
TERM=dumb
# Specialize gradle build to use an up-to-date gradle and the /gradle
# directory.
# The "travis_wait 45" lets our build spend up to 45 minutes without writing
# output, instead of the default 10.
# See notes on the CREDZ and REPORT_GCS_BUCKET environment variable in the
# comments at the top of the file.
script: cd gradle && echo "$CREDZ" >credz.json && chmod 755 ./gradlew && travis_wait 45 ./gradlew build --continue -P uploaderDestination="$REPORT_GCS_DESTINATION" -P uploaderCredentialsFile=credz.json -P uploaderMultithreadedUpload=yes -P mavenUrl=https://storage.googleapis.com/domain-registry-maven-repository/maven -P pluginsUrl=https://storage.googleapis.com/domain-registry-maven-repository/plugins

View File

@@ -32,3 +32,5 @@ Hans Ridder <hans.ridder@gmail.com>
Justin Graham <justin.af.graham@gmail.com>
Guy Bensky <guyben@google.com>
Weimin Yu <weiminyu@google.com>
Shicong Huang <shicong@google.com>
Gustav Brodman <gbrodman@google.com>

View File

@@ -1,4 +1,3 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/

View File

@@ -1,8 +1,8 @@
# Nomulus
| Bazel | Gradle |
|-------|--------|
|![Build Status](https://storage.googleapis.com/domain-registry-github-build-status/github-ci-status.png)|[![Build Status](https://travis-ci.org/google/nomulus.svg?branch=master)](https://travis-ci.org/google/nomulus)|
| Internal Build | FOSS Build | LGTM | License | Code Search |
|:--------------:|:----------:|:----:|:-------:|:-----------:|
|[![Build Status for Google Registry internal build](https://storage.googleapis.com/domain-registry-kokoro/internal/build.svg)](https://storage.googleapis.com/domain-registry-kokoro/internal/index.html)|[![Build Status for the open source build](https://storage.googleapis.com/domain-registry-kokoro/foss/build.svg)](https://storage.googleapis.com/domain-registry-kokoro/foss/index.html)|[![Total alerts](https://img.shields.io/lgtm/alerts/g/google/nomulus.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/google/nomulus/alerts/)|[![License for this repo](https://img.shields.io/github/license/google/nomulus.svg)](https://github.com/google/nomulus/blob/master/LICENSE)|[![Link to Code Search](https://www.gstatic.com/devopsconsole/images/oss/favicons/oss-32x32.png)](https://cs.opensource.google/nomulus/nomulus)|
![Nomulus logo](./nomulus-logo.png)
@@ -30,10 +30,10 @@ running system:
* [Install
guide](https://github.com/google/nomulus/blob/master/docs/install.md)
* View the source code in
[java/google/registry/](https://github.com/google/nomulus/tree/master/java/google/registry)
* View the source code for the [GAE app](https://github.com/google/nomulus/tree/master/core/src/main/java/google/registry)
and for the [GKE proxy](https://github.com/google/nomulus/tree/master/proxy/src/main/java/google/registry)
* [Other docs](https://github.com/google/nomulus/tree/master/docs)
* [Javadoc](https://nomulus.foo/javadoc/latest/)
* [Javadoc](https://javadoc.nomulus.foo/)
* [Nomulus discussion
group](https://groups.google.com/forum/#!forum/nomulus-discuss), for any
other questions

View File

@@ -1,56 +0,0 @@
workspace(name = "domain_registry")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
# https://github.com/bazelbuild/rules_closure/releases/tag/0.8.0
http_archive(
name = "io_bazel_rules_closure",
sha256 = "0e6de40666f2ebb2b30dc0339745a274d9999334a249b05a3b1f46462e489adf",
strip_prefix = "rules_closure-87d24b1df8b62405de8dd059cb604fd9d4b1e395",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/87d24b1df8b62405de8dd059cb604fd9d4b1e395.tar.gz",
"https://github.com/bazelbuild/rules_closure/archive/87d24b1df8b62405de8dd059cb604fd9d4b1e395.tar.gz",
],
)
load("@io_bazel_rules_closure//closure:defs.bzl", "closure_repositories")
closure_repositories(
omit_com_google_auto_factory = True,
omit_com_google_protobuf = True,
omit_com_google_code_findbugs_jsr305 = True,
omit_com_google_guava = True,
omit_com_ibm_icu_icu4j = True,
omit_javax_inject = True,
omit_org_json = True,
)
load("//java/google/registry:repositories.bzl", "domain_registry_repositories")
domain_registry_repositories()
# Setup docker bazel rules
http_archive(
name = "io_bazel_rules_docker",
sha256 = "aed1c249d4ec8f703edddf35cbe9dfaca0b5f5ea6e4cd9e83e99f3b0d1136c3d",
strip_prefix = "rules_docker-0.7.0",
urls = ["https://github.com/bazelbuild/rules_docker/archive/v0.7.0.tar.gz"],
)
load(
"@io_bazel_rules_docker//repositories:repositories.bzl",
container_repositories = "repositories",
)
container_repositories()
load(
"@io_bazel_rules_docker//container:container.bzl",
"container_pull",
)
container_pull(
name = "java_base",
registry = "gcr.io",
repository = "distroless/java",
# 'tag' is also supported, but digest is encouraged for reproducibility.
digest = "sha256:84a63da5da6aba0f021213872de21a4f9829e4bd2801aef051cf40b6f8952e68",
)

110
appengine_war.gradle Normal file
View File

@@ -0,0 +1,110 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
apply plugin: 'war'
def environment = rootProject.environment
def gcpProject = rootProject.gcpProject
// Set this directory before applying the appengine plugin so that the
// plugin will recognize this as an app-engine standard app (and also
// obtains the appengine-web.xml from the correct location)
project.convention.plugins['war'].webAppDirName =
"../../core/src/main/java/google/registry/env/${environment}/${project.name}"
apply plugin: 'com.google.cloud.tools.appengine'
def coreResourcesDir = "${rootDir}/core/build/resources/main"
// Get the web.xml file for the service.
war {
webInf {
from "../../core/src/main/java/google/registry/env/common/${project.name}/WEB-INF"
}
}
war {
from("${coreResourcesDir}/google/registry/ui/html") {
include "*.html"
}
}
if (project.path == ":services:default") {
war {
from("${coreResourcesDir}/google/registry/ui") {
include "registrar_bin.js"
if (environment != "production") {
include "registrar_bin.js.map"
}
into("assets/js")
}
from("${coreResourcesDir}/google/registry/ui/css") {
include "registrar*"
into("assets/css")
}
from("${coreResourcesDir}/google/registry/ui/assets/images") {
include "**/*"
into("assets/images")
}
}
}
appengine {
deploy {
// appengineDeployAll task requires the version to be set. So,
// this config lets gcloud select a version name when deploying
// to alpha or sandbox from our workstation.
if (!rootProject.prodOrSandboxEnv) {
version = 'GCLOUD_CONFIG'
}
// Don't set gcpProject directly, it gets overriden in ./build.gradle.
// Do -P environment={crash,alpha} instead. For sandbox/production,
// use Spinnaker.
projectId = gcpProject
}
}
dependencies {
compile project(path: ':core', configuration: 'deploy_jar')
}
// The tools.jar file gets pulled in from the java environment and for some
// reason gets exploded "readonly", causing subsequent builds to fail when
// they can't overwrite it. The hack below makes the file writable after
// we're done exploding it.
//
// Fun fact: We only use this jar for documentation generation and as such we
// don't need it in our warfile, as it is not used by the application at
// runtime. But it's not clear how to exclude it, as we seem to be
// constructing the jar from the entire WEB-INF directory and per-file
// exclude rules don't seem to work on it. Better solutions are welcome :-)
explodeWar.doLast {
file("${it.explodedAppDirectory}/WEB-INF/lib/tools.jar").setWritable(true)
}
rootProject.deploy.dependsOn appengineDeployAll
rootProject.stage.dependsOn appengineStage
// Impose verification for all of the deployment tasks. We haven't found a
// better way to do this other than to apply to each of them independently.
// If a new task gets added, it will still fail if "environment" is not defined
// because gcpProject is null. We just won't get as friendly an error message.
appengineDeployAll.configure rootProject.verifyDeploymentConfig
appengineDeploy.configure rootProject.verifyDeploymentConfig
appengineDeployCron.configure rootProject.verifyDeploymentConfig
appengineDeployDispatch.configure rootProject.verifyDeploymentConfig
appengineDeployDos.configure rootProject.verifyDeploymentConfig
appengineDeployIndex.configure rootProject.verifyDeploymentConfig
appengineDeployQueue.configure rootProject.verifyDeploymentConfig

473
build.gradle Normal file
View File

@@ -0,0 +1,473 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
import org.gradle.api.tasks.testing.logging.TestExceptionFormat
import org.gradle.api.tasks.testing.logging.TestLogEvent
buildscript {
if (rootProject.enableDependencyLocking.toBoolean()) {
// Lock buildscript dependencies.
configurations.classpath {
resolutionStrategy.activateDependencyLocking()
}
}
dependencies {
classpath 'com.google.cloud.tools:appengine-gradle-plugin:2.0.1'
classpath 'net.ltgt.gradle:gradle-errorprone-plugin:0.6.1'
classpath 'org.sonatype.aether:aether-api:1.13.1'
classpath 'org.sonatype.aether:aether-impl:1.13.1'
}
}
plugins {
// Java static analysis plugins. Keep versions consistent with
// ./buildSrc/build.gradle
id 'nebula.lint' version '16.0.2'
id 'net.ltgt.errorprone' version '0.6.1'
id 'checkstyle'
id 'com.github.johnrengelman.shadow' version '5.1.0'
// NodeJs plugin
id "com.moowork.node" version "1.2.0"
id 'idea'
id 'com.diffplug.gradle.spotless' version '3.25.0'
id 'jacoco'
id 'com.dorongold.task-tree' version '1.5'
}
wrapper {
distributionType = Wrapper.DistributionType.ALL
}
apply plugin: google.registry.gradle.plugin.ReportUploaderPlugin
reportUploader {
// Set the location where we want to upload the build results.
// e.g. -P uploaderDestination=gcs://domain-registry-alpha-build-result-test
//
// If not set - the upload will be skipped
destination = uploaderDestination
// The location of the file containing the OAuth2 Google Cloud credentials.
//
// The file can contain a Service Account key file in JSON format from the
// Google Developers Console or a stored user credential using the format
// supported by the Cloud SDK.
//
// If no file is given - the default credentials are used.
credentialsFile = uploaderCredentialsFile
// If set to 'yes', each file will be uploaded to GCS in a separate thread.
// This is MUCH faster.
multithreadedUpload = uploaderMultithreadedUpload
}
apply from: 'dependencies.gradle'
apply from: 'dependency_lic.gradle'
apply from: 'utils.gradle'
// Custom task to run checkLicense in buildSrc, which is not triggered
// by root project tasks. A shell task is used because buildSrc tasks
// cannot be referenced in the same way as tasks from a regular included
// build.
task checkBuildSrcLicense(type:Exec) {
workingDir "${rootDir}/buildSrc"
commandLine '../gradlew', 'checkLicense'
}
tasks.checkLicense.dependsOn(tasks.checkBuildSrcLicense)
tasks.build.dependsOn(tasks.checkLicense)
// Provide defaults for all of the project properties.
// Only do linting if the build is successful.
gradleLint.autoLintAfterFailure = false
// Paths to main and test sources.
ext.projectRootDir = "${rootDir}"
// Tasks to deploy/stage all App Engine services
task deploy {
group = 'deployment'
description = 'Deploys all services to App Engine.'
}
task stage {
group = 'deployment'
description = 'Generates application directories for all services.'
}
// App-engine environment configuration. We set up all of the variables in
// the root project.
def environments = ['production', 'sandbox', 'alpha', 'crash']
def gcpProject = null
apply from: "${rootDir.path}/projects.gradle"
if (environment == '') {
// Keep the project null, this will prevent deployment. Set the
// environment to "alpha" because other code needs this property to
// explode the war file.
environment = 'alpha'
} else if (environment != 'production' && environment != 'sandbox') {
gcpProject = projects[environment]
if (gcpProject == null) {
throw new GradleException("-Penvironment must be one of " +
"${projects.keySet()}.")
}
}
rootProject.ext.environment = environment
rootProject.ext.gcpProject = gcpProject
rootProject.ext.prodOrSandboxEnv = environment in ['production', 'sandbox']
// Function to verify that the deployment parameters have been set.
def verifyDeploymentParams() {
if (prodOrSandboxEnv) {
// Do not deploy to prod or sandbox. Print a prominent error in bright red.
System.err.println('\033[31;1m-----------------------------------------------------------------')
System.err.println('*** DANGER WILL ROBINSON!')
System.err.println('*** You may not deploy to production or sandbox from gradle. Do a')
System.err.println('*** release from Spinnaker, see deployment playbook.')
System.err.println('-----------------------------------------------------------------')
throw new GradleException('Aborting. See prominent error above.')
} else if (gcpProject == null) {
def error = 'You must specify -P environment={alpha,crash}'
System.err.println("\033[33;1m${error}\033[0m")
throw GradleException("Aborting: ${error}")
}
}
// Closure that we can just drop into all of our deployment tasks.
rootProject.ext.verifyDeploymentConfig = {
doFirst { verifyDeploymentParams() }
}
// Subproject configuration.
allprojects {
// Skip no-op project
if (project.name == 'services') return
repositories {
if (!mavenUrl.isEmpty()) {
maven {
println "Java dependencies: Using repo ${mavenUrl}..."
url mavenUrl
}
} else {
println "Java dependencies: Using Maven Central..."
mavenCentral()
google()
}
}
if (rootProject.enableCrossReferencing.toBoolean()) {
gradle.projectsEvaluated {
tasks.withType(JavaCompile) {
options.fork = true
options.forkOptions.executable =
"${project.rootDir}/kythe/extractors/javac-wrapper.sh"
}
}
}
}
task runPresubmits(type: Exec) {
executable '/usr/bin/python3'
args('config/presubmits.py')
}
def javadocSource = []
def javadocClasspath = []
def javadocDependentTasks = []
subprojects {
// Skip no-op project
if (project.name == 'services') return
ext.createUberJar = {
taskName,
binaryName,
mainClass,
List<Configuration> configs = [project.configurations.runtimeClasspath],
List<SourceSetOutput> srcOutput = [project.sourceSets.main.output],
List<String> excludes = [] ->
project.tasks.create(
taskName, com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) {
mergeServiceFiles()
baseName = binaryName
if (mainClass != '') {
manifest {
attributes 'Main-Class': mainClass
}
}
zip64 = true
classifier = ''
archiveVersion = ''
configurations = configs
from srcOutput
// Excludes signature files that accompany some dependency jars, like
// bonuncycastle. If they are present, only classes from those signed jars are
// made available to the class loader.
// see https://discuss.gradle.org/t/signing-a-custom-gradle-plugin-thats-downloaded-by-the-build-system-from-github/1365
exclude "META-INF/*.SF", "META-INF/*.DSA", "META-INF/*.RSA"
exclude excludes
}
}
if (rootProject.enableDependencyLocking.toBoolean()) {
buildscript {
// Lock buildscript dependencies.
configurations.classpath {
resolutionStrategy.activateDependencyLocking()
}
}
}
afterEvaluate {
if (rootProject.enableDependencyLocking.toBoolean()
&& project.name != 'integration') {
// The ':integration' project runs server/schema integration tests using
// dynamically specified jars with no transitive dependency. Therefore
// dependency-locking does not make sense. Furthermore, during
// evaluation it resolves the 'testRuntime' configuration, making it
// immutable. Locking activation would trigger an invalid operation
// exception.
//
// For all other projects, due to problem with the gradle-license-report
// plugin, the dependencyLicenseReport configuration must opt out of
// dependency-locking. See dependency_lic.gradle for the reason why.
//
// To selectively activate dependency locking without hardcoding them
// in the 'configurations' block, the following code must run after
// project evaluation, when all configurations have been created.
configurations.each {
if (it.name != 'dependencyLicenseReport' && it.name != 'integration') {
it.resolutionStrategy.activateDependencyLocking()
}
}
}
}
def services = [':services:default',
':services:backend',
':services:tools',
':services:pubapi']
// Set up all of the deployment projects.
if (services.contains(project.path)) {
apply from: "${rootDir.path}/appengine_war.gradle"
// Return early, do not apply the settings below.
return
}
apply from: "${rootDir.path}/java_common.gradle"
if (project.name == 'third_party') return
project.tasks.test.dependsOn runPresubmits
def commonlyExcludedResources = ['**/*.java', '**/BUILD']
project.ext.javaDir = "${project.projectDir}/src/main/java"
project.ext.javaTestDir = "${project.projectDir}/src/test/java"
project.ext.resourcesSourceDir = "${project.projectDir}/src/main/resources"
sourceSets {
main {
resources {
srcDirs += project.ext.javaDir
exclude commonlyExcludedResources
}
}
test {
resources {
srcDirs += project.ext.javaTestDir
exclude commonlyExcludedResources
}
}
}
// No need to produce javadoc for the docs subproject, which has no APIs to
// expose to users.
if (project.name != 'docs') {
javadocSource << project.sourceSets.main.allJava
javadocClasspath << project.sourceSets.main.compileClasspath
javadocClasspath << "${buildDir}/generated/sources/annotationProcessor/java/main"
javadocDependentTasks << project.tasks.compileJava
}
}
// If "-P verboseTestOutput=true" is passed in, configure all subprojects to dump all of their
// output and final test status (pass/fail, errors) for each test class.
//
// Note that we can't do this in the main subprojects section above because that's evaluated before
// the subproject build files and the test tasks haven't been defined yet. We have to do it from
// the projectsEvaluted hook, which gets called after the subprojects are configured.
if (verboseTestOutput.toBoolean()) {
gradle.projectsEvaluated({
subprojects {
tasks.withType(Test) {
testLogging {
events TestLogEvent.FAILED, TestLogEvent.PASSED, TestLogEvent.SKIPPED,
TestLogEvent.STANDARD_OUT, TestLogEvent.STANDARD_ERROR
exceptionFormat TestExceptionFormat.FULL
showExceptions true
showCauses true
showStackTraces true
afterSuite { desc, result ->
println "Results: ${result.resultType}, " +
"${result.successfulTestCount}/${result.testCount} tests " +
"passed, ${result.failedTestCount} failures.";
}
}
}
}
})
}
task checkDependenciesDotGradle {
def buildSrcDepsFile = File.createTempFile('buildSrc', 'deps')
buildSrcDepsFile.deleteOnExit()
dependsOn createGetBuildSrcDirectDepsTask(buildSrcDepsFile)
doLast {
Set<String> depsInUse = []
allprojects {
configurations.all {
it.dependencies.findAll { it.group != null }.each {
// Note: .toString() is required since GString should
// not be mixed with Java Strings.
depsInUse.add("${it.group}:${it.name}".toString())
}
}
}
if (buildSrcDepsFile.exists()) {
depsInUse.addAll(buildSrcDepsFile.readLines())
}
def unusedDeps =
rootProject.dependencyMap.keySet()
.findAll { !depsInUse.contains(it) }
.toSorted()
if (unusedDeps.isEmpty()) {
return
}
logger.error(
"Unused dependencies in dependencies.gradle:\n${unusedDeps.toListString()}")
throw new IllegalStateException(
"The dependencies.gradle file should only contain direct dependencies.")
}
}
tasks.build.dependsOn(tasks.checkDependenciesDotGradle)
def createGetBuildSrcDirectDepsTask(outputFileName) {
return tasks
.create(
"getBuildSrcDeps_${java.util.UUID.randomUUID()}".toString(),
Exec) {
workingDir "${rootDir}/buildSrc"
commandLine '../gradlew', 'exportDependencies',
"-PdependencyExportFile=${outputFileName}"
}
}
rootProject.ext {
invokeJavaDiffFormatScript = { action ->
def scriptDir = rootDir.path.endsWith('buildSrc')
? "${rootDir}/../java-format"
: "${rootDir}/java-format"
def workingDir = rootDir.path.endsWith('buildSrc')
? "${rootDir}/.."
: rootDir
def formatDiffScript = "${scriptDir}/google-java-format-git-diff.sh"
return ext.execInBash(
"${formatDiffScript} ${action}", "${workingDir}")
}
}
// Checks if modified lines in Java source files need reformatting.
// Note that this task checks modified Java files in the entire repository.
task javaIncrementalFormatCheck {
doLast {
def checkResult = invokeJavaDiffFormatScript("check")
if (checkResult == 'true') {
throw new IllegalStateException(
"Some Java files need to be reformatted. You may use the "
+ "'javaIncrementalFormatDryRun' task to review\n "
+ "the changes, or the 'javaIncrementalFormatApply' task "
+ "to reformat.")
} else if (checkResult != 'false') {
throw new RuntimeException(
"Failed to invoke format check script:\n" + checkResult)
}
println("Incremental Java format check ok.")
}
}
// Shows how modified lines in Java source files will change after formatting.
// Note that this task checks modified Java files in the entire repository.
task javaIncrementalFormatDryRun {
doLast {
println("${invokeJavaDiffFormatScript("show")}")
}
}
tasks.build.dependsOn(tasks.javaIncrementalFormatCheck)
// Checks if modified lines in Java source files need reformatting.
// Note that this task processes modified Java files in the entire repository.
task javaIncrementalFormatApply {
doLast {
invokeJavaDiffFormatScript("format")
}
}
task javadoc(type: Javadoc) {
source javadocSource
classpath = files(javadocClasspath)
destinationDir = file("${buildDir}/docs/javadoc")
options.encoding = "UTF-8"
// In a lot of places we don't write @return so suppress warnings about that.
options.addBooleanOption('Xdoclint:all,-missing', true)
options.addBooleanOption("-allow-script-in-comments",true)
options.tags = ["type:a:Generic Type",
"error:a:Expected Error"]
}
tasks.build.dependsOn(tasks.javadoc)
// Task for doing development on core Nomulus.
// This fixes code formatting automatically as necessary, builds and tests the
// core Nomulus codebase, and runs all presubmits.
task coreDev {
dependsOn 'javaIncrementalFormatApply'
dependsOn 'javadoc'
dependsOn 'checkDependenciesDotGradle'
dependsOn 'checkLicense'
dependsOn ':core:check'
dependsOn 'assemble'
}
javadocDependentTasks.each { tasks.javadoc.dependsOn(it) }

110
buildSrc/build.gradle Normal file
View File

@@ -0,0 +1,110 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
import static com.google.common.base.Strings.isNullOrEmpty;
buildscript {
if (project.enableDependencyLocking.toBoolean()) {
// Lock buildscript dependencies.
configurations.classpath {
resolutionStrategy.activateDependencyLocking()
}
}
}
plugins {
// Java static analysis plugins. Keep versions consistent with ../build.gradle
id 'nebula.lint' version '16.0.2'
id 'net.ltgt.errorprone' version '0.6.1'
id 'checkstyle'
id 'com.diffplug.gradle.spotless' version '3.25.0'
}
if (rootProject.enableDependencyLocking.toBoolean()) {
// Lock application dependencies.
dependencyLocking {
lockAllConfigurations()
}
}
repositories {
if (isNullOrEmpty(project.ext.properties.mavenUrl)) {
println "Java dependencies: Using Maven central..."
mavenCentral()
google()
} else {
maven {
println "Java dependencies: Using repo ${mavenUrl}..."
url mavenUrl
}
}
}
apply from: '../dependencies.gradle'
apply from: '../dependency_lic.gradle'
apply from: '../java_common.gradle'
sourceSets {
main {
java {
srcDirs += "${project.buildDir}/generated/source/apt/main"
}
}
}
checkstyle {
configDir file('../config/checkstyle')
}
dependencies {
def deps = dependencyMap
compile deps['com.google.auth:google-auth-library-credentials']
compile deps['com.google.auth:google-auth-library-oauth2-http']
compile deps['com.google.cloud:google-cloud-core']
compile deps['com.google.guava:guava']
compile deps['com.google.auto.value:auto-value-annotations']
compile deps['com.google.cloud:google-cloud-storage']
compile deps['org.apache.commons:commons-text']
compile deps['com.google.template:soy']
annotationProcessor deps['com.google.auto.value:auto-value']
testCompile deps['com.google.truth:truth']
testCompile deps['com.google.truth.extensions:truth-java8-extension']
testCompile deps['org.junit.jupiter:junit-jupiter-api']
testCompile deps['org.junit.jupiter:junit-jupiter-engine']
testCompile deps['org.mockito:mockito-core']
}
gradle.projectsEvaluated {
tasks.withType(JavaCompile) {
options.compilerArgs << "-Xlint:unchecked"
}
}
task exportDependencies {
def outputFileProperty = 'dependencyExportFile'
def output = project.hasProperty(outputFileProperty)
? new PrintStream(
new File(project.getProperty(outputFileProperty)))
: System.out
doLast {
project.configurations.all {
it.dependencies.findAll {
it.group != null
}.each {
output.println("${it.group}:${it.name}")
}
}
}
}

View File

@@ -0,0 +1 @@
enableDependencyLocking=true

View File

@@ -0,0 +1,51 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.diffplug.durian:durian-collect:1.2.0
com.diffplug.durian:durian-core:1.2.0
com.diffplug.durian:durian-io:1.2.0
com.diffplug.gradle.spotless:com.diffplug.gradle.spotless.gradle.plugin:3.25.0
com.diffplug.spotless:spotless-lib-extra:1.25.0
com.diffplug.spotless:spotless-lib:1.25.0
com.diffplug.spotless:spotless-plugin-gradle:3.25.0
com.google.guava:guava:19.0
com.googlecode.concurrent-trees:concurrent-trees:2.6.1
com.googlecode.javaewah:JavaEWAH:1.1.6
com.jcraft:jsch:0.1.55
com.jcraft:jzlib:1.1.1
com.netflix.nebula:gradle-lint-plugin:16.0.2
com.netflix.nebula:nebula-gradle-interop:1.0.11
commons-lang:commons-lang:2.6
javax.inject:javax.inject:1
junit:junit:4.12
nebula.lint:nebula.lint.gradle.plugin:16.0.2
net.ltgt.errorprone:net.ltgt.errorprone.gradle.plugin:0.6.1
net.ltgt.gradle:gradle-errorprone-plugin:0.6.1
org.apache.commons:commons-lang3:3.8.1
org.apache.maven:maven-artifact:3.6.2
org.apache.maven:maven-builder-support:3.6.2
org.apache.maven:maven-model-builder:3.6.2
org.apache.maven:maven-model:3.6.2
org.bouncycastle:bcpg-jdk15on:1.61
org.bouncycastle:bcpkix-jdk15on:1.61
org.bouncycastle:bcprov-jdk15on:1.61
org.codehaus.gpars:gpars:1.2.1
org.codehaus.groovy:groovy-xml:2.4.7
org.codehaus.groovy:groovy:2.4.7
org.codehaus.jsr166-mirror:jsr166y:1.7.0
org.codehaus.plexus:plexus-interpolation:1.25
org.codehaus.plexus:plexus-utils:3.2.1
org.eclipse.jgit:org.eclipse.jgit:5.5.0.201909110433-r
org.eclipse.sisu:org.eclipse.sisu.inject:0.3.3
org.hamcrest:hamcrest-core:1.3
org.jetbrains.kotlin:kotlin-stdlib-common:1.3.50
org.jetbrains.kotlin:kotlin-stdlib-jdk7:1.3.50
org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.3.50
org.jetbrains.kotlin:kotlin-stdlib:1.3.50
org.jetbrains:annotations:13.0
org.multiverse:multiverse-core:0.7.0
org.ow2.asm:asm-analysis:7.1
org.ow2.asm:asm-commons:7.1
org.ow2.asm:asm-tree:7.1
org.ow2.asm:asm:7.1
org.slf4j:slf4j-api:1.7.2

View File

@@ -3,16 +3,16 @@
# This file is expected to be part of source control.
antlr:antlr:2.7.7
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.2.0
com.google.errorprone:error_prone_annotations:2.3.2
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.0.1-jre
com.google.guava:guava:28.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.puppycrawl.tools:checkstyle:8.17
commons-beanutils:commons-beanutils:1.9.3
com.google.j2objc:j2objc-annotations:1.3
com.puppycrawl.tools:checkstyle:8.27
commons-beanutils:commons-beanutils:1.9.4
commons-collections:commons-collections:3.2.2
info.picocli:picocli:3.9.0
net.sf.saxon:Saxon-HE:9.9.0-2
info.picocli:picocli:4.1.1
net.sf.saxon:Saxon-HE:9.9.1-5
org.antlr:antlr4-runtime:4.7.2
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.checkerframework:checker-qual:2.8.1
org.codehaus.mojo:animal-sniffer-annotations:1.18

View File

@@ -3,7 +3,7 @@
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.6
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
@@ -11,8 +11,8 @@ com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.12.0
com.google.auth:google-auth-library-oauth2-http:0.12.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
@@ -20,27 +20,27 @@ com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.2.0
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.1-jre
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.27.0
com.google.http-client:google-http-client:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.1
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.10
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.12.0
io.opencensus:opencensus-api:0.15.0
io.opencensus:opencensus-contrib-http-util:0.15.0
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
@@ -48,10 +48,9 @@ javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.5
org.apache.httpcomponents:httpcore:4.4.9
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.checkerframework:checker-qual:2.11.1
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0

View File

@@ -3,7 +3,7 @@
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.6
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
@@ -11,8 +11,8 @@ com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.12.0
com.google.auth:google-auth-library-oauth2-http:0.12.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
@@ -20,27 +20,27 @@ com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.2.0
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.1-jre
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.27.0
com.google.http-client:google-http-client:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.1
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.10
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.12.0
io.opencensus:opencensus-api:0.15.0
io.opencensus:opencensus-contrib-http-util:0.15.0
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
@@ -48,10 +48,9 @@ javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.5
org.apache.httpcomponents:httpcore:4.4.9
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.checkerframework:checker-qual:2.11.1
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0

View File

@@ -0,0 +1,60 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.checkerframework:checker-qual:2.11.1
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.threeten:threetenbp:1.3.3

View File

@@ -0,0 +1,4 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
org.jacoco:org.jacoco.agent:0.8.5

View File

@@ -0,0 +1,11 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
org.jacoco:org.jacoco.agent:0.8.5
org.jacoco:org.jacoco.ant:0.8.5
org.jacoco:org.jacoco.core:0.8.5
org.jacoco:org.jacoco.report:0.8.5
org.ow2.asm:asm-analysis:7.2
org.ow2.asm:asm-commons:7.2
org.ow2.asm:asm-tree:7.2
org.ow2.asm:asm:7.2

View File

@@ -0,0 +1,60 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.checkerframework:checker-qual:2.11.1
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.threeten:threetenbp:1.3.3

View File

@@ -3,7 +3,7 @@
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.6
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
@@ -11,8 +11,8 @@ com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.12.0
com.google.auth:google-auth-library-oauth2-http:0.12.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
@@ -20,27 +20,27 @@ com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.2.0
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.1-jre
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.27.0
com.google.http-client:google-http-client:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.1
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.10
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.12.0
io.opencensus:opencensus-api:0.15.0
io.opencensus:opencensus-contrib-http-util:0.15.0
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
@@ -48,10 +48,9 @@ javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.5
org.apache.httpcomponents:httpcore:4.4.9
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.checkerframework:checker-qual:2.11.1
org.json:json:20160212
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0

View File

@@ -3,7 +3,7 @@
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.6
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
@@ -11,8 +11,8 @@ com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.12.0
com.google.auth:google-auth-library-oauth2-http:0.12.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
@@ -20,49 +20,55 @@ com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.3.1
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.1-jre
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.27.0
com.google.http-client:google-http-client:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.1
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.google.truth.extensions:truth-java8-extension:0.44
com.google.truth:truth:0.44
com.google.truth.extensions:truth-java8-extension:1.0
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.10
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.12.0
io.opencensus:opencensus-api:0.15.0
io.opencensus:opencensus-contrib-http-util:0.15.0
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
junit:junit:4.12
net.bytebuddy:byte-buddy-agent:1.9.7
net.bytebuddy:byte-buddy:1.9.7
net.bytebuddy:byte-buddy-agent:1.10.5
net.bytebuddy:byte-buddy:1.10.5
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.5
org.apache.httpcomponents:httpcore:4.4.9
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.checkerframework:checker-qual:2.11.1
org.hamcrest:hamcrest-core:1.3
org.json:json:20160212
org.mockito:mockito-core:2.25.0
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.3.3
org.objenesis:objenesis:2.6
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0

View File

@@ -3,7 +3,7 @@
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.6
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
@@ -11,8 +11,8 @@ com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.12.0
com.google.auth:google-auth-library-oauth2-http:0.12.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
@@ -20,49 +20,55 @@ com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.3.1
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.1-jre
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.27.0
com.google.http-client:google-http-client:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.1
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.google.truth.extensions:truth-java8-extension:0.44
com.google.truth:truth:0.44
com.google.truth.extensions:truth-java8-extension:1.0
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.10
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.12.0
io.opencensus:opencensus-api:0.15.0
io.opencensus:opencensus-contrib-http-util:0.15.0
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
junit:junit:4.12
net.bytebuddy:byte-buddy-agent:1.9.7
net.bytebuddy:byte-buddy:1.9.7
net.bytebuddy:byte-buddy-agent:1.10.5
net.bytebuddy:byte-buddy:1.10.5
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.5
org.apache.httpcomponents:httpcore:4.4.9
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.checkerframework:checker-qual:2.11.1
org.hamcrest:hamcrest-core:1.3
org.json:json:20160212
org.mockito:mockito-core:2.25.0
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.3.3
org.objenesis:objenesis:2.6
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0

View File

@@ -0,0 +1,77 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.google.truth.extensions:truth-java8-extension:1.0
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
junit:junit:4.12
net.bytebuddy:byte-buddy-agent:1.10.5
net.bytebuddy:byte-buddy:1.10.5
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.hamcrest:hamcrest-core:1.3
org.json:json:20160212
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.3.3
org.objenesis:objenesis:2.6
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0
org.ow2.asm:asm-util:6.0
org.ow2.asm:asm:6.0
org.threeten:threetenbp:1.3.3

View File

@@ -3,7 +3,7 @@
# This file is expected to be part of source control.
aopalliance:aopalliance:1.0
args4j:args4j:2.0.23
com.fasterxml.jackson.core:jackson-core:2.9.6
com.fasterxml.jackson.core:jackson-core:2.9.9
com.google.api-client:google-api-client:1.27.0
com.google.api.grpc:proto-google-common-protos:1.12.0
com.google.api.grpc:proto-google-iam-v1:0.12.0
@@ -11,8 +11,8 @@ com.google.api:api-common:1.7.0
com.google.api:gax-httpjson:0.52.1
com.google.api:gax:1.35.1
com.google.apis:google-api-services-storage:v1-rev20181013-1.27.0
com.google.auth:google-auth-library-credentials:0.12.0
com.google.auth:google-auth-library-oauth2-http:0.12.0
com.google.auth:google-auth-library-credentials:0.16.1
com.google.auth:google-auth-library-oauth2-http:0.16.1
com.google.auto.value:auto-value-annotations:1.6.3
com.google.cloud:google-cloud-core-http:1.59.0
com.google.cloud:google-cloud-core:1.59.0
@@ -20,49 +20,55 @@ com.google.cloud:google-cloud-storage:1.59.0
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.7
com.google.common.html.types:types:1.0.4
com.google.errorprone:error_prone_annotations:2.3.1
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.1-jre
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.0-beta1
com.google.http-client:google-http-client-appengine:1.27.0
com.google.http-client:google-http-client-jackson2:1.27.0
com.google.http-client:google-http-client:1.27.0
com.google.http-client:google-http-client-jackson2:1.30.1
com.google.http-client:google-http-client:1.30.1
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.1
com.google.j2objc:j2objc-annotations:1.3
com.google.oauth-client:google-oauth-client:1.27.0
com.google.protobuf:protobuf-java-util:3.6.1
com.google.protobuf:protobuf-java:3.6.1
com.google.template:soy:2018-03-14
com.google.truth.extensions:truth-java8-extension:0.44
com.google.truth:truth:0.44
com.google.truth.extensions:truth-java8-extension:1.0
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
com.ibm.icu:icu4j:57.1
commons-codec:commons-codec:1.10
commons-codec:commons-codec:1.11
commons-logging:commons-logging:1.2
io.grpc:grpc-context:1.12.0
io.opencensus:opencensus-api:0.15.0
io.opencensus:opencensus-contrib-http-util:0.15.0
io.grpc:grpc-context:1.19.0
io.opencensus:opencensus-api:0.21.0
io.opencensus:opencensus-contrib-http-util:0.21.0
javax.annotation:javax.annotation-api:1.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.validation:validation-api:1.0.0.GA
joda-time:joda-time:2.9.2
junit:junit:4.12
net.bytebuddy:byte-buddy-agent:1.9.7
net.bytebuddy:byte-buddy:1.9.7
net.bytebuddy:byte-buddy-agent:1.10.5
net.bytebuddy:byte-buddy:1.10.5
org.apache.commons:commons-lang3:3.8.1
org.apache.commons:commons-text:1.6
org.apache.httpcomponents:httpclient:4.5.5
org.apache.httpcomponents:httpcore:4.4.9
org.apache.httpcomponents:httpclient:4.5.8
org.apache.httpcomponents:httpcore:4.4.11
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.checkerframework:checker-qual:2.11.1
org.hamcrest:hamcrest-core:1.3
org.json:json:20160212
org.mockito:mockito-core:2.25.0
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.mockito:mockito-core:3.3.3
org.objenesis:objenesis:2.6
org.opentest4j:opentest4j:1.2.0
org.ow2.asm:asm-analysis:6.0
org.ow2.asm:asm-commons:6.0
org.ow2.asm:asm-tree:6.0

View File

@@ -26,8 +26,10 @@ import com.google.common.collect.ImmutableSetMultimap;
import com.google.template.soy.SoyFileSet;
import com.google.template.soy.tofu.SoyTofu;
import google.registry.gradle.plugin.ProjectData.TaskData;
import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Map;
import java.util.function.Supplier;
import java.util.stream.Collectors;
@@ -148,20 +150,28 @@ final class CoverPageGenerator {
/** returns a soy-friendly version of the given task data. */
static ImmutableMap<String, Object> taskDataToSoy(TaskData task) {
// Note that all instances of File.separator are replaced with forward slashes so that we can
// generate a valid href on Windows.
return new ImmutableMap.Builder<String, Object>()
.put("uniqueName", task.uniqueName())
.put("description", task.description())
.put("log", task.log().isPresent() ? getLogPath(task).toString() : "")
.put(
"log",
task.log().isPresent() ? getLogPath(task).toString().replace(File.separator, "/") : "")
.put(
"reports",
task.reports().entrySet().stream()
.collect(
toImmutableMap(
entry -> entry.getKey(),
Map.Entry::getKey,
entry ->
entry.getValue().files().isEmpty()
? ""
: entry.getValue().entryPoint().toString())))
: entry
.getValue()
.entryPoint()
.toString()
.replace(File.separator, "/"))))
.build();
}
@@ -188,8 +198,10 @@ final class CoverPageGenerator {
}
private static Path getLogPath(TaskData task) {
// TODO(guyben):escape uniqueName to guarantee legal file name.
return Paths.get("logs", task.uniqueName() + ".log");
// We replace colons with dashes so that the resulting filename is always valid, even in
// Windows. As a dash is not a valid character in Java identifies, a task name cannot include
// it, so the uniqueness of the name is perserved.
return Paths.get("logs", task.uniqueName().replace(":", "-") + ".log");
}
private static Supplier<byte[]> resourceLoader(Path path) {

View File

@@ -57,6 +57,10 @@ final class GcsPluginUtils {
return file.toPath().toAbsolutePath().normalize();
}
static Path toNormalizedPath(Path file) {
return file.toAbsolutePath().normalize();
}
static String getContentType(String fileName) {
return EXTENSION_TO_CONTENT_TYPE.getOrDefault(
Files.getFileExtension(fileName), DEFAULT_CONTENT_TYPE);
@@ -64,7 +68,8 @@ final class GcsPluginUtils {
static void uploadFileToGcs(
Storage storage, String bucket, Path path, Supplier<byte[]> dataSupplier) {
String filename = path.toString();
// Replace Windows file separators with forward slashes.
String filename = path.toString().replace(File.separator, "/");
storage.create(
BlobInfo.newBuilder(bucket, filename).setContentType(getContentType(filename)).build(),
dataSupplier.get());
@@ -167,7 +172,7 @@ final class GcsPluginUtils {
.collect(
toImmutableMap(
file -> rootDir.relativize(toNormalizedPath(file)),
file -> toByteArraySupplier(file)));
GcsPluginUtils::toByteArraySupplier));
if (files.isEmpty()) {
// The directory exists, but is empty. Return empty FilesWithEntryPoint

View File

@@ -20,17 +20,16 @@ import static com.google.common.truth.Truth8.assertThat;
import static google.registry.gradle.plugin.GcsPluginUtils.toByteArraySupplier;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import google.registry.gradle.plugin.ProjectData.TaskData;
import java.io.File;
import java.nio.file.Paths;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.junit.jupiter.api.Test;
/** Tests for {@link CoverPageGenerator} */
@RunWith(JUnit4.class)
public final class CoverPageGeneratorTest {
final class CoverPageGeneratorTest {
private static final ProjectData EMPTY_PROJECT =
ProjectData.builder()
@@ -63,6 +62,8 @@ public final class CoverPageGeneratorTest {
.setState(TaskData.State.UP_TO_DATE)
.build();
private static final Joiner filenameJoiner = Joiner.on(File.separator);
private ImmutableMap<String, String> getGeneratedFiles(ProjectData project) {
CoverPageGenerator coverPageGenerator = new CoverPageGenerator(project);
FilesWithEntryPoint files = coverPageGenerator.getFilesToUpload();
@@ -84,14 +85,14 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testGetFilesToUpload_entryPoint_isIndexHtml() {
void testGetFilesToUpload_entryPoint_isIndexHtml() {
CoverPageGenerator coverPageGenerator = new CoverPageGenerator(EMPTY_PROJECT);
assertThat(coverPageGenerator.getFilesToUpload().entryPoint())
.isEqualTo(Paths.get("index.html"));
}
@Test
public void testGetFilesToUpload_containsEntryFile() {
void testGetFilesToUpload_containsEntryFile() {
String content = getContentOfGeneratedFile(EMPTY_PROJECT, "index.html");
assertThat(content)
.contains(
@@ -99,7 +100,7 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testCoverPage_showsFailedTask() {
void testCoverPage_showsFailedTask() {
String content = getCoverPage(EMPTY_PROJECT.toBuilder().addTask(EMPTY_TASK_FAILURE).build());
assertThat(content).contains("task-failure");
assertThat(content).contains("<p>FAILURE</p>");
@@ -108,7 +109,7 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testCoverPage_showsSuccessfulTask() {
void testCoverPage_showsSuccessfulTask() {
String content = getCoverPage(EMPTY_PROJECT.toBuilder().addTask(EMPTY_TASK_SUCCESS).build());
assertThat(content).contains("task-success");
assertThat(content).doesNotContain("<p>FAILURE</p>");
@@ -117,7 +118,7 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testCoverPage_showsUpToDateTask() {
void testCoverPage_showsUpToDateTask() {
String content = getCoverPage(EMPTY_PROJECT.toBuilder().addTask(EMPTY_TASK_UP_TO_DATE).build());
assertThat(content).contains("task-up-to-date");
assertThat(content).doesNotContain("<p>FAILURE</p>");
@@ -126,11 +127,10 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testCoverPage_failedAreFirst() {
void testCoverPage_failedAreFirst() {
String content =
getCoverPage(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(EMPTY_TASK_UP_TO_DATE)
.addTask(EMPTY_TASK_FAILURE)
.addTask(EMPTY_TASK_SUCCESS)
@@ -145,11 +145,10 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testCoverPage_failingTask_statusIsFailure() {
void testCoverPage_failingTask_statusIsFailure() {
String content =
getCoverPage(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(EMPTY_TASK_UP_TO_DATE)
.addTask(EMPTY_TASK_FAILURE)
.addTask(EMPTY_TASK_SUCCESS)
@@ -158,11 +157,10 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testCoverPage_noFailingTask_statusIsSuccess() {
void testCoverPage_noFailingTask_statusIsSuccess() {
String content =
getCoverPage(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(EMPTY_TASK_UP_TO_DATE)
.addTask(EMPTY_TASK_SUCCESS)
.build());
@@ -170,52 +168,47 @@ public final class CoverPageGeneratorTest {
}
@Test
public void testGetFilesToUpload_containsCssFile() {
void testGetFilesToUpload_containsCssFile() {
ImmutableMap<String, String> files = getGeneratedFiles(EMPTY_PROJECT);
assertThat(files).containsKey("css/style.css");
assertThat(files.get("css/style.css")).contains("body {");
assertThat(files).containsKey(filenameJoiner.join("css", "style.css"));
assertThat(files.get(filenameJoiner.join("css", "style.css"))).contains("body {");
assertThat(files.get("index.html"))
.contains("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/style.css\">");
}
@Test
public void testCreateReportFiles_taskWithLog() {
void testCreateReportFiles_taskWithLog() {
ImmutableMap<String, String> files =
getGeneratedFiles(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(
EMPTY_TASK_SUCCESS
.toBuilder()
EMPTY_TASK_SUCCESS.toBuilder()
.setUniqueName("my:name")
.setLog(toByteArraySupplier("my log data"))
.build())
.build());
assertThat(files).containsEntry("logs/my:name.log", "my log data");
assertThat(files.get("index.html")).contains("<a href=\"logs/my:name.log\">[log]</a>");
assertThat(files).containsEntry(filenameJoiner.join("logs", "my-name.log"), "my log data");
assertThat(files.get("index.html")).contains("<a href=\"logs/my-name.log\">[log]</a>");
}
@Test
public void testCreateReportFiles_taskWithoutLog() {
void testCreateReportFiles_taskWithoutLog() {
ImmutableMap<String, String> files =
getGeneratedFiles(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(EMPTY_TASK_SUCCESS.toBuilder().setUniqueName("my:name").build())
.build());
assertThat(files).doesNotContainKey("logs/my:name.log");
assertThat(files).doesNotContainKey("logs/my-name.log");
assertThat(files.get("index.html")).contains("<span class=\"report_link_broken\">[log]</span>");
}
@Test
public void testCreateReportFiles_taskWithFilledReport() {
void testCreateReportFiles_taskWithFilledReport() {
ImmutableMap<String, String> files =
getGeneratedFiles(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(
EMPTY_TASK_SUCCESS
.toBuilder()
EMPTY_TASK_SUCCESS.toBuilder()
.putReport(
"someReport",
FilesWithEntryPoint.create(
@@ -225,39 +218,35 @@ public final class CoverPageGeneratorTest {
Paths.get("path", "report.txt")))
.build())
.build());
assertThat(files).containsEntry("path/report.txt", "report content");
assertThat(files).containsEntry(filenameJoiner.join("path", "report.txt"), "report content");
assertThat(files.get("index.html")).contains("<a href=\"path/report.txt\">[someReport]</a>");
}
@Test
public void testCreateReportFiles_taskWithEmptyReport() {
void testCreateReportFiles_taskWithEmptyReport() {
ImmutableMap<String, String> files =
getGeneratedFiles(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(
EMPTY_TASK_SUCCESS
.toBuilder()
EMPTY_TASK_SUCCESS.toBuilder()
.putReport(
"someReport",
FilesWithEntryPoint.create(
ImmutableMap.of(), Paths.get("path", "report.txt")))
.build())
.build());
assertThat(files).doesNotContainKey("path/report.txt");
assertThat(files).doesNotContainKey(filenameJoiner.join("path", "report.txt"));
assertThat(files.get("index.html"))
.contains("<span class=\"report_link_broken\">[someReport]</span>");
}
@Test
public void testCreateReportFiles_taskWithLogAndMultipleReports() {
void testCreateReportFiles_taskWithLogAndMultipleReports() {
ImmutableMap<String, String> files =
getGeneratedFiles(
EMPTY_PROJECT
.toBuilder()
EMPTY_PROJECT.toBuilder()
.addTask(
EMPTY_TASK_SUCCESS
.toBuilder()
EMPTY_TASK_SUCCESS.toBuilder()
.setUniqueName("my:name")
.setLog(toByteArraySupplier("log data"))
.putReport(
@@ -275,12 +264,15 @@ public final class CoverPageGeneratorTest {
ImmutableMap.of(), Paths.get("path-empty", "report.txt")))
.build())
.build());
assertThat(files).containsEntry("path-filled/report.txt", "report content");
assertThat(files).containsEntry("path-filled/other/file.txt", "some other content");
assertThat(files).containsEntry("logs/my:name.log", "log data");
assertThat(files)
.containsEntry(filenameJoiner.join("path-filled", "report.txt"), "report content");
assertThat(files)
.containsEntry(
filenameJoiner.join("path-filled", "other", "file.txt"), "some other content");
assertThat(files).containsEntry(filenameJoiner.join("logs", "my-name.log"), "log data");
assertThat(files.get("index.html"))
.contains("<a href=\"path-filled/report.txt\">[filledReport]</a>");
assertThat(files.get("index.html")).contains("<a href=\"logs/my:name.log\">[log]</a>");
assertThat(files.get("index.html")).contains("<a href=\"logs/my-name.log\">[log]</a>");
assertThat(files.get("index.html"))
.contains("<span class=\"report_link_broken\">[emptyReport]</span>");
}

View File

@@ -0,0 +1,298 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.gradle.plugin;
import static com.google.common.collect.ImmutableMap.toImmutableMap;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.gradle.plugin.GcsPluginUtils.getContentType;
import static google.registry.gradle.plugin.GcsPluginUtils.readFilesWithEntryPoint;
import static google.registry.gradle.plugin.GcsPluginUtils.toByteArraySupplier;
import static google.registry.gradle.plugin.GcsPluginUtils.toNormalizedPath;
import static google.registry.gradle.plugin.GcsPluginUtils.uploadFileToGcs;
import static google.registry.gradle.plugin.GcsPluginUtils.uploadFilesToGcsMultithread;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.nio.file.Files.createDirectories;
import static java.nio.file.Files.createDirectory;
import static java.nio.file.Files.createFile;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableMap;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Optional;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
/** Tests for {@link GcsPluginUtilsTest} */
final class GcsPluginUtilsTest {
private static final Joiner filenameJoiner = Joiner.on(File.separator);
@SuppressWarnings("WeakerAccess")
@TempDir
Path tmpDir;
@Test
void testGetContentType_knownTypes() {
assertThat(getContentType("path/to/file.html")).isEqualTo("text/html");
assertThat(getContentType("path/to/file.htm")).isEqualTo("text/html");
assertThat(getContentType("path/to/file.log")).isEqualTo("text/plain");
assertThat(getContentType("path/to/file.txt")).isEqualTo("text/plain");
assertThat(getContentType("path/to/file.css")).isEqualTo("text/css");
assertThat(getContentType("path/to/file.xml")).isEqualTo("text/xml");
assertThat(getContentType("path/to/file.zip")).isEqualTo("application/zip");
assertThat(getContentType("path/to/file.js")).isEqualTo("text/javascript");
}
@Test
void testGetContentType_unknownTypes() {
assertThat(getContentType("path/to/file.unknown")).isEqualTo("application/octet-stream");
}
@Test
void testUploadFileToGcs() {
Storage storage = mock(Storage.class);
uploadFileToGcs(
storage, "my-bucket", Paths.get("my", "filename.txt"), toByteArraySupplier("my data"));
verify(storage)
.create(
BlobInfo.newBuilder("my-bucket", "my/filename.txt")
.setContentType("text/plain")
.build(),
"my data".getBytes(UTF_8));
verifyNoMoreInteractions(storage);
}
@Test
void testUploadFilesToGcsMultithread() {
Storage storage = mock(Storage.class);
uploadFilesToGcsMultithread(
storage,
"my-bucket",
Paths.get("my", "folder"),
ImmutableMap.of(
Paths.get("some", "index.html"), toByteArraySupplier("some web page"),
Paths.get("some", "style.css"), toByteArraySupplier("some style"),
Paths.get("other", "index.html"), toByteArraySupplier("other web page"),
Paths.get("other", "style.css"), toByteArraySupplier("other style")));
verify(storage)
.create(
BlobInfo.newBuilder("my-bucket", "my/folder/some/index.html")
.setContentType("text/html")
.build(),
"some web page".getBytes(UTF_8));
verify(storage)
.create(
BlobInfo.newBuilder("my-bucket", "my/folder/some/style.css")
.setContentType("text/css")
.build(),
"some style".getBytes(UTF_8));
verify(storage)
.create(
BlobInfo.newBuilder("my-bucket", "my/folder/other/index.html")
.setContentType("text/html")
.build(),
"other web page".getBytes(UTF_8));
verify(storage)
.create(
BlobInfo.newBuilder("my-bucket", "my/folder/other/style.css")
.setContentType("text/css")
.build(),
"other style".getBytes(UTF_8));
verifyNoMoreInteractions(storage);
}
@Test
void testToByteArraySupplier_string() {
assertThat(toByteArraySupplier("my string").get()).isEqualTo("my string".getBytes(UTF_8));
}
@Test
void testToByteArraySupplier_stringSupplier() {
assertThat(toByteArraySupplier(() -> "my string").get()).isEqualTo("my string".getBytes(UTF_8));
}
@Test
void testToByteArraySupplier_file() throws Exception {
Path dir = createDirectory(tmpDir.resolve("arbitrary"));
Path file = createFile(dir.resolve("file.txt"));
Files.write(file, "some data".getBytes(UTF_8));
assertThat(toByteArraySupplier(file.toFile()).get()).isEqualTo("some data".getBytes(UTF_8));
}
private ImmutableMap<String, String> readAllFiles(FilesWithEntryPoint reportFiles) {
return reportFiles.files().entrySet().stream()
.collect(
toImmutableMap(
entry -> entry.getKey().toString(),
entry -> new String(entry.getValue().get(), UTF_8)));
}
@Test
void testCreateReportFiles_destinationIsFile() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
Path somePath = createDirectories(root.resolve("some/path"));
Path destination = createFile(somePath.resolve("file.txt"));
Files.write(destination, "some data".getBytes(UTF_8));
// Since the entry point is obvious here - any hint given is just ignored.
File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination.toFile(), Optional.of(ignoredHint), root);
assertThat(files.entryPoint().toString())
.isEqualTo(filenameJoiner.join("some", "path", "file.txt"));
assertThat(readAllFiles(files))
.containsExactly(filenameJoiner.join("some", "path", "file.txt"), "some data");
}
@Test
void testCreateReportFiles_destinationDoesntExist() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
File destination = root.resolve("non/existing.txt").toFile();
assertThat(destination.isFile()).isFalse();
assertThat(destination.isDirectory()).isFalse();
// Since there are no files, any hint given is obviously wrong and will be ignored.
File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination, Optional.of(ignoredHint), root);
assertThat(files.entryPoint().toString()).isEqualTo(filenameJoiner.join("non", "existing.txt"));
assertThat(files.files()).isEmpty();
}
@Test
void testCreateReportFiles_noFiles() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
Path destination = createDirectories(root.resolve("some/path"));
createDirectories(destination.resolve("a/b"));
createDirectory(destination.resolve("c"));
// Since there are not files, any hint given is obviously wrong and will be ignored.
File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination.toFile(), Optional.of(ignoredHint), root);
assertThat(files.entryPoint().toString()).isEqualTo(filenameJoiner.join("some", "path"));
assertThat(files.files()).isEmpty();
}
@Test
void testCreateReportFiles_oneFile() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
Path destination = createDirectories(root.resolve("some/path"));
createDirectories(destination.resolve("a/b"));
createDirectory(destination.resolve("c"));
Files.write(createFile(destination.resolve("a/file.txt")), "some data".getBytes(UTF_8));
// Since the entry point is obvious here - any hint given is just ignored.
File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination.toFile(), Optional.of(ignoredHint), root);
assertThat(files.entryPoint().toString())
.isEqualTo(filenameJoiner.join("some", "path", "a", "file.txt"));
assertThat(readAllFiles(files))
.containsExactly(filenameJoiner.join("some", "path", "a", "file.txt"), "some data");
}
/**
* Currently tests the "unimplemented" behavior.
*
* <p>TODO(guyben): switch to checking zip file instead.
*/
@Test
void testCreateReportFiles_multipleFiles_noHint() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
Path destination = createDirectories(root.resolve("some/path"));
createDirectories(destination.resolve("a/b"));
createDirectory(destination.resolve("c"));
Files.write(createFile(destination.resolve("index.html")), "some data".getBytes(UTF_8));
Files.write(createFile(destination.resolve("a/index.html")), "wrong index".getBytes(UTF_8));
Files.write(createFile(destination.resolve("c/style.css")), "css file".getBytes(UTF_8));
Files.write(createFile(destination.resolve("my_image.png")), "images".getBytes(UTF_8));
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination.toFile(), Optional.empty(), root);
assertThat(files.entryPoint().toString())
.isEqualTo(filenameJoiner.join("some", "path", "path.zip"));
assertThat(readAllFiles(files).keySet())
.containsExactly(filenameJoiner.join("some", "path", "path.zip"));
}
/**
* Currently tests the "unimplemented" behavior.
*
* <p>TODO(guyben): switch to checking zip file instead.
*/
@Test
void testCreateReportFiles_multipleFiles_withBadHint() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
Path destination = createDirectories(root.resolve("some/path"));
// This entry point points to a directory, which isn't an appropriate entry point
File badEntryPoint = createDirectories(destination.resolve("a/b")).toFile();
createDirectory(destination.resolve("c"));
Files.write(createFile(destination.resolve("index.html")), "some data".getBytes(UTF_8));
Files.write(createFile(destination.resolve("a/index.html")), "wrong index".getBytes(UTF_8));
Files.write(createFile(destination.resolve("c/style.css")), "css file".getBytes(UTF_8));
Files.write(createFile(destination.resolve("my_image.png")), "images".getBytes(UTF_8));
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination.toFile(), Optional.of(badEntryPoint), root);
assertThat(files.entryPoint().toString())
.isEqualTo(filenameJoiner.join("some", "path", "path.zip"));
assertThat(readAllFiles(files).keySet())
.containsExactly(filenameJoiner.join("some", "path", "path.zip"));
}
@Test
void testCreateReportFiles_multipleFiles_withGoodHint() throws Exception {
Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
Path destination = createDirectories(root.resolve("some/path"));
createDirectories(destination.resolve("a/b"));
createDirectory(destination.resolve("c"));
// The hint is an actual file nested in the destination directory!
Path goodEntryPoint = createFile(destination.resolve("index.html"));
Files.write(goodEntryPoint, "some data".getBytes(UTF_8));
Files.write(createFile(destination.resolve("a/index.html")), "wrong index".getBytes(UTF_8));
Files.write(createFile(destination.resolve("c/style.css")), "css file".getBytes(UTF_8));
Files.write(createFile(destination.resolve("my_image.png")), "images".getBytes(UTF_8));
FilesWithEntryPoint files =
readFilesWithEntryPoint(destination.toFile(), Optional.of(goodEntryPoint.toFile()), root);
assertThat(files.entryPoint().toString())
.isEqualTo(filenameJoiner.join("some", "path", "index.html"));
assertThat(readAllFiles(files))
.containsExactly(
filenameJoiner.join("some", "path", "index.html"), "some data",
filenameJoiner.join("some", "path", "a", "index.html"), "wrong index",
filenameJoiner.join("some", "path", "c", "style.css"), "css file",
filenameJoiner.join("some", "path", "my_image.png"), "images");
}
}

7
common/README.md Normal file
View File

@@ -0,0 +1,7 @@
## Summary
This project holds some of the general-purpose utility classes that do not rely
on the registry domain model.
This is an intermediate step in untangling the circular dependencies
between :core and :util subprojects.

66
common/build.gradle Normal file
View File

@@ -0,0 +1,66 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
sourceSets {
testing {
java {
compileClasspath += main.output
runtimeClasspath += main.output
}
}
test {
java {
compileClasspath += testing.output
runtimeClasspath += testing.output
}
}
}
configurations {
testingCompile.extendsFrom compile
testingRuntime.extendsFrom runtime
testCompile.extendsFrom testingCompile
testRuntime.extendsFrom testingRuntime
// All testing util classes. Other projects may declare dependency as:
// testCompile project(path: 'common', configuration: 'testing')
testing
}
task testingJar(type: Jar) {
archiveBaseName = 'testing'
from sourceSets.testing.output
}
artifacts {
testing testingJar
}
dependencies {
def deps = rootProject.dependencyMap
compile deps['com.google.code.findbugs:jsr305']
compile deps['com.google.guava:guava']
compile deps['javax.inject:javax.inject']
compile deps['joda-time:joda-time']
testingCompile deps['com.google.flogger:flogger']
testingRuntime deps['com.google.flogger:flogger-system-backend']
testingCompile deps['com.google.truth:truth']
testingCompile deps['io.github.java-diff-utils:java-diff-utils']
testCompile deps['org.junit.jupiter:junit-jupiter-api']
testCompile deps['org.junit.jupiter:junit-jupiter-engine']
}

View File

@@ -3,16 +3,16 @@
# This file is expected to be part of source control.
antlr:antlr:2.7.7
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.2.0
com.google.errorprone:error_prone_annotations:2.3.2
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.0.1-jre
com.google.guava:guava:28.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.puppycrawl.tools:checkstyle:8.17
commons-beanutils:commons-beanutils:1.9.3
com.google.j2objc:j2objc-annotations:1.3
com.puppycrawl.tools:checkstyle:8.27
commons-beanutils:commons-beanutils:1.9.4
commons-collections:commons-collections:3.2.2
info.picocli:picocli:3.9.0
net.sf.saxon:Saxon-HE:9.9.0-2
info.picocli:picocli:4.1.1
net.sf.saxon:Saxon-HE:9.9.1-5
org.antlr:antlr4-runtime:4.7.2
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.checkerframework:checker-qual:2.8.1
org.codehaus.mojo:animal-sniffer-annotations:1.18

View File

@@ -0,0 +1,12 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
org.checkerframework:checker-qual:2.11.1

View File

@@ -0,0 +1,12 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
org.checkerframework:checker-qual:2.11.1

View File

@@ -0,0 +1,12 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
org.checkerframework:checker-qual:2.11.1

View File

@@ -0,0 +1,12 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
org.checkerframework:checker-qual:2.11.1

View File

@@ -0,0 +1,24 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.github.kevinstern:software-and-algorithms:1.0
com.github.stephenc.jcip:jcip-annotations:1.0-1
com.google.auto:auto-common:0.10
com.google.code.findbugs:jFormatString:3.0.0
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotation:2.3.3
com.google.errorprone:error_prone_annotations:2.3.3
com.google.errorprone:error_prone_check_api:2.3.3
com.google.errorprone:error_prone_core:2.3.3
com.google.errorprone:error_prone_type_annotations:2.3.3
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.0.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.google.protobuf:protobuf-java:3.4.0
com.googlecode.java-diff-utils:diffutils:1.3.0
org.checkerframework:checker-qual:2.5.3
org.checkerframework:dataflow:2.5.3
org.checkerframework:javacutil:2.5.3
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.pcollections:pcollections:2.1.2

View File

@@ -0,0 +1,4 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
org.jacoco:org.jacoco.agent:0.8.5

View File

@@ -0,0 +1,11 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
org.jacoco:org.jacoco.agent:0.8.5
org.jacoco:org.jacoco.ant:0.8.5
org.jacoco:org.jacoco.core:0.8.5
org.jacoco:org.jacoco.report:0.8.5
org.ow2.asm:asm-analysis:7.2
org.ow2.asm:asm-commons:7.2
org.ow2.asm:asm-tree:7.2
org.ow2.asm:asm:7.2

View File

@@ -0,0 +1,12 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
org.checkerframework:checker-qual:2.11.1

View File

@@ -0,0 +1,12 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
org.checkerframework:checker-qual:2.11.1

View File

@@ -0,0 +1,24 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.github.kevinstern:software-and-algorithms:1.0
com.github.stephenc.jcip:jcip-annotations:1.0-1
com.google.auto:auto-common:0.10
com.google.code.findbugs:jFormatString:3.0.0
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotation:2.3.3
com.google.errorprone:error_prone_annotations:2.3.3
com.google.errorprone:error_prone_check_api:2.3.3
com.google.errorprone:error_prone_core:2.3.3
com.google.errorprone:error_prone_type_annotations:2.3.3
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.0.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.google.protobuf:protobuf-java:3.4.0
com.googlecode.java-diff-utils:diffutils:1.3.0
org.checkerframework:checker-qual:2.5.3
org.checkerframework:dataflow:2.5.3
org.checkerframework:javacutil:2.5.3
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.pcollections:pcollections:2.1.2

View File

@@ -0,0 +1,28 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.opentest4j:opentest4j:1.2.0

View File

@@ -0,0 +1,28 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.opentest4j:opentest4j:1.2.0

View File

@@ -0,0 +1,29 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger-system-backend:0.1
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.opentest4j:opentest4j:1.2.0

View File

@@ -0,0 +1,29 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger-system-backend:0.1
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.apiguardian:apiguardian-api:1.1.0
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3
org.junit.jupiter:junit-jupiter-api:5.6.2
org.junit.jupiter:junit-jupiter-engine:5.6.2
org.junit.platform:junit-platform-commons:1.6.2
org.junit.platform:junit-platform-engine:1.6.2
org.junit:junit-bom:5.6.2
org.opentest4j:opentest4j:1.2.0

View File

@@ -0,0 +1,24 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.github.kevinstern:software-and-algorithms:1.0
com.github.stephenc.jcip:jcip-annotations:1.0-1
com.google.auto:auto-common:0.10
com.google.code.findbugs:jFormatString:3.0.0
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotation:2.3.3
com.google.errorprone:error_prone_annotations:2.3.3
com.google.errorprone:error_prone_check_api:2.3.3
com.google.errorprone:error_prone_core:2.3.3
com.google.errorprone:error_prone_type_annotations:2.3.3
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.0.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.google.protobuf:protobuf-java:3.4.0
com.googlecode.java-diff-utils:diffutils:1.3.0
org.checkerframework:checker-qual:2.5.3
org.checkerframework:dataflow:2.5.3
org.checkerframework:javacutil:2.5.3
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.pcollections:pcollections:2.1.2

View File

@@ -0,0 +1,21 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3

View File

@@ -0,0 +1,21 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3

View File

@@ -0,0 +1,22 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger-system-backend:0.1
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3

View File

@@ -0,0 +1,22 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.google.auto.value:auto-value-annotations:1.6.3
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger-system-backend:0.1
com.google.flogger:flogger:0.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.truth:truth:1.0
com.googlecode.java-diff-utils:diffutils:1.3.0
io.github.java-diff-utils:java-diff-utils:4.0
javax.inject:javax.inject:1
joda-time:joda-time:2.9.2
junit:junit:4.12
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
org.hamcrest:hamcrest-core:1.3

View File

@@ -19,6 +19,9 @@ import static com.google.common.base.Preconditions.checkArgument;
import com.google.common.collect.Iterables;
import com.google.common.collect.Lists;
import com.google.common.collect.Ordering;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.util.TimeZone;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
@@ -86,4 +89,23 @@ public class DateTimeUtils {
checkArgument(years >= 0);
return years == 0 ? now : now.minusYears(1).minusYears(years - 1);
}
/**
* Converts a Joda {@link DateTime} object to an equivalent java.time {@link ZonedDateTime}
* object.
*/
public static ZonedDateTime toZonedDateTime(DateTime dateTime) {
java.time.Instant instant = java.time.Instant.ofEpochMilli(dateTime.getMillis());
return ZonedDateTime.ofInstant(instant, ZoneId.of(dateTime.getZone().getID()).normalized());
}
/**
* Converts a java.time {@link ZonedDateTime} object to an equivalent Joda {@link DateTime}
* object.
*/
public static DateTime toJodaDateTime(ZonedDateTime zonedDateTime) {
return new DateTime(
zonedDateTime.toInstant().toEpochMilli(),
DateTimeZone.forTimeZone(TimeZone.getTimeZone(zonedDateTime.getZone())));
}
}

View File

@@ -21,7 +21,6 @@ import org.joda.time.ReadableDuration;
* An object which accepts requests to put the current thread to sleep.
*
* @see SystemSleeper
* @see google.registry.testing.FakeSleeper
*/
@ThreadSafe
public interface Sleeper {

View File

@@ -0,0 +1,104 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.testing.truth;
import static com.google.common.io.Resources.getResource;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.testing.truth.TextDiffSubject.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableList;
import com.google.common.io.Resources;
import google.registry.testing.truth.TextDiffSubject.DiffFormat;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.junit.jupiter.api.Test;
/** Unit tests for {@link TextDiffSubject}. */
class TextDiffSubjectTest {
private static final String RESOURCE_FOLDER = "google/registry/testing/truth/";
// Resources for input data.
private static final String ACTUAL_RESOURCE = RESOURCE_FOLDER + "text-diff-actual.txt";
private static final String EXPECTED_RESOURCE = RESOURCE_FOLDER + "text-diff-expected.txt";
// Resources for expected diff texts.
private static final String UNIFIED_DIFF_RESOURCE = RESOURCE_FOLDER + "text-unified-diff.txt";
private static final String SIDE_BY_SIDE_DIFF_RESOURCE =
RESOURCE_FOLDER + "text-sidebyside-diff.txt";
@Test
void unifiedDiff_equal() throws IOException {
assertThat(getResource(ACTUAL_RESOURCE))
.withDiffFormat(DiffFormat.UNIFIED_DIFF)
.hasSameContentAs(getResource(ACTUAL_RESOURCE));
}
@Test
void sideBySideDiff_equal() throws IOException {
assertThat(getResource(ACTUAL_RESOURCE))
.withDiffFormat(DiffFormat.SIDE_BY_SIDE_MARKDOWN)
.hasSameContentAs(getResource(ACTUAL_RESOURCE));
}
@Test
void unifedDiff_notEqual() throws IOException {
assertThrows(
AssertionError.class,
() ->
assertThat(getResource(ACTUAL_RESOURCE))
.withDiffFormat(DiffFormat.UNIFIED_DIFF)
.hasSameContentAs(getResource(EXPECTED_RESOURCE)));
}
@Test
void sideBySideDiff_notEqual() throws IOException {
assertThrows(
AssertionError.class,
() ->
assertThat(getResource(ACTUAL_RESOURCE))
.withDiffFormat(DiffFormat.SIDE_BY_SIDE_MARKDOWN)
.hasSameContentAs(getResource(EXPECTED_RESOURCE)));
}
@Test
void displayed_unifiedDiff_noDiff() throws IOException {
ImmutableList<String> actual = readAllLinesFromResource(ACTUAL_RESOURCE);
assertThat(TextDiffSubject.generateUnifiedDiff(actual, actual)).isEqualTo("");
}
@Test
void displayed_unifiedDiff_hasDiff() throws IOException {
ImmutableList<String> actual = readAllLinesFromResource(ACTUAL_RESOURCE);
ImmutableList<String> expected = readAllLinesFromResource(EXPECTED_RESOURCE);
String diff = Joiner.on('\n').join(readAllLinesFromResource(UNIFIED_DIFF_RESOURCE));
assertThat(TextDiffSubject.generateUnifiedDiff(expected, actual)).isEqualTo(diff);
}
@Test
void displayed_sideBySideDiff_hasDiff() throws IOException {
ImmutableList<String> actual = readAllLinesFromResource(ACTUAL_RESOURCE);
ImmutableList<String> expected = readAllLinesFromResource(EXPECTED_RESOURCE);
String diff = Joiner.on('\n').join(readAllLinesFromResource(SIDE_BY_SIDE_DIFF_RESOURCE));
assertThat(TextDiffSubject.generateSideBySideDiff(expected, actual)).isEqualTo(diff);
}
private static ImmutableList<String> readAllLinesFromResource(String resourceName)
throws IOException {
return ImmutableList.copyOf(
Resources.readLines(getResource(resourceName), StandardCharsets.UTF_8));
}
}

View File

@@ -0,0 +1,2 @@
This is a random file,
with two lines and terminates with a newline.

View File

@@ -0,0 +1,3 @@
This is a random file,
with three lines and terminates without a newline.

View File

@@ -0,0 +1,5 @@
|Expected |Actual |
|------------------------------------------------------|-----------------------------------------------------|
|This is a random file, |This is a random file, |
| |with **two** lines and terminates **with** a newline.|
|with ~three~ lines and terminates ~without~ a newline.| |

View File

@@ -0,0 +1,6 @@
--- expected
+++ actual
@@ -2,2 +2,1 @@
-
-with three lines and terminates without a newline.
+with two lines and terminates with a newline.

View File

@@ -0,0 +1,223 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.testing.truth;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.truth.Truth.assertAbout;
import static com.google.common.truth.Truth.assertWithMessage;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.github.difflib.DiffUtils;
import com.github.difflib.UnifiedDiffUtils;
import com.github.difflib.algorithm.DiffException;
import com.github.difflib.patch.Patch;
import com.github.difflib.text.DiffRow;
import com.github.difflib.text.DiffRowGenerator;
import com.google.common.base.Ascii;
import com.google.common.base.Joiner;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.io.Resources;
import com.google.common.truth.Fact;
import com.google.common.truth.FailureMetadata;
import com.google.common.truth.SimpleSubjectBuilder;
import com.google.common.truth.Subject;
import java.io.IOException;
import java.net.URL;
import java.util.Collection;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
/**
* Compares two multi-line text blocks, and displays their diffs in readable formats.
*
* <p>User may choose one of the following diff formats:
*
* <ul>
* <li>{@link DiffFormat#UNIFIED_DIFF} displays the differences in the unified-diff format
* <li>{@link DiffFormat#SIDE_BY_SIDE_MARKDOWN} displays the two text blocks side by side, with
* markdown annotations to highlight the differences.
* </ul>
*
* <p>Note that if one text block has one trailing newline at the end while another has none, this
* difference will not be shown in the generated diffs. This is the case where two texts may be
* reported as unequal but the diffs appear equal. Fixing this requires special treatment of the
* last line of text. The fix would not be useful in our environment, where all important files are
* covered by a style checker that ensures the presence of a trailing newline.
*/
// TODO(weiminyu): move this class and test to a standalone 'testing' project. Note that the util
// project is not good since it depends back to core.
public class TextDiffSubject extends Subject {
private final ImmutableList<String> actual;
private DiffFormat diffFormat = DiffFormat.SIDE_BY_SIDE_MARKDOWN;
protected TextDiffSubject(FailureMetadata metadata, List<String> actual) {
super(metadata, actual);
this.actual = ImmutableList.copyOf(actual);
}
protected TextDiffSubject(FailureMetadata metadata, URL actual) {
super(metadata, actual);
try {
this.actual = ImmutableList.copyOf(Resources.asCharSource(actual, UTF_8).readLines());
} catch (IOException e) {
throw new RuntimeException(e);
}
}
public TextDiffSubject withDiffFormat(DiffFormat format) {
this.diffFormat = format;
return this;
}
public void hasSameContentAs(List<String> expectedContent) {
checkNotNull(expectedContent, "expectedContent");
ImmutableList<String> expected = ImmutableList.copyOf(expectedContent);
if (expected.equals(actual)) {
return;
}
String diffString = diffFormat.generateDiff(expected, actual);
failWithoutActual(
Fact.simpleFact(
Joiner.on('\n')
.join(
"Files differ in content. Displaying " + Ascii.toLowerCase(diffFormat.name()),
diffString)));
}
public void hasSameContentAs(URL resourceUrl) throws IOException {
hasSameContentAs(Resources.asCharSource(resourceUrl, UTF_8).readLines());
}
public static TextDiffSubject assertThat(List<String> actual) {
return assertAbout(textFactory()).that(ImmutableList.copyOf(checkNotNull(actual, "actual")));
}
public static TextDiffSubject assertThat(URL resourceUrl) throws IOException {
return assertThat(Resources.asCharSource(resourceUrl, UTF_8).readLines());
}
public static SimpleSubjectBuilder<TextDiffSubject, URL> assertWithMessageAboutUrlSource(
String format, Object... params) {
return assertWithMessage(format, params).about(urlFactory());
}
private static final Subject.Factory<TextDiffSubject, ImmutableList<String>>
TEXT_DIFF_SUBJECT_TEXT_FACTORY = TextDiffSubject::new;
public static Subject.Factory<TextDiffSubject, ImmutableList<String>> textFactory() {
return TEXT_DIFF_SUBJECT_TEXT_FACTORY;
}
private static final Subject.Factory<TextDiffSubject, URL> TEXT_DIFF_SUBJECT_URL_FACTORY =
TextDiffSubject::new;
public static Subject.Factory<TextDiffSubject, URL> urlFactory() {
return TEXT_DIFF_SUBJECT_URL_FACTORY;
}
static String generateUnifiedDiff(
ImmutableList<String> expectedContent, ImmutableList<String> actualContent) {
Patch<String> diff;
try {
diff = DiffUtils.diff(expectedContent, actualContent);
} catch (DiffException e) {
throw new RuntimeException(e);
}
List<String> unifiedDiff =
UnifiedDiffUtils.generateUnifiedDiff("expected", "actual", expectedContent, diff, 0);
return Joiner.on('\n').join(unifiedDiff);
}
static String generateSideBySideDiff(
ImmutableList<String> expectedContent, ImmutableList<String> actualContent) {
DiffRowGenerator generator =
DiffRowGenerator.create()
.showInlineDiffs(true)
.inlineDiffByWord(true)
.oldTag(f -> "~")
.newTag(f -> "**")
.build();
List<DiffRow> rows;
try {
rows = generator.generateDiffRows(expectedContent, actualContent);
} catch (DiffException e) {
throw new RuntimeException(e);
}
int maxExpectedLineLength =
findMaxLineLength(rows.stream().map(DiffRow::getOldLine).collect(Collectors.toList()));
int maxActualLineLength =
findMaxLineLength(rows.stream().map(DiffRow::getNewLine).collect(Collectors.toList()));
SideBySideRowFormatter sideBySideRowFormatter =
new SideBySideRowFormatter(maxExpectedLineLength, maxActualLineLength);
return Joiner.on('\n')
.join(
sideBySideRowFormatter.formatRow("Expected", "Actual", ' '),
sideBySideRowFormatter.formatRow("", "", '-'),
rows.stream()
.map(
row ->
sideBySideRowFormatter.formatRow(row.getOldLine(), row.getNewLine(), ' '))
.toArray());
}
private static int findMaxLineLength(Collection<String> lines) {
return lines.stream()
.max(Comparator.comparingInt(String::length))
.map(String::length)
.orElse(0);
}
private static class SideBySideRowFormatter {
private final int maxExpectedLineLength;
private final int maxActualLineLength;
private SideBySideRowFormatter(int maxExpectedLineLength, int maxActualLineLength) {
this.maxExpectedLineLength = maxExpectedLineLength;
this.maxActualLineLength = maxActualLineLength;
}
public String formatRow(String expected, String actual, char padChar) {
return String.format(
"|%s|%s|",
Strings.padEnd(expected, maxExpectedLineLength, padChar),
Strings.padEnd(actual, maxActualLineLength, padChar));
}
}
/** The format used to display diffs when two text blocks are different. */
public enum DiffFormat {
UNIFIED_DIFF {
@Override
String generateDiff(ImmutableList<String> expected, ImmutableList<String> actual) {
return generateUnifiedDiff(expected, actual);
}
},
SIDE_BY_SIDE_MARKDOWN {
@Override
String generateDiff(ImmutableList<String> expected, ImmutableList<String> actual) {
return generateSideBySideDiff(expected, actual);
}
};
abstract String generateDiff(ImmutableList<String> expected, ImmutableList<String> actual);
}
}

View File

@@ -25,6 +25,60 @@ by Joshua Bloch in his book Effective Java -->
<property name="message" value='TODO is preferred to FIXME. e.g. "TODO(johndoe): Refactor when v2 is released."' />
</module>
<!-- Checks that String.toUpper/LowerCase() is never used without locale. -->
<module name="RegexpSingleline">
<property name="format" value="\.to(Upper|Lower)Case\(\)"/>
<property name="message" value="String.toUpper/LowerCase() can have unexpected results depending on locale. Either set the locale explicitly - e.g., toUpperCase(Locale.ENGLISH) - or use Ascii.toUpper/LowerCase(...) instead." />
</module>
<!-- Checks that DateTime.now(...) is always passed a DateTimeZone parameter. -->
<module name="RegexpSingleline">
<property name="format" value="DateTime\.now\(\)"/>
<property name="message" value="DateTime.now() must be called with a DateTimeZone parameter, e.g. DateTime.now(UTC)" />
</module>
<!-- Checks that Javadoc does not include a malformed @see tag. -->
<module name="RegexpSingleline">
<property name="format" value='@see\s\"http(|s)://'/>
<property name="message" value='Your Javadocs appear to use invalid &lt;a&gt; tag syntax in @see tags. Please use the correct syntax: @see &lt;a href="http(s)://your_url"&gt;url_description&lt;/a&gt;'/>
</module>
<!-- Checks that our Ofy wrapper is used instead of the "real" ofy(). -->
<module name="RegexpSingleline">
<property name="format" value="com\.googlecode\.objectify\.ObjectifyService\.ofy"/>
<property name="message" value="Use google.registry.model.ofy.ObjectifyService.ofy(). Do not use com.googlecode.objectify.v4.ObjectifyService.ofy()."/>
</module>
<!-- Checks that java.util.Optional is used instead of Guava's Optional. -->
<module name="RegexpSingleline">
<property name="format" value="com\.google\.common\.base\.Optional"/>
<property name="message" value="Use java.util.Optional instead of Guava's Optional."/>
</module>
<!-- Checks that the deprecated ExpectedException is not used. -->
<module name="RegexpSingleline">
<property name="format" value="org\.junit\.rules\.ExpectedException"/>
<property name="message" value="Use assertThrows and expectThrows from JUnitBackports instead of the deprecated methods on ExpectedException."/>
</module>
<!-- Checks that the deprecated MockitoJUnitRunner is not used. -->
<module name="RegexpSingleline">
<property name="format" value="MockitoJUnitRunner"/>
<property name="message" value="MockitoJUnitRunner is deprecated. Use @RunWith(JUnit4.class) and MockitoRule instead."/>
</module>
<module name="LineLength">
<!-- Checks if a line is too long. -->
<property name="max" value="${com.puppycrawl.tools.checkstyle.checks.sizes.LineLength.max}" default="100"/>
<property name="severity" value="error"/>
<!-- Ignore lines that have any series of 80 or more non-whitespace characters.
These lines likely cannot be broken.
-->
<property name="ignorePattern"
value="${com.puppycrawl.tools.checkstyle.checks.sizes.LineLength.ignorePattern}"
default=".*[^ ]{80,}.*"/>
</module>
<!-- All Java AST specific tests live under TreeWalker module. -->
<module name="TreeWalker">
@@ -137,19 +191,6 @@ by Joshua Bloch in his book Effective Java -->
LENGTH and CODING CHECKS
-->
<module name="LineLength">
<!-- Checks if a line is too long. -->
<property name="max" value="${com.puppycrawl.tools.checkstyle.checks.sizes.LineLength.max}" default="100"/>
<property name="severity" value="error"/>
<!-- Ignore lines that have any series of 80 or more non-whitespace characters.
These lines likely cannot be broken.
-->
<property name="ignorePattern"
value="${com.puppycrawl.tools.checkstyle.checks.sizes.LineLength.ignorePattern}"
default=".*[^ ]{80,}.*"/>
</module>
<module name="OperatorWrap">
<property name="option" value="nl"/>
<property name="tokens" value="QUESTION, EQUAL, NOT_EQUAL, DIV, PLUS, MINUS, STAR, MOD, SR, BSR, GE, GT, SL, LE, LT, BXOR, BOR, LOR, BAND, LAND, TYPE_EXTENSION_AND, LITERAL_INSTANCEOF"/>

View File

@@ -8,5 +8,7 @@
<!-- Ignore generated files in Checkstyle -->
<suppress files="[/\\].*[/\\]generated.*[/\\]" checks="."/>
<!-- Ignore Javadoc checks in test files -->
<suppress files="[/\\].*[/\\]javatests.*[/\\]" checks="JavadocType"/>
<suppress files="[/\\].*[/\\]src/test/java/.*[/\\]" checks="JavadocType"/>
<!-- ofy() regex check doesn't apply to these files -->
<suppress files="AugmentedDeleter.java|AugmentedSaver.java|Ofy.java" checks="RegexpSingleline"/>
</suppressions>

View File

@@ -0,0 +1,16 @@
## Summary
This folder contains configuration files for the gradle-license-report plugin:
* allowed_licenses.json declares the acceptable licenses. A license may have
multiple entries in this file, since the 'moduleLicense' property value must
match exactly the phrases found in pom or manifest files.
* license_normalizer_bundle.json configures normalization rules for license
reporting.
## Notes About Adding New Licenses
* The WTFPL license is not allowed.
* Each 'Public Domain' license entry must include a specific 'moduleName'. Do
not omit moduleName or use wildcards.

View File

@@ -0,0 +1,242 @@
{
"allowedLicenses": [
{
"moduleLicense": "Apache Software License, Version 1.1"
},
{
"moduleLicense": "Apache Software License, version 1.1"
},
{
"moduleLicense": "Apache 2"
},
{
"moduleLicense": "Apache 2.0"
},
{
"moduleLicense": "Apache-2.0"
},
{
"moduleLicense": "Apache License"
},
{
"moduleLicense": "Apache License 2.0"
},
{
"moduleLicense": "Apache License v2.0"
},
{
"moduleLicense": "Apache License, Version 2.0"
},
{
"moduleLicense": "Apache Software License - Version 2.0"
},
{
"moduleLicense": "The Apache License, Version 2.0"
},
{
"moduleLicense": "The Apache Software License, Version 2.0"
},
{
"moduleLicense": "The Apache Software License, version 2.0"
},
{
"moduleLicense": "Bouncy Castle Licence"
},
{
"moduleLicense": "BSD"
},
{
"moduleLicense": "BSD-2-Clause"
},
{
"moduleLicense": "BSD 2-Clause License"
},
{
"moduleLicense": "BSD 2-Clause license"
},
{
"moduleLicense": "BSD 3-clause"
},
{
"moduleLicense": "BSD 3-Clause"
},
{
"moduleLicense": "BSD 3-Clause License"
},
{
"moduleLicense": "BSD 3-clause New License"
},
{
"moduleLicense": "3-Clause BSD License"
},
{
"moduleLicense": "The 3-Clause BSD License"
},
{
"moduleLicense": "BSD License"
},
{
"moduleLicense": "BSD New License"
},
{
"moduleLicense": "BSD New license"
},
{
"moduleLicense": "BSD style"
},
{
"moduleLicense": "BSD-2-Clause"
},
{
"moduleLicense": "New BSD License"
},
{
"moduleLicense": "Revised BSD"
},
{
"moduleLicense": "The BSD License"
},
{
"moduleLicense": "The New BSD License"
},
{
"moduleLicense": "The PostgreSQL License"
},
{
"moduleLicense": "CC0 1.0 Universal License"
},
{
"moduleLicense": "CDDL 1.1"
},
{
"moduleLicense": "CDDL + GPLv2 with classpath exception"
},
{
"moduleLicense": "COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0"
},
{
"moduleLicense": "https://glassfish.java.net/public/CDDL+GPL_1_1.html, https://glassfish.java.net/public/CDDL+GPL_1_1.html"
},
{
"moduleLicense": "CDDL+GPL License"
},
{
"moduleLicense": "\\n Dual license consisting of the CDDL v1.1 and GPL v2\\n "
},
{
"moduleLicense": "Eclipse Distribution License (New BSD License)"
},
{
"moduleLicense": "Eclipse Distribution License v. 1.0"
},
{
"moduleLicense": "Eclipse Public License - Version 1.0"
},
{
"moduleLicense": "Eclipse Public License - v 1.0"
},
{
"moduleLicense": "Eclipse Public License 1.0"
},
{
"moduleLicense": "Eclipse Public License v1.0"
},
{
"moduleLicense": "Eclipse Public License - v 2.0"
},
{
"moduleLicense": "Eclipse Public License 2.0"
},
{
"moduleLicense": "Eclipse Public License v2.0"
},
{
"moduleLicense": "https://www.eclipse.org/legal/epl-2.0/, http://www.gnu.org/copyleft/gpl.html, http://www.gnu.org/licenses/lgpl.html"
},
{
"moduleLicense": "Google App Engine Terms of Service"
},
{
"moduleLicense": "GNU General Public License Version 2"
},
{
"moduleLicense": "GNU General Public License, version 2, with the Classpath Exception"
},
{
"moduleLicense": "GNU GENERAL PUBLIC LICENSE, Version 2 + Classpath Exception"
},
{
"moduleLicense": "GNU Library General Public License v2.1 or later"
},
{
"moduleLicense": "GPL2 w/ CPE"
},
{
"moduleLicense": "GNU Lesser Public License"
},
{
"moduleLicense": "GNU Lesser General Public License Version 2.1"
},
{
"moduleLicense": "GNU Library General Public License v2.1 or later"
},
{
"moduleLicense": "The Go license"
},
{
"moduleLicense": "GWT Terms"
},
{
"moduleLicense": "ICU License"
},
{
"moduleLicense": "The JSON License"
},
{
"moduleLicense": "LGPL-2.1+"
},
{
"moduleLicense": "LGPL, version 2.1"
},
{
"moduleLicense": "MIT"
},
{
"moduleLicense": "MIT License"
},
{
"moduleLicense": "MIT license"
},
{
"moduleLicense": "The MIT License"
},
{
"moduleLicense": "The MIT license"
},
{
"moduleLicense": "The MIT License (MIT)"
},
{
"moduleLicense": "The PostgreSQL License"
},
{
"moduleLicense": "Mozilla Public License Version 2.0"
},
{
"moduleLicense": "Public Domain",
"moduleName": "aopalliance:aopalliance"
},
{
"moduleLicense": "Public Domain",
"moduleName": "org.tukaani:xz"
},
{
// Actually Eclipse Public License v2.0
"moduleLicense": null,
"moduleName": "org.junit:junit-bom"
},
{
"moduleLicense": "The W3C Software License"
}
]
}

View File

@@ -0,0 +1,32 @@
{
"bundles" : [
{
"bundleName" : "Apache 2.0",
"licenseName" : "Apache 2.0",
"licenseUrl" : "http://www.apache.org/licenses/LICENSE-2.0"
},
{
"bundleName" : "MIT License",
"licenseName" : "MIT License",
"licenseUrl" : "http://www.opensource.org/licenses/mit-license.php"
}
],
"transformationRules" : [
{
"bundleName" : "Apache 2.0",
"licenseUrlPattern" : "http://www.apache.org/licenses/LICENSE-2.0"
},
{
"bundleName" : "Apache 2.0",
"licenseUrlPattern" : "http://www.apache.org/licenses/LICENSE-2.0.txt"
},
{
"bundleName" : "MIT License",
"licenseUrlPattern" : "http://www.opensource.org/licenses/mit-license.php"
},
{
"bundleName" : "MIT License",
"licenseUrlPattern" : "http://www.opensource.org/licenses/MIT"
}
]
}

413
config/nom_build.py Normal file
View File

@@ -0,0 +1,413 @@
# Copyright 2020 The Nomulus Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Script to generate dr-build and the properties file.
"""
import argparse
import dataclasses
import io
import os
import shutil
import subprocess
import sys
from typing import List, Union
@dataclasses.dataclass
class Property:
name : str = ''
desc : str = ''
default : str = ''
constraints : type = str
def validate(self, value: str):
"""Verify that "value" is appropriate for the property."""
if type is bool:
if value not in ('true', 'false'):
raise ValidationError('value of {self.name} must be "true" or '
'"false".')
@dataclasses.dataclass
class GradleFlag:
flags : Union[str, List[str]]
desc : str
has_arg : bool = False
PROPERTIES_HEADER = """\
# This file defines properties used by the gradle build. It must be kept in
# sync with config/nom_build.py.
#
# To regenerate, run ./nom_build --generate-gradle-properties
#
# To view property descriptions (which are command line flags for
# nom_build), run ./nom_build --help.
#
# DO NOT EDIT THIS FILE BY HAND
org.gradle.jvmargs=-Xmx1024m
"""
# Help text to be displayed (in addition to the synopsis and flag help, which
# are displayed automatically).
HELP_TEXT = """\
A wrapper around the gradle build that provides the following features:
- Converts properties into flags to guard against property name spelling errors
and to provide help descriptions for all properties.
- Provides pseudo-commands (with the ":nom:" prefix) that encapsulate common
actions that are difficult to implement in gradle.
Pseudo-commands:
:nom:generate_golden_file - regenerates the golden file from the current
set of flyway files.
"""
# Define all of our special gradle properties here.
# TODO(b/169318491): use consistent naming style for properties and variables.
PROPERTIES = [
Property('mavenUrl',
'URL to use for the main maven repository (defaults to maven '
'central). This can be http(s) or a "gcs" repo.'),
Property('pluginsUrl',
'URL to use for the gradle plugins repository (defaults to maven '
'central, see also mavenUrl'),
Property('uploaderDestination',
'Location to upload test reports to. Normally this should be a '
'GCS url (see also uploaderCredentialsFile)'),
Property('uploaderCredentialsFile',
'json credentials file to use to upload test reports.'),
Property('uploaderMultithreadedUpload',
'Whether to enable multithread upload.'),
Property('verboseTestOutput',
'If true, show all test output in near-realtime.',
'false',
bool),
Property('flowDocsFile',
'Output filename for the flowDocsTool command.'),
Property('enableDependencyLocking',
'Enables dependency locking.',
'true',
bool),
Property('enableCrossReferencing',
'generate metadata during java compile (used for kythe source '
'reference generation).',
'false'),
Property('testFilter',
'Comma separated list of test patterns, if specified run only '
'these.'),
Property('environment', 'GAE Environment for deployment and staging.'),
# Cloud SQL properties
Property('dbServer',
'A registry environment name (e.g., "alpha") or a host[:port] '
'string'),
Property('dbName',
'Database name to use in connection.',
'postgres'),
Property('dbUser', 'Database user name for use in connection'),
Property('dbPassword', 'Database password for use in connection'),
Property('publish_repo',
'Maven repository that hosts the Cloud SQL schema jar and the '
'registry server test jars. Such jars are needed for '
'server/schema integration tests. Please refer to <a '
'href="./integration/README.md">integration project</a> for more '
'information.'),
Property('baseSchemaTag',
'The nomulus version tag of the schema for use in the schema'
'deployment integration test (:db:schemaIncrementalDeployTest)'),
Property('schema_version',
'The nomulus version tag of the schema for use in a database'
'integration test.'),
Property('nomulus_version',
'The version of nomulus to test against in a database '
'integration test.'),
Property('dot_path',
'The path to "dot", part of the graphviz package that converts '
'a BEAM pipeline to image. Setting this property to empty string '
'will disable image generation.',
'/usr/bin/dot'),
]
GRADLE_FLAGS = [
GradleFlag(['-a', '--no-rebuild'],
'Do not rebuild project dependencies.'),
GradleFlag(['-b', '--build-file'], 'Specify the build file.', True),
GradleFlag(['--build-cache'],
'Enables the Gradle build cache. Gradle will try to reuse '
'outputs from previous builds.'),
GradleFlag(['-c', '--settings-file'], 'Specify the settings file.', True),
GradleFlag(['--configure-on-demand'],
'Configure necessary projects only. Gradle will attempt to '
'reduce configuration time for large multi-project builds. '
'[incubating]'),
GradleFlag(['--console'],
'Specifies which type of console output to generate. Values '
"are 'plain', 'auto' (default), 'rich' or 'verbose'.",
True),
GradleFlag(['--continue'], 'Continue task execution after a task failure.'),
GradleFlag(['-D', '--system-prop'],
'Set system property of the JVM (e.g. -Dmyprop=myvalue).',
True),
GradleFlag(['-d', '--debug'],
'Log in debug mode (includes normal stacktrace).'),
GradleFlag(['--daemon'],
'Uses the Gradle Daemon to run the build. Starts the Daemon '
'if not running.'),
GradleFlag(['--foreground'], 'Starts the Gradle Daemon in the foreground.'),
GradleFlag(['-g', '--gradle-user-home'],
'Specifies the gradle user home directory.',
True),
GradleFlag(['-I', '--init-script'], 'Specify an initialization script.',
True),
GradleFlag(['-i', '--info'], 'Set log level to info.'),
GradleFlag(['--include-build'],
'Include the specified build in the composite.',
True),
GradleFlag(['-m', '--dry-run'],
'Run the builds with all task actions disabled.'),
GradleFlag(['--max-workers'],
'Configure the number of concurrent workers Gradle is '
'allowed to use.',
True),
GradleFlag(['--no-build-cache'], 'Disables the Gradle build cache.'),
GradleFlag(['--no-configure-on-demand'],
'Disables the use of configuration on demand. [incubating]'),
GradleFlag(['--no-daemon'],
'Do not use the Gradle daemon to run the build. Useful '
'occasionally if you have configured Gradle to always run '
'with the daemon by default.'),
GradleFlag(['--no-parallel'],
'Disables parallel execution to build projects.'),
GradleFlag(['--no-scan'],
'Disables the creation of a build scan. For more information '
'about build scans, please visit '
'https://gradle.com/build-scans.'),
GradleFlag(['--offline'],
'Execute the build without accessing network resources.'),
GradleFlag(['-P', '--project-prop'],
'Set project property for the build script (e.g. '
'-Pmyprop=myvalue).',
True),
GradleFlag(['-p', '--project-dir'],
'Specifies the start directory for Gradle. Defaults to '
'current directory.'),
GradleFlag(['--parallel'],
'Build projects in parallel. Gradle will attempt to '
'determine the optimal number of executor threads to use.'),
GradleFlag(['--priority'],
'Specifies the scheduling priority for the Gradle daemon and '
"all processes launched by it. Values are 'normal' (default) "
"or 'low' [incubating]",
True),
GradleFlag(['--profile'],
'Profile build execution time and generates a report in the '
'<build_dir>/reports/profile directory.'),
GradleFlag(['--project-cache-dir'],
'Specify the project-specific cache directory. Defaults to '
'.gradle in the root project directory.',
True),
GradleFlag(['-q', '--quiet'], 'Log errors only.'),
GradleFlag(['--refresh-dependencies'], 'Refresh the state of dependencies.'),
GradleFlag(['--rerun-tasks'], 'Ignore previously cached task results.'),
GradleFlag(['-S', '--full-stacktrace'],
'Print out the full (very verbose) stacktrace for all '
'exceptions.'),
GradleFlag(['-s', '--stacktrace'],
'Print out the stacktrace for all exceptions.'),
GradleFlag(['--scan'],
'Creates a build scan. Gradle will emit a warning if the '
'build scan plugin has not been applied. '
'(https://gradle.com/build-scans)'),
GradleFlag(['--status'],
'Shows status of running and recently stopped Gradle '
'Daemon(s).'),
GradleFlag(['--stop'], 'Stops the Gradle Daemon if it is running.'),
GradleFlag(['-t', '--continuous'],
'Enables continuous build. Gradle does not exit and will '
're-execute tasks when task file inputs change.'),
GradleFlag(['--update-locks'],
'Perform a partial update of the dependency lock, letting '
'passed in module notations change version. [incubating]'),
GradleFlag(['-v', '--version'], 'Print version info.'),
GradleFlag(['-w', '--warn'], 'Set log level to warn.'),
GradleFlag(['--warning-mode'],
'Specifies which mode of warnings to generate. Values are '
"'all', 'fail', 'summary'(default) or 'none'",
True),
GradleFlag(['--write-locks'],
'Persists dependency resolution for locked configurations, '
'ignoring existing locking information if it exists '
'[incubating]'),
GradleFlag(['-x', '--exclude-task'],
'Specify a task to be excluded from execution.',
True),
]
def generate_gradle_properties() -> str:
"""Returns the expected contents of gradle.properties."""
out = io.StringIO()
out.write(PROPERTIES_HEADER)
for prop in PROPERTIES:
out.write(f'{prop.name}={prop.default}\n')
return out.getvalue()
def get_root() -> str:
"""Returns the root of the nomulus build tree."""
cur_dir = os.getcwd()
if not os.path.exists(os.path.join(cur_dir, '.git')) or \
not os.path.exists(os.path.join(cur_dir, 'core')) or \
not os.path.exists(os.path.join(cur_dir, 'gradle.properties')):
raise Exception('You must run this script from the root directory')
return cur_dir
class Abort(Exception):
"""Raised to terminate the process with a non-zero error code.
Parameters are ignored.
"""
def do_pseudo_task(task: str) -> None:
root = get_root()
if task == ':nom:generate_golden_file':
if not subprocess.call([f'{root}/gradlew', ':db:test']):
print('\033[33mWARNING:\033[0m Golden schema appears to be '
'up-to-date. If you are making schema changes, be sure to '
'add a flyway file for them.')
return
print('\033[33mWARNING:\033[0m Ignore the above failure, it is '
'expected.')
# Copy the new schema into place.
shutil.copy(f'{root}/db/build/resources/test/testcontainer/'
'mount/dump.txt',
f'{root}/db/src/main/resources/sql/schema/'
'nomulus.golden.sql')
if subprocess.call([f'{root}/gradlew', ':db:test']):
print('\033[31mERROR:\033[0m Golden file test failed after '
'copying schema. Please check your flyway files.')
raise Abort()
else:
print(f'\033[31mERROR:\033[0m Unknown task {task}')
raise Abort()
def main(args) -> int:
parser = argparse.ArgumentParser('nom_build', description=HELP_TEXT,
formatter_class=argparse.RawTextHelpFormatter)
for prop in PROPERTIES:
parser.add_argument('--' + prop.name, default=prop.default,
help=prop.desc)
# Add Gradle flags. We set 'dest' to the first flag to get a name that is
# predictable for getattr (even though it will have a leading '-' and thus
# we can't use normal python attribute syntax to get it).
for flag in GRADLE_FLAGS:
if flag.has_arg:
parser.add_argument(*flag.flags, dest=flag.flags[0],
help=flag.desc)
else:
parser.add_argument(*flag.flags, dest=flag.flags[0],
help=flag.desc,
action='store_true')
# Add a flag to regenerate the gradle properties file.
parser.add_argument('--generate-gradle-properties',
help='Regenerate the gradle.properties file. This '
'file must be regenerated when changes are made to '
'config/nom_build.py, and should not be updated by '
'hand.',
action='store_true')
# Consume the remaining non-flag arguments.
parser.add_argument('non_flag_args', nargs='*')
# Parse command line arguments. Note that this exits the program and
# prints usage if either of the help options (-h, --help) are specified.
args = parser.parse_args(args)
gradle_properties = generate_gradle_properties()
root = get_root()
# If we're regenerating properties, do so and exit.
if args.generate_gradle_properties:
with open(f'{root}/gradle.properties', 'w') as dst:
dst.write(gradle_properties)
return 0
# Verify that the gradle properties file is what we expect it to be.
with open(f'{root}/gradle.properties') as src:
if src.read() != gradle_properties:
print('\033[33mWARNING:\033[0m Gradle properties out of sync '
'with nom_build. Run with --generate-gradle-properties '
'to regenerate.')
# Add properties to the gradle argument list.
gradle_command = [f'{root}/gradlew']
for prop in PROPERTIES:
arg_val = getattr(args, prop.name)
if arg_val != prop.default:
prop.validate(arg_val)
gradle_command.extend(['-P', f'{prop.name}={arg_val}'])
# Add Gradle flags to the gradle argument list.
for flag in GRADLE_FLAGS:
arg_val = getattr(args, flag.flags[0])
if arg_val:
gradle_command.append(flag.flags[-1])
if flag.has_arg:
gradle_command.append(arg_val)
# See if there are any special ":nom:" pseudo-tasks specified.
got_non_pseudo_tasks = False
for arg in args.non_flag_args[1:]:
if arg.startswith(':nom:'):
if got_non_pseudo_tasks:
# We can't currently deal with the situation of gradle tasks
# before pseudo-tasks. This could be implemented by invoking
# gradle for only the set of gradle tasks before the pseudo
# task, but that's overkill for now.
print(f'\033[31mERROR:\033[0m Pseudo task ({arg}) must be '
'specified prior to all actual gradle tasks. Aborting.')
return 1
do_pseudo_task(arg)
else:
got_non_pseudo_tasks = True
non_flag_args = [
arg for arg in args.non_flag_args[1:] if not arg.startswith(':nom:')]
if not non_flag_args:
if not got_non_pseudo_tasks:
print('\033[33mWARNING:\033[0m No tasks specified. Not '
'doing anything')
return 0
# Add the non-flag args (we exclude the first, which is the command name
# itself) and run.
gradle_command.extend(non_flag_args)
return subprocess.call(gradle_command)
if __name__ == '__main__':
try:
sys.exit(main(sys.argv))
except Abort as ex:
sys.exit(1)

126
config/nom_build_test.py Normal file
View File

@@ -0,0 +1,126 @@
# Copyright 2020 The Nomulus Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import io
import os
import shutil
import unittest
from unittest import mock
import nom_build
import subprocess
FAKE_PROPERTIES = [
nom_build.Property('foo', 'help text'),
nom_build.Property('bar', 'more text', 'true', bool),
]
FAKE_PROP_CONTENTS = nom_build.PROPERTIES_HEADER + 'foo=\nbar=true\n'
PROPERTIES_FILENAME = '/tmp/rootdir/gradle.properties'
GRADLEW = '/tmp/rootdir/gradlew'
class FileFake(io.StringIO):
"""File fake that writes file contents to the dictionary on close."""
def __init__(self, contents_dict, filename):
self.dict = contents_dict
self.filename = filename
super(FileFake, self).__init__()
def close(self):
self.dict[self.filename] = self.getvalue()
super(FileFake, self).close()
class MyTest(unittest.TestCase):
def open_fake(self, filename, action='r'):
if action == 'r':
return io.StringIO(self.file_contents.get(filename, ''))
elif action == 'w':
result = self.file_contents[filename] = (
FileFake(self.file_contents, filename))
return result
else:
raise Exception(f'Unexpected action {action}')
def print_fake(self, data):
self.printed.append(data)
def setUp(self):
self.addCleanup(mock.patch.stopall)
self.exists_mock = mock.patch.object(os.path, 'exists').start()
self.getcwd_mock = mock.patch.object(os, 'getcwd').start()
self.getcwd_mock.return_value = '/tmp/rootdir'
self.open_mock = (
mock.patch.object(nom_build, 'open', self.open_fake).start())
self.print_mock = (
mock.patch.object(nom_build, 'print', self.print_fake).start())
self.call_mock = mock.patch.object(subprocess, 'call').start()
self.copy_mock = mock.patch.object(shutil, 'copy').start()
self.file_contents = {
# Prefil with the actual file contents.
PROPERTIES_FILENAME: nom_build.generate_gradle_properties()
}
self.printed = []
@mock.patch.object(nom_build, 'PROPERTIES', FAKE_PROPERTIES)
def test_property_generation(self):
self.assertEqual(nom_build.generate_gradle_properties(),
FAKE_PROP_CONTENTS)
@mock.patch.object(nom_build, 'PROPERTIES', FAKE_PROPERTIES)
def test_property_file_write(self):
nom_build.main(['nom_build', '--generate-gradle-properties'])
self.assertEqual(self.file_contents[PROPERTIES_FILENAME],
FAKE_PROP_CONTENTS)
def test_property_file_incorrect(self):
self.file_contents[PROPERTIES_FILENAME] = 'bad contents'
nom_build.main(['nom_build'])
self.assertIn('', self.printed[0])
def test_no_args(self):
nom_build.main(['nom_build'])
self.assertEqual(self.printed,
['\x1b[33mWARNING:\x1b[0m No tasks specified. Not '
'doing anything'])
def test_property_calls(self):
nom_build.main(['nom_build', 'task-name', '--testFilter=foo'])
self.call_mock.assert_called_with([GRADLEW, '-P', 'testFilter=foo',
'task-name'])
def test_gradle_flags(self):
nom_build.main(['nom_build', 'task-name', '-d', '-b', 'foo'])
self.call_mock.assert_called_with([GRADLEW, '--build-file', 'foo',
'--debug', 'task-name'])
def test_generate_golden_file(self):
self.call_mock.side_effect = [1, 0]
nom_build.main(['nom_build', ':nom:generate_golden_file'])
self.call_mock.assert_has_calls([
mock.call([GRADLEW, ':db:test']),
mock.call([GRADLEW, ':db:test'])
])
def test_generate_golden_file_nofail(self):
self.call_mock.return_value = 0
nom_build.main(['nom_build', ':nom:generate_golden_file'])
self.call_mock.assert_has_calls([mock.call([GRADLEW, ':db:test'])])
unittest.main()

291
config/presubmits.py Normal file
View File

@@ -0,0 +1,291 @@
# Copyright 2019 The Nomulus Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Google-style presubmits for the Nomulus project.
These aren't built in to the static code analysis tools we use (e.g. Checkstyle,
Error Prone) so we must write them manually.
"""
import os
from typing import List, Tuple
import sys
import re
# We should never analyze any generated files
UNIVERSALLY_SKIPPED_PATTERNS = {"/build/", "cloudbuild-caches", "/out/", ".git/"}
# We can't rely on CI to have the Enum package installed so we do this instead.
FORBIDDEN = 1
REQUIRED = 2
class PresubmitCheck:
def __init__(self,
regex,
included_extensions,
skipped_patterns,
regex_type=FORBIDDEN):
"""Define a presubmit check for a particular set of files,
The provided prefix should always or never be included in the files.
Args:
regex: the regular expression to forbid or require
included_extensions: a tuple of extensions that define which files
we run over
skipped_patterns: a set of patterns that will cause any files that
include them to be skipped
regex_type: either FORBIDDEN or REQUIRED--whether or not the regex
must be present or cannot be present.
"""
self.regex = regex
self.included_extensions = included_extensions
self.skipped_patterns = UNIVERSALLY_SKIPPED_PATTERNS.union(skipped_patterns)
self.regex_type = regex_type
def fails(self, file):
""" Determine whether or not this file fails the regex check.
Args:
file: the full path of the file to check
"""
if not file.endswith(self.included_extensions):
return False
for pattern in self.skipped_patterns:
if pattern in file:
return False
with open(file, "r") as f:
file_content = f.read()
matches = re.match(self.regex, file_content, re.DOTALL)
if self.regex_type == FORBIDDEN:
return matches
return not matches
PRESUBMITS = {
# License check
PresubmitCheck(
r".*Copyright 20\d{2} The Nomulus Authors\. All Rights Reserved\.",
("java", "js", "soy", "sql", "py", "sh", "gradle"), {
".git", "/build/", "/generated/", "/generated_tests/",
"node_modules/", "JUnitBackports.java", "registrar_bin.",
"registrar_dbg.", "google-java-format-diff.py",
"nomulus.golden.sql", "soyutils_usegoog.js"
}, REQUIRED):
"File did not include the license header.",
# Files must end in a newline
PresubmitCheck(r".*\n$", ("java", "js", "soy", "sql", "py", "sh", "gradle"),
{"node_modules/"}, REQUIRED):
"Source files must end in a newline.",
# System.(out|err).println should only appear in tools/
PresubmitCheck(
r".*\bSystem\.(out|err)\.print", "java", {
"StackdriverDashboardBuilder.java", "/tools/", "/example/",
"RegistryTestServerMain.java", "TestServerExtension.java",
"FlowDocumentationTool.java"
}):
"System.(out|err).println is only allowed in tools/ packages. Please "
"use a logger instead.",
# ObjectifyService.register is restricted to main/ or AppEngineExtension.
PresubmitCheck(
r".*\bObjectifyService\.register", "java", {
"/build/", "/generated/", "node_modules/", "src/main/",
"AppEngineExtension.java"
}):
"ObjectifyService.register(...) is not allowed in tests. Please use "
"AppEngineExtension.register(...) instead.",
# PostgreSQLContainer instantiation must specify docker tag
PresubmitCheck(
r"[\s\S]*new\s+PostgreSQLContainer(<[\s\S]*>)?\(\s*\)[\s\S]*",
"java", {}):
"PostgreSQLContainer instantiation must specify docker tag.",
# Various Soy linting checks
PresubmitCheck(
r".* (/\*)?\* {?@param ",
"soy",
{},
):
"In SOY please use the ({@param name: string} /** User name. */) style"
" parameter passing instead of the ( * @param name User name.) style "
"parameter pasing.",
PresubmitCheck(
r'.*\{[^}]+\w+:\s+"',
"soy",
{},
):
"Please don't use double-quoted string literals in Soy parameters",
PresubmitCheck(
r'.*autoescape\s*=\s*"[^s]',
"soy",
{},
):
"All soy templates must use strict autoescaping",
PresubmitCheck(
r".*noAutoescape",
"soy",
{},
):
"All soy templates must use strict autoescaping",
# various JS linting checks
PresubmitCheck(
r".*goog\.base\(",
"js",
{"/node_modules/"},
):
"Use of goog.base is not allowed.",
PresubmitCheck(
r".*goog\.dom\.classes",
"js",
{"/node_modules/"},
):
"Instead of goog.dom.classes, use goog.dom.classlist which is smaller "
"and faster.",
PresubmitCheck(
r".*goog\.getMsg",
"js",
{"/node_modules/"},
):
"Put messages in Soy, instead of using goog.getMsg().",
PresubmitCheck(
r".*(innerHTML|outerHTML)\s*(=|[+]=)([^=]|$)",
"js",
{"/node_modules/", "registrar_bin."},
):
"Do not assign directly to the dom. Use goog.dom.setTextContent to set"
" to plain text, goog.dom.removeChildren to clear, or "
"soy.renderElement to render anything else",
PresubmitCheck(
r".*console\.(log|info|warn|error)",
"js",
{"/node_modules/", "google/registry/ui/js/util.js", "registrar_bin."},
):
"JavaScript files should not include console logging."
}
# Note that this regex only works for one kind of Flyway file. If we want to
# start using "R" and "U" files we'll need to update this script.
FLYWAY_FILE_RX = re.compile(r'V(\d+)__.*')
def get_seqnum(filename: str, location: str) -> int:
"""Extracts the sequence number from a filename."""
m = FLYWAY_FILE_RX.match(filename)
if m is None:
raise ValueError('Illegal Flyway filename: %s in %s' % (filename, location))
return int(m.group(1))
def files_by_seqnum(files: List[str], location: str) -> List[Tuple[int, str]]:
"""Returns the list of seqnum, filename sorted by sequence number."""
return [(get_seqnum(filename, location), filename) for filename in files]
def has_valid_order(indexed_files: List[Tuple[int, str]], location: str) -> bool:
"""Verify that sequence numbers are in order without gaps or duplicates.
Args:
files: List of seqnum, filename for a list of Flyway files.
location: Where the list of files came from (for error reporting).
Returns:
True if the file list is valid.
"""
last_index = 0
valid = True
for seqnum, filename in indexed_files:
if seqnum == last_index:
print('duplicate Flyway file sequence number found in %s: %s' %
(location, filename))
valid = False
elif seqnum < last_index:
print('File %s in %s is out of order.' % (filename, location))
valid = False
elif seqnum != last_index + 1:
print('Missing Flyway sequence number %d in %s. Next file is %s' %
(last_index + 1, location, filename))
valid = False
last_index = seqnum
return valid
def verify_flyway_index():
"""Verifies that the Flyway index file is in sync with the directory."""
success = True
# Sort the files in the Flyway directory by their sequence number.
files = sorted(
files_by_seqnum(os.listdir('db/src/main/resources/sql/flyway'),
'Flyway directory'))
# Make sure that there are no gaps and no duplicate sequence numbers in the
# files themselves.
if not has_valid_order(files, 'Flyway directory'):
success = False
# Remove the sequence numbers and compare against the index file contents.
files = [filename[1] for filename in sorted(files)]
with open('db/src/main/resources/sql/flyway.txt') as index:
indexed_files = index.read().splitlines()
if files != indexed_files:
unindexed = set(files) - set(indexed_files)
if unindexed:
print('The following Flyway files are not in flyway.txt: %s' % unindexed)
nonexistent = set(indexed_files) - set(files)
if nonexistent:
print('The following files are in flyway.txt but not in the Flyway '
'directory: %s' % nonexistent)
# Do an ordering check on the index file (ignore the result, we're failing
# anyway).
has_valid_order(files_by_seqnum(indexed_files, 'flyway.txt'), 'flyway.txt')
success = False
if not success:
print('Please fix any conflicts and run "./nom_build :db:generateFlywayIndex"')
return not success
def get_files():
for root, dirnames, filenames in os.walk("."):
for filename in filenames:
yield os.path.join(root, filename)
if __name__ == "__main__":
failed = False
for file in get_files():
error_messages = []
for presubmit, error_message in PRESUBMITS.items():
if presubmit.fails(file):
error_messages.append(error_message)
if error_messages:
failed = True
print("%s had errors: \n %s" % (file, "\n ".join(error_messages)))
# And now for something completely different: check to see if the Flyway
# index is up-to-date. It's quicker to do it here than in the unit tests:
# when we put it here it fails fast before all of the tests are run.
failed |= verify_flyway_index()
if failed:
sys.exit(1)

3
core/Dockerfile Normal file
View File

@@ -0,0 +1,3 @@
FROM gcr.io/distroless/java:debug
ADD build/libs/nomulus.jar /nomulus.jar
ENTRYPOINT ["/usr/bin/java", "-jar", "/nomulus.jar"]

Binary file not shown.

View File

@@ -1,16 +1,35 @@
// Copyright 2019 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
import java.lang.reflect.Constructor
import java.util.Optional
plugins {
id 'java-library'
id 'maven-publish'
}
// Path to code generated by ad hoc tasks in this project. A separate path is
// used for easy inspection.
def generatedDir = "${project.buildDir}/generated/source/custom/main"
def generatedDir = "${project.buildDir}/generated/sources/custom/java/main"
def resourcesDir = "${project.buildDir}/resources/main"
def screenshotsDir = "${project.buildDir}/screenshots"
def screenshotsForGoldensDir = "${project.buildDir}/screenshots_for_goldens"
def newGoldensDir = "${project.buildDir}/new_golden_images"
def goldensDir =
"${javatestsDir}/google/registry/webdriver/goldens/chrome-linux"
"${javaTestDir}/google/registry/webdriver/goldens/chrome-linux"
def jsDir = "${project.projectDir}/src/main/javascript"
// Tests that conflict with (mostly unidentified) members of the main test
// suite. It is unclear if they are offenders (i.e., those that pollute global
@@ -28,6 +47,24 @@ def outcastTestPatterns = [
"google/registry/tools/server/CreatePremiumListActionTest.*",
]
// Tests that fail when running Gradle in a docker container, e. g. when
// building the release artifacts in Google Cloud Build.
def dockerIncompatibleTestPatterns = [
// The webdriver tests start headless Chrome in a Docker container,
// resulting in Docker-in-Docker complications. Likewise,
// GenerateSqlSchemaCommandTest and DumpGoldenSchemaCommandTest launch
// postgresql in a docker container.
"google/registry/webdriver/*",
"google/registry/tools/GenerateSqlSchemaCommandTest.*",
"google/registry/tools/DumpGoldenSchemaCommandTest.*",
// PathParameterTest includes tests which validate that file permissions are
// respected. However when running in Docker the user is root by default, so
// every file is read/write-able. There is no way to exclude specific test
// methods, so we exclude the whole test class.
"google/registry/tools/params/PathParameterTest.*",
"google/registry/persistence/PersistenceModuleTest.*",
]
// Tests that conflict with members of both the main test suite and the
// outcast suite. They seem to be affected by global states outside of
// Nomulus classes, e.g., threads and objects retained by frameworks.
@@ -38,21 +75,8 @@ def fragileTestPatterns = [
// Test Datastore inexplicably aborts transaction.
"google/registry/model/tmch/ClaimsListShardTest.*",
// Creates large object (64MBytes), occasionally throws OOM error.
"google/registry/model/server/KmsSecretRevisionTest.*"
]
// Tests that fail when running Gradle in a docker container, e. g. when
// building the release artifacts in Google Cloud Build.
def dockerIncompatibleTestPatterns = [
// The webdriver tests start headless Chrome in a Docker container,
// resulting in Docker-in-Docker complications.
"google/registry/webdriver/*",
// PathParameterTest includes tests which validate that file permissions are
// respected. However when running in Docker the user is root by default, so
// every file is read/write-able. There is no way to exclude specific test
// methods, so we exclude the whole test class.
"google/registry/tools/params/PathParameterTest.*"
]
"google/registry/model/server/KmsSecretRevisionTest.*",
] + dockerIncompatibleTestPatterns
sourceSets {
main {
@@ -63,22 +87,54 @@ sourceSets {
exclude '**/*.xjb'
}
}
nonprod {
java {
compileClasspath += main.output
// Add the DB runtime classpath to nonprod so we can load the flyway
// scripts.
runtimeClasspath += main.output +
rootProject.project(":db").sourceSets.main.runtimeClasspath
}
}
test {
java {
compileClasspath += nonprod.output
runtimeClasspath += nonprod.output
// Javadoc API is deprecated in Java 11 and removed in Java 12.
// TODO(jianglai): re-enable after migrating to the new Javadoc API
if ((JavaVersion.current().majorVersion as Integer) >= 11) {
exclude 'google/registry/documentation/**'
}
}
resources {
exclude '**/*.xjb', '**/*.xsd'
}
}
}
processTestResources {
exclude '**/webdriver/*'
}
configurations {
css
jaxb
soy
// Label for all dependencies inherited from Bazel build but not used in
// either compile or testRuntime. However, they may be needed at runtime.
// TODO(weiminyu): identify runtime dependencies and remove the rest.
maybeRuntime
closureCompiler
devtool
nonprodCompile.extendsFrom compile
nonprodRuntime.extendsFrom runtime
testCompile.extendsFrom nonprodCompile
testRuntime.extendsFrom nonprodRuntime
// Published jars that are used for server/schema compatibility tests.
// See <a href="../integration/README.md">the integration project</a>
// for details.
nomulus_test
// Exclude non-canonical servlet-api jars. Our AppEngine deployment uses
// javax.servlet:servlet-api:2.5
@@ -109,59 +165,44 @@ dependencies {
// Custom-built objectify jar at commit ecd5165, included in Nomulus
// release.
implementation files(
"${rootDir}/../third_party/objectify/v4_1/objectify-4.1.3.jar")
testImplementation project(':third_party')
compile files(
"${rootDir}/third_party/objectify/v4_1/objectify-4.1.3.jar")
testRuntime files(sourceSets.test.resources.srcDirs)
compile deps['com.beust:jcommander']
compile deps['com.google.api-client:google-api-client']
maybeRuntime deps['com.google.api-client:google-api-client-appengine']
maybeRuntime deps['com.google.api-client:google-api-client-jackson2']
compile deps['com.google.api-client:google-api-client-appengine']
compile deps['com.google.api-client:google-api-client-servlet']
compile deps['com.google.monitoring-client:metrics']
compile deps['com.google.monitoring-client:stackdriver']
compile deps['com.google.api-client:google-api-client-java6']
maybeRuntime deps['com.google.api-client:google-api-client-servlet']
compile deps['com.google.apis:google-api-services-admin-directory']
compile deps['com.google.apis:google-api-services-appengine']
compile deps['com.google.apis:google-api-services-bigquery']
maybeRuntime deps['com.google.apis:google-api-services-clouddebugger']
compile deps['com.google.apis:google-api-services-cloudkms']
maybeRuntime deps['com.google.apis:google-api-services-cloudresourcemanager']
compile deps['com.google.apis:google-api-services-dataflow']
compile deps['com.google.apis:google-api-services-dns']
compile deps['com.google.apis:google-api-services-drive']
compile deps['com.google.apis:google-api-services-groupssettings']
compile deps['com.google.apis:google-api-services-monitoring']
compile deps['com.google.apis:google-api-services-sheets']
maybeRuntime deps['com.google.apis:google-api-services-storage']
// TODO(b/71631624): change appengine:appengine-api-1.0-sdk to
// testCompileOnly after BillingEmailUtilsTest.java is fixed.
compile deps['com.google.appengine:appengine-api-1.0-sdk']
maybeRuntime deps['com.google.appengine:appengine-api-labs']
maybeRuntime deps['com.google.appengine:appengine-api-stubs']
testCompile deps['com.google.appengine:appengine-api-stubs']
compile deps['com.google.appengine.tools:appengine-gcs-client']
compile deps['com.google.appengine.tools:appengine-mapreduce']
compile deps['com.google.appengine.tools:appengine-pipeline']
compile deps['com.google.appengine:appengine-remote-api']
maybeRuntime deps['com.google.appengine:appengine-tools-sdk']
compile deps['com.google.auth:google-auth-library-credentials']
compile deps['com.google.auth:google-auth-library-oauth2-http']
maybeRuntime deps['com.google.auto:auto-common']
maybeRuntime deps['com.google.auto.factory:auto-factory']
compile deps['com.google.cloud.sql:jdbc-socket-factory-core']
runtimeOnly deps['com.google.cloud.sql:postgres-socket-factory']
compile deps['com.google.code.gson:gson']
compile deps['com.google.auto.value:auto-value-annotations']
maybeRuntime deps['com.google.cloud.bigdataoss:gcsio']
maybeRuntime deps['com.google.cloud.bigdataoss:util']
compile deps['com.google.code.findbugs:jsr305']
compile deps['com.google.dagger:dagger']
maybeRuntime deps['com.google.dagger:dagger-producers']
compile deps['com.google.errorprone:error_prone_annotations']
maybeRuntime deps['com.google.errorprone:javac-shaded']
compile deps['com.google.flogger:flogger']
runtime deps['com.google.flogger:flogger-system-backend']
maybeRuntime deps['com.google.gdata:core']
maybeRuntime deps['com.google.googlejavaformat:google-java-format']
compile deps['com.google.guava:guava']
gradleLint.ignore('unused-dependency') {
compile deps['com.google.gwt:gwt-user']
@@ -170,95 +211,80 @@ dependencies {
compile deps['com.google.http-client:google-http-client-appengine']
compile deps['com.google.http-client:google-http-client-jackson2']
compile deps['com.google.oauth-client:google-oauth-client']
maybeRuntime deps['com.google.oauth-client:google-oauth-client-appengine']
compile deps['com.google.oauth-client:google-oauth-client-java6']
compile deps['com.google.oauth-client:google-oauth-client-jetty']
maybeRuntime deps['com.google.oauth-client:google-oauth-client-servlet']
maybeRuntime deps['com.google.protobuf:protobuf-java']
compile deps['com.google.oauth-client:google-oauth-client-appengine']
compile deps['com.google.oauth-client:google-oauth-client-servlet']
compile deps['com.google.re2j:re2j']
compile deps['com.google.template:soy']
maybeRuntime deps['com.googlecode.charts4j:charts4j']
compile deps['com.googlecode.json-simple:json-simple']
compile deps['com.jcraft:jsch']
maybeRuntime deps['com.jcraft:jzlib']
maybeRuntime deps['com.squareup:javapoet']
maybeRuntime deps['com.squareup:javawriter']
maybeRuntime deps['com.sun.activation:javax.activation']
maybeRuntime deps['com.thoughtworks.paranamer:paranamer']
testCompile deps['com.thoughtworks.qdox:qdox']
maybeRuntime deps['commons-codec:commons-codec']
maybeRuntime deps['commons-logging:commons-logging']
compile deps['dnsjava:dnsjava']
maybeRuntime deps['io.netty:netty-buffer']
maybeRuntime deps['io.netty:netty-codec']
maybeRuntime deps['io.netty:netty-codec-http']
maybeRuntime deps['io.netty:netty-common']
maybeRuntime deps['io.netty:netty-handler']
maybeRuntime deps['io.netty:netty-resolver']
maybeRuntime deps['io.netty:netty-tcnative']
maybeRuntime deps['io.netty:netty-tcnative-boringssl-static']
maybeRuntime deps['io.netty:netty-transport']
maybeRuntime deps['it.unimi.dsi:fastutil']
maybeRuntime deps['javax.annotation:jsr250-api']
runtime deps['org.glassfish.jaxb:jaxb-runtime']
testCompile deps['io.github.classgraph:classgraph']
testRuntime deps['io.github.java-diff-utils:java-diff-utils']
testCompile deps['javax.annotation:javax.annotation-api']
testCompile deps['javax.annotation:jsr250-api']
compile deps['javax.inject:javax.inject']
compile deps['javax.mail:mail']
compile deps['javax.inject:javax.inject']
compile deps['javax.persistence:javax.persistence-api']
compile deps['javax.servlet:servlet-api']
compile deps['javax.xml.bind:jaxb-api']
maybeRuntime deps['javax.xml.soap:javax.xml.soap-api']
compile deps['jline:jline']
compile deps['joda-time:joda-time']
compile deps['org.apache.avro:avro']
maybeRuntime deps['org.apache.beam:beam-runners-direct-java']
testCompile deps['org.apache.beam:beam-runners-core-construction-java']
testCompile deps['org.apache.beam:beam-runners-direct-java']
compile deps['org.apache.beam:beam-runners-google-cloud-dataflow-java']
maybeRuntime deps['org.apache.beam:beam-sdks-common-runner-api']
compile deps['org.apache.beam:beam-sdks-java-core']
compile deps['org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core']
compile deps['org.apache.beam:beam-sdks-java-io-google-cloud-platform']
maybeRuntime deps['org.apache.commons:commons-compress']
testCompile deps['org.apache.commons:commons-lang3']
testCompile deps['org.apache.commons:commons-text']
maybeRuntime deps['org.apache.ftpserver:ftplet-api']
testCompile deps['org.apache.ftpserver:ftplet-api']
maybeRuntime deps['org.apache.ftpserver:ftpserver-core']
testCompile deps['org.apache.ftpserver:ftpserver-core']
compile deps['org.apache.httpcomponents:httpclient']
compile deps['org.apache.httpcomponents:httpcore']
maybeRuntime deps['org.apache.mina:mina-core']
maybeRuntime deps['org.apache.sshd:sshd-core']
runtime deps['org.apache.logging.log4j:log4j-core']
testCompile deps['org.apache.sshd:sshd-core']
maybeRuntime deps['org.apache.sshd:sshd-scp']
testCompile deps['org.apache.sshd:sshd-scp']
maybeRuntime deps['org.apache.sshd:sshd-sftp']
testCompile deps['org.apache.sshd:sshd-sftp']
maybeRuntime deps['org.apache.tomcat:tomcat-annotations-api']
testCompile deps['org.apache.tomcat:tomcat-annotations-api']
compile deps['org.bouncycastle:bcpg-jdk15on']
testCompile deps['org.bouncycastle:bcpkix-jdk15on']
compile deps['org.bouncycastle:bcprov-jdk15on']
maybeRuntime deps['org.codehaus.jackson:jackson-core-asl']
maybeRuntime deps['org.codehaus.jackson:jackson-mapper-asl']
testCompile deps['com.fasterxml.jackson.core:jackson-databind']
runtime deps['org.glassfish.jaxb:jaxb-runtime']
compile deps['org.hibernate:hibernate-core']
compile deps['org.joda:joda-money']
compile deps['org.json:json']
maybeRuntime deps['org.khronos:opengl-api']
maybeRuntime deps['org.mortbay.jetty:jetty']
testCompile deps['org.mortbay.jetty:jetty']
maybeRuntime deps['org.mortbay.jetty:jetty-util']
compile deps['org.postgresql:postgresql']
testCompile deps['org.seleniumhq.selenium:selenium-api']
testCompile deps['org.seleniumhq.selenium:selenium-chrome-driver']
testCompile deps['org.seleniumhq.selenium:selenium-java']
testCompile deps['org.seleniumhq.selenium:selenium-remote-driver']
maybeRuntime deps['org.slf4j:slf4j-api']
runtimeOnly deps['org.slf4j:slf4j-jdk14']
testCompile deps['org.testcontainers:jdbc']
testCompile deps['org.testcontainers:junit-jupiter']
compile deps['org.testcontainers:postgresql']
testCompile deps['org.testcontainers:selenium']
maybeRuntime deps['org.tukaani:xz']
maybeRuntime deps['org.xerial.snappy:snappy-java']
testCompile deps['org.testcontainers:testcontainers']
compile deps['us.fatehi:schemacrawler']
compile deps['us.fatehi:schemacrawler-diagram']
compile deps['xerces:xmlParserAPIs']
compile deps['xpp3:xpp3']
// This dependency must come after javax.mail:mail as it would otherwise
// shadow classes in package javax.mail with its own implementation.
compile deps['com.google.appengine:appengine-api-1.0-sdk']
// Known issue: nebula-lint misses inherited dependency.
compile project(':third_party')
compile project(':common')
testCompile project(path: ':common', configuration: 'testing')
compile project(':util')
// Import NomulusPostreSql from ':db' for compile but exclude dependencies.
compile project(path: ':db', configuration: 'compileApi')
testRuntime project(':db')
// Include auto-value in compile until nebula-lint understands
// annotationProcessor
@@ -269,6 +295,8 @@ dependencies {
testAnnotationProcessor deps['com.google.auto.value:auto-value']
annotationProcessor deps['com.google.dagger:dagger-compiler']
testAnnotationProcessor deps['com.google.dagger:dagger-compiler']
annotationProcessor project(':processor')
testAnnotationProcessor project(':processor')
testCompile deps['com.google.appengine:appengine-testing']
testCompile deps['com.google.guava:guava-testlib']
@@ -278,8 +306,18 @@ dependencies {
testCompile deps['org.hamcrest:hamcrest-all']
testCompile deps['org.hamcrest:hamcrest-core']
testCompile deps['org.hamcrest:hamcrest-library']
compile deps['org.hibernate:hibernate-hikaricp']
testCompile deps['junit:junit']
testCompile deps['org.junit.jupiter:junit-jupiter-api']
testCompile deps['org.junit.jupiter:junit-jupiter-engine']
testCompile deps['org.junit.jupiter:junit-jupiter-migrationsupport']
testCompile deps['org.junit.jupiter:junit-jupiter-params']
testCompile deps['org.junit-pioneer:junit-pioneer']
testCompile deps['org.junit.platform:junit-platform-runner']
testCompile deps['org.junit.platform:junit-platform-suite-api']
testCompile deps['org.mockito:mockito-core']
testCompile deps['org.mockito:mockito-junit-jupiter']
runtime deps['org.postgresql:postgresql']
// Indirect dependency found by undeclared-dependency check. Such
// dependencies should go after all other compile and testCompile
@@ -305,18 +343,21 @@ dependencies {
// Tool dependencies. used for doc generation.
compile files("${System.properties['java.home']}/../lib/tools.jar")
// Flyway classes needed to generate the golden file.
compile deps['org.flywaydb:flyway-core']
closureCompiler deps['com.google.javascript:closure-compiler']
}
task jaxbToJava {
def xsdFilesDir = "${javaDir}/google/registry/xml/xsd"
def bindingsFile = "${javaDir}/google/registry/xjc/bindings.xjb"
def pkgInfoGenerator = "${javaDir}/google/registry/xjc/make_pkginfo.sh"
def pkgInfoTemplate = "${javaDir}/google/registry/xjc/package-info.java.in"
def pkgInfoMap = "${javaDir}/google/registry/xjc/package-info.map"
def outputDir = "${generatedDir}/google/registry/xjc"
inputs.dir xsdFilesDir
inputs.files bindingsFile, pkgInfoTemplate, pkgInfoGenerator
inputs.files bindingsFile, pkgInfoTemplate, pkgInfoMap
outputs.dir outputDir
doLast {
@@ -354,12 +395,21 @@ task jaxbToJava {
// below.
arg(line: '-npa -quiet -extension')
}
exec {
workingDir "${generatedDir}"
executable pkgInfoGenerator
args pkgInfoTemplate, outputDir
new File(pkgInfoMap).eachLine { line ->
def (packageName, namespace) = line.split()
ant.copy(
tofile: "${outputDir}/${packageName}/package-info.java",
overwrite: true,
file: "${pkgInfoTemplate}") {
filterSet() {
filter(token: 'PACKAGE', value: packageName)
filter(token: 'NAMESPACE', value: namespace)
}
}
}
execInBash(
'find . -name *.java -exec sed -i /\\*\\ \\<p\\>\\$/d {} +',
generatedDir)
}
}
@@ -374,7 +424,7 @@ task soyToJava {
spec11SoyDir, toolsSoyDir, uiSoyDir, registrarSoyDir,
]
soyRelativeDirs.each {
inputs.dir "${javaDir}/${it}"
inputs.dir "${resourcesSourceDir}/${it}"
outputs.dir "${generatedDir}/${it}"
}
@@ -394,19 +444,19 @@ task soyToJava {
soyToJava('google.registry.tools.soy',
"${generatedDir}/${toolsSoyDir}",
fileTree(
dir: "${javaDir}/${toolsSoyDir}",
dir: "${resourcesSourceDir}/${toolsSoyDir}",
include: ['**/*.soy']))
soyToJava('google.registry.ui.soy.registrar',
"${generatedDir}/${registrarSoyDir}",
fileTree(
dir: "${javaDir}/${registrarSoyDir}",
dir: "${resourcesSourceDir}/${registrarSoyDir}",
include: ['**/*.soy']))
soyToJava('google.registry.ui.soy',
"${generatedDir}/${uiSoyDir}",
files {
file("${javaDir}/${uiSoyDir}").listFiles()
file("${resourcesSourceDir}/${uiSoyDir}").listFiles()
}.filter {
it.name.endsWith(".soy")
})
@@ -414,13 +464,13 @@ task soyToJava {
soyToJava('google.registry.reporting.spec11.soy',
"${generatedDir}/${spec11SoyDir}",
fileTree(
dir: "${javaDir}/${spec11SoyDir}",
dir: "${resourcesSourceDir}/${spec11SoyDir}",
include: ['**/*.soy']))
}
}
task soyToJS {
def rootSoyDirectory = "${javaDir}/google/registry/ui/soy"
def rootSoyDirectory = "${resourcesSourceDir}/google/registry/ui/soy"
def outputSoyDirectory = "${generatedDir}/google/registry/ui/soy"
inputs.dir rootSoyDirectory
outputs.dir outputSoyDirectory
@@ -434,8 +484,10 @@ task soyToJS {
"--allowExternalCalls", "false",
"--srcs", "${soyFiles.join(',')}",
"--shouldProvideRequireSoyNamespaces", "true",
"--compileTimeGlobalsFile", "${javaDir}/google/registry/ui/globals.txt",
"--deps", "${deps.join(',')}"
"--compileTimeGlobalsFile", "${resourcesSourceDir}/google/registry/ui/globals.txt"
if (deps != "") {
args "--deps", "${deps.join(',')}"
}
}
}
@@ -456,7 +508,7 @@ task soyToJS {
}
task stylesheetsToJavascript {
def cssSourceDir = "${javaDir}/google/registry/ui/css"
def cssSourceDir = "${jsDir}/google/registry/ui/css"
def outputDir = "${resourcesDir}/google/registry/ui/css"
inputs.dir cssSourceDir
outputs.dir outputDir
@@ -509,8 +561,8 @@ task compileProdJS(type: JavaExec) {
def outputDir = "${resourcesDir}/google/registry/ui"
def nodeModulesDir = "${rootDir}/node_modules"
def cssSourceDir = "${resourcesDir}/google/registry/ui/css"
def jsSourceDir = "${javaDir}/google/registry/ui/js"
def externsDir = "${javaDir}/google/registry/ui/externs"
def jsSourceDir = "${jsDir}/google/registry/ui/js"
def externsDir = "${jsDir}/google/registry/ui/externs"
def soySourceDir = "${generatedDir}/google/registry/ui/soy"
[nodeModulesDir, cssSourceDir, jsSourceDir, externsDir, soySourceDir].each {
@@ -536,7 +588,7 @@ task compileProdJS(type: JavaExec) {
// manually include all the required js files
closureArgs << "--js=${nodeModulesDir}/google-closure-library/**.js"
closureArgs << "--js=${nodeModulesDir}/soyutils_usegoog.js"
closureArgs << "--js=${jsDir}/soyutils_usegoog.js"
closureArgs << "--js=${cssSourceDir}/registrar_bin.css.js"
closureArgs << "--js=${jsSourceDir}/**.js"
// TODO(shicong) Verify the compiled JS file works in Alpha
@@ -547,7 +599,12 @@ task compileProdJS(type: JavaExec) {
compileJava.dependsOn jaxbToJava
compileJava.dependsOn soyToJava
compileJava.dependsOn compileProdJS
// The Closure JS compiler does not support Windows. It is fine to disable it if
// all we want to do is to complile the Java code on Windows.
if (!System.properties['os.name'].toLowerCase().contains('windows')) {
compileJava.dependsOn compileProdJS
assemble.dependsOn compileProdJS
}
// stylesheetsToJavascript must happen after processResources, which wipes the
// resources folder before copying data into it.
@@ -558,7 +615,15 @@ compileProdJS.dependsOn rootProject.npmInstall
compileProdJS.dependsOn processResources
compileProdJS.dependsOn processTestResources
compileProdJS.dependsOn soyToJS
assemble.dependsOn compileProdJS
task karmaTest(type: Exec) {
dependsOn ':npmInstall'
workingDir rootProject.projectDir
executable 'node_modules/karma/bin/karma'
args('start', "${project.projectDir}/karma.conf.js")
}
test.dependsOn karmaTest
// Make testing artifacts available to be depended up on by other projects.
// TODO: factor out google.registry.testing to be a separate project.
@@ -571,27 +636,6 @@ artifacts {
testRuntime testJar
}
task fragileTest(type: Test) {
// Common exclude pattern. See README in parent directory for explanation.
exclude "**/*TestCase.*", "**/*TestSuite.*"
include fragileTestPatterns
// Run every test class in a freshly started process.
forkEvery 1
// Uncomment to see test outputs in stdout.
//testLogging.showStandardStreams = true
}
task outcastTest(type: Test) {
// Common exclude pattern. See README in parent directory for explanation.
exclude "**/*TestCase.*", "**/*TestSuite.*"
include outcastTestPatterns
// Sets the maximum number of test executors that may exist at the same time.
maxParallelForks 5
}
task findGoldenImages(type: JavaExec) {
classpath = sourceSets.test.runtimeClasspath
main = 'google.registry.webdriver.GoldenImageFinder'
@@ -606,10 +650,97 @@ task findGoldenImages(type: JavaExec) {
args arguments
}
task generateGoldenImages(type: Test) {
// Common exclude pattern. See README in parent directory for explanation.
exclude "**/*TestCase.*", "**/*TestSuite.*"
include "**/webdriver/*"
Optional<List<String>> getToolArgsList() {
// If "-PtoolArgs=..." is present in the command line, use it to set the args,
// otherwise use the default flag, which is "--args" to set the args.
def toolArgs = rootProject.findProperty("toolArgs")
if (toolArgs != null) {
def delimiter = '|'
toolArgs += delimiter
def argsList = []
def currArg = ''
for (def i = 0; i < toolArgs.length(); i++) {
def currChar = toolArgs[i]
if (currChar != delimiter) {
currArg += currChar
} else if (i != 0 && toolArgs[i - 1] == '\\') {
currArg = currArg.substring(0, currArg.length() - 1) + currChar
} else {
argsList.add(currArg)
currArg = ''
}
}
return Optional.of(argsList)
}
return Optional.empty()
}
// To run the nomulus tools with these command line tokens:
// "--foo", "bar baz", "--qux=quz"
// gradle registryTool --args="--foo 'bar baz' --qux=quz"
// or:
// gradle registryTool --PtoolArgs="--foo|bar baz|--qux=quz"
// Note that the delimiting pipe can be backslash escaped if it is part of a
// parameter.
ext.createToolTask = {
taskName,
mainClass,
sourceSet = sourceSets.main ->
project.tasks.create(taskName, JavaExec) {
classpath = sourceSet.runtimeClasspath
main = mainClass
doFirst {
getToolArgsList().ifPresent {
args it
}
}
}
}
createToolTask('registryTool', 'google.registry.tools.RegistryTool')
createToolTask(
'devTool',
'google.registry.tools.DevTool',
sourceSets.nonprod)
project.tasks.create('initSqlPipeline', JavaExec) {
main = 'google.registry.beam.initsql.InitSqlPipeline'
doFirst {
getToolArgsList().ifPresent {
args it
}
def isDirectRunner =
args.contains('DirectRunner') || args.contains('--runner=DirectRunner')
// The dependency containing DirectRunner is intentionally excluded from the
// production binary, so that it won't be chosen by mistake: we definitely do
// not want to use it for the real jobs, yet DirectRunner is the default if
// the user forgets to override it.
// DirectRunner is required for tests and is already on testRuntimeClasspath.
// For simplicity, we add testRuntimeClasspath to this task's classpath instead
// of defining a new configuration just for the DirectRunner dependency.
classpath =
isDirectRunner
? sourceSets.main.runtimeClasspath.plus(sourceSets.test.runtimeClasspath)
: sourceSets.main.runtimeClasspath
}
}
project.tasks.create('generateSqlSchema', JavaExec) {
classpath = sourceSets.nonprod.runtimeClasspath
main = 'google.registry.tools.DevTool'
args = [
'-e', 'alpha',
'generate_sql_schema', '--start_postgresql', '-o',
"${rootProject.projectRootDir}/db/src/main/resources/sql/schema/" +
"db-schema.sql.generated"
]
}
task generateGoldenImages(type: FilteringTest) {
tests = ["**/webdriver/*"]
// Sets the maximum number of test executors that may exist at the same time.
maxParallelForks 5
@@ -624,27 +755,254 @@ task generateGoldenImages(type: Test) {
}
generateGoldenImages.finalizedBy(findGoldenImages)
task flowDocsTool(type: JavaExec) {
systemProperty 'test.projectRoot', rootProject.projectRootDir
systemProperty 'test.resourcesDir', resourcesDir
createUberJar('nomulus', 'nomulus', 'google.registry.tools.RegistryTool')
classpath = sourceSets.main.runtimeClasspath
main = 'google.registry.documentation.FlowDocumentationTool'
def arguments = []
if (rootProject.flowDocsFile) {
arguments << "--output_file=${rootProject.flowDocsFile}"
} else {
arguments << "--output_file=${rootProject.projectRootDir}/docs/flows.md"
// A jar with classes and resources from main sourceSet, excluding internal
// data. See comments on configurations.nomulus_test above for details.
// TODO(weiminyu): release process should build this using the public repo to eliminate the need
// for excludes.
task nomulusFossJar (type: Jar) {
archiveBaseName = 'nomulus'
archiveClassifier = 'public'
from (project.sourceSets.main.output) {
exclude 'google/registry/config/files/**'
exclude 'google/registry/proxy/config/**'
}
from (project.sourceSets.main.output) {
include 'google/registry/config/files/default-config.yaml'
include 'google/registry/config/files/nomulus-config-unittest.yaml'
}
args arguments
}
test {
// An UberJar of registry test classes, resources and all dependencies.
// See comments on configurations.nomulus_test above for details.
createUberJar(
'testUberJar',
'nomulus-tests',
'',
[project.configurations.testRuntimeClasspath],
[project.sourceSets.test.output],
[
// Exclude SQL schema, which is a test dependency.
'sql/flyway/**',
// ShadowJar includes java source files when used on
// sourceSets.test.output.
'**/*.java'
])
tasks.testUberJar {
archiveClassifier = 'alldeps'
}
artifacts {
nomulus_test nomulusFossJar
nomulus_test testUberJar
}
publishing {
repositories {
maven {
url project.publish_repo
}
}
publications {
nomulusTestsPublication(MavenPublication) {
groupId 'google.registry'
artifactId 'nomulus_test'
version project.nomulus_version
artifact nomulusFossJar
artifact testUberJar
}
}
}
task buildToolImage(dependsOn: nomulus, type: Exec) {
commandLine 'docker', 'build', '-t', 'nomulus-tool', '.'
}
task generateInitSqlPipelineGraph(type: Test) {
include "**/InitSqlPipelineGraphTest.*"
testNameIncludePatterns = ["**createPipeline_compareGraph"]
ignoreFailures = true
}
task updateInitSqlPipelineGraph(type: Copy) {
def graphRelativePath = 'google/registry/beam/initsql/'
from ("${projectDir}/build/resources/test/${graphRelativePath}") {
include 'pipeline_curr.dot'
rename 'curr', 'golden'
}
into "src/test/resources/${graphRelativePath}"
dependsOn generateInitSqlPipelineGraph
doLast {
if (com.google.common.base.Strings.isNullOrEmpty(project.dot_path)) {
getLogger().info('Property dot_path is null. Not creating image for pipeline graph.')
}
def dotPath = project.dot_path
if (!new File(dotPath).exists()) {
throw new RuntimeException(
"""\
${dotPath} not found. Make sure graphviz is installed
and the dot_path property is set correctly."""
.stripIndent())
}
def goldenGraph = "src/test/resources/${graphRelativePath}/pipeline_golden.dot"
def goldenImage = "src/test/resources/${graphRelativePath}/pipeline_golden.png"
def cmd = "${dotPath} -Tpng -o \"${goldenImage}\" \"${goldenGraph}\""
try {
rootProject.ext.execInBash(cmd, projectDir)
} catch (Throwable throwable) {
throw new RuntimeException(
"""\
Failed to generate golden image with command ${cmd}
Error: ${throwable.getMessage()}
""")
}
}
}
// Build the devtool jar.
createUberJar(
'devtool',
'devtool',
'google.registry.tools.DevTool',
[ project.configurations.nonprodRuntimeClasspath ],
[ project.sourceSets.nonprod.output, sourceSets.main.output ],
[ '**/*.java' ])
artifacts {
devtool devtool
}
task copyJsFilesForTestServer(dependsOn: assemble, type: Copy) {
// Unfortunately the test server relies on having some compiled JS/CSS
// in place, so copy it over here
from "${resourcesDir}/google/registry/ui/"
include '**/*.js'
include '**/*.css'
into "${project.projectDir}/src/main/resources/google/registry/ui/"
}
task runTestServer(dependsOn: copyJsFilesForTestServer, type: JavaExec) {
main = 'google.registry.server.RegistryTestServerMain'
classpath = sourceSets.test.runtimeClasspath
}
/**
* We have to break out the test suites because some of the tests conflict
* with one another, but unfortunately this breaks the "--tests" flag. The
* --tests flag only applies to the task named on the command line (usually
* just "test"), not for all tasks of type "Test".
*
* As a better solution, FilteringTest sets testNameIncludePatterns (the
* internal property that --tests sets) from the value of the "testFilter"
* property, allowing us to filter across all the tests in core without
* explicitly specifying a test task or causing errors because there are no
* matching tests in the main task.
*
* To use it, define "testFilter" to be a comma-separated collection of class
* names (wildcards are allowed):
*
* ./gradlew test -P testFilter=*.FooBar,google.registry.tools.ShellCommandTest
*/
class FilteringTest extends Test {
FilteringTest() {
useJUnitPlatform();
}
private void applyTestFilter() {
if (project.testFilter) {
testNameIncludePatterns = project.testFilter.split(',')
// By default, gradle test tasks will produce a failure if no tests
// match the include/exclude/filter rules. Since test filtering allows us
// to select a set of tests from a particular task, we don't want this
// behavior.
filter.failOnNoMatchingTests = false
}
}
/**
* Set to false if you also want to include TestCase and TestSuite classes.
*
* <p>Must be defined before "test", if at all.
*/
boolean excludeTestCases = true
void setTests(List<String> tests) {
// Common exclude pattern. See README in parent directory for explanation.
if (excludeTestCases) {
exclude "**/*TestCase.*", "**/*TestSuite.*"
}
include tests
applyTestFilter()
}
/**
* Include all of the tests (except Test{Case,TestSuite}). This actually
* doesn't explicitly "include" anything, in which cast the Test class tries
* to include everything that is not explicitly excluded.
*/
void includeAllTests() {
exclude "**/*TestCase.*", "**/*TestSuite.*"
applyTestFilter()
}
}
task fragileTest(type: FilteringTest) {
// Common exclude pattern. See README in parent directory for explanation.
exclude "**/*TestCase.*", "**/*TestSuite.*"
tests = fragileTestPatterns
if (rootProject.findProperty("skipDockerIncompatibleTests") == "true") {
exclude dockerIncompatibleTestPatterns
}
// Run every test class in a freshly started process.
forkEvery 1
doFirst {
new File(screenshotsDir).deleteDir()
}
}
task outcastTest(type: FilteringTest) {
tests = outcastTestPatterns
// Sets the maximum number of test executors that may exist at the same time.
// Note that this number appears to contribute to NoClassDefFoundError
// exceptions on certain machines and distros. The root cause is unclear.
// Try reducing this number if you experience similar problems.
maxParallelForks 3
}
// Dedicated test suite for schema-dependent tests.
task sqlIntegrationTest(type: FilteringTest) {
// TestSuite still requires a JUnit 4 runner, which knows how to handle JUnit 5 tests.
// Here we need to override parent's choice of JUnit 5. If changing this, remember to
// change :integration:sqlIntegrationTest too.
useJUnit()
excludeTestCases = false
tests = ['google/registry/schema/integration/SqlIntegrationTestSuite.*']
}
// Verifies that RegistryTool can be instantiated:
// - All dependencies are packaged in nomulus.jar
// - JPA setup succeeds.
task registryToolIntegrationTest(dependsOn: nomulus, type: FilteringTest) {
tests = ['google/registry/tools/RegistryToolTest.*']
testClassesDirs = sourceSets.test.output.classesDirs
classpath = nomulus.outputs.files.plus(configurations.testRuntimeClasspath)
.plus(files(testClassesDirs))
}
task standardTest(type: FilteringTest) {
includeAllTests()
exclude fragileTestPatterns
exclude outcastTestPatterns
// See SqlIntegrationTestSuite.java
exclude '**/*BeforeSuiteTest.*', '**/*AfterSuiteTest.*'
if (rootProject.findProperty("skipDockerIncompatibleTests") == "true") {
exclude dockerIncompatibleTestPatterns
@@ -656,33 +1014,22 @@ test {
// forkEvery 1
// Sets the maximum number of test executors that may exist at the same time.
// Also, Gradle executes tests in 1 thread and some of our test infrastructures
// depend on that, e.g. DualDatabaseTestInvocationContextProvider injects
// different implementation of TransactionManager into TransactionManagerFactory.
maxParallelForks 5
systemProperty 'test.projectRoot', rootProject.projectRootDir
systemProperty 'test.resourcesDir', resourcesDir
doFirst {
new File(screenshotsDir).deleteDir()
}
}.dependsOn(fragileTest, outcastTest)
task nomulus(type: Jar) {
manifest {
attributes 'Main-Class': 'google.registry.tools.RegistryTool'
}
zip64 = true
baseName = 'nomulus'
version = null
from {
configurations.runtimeClasspath.collect {
it.isDirectory() ? it : zipTree(it)
}
}
// Excludes signature files that accompany some dependency jars, like
// bonuncycastle. It they are present, only classes from those signed jars are
// made available to the class loader.
// see https://discuss.gradle.org/t/signing-a-custom-gradle-plugin-thats-downloaded-by-the-build-system-from-github/1365
exclude "META-INF/*.SF", "META-INF/*.DSA", "META-INF/*.RSA"
with jar
dependsOn project(':third_party').jar
}
test {
// Don't run any tests from this task, all testing gets done in the
// FilteringTest tasks.
exclude "**"
// TODO(weiminyu): Remove dependency on sqlIntegrationTest
}.dependsOn(fragileTest, outcastTest, standardTest, registryToolIntegrationTest, sqlIntegrationTest)
project.build.dependsOn devtool
project.build.dependsOn buildToolImage
project.build.dependsOn ':stage'

View File

@@ -0,0 +1,40 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
com.github.kevinstern:software-and-algorithms:1.0
com.github.stephenc.jcip:jcip-annotations:1.0-1
com.google.auto.value:auto-value:1.6.3
com.google.auto:auto-common:0.10
com.google.code.findbugs:jFormatString:3.0.0
com.google.code.findbugs:jsr305:3.0.2
com.google.dagger:dagger-compiler:2.28
com.google.dagger:dagger-producers:2.28
com.google.dagger:dagger-spi:2.28
com.google.dagger:dagger:2.28
com.google.errorprone:error_prone_annotation:2.3.3
com.google.errorprone:error_prone_annotations:2.3.4
com.google.errorprone:error_prone_check_api:2.3.3
com.google.errorprone:error_prone_core:2.3.3
com.google.errorprone:error_prone_type_annotations:2.3.3
com.google.errorprone:javac-shaded:9-dev-r4023-3
com.google.googlejavaformat:google-java-format:1.5
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.3
com.google.protobuf:protobuf-java:3.4.0
com.googlecode.java-diff-utils:diffutils:1.3.0
com.squareup:javapoet:1.12.1
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.persistence:javax.persistence-api:2.2
net.ltgt.gradle.incap:incap:0.2
org.checkerframework:checker-compat-qual:2.5.3
org.checkerframework:checker-qual:2.11.1
org.checkerframework:dataflow:2.5.3
org.checkerframework:javacutil:2.5.3
org.jetbrains.kotlin:kotlin-stdlib-common:1.3.61
org.jetbrains.kotlin:kotlin-stdlib:1.3.61
org.jetbrains.kotlinx:kotlinx-metadata-jvm:0.1.0
org.jetbrains:annotations:13.0
org.pcollections:pcollections:2.1.2

View File

@@ -3,16 +3,16 @@
# This file is expected to be part of source control.
antlr:antlr:2.7.7
com.google.code.findbugs:jsr305:3.0.2
com.google.errorprone:error_prone_annotations:2.2.0
com.google.errorprone:error_prone_annotations:2.3.2
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:27.0.1-jre
com.google.guava:guava:28.1-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.puppycrawl.tools:checkstyle:8.17
commons-beanutils:commons-beanutils:1.9.3
com.google.j2objc:j2objc-annotations:1.3
com.puppycrawl.tools:checkstyle:8.27
commons-beanutils:commons-beanutils:1.9.4
commons-collections:commons-collections:3.2.2
info.picocli:picocli:3.9.0
net.sf.saxon:Saxon-HE:9.9.0-2
info.picocli:picocli:4.1.1
net.sf.saxon:Saxon-HE:9.9.1-5
org.antlr:antlr4-runtime:4.7.2
org.checkerframework:checker-qual:2.5.2
org.codehaus.mojo:animal-sniffer-annotations:1.17
org.checkerframework:checker-qual:2.8.1
org.codehaus.mojo:animal-sniffer-annotations:1.18

View File

@@ -0,0 +1,259 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
antlr:antlr:2.7.7
aopalliance:aopalliance:1.0
args4j:args4j:2.33
cglib:cglib-nodep:2.2
com.beust:jcommander:1.60
com.fasterxml.jackson.core:jackson-annotations:2.10.2
com.fasterxml.jackson.core:jackson-core:2.10.2
com.fasterxml.jackson.core:jackson-databind:2.10.2
com.fasterxml:classmate:1.5.1
com.github.jnr:jffi:1.2.23
com.github.jnr:jnr-a64asm:1.0.0
com.github.jnr:jnr-constants:0.9.15
com.github.jnr:jnr-enxio:0.26
com.github.jnr:jnr-ffi:2.1.13
com.github.jnr:jnr-posix:3.0.55
com.github.jnr:jnr-unixsocket:0.30
com.github.jnr:jnr-x86asm:1.0.2
com.google.android:annotations:4.1.1.4
com.google.api-client:google-api-client-appengine:1.30.8
com.google.api-client:google-api-client-jackson2:1.30.9
com.google.api-client:google-api-client-java6:1.30.9
com.google.api-client:google-api-client-servlet:1.30.8
com.google.api-client:google-api-client:1.30.9
com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:1.9.1
com.google.api.grpc:grpc-google-cloud-bigtable-v2:1.9.1
com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.85.1
com.google.api.grpc:grpc-google-common-protos:1.17.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:0.90.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.90.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:0.90.0
com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:1.9.1
com.google.api.grpc:proto-google-cloud-bigtable-v2:1.9.1
com.google.api.grpc:proto-google-cloud-datastore-v1:0.85.0
com.google.api.grpc:proto-google-cloud-pubsub-v1:1.85.1
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:1.49.1
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:1.49.1
com.google.api.grpc:proto-google-cloud-spanner-v1:1.49.1
com.google.api.grpc:proto-google-common-protos:1.18.0
com.google.api.grpc:proto-google-iam-v1:0.13.0
com.google.api:api-common:1.8.1
com.google.api:gax-grpc:1.54.0
com.google.api:gax-httpjson:0.70.1
com.google.api:gax:1.54.0
com.google.apis:google-api-services-admin-directory:directory_v1-rev72-1.22.0
com.google.apis:google-api-services-appengine:v1-rev101-1.25.0
com.google.apis:google-api-services-bigquery:v2-rev20191211-1.30.9
com.google.apis:google-api-services-clouddebugger:v2-rev20200313-1.30.9
com.google.apis:google-api-services-cloudkms:v1-rev12-1.22.0
com.google.apis:google-api-services-cloudresourcemanager:v1-rev20200311-1.30.9
com.google.apis:google-api-services-dataflow:v1b3-rev20200305-1.30.9
com.google.apis:google-api-services-dns:v2beta1-rev6-1.22.0
com.google.apis:google-api-services-drive:v2-rev160-1.19.1
com.google.apis:google-api-services-groupssettings:v1-rev60-1.22.0
com.google.apis:google-api-services-healthcare:v1beta1-rev20200525-1.30.9
com.google.apis:google-api-services-monitoring:v3-rev426-1.23.0
com.google.apis:google-api-services-pubsub:v1-rev20200312-1.30.9
com.google.apis:google-api-services-sheets:v4-rev483-1.22.0
com.google.apis:google-api-services-sqladmin:v1beta4-rev20190827-1.30.1
com.google.apis:google-api-services-storage:v1-rev20200326-1.30.9
com.google.appengine.tools:appengine-gcs-client:0.6
com.google.appengine.tools:appengine-mapreduce:0.9
com.google.appengine.tools:appengine-pipeline:0.2.13
com.google.appengine:appengine-api-1.0-sdk:1.9.81
com.google.appengine:appengine-remote-api:1.9.81
com.google.appengine:appengine-testing:1.9.81
com.google.auth:google-auth-library-credentials:0.20.0
com.google.auth:google-auth-library-oauth2-http:0.20.0
com.google.auto.value:auto-value-annotations:1.7.1
com.google.auto.value:auto-value:1.6.3
com.google.cloud.bigdataoss:gcsio:2.1.3
com.google.cloud.bigdataoss:util:2.1.3
com.google.cloud.bigtable:bigtable-client-core:1.13.0
com.google.cloud.datastore:datastore-v1-proto-client:1.6.3
com.google.cloud.sql:jdbc-socket-factory-core:1.0.16
com.google.cloud:google-cloud-bigquery:1.106.0
com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta
com.google.cloud:google-cloud-bigtable:1.9.1
com.google.cloud:google-cloud-core-grpc:1.92.2
com.google.cloud:google-cloud-core-http:1.92.4
com.google.cloud:google-cloud-core:1.92.5
com.google.cloud:google-cloud-spanner:1.49.1
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.dagger:dagger:2.28
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger-system-backend:0.5.1
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.2
com.google.http-client:google-http-client-appengine:1.34.1
com.google.http-client:google-http-client-jackson2:1.34.2
com.google.http-client:google-http-client-protobuf:1.33.0
com.google.http-client:google-http-client:1.34.2
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.2
com.google.monitoring-client:metrics:1.0.7
com.google.monitoring-client:stackdriver:1.0.7
com.google.oauth-client:google-oauth-client-appengine:1.30.5
com.google.oauth-client:google-oauth-client-java6:1.30.6
com.google.oauth-client:google-oauth-client-jetty:1.28.0
com.google.oauth-client:google-oauth-client-servlet:1.30.5
com.google.oauth-client:google-oauth-client:1.30.6
com.google.protobuf:protobuf-java-util:3.11.4
com.google.protobuf:protobuf-java:3.11.4
com.google.re2j:re2j:1.1
com.google.template:soy:2018-03-14
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:57.1
com.jcraft:jsch:0.1.55
com.kohlschutter.junixsocket:junixsocket-common:2.0.4
com.kohlschutter.junixsocket:junixsocket-native-common:2.0.4
com.squareup.okhttp:okhttp:2.5.0
com.squareup.okio:okio:1.13.0
com.sun.istack:istack-commons-runtime:3.0.7
com.sun.xml.fastinfoset:FastInfoset:1.2.15
com.thoughtworks.paranamer:paranamer:2.7
com.zaxxer:HikariCP:3.2.0
commons-codec:commons-codec:1.13
commons-logging:commons-logging:1.2
dnsjava:dnsjava:2.1.7
io.dropwizard.metrics:metrics-core:3.2.6
io.github.classgraph:classgraph:4.8.65
io.grpc:grpc-all:1.27.2
io.grpc:grpc-alts:1.29.0
io.grpc:grpc-api:1.29.0
io.grpc:grpc-auth:1.29.0
io.grpc:grpc-context:1.29.0
io.grpc:grpc-core:1.29.0
io.grpc:grpc-grpclb:1.29.0
io.grpc:grpc-netty-shaded:1.29.0
io.grpc:grpc-netty:1.27.2
io.grpc:grpc-okhttp:1.27.2
io.grpc:grpc-protobuf-lite:1.29.0
io.grpc:grpc-protobuf:1.29.0
io.grpc:grpc-stub:1.29.0
io.grpc:grpc-testing:1.27.2
io.netty:netty-buffer:4.1.42.Final
io.netty:netty-codec-http2:4.1.42.Final
io.netty:netty-codec-http:4.1.42.Final
io.netty:netty-codec-socks:4.1.42.Final
io.netty:netty-codec:4.1.42.Final
io.netty:netty-common:4.1.42.Final
io.netty:netty-handler-proxy:4.1.42.Final
io.netty:netty-handler:4.1.42.Final
io.netty:netty-resolver:4.1.42.Final
io.netty:netty-tcnative-boringssl-static:2.0.17.Final
io.netty:netty-transport:4.1.42.Final
io.opencensus:opencensus-api:0.24.0
io.opencensus:opencensus-contrib-grpc-util:0.24.0
io.opencensus:opencensus-contrib-http-util:0.24.0
io.perfmark:perfmark-api:0.19.0
it.unimi.dsi:fastutil:6.5.16
javax.activation:activation:1.1
javax.activation:javax.activation-api:1.2.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.jdo:jdo2-api:2.3-eb
javax.mail:mail:1.4
javax.persistence:javax.persistence-api:2.2
javax.servlet:servlet-api:2.5
javax.transaction:transaction-api:1.1
javax.validation:validation-api:1.0.0.GA
javax.xml.bind:jaxb-api:2.3.1
jline:jline:1.0
joda-time:joda-time:2.10.5
junit:junit:4.12
net.bytebuddy:byte-buddy:1.10.10
net.java.dev.jna:jna-platform:5.5.0
net.java.dev.jna:jna:5.5.0
org.apache.avro:avro:1.8.2
org.apache.beam:beam-model-fn-execution:2.23.0
org.apache.beam:beam-model-job-management:2.23.0
org.apache.beam:beam-model-pipeline:2.23.0
org.apache.beam:beam-runners-core-construction-java:2.23.0
org.apache.beam:beam-runners-google-cloud-dataflow-java:2.23.0
org.apache.beam:beam-runners-java-fn-execution:2.23.0
org.apache.beam:beam-sdks-java-core:2.23.0
org.apache.beam:beam-sdks-java-expansion-service:2.23.0
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:2.23.0
org.apache.beam:beam-sdks-java-extensions-protobuf:2.23.0
org.apache.beam:beam-sdks-java-fn-execution:2.23.0
org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.23.0
org.apache.beam:beam-vendor-bytebuddy-1_10_8:0.1
org.apache.beam:beam-vendor-grpc-1_26_0:0.3
org.apache.beam:beam-vendor-guava-26_0-jre:0.1
org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
org.apache.commons:commons-compress:1.20
org.apache.commons:commons-lang3:3.5
org.apache.httpcomponents:httpclient:4.5.11
org.apache.httpcomponents:httpcore:4.4.13
org.apache.logging.log4j:log4j-api:2.6.2
org.bouncycastle:bcpg-jdk15on:1.61
org.bouncycastle:bcpkix-jdk15on:1.61
org.bouncycastle:bcprov-jdk15on:1.61
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.codehaus.jackson:jackson-core-asl:1.9.13
org.codehaus.jackson:jackson-mapper-asl:1.9.13
org.codehaus.mojo:animal-sniffer-annotations:1.18
org.conscrypt:conscrypt-openjdk-uber:2.2.1
org.dom4j:dom4j:2.1.3
org.easymock:easymock:3.0
org.flywaydb:flyway-core:5.2.4
org.glassfish.jaxb:jaxb-runtime:2.3.1
org.glassfish.jaxb:txw2:2.3.1
org.hamcrest:hamcrest-core:1.3
org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
org.hibernate:hibernate-core:5.4.17.Final
org.hibernate:hibernate-hikaricp:5.4.17.Final
org.javassist:javassist:3.24.0-GA
org.jboss.logging:jboss-logging:3.3.2.Final
org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
org.jboss:jandex:2.1.3.Final
org.jetbrains:annotations:19.0.0
org.joda:joda-money:1.0.1
org.json:json:20160810
org.jvnet.staxex:stax-ex:1.8
org.mortbay.jetty:jetty-util:6.1.26
org.mortbay.jetty:jetty:6.1.26
org.objenesis:objenesis:1.2
org.ow2.asm:asm-analysis:8.0.1
org.ow2.asm:asm-commons:7.1
org.ow2.asm:asm-tree:8.0.1
org.ow2.asm:asm-util:8.0.1
org.ow2.asm:asm:8.0.1
org.postgresql:postgresql:42.2.14
org.rnorth.duct-tape:duct-tape:1.0.8
org.rnorth.visible-assertions:visible-assertions:2.1.2
org.rnorth:tcp-unix-socket-proxy:1.0.2
org.scijava:native-lib-loader:2.0.2
org.slf4j:slf4j-api:1.7.30
org.testcontainers:database-commons:1.14.3
org.testcontainers:jdbc:1.14.3
org.testcontainers:postgresql:1.14.3
org.testcontainers:testcontainers:1.14.3
org.threeten:threetenbp:1.4.1
org.tukaani:xz:1.8
org.w3c.css:sac:1.3
org.xerial.snappy:snappy-java:1.1.4
org.yaml:snakeyaml:1.17
us.fatehi:schemacrawler-api:16.10.1
us.fatehi:schemacrawler-diagram:16.10.1
us.fatehi:schemacrawler-tools:16.10.1
us.fatehi:schemacrawler-utility:16.10.1
us.fatehi:schemacrawler:16.10.1
xerces:xmlParserAPIs:2.6.2
xpp3:xpp3:1.1.4c

View File

@@ -0,0 +1,256 @@
# This is a Gradle generated file for dependency locking.
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
antlr:antlr:2.7.7
aopalliance:aopalliance:1.0
args4j:args4j:2.33
cglib:cglib-nodep:2.2
com.beust:jcommander:1.60
com.fasterxml.jackson.core:jackson-annotations:2.10.2
com.fasterxml.jackson.core:jackson-core:2.10.2
com.fasterxml.jackson.core:jackson-databind:2.10.2
com.fasterxml:classmate:1.5.1
com.github.jnr:jffi:1.2.23
com.github.jnr:jnr-a64asm:1.0.0
com.github.jnr:jnr-constants:0.9.15
com.github.jnr:jnr-enxio:0.26
com.github.jnr:jnr-ffi:2.1.13
com.github.jnr:jnr-posix:3.0.55
com.github.jnr:jnr-unixsocket:0.30
com.github.jnr:jnr-x86asm:1.0.2
com.google.android:annotations:4.1.1.4
com.google.api-client:google-api-client-appengine:1.30.8
com.google.api-client:google-api-client-jackson2:1.30.9
com.google.api-client:google-api-client-java6:1.30.9
com.google.api-client:google-api-client-servlet:1.30.8
com.google.api-client:google-api-client:1.30.9
com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:1.9.1
com.google.api.grpc:grpc-google-cloud-bigtable-v2:1.9.1
com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.85.1
com.google.api.grpc:grpc-google-common-protos:1.17.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:0.90.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha2:0.90.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.90.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:0.90.0
com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:1.9.1
com.google.api.grpc:proto-google-cloud-bigtable-v2:1.9.1
com.google.api.grpc:proto-google-cloud-datastore-v1:0.85.0
com.google.api.grpc:proto-google-cloud-pubsub-v1:1.85.1
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:1.49.1
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:1.49.1
com.google.api.grpc:proto-google-cloud-spanner-v1:1.49.1
com.google.api.grpc:proto-google-common-protos:1.18.0
com.google.api.grpc:proto-google-iam-v1:0.13.0
com.google.api:api-common:1.8.1
com.google.api:gax-grpc:1.54.0
com.google.api:gax-httpjson:0.70.1
com.google.api:gax:1.54.0
com.google.apis:google-api-services-admin-directory:directory_v1-rev72-1.22.0
com.google.apis:google-api-services-appengine:v1-rev101-1.25.0
com.google.apis:google-api-services-bigquery:v2-rev20191211-1.30.9
com.google.apis:google-api-services-clouddebugger:v2-rev20200313-1.30.9
com.google.apis:google-api-services-cloudkms:v1-rev12-1.22.0
com.google.apis:google-api-services-cloudresourcemanager:v1-rev20200311-1.30.9
com.google.apis:google-api-services-dataflow:v1b3-rev20200305-1.30.9
com.google.apis:google-api-services-dns:v2beta1-rev6-1.22.0
com.google.apis:google-api-services-drive:v2-rev160-1.19.1
com.google.apis:google-api-services-groupssettings:v1-rev60-1.22.0
com.google.apis:google-api-services-healthcare:v1beta1-rev20200525-1.30.9
com.google.apis:google-api-services-monitoring:v3-rev426-1.23.0
com.google.apis:google-api-services-pubsub:v1-rev20200312-1.30.9
com.google.apis:google-api-services-sheets:v4-rev483-1.22.0
com.google.apis:google-api-services-sqladmin:v1beta4-rev20190827-1.30.1
com.google.apis:google-api-services-storage:v1-rev20200326-1.30.9
com.google.appengine.tools:appengine-gcs-client:0.6
com.google.appengine.tools:appengine-mapreduce:0.9
com.google.appengine.tools:appengine-pipeline:0.2.13
com.google.appengine:appengine-api-1.0-sdk:1.9.81
com.google.appengine:appengine-remote-api:1.9.81
com.google.appengine:appengine-testing:1.9.81
com.google.auth:google-auth-library-credentials:0.20.0
com.google.auth:google-auth-library-oauth2-http:0.20.0
com.google.auto.value:auto-value-annotations:1.7.1
com.google.auto.value:auto-value:1.6.3
com.google.cloud.bigdataoss:gcsio:2.1.3
com.google.cloud.bigdataoss:util:2.1.3
com.google.cloud.bigtable:bigtable-client-core:1.13.0
com.google.cloud.datastore:datastore-v1-proto-client:1.6.3
com.google.cloud.sql:jdbc-socket-factory-core:1.0.16
com.google.cloud:google-cloud-bigquery:1.106.0
com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta
com.google.cloud:google-cloud-bigtable:1.9.1
com.google.cloud:google-cloud-core-grpc:1.92.2
com.google.cloud:google-cloud-core-http:1.92.4
com.google.cloud:google-cloud-core:1.92.5
com.google.cloud:google-cloud-spanner:1.49.1
com.google.code.findbugs:jsr305:3.0.2
com.google.code.gson:gson:2.8.6
com.google.common.html.types:types:1.0.4
com.google.dagger:dagger:2.28
com.google.errorprone:error_prone_annotations:2.3.4
com.google.flogger:flogger:0.5.1
com.google.flogger:google-extensions:0.5.1
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:29.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.gwt:gwt-user:2.8.2
com.google.http-client:google-http-client-appengine:1.34.1
com.google.http-client:google-http-client-jackson2:1.34.2
com.google.http-client:google-http-client-protobuf:1.33.0
com.google.http-client:google-http-client:1.34.2
com.google.inject.extensions:guice-multibindings:4.1.0
com.google.inject:guice:4.1.0
com.google.j2objc:j2objc-annotations:1.3
com.google.jsinterop:jsinterop-annotations:1.0.2
com.google.monitoring-client:metrics:1.0.7
com.google.monitoring-client:stackdriver:1.0.7
com.google.oauth-client:google-oauth-client-appengine:1.30.5
com.google.oauth-client:google-oauth-client-java6:1.30.6
com.google.oauth-client:google-oauth-client-jetty:1.28.0
com.google.oauth-client:google-oauth-client-servlet:1.30.5
com.google.oauth-client:google-oauth-client:1.30.6
com.google.protobuf:protobuf-java-util:3.11.4
com.google.protobuf:protobuf-java:3.11.4
com.google.re2j:re2j:1.1
com.google.template:soy:2018-03-14
com.googlecode.charts4j:charts4j:1.3
com.googlecode.json-simple:json-simple:1.1.1
com.ibm.icu:icu4j:57.1
com.jcraft:jsch:0.1.55
com.kohlschutter.junixsocket:junixsocket-common:2.0.4
com.kohlschutter.junixsocket:junixsocket-native-common:2.0.4
com.squareup.okhttp:okhttp:2.5.0
com.squareup.okio:okio:1.13.0
com.sun.istack:istack-commons-runtime:3.0.7
com.sun.xml.fastinfoset:FastInfoset:1.2.15
com.thoughtworks.paranamer:paranamer:2.7
com.zaxxer:HikariCP:3.2.0
commons-codec:commons-codec:1.13
commons-logging:commons-logging:1.2
dnsjava:dnsjava:2.1.7
io.dropwizard.metrics:metrics-core:3.2.6
io.github.classgraph:classgraph:4.8.65
io.grpc:grpc-all:1.27.2
io.grpc:grpc-alts:1.29.0
io.grpc:grpc-api:1.29.0
io.grpc:grpc-auth:1.29.0
io.grpc:grpc-context:1.29.0
io.grpc:grpc-core:1.29.0
io.grpc:grpc-grpclb:1.29.0
io.grpc:grpc-netty-shaded:1.29.0
io.grpc:grpc-netty:1.27.2
io.grpc:grpc-okhttp:1.27.2
io.grpc:grpc-protobuf-lite:1.29.0
io.grpc:grpc-protobuf:1.29.0
io.grpc:grpc-stub:1.29.0
io.grpc:grpc-testing:1.27.2
io.netty:netty-buffer:4.1.42.Final
io.netty:netty-codec-http2:4.1.42.Final
io.netty:netty-codec-http:4.1.42.Final
io.netty:netty-codec-socks:4.1.42.Final
io.netty:netty-codec:4.1.42.Final
io.netty:netty-common:4.1.42.Final
io.netty:netty-handler-proxy:4.1.42.Final
io.netty:netty-handler:4.1.42.Final
io.netty:netty-resolver:4.1.42.Final
io.netty:netty-tcnative-boringssl-static:2.0.17.Final
io.netty:netty-transport:4.1.42.Final
io.opencensus:opencensus-api:0.24.0
io.opencensus:opencensus-contrib-grpc-util:0.24.0
io.opencensus:opencensus-contrib-http-util:0.24.0
io.perfmark:perfmark-api:0.19.0
it.unimi.dsi:fastutil:6.5.16
javax.activation:activation:1.1
javax.activation:javax.activation-api:1.2.0
javax.annotation:javax.annotation-api:1.3.2
javax.annotation:jsr250-api:1.0
javax.inject:javax.inject:1
javax.jdo:jdo2-api:2.3-eb
javax.mail:mail:1.4
javax.persistence:javax.persistence-api:2.2
javax.servlet:servlet-api:2.5
javax.transaction:transaction-api:1.1
javax.validation:validation-api:1.0.0.GA
javax.xml.bind:jaxb-api:2.3.1
jline:jline:1.0
joda-time:joda-time:2.10.5
net.bytebuddy:byte-buddy:1.10.10
net.java.dev.jna:jna-platform:5.5.0
net.java.dev.jna:jna:5.5.0
org.apache.avro:avro:1.8.2
org.apache.beam:beam-model-fn-execution:2.23.0
org.apache.beam:beam-model-job-management:2.23.0
org.apache.beam:beam-model-pipeline:2.23.0
org.apache.beam:beam-runners-core-construction-java:2.23.0
org.apache.beam:beam-runners-google-cloud-dataflow-java:2.23.0
org.apache.beam:beam-runners-java-fn-execution:2.23.0
org.apache.beam:beam-sdks-java-core:2.23.0
org.apache.beam:beam-sdks-java-expansion-service:2.23.0
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:2.23.0
org.apache.beam:beam-sdks-java-extensions-protobuf:2.23.0
org.apache.beam:beam-sdks-java-fn-execution:2.23.0
org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.23.0
org.apache.beam:beam-vendor-bytebuddy-1_10_8:0.1
org.apache.beam:beam-vendor-grpc-1_26_0:0.3
org.apache.beam:beam-vendor-guava-26_0-jre:0.1
org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
org.apache.commons:commons-compress:1.20
org.apache.commons:commons-lang3:3.5
org.apache.httpcomponents:httpclient:4.5.11
org.apache.httpcomponents:httpcore:4.4.13
org.apache.logging.log4j:log4j-api:2.6.2
org.bouncycastle:bcpg-jdk15on:1.61
org.bouncycastle:bcpkix-jdk15on:1.61
org.bouncycastle:bcprov-jdk15on:1.61
org.checkerframework:checker-compat-qual:2.5.5
org.checkerframework:checker-qual:2.11.1
org.codehaus.jackson:jackson-core-asl:1.9.13
org.codehaus.jackson:jackson-mapper-asl:1.9.13
org.codehaus.mojo:animal-sniffer-annotations:1.18
org.conscrypt:conscrypt-openjdk-uber:2.2.1
org.dom4j:dom4j:2.1.3
org.easymock:easymock:3.0
org.flywaydb:flyway-core:5.2.4
org.glassfish.jaxb:jaxb-runtime:2.3.1
org.glassfish.jaxb:txw2:2.3.1
org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
org.hibernate:hibernate-core:5.4.17.Final
org.hibernate:hibernate-hikaricp:5.4.17.Final
org.javassist:javassist:3.24.0-GA
org.jboss.logging:jboss-logging:3.3.2.Final
org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
org.jboss:jandex:2.1.3.Final
org.jetbrains:annotations:19.0.0
org.joda:joda-money:1.0.1
org.json:json:20160810
org.jvnet.staxex:stax-ex:1.8
org.mortbay.jetty:jetty-util:6.1.26
org.mortbay.jetty:jetty:6.1.26
org.objenesis:objenesis:1.2
org.ow2.asm:asm-analysis:8.0.1
org.ow2.asm:asm-commons:7.1
org.ow2.asm:asm-tree:8.0.1
org.ow2.asm:asm-util:8.0.1
org.ow2.asm:asm:8.0.1
org.postgresql:postgresql:42.2.14
org.rnorth.duct-tape:duct-tape:1.0.8
org.rnorth.visible-assertions:visible-assertions:2.1.2
org.rnorth:tcp-unix-socket-proxy:1.0.2
org.scijava:native-lib-loader:2.0.2
org.slf4j:slf4j-api:1.7.30
org.testcontainers:database-commons:1.14.3
org.testcontainers:jdbc:1.14.3
org.testcontainers:postgresql:1.14.3
org.testcontainers:testcontainers:1.14.3
org.threeten:threetenbp:1.4.1
org.tukaani:xz:1.8
org.w3c.css:sac:1.3
org.xerial.snappy:snappy-java:1.1.4
org.yaml:snakeyaml:1.17
us.fatehi:schemacrawler-api:16.10.1
us.fatehi:schemacrawler-diagram:16.10.1
us.fatehi:schemacrawler-tools:16.10.1
us.fatehi:schemacrawler-utility:16.10.1
us.fatehi:schemacrawler:16.10.1
xerces:xmlParserAPIs:2.6.2
xpp3:xpp3:1.1.4c

Some files were not shown because too many files have changed in this diff Show More