- Reuse DS record format processing from the create/update domain commands
(BIND format, commonly used in URS requests)
- Remove the CLIENT_HOLD status from domains that have it (this blocks us from
serving the new nameservers and DS record)
* Integrate transaction persistence into JpaTM
Store the serialized transaction whenever we commit from the JPA transaction
manager. This change also adds:
- The Transaction table.
- The TransactionEntity which is stored in it.
- Changes to the test infrastructure to register the TransactionEntity for
tests where we don't load the nomulus schema.
- A new configuration variable to allow us to turn the transaction
persistence functionality on and off (default is "off").
* Changes for review.
* Incremented sequence number of flyway file
Add a "pseudo-task" in nom_build to do the three step process of generating
the golden schema. In the course of this, add support for pseudo-tasks in
general, improve the database directory readme and make nom_build not call
gradlew if there are no tasks.
* Make EppResource.loadCached() use batched fetch
Use a batched fetch (ofy().load().keys(...)) from datastore in
EppResource.loadCached().
To support this, convert TransactionManager.load(Iterable<VKey>) to accept the
more flexible generic parameters and return a map.
* Simplify datastore key streaming
* Changes requested in review.
* Implement a persistable Transaction object
Implement Transaction, which encapsulates a sequence of datastore mutations
that can be serialized and written to the Cloud SQL Transaction table and
subsequently replayed to Datastore from a backend cron job.
* Changes requested in review
* Add a mujtation count to the persisted format
* Fix cookie processing for RDAP URL update
The existing code only does cookie processing on the _first_ Set-Cookie
header. Therefore, if the "id" cookie used for authentication is defined in
anything other than the first Set-Cookie header (as it now is), we don't find
it.
Replace the cookie processing stanza with a line that processes all cookies in
all Set-Cookie headers.
* Allow class-specific creation of symmetrict VKeys
When translating from a datastore Key to a VKey, see if the "kind" class
contains a createVKey(com.googlecode.objectify.Key) static method and if it
does, use it to construct a symmetric VKey instead of simply creating an
objectify-sided asymmetric VKey.
As a test case for this, implement the createVKey() static function for
DomainBase. Also, create unit tests for VKeyTranslatorFactory, which
continues to house the functionality.
* Use TransactionManager for hosts and contacts
Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts. In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.
* Minor formatting fix
* Make VKey persist to datastore as a key
Convert nsHosts entirely to VKey as a proof-of-concept.
Tested as follows:
1) Deployed to crash, verified that nameservers were visible for several
domains (indicating that we are able to load a set of Keys as VKeys)
2) Updated the set of nameservers for a domain (removing some initial
hosts) and verified that the changes went through.
3) Deployed the old version to crash, verified that I was able to retrieve
the newly saved VKeys as Keys.
4) Modified the hosts for the same domain (adding back one of the hosts)
and verified that the change took effect.
5) Redeployed this change to crash, again updated the nameservers to add
another host.
6) Again restored the old version, verified that the new hosts were
visible.
* Changes in response to review
* Convert to a single VKeyTranslatorFactory instance
* Moved vkey field rename to V25
* Implement all DatastoreTransactionManager methods
In the course of this:
- Make assertDelete() specific to JpaTransactionManager, remove the return
value from delete()
- Converter "in transaction" assertion to IllegalStateException, which is less
JPA specific.
* Upgraded DatastoreTransactionManagerTest to junit5
* Convert DomainBase's contacts to VKeys
Convert usage of DomainBase contacts from Key to VKey. This is the same
change as done for nameserver hosts, as it affects all external interfaces.
As with nameserver hosts, we preserve the existing representation so as not to
afffect the datastore representation.
Since we rarely (if ever) want to check the result of a single element load,
make TransactionManager.load(VKey) return non-optional, non-nullable and just
throw a NoSuchElementException if the entity is not found.
Also add a maybeLoad() that does return an optional in case we ever want to do
this (exists() should suffice for an existence check).
* Remove separate deployment of persistence.xml
We added a step to explicitly copy persistence.xml because for some reason it
wasn't originally getting deployed to app-engine, resulting in failures on
startup. However, this file is now included in core.jar and we are now
getting a warning about multiple persistence units with the same name as it
reads the files from both the filesystem and core.jar.
* Implement VKeyConverter
Implement a SQL converter that can be used for VKey objects.
Caveats:
- This only works with string columns (there's an excellent chance that all of
our VKeys will use SQL string columns).
- Using this dpesn't establish a foreign key constraint between the referenced
type (the "T" in VKey<T>) and the entity itself: this needs to be
defined manually in the schema.
* Improve error information in coverage test.
If the golden schema isn't up-to-date with the persistence model, the coverage
tests fail with an exception chain that ends in a PSQLException 'relation
"TableName" does not exist' which is kind of misleading when the problem is
that your golden schema isn't up-to-date.
Check for this error in the coverage tests and generate a more informative
error message indicating a likely root cause.
* Key to VKey conversion for Nameserver
This change illustrates the conversion of a single key in the system
(Key<HostResource> as used in the "nameservers" field of DomainBase) to a
VKey.
It currently builds, but had some curious (possibly unrelated?) test failures
that I have not fully investigated.
* Latest round of changes, all tests pass.
* Changes requested in review.
* Fix problems with null check in VKey accessors
Add maybeGet versions of getSqlKey() and getOfyKey() that return Optional
objects and make the nameserver management routines use those instead.
* Disambiguate naming of VKey.create() overloads
It was discovered in the course of trying to convert the larger codebase
to VKey.create() calls that method overloading isn't a very effective
discriminator in cases where "Object" is one of the distinguishing
argument types:-)
Convert the two specialized create() methods to createOfy() and
createSql() so that (at least in the former case) we'll get a
compile-time error if we aim to create a VKey for an Ofy key from an
object of the incorrect type.
* Create a nom_build wrapper script
nom_build is a wrapper around ./gradlew. It's purpose is to help us
deal with properties. The main problem that it is trying to solve is
that when properties are specified using -P, we don't get an error if
the property we specify isn't correct. As a result, a user or a build
agent can launch a build with unintended parameters.
nom_build consolidates all of the properties that we define into a
python script where the properties are translated to flags (actual
gradlew flags are also proxied). It also generates the property file
and warns the user if the current properties file is out of sync with
the script and includes documentation on each of the properties.
* Implement dump_golden_schema command in devtool
Add a dump_golden_schema command so that we can generate the golden schema
in-place without having to do the test -> fail -> copy -> test dance.
Refactor the SQL container functionality from GenerateSqlCommand. There is
some duplication of code between the dump command and SchemaTest which should
be dealt with in a subsequent PR.
* Reformatted and changes in response to review
* Fix getDockerTag() usage
* Fix "leaked resource"
* Implement a hibernate type for enums
Leverage GenericCollectionUserType to store enum fields as an array of
strings.
* Changes requested in review
* Remove extraneous import
Curiously, this causes checkstyle to fail now that the import was /removed
from a comment./
The segregated test targets in core break the --tests filter. Fix this by
defining a "testFilter" property and creating the FilteringTest task type that
applies it to the property set by "--tests".
* Add persistence.xml to the war files
* Always use the DummyJpaTransactionManager
Use the DJTM until we get all of the dependencies set up for all of the
environments.
This shouldn't affect any of the unit tests, these use the
JpaTransactionManagerRule to set up a local database and connection.
This fixes the App Engine build.
* Implement CreateAutoTimestampConverter
Implement a JPA-based converter for CreateAutoTimestamp, allowing us to
persist instances of this class.
Note that converters appear to be required to convert to and from database
types that are generally known to JDBC. For example, conversion to Timestamp
works, conversion to OffsetDateTime does not (even though this works through
the JDBC interface directly).
For some reason tools.jar (which we don't need in the runtime in the first
place) gets exploded readonly, causing subsequent builds to fail since they
can't overwrite it. This hack makes it writable at the end of explodeWar.
* Output command test output as well as consuming it
CommandTestCase currently consumes stdout & stderr for the command being
tested. Unfortunately, this results in us not being able to see the command
output. Add an output splitter so that output gets written to the original
stream in addition to being captured.
A simpler approach would be to print the captured data after command
completion. However, this won't work for tests that become hung and also
won't display results in real-time.
Tested: Ran a command test with verboseTestOutput=true, verified that standard
output was visible.
* Save and restore original stdout/err in cmd tests
We have to restore the original stdout/stderr print streams otherwise we end
up nesting them across tests which eventually causes the RDE tests to OOM.
* Generate basic schema for all of DomainBase
Generate a basic schema for DomainBase and everything that is part of it.
This still isn't complete, in particular it lacks:
- Correct conversions for problematic types (e.g. DateTime, Key...)
- Schema generation for history records.
- Name translation.
* Add a generate_schema command
Add a generate_schema command to nomulus tool and add the necessary
instrumentation to EppResource and DomainBase to allow us to generate a
proof-of-concept schema for DomainBase.
* Added forgotten command description
* Revert "Added forgotten command description"
This reverts commit 09326cb8ac.
(checked in the wrong file)
* Added fixes requested during review
* Add a todo to start postgresql container
Add a todo to start a postgresql container from generate_sql_command.
Per further discussion on b/130533696, we've decided to increase these so
tenants can have greater resilience to an outage in our DNS servers.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=248546853
We need this as a precursor to running refreshDnsForAllDomains to change our
TTLs from 3 minutes to an hour (see the associated bug for details).
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=246011747
The latest version of the selenium/standalone-chrome docker image appears to
be out of sync with the API that we're using and there is no more recent
version of the API available in maven central.
This change pins us to the earlier version and fixes our CI build.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=245286532
This signs an attestation that the proxy image was built by cloud build,
allowing the image to be used on a bin-auth restricted kubernetes cluster.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=242917524
Since we're now sending these to actual registrars, change the text to make it
a little bit clearer what's going on.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=241992717
Collect the set of dependencies using the gradle proxy and push to GCS using
gcs_sync.
TESTED: Verified that the script works against both unupdated and up-to-date
dependency sets, verified that the proxy server is destroyed after completion.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=241529357
Note that this also requires setting the CREDZ environment variable to the
contents of the service account credentials file (wrapped in single quotes for
shell command protection)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=234993149
The correct way to override the plugins repo is through the pluginManagement
section in the gradle settings file. Also make use of the gradle.properties
file to initialize repositoryUrl and also publishUrl so we don't have to mess
around with finding and assigning them in the main gradle file.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233778270
It looks like I must have somehow duplicated the absolute path when I copied
these, this puts them in the right place (I think...)
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=233412326
These files are unused by the tests in this directory, have no obvious purpose
and have no README explaining their existence.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232014732
This should have gone in with the implementation of BaseDnsWriter.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=232009971
Remove the GoogleInternal designator on the premium/reserved list tests, move
the lists themselves under the config/files directory, and fix all of the
surrounding infrastructure.
The lists do not get exported. Only the "example.txt" list does, allowing the
test to function on the open source build.
TESTED:
Verified that the gradle build works and that only the example.txt file is exported.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=229940659
Since we seem to be experiencing weird interference between tests on Travis,
run every test in its own JVM until we find a better solution.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228738848
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
Files.copy() attempts to delete the file if it already exists, which obviously
won't work very well for /dev/stdout. Instead copy directly from the decoder
to standard output.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=228384575
Recent changes cause a bunch of new tests to have runtime conflicts.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=226499557
Added a property to show all test output, useful for debugging.
Usage: ./gradlew -PshowAllOutput=true build
Tested:
Verified that the property does the right thing for true and false values
and verified that the build doesn't show test output if it's omitted.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225854375
Modularize the code for DNS count reporting to allow it to be customized for
more flexible systems.
Tested:
Uploaded to alpha with hacks to allow admin initiating and logging from the
DnsCountQueryCoordinatorModule, verified that the provider function is invoked and
that the action runs successfully.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=225225587
Now that we're shipping gradlew, use it from travis. This is the prescribed
approach in the travis docs.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223976622
Remove the application and version tags and replace "module" with "service".
The old tags have been deprecated for ages, and they cause warnings when we
do gradle deployment/staging.
TESTED:
Deployed to crash, everything still seemed to work to the extent that crash
can be expected to, service variations still happened, all services got
updated. Minimal risk to the other environments.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223799777
Add a first pass add Nomulus GAE deployment to our gradle build scripts. This
attempt could stand some improvement. In particular:
- It currently only deploys to crash, per discussion we will make the
environment a parameter.
- This uses a different import mechanism from the other plugins (via a
"buildscript" section) and it looks like the lint plugin is very similar.
I'll experiment with getting it to work that way.
- There is a lot of redundancy in the deployment scripts for each module, and
given that we have the full power of a general programming language it
should be possible for us to reuse the common parts.
But that said, this is a pretty good first step and I wanted to put it out
there.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=223366370
Since we're explicitly downloading a version of travis anyway, just run that
and don't bother with creating gradlew which does nothing but download the
same version over again.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=220618139
Add the Travis-CI configuration file, which will allow our Gradle build to be
tested from Github with travis.
TESTED:
1) Repeatedly tweaked this on own local fork of google/nomulus until I was able to successfully build.
2) Verified that this shows up in the tarball after "blaze build :opensource-staging"
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219678030
Move the shell output encapsulation so that we don't double-wrap on a
premature exit.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=219136896
com_google_truth has a dependency on diffutils that we're not exposing. This
becomes problematic in cases where certain tests fail and the equality check
can't show the difference. If this happens, instead of the original failure
all we see is a failure to load diffutils.
Note that com_google_truth appears to have some other dependencies that we are
also not exposing, but that so far these have not been problematic.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=214774587
ServerSideCommand now just aggregates CommandWithConnection and
CommandWithRemoteApi, so it's arguably clearer for commands to just implement
both of these.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211670031
This change required several things:
- Separating out the interfaces that merely do HTTP calls to the backend from those
that require the remote API (only load the remote API for the latter). Only the
tools service provides the remote api endpoint.
- Removing the XSRF token as an authentication mechanism (with OAUTH, we no longer
need this, and trying to provide it requires initialization of the datastore
code which requires the remote API)
I can't think of a compelling unit test for this beyond what already exists.
Tested:
Verified that:
- nomulus tool commands (e.g. "list_tlds") work against the tools service as they
currently do
- The "curl" command hits endpoints on "tools" by default.
- We can use --server to specify endpoints on the default service.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=211510454
This was only supposed to stay commented out until load-testing was complete.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=210087917
Do not allow the user to create TLDs on sandbox that aren't of the form
"*.test.". If real TLDs are created, they will block users from registering
names under that TLD for the nameserver set that we're using for sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=209983482
Removing this stanza from the config will cause sandbox to write to production
cloud dns, which is what we want.
Likewise, exclude sandbox in addition to production in the create_cdns_tld
command from the environments that point to staging.
Cloud DNS has 3 environments that we would consider using:
- staging which is reset every week, so we can't use it for sandbox
- testing, which is not accessible from external App Engine
- production
Because of the difficulties with the first two, we've decided to use production.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208834786
Create a command to send arbitrary, authenticated HTTP requests to the backend
and remove the existing commands that are basically just wrappers around this.
Tested:
In addition to the unit tests, verified both get and post requests against
alpha.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207756509
It turns out that SourceCodeEscapers has still not been released to github.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196721579
Since the ConsoleReader now controls the display of the prompt, there is no
longer any way for an external program interacting with the nomulus shell to
recognize when the last command issued has been completed.
To remedy this, we introduce an "--encapsulate_output" flag, which causes
standard output and standard error to be wrapped in a class that precedes
all normal lines of output with a prefix ("out: " or "err: ", accordingly)
and allows the command processor to insert a "SUCCESS" or "FAILURE"
line (with no special prefix) after completing the processing of a
command.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=196702338
Increase the instances on alpha to achieve parity with sandbox.
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=194980588
Michael MullerandGitHub
