Add a tools command to launch SQL validation job (#1526)

* Add a tools command to launch SQL validation job

Stopping using Pipeline.run().waitUntilFinish in
ValidateDatastorePipeline. Flex-templalate does not support blocking
wait in the main thread.

This PR adds a new ValidateSqlCommand that launches the pipeline and
maintains the SQL snapshot while the pipeline is running.

This PR also added more parameters to both ValidateSqlCommand and
ValidateDatastoreCommand:
- The -c option to supply an optional incremental comparison start time
- The -r option to supply an optional release tag that is not 'live',
  e.g., nomulus-DDDDYYMM-RC00

If the manual launch option (-m) is enabled, the commands will print the
gcloud command that can launch the pipeline.

Tested with sandbox, qa and the dev project.
This commit is contained in:
Weimin Yu
2022-02-28 13:14:57 -05:00
committed by GitHub
parent d6bb83f6d3
commit 4f33de10f3
17 changed files with 482 additions and 521 deletions
@@ -1,6 +1,6 @@
{
"name": "Validate Datastore with Cloud SQL",
"description": "An Apache Beam batch pipeline that compares Datastore with the primary Cloud SQL database.",
"name": "Validate Datastore and Cloud SQL",
"description": "An Apache Beam batch pipeline that compares the data in Datastore and Cloud SQL database.",
"parameters": [
{
"name": "registryEnvironment",
@@ -1,21 +0,0 @@
{
"name": "Validate Cloud SQL with Datastore being primary",
"description": "An Apache Beam batch pipeline that compares Cloud SQL with the primary Datastore.",
"parameters": [
{
"name": "registryEnvironment",
"label": "The Registry environment.",
"helpText": "The Registry environment.",
"is_optional": false,
"regexes": [
"^PRODUCTION|SANDBOX|CRASH|QA|ALPHA$"
]
},
{
"name": "comparisonStartTimestamp",
"label": "Only entities updated at or after this time are included for validation.",
"helpText": "The earliest entity update time allowed for inclusion in validation, in ISO8601 format.",
"is_optional": true
}
]
}