Change default beam job region (#1937)

For reasons that I cannot explain, the same expand recurring billing
event pipeline would fail in us-east1 but succeed in us-central1.

See:

https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-02-09_14_52_24-162498476138221714;graphView=0?project=domain-registry

https://pantheon.corp.google.com/dataflow/jobs/us-east1/2023-02-09_14_26_07-4564782062878841960;graphView=1?project=domain-registry

Also improved how the accuracy of the metrics:

It is observed that both counters are consistently higher for the same
start and end times when running in dry run mode. There is no way to
test for consistency when not running in dry run, for obviously reasons.

I can make the recurrings in scope counter consistent by not updating it
in a side-effect-causing transaction, but there is no way around the
other counter. It can only be trusted when running in dry run mode,
unfortunately.
This commit is contained in:
Lai Jiang
2023-02-13 15:57:32 -05:00
committed by GitHub
parent b1cd8c5a6f
commit ff39a4a763
3 changed files with 12 additions and 15 deletions
@@ -23,12 +23,6 @@
"helpText": "The exclusive upper bound of the operation window, in ISO 8601 format.",
"is_optional": false
},
{
"name": "shard",
"label": "The exclusive upper bound of the operation window.",
"helpText": "The exclusive upper bound of the operation window, in ISO 8601 format.",
"is_optional": true
},
{
"name": "isDryRun",
"label": "Whether this job is a dry run.",