Add SQL functionality to DeleteProberDataAction (#1218 )

This includes a change to how the JPA transaction manager handles existence and load checks for entities with compound IDs. Previously, we relied on the fields all being named the same in the ID entity and the parent entity. This didn't work for History objects (e.g. DomainHistory) so existence checks were broken. Now, we use the methods the same way that Hibernate does (if possible). Note as well that there's a bit of semi-duplicated logic in DeleteProberDataAction (between the mapper and the SQL logic). The mapper code will be deleted once we've shifted to SQL, and for now it's better to keep it in place for logging purposes.
Implement read-only transaction manager modes for R3.0 migration (#1241 )
2026-07-06 08:06:33 +00:00 · 2021-08-27 21:09:08 -04:00 · 2021-08-27 15:59:16 -04:00 · 2021-08-27 13:11:54 -04:00 · 2021-08-26 16:15:56 -04:00 · 2021-08-25 14:45:05 -06:00
223 changed files with 6654 additions and 3241 deletions
@@ -215,6 +215,7 @@ PRESUBMITS = {
         "RdapDomainSearchAction.java",
         "RdapNameserverSearchAction.java",
         "RdapSearchActionBase.java",
+         "ReadOnlyCheckingEntityManager.java",
         "RegistryQuery",
         },
    ):
@@ -44,7 +44,6 @@ def outcastTestPatterns = [
    "google/registry/flows/domain/DomainCreateFlowTest.*",
    "google/registry/flows/domain/DomainUpdateFlowTest.*",
    "google/registry/tools/CreateDomainCommandTest.*",
-    "google/registry/tools/server/CreatePremiumListActionTest.*",
 ]

 // Tests that fail when running Gradle in a docker container, e. g. when
@@ -70,8 +69,6 @@ def dockerIncompatibleTestPatterns = [
 // Nomulus classes, e.g., threads and objects retained by frameworks.
 // TODO(weiminyu): identify cause and fix offending tests.
 def fragileTestPatterns = [
-    // Problem seems to lie with AppEngine TaskQueue for test.
-    "google/registry/cron/TldFanoutActionTest.*",
    // Test Datastore inexplicably aborts transaction.
    "google/registry/model/tmch/ClaimsListShardTest.*",
    // Changes cache timeouts and for some reason appears to have contention
@@ -185,6 +182,7 @@ dependencies {
  compile deps['com.google.monitoring-client:metrics']
  compile deps['com.google.monitoring-client:stackdriver']
  compile deps['com.google.api-client:google-api-client-java6']
+  compile deps['com.google.api.grpc:proto-google-cloud-tasks-v2']
  compile deps['com.google.apis:google-api-services-admin-directory']
  compile deps['com.google.apis:google-api-services-appengine']
  compile deps['com.google.apis:google-api-services-bigquery']
@@ -217,11 +215,13 @@ dependencies {
  compile deps['com.google.flogger:flogger']
  runtime deps['com.google.flogger:flogger-system-backend']
  compile deps['com.google.guava:guava']
+  compile deps['com.google.protobuf:protobuf-java']
  gradleLint.ignore('unused-dependency') {
    compile deps['com.google.gwt:gwt-user']
  }
  compile deps['com.google.cloud:google-cloud-core']
  compile deps['com.google.cloud:google-cloud-storage']
+  compile deps['com.google.cloud:google-cloud-tasks']
  compile deps['com.google.http-client:google-http-client']
  compile deps['com.google.http-client:google-http-client-appengine']
  compile deps['com.google.http-client:google-http-client-jackson2']
@@ -54,12 +54,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -83,10 +86,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -104,11 +107,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -119,10 +123,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -134,8 +138,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -151,17 +155,17 @@ commons-logging:commons-logging:1.2
 dnsjava:dnsjava:3.3.1
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -259,7 +263,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
@@ -53,12 +53,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -82,10 +85,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -103,11 +106,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger:0.5.1
 com.google.flogger:google-extensions:0.5.1
@@ -117,10 +121,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -133,7 +137,7 @@ com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
 com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -149,17 +153,17 @@ commons-logging:commons-logging:1.2
 dnsjava:dnsjava:3.3.1
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -252,7 +256,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
@@ -58,12 +58,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -87,10 +90,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -109,11 +112,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -124,10 +128,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -139,8 +143,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -159,17 +163,17 @@ guru.nidi:graphviz-java-all-j2v8:0.17.0
 guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -273,7 +277,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.webjars.npm:viz.js-for-graphviz-java:2.1.3
@@ -58,12 +58,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -87,10 +90,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -109,11 +112,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -124,10 +128,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -139,8 +143,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -159,17 +163,17 @@ guru.nidi:graphviz-java-all-j2v8:0.17.0
 guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -272,7 +276,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.webjars.npm:viz.js-for-graphviz-java:2.1.3
@@ -54,12 +54,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -83,10 +86,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -104,11 +107,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -119,10 +123,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -134,8 +138,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -151,17 +155,17 @@ commons-logging:commons-logging:1.2
 dnsjava:dnsjava:3.3.1
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -259,7 +263,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
@@ -53,12 +53,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -82,10 +85,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -103,11 +106,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger:0.5.1
 com.google.flogger:google-extensions:0.5.1
@@ -117,10 +121,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -133,7 +137,7 @@ com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
 com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -149,17 +153,17 @@ commons-logging:commons-logging:1.2
 dnsjava:dnsjava:3.3.1
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -253,7 +257,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
@@ -58,12 +58,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -87,10 +90,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -108,11 +111,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -123,10 +127,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -138,8 +142,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -158,17 +162,17 @@ guru.nidi:graphviz-java-all-j2v8:0.17.0
 guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -271,7 +275,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.webjars.npm:viz.js-for-graphviz-java:2.1.3
@@ -58,12 +58,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -87,10 +90,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -108,11 +111,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -123,10 +127,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -138,8 +142,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -158,17 +162,17 @@ guru.nidi:graphviz-java-all-j2v8:0.17.0
 guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -271,7 +275,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.webjars.npm:viz.js-for-graphviz-java:2.1.3
@@ -58,12 +58,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -87,10 +90,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -108,11 +111,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -123,10 +127,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -138,8 +142,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -158,17 +162,17 @@ guru.nidi:graphviz-java-all-j2v8:0.17.0
 guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -271,7 +275,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.webjars.npm:viz.js-for-graphviz-java:2.1.3
@@ -58,12 +58,15 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
-com.google.api.grpc:proto-google-common-protos:2.1.0
-com.google.api.grpc:proto-google-iam-v1:1.0.10
-com.google.api:api-common:1.10.1
-com.google.api:gax-grpc:1.62.0
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
+com.google.api.grpc:proto-google-common-protos:2.3.2
+com.google.api.grpc:proto-google-iam-v1:1.0.14
+com.google.api:api-common:1.10.4
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.79.0
-com.google.api:gax:1.62.0
+com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
 com.google.apis:google-api-services-appengine:v1-rev130-1.25.0
 com.google.apis:google-api-services-bigquery:v2-rev20200916-1.30.10
@@ -87,10 +90,10 @@ com.google.appengine.tools:appengine-pipeline:0.2.13
 com.google.appengine:appengine-api-1.0-sdk:1.9.86
 com.google.appengine:appengine-remote-api:1.9.86
 com.google.appengine:appengine-testing:1.9.86
-com.google.auth:google-auth-library-credentials:0.24.1
-com.google.auth:google-auth-library-oauth2-http:0.24.1
+com.google.auth:google-auth-library-credentials:0.26.0
+com.google.auth:google-auth-library-oauth2-http:0.26.0
 com.google.auto.service:auto-service-annotations:1.0-rc7
-com.google.auto.value:auto-value-annotations:1.7.4
+com.google.auto.value:auto-value-annotations:1.8.1
 com.google.auto.value:auto-value:1.7.4
 com.google.cloud.bigdataoss:gcsio:2.1.6
 com.google.cloud.bigdataoss:util:2.1.6
@@ -109,11 +112,12 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.113.12
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
-com.google.code.gson:gson:2.8.6
+com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
 com.google.dagger:dagger:2.33
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.7.1
 com.google.escapevelocity:escapevelocity:0.9.1
 com.google.flogger:flogger-system-backend:0.5.1
 com.google.flogger:flogger:0.5.1
@@ -124,10 +128,10 @@ com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.gwt:gwt-user:2.9.0
 com.google.http-client:google-http-client-apache-v2:1.39.0
 com.google.http-client:google-http-client-appengine:1.39.0
-com.google.http-client:google-http-client-gson:1.39.0
+com.google.http-client:google-http-client-gson:1.39.2
 com.google.http-client:google-http-client-jackson2:1.39.0
 com.google.http-client:google-http-client-protobuf:1.33.0
-com.google.http-client:google-http-client:1.39.0
+com.google.http-client:google-http-client:1.39.2
 com.google.inject.extensions:guice-multibindings:4.1.0
 com.google.inject:guice:4.1.0
 com.google.j2objc:j2objc-annotations:1.3
@@ -139,8 +143,8 @@ com.google.oauth-client:google-oauth-client-java6:1.31.4
 com.google.oauth-client:google-oauth-client-jetty:1.31.4
 com.google.oauth-client:google-oauth-client-servlet:1.31.4
 com.google.oauth-client:google-oauth-client:1.31.4
-com.google.protobuf:protobuf-java-util:3.15.3
-com.google.protobuf:protobuf-java:3.15.3
+com.google.protobuf:protobuf-java-util:3.17.3
+com.google.protobuf:protobuf-java:3.17.3
 com.google.re2j:re2j:1.6
 com.google.template:soy:2021-02-01
 com.googlecode.charts4j:charts4j:1.3
@@ -159,17 +163,17 @@ guru.nidi:graphviz-java-all-j2v8:0.17.0
 guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
-io.grpc:grpc-context:1.36.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
+io.grpc:grpc-context:1.39.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -272,7 +276,7 @@ org.testcontainers:database-commons:1.15.2
 org.testcontainers:jdbc:1.15.2
 org.testcontainers:postgresql:1.15.2
 org.testcontainers:testcontainers:1.15.2
-org.threeten:threetenbp:1.5.0
+org.threeten:threetenbp:1.5.1
 org.tukaani:xz:1.5
 org.w3c.css:sac:1.3
 org.webjars.npm:viz.js-for-graphviz-java:2.1.3
@@ -54,10 +54,13 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
 com.google.api.grpc:proto-google-common-protos:2.3.2
 com.google.api.grpc:proto-google-iam-v1:1.0.14
 com.google.api:api-common:1.10.4
-com.google.api:gax-grpc:1.62.0
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.83.0
 com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
@@ -106,6 +109,7 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.118.0
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
 com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
@@ -160,17 +164,17 @@ commons-logging:commons-logging:1.2
 dnsjava:dnsjava:3.3.1
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.102
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
 io.grpc:grpc-context:1.39.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -53,10 +53,13 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
 com.google.api.grpc:proto-google-common-protos:2.3.2
 com.google.api.grpc:proto-google-iam-v1:1.0.14
 com.google.api:api-common:1.10.4
-com.google.api:gax-grpc:1.62.0
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.83.0
 com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
@@ -105,6 +108,7 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.118.0
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
 com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
@@ -158,17 +162,17 @@ commons-logging:commons-logging:1.2
 dnsjava:dnsjava:3.3.1
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.102
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
 io.grpc:grpc-context:1.39.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -58,10 +58,13 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
 com.google.api.grpc:proto-google-common-protos:2.3.2
 com.google.api.grpc:proto-google-iam-v1:1.0.14
 com.google.api:api-common:1.10.4
-com.google.api:gax-grpc:1.62.0
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.83.0
 com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
@@ -111,6 +114,7 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.118.0
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
 com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
@@ -169,17 +173,17 @@ guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.102
 io.github.java-diff-utils:java-diff-utils:4.9
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
 io.grpc:grpc-context:1.39.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -58,10 +58,13 @@ com.google.api.grpc:proto-google-cloud-secretmanager-v1beta1:1.4.0
 com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:2.0.2
 com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2
+com.google.api.grpc:proto-google-cloud-tasks-v2:1.33.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta2:0.89.2
+com.google.api.grpc:proto-google-cloud-tasks-v2beta3:0.89.2
 com.google.api.grpc:proto-google-common-protos:2.3.2
 com.google.api.grpc:proto-google-iam-v1:1.0.14
 com.google.api:api-common:1.10.4
-com.google.api:gax-grpc:1.62.0
+com.google.api:gax-grpc:1.66.0
 com.google.api:gax-httpjson:0.83.0
 com.google.api:gax:1.66.0
 com.google.apis:google-api-services-admin-directory:directory_v1-rev118-1.25.0
@@ -111,6 +114,7 @@ com.google.cloud:google-cloud-pubsublite:0.7.0
 com.google.cloud:google-cloud-secretmanager:1.4.0
 com.google.cloud:google-cloud-spanner:2.0.2
 com.google.cloud:google-cloud-storage:1.118.0
+com.google.cloud:google-cloud-tasks:1.33.2
 com.google.code.findbugs:jsr305:3.0.2
 com.google.code.gson:gson:2.8.7
 com.google.common.html.types:types:1.0.6
@@ -169,17 +173,17 @@ guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.102
 io.github.java-diff-utils:java-diff-utils:4.9
-io.grpc:grpc-alts:1.36.0
-io.grpc:grpc-api:1.36.0
-io.grpc:grpc-auth:1.36.0
+io.grpc:grpc-alts:1.39.0
+io.grpc:grpc-api:1.39.0
+io.grpc:grpc-auth:1.39.0
 io.grpc:grpc-context:1.39.0
-io.grpc:grpc-core:1.36.0
-io.grpc:grpc-grpclb:1.36.0
-io.grpc:grpc-netty-shaded:1.36.0
+io.grpc:grpc-core:1.39.0
+io.grpc:grpc-grpclb:1.39.0
+io.grpc:grpc-netty-shaded:1.39.0
 io.grpc:grpc-netty:1.32.2
-io.grpc:grpc-protobuf-lite:1.36.0
-io.grpc:grpc-protobuf:1.36.0
-io.grpc:grpc-stub:1.36.0
+io.grpc:grpc-protobuf-lite:1.39.0
+io.grpc:grpc-protobuf:1.39.0
+io.grpc:grpc-stub:1.39.0
 io.netty:netty-buffer:4.1.51.Final
 io.netty:netty-codec-http2:4.1.51.Final
 io.netty:netty-codec-http:4.1.51.Final
@@ -260,7 +260,7 @@ public class ReplayCommitLogsToSqlAction implements Runnable {
            .ifPresent(
                sqlEntity -> {
                  sqlEntity.beforeSqlSaveOnReplay();
-                  jpaTm().put(sqlEntity);
+                  jpaTm().putIgnoringReadOnly(sqlEntity);
                });
      } else {
        // this should never happen, but we shouldn't fail on it
@@ -293,7 +293,7 @@ public class ReplayCommitLogsToSqlAction implements Runnable {
          && !DatastoreOnlyEntity.class.isAssignableFrom(entityClass)
          && entityClass.getAnnotation(javax.persistence.Entity.class) != null) {
        ReplaySpecializer.beforeSqlDelete(entityVKey);
-        jpaTm().delete(entityVKey);
+        jpaTm().deleteIgnoringReadOnly(entityVKey);
      }
    } catch (Throwable t) {
      logger.atSevere().log("Error when deleting key %s", entityVKey);
@@ -23,6 +23,7 @@ import static google.registry.model.ResourceTransferUtils.updateForeignKeyIndexD
 import static google.registry.model.ofy.ObjectifyService.auditedOfy;
 import static google.registry.model.reporting.HistoryEntry.Type.DOMAIN_DELETE;
 import static google.registry.model.tld.Registries.getTldsOfType;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.request.Action.Method.POST;
 import static google.registry.request.RequestParameters.PARAM_TLDS;
@@ -42,6 +43,7 @@ import google.registry.config.RegistryEnvironment;
 import google.registry.dns.DnsQueue;
 import google.registry.mapreduce.MapreduceRunner;
 import google.registry.mapreduce.inputs.EppResourceInputs;
+import google.registry.model.CreateAutoTimestamp;
 import google.registry.model.EppResourceUtils;
 import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.DomainHistory;
@@ -54,15 +56,18 @@ import google.registry.request.Parameter;
 import google.registry.request.Response;
 import google.registry.request.auth.Auth;
 import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
 import javax.inject.Inject;
+import org.hibernate.CacheMode;
+import org.hibernate.ScrollMode;
+import org.hibernate.ScrollableResults;
+import org.hibernate.query.Query;
 import org.joda.time.DateTime;
 import org.joda.time.Duration;

 /**
- * Deletes all prober DomainBases and their subordinate history entries, poll messages, and
- * billing events, along with their ForeignKeyDomainIndex and EppResourceIndex entities.
- *
- * <p>See: https://www.youtube.com/watch?v=xuuv0syoHnM
+ * Deletes all prober DomainBases and their subordinate history entries, poll messages, and billing
+ * events, along with their ForeignKeyDomainIndex and EppResourceIndex entities.
 */
@Action(
    service = Action.Service.BACKEND,
@@ -73,10 +78,51 @@ public class DeleteProberDataAction implements Runnable {

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

+  /**
+   * The maximum amount of time we allow a prober domain to be in use.
+   *
+   * <p>In practice, the prober's connection will time out well before this duration. This includes
+   * a decent buffer.
+   */
+  private static final Duration DOMAIN_USED_DURATION = Duration.standardHours(1);
+
+  /**
+   * The minimum amount of time we want a domain to be "soft deleted".
+   *
+   * <p>The domain has to remain soft deleted for at least enough time for the DNS task to run and
+   * remove it from DNS itself. This is probably on the order of minutes.
+   */
+  private static final Duration SOFT_DELETE_DELAY = Duration.standardHours(1);
+
+  private static final DnsQueue dnsQueue = DnsQueue.create();
+
+  // Domains to delete must:
+  // 1. Be in one of the prober TLDs
+  // 2. Not be a nic domain
+  // 3. Have no subordinate hosts
+  // 4. Not still be used (within an hour of creation time)
+  // 5. Either be active (creationTime <= now < deletionTime) or have been deleted a while ago (this
+  //    prevents accidental double-map with the same key from immediately deleting active domains)
+  //
+  // Note: creationTime must be compared to a Java object (CreateAutoTimestamp) but deletionTime can
+  // be compared directly to the SQL timestamp (it's a DateTime)
+  private static final String DOMAIN_QUERY_STRING =
+      "FROM Domain d WHERE d.tld IN :tlds AND d.fullyQualifiedDomainName NOT LIKE 'nic.%' AND"
+          + " (d.subordinateHosts IS EMPTY OR d.subordinateHosts IS NULL) AND d.creationTime <"
+          + " :creationTimeCutoff AND ((d.creationTime <= :nowAutoTimestamp AND d.deletionTime >"
+          + " current_timestamp()) OR d.deletionTime < :nowMinusSoftDeleteDelay) ORDER BY d.repoId";
+
+  /** Number of domains to retrieve and delete per SQL transaction. */
+  private static final int BATCH_SIZE = 1000;
+
  @Inject @Parameter(PARAM_DRY_RUN) boolean isDryRun;
  /** List of TLDs to work on. If empty - will work on all TLDs that end with .test. */
  @Inject @Parameter(PARAM_TLDS) ImmutableSet<String> tlds;
-  @Inject @Config("registryAdminClientId") String registryAdminClientId;
+
+  @Inject
+  @Config("registryAdminClientId")
+  String registryAdminRegistrarId;
+
  @Inject MapreduceRunner mrRunner;
  @Inject Response response;
  @Inject DeleteProberDataAction() {}
@@ -84,25 +130,14 @@ public class DeleteProberDataAction implements Runnable {
  @Override
  public void run() {
    checkState(
-        !Strings.isNullOrEmpty(registryAdminClientId),
+        !Strings.isNullOrEmpty(registryAdminRegistrarId),
        "Registry admin client ID must be configured for prober data deletion to work");
-    mrRunner
-        .setJobName("Delete prober data")
-        .setModuleName("backend")
-        .runMapOnly(
-            new DeleteProberDataMapper(getProberRoidSuffixes(), isDryRun, registryAdminClientId),
-            ImmutableList.of(EppResourceInputs.createKeyInput(DomainBase.class)))
-        .sendLinkToMapreduceConsole(response);
-  }
-
-  private ImmutableSet<String> getProberRoidSuffixes() {
    checkArgument(
        !PRODUCTION.equals(RegistryEnvironment.get())
            || tlds.stream().allMatch(tld -> tld.endsWith(".test")),
        "On production, can only work on TLDs that end with .test");
    ImmutableSet<String> deletableTlds =
-        getTldsOfType(TldType.TEST)
-            .stream()
+        getTldsOfType(TldType.TEST).stream()
            .filter(tld -> tlds.isEmpty() ? tld.endsWith(".test") : tlds.contains(tld))
            .collect(toImmutableSet());
    checkArgument(
@@ -110,10 +145,161 @@ public class DeleteProberDataAction implements Runnable {
        "If tlds are given, they must all exist and be TEST tlds. Given: %s, not found: %s",
        tlds,
        Sets.difference(tlds, deletableTlds));
-    return deletableTlds
-        .stream()
-        .map(tld -> Registry.get(tld).getRoidSuffix())
-        .collect(toImmutableSet());
+    ImmutableSet<String> proberRoidSuffixes =
+        deletableTlds.stream()
+            .map(tld -> Registry.get(tld).getRoidSuffix())
+            .collect(toImmutableSet());
+    if (tm().isOfy()) {
+      mrRunner
+          .setJobName("Delete prober data")
+          .setModuleName("backend")
+          .runMapOnly(
+              new DeleteProberDataMapper(proberRoidSuffixes, isDryRun, registryAdminRegistrarId),
+              ImmutableList.of(EppResourceInputs.createKeyInput(DomainBase.class)))
+          .sendLinkToMapreduceConsole(response);
+    } else {
+      runSqlJob(deletableTlds);
+    }
+  }
+
+  private void runSqlJob(ImmutableSet<String> deletableTlds) {
+    AtomicInteger softDeletedDomains = new AtomicInteger();
+    AtomicInteger hardDeletedDomains = new AtomicInteger();
+    jpaTm().transact(() -> processDomains(deletableTlds, softDeletedDomains, hardDeletedDomains));
+    logger.atInfo().log(
+        "%s %d domains.",
+        isDryRun ? "Would have soft-deleted" : "Soft-deleted", softDeletedDomains.get());
+    logger.atInfo().log(
+        "%s %d domains.",
+        isDryRun ? "Would have hard-deleted" : "Hard-deleted", hardDeletedDomains.get());
+  }
+
+  private void processDomains(
+      ImmutableSet<String> deletableTlds,
+      AtomicInteger softDeletedDomains,
+      AtomicInteger hardDeletedDomains) {
+    DateTime now = tm().getTransactionTime();
+    // Scroll through domains, soft-deleting as necessary (very few will be soft-deleted) and
+    // keeping track of which domains to hard-delete (there can be many, so we batch them up)
+    ScrollableResults scrollableResult =
+        jpaTm()
+            .query(DOMAIN_QUERY_STRING, DomainBase.class)
+            .setParameter("tlds", deletableTlds)
+            .setParameter(
+                "creationTimeCutoff", CreateAutoTimestamp.create(now.minus(DOMAIN_USED_DURATION)))
+            .setParameter("nowMinusSoftDeleteDelay", now.minus(SOFT_DELETE_DELAY))
+            .setParameter("nowAutoTimestamp", CreateAutoTimestamp.create(now))
+            .unwrap(Query.class)
+            .setCacheMode(CacheMode.IGNORE)
+            .scroll(ScrollMode.FORWARD_ONLY);
+    ImmutableList.Builder<String> domainRepoIdsToHardDelete = new ImmutableList.Builder<>();
+    ImmutableList.Builder<String> hostNamesToHardDelete = new ImmutableList.Builder<>();
+    for (int i = 1; scrollableResult.next(); i = (i + 1) % BATCH_SIZE) {
+      DomainBase domain = (DomainBase) scrollableResult.get(0);
+      processDomain(
+          domain,
+          domainRepoIdsToHardDelete,
+          hostNamesToHardDelete,
+          softDeletedDomains,
+          hardDeletedDomains);
+      // Batch the deletion and DB flush + session clearing so we don't OOM
+      if (i == 0) {
+        hardDeleteDomainsAndHosts(domainRepoIdsToHardDelete.build(), hostNamesToHardDelete.build());
+        domainRepoIdsToHardDelete = new ImmutableList.Builder<>();
+        hostNamesToHardDelete = new ImmutableList.Builder<>();
+        jpaTm().getEntityManager().flush();
+        jpaTm().getEntityManager().clear();
+      }
+    }
+    // process the remainder
+    hardDeleteDomainsAndHosts(domainRepoIdsToHardDelete.build(), hostNamesToHardDelete.build());
+  }
+
+  private void processDomain(
+      DomainBase domain,
+      ImmutableList.Builder<String> domainRepoIdsToHardDelete,
+      ImmutableList.Builder<String> hostNamesToHardDelete,
+      AtomicInteger softDeletedDomains,
+      AtomicInteger hardDeletedDomains) {
+    // If the domain is still active, that means that the prober encountered a failure and did not
+    // successfully soft-delete the domain (thus leaving its DNS entry published). We soft-delete
+    // it now so that the DNS entry can be handled. The domain will then be hard-deleted the next
+    // time the job is run.
+    if (EppResourceUtils.isActive(domain, tm().getTransactionTime())) {
+      if (isDryRun) {
+        logger.atInfo().log(
+            "Would soft-delete the active domain: %s (%s)",
+            domain.getDomainName(), domain.getRepoId());
+      } else {
+        softDeleteDomain(domain, registryAdminRegistrarId, dnsQueue);
+      }
+      softDeletedDomains.incrementAndGet();
+    } else {
+      if (isDryRun) {
+        logger.atInfo().log(
+            "Would hard-delete the non-active domain: %s (%s) and its dependents",
+            domain.getDomainName(), domain.getRepoId());
+      } else {
+        domainRepoIdsToHardDelete.add(domain.getRepoId());
+        hostNamesToHardDelete.addAll(domain.getSubordinateHosts());
+      }
+      hardDeletedDomains.incrementAndGet();
+    }
+  }
+
+  private void hardDeleteDomainsAndHosts(
+      ImmutableList<String> domainRepoIds, ImmutableList<String> hostNames) {
+    jpaTm()
+        .query("DELETE FROM Host WHERE fullyQualifiedHostName IN :hostNames")
+        .setParameter("hostNames", hostNames)
+        .executeUpdate();
+    jpaTm()
+        .query("DELETE FROM BillingEvent WHERE domainRepoId IN :repoIds")
+        .setParameter("repoIds", domainRepoIds)
+        .executeUpdate();
+    jpaTm()
+        .query("DELETE FROM BillingRecurrence WHERE domainRepoId IN :repoIds")
+        .setParameter("repoIds", domainRepoIds)
+        .executeUpdate();
+    jpaTm()
+        .query("DELETE FROM BillingCancellation WHERE domainRepoId IN :repoIds")
+        .setParameter("repoIds", domainRepoIds)
+        .executeUpdate();
+    jpaTm()
+        .query("DELETE FROM DomainHistory WHERE domainRepoId IN :repoIds")
+        .setParameter("repoIds", domainRepoIds)
+        .executeUpdate();
+    jpaTm()
+        .query("DELETE FROM PollMessage WHERE domainRepoId IN :repoIds")
+        .setParameter("repoIds", domainRepoIds)
+        .executeUpdate();
+    jpaTm()
+        .query("DELETE FROM Domain WHERE repoId IN :repoIds")
+        .setParameter("repoIds", domainRepoIds)
+        .executeUpdate();
+  }
+
+  // Take a DNS queue + admin registrar id as input so that it can be called from the mapper as well
+  private static void softDeleteDomain(
+      DomainBase domain, String registryAdminRegistrarId, DnsQueue localDnsQueue) {
+    DomainBase deletedDomain =
+        domain.asBuilder().setDeletionTime(tm().getTransactionTime()).setStatusValues(null).build();
+    DomainHistory historyEntry =
+        new DomainHistory.Builder()
+            .setDomain(domain)
+            .setType(DOMAIN_DELETE)
+            .setModificationTime(tm().getTransactionTime())
+            .setBySuperuser(true)
+            .setReason("Deletion of prober data")
+            .setClientId(registryAdminRegistrarId)
+            .build();
+    // Note that we don't bother handling grace periods, billing events, pending transfers, poll
+    // messages, or auto-renews because those will all be hard-deleted the next time the job runs
+    // anyway.
+    tm().putAllWithoutBackup(ImmutableList.of(deletedDomain, historyEntry));
+    // updating foreign keys is a no-op in SQL
+    updateForeignKeyIndexDeletionTime(deletedDomain);
+    localDnsQueue.addDomainRefreshTask(deletedDomain.getDomainName());
  }

  /** Provides the map method that runs for each existing DomainBase entity. */
@@ -122,32 +308,17 @@ public class DeleteProberDataAction implements Runnable {
    private static final DnsQueue dnsQueue = DnsQueue.create();
    private static final long serialVersionUID = -7724537393697576369L;

-    /**
-     * The maximum amount of time we allow a prober domain to be in use.
-     *
-     * In practice, the prober's connection will time out well before this duration. This includes a
-     * decent buffer.
-     *
-     */
-    private static final Duration DOMAIN_USED_DURATION = Duration.standardHours(1);
-
-    /**
-     * The minimum amount of time we want a domain to be "soft deleted".
-     *
-     * The domain has to remain soft deleted for at least enough time for the DNS task to run and
-     * remove it from DNS itself. This is probably on the order of minutes.
-     */
-    private static final Duration SOFT_DELETE_DELAY = Duration.standardHours(1);
-
    private final ImmutableSet<String> proberRoidSuffixes;
    private final Boolean isDryRun;
-    private final String registryAdminClientId;
+    private final String registryAdminRegistrarId;

    public DeleteProberDataMapper(
-        ImmutableSet<String> proberRoidSuffixes, Boolean isDryRun, String registryAdminClientId) {
+        ImmutableSet<String> proberRoidSuffixes,
+        Boolean isDryRun,
+        String registryAdminRegistrarId) {
      this.proberRoidSuffixes = proberRoidSuffixes;
      this.isDryRun = isDryRun;
-      this.registryAdminClientId = registryAdminClientId;
+      this.registryAdminRegistrarId = registryAdminRegistrarId;
    }

    @Override
@@ -203,7 +374,7 @@ public class DeleteProberDataAction implements Runnable {
          logger.atInfo().log(
              "Would soft-delete the active domain: %s (%s)", domainName, domainKey);
        } else {
-          softDeleteDomain(domain);
+          tm().transact(() -> softDeleteDomain(domain, registryAdminRegistrarId, dnsQueue));
        }
        getContext().incrementCounter("domains soft-deleted");
        return;
@@ -223,8 +394,7 @@ public class DeleteProberDataAction implements Runnable {
          tm().transact(
                  () -> {
                    // This ancestor query selects all descendant HistoryEntries, BillingEvents,
-                    // PollMessages,
-                    // and TLD-specific entities, as well as the domain itself.
+                    // PollMessages, and TLD-specific entities, as well as the domain itself.
                    List<Key<Object>> domainAndDependentKeys =
                        auditedOfy().load().ancestor(domainKey).keys().list();
                    ImmutableSet<Key<?>> allKeys =
@@ -243,32 +413,5 @@ public class DeleteProberDataAction implements Runnable {
      getContext().incrementCounter("domains hard-deleted");
      getContext().incrementCounter("total entities hard-deleted", entitiesDeleted);
    }
-
-    private void softDeleteDomain(final DomainBase domain) {
-      tm().transactNew(
-              () -> {
-                DomainBase deletedDomain =
-                    domain
-                        .asBuilder()
-                        .setDeletionTime(tm().getTransactionTime())
-                        .setStatusValues(null)
-                        .build();
-                DomainHistory historyEntry =
-                    new DomainHistory.Builder()
-                        .setDomain(domain)
-                        .setType(DOMAIN_DELETE)
-                        .setModificationTime(tm().getTransactionTime())
-                        .setBySuperuser(true)
-                        .setReason("Deletion of prober data")
-                        .setClientId(registryAdminClientId)
-                        .build();
-                // Note that we don't bother handling grace periods, billing events, pending
-                // transfers, poll messages, or auto-renews because these will all be hard-deleted
-                // the next time the mapreduce runs anyway.
-                tm().putAll(deletedDomain, historyEntry);
-                updateForeignKeyIndexDeletionTime(deletedDomain);
-                dnsQueue.addDomainRefreshTask(deletedDomain.getDomainName());
-              });
-    }
  }
 }
@@ -0,0 +1,337 @@
+// Copyright 2021 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.batch;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
+import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
+import static org.apache.http.HttpStatus.SC_INTERNAL_SERVER_ERROR;
+import static org.apache.http.HttpStatus.SC_OK;
+import static org.joda.time.DateTimeZone.UTC;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedSet;
+import com.google.common.collect.Streams;
+import com.google.common.flogger.FluentLogger;
+import com.google.common.net.MediaType;
+import google.registry.config.RegistryConfig.Config;
+import google.registry.flows.certs.CertificateChecker;
+import google.registry.model.registrar.Registrar;
+import google.registry.model.registrar.RegistrarContact;
+import google.registry.model.registrar.RegistrarContact.Type;
+import google.registry.request.Action;
+import google.registry.request.Response;
+import google.registry.request.auth.Auth;
+import google.registry.util.EmailMessage;
+import google.registry.util.SendEmailService;
+import java.util.Date;
+import java.util.Optional;
+import javax.inject.Inject;
+import javax.mail.internet.AddressException;
+import javax.mail.internet.InternetAddress;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+/** An action that sends notification emails to registrars whose certificates are expiring soon. */
+@Action(
+    service = Action.Service.BACKEND,
+    path = SendExpiringCertificateNotificationEmailAction.PATH,
+    auth = Auth.AUTH_INTERNAL_OR_ADMIN)
+public class SendExpiringCertificateNotificationEmailAction implements Runnable {
+  public static final String PATH = "/_dr/task/sendExpiringCertificateNotificationEmail";
+  /**
+   * Used as an offset when storing the last notification email sent date.
+   *
+   * <p>This is used to handle edges cases when the update happens in between the day switch. For
+   * instance,if the job starts at 2:00 am every day and it finishes at 2:03 of the same day, then
+   * next day at 2am, the date difference will be less than a day, which will lead to the date
+   * difference between two successive email sent date being the expected email interval days + 1;
+   */
+  protected static final Duration UPDATE_TIME_OFFSET = Duration.standardMinutes(10);
+
+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+  private static final DateTimeFormatter DATE_FORMATTER = DateTimeFormat.forPattern("yyyy-MM-dd");
+
+  private final CertificateChecker certificateChecker;
+  private final String expirationWarningEmailBodyText;
+  private final SendEmailService sendEmailService;
+  private final String expirationWarningEmailSubjectText;
+  private final InternetAddress gSuiteOutgoingEmailAddress;
+  private final Response response;
+
+  @Inject
+  public SendExpiringCertificateNotificationEmailAction(
+      @Config("expirationWarningEmailBodyText") String expirationWarningEmailBodyText,
+      @Config("expirationWarningEmailSubjectText") String expirationWarningEmailSubjectText,
+      @Config("gSuiteOutgoingEmailAddress") InternetAddress gSuiteOutgoingEmailAddress,
+      SendEmailService sendEmailService,
+      CertificateChecker certificateChecker,
+      Response response) {
+    this.certificateChecker = certificateChecker;
+    this.expirationWarningEmailSubjectText = expirationWarningEmailSubjectText;
+    this.sendEmailService = sendEmailService;
+    this.gSuiteOutgoingEmailAddress = gSuiteOutgoingEmailAddress;
+    this.expirationWarningEmailBodyText = expirationWarningEmailBodyText;
+    this.response = response;
+  }
+
+  @Override
+  public void run() {
+    response.setContentType(MediaType.PLAIN_TEXT_UTF_8);
+    try {
+      sendNotificationEmails();
+      response.setStatus(SC_OK);
+    } catch (Exception e) {
+      logger.atWarning().withCause(e).log(
+          "Exception thrown when sending expiring certificate notification emails.");
+      response.setStatus(SC_INTERNAL_SERVER_ERROR);
+      response.setPayload(String.format("Exception thrown with cause: %s", e));
+    }
+  }
+
+  /**
+   * Returns a list of registrars that should receive expiring notification emails. There are two
+   * certificates that should be considered (the main certificate and failOver certificate). The
+   * registrars should receive notifications if one of the certificate checks returns true.
+   */
+  @VisibleForTesting
+  ImmutableList<RegistrarInfo> getRegistrarsWithExpiringCertificates() {
+    logger.atInfo().log(
+        "Getting a list of registrars that should receive expiring notification emails.");
+    return Streams.stream(Registrar.loadAllCached())
+        .map(
+            registrar ->
+                RegistrarInfo.create(
+                    registrar,
+                    registrar.getClientCertificate().isPresent()
+                        && certificateChecker.shouldReceiveExpiringNotification(
+                            registrar.getLastExpiringCertNotificationSentDate(),
+                            registrar.getClientCertificate().get()),
+                    registrar.getFailoverClientCertificate().isPresent()
+                        && certificateChecker.shouldReceiveExpiringNotification(
+                            registrar.getLastExpiringFailoverCertNotificationSentDate(),
+                            registrar.getFailoverClientCertificate().get())))
+        .filter(
+            registrarInfo ->
+                registrarInfo.isCertExpiring() || registrarInfo.isFailOverCertExpiring())
+        .collect(toImmutableList());
+  }
+
+  /**
+   * Sends a notification email to the registrar regarding the expiring certificate and returns true
+   * if it's sent successfully.
+   */
+  @VisibleForTesting
+  boolean sendNotificationEmail(
+      Registrar registrar,
+      DateTime lastExpiringCertNotificationSentDate,
+      CertificateType certificateType,
+      Optional<String> certificate) {
+    if (!certificate.isPresent()
+        || !certificateChecker.shouldReceiveExpiringNotification(
+            lastExpiringCertNotificationSentDate, certificate.get())) {
+      return false;
+    }
+    try {
+      ImmutableSet<InternetAddress> recipients = getEmailAddresses(registrar, Type.TECH);
+      if (recipients.isEmpty()) {
+        logger.atWarning().log(
+            "Registrar %s contains no email addresses to receive notification email.",
+            registrar.getRegistrarName());
+        return false;
+      }
+      sendEmailService.sendEmail(
+          EmailMessage.newBuilder()
+              .setFrom(gSuiteOutgoingEmailAddress)
+              .setSubject(expirationWarningEmailSubjectText)
+              .setBody(
+                  getEmailBody(
+                      registrar.getRegistrarName(),
+                      certificateType,
+                      certificateChecker.getCertificate(certificate.get()).getNotAfter()))
+              .setRecipients(recipients)
+              .setCcs(getEmailAddresses(registrar, Type.ADMIN))
+              .build());
+      logger.atInfo().log(
+          "Sent an email to inform registrar %s that its %s SSL certificate will expire on %s.",
+          registrar.getRegistrarName(),
+          certificateType.getDisplayName(),
+          DATE_FORMATTER.print(lastExpiringCertNotificationSentDate));
+      /*
+       * A duration time offset is used here to ensure that date comparison between two
+       * successive dates is always greater than 1 day. This date is set as last updated date,
+       * for applicable certificate.
+       */
+      updateLastNotificationSentDate(
+          registrar,
+          DateTime.now(UTC).minusMinutes((int) UPDATE_TIME_OFFSET.getStandardMinutes()),
+          certificateType);
+      return true;
+    } catch (Exception e) {
+      throw new RuntimeException(
+          String.format(
+              "Failed to send expiring certificate notification email to registrar %s.",
+              registrar.getRegistrarName()));
+    }
+  }
+
+  /** Updates the last notification sent date in database. */
+  @VisibleForTesting
+  void updateLastNotificationSentDate(
+      Registrar registrar, DateTime now, CertificateType certificateType) {
+    try {
+      tm().transact(
+              () -> {
+                Registrar.Builder newRegistrar = tm().loadByEntity(registrar).asBuilder();
+                switch (certificateType) {
+                  case PRIMARY:
+                    newRegistrar.setLastExpiringCertNotificationSentDate(now);
+                    tm().put(newRegistrar.build());
+                    logger.atInfo().log(
+                        "Updated last notification email sent date to %s for %s certificate of "
+                            + "registrar %s.",
+                        DATE_FORMATTER.print(now),
+                        certificateType.getDisplayName(),
+                        registrar.getRegistrarName());
+                    break;
+                  case FAILOVER:
+                    newRegistrar.setLastExpiringFailoverCertNotificationSentDate(now);
+                    tm().put(newRegistrar.build());
+                    logger.atInfo().log(
+                        "Updated last notification email sent date to %s for %s certificate of "
+                            + "registrar %s.",
+                        DATE_FORMATTER.print(now),
+                        certificateType.getDisplayName(),
+                        registrar.getRegistrarName());
+                    break;
+                  default:
+                    throw new IllegalArgumentException(
+                        String.format(
+                            "Unsupported certificate type: %s being passed in when updating "
+                                + "the last notification sent date to registrar %s.",
+                            certificateType.toString(), registrar.getRegistrarName()));
+                }
+              });
+    } catch (Exception e) {
+      throw new RuntimeException(
+          String.format(
+              "Failed to update the last notification sent date to Registrar %s for the %s "
+                  + "certificate.",
+              registrar.getRegistrarName(), certificateType.getDisplayName()));
+    }
+  }
+
+  /** Sends notification emails to registrars with expiring certificates. */
+  @VisibleForTesting
+  int sendNotificationEmails() {
+    int emailsSent = 0;
+    for (RegistrarInfo registrarInfo : getRegistrarsWithExpiringCertificates()) {
+      Registrar registrar = registrarInfo.registrar();
+      if (registrarInfo.isCertExpiring()) {
+        sendNotificationEmail(
+            registrar,
+            registrar.getLastExpiringCertNotificationSentDate(),
+            CertificateType.PRIMARY,
+            registrar.getClientCertificate());
+        emailsSent++;
+      }
+      if (registrarInfo.isFailOverCertExpiring()) {
+        sendNotificationEmail(
+            registrar,
+            registrar.getLastExpiringFailoverCertNotificationSentDate(),
+            CertificateType.FAILOVER,
+            registrar.getFailoverClientCertificate());
+        emailsSent++;
+      }
+    }
+    logger.atInfo().log(
+        "Attempted to send %d expiring certificate notification emails.", emailsSent);
+    return emailsSent;
+  }
+
+  /** Returns a list of email addresses of the registrar that should receive a notification email */
+  @VisibleForTesting
+  ImmutableSet<InternetAddress> getEmailAddresses(Registrar registrar, Type contactType) {
+    ImmutableSortedSet<RegistrarContact> contacts = registrar.getContactsOfType(contactType);
+    ImmutableSet.Builder<InternetAddress> recipientEmails = new ImmutableSet.Builder<>();
+    for (RegistrarContact contact : contacts) {
+      try {
+        recipientEmails.add(new InternetAddress(contact.getEmailAddress()));
+      } catch (AddressException e) {
+        logger.atWarning().withCause(e).log(
+            "Registrar Contact email address %s of Registrar %s is invalid; skipping.",
+            contact.getEmailAddress(), registrar.getRegistrarName());
+      }
+    }
+    return recipientEmails.build();
+  }
+
+  /**
+   * Generates email content by taking registrar name, certificate type and expiration date as
+   * parameters.
+   */
+  @VisibleForTesting
+  @SuppressWarnings("lgtm[java/dereferenced-value-may-be-null]")
+  String getEmailBody(String registrarName, CertificateType type, Date expirationDate) {
+    checkArgumentNotNull(expirationDate, "Expiration date cannot be null");
+    checkArgumentNotNull(type, "Certificate type cannot be null");
+    return String.format(
+        expirationWarningEmailBodyText,
+        registrarName,
+        type.getDisplayName(),
+        DATE_FORMATTER.print(new DateTime(expirationDate)));
+  }
+
+  /**
+   * Certificate types for X509Certificate.
+   *
+   * <p><b>Note:</b> These types are only used to indicate the type of expiring certificate in
+   * notification emails.
+   */
+  protected enum CertificateType {
+    PRIMARY("primary"),
+    FAILOVER("fail-over");
+
+    private final String displayName;
+
+    CertificateType(String displayName) {
+      this.displayName = displayName;
+    }
+
+    public String getDisplayName() {
+      return displayName;
+    }
+  }
+
+  @AutoValue
+  public abstract static class RegistrarInfo {
+    static RegistrarInfo create(
+        Registrar registrar, boolean isCertExpiring, boolean isFailOverCertExpiring) {
+      return new AutoValue_SendExpiringCertificateNotificationEmailAction_RegistrarInfo(
+          registrar, isCertExpiring, isFailOverCertExpiring);
+    }
+
+    public abstract Registrar registrar();
+
+    public abstract boolean isCertExpiring();
+
+    public abstract boolean isFailOverCertExpiring();
+  }
+}
@@ -0,0 +1,65 @@
+// Copyright 2021 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.config;
+
+import com.google.api.gax.core.FixedCredentialsProvider;
+import com.google.cloud.tasks.v2.CloudTasksClient;
+import com.google.cloud.tasks.v2.CloudTasksSettings;
+import dagger.Module;
+import dagger.Provides;
+import google.registry.config.CredentialModule.DefaultCredential;
+import google.registry.config.RegistryConfig.Config;
+import google.registry.util.CloudTasksUtils;
+import google.registry.util.GoogleCredentialsBundle;
+import google.registry.util.Retrier;
+import java.io.IOException;
+import javax.inject.Provider;
+import javax.inject.Singleton;
+
+/**
+ * A {@link Module} that provides {@link CloudTasksUtils}.
+ *
+ * <p>The class itself cannot be annotated as {@code Inject} because its requested dependencies use
+ * the {@link Config} qualifier which is not available in the {@code util} package.
+ */
+@Module
+public abstract class CloudTasksUtilsModule {
+
+  @Singleton
+  @Provides
+  public static CloudTasksUtils provideCloudTasksUtils(
+      @Config("projectId") String projectId,
+      @Config("locationId") String locationId,
+      // Use a provider so that we can use try-with-resources with the client, which implements
+      // Autocloseable.
+      Provider<CloudTasksClient> clientProvider,
+      Retrier retrier) {
+    return new CloudTasksUtils(retrier, projectId, locationId, clientProvider);
+  }
+
+  @Provides
+  public static CloudTasksClient provideCloudTasksClient(
+      @DefaultCredential GoogleCredentialsBundle credentials) {
+    try {
+      return CloudTasksClient.create(
+          CloudTasksSettings.newBuilder()
+              .setCredentialsProvider(
+                  FixedCredentialsProvider.create(credentials.getGoogleCredentials()))
+              .build());
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+}
@@ -105,6 +105,12 @@ public final class RegistryConfig {
      return config.appEngine.projectId;
    }

+    @Provides
+    @Config("locationId")
+    public static String provideLocationId(RegistryConfigSettings config) {
+      return config.appEngine.locationId;
+    }
+
    /**
     * The filename of the logo to be displayed in the header of the registrar console.
     *
@@ -45,6 +45,7 @@ public class RegistryConfigSettings {
  /** Configuration options that apply to the entire App Engine project. */
  public static class AppEngine {
    public String projectId;
+    public String locationId;
    public boolean isLocal;
    public String defaultServiceUrl;
    public String backendServiceUrl;
@@ -8,6 +8,10 @@
 appEngine:
  # Globally unique App Engine project ID
  projectId: registry-project-id
+  # Location of the App engine project, note that us-central1 and europe-west1 are special in that
+  # they are used without the trailing number in App Engine commands and Google Cloud Console.
+  # See: https://cloud.google.com/appengine/docs/locations
+  locationId: registry-location-id

  # whether to use local/test credentials when connecting to the servers
  isLocal: true
@@ -14,14 +14,10 @@

 package google.registry.cron;

-import static com.google.appengine.api.taskqueue.QueueFactory.getQueue;
-import static com.google.appengine.api.taskqueue.TaskOptions.Builder.withUrl;
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Predicates.in;
 import static com.google.common.base.Predicates.not;
-import static com.google.common.base.Strings.nullToEmpty;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
-import static com.google.common.collect.Iterables.getFirst;
 import static com.google.common.collect.Multimaps.filterKeys;
 import static com.google.common.net.MediaType.PLAIN_TEXT_UTF_8;
 import static google.registry.cron.CronModule.ENDPOINT_PARAM;
@@ -36,21 +32,24 @@ import static google.registry.model.tld.Registry.TldType.REAL;
 import static google.registry.model.tld.Registry.TldType.TEST;
 import static java.util.concurrent.TimeUnit.SECONDS;

-import com.google.appengine.api.taskqueue.Queue;
-import com.google.appengine.api.taskqueue.TaskHandle;
-import com.google.appengine.api.taskqueue.TaskOptions;
+import com.google.cloud.tasks.v2.Task;
+import com.google.common.collect.ArrayListMultimap;
 import com.google.common.collect.ImmutableListMultimap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Multimap;
 import com.google.common.collect.Streams;
 import com.google.common.flogger.FluentLogger;
+import com.google.protobuf.Timestamp;
 import google.registry.request.Action;
+import google.registry.request.Action.Service;
 import google.registry.request.Parameter;
 import google.registry.request.ParameterMap;
 import google.registry.request.RequestParameters;
 import google.registry.request.Response;
 import google.registry.request.auth.Auth;
-import google.registry.util.TaskQueueUtils;
+import google.registry.util.Clock;
+import google.registry.util.CloudTasksUtils;
+import java.time.Instant;
 import java.util.Optional;
 import java.util.Random;
 import java.util.stream.Stream;
@@ -105,7 +104,8 @@ public final class TldFanoutAction implements Runnable {

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

-  @Inject TaskQueueUtils taskQueueUtils;
+  @Inject Clock clock;
+  @Inject CloudTasksUtils cloudTasksUtils;
  @Inject Response response;
  @Inject @Parameter(ENDPOINT_PARAM) String endpoint;
  @Inject @Parameter(QUEUE_PARAM) String queue;
@@ -115,7 +115,9 @@ public final class TldFanoutAction implements Runnable {
  @Inject @Parameter(EXCLUDE_PARAM) ImmutableSet<String> excludes;
  @Inject @Parameter(JITTER_SECONDS_PARAM) Optional<Integer> jitterSeconds;
  @Inject @ParameterMap ImmutableListMultimap<String, String> params;
-  @Inject TldFanoutAction() {}
+
+  @Inject
+  TldFanoutAction() {}

  @Override
  public void run() {
@@ -126,8 +128,7 @@ public final class TldFanoutAction implements Runnable {
        runInEmpty || forEachTestTld || forEachRealTld,
        "At least one of runInEmpty, forEachTestTld, forEachRealTld must be given");
    checkArgument(
-        !(runInEmpty && !excludes.isEmpty()),
-        "Can't specify 'exclude' with 'runInEmpty'");
+        !(runInEmpty && !excludes.isEmpty()), "Can't specify 'exclude' with 'runInEmpty'");
    ImmutableSet<String> tlds =
        runInEmpty
            ? ImmutableSet.of("")
@@ -137,7 +138,6 @@ public final class TldFanoutAction implements Runnable {
                .filter(not(in(excludes)))
                .collect(toImmutableSet());
    Multimap<String, String> flowThruParams = filterKeys(params, not(in(CONTROL_PARAMS)));
-    Queue taskQueue = getQueue(queue);
    StringBuilder outputPayload =
        new StringBuilder(
            String.format("OK: Launched the following %d tasks in queue %s\n", tlds.size(), queue));
@@ -146,33 +146,41 @@ public final class TldFanoutAction implements Runnable {
      logger.atWarning().log("No TLDs to fan-out!");
    }
    for (String tld : tlds) {
-      TaskOptions taskOptions = createTaskOptions(tld, flowThruParams);
-      TaskHandle taskHandle = taskQueueUtils.enqueue(taskQueue, taskOptions);
+      Task task = createTask(tld, flowThruParams);
+      Task createdTask = cloudTasksUtils.enqueue(queue, task);
      outputPayload.append(
          String.format(
              "- Task: '%s', tld: '%s', endpoint: '%s'\n",
-              taskHandle.getName(), tld, taskOptions.getUrl()));
+              createdTask.getName(), tld, createdTask.getAppEngineHttpRequest().getRelativeUri()));
      logger.atInfo().log(
-          "Task: '%s', tld: '%s', endpoint: '%s'", taskHandle.getName(), tld, taskOptions.getUrl());
+          "Task: '%s', tld: '%s', endpoint: '%s'",
+          createdTask.getName(), tld, createdTask.getAppEngineHttpRequest().getRelativeUri());
    }
    response.setContentType(PLAIN_TEXT_UTF_8);
    response.setPayload(outputPayload.toString());
  }

-  private TaskOptions createTaskOptions(String tld, Multimap<String, String> params) {
-    TaskOptions options =
-        withUrl(endpoint)
-            .countdownMillis(
-                jitterSeconds
-                    .map(seconds -> random.nextInt((int) SECONDS.toMillis(seconds)))
-                    .orElse(0));
+  private Task createTask(String tld, Multimap<String, String> params) {
    if (!tld.isEmpty()) {
-      options.param(RequestParameters.PARAM_TLD, tld);
+      params = ArrayListMultimap.create(params);
+      params.put(RequestParameters.PARAM_TLD, tld);
    }
-    for (String param : params.keySet()) {
-      // TaskOptions.param() does not accept null values.
-      options.param(param, nullToEmpty(getFirst(params.get(param), null)));
-    }
-    return options;
+    Instant scheduleTime =
+        Instant.ofEpochMilli(
+            clock
+                .nowUtc()
+                .plusMillis(
+                    jitterSeconds
+                        .map(seconds -> random.nextInt((int) SECONDS.toMillis(seconds)))
+                        .orElse(0))
+                .getMillis());
+    return Task.newBuilder(
+            CloudTasksUtils.createPostTask(endpoint, Service.BACKEND.toString(), params))
+        .setScheduleTime(
+            Timestamp.newBuilder()
+                .setSeconds(scheduleTime.getEpochSecond())
+                .setNanos(scheduleTime.getNano())
+                .build())
+        .build();
  }
 }
@@ -355,6 +355,12 @@
    <url-pattern>/_dr/task/deleteExpiredDomains</url-pattern>
  </servlet-mapping>

+  <!-- Background action to send notification emails to registrars with expiring certificate. -->
+  <servlet-mapping>
+    <servlet-name>backend-servlet</servlet-name>
+    <url-pattern>/_dr/task/sendExpiringCertificateNotificationEmail</url-pattern>
+  </servlet-mapping>
+
  <!-- Mapreduce to import contacts from escrow file -->
  <servlet-mapping>
    <servlet-name>backend-servlet</servlet-name>
@@ -143,7 +143,7 @@
      It also enqueues a new task to wait on the completion of that job and then load the resulting
      snapshot into bigquery.
    </description>
-    <!--
+    <!- -
      Keep google.registry.export.CheckBackupAction.MAXIMUM_BACKUP_RUNNING_TIME less than
      this interval. - ->
    <schedule>every day 06:00</schedule>
@@ -330,4 +330,13 @@
    <schedule>every day 15:00</schedule>
    <target>backend</target>
  </cron>
+
+  <cron>
+    <url><![CDATA[/_dr/cron/fanout?queue=replay-commit-logs-to-sql&endpoint=/_dr/task/replayCommitLogsToSql&runInEmpty]]></url>
+    <description>
+      Replays recent commit logs from Datastore to the SQL secondary backend.
+    </description>
+    <schedule>every 3 minutes</schedule>
+    <target>backend</target>
+  </cron>
 </cronentries>
@@ -168,6 +168,15 @@
    <target>backend</target>
  </cron>

+  <cron>
+    <url><![CDATA[/_dr/task/sendExpiringCertificateNotificationEmail]]></url>
+    <description>
+      This job runs an action that sends emails to partners if their certificates are expiring soon.
+    </description>
+    <schedule>every day 04:30</schedule>
+    <target>backend</target>
+  </cron>
+
  <cron>
    <url><![CDATA[/_dr/cron/fanout?queue=export-snapshot&endpoint=/_dr/task/backupDatastore&runInEmpty]]></url>
    <description>
@@ -43,10 +43,6 @@ import google.registry.model.reporting.HistoryEntry;
 import google.registry.model.server.Lock;
 import google.registry.model.server.ServerSecret;
 import google.registry.model.tld.Registry;
-import google.registry.model.tld.label.ReservedList;
-import google.registry.model.tmch.ClaimsList;
-import google.registry.model.tmch.ClaimsList.ClaimsListRevision;
-import google.registry.model.tmch.ClaimsList.ClaimsListSingleton;
 import google.registry.model.tmch.TmchCrl;

 /** Sets of classes of the Objectify-registered entities in use throughout the model. */
@@ -60,9 +56,6 @@ public final class EntityClasses {
          BillingEvent.Modification.class,
          BillingEvent.OneTime.class,
          BillingEvent.Recurring.class,
-          ClaimsList.class,
-          ClaimsListRevision.class,
-          ClaimsListSingleton.class,
          CommitLogBucket.class,
          CommitLogCheckpoint.class,
          CommitLogCheckpointRoot.class,
@@ -92,7 +85,6 @@ public final class EntityClasses {
          Registrar.class,
          RegistrarContact.class,
          Registry.class,
-          ReservedList.class,
          ServerSecret.class,
          TmchCrl.class);

@@ -20,6 +20,7 @@ import static com.google.common.collect.Sets.difference;
 import static com.google.common.collect.Sets.union;
 import static google.registry.config.RegistryConfig.getEppResourceCachingDuration;
 import static google.registry.config.RegistryConfig.getEppResourceMaxCachedEntries;
+import static google.registry.persistence.transaction.TransactionManagerFactory.ofyTm;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.CollectionUtils.nullToEmpty;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
@@ -37,6 +38,8 @@ import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Index;
 import google.registry.config.RegistryConfig;
 import google.registry.model.eppcommon.StatusValue;
+import google.registry.model.index.EppResourceIndex;
+import google.registry.model.index.ForeignKeyIndex;
 import google.registry.model.ofy.CommitLogManifest;
 import google.registry.model.transfer.TransferData;
 import google.registry.persistence.VKey;
@@ -219,6 +222,14 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
  @Override
  public abstract Builder<?, ?> asBuilder();

+  /** Used when replaying from SQL to DS to populate the Datastore indexes. */
+  protected void saveIndexesToDatastore() {
+    ofyTm()
+        .putAll(
+            ForeignKeyIndex.create(this, getDeletionTime()),
+            EppResourceIndex.create(Key.create(this)));
+  }
+
  /** EppResources that are loaded via foreign keys should implement this marker interface. */
  public interface ForeignKeyedEppResource {}

@@ -36,8 +36,8 @@ import org.joda.time.DateTime;
 /**
 * A wrapper object representing the stage-to-time mapping of the Registry 3.0 Cloud SQL migration.
 *
- * <p>The entity is stored in Datastore throughout the entire migration so as to have a single point
- * of access.
+ * <p>The entity is stored in SQL throughout the entire migration so as to have a single point of
+ * access.
 */
@Entity
 public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements SqlOnlyEntity {
@@ -187,12 +187,12 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
  private DatabaseMigrationStateSchedule() {}

  @VisibleForTesting
-  DatabaseMigrationStateSchedule(
+  public DatabaseMigrationStateSchedule(
      TimedTransitionProperty<MigrationState, MigrationStateTransition> migrationTransitions) {
    this.migrationTransitions = migrationTransitions;
  }

-  /** Sets and persists to Datastore the provided migration transition schedule. */
+  /** Sets and persists to SQL the provided migration transition schedule. */
  public static void set(ImmutableSortedMap<DateTime, MigrationState> migrationTransitionMap) {
    jpaTm().assertInTransaction();
    TimedTransitionProperty<MigrationState, MigrationStateTransition> transitions =
@@ -204,7 +204,7 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
            MigrationState.DATASTORE_ONLY,
            "migrationTransitionMap must start with DATASTORE_ONLY");
    validateTransitionAtCurrentTime(transitions);
-    jpaTm().put(new DatabaseMigrationStateSchedule(transitions));
+    jpaTm().putIgnoringReadOnly(new DatabaseMigrationStateSchedule(transitions));
    CACHE.invalidateAll();
  }

@@ -218,7 +218,7 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
    return get().getValueAtTime(dateTime);
  }

-  /** Loads the currently-set migration schedule from Datastore, or the default if none exists. */
+  /** Loads the currently-set migration schedule from SQL, or the default if none exists. */
  @VisibleForTesting
  static TimedTransitionProperty<MigrationState, MigrationStateTransition> getUncached() {
    return jpaTm()
@@ -66,6 +66,11 @@ public class ContactResource extends ContactBase
    return ContactBase.cloneContactProjectedAtTime(this, now);
  }

+  @Override
+  public void beforeDatastoreSaveOnReplay() {
+    saveIndexesToDatastore();
+  }
+
  @Override
  public Builder asBuilder() {
    return new Builder(clone(this));
@@ -163,6 +163,11 @@ public class DomainBase extends DomainContent
    return cloneDomainProjectedAtTime(this, now);
  }

+  @Override
+  public void beforeDatastoreSaveOnReplay() {
+    saveIndexesToDatastore();
+  }
+
  public static VKey<DomainBase> createVKey(Key<DomainBase> key) {
    return VKey.create(DomainBase.class, key.getName(), key);
  }
@@ -51,6 +51,11 @@ public class HostResource extends HostBase
    return VKey.create(HostResource.class, getRepoId(), Key.create(this));
  }

+  @Override
+  public void beforeDatastoreSaveOnReplay() {
+    saveIndexesToDatastore();
+  }
+
  @Override
  public Builder asBuilder() {
    return new Builder(clone(this));
@@ -336,7 +336,7 @@ public class DatastoreTransactionManager implements TransactionManager {

  @Override
  public <T> QueryComposer<T> createQueryComposer(Class<T> entity) {
-    return new DatastoreQueryComposerImpl(entity);
+    return new DatastoreQueryComposerImpl<>(entity);
  }

  @Override
@@ -349,6 +349,16 @@ public class DatastoreTransactionManager implements TransactionManager {
    return true;
  }

+  @Override
+  public void putIgnoringReadOnly(Object entity) {
+    syncIfTransactionless(getOfy().saveIgnoringReadOnly().entities(toDatastoreEntity(entity)));
+  }
+
+  @Override
+  public void deleteIgnoringReadOnly(VKey<?> key) {
+    syncIfTransactionless(getOfy().deleteIgnoringReadOnly().key(key.getOfyKey()));
+  }
+
  /**
   * Executes the given {@link Result} instance synchronously if not in a transaction.
   *
@@ -19,6 +19,7 @@ import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.collect.Maps.uniqueIndex;
 import static com.googlecode.objectify.ObjectifyService.ofy;
 import static google.registry.config.RegistryConfig.getBaseOfyRetryDuration;
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertNotReadOnlyMode;
 import static google.registry.util.CollectionUtils.union;

 import com.google.appengine.api.datastore.DatastoreFailureException;
@@ -131,6 +132,7 @@ public class Ofy {
   * <p>We only allow this in transactions so commit logs can be written in tandem with the delete.
   */
  public Deleter delete() {
+    assertNotReadOnlyMode();
    return new AugmentedDeleter() {
      @Override
      protected void handleDeletion(Iterable<Key<?>> keys) {
@@ -148,12 +150,8 @@ public class Ofy {
   * <p>No backups get written.
   */
  public Deleter deleteWithoutBackup() {
-    return new AugmentedDeleter() {
-      @Override
-      protected void handleDeletion(Iterable<Key<?>> keys) {
-        checkProhibitedAnnotations(keys, VirtualEntity.class);
-      }
-    };
+    assertNotReadOnlyMode();
+    return deleteIgnoringReadOnly();
  }

  /**
@@ -163,6 +161,7 @@ public class Ofy {
   * <p>We only allow this in transactions so commit logs can be written in tandem with the save.
   */
  public Saver save() {
+    assertNotReadOnlyMode();
    return new AugmentedSaver() {
      @Override
      protected void handleSave(Iterable<?> entities) {
@@ -182,6 +181,12 @@ public class Ofy {
   * <p>No backups get written.
   */
  public Saver saveWithoutBackup() {
+    assertNotReadOnlyMode();
+    return saveIgnoringReadOnly();
+  }
+
+  /** Save, ignoring any backups or any read-only settings. */
+  public Saver saveIgnoringReadOnly() {
    return new AugmentedSaver() {
      @Override
      protected void handleSave(Iterable<?> entities) {
@@ -190,6 +195,16 @@ public class Ofy {
    };
  }

+  /** Delete, ignoring any backups or any read-only settings. */
+  public Deleter deleteIgnoringReadOnly() {
+    return new AugmentedDeleter() {
+      @Override
+      protected void handleDeletion(Iterable<Key<?>> keys) {
+        checkProhibitedAnnotations(keys, VirtualEntity.class);
+      }
+    };
+  }
+
  private Clock getClock() {
    return injectedClock == null ? clock : injectedClock;
  }
@@ -39,7 +39,7 @@ public final class RdeNamingUtils {
  }

  /** Returns same thing as {@link #makeRydeFilename} except without the series and revision. */
-  static String makePartialName(String tld, DateTime date, RdeMode mode) {
+  public static String makePartialName(String tld, DateTime date, RdeMode mode) {
    return String.format("%s_%s_%s",
        checkNotNull(tld), formatDate(date), mode.getFilenameComponent());
  }
@@ -27,4 +27,7 @@ import java.util.Optional;
 public interface DatastoreEntity {

  Optional<SqlEntity> toSqlEntity();
+
+  /** A method called before the object is saved to Datastore in asynchronous replay. */
+  default void beforeDatastoreSaveOnReplay() {}
 }
@@ -22,6 +22,7 @@ import static google.registry.request.Action.Method.GET;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableList;
 import com.google.common.flogger.FluentLogger;
+import google.registry.model.UpdateAutoTimestamp;
 import google.registry.model.common.DatabaseMigrationStateSchedule;
 import google.registry.model.common.DatabaseMigrationStateSchedule.MigrationState;
 import google.registry.model.common.DatabaseMigrationStateSchedule.ReplayDirection;
@@ -64,7 +65,7 @@ public class ReplicateToDatastoreAction implements Runnable {
  @VisibleForTesting
  public List<TransactionEntity> getTransactionBatch() {
    // Get the next batch of transactions that we haven't replicated.
-    LastSqlTransaction lastSqlTxnBeforeBatch = ofyTm().transact(() -> LastSqlTransaction.load());
+    LastSqlTransaction lastSqlTxnBeforeBatch = ofyTm().transact(LastSqlTransaction::load);
    try {
      return jpaTm()
          .transactWithoutBackup(
@@ -85,53 +86,57 @@ public class ReplicateToDatastoreAction implements Runnable {
  /**
   * Apply a transaction to Datastore, returns true if there was a fatal error and the batch should
   * be aborted.
+   *
+   * <p>Throws an exception if a fatal error occurred and the batch should be aborted
   */
  @VisibleForTesting
-  public boolean applyTransaction(TransactionEntity txnEntity) {
+  public void applyTransaction(TransactionEntity txnEntity) {
    logger.atInfo().log("Applying a single transaction Cloud SQL -> Cloud Datastore");
-    return ofyTm()
-        .transact(
-            () -> {
-              // Reload the last transaction id, which could possibly have changed.
-              LastSqlTransaction lastSqlTxn = LastSqlTransaction.load();
-              long nextTxnId = lastSqlTxn.getTransactionId() + 1;
-              if (nextTxnId < txnEntity.getId()) {
-                // We're missing a transaction.  This is bad.  Transaction ids are supposed to
-                // increase monotonically, so we abort rather than applying anything out of
-                // order.
-                logger.atSevere().log(
-                    "Missing transaction: last transaction id = %s, next available transaction "
-                        + "= %s",
-                    nextTxnId - 1, txnEntity.getId());
-                return true;
-              } else if (nextTxnId > txnEntity.getId()) {
-                // We've already replayed this transaction.  This shouldn't happen, as GAE cron
-                // is supposed to avoid overruns and this action shouldn't be executed from any
-                // other context, but it's not harmful as we can just ignore the transaction.  Log
-                // it so that we know about it and move on.
-                logger.atWarning().log(
-                    "Ignoring transaction %s, which appears to have already been applied.",
-                    txnEntity.getId());
-                return false;
-              }
+    try (UpdateAutoTimestamp.DisableAutoUpdateResource disabler =
+        UpdateAutoTimestamp.disableAutoUpdate()) {
+      ofyTm()
+          .transact(
+              () -> {
+                // Reload the last transaction id, which could possibly have changed.
+                LastSqlTransaction lastSqlTxn = LastSqlTransaction.load();
+                long nextTxnId = lastSqlTxn.getTransactionId() + 1;
+                if (nextTxnId < txnEntity.getId()) {
+                  // We're missing a transaction.  This is bad.  Transaction ids are supposed to
+                  // increase monotonically, so we abort rather than applying anything out of
+                  // order.
+                  throw new IllegalStateException(
+                      String.format(
+                          "Missing transaction: last txn id = %s, next available txn = %s",
+                          nextTxnId - 1, txnEntity.getId()));
+                } else if (nextTxnId > txnEntity.getId()) {
+                  // We've already replayed this transaction.  This shouldn't happen, as GAE cron
+                  // is supposed to avoid overruns and this action shouldn't be executed from any
+                  // other context, but it's not harmful as we can just ignore the transaction.  Log
+                  // it so that we know about it and move on.
+                  logger.atWarning().log(
+                      "Ignoring transaction %s, which appears to have already been applied.",
+                      txnEntity.getId());
+                  return;
+                }

-              logger.atInfo().log("Applying transaction %s to Cloud Datastore", txnEntity.getId());
+                logger.atInfo().log(
+                    "Applying transaction %s to Cloud Datastore", txnEntity.getId());

-              // At this point, we know txnEntity is the correct next transaction, so write it
-              // to datastore.
-              try {
-                Transaction.deserialize(txnEntity.getContents()).writeToDatastore();
-              } catch (IOException e) {
-                throw new RuntimeException("Error during transaction deserialization.", e);
-              }
+                // At this point, we know txnEntity is the correct next transaction, so write it
+                // to datastore.
+                try {
+                  Transaction.deserialize(txnEntity.getContents()).writeToDatastore();
+                } catch (IOException e) {
+                  throw new RuntimeException("Error during transaction deserialization.", e);
+                }

-              // Write the updated last transaction id to datastore as part of this datastore
-              // transaction.
-              auditedOfy().save().entity(lastSqlTxn.cloneWithNewTransactionId(nextTxnId));
-              logger.atInfo().log(
-                  "Finished applying single transaction Cloud SQL -> Cloud Datastore");
-              return false;
-            });
+                // Write the updated last transaction id to datastore as part of this datastore
+                // transaction.
+                auditedOfy().save().entity(lastSqlTxn.cloneWithNewTransactionId(nextTxnId));
+                logger.atInfo().log(
+                    "Finished applying single transaction Cloud SQL -> Cloud Datastore");
+              });
+    }
  }

  @Override
@@ -139,18 +144,23 @@ public class ReplicateToDatastoreAction implements Runnable {
    MigrationState state = DatabaseMigrationStateSchedule.getValueAtTime(clock.nowUtc());
    if (!state.getReplayDirection().equals(ReplayDirection.SQL_TO_DATASTORE)) {
      logger.atInfo().log(
-          String.format(
-              "Skipping ReplicateToDatastoreAction because we are in migration phase %s.", state));
+          "Skipping ReplicateToDatastoreAction because we are in migration phase %s.", state);
      return;
    }
    // TODO(b/181758163): Deal with objects that don't exist in Cloud SQL, e.g. ForeignKeyIndex,
    // EppResourceIndex.
    logger.atInfo().log("Processing transaction replay batch Cloud SQL -> Cloud Datastore");
+    int numTransactionsReplayed = 0;
    for (TransactionEntity txnEntity : getTransactionBatch()) {
-      if (applyTransaction(txnEntity)) {
-        break;
+      try {
+        applyTransaction(txnEntity);
+      } catch (Throwable t) {
+        logger.atSevere().withCause(t).log("Errored out replaying files");
+        return;
      }
+      numTransactionsReplayed++;
    }
-    logger.atInfo().log("Done processing transaction replay batch Cloud SQL -> Cloud Datastore");
+    logger.atInfo().log(
+        "Replayed %d transactions from Cloud SQL -> Datastore", numTransactionsReplayed);
  }
 }
@@ -29,10 +29,10 @@ public interface SqlEntity {

  Optional<DatastoreEntity> toDatastoreEntity();

-  /** A method that will ber called before the object is saved to SQL in asynchronous replay. */
+  /** A method that will be called before the object is saved to SQL in asynchronous replay. */
  default void beforeSqlSaveOnReplay() {}

-  /* Returns this entity's primary key field(s) in a string. */
+  /** Returns this entity's primary key field(s) in a string. */
  default String getPrimaryKeyString() {
    return jpaTm()
        .transact(
@@ -41,6 +41,6 @@ public class SqlReplayCheckpoint extends CrossTldSingleton implements SqlOnlyEnt
    SqlReplayCheckpoint checkpoint = new SqlReplayCheckpoint();
    checkpoint.lastReplayTime = lastReplayTime;
    // this will overwrite the existing object due to the constant revisionId
-    jpaTm().put(checkpoint);
+    jpaTm().putIgnoringReadOnly(checkpoint);
  }
 }
@@ -250,7 +250,7 @@ public class Lock extends ImmutableObject implements DatastoreAndSqlEntity, Seri
                          resourceName, scope, requestStatusChecker.getLogId(), now, leaseLength);
                  // Locks are not parented under an EntityGroupRoot (so as to avoid write
                  // contention) and don't need to be backed up.
-                  tm().putWithoutBackup(newLock);
+                  tm().putIgnoringReadOnly(newLock);

                  return AcquireResult.create(now, lock, newLock, lockState);
                });
@@ -269,18 +269,15 @@ public class Lock extends ImmutableObject implements DatastoreAndSqlEntity, Seri
              // delete it. If the lock in Datastore was different then this lock is gone already;
              // this can happen if release() is called around the expiration time and the lock
              // expires underneath us.
-              Lock loadedLock =
-                  tm().loadByKeyIfPresent(
-                          VKey.create(
-                              Lock.class,
-                              new LockId(resourceName, tld),
-                              Key.create(Lock.class, lockId)))
-                      .orElse(null);
+              VKey<Lock> key =
+                  VKey.create(
+                      Lock.class, new LockId(resourceName, tld), Key.create(Lock.class, lockId));
+              Lock loadedLock = tm().loadByKeyIfPresent(key).orElse(null);
              if (Lock.this.equals(loadedLock)) {
                // Use deleteWithoutBackup() so that we don't create a commit log entry for deleting
                // the lock.
                logger.atInfo().log("Deleting lock: %s", lockId);
-                tm().deleteWithoutBackup(Lock.this);
+                tm().deleteIgnoringReadOnly(key);

                lockMetrics.recordRelease(
                    resourceName, tld, new Duration(acquiredTime, tm().getTransactionTime()));
@@ -33,13 +33,8 @@ import com.google.common.cache.LoadingCache;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.util.concurrent.UncheckedExecutionException;
-import com.googlecode.objectify.annotation.Embed;
-import com.googlecode.objectify.annotation.Entity;
-import com.googlecode.objectify.annotation.Mapify;
-import com.googlecode.objectify.mapper.Mapper;
 import google.registry.model.Buildable;
-import google.registry.model.annotations.ReportedOn;
-import google.registry.model.replay.NonReplicatedEntity;
+import google.registry.model.replay.SqlOnlyEntity;
 import google.registry.model.tld.Registry;
 import google.registry.model.tld.label.DomainLabelMetrics.MetricsReservedListMatch;
 import java.io.Serializable;
@@ -65,13 +60,11 @@ import org.joda.time.DateTime;
 * succeeds, we will end up with having two exact same reserved lists that differ only by
 * revisionId. This is fine though, because we only use the list with the highest revisionId.
 */
-@Entity
-@ReportedOn
@javax.persistence.Entity
@Table(indexes = {@Index(columnList = "name", name = "reservedlist_name_idx")})
 public final class ReservedList
    extends BaseDomainLabelList<ReservationType, ReservedList.ReservedListEntry>
-    implements NonReplicatedEntity {
+    implements SqlOnlyEntity {

  /**
   * Mapping from domain name to its reserved list info.
@@ -80,7 +73,6 @@ public final class ReservedList
   * from the immutability contract so we can modify it after construction and we have to handle the
   * database processing on our own so we can detach it after load.
   */
-  @Mapify(ReservedListEntry.LabelMapper.class)
  @Insignificant
  @Transient
  Map<String, ReservedListEntry> reservedListMap;
@@ -121,10 +113,9 @@ public final class ReservedList
   * A reserved list entry entity, persisted to Datastore, that represents a single label and its
   * reservation type.
   */
-  @Embed
  @javax.persistence.Entity(name = "ReservedEntry")
  public static class ReservedListEntry extends DomainLabelEntry<ReservationType, ReservedListEntry>
-      implements Buildable, NonReplicatedEntity, Serializable {
+      implements Buildable, SqlOnlyEntity, Serializable {

    @Insignificant @Id Long revisionId;

@@ -133,15 +124,6 @@ public final class ReservedList

    String comment;

-    /** Mapper for use with @Mapify */
-    static class LabelMapper implements Mapper<String, ReservedListEntry> {
-
-      @Override
-      public String getKey(ReservedListEntry entry) {
-        return entry.getDomainLabel();
-      }
-    }
-
    public String getComment(String comment) {
      return comment;
    }
@@ -239,7 +221,7 @@ public final class ReservedList
   * @return An Optional&lt;ReservedList&gt; that has a value if a reserved list exists by the given
   *     name, or absent if not.
   * @throws UncheckedExecutionException if some other error occurs while trying to load the
-   *     ReservedList from the cache or Datastore.
+   *     ReservedList from the cache or database.
   */
  public static Optional<ReservedList> get(String listName) {
    return getFromCache(listName, cache);
@@ -17,34 +17,21 @@ package google.registry.model.tmch;
 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.collect.ImmutableMap.toImmutableMap;
-import static google.registry.model.IdService.allocateId;
-import static google.registry.model.ofy.ObjectifyService.auditedOfy;
 import static google.registry.persistence.transaction.QueryComposer.Comparator.EQ;
 import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;

-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableMap;
-import com.googlecode.objectify.Key;
-import com.googlecode.objectify.annotation.Entity;
-import com.googlecode.objectify.annotation.Id;
-import com.googlecode.objectify.annotation.Ignore;
-import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.CreateAutoTimestamp;
 import google.registry.model.ImmutableObject;
-import google.registry.model.annotations.InCrossTld;
-import google.registry.model.annotations.NotBackedUp;
-import google.registry.model.annotations.NotBackedUp.Reason;
-import google.registry.model.annotations.VirtualEntity;
-import google.registry.model.common.CrossTldSingleton;
-import google.registry.model.replay.DatastoreOnlyEntity;
-import google.registry.model.replay.NonReplicatedEntity;
+import google.registry.model.replay.SqlOnlyEntity;
 import google.registry.model.tld.label.ReservedList.ReservedListEntry;
 import java.util.Map;
 import java.util.Optional;
-import javax.annotation.Nullable;
 import javax.persistence.Column;
+import javax.persistence.Entity;
 import javax.persistence.GeneratedValue;
 import javax.persistence.GenerationType;
+import javax.persistence.Id;
 import javax.persistence.PostPersist;
 import javax.persistence.PostUpdate;
 import javax.persistence.PreRemove;
@@ -60,26 +47,15 @@ import org.joda.time.DateTime;
 * succeeds, we will end up with having two exact same claims list with only different {@link
 * #revisionId}. However, this is not an actual problem because we only use the claims list with
 * highest {@link #revisionId}.
- *
- * <p>TODO(b/162007765): Remove Datastore related fields and methods.
 */
-@Entity
-@NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
-@javax.persistence.Entity(name = "ClaimsList")
+@Entity(name = "ClaimsList")
@Table
-@InCrossTld
-public class ClaimsList extends ImmutableObject implements NonReplicatedEntity {
+public class ClaimsList extends ImmutableObject implements SqlOnlyEntity {

-  @Transient @Id long id;
-
-  @Transient @Parent Key<ClaimsListRevision> parent;
-
-  @Ignore
-  @javax.persistence.Id
+  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  Long revisionId;

-  @Ignore
  @Column(nullable = false)
  CreateAutoTimestamp creationTimestamp = CreateAutoTimestamp.create(null);

@@ -217,65 +193,8 @@ public class ClaimsList extends ImmutableObject implements NonReplicatedEntity {
  public static ClaimsList create(
      DateTime tmdbGenerationTime, ImmutableMap<String, String> labelsToKeys) {
    ClaimsList instance = new ClaimsList();
-    instance.id = allocateId();
    instance.creationTime = checkNotNull(tmdbGenerationTime);
    instance.labelsToKeys = checkNotNull(labelsToKeys);
    return instance;
  }
-
-  /** Virtual parent entity for claims list shards of a specific revision. */
-  @Entity
-  @VirtualEntity
-  public static class ClaimsListRevision extends ImmutableObject implements DatastoreOnlyEntity {
-    @Parent Key<ClaimsListSingleton> parent;
-
-    @Id long versionId;
-
-    @VisibleForTesting
-    public static Key<ClaimsListRevision> createKey(ClaimsListSingleton singleton) {
-      ClaimsListRevision revision = new ClaimsListRevision();
-      revision.versionId = allocateId();
-      revision.parent = Key.create(singleton);
-      return Key.create(revision);
-    }
-
-    @VisibleForTesting
-    public static Key<ClaimsListRevision> createKey() {
-      return createKey(new ClaimsListSingleton());
-    }
-  }
-
-  /**
-   * Serves as the coordinating claims list singleton linking to the {@link ClaimsListRevision} that
-   * is live.
-   */
-  @Entity
-  @NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
-  public static class ClaimsListSingleton extends CrossTldSingleton implements DatastoreOnlyEntity {
-    Key<ClaimsListRevision> activeRevision;
-
-    static ClaimsListSingleton create(Key<ClaimsListRevision> revision) {
-      ClaimsListSingleton instance = new ClaimsListSingleton();
-      instance.activeRevision = revision;
-      return instance;
-    }
-
-    @VisibleForTesting
-    public void setActiveRevision(Key<ClaimsListRevision> revision) {
-      activeRevision = revision;
-    }
-  }
-
-  /**
-   * Returns the current ClaimsListRevision if there is one, or null if no claims list revisions
-   * have ever been persisted yet.
-   */
-  @Nullable
-  public static Key<ClaimsListRevision> getCurrentRevision() {
-    ClaimsListSingleton singleton = auditedOfy().load().entity(new ClaimsListSingleton()).now();
-    return singleton == null ? null : singleton.activeRevision;
-  }
-
-  /** Exception when trying to directly save a {@link ClaimsList} without sharding. */
-  public static class UnshardedSaveException extends RuntimeException {}
 }
@@ -157,10 +157,12 @@ public class DomainTransferData extends TransferData<DomainTransferData.Builder>
        DomainBase.restoreOfyFrom(rootKey, billingCancellationId, billingCancellationHistoryId);

    // Reconstruct server approve entities.  We currently have to call postLoad() a _second_ time
-    // if the billing cancellation id has been reconstituted, as it is part of that set.
+    // if any of the billing objects have been reconstituted, as they are part of that set.
    // TODO(b/183010623): Normalize the approaches to VKey reconstitution for the TransferData
    // hierarchy (the logic currently lives either in PostLoad or here, depending on the key).
-    if (billingCancellationId != null) {
+    if (billingCancellationId != null
+        || serverApproveBillingEvent != null
+        || serverApproveAutorenewEvent != null) {
      serverApproveEntities = null;
      postLoad();
    }
@@ -18,6 +18,7 @@ import com.google.monitoring.metrics.MetricReporter;
 import dagger.Component;
 import dagger.Lazy;
 import google.registry.bigquery.BigqueryModule;
+import google.registry.config.CloudTasksUtilsModule;
 import google.registry.config.CredentialModule;
 import google.registry.config.RegistryConfig.ConfigModule;
 import google.registry.dns.writer.VoidDnsWriterModule;
@@ -56,6 +57,7 @@ import javax.inject.Singleton;
      BackendRequestComponentModule.class,
      BigqueryModule.class,
      ConfigModule.class,
+      CloudTasksUtilsModule.class,
      CredentialModule.class,
      CustomLogicFactoryModule.class,
      DatastoreAdminModule.class,
@@ -31,6 +31,7 @@ import google.registry.batch.RefreshDnsOnHostRenameAction;
 import google.registry.batch.RelockDomainAction;
 import google.registry.batch.ResaveAllEppResourcesAction;
 import google.registry.batch.ResaveEntityAction;
+import google.registry.batch.SendExpiringCertificateNotificationEmailAction;
 import google.registry.batch.WipeOutCloudSqlAction;
 import google.registry.batch.WipeoutDatastoreAction;
 import google.registry.cron.CommitLogFanoutAction;
@@ -193,6 +194,8 @@ interface BackendRequestComponent {

  ResaveEntityAction resaveEntityAction();

+  SendExpiringCertificateNotificationEmailAction sendExpiringCertificateNotificationEmailAction();
+
  SyncGroupMembersAction syncGroupMembersAction();

  SyncRegistrarsSheetAction syncRegistrarsSheetAction();
@@ -30,7 +30,6 @@ import google.registry.request.RequestComponentBuilder;
 import google.registry.request.RequestModule;
 import google.registry.request.RequestScope;
 import google.registry.tools.server.CreateGroupsAction;
-import google.registry.tools.server.CreatePremiumListAction;
 import google.registry.tools.server.GenerateZoneFilesAction;
 import google.registry.tools.server.KillAllCommitLogsAction;
 import google.registry.tools.server.KillAllEppResourcesAction;
@@ -43,7 +42,6 @@ import google.registry.tools.server.ListTldsAction;
 import google.registry.tools.server.RefreshDnsForAllDomainsAction;
 import google.registry.tools.server.ResaveAllHistoryEntriesAction;
 import google.registry.tools.server.ToolsServerModule;
-import google.registry.tools.server.UpdatePremiumListAction;
 import google.registry.tools.server.VerifyOteAction;

 /** Dagger component with per-request lifetime for "tools" App Engine module. */
@@ -61,7 +59,6 @@ import google.registry.tools.server.VerifyOteAction;
    })
 interface ToolsRequestComponent {
  CreateGroupsAction createGroupsAction();
-  CreatePremiumListAction createPremiumListAction();
  EppToolAction eppToolAction();
  FlowComponent.Builder flowComponentBuilder();
  GenerateZoneFilesAction generateZoneFilesAction();
@@ -77,7 +74,6 @@ interface ToolsRequestComponent {
  RefreshDnsForAllDomainsAction refreshDnsForAllDomainsAction();
  ResaveAllHistoryEntriesAction resaveAllHistoryEntriesAction();
  RestoreCommitLogsAction restoreCommitLogsAction();
-  UpdatePremiumListAction updatePremiumListAction();
  VerifyOteAction verifyOteAction();

  @Subcomponent.Builder
@@ -1,36 +0,0 @@
-// Copyright 2020 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.persistence.converter;
-
-import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
-
-import com.googlecode.objectify.Key;
-import google.registry.model.tld.label.ReservedList;
-import javax.persistence.Converter;
-
-/** JPA converter for a set of {@link Key} containing a {@link ReservedList} */
-@Converter(autoApply = true)
-public class ReservedListKeySetConverter extends StringSetConverterBase<Key<ReservedList>> {
-
-  @Override
-  String toString(Key<ReservedList> key) {
-    return key.getName();
-  }
-
-  @Override
-  Key<ReservedList> fromString(String value) {
-    return Key.create(getCrossTldKey(), ReservedList.class, value);
-  }
-}
@@ -40,7 +40,6 @@ import google.registry.model.index.ForeignKeyIndex.ForeignKeyHostIndex;
 import google.registry.model.ofy.DatastoreTransactionManager;
 import google.registry.model.replay.NonReplicatedEntity;
 import google.registry.model.replay.SqlOnlyEntity;
-import google.registry.model.tmch.ClaimsList.ClaimsListSingleton;
 import google.registry.persistence.JpaRetries;
 import google.registry.persistence.VKey;
 import google.registry.util.Clock;
@@ -49,6 +48,8 @@ import google.registry.util.SystemSleeper;
 import java.io.Serializable;
 import java.lang.reflect.Array;
 import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
 import java.util.Calendar;
 import java.util.Collections;
 import java.util.Date;
@@ -74,7 +75,6 @@ import javax.persistence.TemporalType;
 import javax.persistence.TypedQuery;
 import javax.persistence.criteria.CriteriaQuery;
 import javax.persistence.metamodel.EntityType;
-import javax.persistence.metamodel.SingularAttribute;
 import org.joda.time.DateTime;

 /** Implementation of {@link JpaTransactionManager} for JPA compatible database. */
@@ -90,7 +90,6 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  // TODO(b/176108270): Remove this property after database migration.
  private static final ImmutableSet<Class<? extends ImmutableObject>> IGNORED_ENTITY_CLASSES =
      ImmutableSet.of(
-          ClaimsListSingleton.class,
          EppResourceIndex.class,
          ForeignKeyContactIndex.class,
          ForeignKeyDomainIndex.class,
@@ -121,22 +120,23 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {

  @Override
  public EntityManager getEntityManager() {
-    if (transactionInfo.get().entityManager == null) {
+    EntityManager entityManager = transactionInfo.get().entityManager;
+    if (entityManager == null) {
      throw new PersistenceException(
          "No EntityManager has been initialized. getEntityManager() must be invoked in the scope"
              + " of a transaction");
    }
-    return transactionInfo.get().entityManager;
+    return entityManager;
  }

  @Override
  public <T> TypedQuery<T> query(String sqlString, Class<T> resultClass) {
-    return new DetachingTypedQuery(getEntityManager().createQuery(sqlString, resultClass));
+    return new DetachingTypedQuery<>(getEntityManager().createQuery(sqlString, resultClass));
  }

  @Override
  public <T> TypedQuery<T> query(CriteriaQuery<T> criteriaQuery) {
-    return new DetachingTypedQuery(getEntityManager().createQuery(criteriaQuery));
+    return new DetachingTypedQuery<>(getEntityManager().createQuery(criteriaQuery));
  }

  @Override
@@ -173,7 +173,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
            return work.get();
          }
          TransactionInfo txnInfo = transactionInfo.get();
-          txnInfo.entityManager = emf.createEntityManager();
+          txnInfo.entityManager = createReadOnlyCheckingEntityManager();
          EntityTransaction txn = txnInfo.entityManager.getTransaction();
          try {
            txn.begin();
@@ -205,7 +205,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
      return work.get();
    }
    TransactionInfo txnInfo = transactionInfo.get();
-    txnInfo.entityManager = emf.createEntityManager();
+    txnInfo.entityManager = createReadOnlyCheckingEntityManager();
    EntityTransaction txn = txnInfo.entityManager.getTransaction();
    try {
      txn.begin();
@@ -596,7 +596,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {

  @Override
  public <T> QueryComposer<T> createQueryComposer(Class<T> entity) {
-    return new JpaQueryComposerImpl<T>(entity);
+    return new JpaQueryComposerImpl<>(entity);
  }

  @Override
@@ -609,6 +609,38 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    return false;
  }

+  @Override
+  public void putIgnoringReadOnly(Object entity) {
+    checkArgumentNotNull(entity);
+    if (isEntityOfIgnoredClass(entity)) {
+      return;
+    }
+    assertInTransaction();
+    // Necessary due to the changes in HistoryEntry representation during the migration to SQL
+    Object toPersist = toSqlEntity(entity);
+    TransactionInfo txn = transactionInfo.get();
+    Object merged = txn.entityManager.mergeIgnoringReadOnly(toPersist);
+    txn.objectsToSave.add(merged);
+    txn.addUpdate(toPersist);
+  }
+
+  @Override
+  public void deleteIgnoringReadOnly(VKey<?> key) {
+    checkArgumentNotNull(key, "key must be specified");
+    assertInTransaction();
+    if (IGNORED_ENTITY_CLASSES.contains(key.getKind())) {
+      return;
+    }
+    EntityType<?> entityType = getEntityType(key.getKind());
+    ImmutableSet<EntityId> entityIds = getEntityIdsFromSqlKey(entityType, key.getSqlKey());
+    String sql =
+        String.format("DELETE FROM %s WHERE %s", entityType.getName(), getAndClause(entityIds));
+    ReadOnlyCheckingQuery query = transactionInfo.get().entityManager.createQuery(sql);
+    entityIds.forEach(entityId -> query.setParameter(entityId.name, entityId.value));
+    transactionInfo.get().addDelete(key);
+    query.executeUpdateIgnoringReadOnly();
+  }
+
  @Override
  public <T> void assertDelete(VKey<T> key) {
    if (internalDelete(key) != 1) {
@@ -617,6 +649,10 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    }
  }

+  private ReadOnlyCheckingEntityManager createReadOnlyCheckingEntityManager() {
+    return new ReadOnlyCheckingEntityManager(emf.createEntityManager());
+  }
+
  private <T> EntityType<T> getEntityType(Class<T> clazz) {
    return emf.getMetamodel().entity(clazz);
  }
@@ -659,10 +695,22 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  private static ImmutableSet<EntityId> getEntityIdsFromIdContainer(
      EntityType<?> entityType, Object idContainer) {
    return entityType.getIdClassAttributes().stream()
-        .map(SingularAttribute::getName)
        .map(
-            idName -> {
-              Object idValue = getFieldValue(idContainer, idName);
+            attribute -> {
+              String idName = attribute.getName();
+              // The object may use either Java getters or field names to represent the ID object.
+              // Attempt the Java getter, then fall back to the field name if that fails.
+              String methodName = attribute.getJavaMember().getName();
+              Object idValue;
+              try {
+                Method method = idContainer.getClass().getDeclaredMethod(methodName);
+                method.setAccessible(true);
+                idValue = method.invoke(idContainer);
+              } catch (NoSuchMethodException
+                  | IllegalAccessException
+                  | InvocationTargetException e) {
+                idValue = getFieldValue(idContainer, idName);
+              }
              return new EntityId(idName, idValue);
            })
        .collect(toImmutableSet());
@@ -752,7 +800,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  }

  private static class TransactionInfo {
-    EntityManager entityManager;
+    ReadOnlyCheckingEntityManager entityManager;
    boolean inTransaction = false;
    DateTime transactionTime;

@@ -0,0 +1,320 @@
+// Copyright 2021 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.transaction;
+
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertNotReadOnlyMode;
+
+import java.util.List;
+import java.util.Map;
+import javax.persistence.EntityGraph;
+import javax.persistence.EntityManager;
+import javax.persistence.EntityManagerFactory;
+import javax.persistence.EntityTransaction;
+import javax.persistence.FlushModeType;
+import javax.persistence.LockModeType;
+import javax.persistence.Query;
+import javax.persistence.StoredProcedureQuery;
+import javax.persistence.TypedQuery;
+import javax.persistence.criteria.CriteriaBuilder;
+import javax.persistence.criteria.CriteriaDelete;
+import javax.persistence.criteria.CriteriaQuery;
+import javax.persistence.criteria.CriteriaUpdate;
+import javax.persistence.metamodel.Metamodel;
+
+/** An {@link EntityManager} that throws exceptions on write actions if in read-only mode. */
+public class ReadOnlyCheckingEntityManager implements EntityManager {
+
+  private final EntityManager delegate;
+
+  public ReadOnlyCheckingEntityManager(EntityManager delegate) {
+    this.delegate = delegate;
+  }
+
+  @Override
+  public void persist(Object entity) {
+    assertNotReadOnlyMode();
+    delegate.persist(entity);
+  }
+
+  @Override
+  public <T> T merge(T entity) {
+    assertNotReadOnlyMode();
+    return delegate.merge(entity);
+  }
+
+  @Override
+  public void remove(Object entity) {
+    assertNotReadOnlyMode();
+    delegate.remove(entity);
+  }
+
+  @Override
+  public <T> T find(Class<T> entityClass, Object primaryKey) {
+    return delegate.find(entityClass, primaryKey);
+  }
+
+  @Override
+  public <T> T find(Class<T> entityClass, Object primaryKey, Map<String, Object> properties) {
+    return delegate.find(entityClass, primaryKey, properties);
+  }
+
+  @Override
+  public <T> T find(Class<T> entityClass, Object primaryKey, LockModeType lockMode) {
+    return delegate.find(entityClass, primaryKey, lockMode);
+  }
+
+  @Override
+  public <T> T find(
+      Class<T> entityClass,
+      Object primaryKey,
+      LockModeType lockMode,
+      Map<String, Object> properties) {
+    return delegate.find(entityClass, primaryKey, lockMode, properties);
+  }
+
+  @Override
+  public <T> T getReference(Class<T> entityClass, Object primaryKey) {
+    return delegate.getReference(entityClass, primaryKey);
+  }
+
+  @Override
+  public void flush() {
+    delegate.flush();
+  }
+
+  @Override
+  public void setFlushMode(FlushModeType flushMode) {
+    delegate.setFlushMode(flushMode);
+  }
+
+  @Override
+  public FlushModeType getFlushMode() {
+    return delegate.getFlushMode();
+  }
+
+  @Override
+  public void lock(Object entity, LockModeType lockMode) {
+    assertNotReadOnlyMode();
+    delegate.lock(entity, lockMode);
+  }
+
+  @Override
+  public void lock(Object entity, LockModeType lockMode, Map<String, Object> properties) {
+    assertNotReadOnlyMode();
+    delegate.lock(entity, lockMode, properties);
+  }
+
+  @Override
+  public void refresh(Object entity) {
+    delegate.refresh(entity);
+  }
+
+  @Override
+  public void refresh(Object entity, Map<String, Object> properties) {
+    delegate.refresh(entity, properties);
+  }
+
+  @Override
+  public void refresh(Object entity, LockModeType lockMode) {
+    delegate.refresh(entity, lockMode);
+  }
+
+  @Override
+  public void refresh(Object entity, LockModeType lockMode, Map<String, Object> properties) {
+    delegate.refresh(entity, lockMode, properties);
+  }
+
+  @Override
+  public void clear() {
+    delegate.clear();
+  }
+
+  @Override
+  public void detach(Object entity) {
+    delegate.detach(entity);
+  }
+
+  @Override
+  public boolean contains(Object entity) {
+    return delegate.contains(entity);
+  }
+
+  @Override
+  public LockModeType getLockMode(Object entity) {
+    return delegate.getLockMode(entity);
+  }
+
+  @Override
+  public void setProperty(String propertyName, Object value) {
+    delegate.setProperty(propertyName, value);
+  }
+
+  @Override
+  public Map<String, Object> getProperties() {
+    return delegate.getProperties();
+  }
+
+  @Override
+  public ReadOnlyCheckingQuery createQuery(String qlString) {
+    return new ReadOnlyCheckingQuery(delegate.createQuery(qlString));
+  }
+
+  @Override
+  public <T> TypedQuery<T> createQuery(CriteriaQuery<T> criteriaQuery) {
+    return new ReadOnlyCheckingTypedQuery<>(delegate.createQuery(criteriaQuery));
+  }
+
+  @Override
+  public Query createQuery(CriteriaUpdate updateQuery) {
+    assertNotReadOnlyMode();
+    return delegate.createQuery(updateQuery);
+  }
+
+  @Override
+  public Query createQuery(CriteriaDelete deleteQuery) {
+    assertNotReadOnlyMode();
+    return delegate.createQuery(deleteQuery);
+  }
+
+  @Override
+  public <T> TypedQuery<T> createQuery(String qlString, Class<T> resultClass) {
+    return new ReadOnlyCheckingTypedQuery<>(delegate.createQuery(qlString, resultClass));
+  }
+
+  @Override
+  public Query createNamedQuery(String name) {
+    return new ReadOnlyCheckingQuery(delegate.createNamedQuery(name));
+  }
+
+  @Override
+  public <T> TypedQuery<T> createNamedQuery(String name, Class<T> resultClass) {
+    return new ReadOnlyCheckingTypedQuery<>(delegate.createNamedQuery(name, resultClass));
+  }
+
+  @Override
+  public Query createNativeQuery(String sqlString) {
+    return new ReadOnlyCheckingQuery(delegate.createNativeQuery(sqlString));
+  }
+
+  @Override
+  public Query createNativeQuery(String sqlString, Class resultClass) {
+    return new ReadOnlyCheckingQuery(delegate.createNativeQuery(sqlString, resultClass));
+  }
+
+  @Override
+  public Query createNativeQuery(String sqlString, String resultSetMapping) {
+    return new ReadOnlyCheckingQuery(delegate.createNativeQuery(sqlString, resultSetMapping));
+  }
+
+  @Override
+  public StoredProcedureQuery createNamedStoredProcedureQuery(String name) {
+    assertNotReadOnlyMode();
+    return delegate.createNamedStoredProcedureQuery(name);
+  }
+
+  @Override
+  public StoredProcedureQuery createStoredProcedureQuery(String procedureName) {
+    assertNotReadOnlyMode();
+    return delegate.createStoredProcedureQuery(procedureName);
+  }
+
+  @Override
+  public StoredProcedureQuery createStoredProcedureQuery(
+      String procedureName, Class... resultClasses) {
+    assertNotReadOnlyMode();
+    return delegate.createStoredProcedureQuery(procedureName, resultClasses);
+  }
+
+  @Override
+  public StoredProcedureQuery createStoredProcedureQuery(
+      String procedureName, String... resultSetMappings) {
+    assertNotReadOnlyMode();
+    return delegate.createStoredProcedureQuery(procedureName, resultSetMappings);
+  }
+
+  @Override
+  public void joinTransaction() {
+    delegate.joinTransaction();
+  }
+
+  @Override
+  public boolean isJoinedToTransaction() {
+    return delegate.isJoinedToTransaction();
+  }
+
+  @Override
+  public <T> T unwrap(Class<T> cls) {
+    return delegate.unwrap(cls);
+  }
+
+  @Override
+  public Object getDelegate() {
+    return delegate.getDelegate();
+  }
+
+  @Override
+  public void close() {
+    delegate.close();
+  }
+
+  @Override
+  public boolean isOpen() {
+    return delegate.isOpen();
+  }
+
+  @Override
+  public EntityTransaction getTransaction() {
+    return delegate.getTransaction();
+  }
+
+  @Override
+  public EntityManagerFactory getEntityManagerFactory() {
+    return delegate.getEntityManagerFactory();
+  }
+
+  @Override
+  public CriteriaBuilder getCriteriaBuilder() {
+    return delegate.getCriteriaBuilder();
+  }
+
+  @Override
+  public Metamodel getMetamodel() {
+    return delegate.getMetamodel();
+  }
+
+  @Override
+  public <T> EntityGraph<T> createEntityGraph(Class<T> rootType) {
+    return delegate.createEntityGraph(rootType);
+  }
+
+  @Override
+  public EntityGraph<?> createEntityGraph(String graphName) {
+    return delegate.createEntityGraph(graphName);
+  }
+
+  @Override
+  public EntityGraph<?> getEntityGraph(String graphName) {
+    return delegate.getEntityGraph(graphName);
+  }
+
+  @Override
+  public <T> List<EntityGraph<? super T>> getEntityGraphs(Class<T> entityClass) {
+    return delegate.getEntityGraphs(entityClass);
+  }
+
+  public <T> T mergeIgnoringReadOnly(T entity) {
+    return delegate.merge(entity);
+  }
+}
@@ -0,0 +1,203 @@
+// Copyright 2021 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.transaction;
+
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertNotReadOnlyMode;
+
+import java.util.Calendar;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.persistence.FlushModeType;
+import javax.persistence.LockModeType;
+import javax.persistence.Parameter;
+import javax.persistence.Query;
+import javax.persistence.TemporalType;
+
+/** A {@link Query} that throws exceptions on write actions if in read-only mode. */
+class ReadOnlyCheckingQuery implements Query {
+
+  private final Query delegate;
+
+  ReadOnlyCheckingQuery(Query delegate) {
+    this.delegate = delegate;
+  }
+
+  @Override
+  public List getResultList() {
+    return delegate.getResultList();
+  }
+
+  @Override
+  public Object getSingleResult() {
+    return delegate.getSingleResult();
+  }
+
+  @Override
+  public int executeUpdate() {
+    assertNotReadOnlyMode();
+    return delegate.executeUpdate();
+  }
+
+  @Override
+  public Query setMaxResults(int maxResult) {
+    return delegate.setMaxResults(maxResult);
+  }
+
+  @Override
+  public int getMaxResults() {
+    return delegate.getMaxResults();
+  }
+
+  @Override
+  public Query setFirstResult(int startPosition) {
+    return delegate.setFirstResult(startPosition);
+  }
+
+  @Override
+  public int getFirstResult() {
+    return delegate.getFirstResult();
+  }
+
+  @Override
+  public Query setHint(String hintName, Object value) {
+    return delegate.setHint(hintName, value);
+  }
+
+  @Override
+  public Map<String, Object> getHints() {
+    return delegate.getHints();
+  }
+
+  @Override
+  public <T> Query setParameter(Parameter<T> param, T value) {
+    return delegate.setParameter(param, value);
+  }
+
+  @Override
+  public Query setParameter(Parameter<Calendar> param, Calendar value, TemporalType temporalType) {
+    return delegate.setParameter(param, value, temporalType);
+  }
+
+  @Override
+  public Query setParameter(Parameter<Date> param, Date value, TemporalType temporalType) {
+    return delegate.setParameter(param, value, temporalType);
+  }
+
+  @Override
+  public Query setParameter(String name, Object value) {
+    return delegate.setParameter(name, value);
+  }
+
+  @Override
+  public Query setParameter(String name, Calendar value, TemporalType temporalType) {
+    return delegate.setParameter(name, value, temporalType);
+  }
+
+  @Override
+  public Query setParameter(String name, Date value, TemporalType temporalType) {
+    return delegate.setParameter(name, value, temporalType);
+  }
+
+  @Override
+  public Query setParameter(int position, Object value) {
+    return delegate.setParameter(position, value);
+  }
+
+  @Override
+  public Query setParameter(int position, Calendar value, TemporalType temporalType) {
+    return delegate.setParameter(position, value, temporalType);
+  }
+
+  @Override
+  public Query setParameter(int position, Date value, TemporalType temporalType) {
+    return delegate.setParameter(position, value, temporalType);
+  }
+
+  @Override
+  public Set<Parameter<?>> getParameters() {
+    return delegate.getParameters();
+  }
+
+  @Override
+  public Parameter<?> getParameter(String name) {
+    return delegate.getParameter(name);
+  }
+
+  @Override
+  public <T> Parameter<T> getParameter(String name, Class<T> type) {
+    return delegate.getParameter(name, type);
+  }
+
+  @Override
+  public Parameter<?> getParameter(int position) {
+    return delegate.getParameter(position);
+  }
+
+  @Override
+  public <T> Parameter<T> getParameter(int position, Class<T> type) {
+    return delegate.getParameter(position, type);
+  }
+
+  @Override
+  public boolean isBound(Parameter<?> param) {
+    return delegate.isBound(param);
+  }
+
+  @Override
+  public <T> T getParameterValue(Parameter<T> param) {
+    return delegate.getParameterValue(param);
+  }
+
+  @Override
+  public Object getParameterValue(String name) {
+    return delegate.getParameterValue(name);
+  }
+
+  @Override
+  public Object getParameterValue(int position) {
+    return delegate.getParameterValue(position);
+  }
+
+  @Override
+  public Query setFlushMode(FlushModeType flushMode) {
+    return delegate.setFlushMode(flushMode);
+  }
+
+  @Override
+  public FlushModeType getFlushMode() {
+    return delegate.getFlushMode();
+  }
+
+  @Override
+  public Query setLockMode(LockModeType lockMode) {
+    return delegate.setLockMode(lockMode);
+  }
+
+  @Override
+  public LockModeType getLockMode() {
+    return delegate.getLockMode();
+  }
+
+  @Override
+  public <T> T unwrap(Class<T> cls) {
+    return delegate.unwrap(cls);
+  }
+
+  public int executeUpdateIgnoringReadOnly() {
+    return delegate.executeUpdate();
+  }
+}
@@ -0,0 +1,200 @@
+// Copyright 2021 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.transaction;
+
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertNotReadOnlyMode;
+
+import java.util.Calendar;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.persistence.FlushModeType;
+import javax.persistence.LockModeType;
+import javax.persistence.Parameter;
+import javax.persistence.TemporalType;
+import javax.persistence.TypedQuery;
+
+/** A {@link TypedQuery <T>} that throws exceptions on write actions if in read-only mode. */
+class ReadOnlyCheckingTypedQuery<T> implements TypedQuery<T> {
+
+  private final TypedQuery<T> delegate;
+
+  ReadOnlyCheckingTypedQuery(TypedQuery<T> delegate) {
+    this.delegate = delegate;
+  }
+
+  @Override
+  public List<T> getResultList() {
+    return delegate.getResultList();
+  }
+
+  @Override
+  public T getSingleResult() {
+    return delegate.getSingleResult();
+  }
+
+  @Override
+  public int executeUpdate() {
+    assertNotReadOnlyMode();
+    return delegate.executeUpdate();
+  }
+
+  @Override
+  public TypedQuery<T> setMaxResults(int maxResult) {
+    return delegate.setMaxResults(maxResult);
+  }
+
+  @Override
+  public int getMaxResults() {
+    return delegate.getMaxResults();
+  }
+
+  @Override
+  public TypedQuery<T> setFirstResult(int startPosition) {
+    return delegate.setFirstResult(startPosition);
+  }
+
+  @Override
+  public int getFirstResult() {
+    return delegate.getFirstResult();
+  }
+
+  @Override
+  public TypedQuery<T> setHint(String hintName, Object value) {
+    return delegate.setHint(hintName, value);
+  }
+
+  @Override
+  public Map<String, Object> getHints() {
+    return delegate.getHints();
+  }
+
+  @Override
+  public <T1> TypedQuery<T> setParameter(Parameter<T1> param, T1 value) {
+    return delegate.setParameter(param, value);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(
+      Parameter<Calendar> param, Calendar value, TemporalType temporalType) {
+    return delegate.setParameter(param, value, temporalType);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(Parameter<Date> param, Date value, TemporalType temporalType) {
+    return delegate.setParameter(param, value, temporalType);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(String name, Object value) {
+    return delegate.setParameter(name, value);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(String name, Calendar value, TemporalType temporalType) {
+    return delegate.setParameter(name, value, temporalType);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(String name, Date value, TemporalType temporalType) {
+    return delegate.setParameter(name, value, temporalType);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(int position, Object value) {
+    return delegate.setParameter(position, value);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(int position, Calendar value, TemporalType temporalType) {
+    return delegate.setParameter(position, value, temporalType);
+  }
+
+  @Override
+  public TypedQuery<T> setParameter(int position, Date value, TemporalType temporalType) {
+    return delegate.setParameter(position, value, temporalType);
+  }
+
+  @Override
+  public Set<Parameter<?>> getParameters() {
+    return delegate.getParameters();
+  }
+
+  @Override
+  public Parameter<?> getParameter(String name) {
+    return delegate.getParameter(name);
+  }
+
+  @Override
+  public <X> Parameter<X> getParameter(String name, Class<X> type) {
+    return delegate.getParameter(name, type);
+  }
+
+  @Override
+  public Parameter<?> getParameter(int position) {
+    return delegate.getParameter(position);
+  }
+
+  @Override
+  public <X> Parameter<X> getParameter(int position, Class<X> type) {
+    return delegate.getParameter(position, type);
+  }
+
+  @Override
+  public boolean isBound(Parameter<?> param) {
+    return delegate.isBound(param);
+  }
+
+  @Override
+  public <X> X getParameterValue(Parameter<X> param) {
+    return delegate.getParameterValue(param);
+  }
+
+  @Override
+  public Object getParameterValue(String name) {
+    return delegate.getParameterValue(name);
+  }
+
+  @Override
+  public Object getParameterValue(int position) {
+    return delegate.getParameterValue(position);
+  }
+
+  @Override
+  public TypedQuery<T> setFlushMode(FlushModeType flushMode) {
+    return delegate.setFlushMode(flushMode);
+  }
+
+  @Override
+  public FlushModeType getFlushMode() {
+    return delegate.getFlushMode();
+  }
+
+  @Override
+  public TypedQuery<T> setLockMode(LockModeType lockMode) {
+    return delegate.setLockMode(lockMode);
+  }
+
+  @Override
+  public LockModeType getLockMode() {
+    return delegate.getLockMode();
+  }
+
+  @Override
+  public <X> X unwrap(Class<X> cls) {
+    return delegate.unwrap(cls);
+  }
+}
@@ -21,10 +21,13 @@ import static google.registry.persistence.transaction.TransactionManagerFactory.

 import com.google.appengine.api.datastore.Entity;
 import com.google.appengine.api.datastore.EntityTranslator;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableList;
 import com.google.storage.onestore.v3.OnestoreEntity.EntityProto;
 import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
+import google.registry.model.replay.DatastoreEntity;
+import google.registry.model.replay.SqlEntity;
 import google.registry.persistence.VKey;
 import java.io.ByteArrayInputStream;
 import java.io.ByteArrayOutputStream;
@@ -57,6 +60,11 @@ public class Transaction extends ImmutableObject implements Buildable {

  private transient ImmutableList<Mutation> mutations;

+  @VisibleForTesting
+  public ImmutableList<Mutation> getMutations() {
+    return mutations;
+  }
+
  /** Write the entire transaction to the datastore in a datastore transaction. */
  public void writeToDatastore() {
    ofyTm()
@@ -128,7 +136,7 @@ public class Transaction extends ImmutableObject implements Buildable {
   * Returns true if we are serializing a transaction in the current thread.
   *
   * <p>This should be checked by any Ofy translators prior to making any changes to an entity's
-   * state representation based on the assumption that we are currently pseristing the entity to
+   * state representation based on the assumption that we are currently persisting the entity to
   * datastore.
   */
  public static boolean inSerializationMode() {
@@ -146,7 +154,7 @@ public class Transaction extends ImmutableObject implements Buildable {

  public static class Builder extends GenericBuilder<Transaction, Builder> {

-    ImmutableList.Builder listBuilder = new ImmutableList.Builder();
+    ImmutableList.Builder<Mutation> listBuilder = new ImmutableList.Builder<>();

    Builder() {}

@@ -224,11 +232,16 @@ public class Transaction extends ImmutableObject implements Buildable {
    private Object entity;

    Update(Object entity) {
-      this.entity = entity;
+      this.entity =
+          (entity instanceof SqlEntity) ? ((SqlEntity) entity).toDatastoreEntity().get() : entity;
    }

    @Override
    public void writeToDatastore() {
+      // this should always be the case, but check just in case
+      if (entity instanceof DatastoreEntity) {
+        ((DatastoreEntity) entity).beforeDatastoreSaveOnReplay();
+      }
      ofyTm().put(entity);
    }

@@ -241,6 +254,11 @@ public class Transaction extends ImmutableObject implements Buildable {
      proto.writeDelimitedTo(out);
    }

+    @VisibleForTesting
+    public Object getEntity() {
+      return entity;
+    }
+
    public static Update deserializeFrom(ObjectInputStream in) throws IOException {
      EntityProto proto = new EntityProto();
      proto.parseDelimitedFrom(in);
@@ -273,9 +291,14 @@ public class Transaction extends ImmutableObject implements Buildable {
      out.writeObject(key);
    }

+    @VisibleForTesting
+    public VKey<?> getKey() {
+      return key;
+    }
+
    public static Delete deserializeFrom(ObjectInputStream in) throws IOException {
      try {
-        return new Delete((VKey) in.readObject());
+        return new Delete((VKey<?>) in.readObject());
      } catch (ClassNotFoundException e) {
        throw new IllegalArgumentException(e);
      }
@@ -14,6 +14,7 @@

 package google.registry.persistence.transaction;

+import google.registry.model.ImmutableObject;
 import google.registry.model.replay.SqlOnlyEntity;
 import javax.persistence.Entity;
 import javax.persistence.GeneratedValue;
@@ -28,7 +29,7 @@ import javax.persistence.Table;
 */
@Entity
@Table(name = "Transaction")
-public class TransactionEntity implements SqlOnlyEntity {
+public class TransactionEntity extends ImmutableObject implements SqlOnlyEntity {

  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
@@ -307,4 +307,10 @@ public interface TransactionManager {

  /** Returns true if the transaction manager is DatastoreTransactionManager, false otherwise. */
  boolean isOfy();
+
+  /** Performs the given write ignoring any read-only restrictions, for use only in replay. */
+  void putIgnoringReadOnly(Object entity);
+
+  /** Performs the given delete ignoring any read-only restrictions, for use only in replay. */
+  void deleteIgnoringReadOnly(VKey<?> key);
 }
@@ -86,9 +86,11 @@ public class TransactionManagerFactory {
    if (tmForTest.isPresent()) {
      return tmForTest.get();
    }
-    PrimaryDatabase primaryDatabase =
-        DatabaseMigrationStateSchedule.getValueAtTime(DateTime.now(UTC)).getPrimaryDatabase();
-    return primaryDatabase.equals(PrimaryDatabase.DATASTORE) ? ofyTm() : jpaTm();
+    return DatabaseMigrationStateSchedule.getValueAtTime(DateTime.now(UTC))
+            .getPrimaryDatabase()
+            .equals(PrimaryDatabase.DATASTORE)
+        ? ofyTm()
+        : jpaTm();
  }

  /**
@@ -141,4 +143,17 @@ public class TransactionManagerFactory {
  public static void removeTmOverrideForTest() {
    tmForTest = Optional.empty();
  }
+
+  public static void assertNotReadOnlyMode() {
+    if (DatabaseMigrationStateSchedule.getValueAtTime(DateTime.now(UTC)).isReadOnly()) {
+      throw new ReadOnlyModeException();
+    }
+  }
+
+  /** Thrown when a write is attempted when the DB is in read-only mode. */
+  public static class ReadOnlyModeException extends IllegalStateException {
+    public ReadOnlyModeException() {
+      super("Registry is currently in read-only mode");
+    }
+  }
 }
@@ -46,7 +46,7 @@ public final class TmchDnlAction implements Runnable {
  @Inject @Key("marksdbDnlLoginAndPassword") Optional<String> marksdbDnlLoginAndPassword;
  @Inject TmchDnlAction() {}

-  /** Synchronously fetches latest domain name list and saves it to Datastore. */
+  /** Synchronously fetches latest domain name list and saves it to Cloud SQL. */
  @Override
  public void run() {
    List<String> lines;
@@ -26,7 +26,8 @@ import javax.annotation.Nullable;
 * Base class for specification of command line parameters common to creating and updating reserved
 * lists.
 */
-public abstract class CreateOrUpdateReservedListCommand extends MutatingCommand {
+public abstract class CreateOrUpdateReservedListCommand extends ConfirmingCommand
+    implements CommandWithRemoteApi {

  static final FluentLogger logger = FluentLogger.forEnclosingClass();

@@ -43,7 +43,6 @@ public class CreatePremiumListCommand extends CreateOrUpdatePremiumListCommand {
  String currencyUnit;

  @Override
-  // Using CreatePremiumListAction.java as reference;
  protected String prompt() throws Exception {
    currency = CurrencyUnit.of(currencyUnit);
    name = Strings.isNullOrEmpty(name) ? convertFilePathToName(inputFile) : name;
@@ -51,7 +50,6 @@ public class CreatePremiumListCommand extends CreateOrUpdatePremiumListCommand {
        !PremiumListDao.getLatestRevision(name).isPresent(),
        "A premium list already exists by this name");
    if (!override) {
-      // refer to CreatePremiumListAction.java
      assertTldExists(
          name,
          "Premium names must match the name of the TLD they are intended to be used on"
@@ -25,9 +25,7 @@ import com.beust.jcommander.Parameters;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Splitter;
 import com.google.common.base.Strings;
-import com.googlecode.objectify.Key;
 import google.registry.model.tld.label.ReservedList;
-import google.registry.persistence.VKey;
 import java.nio.file.Files;
 import java.util.List;
 import java.util.stream.Collectors;
@@ -48,7 +46,7 @@ final class CreateReservedListCommand extends CreateOrUpdateReservedListCommand
  boolean override;

  @Override
-  protected void init() throws Exception {
+  protected String prompt() throws Exception {
    name = Strings.isNullOrEmpty(name) ? convertFilePathToName(input) : name;
    checkArgument(
        !ReservedList.get(name).isPresent(), "A reserved list already exists by this name");
@@ -66,35 +64,11 @@ final class CreateReservedListCommand extends CreateOrUpdateReservedListCommand
            .setCreationTimestamp(now)
            .build();

-    // calls the stageEntityChange method that takes old entity, new entity and a new vkey;
-    // Because ReservedList is a sqlEntity and its primary key field (revisionId) is only set when
-    // it's being persisted; a vkey has to be created here explicitly for ReservedList instances.
-    stageEntityChange(
-        null, reservedList, VKey.createOfy(ReservedList.class, Key.create(reservedList)));
-  }
-
-  @Override
-  protected String prompt() {
-    return getChangedEntities().isEmpty()
-        ? "No entity changes to apply."
-        : getChangedEntities().stream()
-            .map(
-                entity -> {
-                  if (entity instanceof ReservedList) {
-                    // Format the entries of the reserved list as well.
-                    String entries =
-                        ((ReservedList) entity)
-                            .getReservedListEntries().entrySet().stream()
-                                .map(
-                                    entry ->
-                                        String.format("%s=%s", entry.getKey(), entry.getValue()))
-                                .collect(Collectors.joining(", "));
-                    return String.format("%s\nreservedListMap={%s}\n", entity, entries);
-                  } else {
-                    return entity.toString();
-                  }
-                })
-            .collect(Collectors.joining("\n"));
+    String entries =
+        reservedList.getReservedListEntries().entrySet().stream()
+            .map(entry -> String.format("%s=%s", entry.getKey(), entry.getValue()))
+            .collect(Collectors.joining(", "));
+    return String.format("%s\nreservedListMap={%s}\n", reservedList, entries);
  }

  private static void validateListName(String name) {
@@ -37,7 +37,6 @@ import java.util.Optional;
 class UpdatePremiumListCommand extends CreateOrUpdatePremiumListCommand {

  @Override
-  // Using UpdatePremiumListAction.java as reference;
  protected String prompt() throws Exception {
    name = Strings.isNullOrEmpty(name) ? convertFilePathToName(inputFile) : name;
    Optional<PremiumList> list = PremiumListDao.getLatestRevision(name);
@@ -19,18 +19,17 @@ import static java.nio.charset.StandardCharsets.UTF_8;

 import com.beust.jcommander.Parameters;
 import com.google.common.base.Strings;
-import com.googlecode.objectify.Key;
 import google.registry.model.tld.label.ReservedList;
-import google.registry.persistence.VKey;
 import java.nio.file.Files;
 import java.util.List;
+import java.util.stream.Collectors;

 /** Command to safely update {@link ReservedList}. */
@Parameters(separators = " =", commandDescription = "Update a ReservedList.")
 final class UpdateReservedListCommand extends CreateOrUpdateReservedListCommand {

  @Override
-  protected void init() throws Exception {
+  protected String prompt() throws Exception {
    name = Strings.isNullOrEmpty(name) ? convertFilePathToName(input) : name;
    ReservedList existingReservedList =
        ReservedList.get(name)
@@ -53,15 +52,18 @@ final class UpdateReservedListCommand extends CreateOrUpdateReservedListCommand
    if (!existingReservedList
        .getReservedListEntries()
        .equals(reservedList.getReservedListEntries())) {
-      // calls the stageEntityChange method that takes old entity, new entity and a new vkey;
-      // a vkey has to be created here explicitly for ReservedList instances.
-      // ReservedList is a sqlEntity; it triggers the static method Vkey.create(Key<?> ofyCall),
-      // which invokes a static ReservedList.createVkey(Key ofyKey) method that does not exist.
-      // the sql primary key field (revisionId) is only set when it's being persisted;
-      stageEntityChange(
-          existingReservedList,
-          reservedList,
-          VKey.createOfy(ReservedList.class, Key.create(existingReservedList)));
+      String oldEntries =
+          existingReservedList.getReservedListEntries().entrySet().stream()
+              .map(entry -> String.format("%s=%s", entry.getKey(), entry.getValue()))
+              .collect(Collectors.joining(", "));
+      String newEntries =
+          reservedList.getReservedListEntries().entrySet().stream()
+              .map(entry -> String.format("%s=%s", entry.getKey(), entry.getValue()))
+              .collect(Collectors.joining(", "));
+      return String.format(
+          "Update reserved list for %s?\nOld List: %s\n New List: %s",
+          name, oldEntries, newEntries);
    }
+    return "No entity changes to apply.";
  }
 }
@@ -1,76 +0,0 @@
-// Copyright 2017 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.tools.server;
-
-import static com.google.common.flogger.LazyArgs.lazy;
-import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
-
-import com.google.common.collect.ImmutableMap;
-import com.google.common.flogger.FluentLogger;
-import google.registry.request.JsonResponse;
-import google.registry.request.Parameter;
-import javax.inject.Inject;
-
-/** Abstract base class for actions that update premium lists. */
-public abstract class CreateOrUpdatePremiumListAction implements Runnable {
-
-  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
-
-  private static final int MAX_LOGGING_PREMIUM_LIST_LENGTH = 1000;
-
-  public static final String NAME_PARAM = "name";
-  public static final String INPUT_PARAM = "inputData";
-
-  @Inject JsonResponse response;
-
-  @Inject
-  @Parameter("premiumListName")
-  String name;
-
-  @Inject
-  @Parameter(INPUT_PARAM)
-  String inputData;
-
-  @Override
-  public void run() {
-    try {
-      checkArgumentNotNull(inputData, "Input data must not be null");
-      save();
-    } catch (IllegalArgumentException e) {
-      logger.atInfo().withCause(e).log(
-          "Usage error in attempting to save premium list from nomulus tool command");
-      response.setPayload(ImmutableMap.of("error", e.getMessage(), "status", "error"));
-    } catch (Exception e) {
-      logger.atSevere().withCause(e).log(
-          "Unexpected error saving premium list to Datastore from nomulus tool command");
-      response.setPayload(ImmutableMap.of("error", e.getMessage(), "status", "error"));
-    }
-  }
-
-  /** Logs the premium list data at INFO, truncated if too long. */
-  void logInputData() {
-    String logData = (inputData == null) ? "(null)" : inputData;
-    logger.atInfo().log(
-        "Received the following input data: %s",
-        lazy(
-            () ->
-                (logData.length() < MAX_LOGGING_PREMIUM_LIST_LENGTH)
-                    ? logData
-                    : (logData.substring(0, MAX_LOGGING_PREMIUM_LIST_LENGTH) + "<truncated>")));
-  }
-
-  /** Saves the premium list to both Datastore and Cloud SQL. */
-  protected abstract void save();
-}
@@ -1,79 +0,0 @@
-// Copyright 2017 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.tools.server;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static google.registry.model.tld.Registries.assertTldExists;
-import static google.registry.request.Action.Method.POST;
-
-import com.google.common.base.Splitter;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.flogger.FluentLogger;
-import google.registry.model.tld.label.PremiumListDao;
-import google.registry.request.Action;
-import google.registry.request.Parameter;
-import google.registry.request.auth.Auth;
-import java.util.List;
-import javax.inject.Inject;
-import org.joda.money.CurrencyUnit;
-
-/**
- * An action that creates a premium list, for use by the {@code nomulus create_premium_list}
- * command.
- */
-@Action(
-    service = Action.Service.TOOLS,
-    path = CreatePremiumListAction.PATH,
-    method = POST,
-    auth = Auth.AUTH_INTERNAL_OR_ADMIN)
-public class CreatePremiumListAction extends CreateOrUpdatePremiumListAction {
-
-  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
-
-  public static final String OVERRIDE_PARAM = "override";
-  public static final String PATH = "/_dr/admin/createPremiumList";
-  public static final String CURRENCY = "currency";
-
-  @Inject @Parameter(OVERRIDE_PARAM) boolean override;
-
-  @Inject
-  @Parameter("currency")
-  CurrencyUnit currency;
-
-  @Inject CreatePremiumListAction() {}
-
-  @Override
-  protected void save() {
-    checkArgument(
-        !PremiumListDao.getLatestRevision(name).isPresent(),
-        "A premium list of this name already exists: %s",
-        name);
-    if (!override) {
-      assertTldExists(
-          name,
-          "Premium names must match the name of the TLD they are intended to be used on"
-              + " (unless --override is specified), yet TLD %s does not exist");
-    }
-    logger.atInfo().log("Saving premium list for TLD %s", name);
-    logInputData();
-    List<String> inputDataPreProcessed =
-        Splitter.on('\n').omitEmptyStrings().splitToList(inputData);
-    PremiumListDao.save(name, currency, inputDataPreProcessed);
-    String message =
-        String.format("Saved premium list %s with %d entries", name, inputDataPreProcessed.size());
-    logger.atInfo().log(message);
-    response.setPayload(ImmutableMap.of("status", "success", "message", message));
-  }
-}
@@ -15,7 +15,6 @@
 package google.registry.tools.server;

 import static com.google.common.base.Strings.emptyToNull;
-import static google.registry.request.RequestParameters.extractBooleanParameter;
 import static google.registry.request.RequestParameters.extractIntParameter;
 import static google.registry.request.RequestParameters.extractOptionalParameter;
 import static google.registry.request.RequestParameters.extractRequiredParameter;
@@ -28,7 +27,6 @@ import google.registry.request.Parameter;
 import google.registry.request.RequestParameters;
 import java.util.Optional;
 import javax.servlet.http.HttpServletRequest;
-import org.joda.money.CurrencyUnit;

 /**
 * Dagger module for the tools package.
@@ -55,30 +53,6 @@ public class ToolsServerModule {
    return (s == null) ? Optional.empty() : Optional.of(Boolean.parseBoolean(s));
  }

-  @Provides
-  @Parameter("inputData")
-  static String provideInput(HttpServletRequest req) {
-    return extractRequiredParameter(req, CreatePremiumListAction.INPUT_PARAM);
-  }
-
-  @Provides
-  @Parameter("premiumListName")
-  static String provideName(HttpServletRequest req) {
-    return extractRequiredParameter(req, CreatePremiumListAction.NAME_PARAM);
-  }
-
-  @Provides
-  @Parameter("currency")
-  static CurrencyUnit provideCurrency(HttpServletRequest req) {
-    return CurrencyUnit.of(extractRequiredParameter(req, CreatePremiumListAction.CURRENCY));
-  }
-
-  @Provides
-  @Parameter("override")
-  static boolean provideOverride(HttpServletRequest req) {
-   return extractBooleanParameter(req, CreatePremiumListAction.OVERRIDE_PARAM);
-  }
-
  @Provides
  @Parameter("printHeaderRow")
  static Optional<Boolean> providePrintHeaderRow(HttpServletRequest req) {
@@ -1,70 +0,0 @@
-// Copyright 2017 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.tools.server;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static google.registry.request.Action.Method.POST;
-
-import com.google.common.base.Splitter;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.flogger.FluentLogger;
-import google.registry.model.tld.label.PremiumList;
-import google.registry.model.tld.label.PremiumListDao;
-import google.registry.request.Action;
-import google.registry.request.auth.Auth;
-import java.util.List;
-import java.util.Optional;
-import javax.inject.Inject;
-
-/**
- * An action that creates a premium list, for use by the {@code nomulus create_premium_list}
- * command.
- */
-@Action(
-    service = Action.Service.TOOLS,
-    path = UpdatePremiumListAction.PATH,
-    method = POST,
-    auth = Auth.AUTH_INTERNAL_OR_ADMIN)
-public class UpdatePremiumListAction extends CreateOrUpdatePremiumListAction {
-
-  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
-
-  public static final String PATH = "/_dr/admin/updatePremiumList";
-
-  @Inject UpdatePremiumListAction() {}
-
-  @Override
-  protected void save() {
-    Optional<PremiumList> existingList = PremiumListDao.getLatestRevision(name);
-    checkArgument(
-        existingList.isPresent(),
-        "Could not update premium list %s because it doesn't exist.",
-        name);
-
-    logger.atInfo().log("Updating premium list for TLD %s", name);
-    logInputData();
-    List<String> inputDataPreProcessed =
-        Splitter.on('\n').omitEmptyStrings().splitToList(inputData);
-    PremiumList newPremiumList =
-        PremiumListDao.save(name, existingList.get().getCurrency(), inputDataPreProcessed);
-
-    String message =
-        String.format(
-            "Updated premium list %s with %d entries.",
-            newPremiumList.getName(), inputDataPreProcessed.size());
-    logger.atInfo().log(message);
-    response.setPayload(ImmutableMap.of("status", "success", "message", message));
-  }
-}
@@ -93,7 +93,6 @@
    <class>google.registry.persistence.converter.LocalDateConverter</class>
    <class>google.registry.persistence.converter.PostalInfoChoiceListConverter</class>
    <class>google.registry.persistence.converter.RegistrarPocSetConverter</class>
-    <class>google.registry.persistence.converter.ReservedListKeySetConverter</class>
    <class>google.registry.persistence.converter.Spec11ThreatMatchThreatTypeSetConverter</class>
    <class>google.registry.persistence.converter.StatusValueSetConverter</class>
    <class>google.registry.persistence.converter.StringListConverter</class>
@@ -38,9 +38,9 @@ import static org.mockito.Mockito.verify;

 import com.google.cloud.storage.contrib.nio.testing.LocalStorageHelper;
 import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedMap;
+import com.google.common.collect.Iterables;
 import com.google.common.truth.Truth8;
 import com.google.common.util.concurrent.MoreExecutors;
 import com.googlecode.objectify.Key;
@@ -55,13 +55,15 @@ import google.registry.model.index.ForeignKeyIndex;
 import google.registry.model.ofy.CommitLogBucket;
 import google.registry.model.ofy.CommitLogManifest;
 import google.registry.model.ofy.CommitLogMutation;
+import google.registry.model.rde.RdeMode;
+import google.registry.model.rde.RdeNamingUtils;
+import google.registry.model.rde.RdeRevision;
 import google.registry.model.registrar.RegistrarContact;
 import google.registry.model.replay.SqlReplayCheckpoint;
 import google.registry.model.server.Lock;
 import google.registry.model.tld.label.PremiumList;
 import google.registry.model.tld.label.PremiumList.PremiumEntry;
-import google.registry.model.tld.label.ReservedList;
-import google.registry.model.tmch.ClaimsList;
+import google.registry.model.tmch.TmchCrl;
 import google.registry.model.translators.VKeyTranslatorFactory;
 import google.registry.persistence.VKey;
 import google.registry.persistence.transaction.JpaTransactionManager;
@@ -356,10 +358,10 @@ public class ReplayCommitLogsToSqlActionTest {
    //    even though the domain came first in the file
    // 2. that the allocation token delete occurred after the insertions
    InOrder inOrder = Mockito.inOrder(spy);
-    inOrder.verify(spy).put(any(ContactResource.class));
-    inOrder.verify(spy).put(any(DomainBase.class));
-    inOrder.verify(spy).delete(toDelete.createVKey());
-    inOrder.verify(spy).put(any(SqlReplayCheckpoint.class));
+    inOrder.verify(spy).putIgnoringReadOnly(any(ContactResource.class));
+    inOrder.verify(spy).putIgnoringReadOnly(any(DomainBase.class));
+    inOrder.verify(spy).deleteIgnoringReadOnly(toDelete.createVKey());
+    inOrder.verify(spy).putIgnoringReadOnly(any(SqlReplayCheckpoint.class));
  }

  @Test
@@ -398,8 +400,8 @@ public class ReplayCommitLogsToSqlActionTest {
    // deletes have higher weight
    ArgumentCaptor<Object> putCaptor = ArgumentCaptor.forClass(Object.class);
    InOrder inOrder = Mockito.inOrder(spy);
-    inOrder.verify(spy).delete(contact.createVKey());
-    inOrder.verify(spy).put(putCaptor.capture());
+    inOrder.verify(spy).deleteIgnoringReadOnly(contact.createVKey());
+    inOrder.verify(spy).putIgnoringReadOnly(putCaptor.capture());
    assertThat(putCaptor.getValue().getClass()).isEqualTo(ContactResource.class);
    assertThat(jpaTm().transact(() -> jpaTm().loadByKey(contact.createVKey()).getEmailAddress()))
        .isEqualTo("replay@example.tld");
@@ -419,8 +421,9 @@ public class ReplayCommitLogsToSqlActionTest {

    createTld("tld");
    // Have a commit log with a couple objects that shouldn't be replayed
-    ReservedList reservedList =
-        new ReservedList.Builder().setReservedListMap(ImmutableMap.of()).setName("name").build();
+    String triplet = RdeNamingUtils.makePartialName("tld", fakeClock.nowUtc(), RdeMode.FULL);
+    RdeRevision rdeRevision =
+        RdeRevision.create(triplet, "tld", fakeClock.nowUtc().toLocalDate(), RdeMode.FULL, 1);
    ForeignKeyIndex<DomainBase> fki = ForeignKeyIndex.create(newDomainBase("foo.tld"), now);
    tm().transact(
            () -> {
@@ -430,8 +433,8 @@ public class ReplayCommitLogsToSqlActionTest {
                    createCheckpoint(now.minusMinutes(1)),
                    CommitLogManifest.create(
                        getBucketKey(1), now.minusMinutes(1), ImmutableSet.of()),
-                    // Reserved list is dually-written non-replicated
-                    CommitLogMutation.create(manifestKey, reservedList),
+                    // RDE Revisions are not replicated
+                    CommitLogMutation.create(manifestKey, rdeRevision),
                    // FKIs aren't replayed to SQL at all
                    CommitLogMutation.create(manifestKey, fki));
              } catch (IOException e) {
@@ -439,9 +442,9 @@ public class ReplayCommitLogsToSqlActionTest {
              }
            });
    runAndAssertSuccess(now.minusMinutes(1), 1, 1);
-    // jpaTm()::put should only have been called with the checkpoint
-    verify(spy, times(2)).put(any(SqlReplayCheckpoint.class));
-    verify(spy, times(2)).put(any());
+    // jpaTm()::putIgnoringReadOnly should only have been called with the checkpoint
+    verify(spy, times(2)).putIgnoringReadOnly(any(SqlReplayCheckpoint.class));
+    verify(spy, times(2)).putIgnoringReadOnly(any());
  }

  @Test
@@ -453,7 +456,7 @@ public class ReplayCommitLogsToSqlActionTest {

    jpaTm().transact(() -> SqlReplayCheckpoint.set(now.minusMinutes(1).minusMillis(1)));
    // Save a couple deletes that aren't propagated to SQL (the objects deleted are irrelevant)
-    Key<ClaimsList> claimsListKey = Key.create(ClaimsList.class, 1L);
+    Key<TmchCrl> tmchCrlKey = Key.create(TmchCrl.class, 1L);
    saveDiffFile(
        gcsUtils,
        createCheckpoint(now.minusMinutes(1)),
@@ -461,7 +464,7 @@ public class ReplayCommitLogsToSqlActionTest {
            getBucketKey(1),
            now.minusMinutes(1),
            // one object only exists in Datastore, one is dually-written (so isn't replicated)
-            ImmutableSet.of(getCrossTldKey(), claimsListKey)));
+            ImmutableSet.of(getCrossTldKey(), tmchCrlKey)));

    runAndAssertSuccess(now.minusMinutes(1), 1, 1);
    verify(spy, times(0)).delete(any(VKey.class));
@@ -554,6 +557,34 @@ public class ReplayCommitLogsToSqlActionTest {
            });
  }

+  @Test
+  void testReplay_duringReadOnly() throws Exception {
+    DateTime now = fakeClock.nowUtc();
+    jpaTm()
+        .transact(
+            () -> {
+              jpaTm().insertWithoutBackup(TestObject.create("previous to delete"));
+              SqlReplayCheckpoint.set(now.minusMinutes(2));
+            });
+    Key<CommitLogManifest> manifestKey =
+        CommitLogManifest.createKey(getBucketKey(1), now.minusMinutes(1));
+    saveDiffFile(
+        gcsUtils,
+        createCheckpoint(now.minusMinutes(1)),
+        CommitLogManifest.create(
+            getBucketKey(1),
+            now.minusMinutes(1),
+            ImmutableSet.of(Key.create(TestObject.create("previous to delete")))),
+        CommitLogMutation.create(manifestKey, TestObject.create("a")));
+    DatabaseHelper.setMigrationScheduleToDatastorePrimaryReadOnly(fakeClock);
+    runAndAssertSuccess(now.minusMinutes(1), 1, 1);
+    jpaTm()
+        .transact(
+            () ->
+                assertThat(Iterables.getOnlyElement(jpaTm().loadAllOf(TestObject.class)).getId())
+                    .isEqualTo("a"));
+  }
+
  @Test
  void testReplay_deleteAndResaveCascade_withOtherDeletion_noErrors() throws Exception {
    createTld("tld");
@@ -15,10 +15,13 @@
 package google.registry.batch;

 import static com.google.common.truth.Truth.assertThat;
+import static com.google.common.truth.Truth.assertWithMessage;
 import static com.google.common.truth.Truth8.assertThat;
 import static google.registry.model.EppResourceUtils.loadByForeignKey;
-import static google.registry.model.ofy.ObjectifyService.auditedOfy;
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.testing.DatabaseHelper.createTld;
+import static google.registry.testing.DatabaseHelper.loadByEntitiesIfPresent;
+import static google.registry.testing.DatabaseHelper.loadByEntity;
 import static google.registry.testing.DatabaseHelper.newDomainBase;
 import static google.registry.testing.DatabaseHelper.persistActiveDomain;
 import static google.registry.testing.DatabaseHelper.persistActiveHost;
@@ -46,18 +49,20 @@ import google.registry.model.poll.PollMessage;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.model.tld.Registry;
 import google.registry.model.tld.Registry.TldType;
+import google.registry.testing.DualDatabaseTest;
 import google.registry.testing.FakeResponse;
 import google.registry.testing.SystemPropertyExtension;
+import google.registry.testing.TestOfyAndSql;
 import google.registry.testing.mapreduce.MapreduceTestCase;
 import java.util.Optional;
 import java.util.Set;
 import org.joda.money.Money;
 import org.joda.time.DateTime;
 import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.RegisterExtension;

 /** Unit tests for {@link DeleteProberDataAction}. */
+@DualDatabaseTest
 class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataAction> {

  private static final DateTime DELETION_TIME = DateTime.parse("2010-01-01T00:00:00.000Z");
@@ -93,7 +98,7 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    action.response = new FakeResponse();
    action.isDryRun = false;
    action.tlds = ImmutableSet.of();
-    action.registryAdminClientId = "TheRegistrar";
+    action.registryAdminRegistrarId = "TheRegistrar";
    RegistryEnvironment.SANDBOX.setup(systemPropertyExtension);
  }

@@ -102,7 +107,7 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    executeTasksUntilEmpty("mapreduce");
  }

-  @Test
+  @TestOfyAndSql
  void test_deletesAllAndOnlyProberData() throws Exception {
    Set<ImmutableObject> tldEntities = persistLotsOfDomains("tld");
    Set<ImmutableObject> exampleEntities = persistLotsOfDomains("example");
@@ -110,14 +115,14 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    Set<ImmutableObject> ibEntities = persistLotsOfDomains("ib-any.test");
    Set<ImmutableObject> oaEntities = persistLotsOfDomains("oa-canary.test");
    runMapreduce();
-    assertNotDeleted(tldEntities);
-    assertNotDeleted(exampleEntities);
-    assertNotDeleted(notTestEntities);
-    assertDeleted(ibEntities);
-    assertDeleted(oaEntities);
+    assertAllExist(tldEntities);
+    assertAllExist(exampleEntities);
+    assertAllExist(notTestEntities);
+    assertAllAbsent(ibEntities);
+    assertAllAbsent(oaEntities);
  }

-  @Test
+  @TestOfyAndSql
  void testSuccess_deletesAllAndOnlyGivenTlds() throws Exception {
    Set<ImmutableObject> tldEntities = persistLotsOfDomains("tld");
    Set<ImmutableObject> exampleEntities = persistLotsOfDomains("example");
@@ -126,14 +131,14 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    Set<ImmutableObject> oaEntities = persistLotsOfDomains("oa-canary.test");
    action.tlds = ImmutableSet.of("example", "ib-any.test");
    runMapreduce();
-    assertNotDeleted(tldEntities);
-    assertNotDeleted(notTestEntities);
-    assertNotDeleted(oaEntities);
-    assertDeleted(exampleEntities);
-    assertDeleted(ibEntities);
+    assertAllExist(tldEntities);
+    assertAllExist(notTestEntities);
+    assertAllExist(oaEntities);
+    assertAllAbsent(exampleEntities);
+    assertAllAbsent(ibEntities);
  }

-  @Test
+  @TestOfyAndSql
  void testFail_givenNonTestTld() {
    action.tlds = ImmutableSet.of("not-test.test");
    IllegalArgumentException thrown =
@@ -143,7 +148,7 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
        .contains("If tlds are given, they must all exist and be TEST tlds");
  }

-  @Test
+  @TestOfyAndSql
  void testFail_givenNonExistentTld() {
    action.tlds = ImmutableSet.of("non-existent.test");
    IllegalArgumentException thrown =
@@ -153,7 +158,7 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
        .contains("If tlds are given, they must all exist and be TEST tlds");
  }

-  @Test
+  @TestOfyAndSql
  void testFail_givenNonDotTestTldOnProd() {
    action.tlds = ImmutableSet.of("example");
    RegistryEnvironment.PRODUCTION.setup(systemPropertyExtension);
@@ -164,44 +169,46 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
        .contains("On production, can only work on TLDs that end with .test");
  }

-  @Test
+  @TestOfyAndSql
  void testSuccess_doesntDeleteNicDomainForProbers() throws Exception {
    DomainBase nic = persistActiveDomain("nic.ib-any.test");
    ForeignKeyIndex<DomainBase> fkiNic =
        ForeignKeyIndex.load(DomainBase.class, "nic.ib-any.test", START_OF_TIME);
    Set<ImmutableObject> ibEntities = persistLotsOfDomains("ib-any.test");
    runMapreduce();
-    assertDeleted(ibEntities);
-    assertNotDeleted(ImmutableSet.of(nic, fkiNic));
+    assertAllAbsent(ibEntities);
+    assertAllExist(ImmutableSet.of(nic));
+    if (tm().isOfy()) {
+      assertAllExist(ImmutableSet.of(fkiNic));
+    }
  }

-  @Test
+  @TestOfyAndSql
  void testDryRun_doesntDeleteData() throws Exception {
    Set<ImmutableObject> tldEntities = persistLotsOfDomains("tld");
    Set<ImmutableObject> oaEntities = persistLotsOfDomains("oa-canary.test");
    action.isDryRun = true;
    runMapreduce();
-    assertNotDeleted(tldEntities);
-    assertNotDeleted(oaEntities);
+    assertAllExist(tldEntities);
+    assertAllExist(oaEntities);
  }

-  @Test
+  @TestOfyAndSql
  void testSuccess_activeDomain_isSoftDeleted() throws Exception {
-    DomainBase domain = persistResource(
-        newDomainBase("blah.ib-any.test")
-            .asBuilder()
-            .setCreationTimeForTest(DateTime.now(UTC).minusYears(1))
-            .build());
+    DomainBase domain =
+        persistResource(
+            newDomainBase("blah.ib-any.test")
+                .asBuilder()
+                .setCreationTimeForTest(DateTime.now(UTC).minusYears(1))
+                .build());
    runMapreduce();
    DateTime timeAfterDeletion = DateTime.now(UTC);
-    assertThat(loadByForeignKey(DomainBase.class, "blah.ib-any.test", timeAfterDeletion))
-        .isEmpty();
-    assertThat(auditedOfy().load().entity(domain).now().getDeletionTime())
-        .isLessThan(timeAfterDeletion);
+    assertThat(loadByForeignKey(DomainBase.class, "blah.ib-any.test", timeAfterDeletion)).isEmpty();
+    assertThat(loadByEntity(domain).getDeletionTime()).isLessThan(timeAfterDeletion);
    assertDnsTasksEnqueued("blah.ib-any.test");
  }

-  @Test
+  @TestOfyAndSql
  void testSuccess_activeDomain_doubleMapSoftDeletes() throws Exception {
    DomainBase domain = persistResource(
        newDomainBase("blah.ib-any.test")
@@ -214,12 +221,11 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    runMapreduce();
    assertThat(loadByForeignKey(DomainBase.class, "blah.ib-any.test", timeAfterDeletion))
        .isEmpty();
-    assertThat(auditedOfy().load().entity(domain).now().getDeletionTime())
-        .isLessThan(timeAfterDeletion);
+    assertThat(loadByEntity(domain).getDeletionTime()).isLessThan(timeAfterDeletion);
    assertDnsTasksEnqueued("blah.ib-any.test");
  }

-  @Test
+  @TestOfyAndSql
  void test_recentlyCreatedDomain_isntDeletedYet() throws Exception {
    persistResource(
        newDomainBase("blah.ib-any.test")
@@ -233,19 +239,20 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    assertThat(domain.get().getDeletionTime()).isEqualTo(END_OF_TIME);
  }

-  @Test
+  @TestOfyAndSql
  void testDryRun_doesntSoftDeleteData() throws Exception {
-    DomainBase domain = persistResource(
-        newDomainBase("blah.ib-any.test")
-            .asBuilder()
-            .setCreationTimeForTest(DateTime.now(UTC).minusYears(1))
-            .build());
+    DomainBase domain =
+        persistResource(
+            newDomainBase("blah.ib-any.test")
+                .asBuilder()
+                .setCreationTimeForTest(DateTime.now(UTC).minusYears(1))
+                .build());
    action.isDryRun = true;
    runMapreduce();
-    assertThat(auditedOfy().load().entity(domain).now().getDeletionTime()).isEqualTo(END_OF_TIME);
+    assertThat(loadByEntity(domain).getDeletionTime()).isEqualTo(END_OF_TIME);
  }

-  @Test
+  @TestOfyAndSql
  void test_domainWithSubordinateHosts_isSkipped() throws Exception {
    persistActiveHost("ns1.blah.ib-any.test");
    DomainBase nakedDomain =
@@ -258,18 +265,19 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
                .build(),
            DateTime.now(UTC).minusYears(1));
    runMapreduce();
-    assertThat(auditedOfy().load().entity(domainWithSubord).now()).isNotNull();
-    assertThat(auditedOfy().load().entity(nakedDomain).now()).isNull();
+
+    assertAllExist(ImmutableSet.of(domainWithSubord));
+    assertAllAbsent(ImmutableSet.of(nakedDomain));
  }

-  @Test
+  @TestOfyAndSql
  void testFailure_registryAdminClientId_isRequiredForSoftDeletion() {
    persistResource(
        newDomainBase("blah.ib-any.test")
            .asBuilder()
            .setCreationTimeForTest(DateTime.now(UTC).minusYears(1))
            .build());
-    action.registryAdminClientId = null;
+    action.registryAdminRegistrarId = null;
    IllegalStateException thrown = assertThrows(IllegalStateException.class, this::runMapreduce);
    assertThat(thrown).hasMessageThat().contains("Registry admin client ID must be configured");
  }
@@ -299,19 +307,26 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
            .setEventTime(DELETION_TIME)
            .setTargetId(fqdn)
            .build());
-    PollMessage.OneTime pollMessage = persistSimpleResource(
-        new PollMessage.OneTime.Builder()
-            .setParent(historyEntry)
-            .setEventTime(DELETION_TIME)
-            .setClientId("TheRegistrar")
-            .setMsg("Domain registered")
-            .build());
-    ForeignKeyIndex<DomainBase> fki =
-        ForeignKeyIndex.load(DomainBase.class, fqdn, START_OF_TIME);
-    EppResourceIndex eppIndex =
-        auditedOfy().load().entity(EppResourceIndex.create(Key.create(domain))).now();
-    return ImmutableSet.of(
-        domain, historyEntry, billingEvent, pollMessage, fki, eppIndex);
+    PollMessage.OneTime pollMessage =
+        persistSimpleResource(
+            new PollMessage.OneTime.Builder()
+                .setParent(historyEntry)
+                .setEventTime(DELETION_TIME)
+                .setClientId("TheRegistrar")
+                .setMsg("Domain registered")
+                .build());
+    ImmutableSet.Builder<ImmutableObject> builder =
+        new ImmutableSet.Builder<ImmutableObject>()
+            .add(domain)
+            .add(historyEntry)
+            .add(billingEvent)
+            .add(pollMessage);
+    if (tm().isOfy()) {
+      builder
+          .add(ForeignKeyIndex.load(DomainBase.class, fqdn, START_OF_TIME))
+          .add(loadByEntity(EppResourceIndex.create(Key.create(domain))));
+    }
+    return builder.build();
  }

  private static Set<ImmutableObject> persistLotsOfDomains(String tld) {
@@ -322,15 +337,15 @@ class DeleteProberDataActionTest extends MapreduceTestCase<DeleteProberDataActio
    return persistedObjects.build();
  }

-  private static void assertNotDeleted(Iterable<ImmutableObject> entities) {
-    for (ImmutableObject entity : entities) {
-      assertThat(auditedOfy().load().entity(entity).now()).isNotNull();
-    }
+  private static void assertAllExist(Iterable<ImmutableObject> entities) {
+    assertWithMessage("Expected entities to exist in the DB but they were deleted")
+        .that(loadByEntitiesIfPresent(entities))
+        .containsExactlyElementsIn(entities);
  }

-  private static void assertDeleted(Iterable<ImmutableObject> entities) {
-    for (ImmutableObject entity : entities) {
-      assertThat(auditedOfy().load().entity(entity).now()).isNull();
-    }
+  private static void assertAllAbsent(Iterable<ImmutableObject> entities) {
+    assertWithMessage("Expected entities to not exist in the DB, but they did")
+        .that(loadByEntitiesIfPresent(entities))
+        .isEmpty();
  }
 }
@@ -85,7 +85,7 @@ public class ExpandRecurringBillingEventsActionTest

  @Order(Order.DEFAULT - 2)
  @RegisterExtension
-  public final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  public final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  private DomainBase domain;
  private DomainHistory historyEntry;
@@ -0,0 +1,607 @@
+// Copyright 2021 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.batch;
+
+import static com.google.common.truth.Truth.assertThat;
+import static google.registry.testing.AppEngineExtension.makeRegistrar1;
+import static google.registry.testing.DatabaseHelper.loadByEntity;
+import static google.registry.testing.DatabaseHelper.persistResource;
+import static google.registry.testing.DatabaseHelper.persistSimpleResources;
+import static google.registry.util.DateTimeUtils.START_OF_TIME;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedMap;
+import google.registry.batch.SendExpiringCertificateNotificationEmailAction.CertificateType;
+import google.registry.batch.SendExpiringCertificateNotificationEmailAction.RegistrarInfo;
+import google.registry.flows.certs.CertificateChecker;
+import google.registry.model.registrar.Registrar;
+import google.registry.model.registrar.RegistrarAddress;
+import google.registry.model.registrar.RegistrarContact;
+import google.registry.model.registrar.RegistrarContact.Type;
+import google.registry.request.Response;
+import google.registry.testing.AppEngineExtension;
+import google.registry.testing.DualDatabaseTest;
+import google.registry.testing.FakeClock;
+import google.registry.testing.InjectExtension;
+import google.registry.testing.TestOfyAndSql;
+import google.registry.util.SelfSignedCaCertificate;
+import google.registry.util.SendEmailService;
+import java.security.cert.X509Certificate;
+import java.util.Optional;
+import javax.annotation.Nullable;
+import javax.mail.internet.InternetAddress;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+/** Unit tests for {@link SendExpiringCertificateNotificationEmailAction}. */
+@DualDatabaseTest
+class SendExpiringCertificateNotificationEmailActionTest {
+
+  @RegisterExtension
+  public final AppEngineExtension appEngine =
+      AppEngineExtension.builder().withDatastoreAndCloudSql().withTaskQueue().build();
+
+  @RegisterExtension public final InjectExtension inject = new InjectExtension();
+  private final FakeClock clock = new FakeClock(DateTime.parse("2021-05-24T20:21:22Z"));
+  private final SendEmailService sendEmailService = mock(SendEmailService.class);
+  private CertificateChecker certificateChecker;
+  private SendExpiringCertificateNotificationEmailAction action;
+  private Registrar sampleRegistrar;
+  private Response response;
+
+  @BeforeEach
+  void beforeEach() throws Exception {
+    certificateChecker =
+        new CertificateChecker(
+            ImmutableSortedMap.of(START_OF_TIME, 825, DateTime.parse("2020-09-01T00:00:00Z"), 398),
+            30,
+            15,
+            2048,
+            ImmutableSet.of("secp256r1", "secp384r1"),
+            clock);
+    String expirationWarningEmailBodyText =
+        " Hello Registrar %s,\n" + "       The %s certificate is expiring on %s.";
+    String expirationWarningEmailSubjectText = "expiring certificate notification email";
+
+    action =
+        new SendExpiringCertificateNotificationEmailAction(
+            expirationWarningEmailBodyText,
+            expirationWarningEmailSubjectText,
+            new InternetAddress("test@example.com"),
+            sendEmailService,
+            certificateChecker,
+            response);
+
+    sampleRegistrar =
+        persistResource(createRegistrar("clientId", "sampleRegistrar", null, null).build());
+  }
+
+  /** Returns a sample registrar with a customized registrar name, client id and certificate* */
+  private Registrar.Builder createRegistrar(
+      String clientId,
+      String registrarName,
+      @Nullable X509Certificate certificate,
+      @Nullable X509Certificate failOverCertificate)
+      throws Exception {
+    // set up only required fields sample test data
+    Registrar.Builder builder =
+        new Registrar.Builder()
+            .setClientId(clientId)
+            .setRegistrarName(registrarName)
+            .setType(Registrar.Type.REAL)
+            .setIanaIdentifier(8L)
+            .setState(Registrar.State.ACTIVE)
+            .setInternationalizedAddress(
+                new RegistrarAddress.Builder()
+                    .setStreet(ImmutableList.of("very fake street"))
+                    .setCity("city")
+                    .setState("state")
+                    .setZip("99999")
+                    .setCountryCode("US")
+                    .build())
+            .setPhoneNumber("+0.000000000")
+            .setFaxNumber("+9.999999999")
+            .setEmailAddress("contact-us@test.example")
+            .setWhoisServer("whois.registrar.example")
+            .setUrl("http://www.test.example");
+
+    if (failOverCertificate != null) {
+      builder.setFailoverClientCertificate(
+          certificateChecker.serializeCertificate(failOverCertificate), clock.nowUtc());
+    }
+    if (certificate != null) {
+      builder.setClientCertificate(
+          certificateChecker.serializeCertificate(certificate), clock.nowUtc());
+    }
+    return builder;
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmail_returnsTrue() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    Optional<String> cert =
+        Optional.of(certificateChecker.serializeCertificate(expiringCertificate));
+    Registrar registrar =
+        persistResource(
+            makeRegistrar1()
+                .asBuilder()
+                .setFailoverClientCertificate(cert.get(), clock.nowUtc())
+                .build());
+    ImmutableList<RegistrarContact> contacts =
+        ImmutableList.of(
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("Will Doe")
+                .setEmailAddress("will@example-registrar.tld")
+                .setPhoneNumber("+1.3105551213")
+                .setFaxNumber("+1.3105551213")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.TECH))
+                .setVisibleInWhoisAsAdmin(true)
+                .setVisibleInWhoisAsTech(false)
+                .build());
+    persistSimpleResources(contacts);
+    persistResource(registrar);
+    assertThat(
+            action.sendNotificationEmail(registrar, START_OF_TIME, CertificateType.FAILOVER, cert))
+        .isEqualTo(true);
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmail_returnsFalse_noEmailRecipients() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-02T00:00:00Z"))
+            .cert();
+    Optional<String> cert =
+        Optional.of(certificateChecker.serializeCertificate(expiringCertificate));
+    assertThat(
+            action.sendNotificationEmail(
+                sampleRegistrar, START_OF_TIME, CertificateType.FAILOVER, cert))
+        .isEqualTo(false);
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmail_throwsRunTimeException() throws Exception {
+    doThrow(new RuntimeException("this is a runtime exception"))
+        .when(sendEmailService)
+        .sendEmail(any());
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    Optional<String> cert =
+        Optional.of(certificateChecker.serializeCertificate(expiringCertificate));
+    Registrar registrar =
+        persistResource(
+            makeRegistrar1()
+                .asBuilder()
+                .setFailoverClientCertificate(cert.get(), clock.nowUtc())
+                .build());
+    ImmutableList<RegistrarContact> contacts =
+        ImmutableList.of(
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("Will Doe")
+                .setEmailAddress("will@example-registrar.tld")
+                .setPhoneNumber("+1.3105551213")
+                .setFaxNumber("+1.3105551213")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.TECH))
+                .setVisibleInWhoisAsAdmin(true)
+                .setVisibleInWhoisAsTech(false)
+                .build());
+    persistSimpleResources(contacts);
+    RuntimeException thrown =
+        assertThrows(
+            RuntimeException.class,
+            () ->
+                action.sendNotificationEmail(
+                    registrar, START_OF_TIME, CertificateType.FAILOVER, cert));
+    assertThat(thrown)
+        .hasMessageThat()
+        .contains(
+            String.format(
+                "Failed to send expiring certificate notification email to registrar %s",
+                registrar.getRegistrarName()));
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmail_returnsFalse_noCertificate() {
+    assertThat(
+            action.sendNotificationEmail(
+                sampleRegistrar, START_OF_TIME, CertificateType.FAILOVER, Optional.empty()))
+        .isEqualTo(false);
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmails_allEmailsBeingAttemptedToSend() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    X509Certificate certificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-10-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    int numOfRegistrarsWithExpiringCertificates = 2;
+    for (int i = 1; i <= numOfRegistrarsWithExpiringCertificates; i++) {
+      persistResource(
+          createRegistrar("oldcert" + i, "name" + i, expiringCertificate, null).build());
+    }
+    for (int i = numOfRegistrarsWithExpiringCertificates; i <= numOfRegistrars; i++) {
+      persistResource(createRegistrar("goodcert" + i, "name" + i, certificate, null).build());
+    }
+    assertThat(action.sendNotificationEmails()).isEqualTo(numOfRegistrarsWithExpiringCertificates);
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmails_allEmailsBeingAttemptedToSend_onlyMainCertificates()
+      throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    for (int i = 1; i <= numOfRegistrars; i++) {
+      persistResource(
+          createRegistrar("oldcert" + i, "name" + i, expiringCertificate, null).build());
+    }
+    assertThat(action.sendNotificationEmails()).isEqualTo(numOfRegistrars);
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmails_allEmailsBeingAttemptedToSend_onlyFailOverCertificates()
+      throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    for (int i = 1; i <= numOfRegistrars; i++) {
+      persistResource(
+          createRegistrar("oldcert" + i, "name" + i, null, expiringCertificate).build());
+    }
+    assertThat(action.sendNotificationEmails()).isEqualTo(numOfRegistrars);
+  }
+
+  @TestOfyAndSql
+  void sendNotificationEmails_allEmailsBeingAttemptedToSend_mixedOfCertificates() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    int numOfExpiringFailOverOnly = 2;
+    int numOfExpiringPrimaryOnly = 3;
+    for (int i = 1; i <= numOfExpiringFailOverOnly; i++) {
+      persistResource(
+          createRegistrar("cl" + i, "expiringFailOverOnly" + i, null, expiringCertificate).build());
+    }
+    for (int i = 1; i <= numOfExpiringPrimaryOnly; i++) {
+      persistResource(
+          createRegistrar("cli" + i, "expiringPrimaryOnly" + i, expiringCertificate, null).build());
+    }
+    for (int i = numOfExpiringFailOverOnly + numOfExpiringPrimaryOnly + 1;
+        i <= numOfRegistrars;
+        i++) {
+      persistResource(
+          createRegistrar("client" + i, "regularReg" + i, expiringCertificate, expiringCertificate)
+              .build());
+    }
+    assertThat(action.sendNotificationEmails())
+        .isEqualTo(numOfRegistrars + numOfExpiringFailOverOnly + numOfExpiringPrimaryOnly);
+  }
+
+  @TestOfyAndSql
+  void updateLastNotificationSentDate_updatedSuccessfully_primaryCertificate() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-02T00:00:00Z"))
+            .cert();
+    Registrar registrar =
+        createRegistrar("testClientId", "registrar", expiringCertificate, null).build();
+    persistResource(registrar);
+    action.updateLastNotificationSentDate(registrar, clock.nowUtc(), CertificateType.PRIMARY);
+    assertThat(loadByEntity(registrar).getLastExpiringCertNotificationSentDate())
+        .isEqualTo(clock.nowUtc());
+  }
+
+  @TestOfyAndSql
+  void updateLastNotificationSentDate_updatedSuccessfully_failOverCertificate() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    Registrar registrar =
+        createRegistrar("testClientId", "registrar", null, expiringCertificate).build();
+    persistResource(registrar);
+    action.updateLastNotificationSentDate(registrar, clock.nowUtc(), CertificateType.FAILOVER);
+    assertThat(loadByEntity(registrar).getLastExpiringFailoverCertNotificationSentDate())
+        .isEqualTo(clock.nowUtc());
+  }
+
+  @TestOfyAndSql
+  void updateLastNotificationSentDate_noUpdates_noLastNotificationSentDate() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    Registrar registrar =
+        createRegistrar("testClientId", "registrar", null, expiringCertificate).build();
+    persistResource(registrar);
+    RuntimeException thrown =
+        assertThrows(
+            RuntimeException.class,
+            () -> action.updateLastNotificationSentDate(registrar, null, CertificateType.FAILOVER));
+    assertThat(thrown)
+        .hasMessageThat()
+        .contains("Failed to update the last notification sent date to Registrar");
+  }
+
+  @TestOfyAndSql
+  void updateLastNotificationSentDate_noUpdates_invalidCertificateType() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    Registrar registrar =
+        createRegistrar("testClientId", "registrar", null, expiringCertificate).build();
+    persistResource(registrar);
+    IllegalArgumentException thrown =
+        assertThrows(
+            IllegalArgumentException.class,
+            () ->
+                action.updateLastNotificationSentDate(
+                    registrar, clock.nowUtc(), CertificateType.valueOf("randomType")));
+    assertThat(thrown).hasMessageThat().contains("No enum constant");
+  }
+
+  @TestOfyAndSql
+  void getRegistrarsWithExpiringCertificates_returnsPartOfRegistrars() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    X509Certificate certificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-10-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    int numOfRegistrarsWithExpiringCertificates = 2;
+    for (int i = 1; i <= numOfRegistrarsWithExpiringCertificates; i++) {
+      persistResource(
+          createRegistrar("oldcert" + i, "name" + i, expiringCertificate, null).build());
+    }
+    for (int i = numOfRegistrarsWithExpiringCertificates; i <= numOfRegistrars; i++) {
+      persistResource(createRegistrar("goodcert" + i, "name" + i, certificate, null).build());
+    }
+
+    ImmutableList<RegistrarInfo> results = action.getRegistrarsWithExpiringCertificates();
+    assertThat(results.size()).isEqualTo(numOfRegistrarsWithExpiringCertificates);
+  }
+
+  @TestOfyAndSql
+  void getRegistrarsWithExpiringCertificates_returnsPartOfRegistrars_failOverCertificateBranch()
+      throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+    X509Certificate certificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-10-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    int numOfRegistrarsWithExpiringCertificates = 2;
+    for (int i = 1; i <= numOfRegistrarsWithExpiringCertificates; i++) {
+      persistResource(
+          createRegistrar("oldcert" + i, "name" + i, null, expiringCertificate).build());
+    }
+    for (int i = numOfRegistrarsWithExpiringCertificates; i <= numOfRegistrars; i++) {
+      persistResource(createRegistrar("goodcert" + i, "name" + i, null, certificate).build());
+    }
+
+    assertThat(action.getRegistrarsWithExpiringCertificates().size())
+        .isEqualTo(numOfRegistrarsWithExpiringCertificates);
+  }
+
+  @TestOfyAndSql
+  void getRegistrarsWithExpiringCertificates_returnsAllRegistrars() throws Exception {
+    X509Certificate expiringCertificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-06-01T00:00:00Z"))
+            .cert();
+
+    int numOfRegistrarsWithExpiringCertificates = 5;
+    for (int i = 1; i <= numOfRegistrarsWithExpiringCertificates; i++) {
+      persistResource(
+          createRegistrar("oldcert" + i, "name" + i, expiringCertificate, null).build());
+    }
+    assertThat(action.getRegistrarsWithExpiringCertificates().size())
+        .isEqualTo(numOfRegistrarsWithExpiringCertificates);
+  }
+
+  @TestOfyAndSql
+  void getRegistrarsWithExpiringCertificates_returnsNoRegistrars() throws Exception {
+    X509Certificate certificate =
+        SelfSignedCaCertificate.create(
+                "www.example.tld",
+                DateTime.parse("2020-09-02T00:00:00Z"),
+                DateTime.parse("2021-10-01T00:00:00Z"))
+            .cert();
+    int numOfRegistrars = 10;
+    for (int i = 1; i <= numOfRegistrars; i++) {
+      persistResource(createRegistrar("goodcert" + i, "name" + i, certificate, null).build());
+    }
+    assertThat(action.getRegistrarsWithExpiringCertificates()).isEmpty();
+  }
+
+  @TestOfyAndSql
+  void getRegistrarsWithExpiringCertificates_noRegistrarsInDatabase() {
+    assertThat(action.getRegistrarsWithExpiringCertificates()).isEmpty();
+  }
+
+  @TestOfyAndSql
+  void getEmailAddresses_success_returnsAnEmptyList() {
+    assertThat(action.getEmailAddresses(sampleRegistrar, Type.TECH)).isEmpty();
+    assertThat(action.getEmailAddresses(sampleRegistrar, Type.ADMIN)).isEmpty();
+  }
+
+  @TestOfyAndSql
+  void getEmailAddresses_success_returnsAListOfEmails() throws Exception {
+    Registrar registrar = persistResource(makeRegistrar1());
+    ImmutableList<RegistrarContact> contacts =
+        ImmutableList.of(
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("John Doe")
+                .setEmailAddress("jd@example-registrar.tld")
+                .setPhoneNumber("+1.3105551213")
+                .setFaxNumber("+1.3105551213")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.TECH))
+                .setVisibleInWhoisAsAdmin(true)
+                .setVisibleInWhoisAsTech(false)
+                .build(),
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("John Smith")
+                .setEmailAddress("js@example-registrar.tld")
+                .setPhoneNumber("+1.1111111111")
+                .setFaxNumber("+1.1111111111")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.TECH))
+                .build(),
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("Will Doe")
+                .setEmailAddress("will@example-registrar.tld")
+                .setPhoneNumber("+1.3105551213")
+                .setFaxNumber("+1.3105551213")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.TECH))
+                .setVisibleInWhoisAsAdmin(true)
+                .setVisibleInWhoisAsTech(false)
+                .build(),
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("Mike Doe")
+                .setEmailAddress("mike@example-registrar.tld")
+                .setPhoneNumber("+1.1111111111")
+                .setFaxNumber("+1.1111111111")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.ADMIN))
+                .build(),
+            new RegistrarContact.Builder()
+                .setParent(registrar)
+                .setName("John T")
+                .setEmailAddress("john@example-registrar.tld")
+                .setPhoneNumber("+1.3105551215")
+                .setFaxNumber("+1.3105551216")
+                .setTypes(ImmutableSet.of(RegistrarContact.Type.ADMIN))
+                .setVisibleInWhoisAsTech(true)
+                .build());
+    persistSimpleResources(contacts);
+    assertThat(action.getEmailAddresses(registrar, Type.TECH))
+        .containsExactly(
+            new InternetAddress("will@example-registrar.tld"),
+            new InternetAddress("jd@example-registrar.tld"),
+            new InternetAddress("js@example-registrar.tld"));
+    assertThat(action.getEmailAddresses(registrar, Type.ADMIN))
+        .containsExactly(
+            new InternetAddress("janedoe@theregistrar.com"), // comes with makeRegistrar1()
+            new InternetAddress("mike@example-registrar.tld"),
+            new InternetAddress("john@example-registrar.tld"));
+  }
+
+  @TestOfyAndSql
+  void getEmailAddresses_failure_returnsPartialListOfEmails_skipInvalidEmails() {
+    // when building a new RegistrarContact object, there's already an email validation process.
+    // if the registrarContact is created successful, the email address of the contact object
+    // should already be validated. Ideally, there should not be an AddressException when creating
+    // a new InternetAddress using the email address string of the contact object.
+  }
+
+  @TestOfyAndSql
+  void getEmailBody_returnsEmailBodyText() {
+    String registrarName = "good registrar";
+    String certExpirationDateStr = "2021-06-15";
+    CertificateType certificateType = CertificateType.PRIMARY;
+    String emailBody =
+        action.getEmailBody(
+            registrarName, certificateType, DateTime.parse(certExpirationDateStr).toDate());
+    assertThat(emailBody).contains(registrarName);
+    assertThat(emailBody).contains(certificateType.getDisplayName());
+    assertThat(emailBody).contains(certExpirationDateStr);
+  }
+
+  @TestOfyAndSql
+  void getEmailBody_throwsIllegalArgumentException_noExpirationDate() {
+    IllegalArgumentException thrown =
+        assertThrows(
+            IllegalArgumentException.class,
+            () -> action.getEmailBody("good registrar", CertificateType.FAILOVER, null));
+    assertThat(thrown).hasMessageThat().contains("Expiration date cannot be null");
+  }
+
+  @TestOfyAndSql
+  void getEmailBody_throwsIllegalArgumentException_noCertificateType() {
+    IllegalArgumentException thrown =
+        assertThrows(
+            IllegalArgumentException.class,
+            () ->
+                action.getEmailBody("good registrar", null, DateTime.parse("2021-06-15").toDate()));
+    assertThat(thrown).hasMessageThat().contains("Certificate type cannot be null");
+  }
+}
@@ -15,27 +15,23 @@
 package google.registry.cron;

 import static com.google.common.collect.ImmutableList.toImmutableList;
-import static com.google.common.collect.Iterables.getLast;
 import static com.google.common.truth.Truth.assertThat;
 import static google.registry.testing.DatabaseHelper.createTlds;
 import static google.registry.testing.DatabaseHelper.persistResource;
-import static google.registry.testing.TaskQueueHelper.assertNoTasksEnqueued;
-import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
 import static org.junit.jupiter.api.Assertions.assertThrows;

-import com.google.appengine.api.taskqueue.dev.QueueStateInfo.TaskStateInfo;
-import com.google.appengine.tools.development.testing.LocalTaskQueueTestConfig;
-import com.google.common.base.Joiner;
+import com.google.cloud.tasks.v2.HttpMethod;
+import com.google.cloud.tasks.v2.Task;
 import com.google.common.base.Splitter;
 import com.google.common.collect.ImmutableListMultimap;
 import com.google.common.collect.ImmutableSet;
 import google.registry.model.tld.Registry;
 import google.registry.model.tld.Registry.TldType;
 import google.registry.testing.AppEngineExtension;
+import google.registry.testing.CloudTasksHelper;
+import google.registry.testing.CloudTasksHelper.TaskMatcher;
+import google.registry.testing.FakeClock;
 import google.registry.testing.FakeResponse;
-import google.registry.testing.TaskQueueHelper.TaskMatcher;
-import google.registry.util.Retrier;
-import google.registry.util.TaskQueueUtils;
 import java.util.List;
 import java.util.Optional;
 import java.util.stream.Stream;
@@ -49,27 +45,14 @@ class TldFanoutActionTest {
  private static final String ENDPOINT = "/the/servlet";
  private static final String QUEUE = "the-queue";
  private final FakeResponse response = new FakeResponse();
+  private final CloudTasksHelper cloudTasksHelper = new CloudTasksHelper();

  @RegisterExtension
  final AppEngineExtension appEngine =
-      AppEngineExtension.builder()
-          .withDatastoreAndCloudSql()
-          .withTaskQueue(
-              Joiner.on('\n')
-                  .join(
-                      "<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
-                      "<queue-entries>",
-                      "  <queue>",
-                      "    <name>the-queue</name>",
-                      "    <rate>1/s</rate>",
-                      "  </queue>",
-                      "</queue-entries>"))
-          .build();
+      AppEngineExtension.builder().withDatastoreAndCloudSql().build();

  private static ImmutableListMultimap<String, String> getParamsMap(String... keysAndValues) {
    ImmutableListMultimap.Builder<String, String> params = new ImmutableListMultimap.Builder<>();
-    params.put("queue", QUEUE);
-    params.put("endpoint", ENDPOINT);
    for (int i = 0; i < keysAndValues.length; i += 2) {
      params.put(keysAndValues[i], keysAndValues[i + 1]);
    }
@@ -78,13 +61,15 @@ class TldFanoutActionTest {

  private void run(ImmutableListMultimap<String, String> params) {
    TldFanoutAction action = new TldFanoutAction();
+    action.clock = new FakeClock();
    action.params = params;
-    action.endpoint = getLast(params.get("endpoint"));
-    action.queue = getLast(params.get("queue"));
-    action.excludes = params.containsKey("exclude")
-        ? ImmutableSet.copyOf(Splitter.on(',').split(params.get("exclude").get(0)))
-        : ImmutableSet.of();
-    action.taskQueueUtils = new TaskQueueUtils(new Retrier(null, 1));
+    action.endpoint = ENDPOINT;
+    action.queue = QUEUE;
+    action.excludes =
+        params.containsKey("exclude")
+            ? ImmutableSet.copyOf(Splitter.on(',').split(params.get("exclude").get(0)))
+            : ImmutableSet.of();
+    action.cloudTasksUtils = cloudTasksHelper.getTestCloudTasksUtils();
    action.response = response;
    action.runInEmpty = params.containsKey("runInEmpty");
    action.forEachRealTld = params.containsKey("forEachRealTld");
@@ -99,20 +84,21 @@ class TldFanoutActionTest {
    persistResource(Registry.get("example").asBuilder().setTldType(TldType.TEST).build());
  }

-  private static void assertTasks(String... tasks) {
-    assertTasksEnqueued(
+  private void assertTasks(String... tasks) {
+    cloudTasksHelper.assertTasksEnqueued(
        QUEUE,
-        Stream.of(tasks).map(
-            namespace ->
-                new TaskMatcher()
-                    .url(ENDPOINT)
-                    .header("content-type", "application/x-www-form-urlencoded")
-                    .param("tld", namespace))
-        .collect(toImmutableList()));
+        Stream.of(tasks)
+            .map(
+                namespace ->
+                    new TaskMatcher()
+                        .url(ENDPOINT)
+                        .header("content-type", "application/x-www-form-urlencoded")
+                        .param("tld", namespace))
+            .collect(toImmutableList()));
  }

-  private static void assertTaskWithoutTld() {
-    assertTasksEnqueued(
+  private void assertTaskWithoutTld() {
+    cloudTasksHelper.assertTasksEnqueued(
        QUEUE,
        new TaskMatcher()
            .url(ENDPOINT)
@@ -120,9 +106,10 @@ class TldFanoutActionTest {
  }

  @Test
-  void testSuccess_methodPostIsDefault() {
+  void testSuccess_methodPostAndServiceBackendAreDefault() {
    run(getParamsMap("runInEmpty", ""));
-    assertTasksEnqueued(QUEUE, new TaskMatcher().method("POST"));
+    cloudTasksHelper.assertTasksEnqueued(
+        QUEUE, new TaskMatcher().method(HttpMethod.POST).service("backend"));
  }

  @Test
@@ -150,9 +137,10 @@ class TldFanoutActionTest {

  @Test
  void testSuccess_forEachTestTldAndForEachRealTld() {
-    run(getParamsMap(
-        "forEachTestTld", "",
-        "forEachRealTld", ""));
+    run(
+        getParamsMap(
+            "forEachTestTld", "",
+            "forEachRealTld", ""));
    assertTasks("com", "net", "org", "example");
  }

@@ -164,26 +152,29 @@ class TldFanoutActionTest {

  @Test
  void testSuccess_excludeRealTlds() {
-    run(getParamsMap(
-        "forEachRealTld", "",
-        "exclude", "com,net"));
+    run(
+        getParamsMap(
+            "forEachRealTld", "",
+            "exclude", "com,net"));
    assertTasks("org");
  }

  @Test
  void testSuccess_excludeTestTlds() {
-    run(getParamsMap(
-        "forEachTestTld", "",
-        "exclude", "example"));
-    assertNoTasksEnqueued(QUEUE);
+    run(
+        getParamsMap(
+            "forEachTestTld", "",
+            "exclude", "example"));
+    cloudTasksHelper.assertNoTasksEnqueued(QUEUE);
  }

  @Test
  void testSuccess_excludeNonexistentTlds() {
-    run(getParamsMap(
-        "forEachTestTld", "",
-        "forEachRealTld", "",
-        "exclude", "foo"));
+    run(
+        getParamsMap(
+            "forEachTestTld", "",
+            "forEachRealTld", "",
+            "exclude", "foo"));
    assertTasks("com", "net", "org", "example");
  }

@@ -223,26 +214,24 @@ class TldFanoutActionTest {
  @Test
  void testSuccess_additionalArgsFlowThroughToPostParams() {
    run(getParamsMap("forEachTestTld", "", "newkey", "newval"));
-    assertTasksEnqueued(QUEUE,
-        new TaskMatcher().url("/the/servlet").param("newkey", "newval"));
+    cloudTasksHelper.assertTasksEnqueued(
+        QUEUE, new TaskMatcher().url("/the/servlet").param("newkey", "newval"));
  }

  @Test
  void testSuccess_returnHttpResponse() {
    run(getParamsMap("forEachRealTld", "", "endpoint", "/the/servlet"));

-    List<TaskStateInfo> taskList =
-        LocalTaskQueueTestConfig.getLocalTaskQueue().getQueueStateInfo().get(QUEUE).getTaskInfo();
+    List<Task> taskList = cloudTasksHelper.getTestTasksFor(QUEUE);

    assertThat(taskList).hasSize(3);
-    String expectedResponse = String.format(
-        "OK: Launched the following 3 tasks in queue the-queue\n"
-            + "- Task: '%s', tld: 'com', endpoint: '/the/servlet'\n"
-            + "- Task: '%s', tld: 'net', endpoint: '/the/servlet'\n"
-            + "- Task: '%s', tld: 'org', endpoint: '/the/servlet'\n",
-        taskList.get(0).getTaskName(),
-        taskList.get(1).getTaskName(),
-        taskList.get(2).getTaskName());
+    String expectedResponse =
+        String.format(
+            "OK: Launched the following 3 tasks in queue the-queue\n"
+                + "- Task: '%s', tld: 'com', endpoint: '/the/servlet'\n"
+                + "- Task: '%s', tld: 'net', endpoint: '/the/servlet'\n"
+                + "- Task: '%s', tld: 'org', endpoint: '/the/servlet'\n",
+            taskList.get(0).getName(), taskList.get(1).getName(), taskList.get(2).getName());
    assertThat(response.getPayload()).isEqualTo(expectedResponse);
  }

@@ -250,14 +239,14 @@ class TldFanoutActionTest {
  void testSuccess_returnHttpResponse_runInEmpty() {
    run(getParamsMap("runInEmpty", "", "endpoint", "/the/servlet"));

-    List<TaskStateInfo> taskList =
-        LocalTaskQueueTestConfig.getLocalTaskQueue().getQueueStateInfo().get(QUEUE).getTaskInfo();
+    List<Task> taskList = cloudTasksHelper.getTestTasksFor(QUEUE);

    assertThat(taskList).hasSize(1);
-    String expectedResponse = String.format(
-        "OK: Launched the following 1 tasks in queue the-queue\n"
-            + "- Task: '%s', tld: '', endpoint: '/the/servlet'\n",
-        taskList.get(0).getTaskName());
+    String expectedResponse =
+        String.format(
+            "OK: Launched the following 1 tasks in queue the-queue\n"
+                + "- Task: '%s', tld: '', endpoint: '/the/servlet'\n",
+            taskList.get(0).getName());
    assertThat(response.getPayload()).isEqualTo(expectedResponse);
  }
 }
@@ -758,11 +758,11 @@ class EppLifecycleDomainTest extends EppTestCase {
        .hasResponse(
            "poll_response_autorenew.xml",
            ImmutableMap.of(
-                "ID", "1-C-EXAMPLE-13-16-2002",
+                "ID", "1-B-EXAMPLE-12-15-2002",
                "QDATE", "2002-06-01T00:04:00Z",
                "DOMAIN", "fakesite.example",
                "EXDATE", "2003-06-01T00:04:00Z"));
-    assertThatCommand("poll_ack.xml", ImmutableMap.of("ID", "1-C-EXAMPLE-13-16-2002"))
+    assertThatCommand("poll_ack.xml", ImmutableMap.of("ID", "1-B-EXAMPLE-12-15-2002"))
        .atTime("2002-07-01T00:02:00Z")
        .hasResponse("poll_ack_response_empty.xml");

@@ -776,13 +776,13 @@ class EppLifecycleDomainTest extends EppTestCase {
        .hasResponse(
            "poll_response_autorenew.xml",
            ImmutableMap.of(
-                "ID", "1-C-EXAMPLE-13-16-2003", // Note -- Year is different from previous ID.
+                "ID", "1-B-EXAMPLE-12-15-2003", // Note -- Year is different from previous ID.
                "QDATE", "2003-06-01T00:04:00Z",
                "DOMAIN", "fakesite.example",
                "EXDATE", "2004-06-01T00:04:00Z"));

    // Ack the second poll message and verify that none remain.
-    assertThatCommand("poll_ack.xml", ImmutableMap.of("ID", "1-C-EXAMPLE-13-16-2003"))
+    assertThatCommand("poll_ack.xml", ImmutableMap.of("ID", "1-B-EXAMPLE-12-15-2003"))
        .atTime("2003-07-01T00:05:05Z")
        .hasResponse("poll_ack_response_empty.xml");
    assertThatCommand("poll.xml")
@@ -812,7 +812,7 @@ class EppLifecycleDomainTest extends EppTestCase {

    // As the losing registrar, read the request poll message, and then ack it.
    assertThatLoginSucceeds("NewRegistrar", "foo-BAR2");
-    String messageId = "1-C-EXAMPLE-19-25-2001";
+    String messageId = "1-B-EXAMPLE-18-24-2001";
    assertThatCommand("poll.xml")
        .atTime("2001-01-01T00:01:00Z")
        .hasResponse("poll_response_domain_transfer_request.xml", ImmutableMap.of("ID", messageId));
@@ -821,7 +821,7 @@ class EppLifecycleDomainTest extends EppTestCase {
        .hasResponse("poll_ack_response_empty.xml");

    // Five days in the future, expect a server approval poll message to the loser, and ack it.
-    messageId = "1-C-EXAMPLE-19-24-2001";
+    messageId = "1-B-EXAMPLE-18-23-2001";
    assertThatCommand("poll.xml")
        .atTime("2001-01-06T00:01:00Z")
        .hasResponse(
@@ -833,7 +833,7 @@ class EppLifecycleDomainTest extends EppTestCase {
    assertThatLogoutSucceeds();

    // Also expect a server approval poll message to the winner, with the transfer request trid.
-    messageId = "1-C-EXAMPLE-19-23-2001";
+    messageId = "1-B-EXAMPLE-18-22-2001";
    assertThatLoginSucceeds("TheRegistrar", "password2");
    assertThatCommand("poll.xml")
        .atTime("2001-01-06T00:02:00Z")
@@ -51,7 +51,8 @@ import org.mockito.Mockito;
 class FlowRunnerTest {

  @RegisterExtension
-  final AppEngineExtension appEngineRule = new AppEngineExtension.Builder().build();
+  final AppEngineExtension appEngineRule =
+      new AppEngineExtension.Builder().withDatastoreAndCloudSql().build();

  private final FlowRunner flowRunner = new FlowRunner();
  private final EppMetric.Builder eppMetricBuilder = EppMetric.builderForRequest(new FakeClock());
@@ -36,7 +36,7 @@ class ContactCheckFlowTest extends ResourceCheckFlowTestCase<ContactCheckFlow, C

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  ContactCheckFlowTest() {
    setEppInput("contact_check.xml");
@@ -45,7 +45,7 @@ class ContactCreateFlowTest extends ResourceFlowTestCase<ContactCreateFlow, Cont

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  ContactCreateFlowTest() {
    setEppInput("contact_create.xml");
@@ -66,7 +66,7 @@ class ContactDeleteFlowTest extends ResourceFlowTestCase<ContactDeleteFlow, Cont

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void initFlowTest() {
@@ -51,7 +51,7 @@ class ContactInfoFlowTest extends ResourceFlowTestCase<ContactInfoFlow, ContactR

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  ContactInfoFlowTest() {
    setEppInput("contact_info.xml");
@@ -54,7 +54,7 @@ class ContactTransferApproveFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -51,7 +51,7 @@ class ContactTransferCancelFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -46,7 +46,7 @@ class ContactTransferQueryFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -53,7 +53,7 @@ class ContactTransferRejectFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -65,7 +65,7 @@ class ContactTransferRequestFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  ContactTransferRequestFlowTest() {
    // We need the transfer to happen at exactly this time in order for the response to match up.
@@ -53,7 +53,7 @@ class ContactUpdateFlowTest extends ResourceFlowTestCase<ContactUpdateFlow, Cont

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  ContactUpdateFlowTest() {
    setEppInput("contact_update.xml");
@@ -93,7 +93,7 @@ class DomainCheckFlowTest extends ResourceCheckFlowTestCase<DomainCheckFlow, Dom

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  DomainCheckFlowTest() {
    setEppInput("domain_check_one_tld.xml");
@@ -50,7 +50,7 @@ public class DomainClaimsCheckFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  DomainClaimsCheckFlowTest() {
    setEppInput("domain_check_claims.xml");
@@ -188,7 +188,7 @@ class DomainCreateFlowTest extends ResourceFlowTestCase<DomainCreateFlow, Domain

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  DomainCreateFlowTest() {
    setEppInput("domain_create.xml", ImmutableMap.of("DOMAIN", "example.tld"));
@@ -119,7 +119,7 @@ class DomainDeleteFlowTest extends ResourceFlowTestCase<DomainDeleteFlow, Domain

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  private DomainBase domain;
  private DomainHistory earlierHistoryEntry;
@@ -87,7 +87,7 @@ class DomainInfoFlowTest extends ResourceFlowTestCase<DomainInfoFlow, DomainBase

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  /**
   * The domain_info_fee.xml default substitutions common to most tests.
@@ -112,7 +112,7 @@ class DomainRenewFlowTest extends ResourceFlowTestCase<DomainRenewFlow, DomainBa

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void initDomainTest() {
@@ -90,7 +90,7 @@ class DomainRestoreRequestFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  private static final ImmutableMap<String, String> FEE_06_MAP =
      ImmutableMap.of("FEE_VERSION", "0.6", "FEE_NS", "fee", "CURRENCY", "USD");
@@ -91,7 +91,7 @@ class DomainTransferApproveFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -71,7 +71,7 @@ class DomainTransferCancelFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -48,7 +48,7 @@ class DomainTransferQueryFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void beforeEach() {
@@ -73,7 +73,7 @@ class DomainTransferRejectFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void setUp() {
@@ -126,7 +126,7 @@ class DomainTransferRequestFlowTest

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  private static final ImmutableMap<String, String> BASE_FEE_MAP =
      new ImmutableMap.Builder<String, String>()
@@ -120,7 +120,7 @@ class DomainUpdateFlowTest extends ResourceFlowTestCase<DomainUpdateFlow, Domain

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  @BeforeEach
  void initDomainTest() {
@@ -36,7 +36,7 @@ class HostCheckFlowTest extends ResourceCheckFlowTestCase<HostCheckFlow, HostRes

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  HostCheckFlowTest() {
    setEppInput("host_check.xml");
@@ -67,7 +67,7 @@ class HostCreateFlowTest extends ResourceFlowTestCase<HostCreateFlow, HostResour

  @Order(value = Order.DEFAULT - 2)
  @RegisterExtension
-  final ReplayExtension replayExtension = ReplayExtension.createWithCompare(clock);
+  final ReplayExtension replayExtension = ReplayExtension.createWithDoubleReplay(clock);

  private void setEppHostCreateInput(String hostName, String hostAddrs) {
    setEppInput(
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
gbrodman	054c0625a8	Add SQL functionality to DeleteProberDataAction (#1218 ) This includes a change to how the JPA transaction manager handles existence and load checks for entities with compound IDs. Previously, we relied on the fields all being named the same in the ID entity and the parent entity. This didn't work for History objects (e.g. DomainHistory) so existence checks were broken. Now, we use the methods the same way that Hibernate does (if possible). Note as well that there's a bit of semi-duplicated logic in DeleteProberDataAction (between the mapper and the SQL logic). The mapper code will be deleted once we've shifted to SQL, and for now it's better to keep it in place for logging purposes.	2021-08-27 21:09:08 -04:00
gbrodman	b03639d7fc	Implement read-only transaction manager modes for R3.0 migration (#1241 ) This involves: - Altering both transaction managers to check for a read-only mode at the start of standard write actions (e.g. delete, put). - Altering both raw layers (entity manager, ofy) to throw exceptions on write actions as well - Implementing bypass routes for reading / setting / removing the schedule itself so that we don't get "stuck"	2021-08-27 15:59:16 -04:00
Rachel Guan	bd9af0de84	Improve logging for SendExpiringCertificateNotificationEmailAction.java (#1302 ) * Improve logging for SendExpiringCertificateNotificationEmailAction.java	2021-08-27 13:11:54 -04:00
gbrodman	ae911a5280	Fix semantic merge conflict accidentally introduced (#1301 )	2021-08-26 16:15:56 -04:00
gbrodman	d57597f40f	Clean up ReplicateToDatastoreAction and tests (#1299 ) * Clean up ReplicateToDatastoreAction and tests 1. applyTransaction should throw an error if it fails; this allows us to have more information in the caller (and it shouldn't usually happen) 2. Set a response code + payload now, since this is an action that is called by cron 3. Add a method to the test log subject that allows us to check if a severe log with a particular Throwable cause was logged (since the cause isn't contained in the log message itself directly)	2021-08-25 14:45:05 -06:00
gbrodman	2641d0d462	Save indexes when replaying EppResources SQL->DS (#1300 ) * Save indexes when replaying EppResources SQL->DS We implement this similarly to how we implement the beforeSqlSaveOnReplay callback in the other direction -- a beforeDatastoreSaveOnReplay method that is called when replaying a Mutation to Datastore. This means that the asynchronous replay will create the relevant ForeignKeyIndex and EppResourceIndex objects for EppResources saved when SQL is primary.	2021-08-25 14:44:44 -06:00
sarahcaseybot	5b41f0b9b6	Remove ClaimsList from Datastore Schema (#1298 ) * Remove ClaimsList from Datastore schema * Remove some Datastore references * Remove unnecessary annotations	2021-08-25 11:58:44 -04:00
Lai Jiang	1a26677d72	Implement a util class to manage push queues using Cloud Tasks API (#1290 ) * Implement a util class to manage push queues using Cloud Tasks API Push queues were part of App Engine when they debuted. As a result the Task Queue API were part of the App Engine SDK and can only be used in App Engine classic runtime. The new Cloud Tasks API can be used in any runtime but it only supports push queues. In this PR we implement a util class (CloudTasksUtils) like TaskQueueUtils to handle enqueuing tasks to push queues using Cloud Tasks. One action (TldFanoutAction) was converted to use the new API as a demo. Mass migration of other call sites of the old API will follow in a separate PR. TESTED=deployed to alpha and verified that tasks are corrected enqueued and executed.	2021-08-24 21:13:54 -04:00
gbrodman	f1beeb4016	Add double-replay to remaining existing ReplayExtension calls (#1297 ) The only other change is that we need to reconstitute serverApproveEntities for DomainTransferData in more situations (to fill out the ofy keys)	2021-08-23 15:08:09 -04:00
gbrodman	5c33286056	Compare SQL and Datastore objects in SQL->DS replay testing (#1291 ) Add double-replay to the Host*Flow tests to show how this works. The only change to the double replay itself is that now we store the Datastore entity in the TransactionEntity object -- this is because we use Objectify to serialize the objects into bytes and we need it to know about the entity in question.	2021-08-23 11:05:14 -04:00
gbrodman	603a95d719	Add DS->SQL replay cron job to production (#1292 ) * Add DS->SQL replay cron job to production This won't do anything until we set the migration schedule to DATASTORE_PRIMARY. Actions in order: 1. Add this cron job (it'll be a no-op) 2. Run the init-sql-pipeline to populate production's SQL DB 3. Set the SqlReplayCheckpoint to a time before the smear backup that was used in step #1 (maybe 30 minutes) 4. Set the database migration schedule to transition to DATASTORE_PRIMARY at some point	2021-08-23 07:59:51 -06:00
gbrodman	0a3774d3f7	Add withDsAndCloudSql to flow test (#1293 ) * Add withDsAndCloudSql to flow test Not sure why this wasn't failing before	2021-08-20 09:07:38 -06:00
Rachel Guan	cc60b27dd3	Add sending notification email mechanism for expiring certificates (#1179 ) * Resolve rebase conflict * Fix and imporove based on feedback.	2021-08-19 12:49:45 -04:00
Rachel Guan	52c18f9967	Remove files that are not longer used for create/update premium list (#1288 ) * Remove files that are not longer used for create/update premium list * Remove comments/notes related to create/update premium list action files	2021-08-18 14:04:57 -04:00
gbrodman	5339b3cb6c	Remove -- from crash cron comment (#1289 ) This is causing the release build to fail, see https://pantheon.corp.google.com/cloud-build/builds;region=global/22ec980b-c2b6-43fe-994a-aa98c0dbc9d4?project=domain-registry-dev	2021-08-18 11:30:01 -04:00
sarahcaseybot	d18dab3327	Remove ReservedList from Datastore schema (#1285 ) * Remove ReservedList from Datastore schema * Remove some Datastore references * Add a different non-replicated entity to ReplayCommitLogsToSqlActionTest	2021-08-17 16:56:00 -04:00