Ignore version UIDs during txn deserialization (#1607 )

* Ignore version UIDs during txn deserialization When deserializing transactions for replay to datastore, ignore class version UIDs that don't match those of the local classes and just use the local class descriptors instead. This is a simple solution for the problem of persisted VKeys containing references to classes where the class has been updated and the serial version UID has changed. Also add a "replay_txns" command that replays the transactions from a given start point so we can verify all transactions are deserializable. TESTED: Ran replay_txns against all transactions on sandbox beginning with transaction id 1828385, which includes Recurring billing events containing both the old and the new serial version UIDs.
Convert more Guava caches to Caffeine (#1603 )
2026-08-14 11:16:14 +00:00 · 2022-04-27 15:40:27 -04:00 · 2022-04-26 11:26:51 -04:00 · 2022-04-25 17:08:58 -04:00 · 2022-04-25 12:39:16 -04:00 · 2022-04-22 16:33:18 -04:00
269 changed files with 5746 additions and 3187 deletions
@@ -36,3 +36,4 @@ Shicong Huang <shicong@google.com>
 Gustav Brodman <gbrodman@google.com>
 Sarah Botwinick <sarahbot@google.com>
 Legina Chen <legina@google.com>
+Rachel Guan <rachelguan@google.com>
@@ -331,6 +331,13 @@ subprojects {

  apply from: "${rootDir.path}/java_common.gradle"

+  if (project.name != 'docs') {
+    compileJava {
+      // TODO: Remove this once we migrate off AppEngine.
+      options.release = 8
+    }
+  }
+
  if (project.name == 'third_party') return

  project.tasks.test.dependsOn runPresubmits
@@ -51,6 +51,7 @@ artifacts {
 dependencies {
  def deps = rootProject.dependencyMap

+  compile deps['com.github.ben-manes.caffeine:caffeine']
  compile deps['com.google.code.findbugs:jsr305']
  compile deps['com.google.guava:guava']
  compile deps['javax.inject:javax.inject']
@@ -1,12 +1,13 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.3
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-org.checkerframework:checker-qual:3.8.0
+org.checkerframework:checker-qual:3.19.0
@@ -1,12 +1,13 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.3
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-org.checkerframework:checker-qual:3.8.0
+org.checkerframework:checker-qual:3.19.0
@@ -1,12 +1,13 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.3
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-org.checkerframework:checker-qual:3.8.0
+org.checkerframework:checker-qual:3.19.0
@@ -1,12 +1,13 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.3
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-org.checkerframework:checker-qual:3.8.0
+org.checkerframework:checker-qual:3.19.0
@@ -1,12 +1,13 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.3
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-org.checkerframework:checker-qual:3.8.0
+org.checkerframework:checker-qual:3.19.0
@@ -1,12 +1,13 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.3
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-org.checkerframework:checker-qual:3.8.0
+org.checkerframework:checker-qual:3.19.0
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
@@ -16,7 +17,7 @@ joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
@@ -16,7 +17,7 @@ joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger-system-backend:0.7.4
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
@@ -17,7 +18,7 @@ joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger-system-backend:0.7.4
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
@@ -17,7 +18,7 @@ joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
@@ -15,6 +16,6 @@ javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.ow2.asm:asm:9.0
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:30.1.1-jre
@@ -15,6 +16,6 @@ javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.ow2.asm:asm:9.0
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger-system-backend:0.7.4
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
@@ -16,6 +17,6 @@ javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.ow2.asm:asm:9.0
@@ -1,9 +1,10 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.google.auto.value:auto-value-annotations:1.7.4
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotations:2.5.1
+com.google.errorprone:error_prone_annotations:2.10.0
 com.google.flogger:flogger-system-backend:0.7.4
 com.google.flogger:flogger:0.7.4
 com.google.guava:failureaccess:1.0.1
@@ -16,6 +17,6 @@ javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
 junit:junit:4.13.1
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:3.9.1
+org.checkerframework:checker-qual:3.19.0
 org.hamcrest:hamcrest-core:1.3
 org.ow2.asm:asm:9.0
@@ -14,9 +14,9 @@

 package google.registry.testing;

-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
+import com.github.benmanes.caffeine.cache.CacheLoader;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
 import com.google.common.flogger.FluentLogger;
 import java.io.IOException;
 import java.util.concurrent.ExecutionException;
@@ -29,7 +29,7 @@ public final class SystemInfo {
  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

  private static final LoadingCache<String, Boolean> hasCommandCache =
-      CacheBuilder.newBuilder()
+      Caffeine.newBuilder()
          .build(
              new CacheLoader<String, Boolean>() {
                @Override
@@ -172,6 +172,7 @@ dependencies {
  testRuntime files(sourceSets.test.resources.srcDirs)

  compile deps['com.beust:jcommander']
+  compile deps['com.github.ben-manes.caffeine:caffeine']
  compile deps['com.google.api:gax']
  compile deps['com.google.api.grpc:proto-google-cloud-datastore-v1']
  compile deps['com.google.api.grpc:proto-google-common-protos']
@@ -799,6 +800,11 @@ if (environment == 'alpha') {
              mainClass: 'google.registry.beam.comparedb.ValidateDatabasePipeline',
              metaData: 'google/registry/beam/validate_database_pipeline_metadata.json'
          ],
+      resaveAllEppResources:
+          [
+              mainClass: 'google.registry.beam.resave.ResaveAllEppResourcesPipeline',
+              metaData: 'google/registry/beam/resave_all_epp_resources_pipeline_metadata.json'
+          ],
  ]
  project.tasks.create("stageBeamPipelines") {
    doLast {
@@ -13,6 +13,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -13,6 +13,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -13,6 +13,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -13,6 +13,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -13,6 +13,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -13,6 +13,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -17,6 +17,7 @@ com.fasterxml.jackson.core:jackson-databind:2.13.0
 com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.13.0
 com.fasterxml.jackson:jackson-bom:2.13.0
 com.fasterxml:classmate:1.5.1
+com.github.ben-manes.caffeine:caffeine:2.9.3
 com.github.docker-java:docker-java-api:3.2.7
 com.github.docker-java:docker-java-transport-zerodep:3.2.7
 com.github.docker-java:docker-java-transport:3.2.7
@@ -112,6 +112,12 @@ public class BatchModule {
        req, ExpandRecurringBillingEventsAction.PARAM_CURSOR_TIME);
  }

+  @Provides
+  @Parameter(ResaveAllEppResourcesPipelineAction.PARAM_FAST)
+  static Optional<Boolean> provideIsFast(HttpServletRequest req) {
+    return extractOptionalBooleanParameter(req, ResaveAllEppResourcesPipelineAction.PARAM_FAST);
+  }
+
  @Provides
  @Named(QUEUE_ASYNC_ACTIONS)
  static Queue provideAsyncActionsPushQueue() {
@@ -0,0 +1,129 @@
+// Copyright 2022 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.batch;
+
+import static google.registry.beam.BeamUtils.createJobName;
+import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
+import static javax.servlet.http.HttpServletResponse.SC_OK;
+
+import com.google.api.services.dataflow.Dataflow;
+import com.google.api.services.dataflow.model.LaunchFlexTemplateParameter;
+import com.google.api.services.dataflow.model.LaunchFlexTemplateRequest;
+import com.google.api.services.dataflow.model.LaunchFlexTemplateResponse;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.flogger.FluentLogger;
+import com.google.common.net.MediaType;
+import google.registry.config.RegistryConfig.Config;
+import google.registry.config.RegistryEnvironment;
+import google.registry.request.Action;
+import google.registry.request.Parameter;
+import google.registry.request.Response;
+import google.registry.request.auth.Auth;
+import google.registry.util.Clock;
+import java.util.Optional;
+import javax.inject.Inject;
+
+/**
+ * Starts a Dataflow pipeline that resaves all EPP resources projected to the current time.
+ *
+ * <p>This is useful for a few situations. First, as a fallback option for resource transfers that
+ * have expired pending transfers (this will resolve them), just in case the enqueued action fails.
+ * Second, it will reflect domain autorenews that have happened. Third, it will remove any expired
+ * grace periods.
+ *
+ * <p>There's also the general principle that it's good to have the data in the database remain as
+ * current as is reasonably possible.
+ *
+ * <p>If the <code>?isFast=true</code> query string parameter is passed as true, the pipeline will
+ * only attempt to load, project, and resave entities where we expect one of the previous situations
+ * has occurred. Otherwise, we will load, project, and resave all EPP resources.
+ *
+ * <p>This runs the {@link google.registry.beam.resave.ResaveAllEppResourcesPipeline}.
+ */
+@Action(
+    service = Action.Service.BACKEND,
+    path = ResaveAllEppResourcesPipelineAction.PATH,
+    auth = Auth.AUTH_INTERNAL_OR_ADMIN)
+public class ResaveAllEppResourcesPipelineAction implements Runnable {
+
+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
+  static final String PATH = "/_dr/task/resaveAllEppResourcesPipeline";
+  static final String PIPELINE_NAME = "resave_all_epp_resources_pipeline";
+
+  public static final String PARAM_FAST = "fast";
+
+  private final String projectId;
+  private final String jobRegion;
+  private final String stagingBucketUrl;
+  private final boolean fast;
+  private final Clock clock;
+  private final Response response;
+  private final Dataflow dataflow;
+
+  @Inject
+  ResaveAllEppResourcesPipelineAction(
+      @Config("projectId") String projectId,
+      @Config("defaultJobRegion") String jobRegion,
+      @Config("beamStagingBucketUrl") String stagingBucketUrl,
+      @Parameter(PARAM_FAST) Optional<Boolean> fast,
+      Clock clock,
+      Response response,
+      Dataflow dataflow) {
+    this.projectId = projectId;
+    this.jobRegion = jobRegion;
+    this.stagingBucketUrl = stagingBucketUrl;
+    this.fast = fast.orElse(false);
+    this.clock = clock;
+    this.response = response;
+    this.dataflow = dataflow;
+  }
+
+  @Override
+  public void run() {
+    response.setContentType(MediaType.PLAIN_TEXT_UTF_8);
+    logger.atInfo().log("Launching ResaveAllEppResourcesPipeline");
+    try {
+      LaunchFlexTemplateParameter parameter =
+          new LaunchFlexTemplateParameter()
+              .setJobName(createJobName("resave-all-epp-resources", clock))
+              .setContainerSpecGcsPath(
+                  String.format("%s/%s_metadata.json", stagingBucketUrl, PIPELINE_NAME))
+              .setParameters(
+                  new ImmutableMap.Builder<String, String>()
+                      .put(PARAM_FAST, Boolean.toString(fast))
+                      .put("registryEnvironment", RegistryEnvironment.get().name())
+                      .build());
+      LaunchFlexTemplateResponse launchResponse =
+          dataflow
+              .projects()
+              .locations()
+              .flexTemplates()
+              .launch(
+                  projectId,
+                  jobRegion,
+                  new LaunchFlexTemplateRequest().setLaunchParameter(parameter))
+              .execute();
+      logger.atInfo().log("Got response: %s", launchResponse.getJob().toPrettyString());
+      String jobId = launchResponse.getJob().getId();
+      response.setStatus(SC_OK);
+      response.setPayload(String.format("Launched resaveAllEppResources pipeline: %s", jobId));
+    } catch (Exception e) {
+      logger.atSevere().withCause(e).log("Template Launch failed.");
+      response.setStatus(SC_INTERNAL_SERVER_ERROR);
+      response.setPayload(String.format("Pipeline launch failed: %s", e.getMessage()));
+    }
+  }
+}
@@ -23,6 +23,7 @@ import com.google.common.collect.Streams;
 import google.registry.beam.common.RegistryQuery.CriteriaQuerySupplier;
 import google.registry.model.UpdateAutoTimestamp;
 import google.registry.model.UpdateAutoTimestamp.DisableAutoUpdateResource;
+import google.registry.model.common.DatabaseMigrationStateSchedule;
 import google.registry.model.replay.SqlEntity;
 import google.registry.persistence.transaction.JpaTransactionManager;
 import google.registry.persistence.transaction.TransactionManagerFactory;
@@ -234,6 +235,10 @@ public final class RegistryJpaIO {

      @ProcessElement
      public void processElement(OutputReceiver<T> outputReceiver) {
+        // Preload the migration schedule into cache, otherwise the cache loading query may happen
+        // before the setDatabaseSnapshot call in the transaction below, causing it to fail.
+        DatabaseMigrationStateSchedule.get();
+
        jpaTm()
            .transactNoRetry(
                () -> {
@@ -158,7 +158,7 @@ public class InitSqlPipeline implements Serializable {
                    .addAll(toKindStrings(PHASE_TWO_ORDERED))
                    .build()));

-    // Set up the pipeline to write entity kinds from PHASE_ONE_ORDERED to SQL. Return a object
+    // Set up the pipeline to write entity kinds from PHASE_ONE_ORDERED to SQL. Return an object
    // that signals the completion of the phase.
    PCollection<Void> blocker =
        scheduleOnePhaseWrites(datastoreSnapshot, PHASE_ONE_ORDERED, Optional.empty(), null);
@@ -22,7 +22,7 @@ import static google.registry.beam.initsql.BackupPaths.getExportFilePatterns;
 import static google.registry.model.ofy.ObjectifyService.auditedOfy;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
 import static google.registry.util.DateTimeUtils.isBeforeOrAt;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static java.util.Comparator.comparing;
 import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
 import static org.apache.beam.sdk.values.TypeDescriptors.strings;
@@ -343,11 +343,11 @@ public final class Transforms {
      // Canonicalize old domain/host names from 2016 and earlier before we were enforcing this.
      entity.setIndexedProperty(
          "fullyQualifiedDomainName",
-          canonicalizeDomainName((String) entity.getProperty("fullyQualifiedDomainName")));
+          canonicalizeHostname((String) entity.getProperty("fullyQualifiedDomainName")));
    } else if (entity.getKind().equals("HostResource")) {
      entity.setIndexedProperty(
          "fullyQualifiedHostName",
-          canonicalizeDomainName((String) entity.getProperty("fullyQualifiedHostName")));
+          canonicalizeHostname((String) entity.getProperty("fullyQualifiedHostName")));
    }
    return entity;
  }
@@ -18,6 +18,7 @@ import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.beam.BeamUtils.getQueryFromFile;
 import static org.apache.beam.sdk.values.TypeDescriptors.strings;

+import com.google.common.flogger.FluentLogger;
 import google.registry.beam.common.RegistryJpaIO;
 import google.registry.beam.common.RegistryJpaIO.Read;
 import google.registry.beam.invoicing.BillingEvent.InvoiceGroupingKey;
@@ -35,6 +36,7 @@ import java.time.LocalTime;
 import java.time.YearMonth;
 import java.time.ZoneId;
 import java.time.format.DateTimeFormatter;
+import java.util.Objects;
 import java.util.Optional;
 import java.util.regex.Pattern;
 import org.apache.beam.sdk.Pipeline;
@@ -53,6 +55,7 @@ import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TypeDescriptor;
+import org.joda.money.CurrencyUnit;

 /**
 * Definition of a Dataflow Flex pipeline template, which generates a given month's invoices.
@@ -73,6 +76,8 @@ public class InvoicingPipeline implements Serializable {
  private static final Pattern SQL_COMMENT_REGEX =
      Pattern.compile("^\\s*--.*\\n", Pattern.MULTILINE);

+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
  private final InvoicingPipelineOptions options;

  InvoicingPipeline(InvoicingPipelineOptions options) {
@@ -113,31 +118,44 @@ public class InvoicingPipeline implements Serializable {
      InvoicingPipelineOptions options, Pipeline pipeline) {
    Read<Object[], BillingEvent> read =
        RegistryJpaIO.read(
-            makeCloudSqlQuery(options.getYearMonth()), false, InvoicingPipeline::parseRow);
+            makeCloudSqlQuery(options.getYearMonth()), false, row -> parseRow(row).orElse(null));

-    return pipeline.apply("Read BillingEvents from Cloud SQL", read);
+    PCollection<BillingEvent> billingEventsWithNulls =
+        pipeline.apply("Read BillingEvents from Cloud SQL", read);
+
+    // Remove null billing events
+    return billingEventsWithNulls.apply(Filter.by(Objects::nonNull));
  }

-  private static BillingEvent parseRow(Object[] row) {
+  private static Optional<BillingEvent> parseRow(Object[] row) {
    google.registry.model.billing.BillingEvent.OneTime oneTime =
        (google.registry.model.billing.BillingEvent.OneTime) row[0];
    Registrar registrar = (Registrar) row[1];
-    return BillingEvent.create(
-        oneTime.getId(),
-        DateTimeUtils.toZonedDateTime(oneTime.getBillingTime(), ZoneId.of("UTC")),
-        DateTimeUtils.toZonedDateTime(oneTime.getEventTime(), ZoneId.of("UTC")),
-        registrar.getRegistrarId(),
-        registrar.getBillingIdentifier().toString(),
-        registrar.getPoNumber().orElse(""),
-        DomainNameUtils.getTldFromDomainName(oneTime.getTargetId()),
-        oneTime.getReason().toString(),
-        oneTime.getTargetId(),
-        oneTime.getDomainRepoId(),
-        Optional.ofNullable(oneTime.getPeriodYears()).orElse(0),
-        oneTime.getCost().getCurrencyUnit().toString(),
-        oneTime.getCost().getAmount().doubleValue(),
-        String.join(
-            " ", oneTime.getFlags().stream().map(Flag::toString).collect(toImmutableSet())));
+    CurrencyUnit currency = oneTime.getCost().getCurrencyUnit();
+    if (!registrar.getBillingAccountMap().containsKey(currency)) {
+      logger.atSevere().log(
+          "Registrar %s does not have a product account key for the currency unit: %s",
+          registrar.getRegistrarId(), currency);
+      return Optional.empty();
+    }
+
+    return Optional.of(
+        BillingEvent.create(
+            oneTime.getId(),
+            DateTimeUtils.toZonedDateTime(oneTime.getBillingTime(), ZoneId.of("UTC")),
+            DateTimeUtils.toZonedDateTime(oneTime.getEventTime(), ZoneId.of("UTC")),
+            registrar.getRegistrarId(),
+            registrar.getBillingAccountMap().get(currency),
+            registrar.getPoNumber().orElse(""),
+            DomainNameUtils.getTldFromDomainName(oneTime.getTargetId()),
+            oneTime.getReason().toString(),
+            oneTime.getTargetId(),
+            oneTime.getDomainRepoId(),
+            Optional.ofNullable(oneTime.getPeriodYears()).orElse(0),
+            oneTime.getCost().getCurrencyUnit().toString(),
+            oneTime.getCost().getAmount().doubleValue(),
+            String.join(
+                " ", oneTime.getFlags().stream().map(Flag::toString).collect(toImmutableSet()))));
  }

  /** Transform that converts a {@code BillingEvent} into an invoice CSV row. */
@@ -15,6 +15,7 @@
 package google.registry.beam.rde;

 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.beam.rde.RdePipeline.TupleTags.DOMAIN_FRAGMENTS;
 import static google.registry.beam.rde.RdePipeline.TupleTags.EXTERNAL_HOST_FRAGMENTS;
@@ -28,10 +29,12 @@ import static google.registry.model.reporting.HistoryEntryDao.RESOURCE_TYPES_TO_
 import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
 import static org.apache.beam.sdk.values.TypeDescriptors.kvs;

+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Sets;
 import com.google.common.collect.Streams;
+import com.google.common.flogger.FluentLogger;
 import com.google.common.io.BaseEncoding;
 import dagger.BindsInstance;
 import dagger.Component;
@@ -198,6 +201,8 @@ public class RdePipeline implements Serializable {
          HostHistory.class,
          "hostBase");

+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
  @Inject
  RdePipeline(RdePipelineOptions options, GcsUtils gcsUtils, CloudTasksUtils cloudTasksUtils) {
    this.options = options;
@@ -357,6 +362,33 @@ public class RdePipeline implements Serializable {
        .setCoder(KvCoder.of(StringUtf8Coder.of(), VarLongCoder.of()));
  }

+  private <T extends HistoryEntry> EppResource loadResourceByHistoryEntryId(
+      Class<T> historyEntryClazz, String repoId, Iterable<Long> revisionIds) {
+    ImmutableList<Long> ids = ImmutableList.copyOf(revisionIds);
+    // The size should always be 1 because we are only getting one repo ID -> revision ID pair per
+    // repo ID from the source transform (the JPA query in the method above). But for some reason
+    // after CoGroupByKey (joining the revision IDs and the pending deposits on repo IDs), in
+    // #removedUnreferencedResources, duplicate revision IDs are sometimes introduced. Here we
+    // attempt to deduplicate the iterable. If it contains multiple revision IDs that are NOT the
+    // same, we have a more serious problem as we cannot be sure which one to use. We should use the
+    // highest revision ID, but we don't even know where it comes from, as the query should
+    // definitively only give us one revision ID per repo ID. In this case we have to abort and
+    // require manual intervention.
+    if (ids.size() != 1) {
+      ImmutableSet<Long> dedupedIds = ImmutableSet.copyOf(ids);
+      checkState(
+          dedupedIds.size() == 1,
+          "Multiple unique revision IDs detected for %s repo ID %s: %s",
+          EPP_RESOURCE_FIELD_NAME.get(historyEntryClazz),
+          repoId,
+          ids);
+      logger.atSevere().log(
+          "Duplicate revision IDs detected for %s repo ID %s: %s",
+          EPP_RESOURCE_FIELD_NAME.get(historyEntryClazz), repoId, ids);
+    }
+    return loadResourceByHistoryEntryId(historyEntryClazz, repoId, ids.get(0));
+  }
+
  private <T extends HistoryEntry> EppResource loadResourceByHistoryEntryId(
      Class<T> historyEntryClazz, String repoId, long revisionId) {
    try {
@@ -516,7 +548,7 @@ public class RdePipeline implements Serializable {
                              loadResourceByHistoryEntryId(
                                  ContactHistory.class,
                                  kv.getKey(),
-                                  kv.getValue().getOnly(REVISION_ID));
+                                  kv.getValue().getAll(REVISION_ID));
                      DepositFragment fragment = marshaller.marshalContact(contact);
                      ImmutableSet<KV<PendingDeposit, DepositFragment>> fragments =
                          Streams.stream(kv.getValue().getAll(PENDING_DEPOSIT))
@@ -549,8 +581,8 @@ public class RdePipeline implements Serializable {
                                loadResourceByHistoryEntryId(
                                    HostHistory.class,
                                    kv.getKey(),
-                                    kv.getValue().getOnly(REVISION_ID));
-                        // When a host is subordinate, we need to find it's superordinate domain and
+                                    kv.getValue().getAll(REVISION_ID));
+                        // When a host is subordinate, we need to find its superordinate domain and
                        // include it in the deposit as well.
                        if (host.isSubordinate()) {
                          subordinateHostCounter.inc();
@@ -627,7 +659,7 @@ public class RdePipeline implements Serializable {
                              loadResourceByHistoryEntryId(
                                  DomainHistory.class,
                                  kv.getKey(),
-                                  kv.getValue().getOnly(REVISION_ID));
+                                  kv.getValue().getAll(REVISION_ID));
                      ImmutableSet.Builder<KV<PendingDeposit, DepositFragment>> results =
                          new ImmutableSet.Builder<>();
                      for (KV<String, CoGbkResult> hostToPendingDeposits :
@@ -637,7 +669,7 @@ public class RdePipeline implements Serializable {
                                loadResourceByHistoryEntryId(
                                    HostHistory.class,
                                    hostToPendingDeposits.getKey(),
-                                    hostToPendingDeposits.getValue().getOnly(REVISION_ID));
+                                    hostToPendingDeposits.getValue().getAll(REVISION_ID));
                        DepositFragment fragment =
                            marshaller.marshalSubordinateHost(host, superordinateDomain);
                        Streams.stream(hostToPendingDeposits.getValue().getAll(PENDING_DEPOSIT))
@@ -696,6 +728,7 @@ public class RdePipeline implements Serializable {
   * CoGbkResult}s are used.
   */
  protected abstract static class TupleTags {
+
    protected static final TupleTag<KV<PendingDeposit, DepositFragment>> DOMAIN_FRAGMENTS =
        new TupleTag<KV<PendingDeposit, DepositFragment>>() {};

@@ -729,10 +762,12 @@ public class RdePipeline implements Serializable {
        UtilsModule.class
      })
  interface RdePipelineComponent {
+
    RdePipeline rdePipeline();

    @Component.Builder
    interface Builder {
+
      @BindsInstance
      Builder options(RdePipelineOptions options);

@@ -0,0 +1,174 @@
+// Copyright 2022 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.beam.resave;
+
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import google.registry.beam.common.RegistryJpaIO;
+import google.registry.beam.common.RegistryJpaIO.Read;
+import google.registry.model.EppResource;
+import google.registry.model.contact.ContactResource;
+import google.registry.model.domain.DomainBase;
+import google.registry.model.host.HostResource;
+import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
+import google.registry.persistence.transaction.CriteriaQueryBuilder;
+import google.registry.util.DateTimeUtils;
+import java.io.Serializable;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupIntoBatches;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.util.ShardedKey;
+import org.apache.beam.sdk.values.KV;
+import org.joda.time.DateTime;
+
+/**
+ * A Dataflow Flex pipeline that resaves changed EPP resources in SQL.
+ *
+ * <p>Due to the way that Hibernate works, if an entity is unchanged by {@link
+ * EppResource#cloneProjectedAtTime(DateTime)} it will not actually be re-persisted to the database.
+ * Thus, the only actual changes occur when objects are changed by projecting them to now, such as
+ * when a pending transfer is resolved.
+ */
+public class ResaveAllEppResourcesPipeline implements Serializable {
+
+  private static final ImmutableSet<Class<? extends EppResource>> EPP_RESOURCE_CLASSES =
+      ImmutableSet.of(ContactResource.class, DomainBase.class, HostResource.class);
+
+  /**
+   * There exist three possible situations where we know we'll want to project domains to the
+   * current point in time:
+   *
+   * <ul>
+   *   <li>A pending domain transfer has expired.
+   *   <li>A domain is past its expiration time without being deleted (this means it autorenewed).
+   *   <li>A domain has expired grace periods.
+   * </ul>
+   *
+   * <p>This command contains all three scenarios so that we can avoid querying the Domain table
+   * multiple times, and to avoid projecting and resaving the same domain multiple times.
+   */
+  private static final String DOMAINS_TO_PROJECT_QUERY =
+      "FROM Domain d WHERE (d.transferData.transferStatus = 'PENDING' AND"
+          + " d.transferData.pendingTransferExpirationTime < current_timestamp()) OR"
+          + " (d.registrationExpirationTime < current_timestamp() AND d.deletionTime ="
+          + " (:END_OF_TIME)) OR (EXISTS (SELECT 1 FROM GracePeriod gp WHERE gp.domainRepoId ="
+          + " d.repoId AND gp.expirationTime < current_timestamp()))";
+
+  private final ResaveAllEppResourcesPipelineOptions options;
+
+  ResaveAllEppResourcesPipeline(ResaveAllEppResourcesPipelineOptions options) {
+    this.options = options;
+  }
+
+  PipelineResult run() {
+    Pipeline pipeline = Pipeline.create(options);
+    setupPipeline(pipeline);
+    return pipeline.run();
+  }
+
+  void setupPipeline(Pipeline pipeline) {
+    options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_COMMITTED);
+    if (options.getFast()) {
+      fastResaveContacts(pipeline);
+      fastResaveDomains(pipeline);
+    } else {
+      EPP_RESOURCE_CLASSES.forEach(clazz -> forceResaveAllResources(pipeline, clazz));
+    }
+  }
+
+  /** Projects to the current time and saves any contacts with expired transfers. */
+  private void fastResaveContacts(Pipeline pipeline) {
+    Read<ContactResource, ContactResource> read =
+        RegistryJpaIO.read(
+            "FROM Contact WHERE transferData.transferStatus = 'PENDING' AND"
+                + " transferData.pendingTransferExpirationTime < current_timestamp()",
+            ContactResource.class,
+            c -> c);
+    projectAndResaveResources(pipeline, ContactResource.class, read);
+  }
+
+  /**
+   * Projects to the current time and saves any domains with expired pending actions (e.g.
+   * transfers, grace periods).
+   *
+   * <p>The logic of what might have changed is paraphrased from {@link
+   * google.registry.model.domain.DomainContent#cloneProjectedAtTime(DateTime)}.
+   */
+  private void fastResaveDomains(Pipeline pipeline) {
+    Read<DomainBase, DomainBase> read =
+        RegistryJpaIO.read(
+            DOMAINS_TO_PROJECT_QUERY,
+            ImmutableMap.of("END_OF_TIME", DateTimeUtils.END_OF_TIME),
+            DomainBase.class,
+            d -> d);
+    projectAndResaveResources(pipeline, DomainBase.class, read);
+  }
+
+  /** Projects all resources to the current time and saves them. */
+  private <T extends EppResource> void forceResaveAllResources(Pipeline pipeline, Class<T> clazz) {
+    Read<T, T> read = RegistryJpaIO.read(() -> CriteriaQueryBuilder.create(clazz).build());
+    projectAndResaveResources(pipeline, clazz, read);
+  }
+
+  /** Projects and re-saves the result of the provided {@link Read}. */
+  private <T extends EppResource> void projectAndResaveResources(
+      Pipeline pipeline, Class<T> clazz, Read<?, T> read) {
+    int numShards = options.getSqlWriteShards();
+    int batchSize = options.getSqlWriteBatchSize();
+    String className = clazz.getSimpleName();
+    pipeline
+        .apply("Read " + className, read)
+        .apply(
+            "Shard data for class" + className,
+            WithKeys.<Integer, T>of(e -> ThreadLocalRandom.current().nextInt(numShards))
+                .withKeyType(integers()))
+        .apply(
+            "Group into batches for class" + className,
+            GroupIntoBatches.<Integer, T>ofSize(batchSize).withShardedKey())
+        .apply("Map " + className + " to now", ParDo.of(new BatchedProjectionFunction<>()))
+        .apply(
+            "Write transformed " + className,
+            RegistryJpaIO.<EppResource>write()
+                .withName("Write transformed " + className)
+                .withBatchSize(batchSize)
+                .withShards(numShards));
+  }
+
+  private static class BatchedProjectionFunction<T extends EppResource>
+      extends DoFn<KV<ShardedKey<Integer>, Iterable<T>>, EppResource> {
+
+    @ProcessElement
+    public void processElement(
+        @Element KV<ShardedKey<Integer>, Iterable<T>> element,
+        OutputReceiver<EppResource> outputReceiver) {
+      jpaTm()
+          .transact(
+              () ->
+                  element
+                      .getValue()
+                      .forEach(
+                          resource ->
+                              outputReceiver.output(
+                                  resource.cloneProjectedAtTime(jpaTm().getTransactionTime()))));
+    }
+  }
+}
@@ -0,0 +1,26 @@
+// Copyright 2022 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.beam.resave;
+
+import google.registry.beam.common.RegistryPipelineOptions;
+import org.apache.beam.sdk.options.Description;
+
+public interface ResaveAllEppResourcesPipelineOptions extends RegistryPipelineOptions {
+
+  @Description("True if we should attempt to run only over potentially out-of-date EPP resources")
+  boolean getFast();
+
+  void setFast(boolean fast);
+}
@@ -576,6 +576,19 @@ public final class RegistryConfig {
      return config.beam.highPerformanceMachineType;
    }

+    /**
+     * Returns initial number of workers used for a Beam pipeline. Autoscaling can still in effect.
+     *
+     * @see <a
+     *     href=https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#horizontal-autoscaling>
+     *     Horizontal Autoscaling </a>
+     */
+    @Provides
+    @Config("initialWorkerCount")
+    public static int provideInitialWorkerCount(RegistryConfigSettings config) {
+      return config.beam.initialWorkerCount;
+    }
+
    /**
     * Returns the default job region to run Apache Beam (Cloud Dataflow) jobs in.
     *
@@ -1425,8 +1438,8 @@ public final class RegistryConfig {
  }

  /** Returns the amount of time a singleton should be cached, before expiring. */
-  public static Duration getSingletonCacheRefreshDuration() {
-    return Duration.standardSeconds(CONFIG_SETTINGS.get().caching.singletonCacheRefreshSeconds);
+  public static java.time.Duration getSingletonCacheRefreshDuration() {
+    return java.time.Duration.ofSeconds(CONFIG_SETTINGS.get().caching.singletonCacheRefreshSeconds);
  }

  /**
@@ -137,6 +137,7 @@ public class RegistryConfigSettings {
  public static class Beam {
    public String defaultJobRegion;
    public String highPerformanceMachineType;
+    public int initialWorkerCount;
    public String stagingBucketUrl;
  }

@@ -442,6 +442,11 @@ beam:
  # core count per machine may be preferable in order to preserve IP addresses.
  # See: https://cloud.google.com/compute/quotas#cpu_quota
  highPerformanceMachineType: n2-standard-4
+  # The initial number of workers requested. This can help speed up the pipeline
+  # which otherwise would take some time to spin up the necessary number of
+  # works. Autoscaling is still in effect to reduce the number of workers if
+  # not in use.
+  initialWorkerCount: 24
  stagingBucketUrl: gcs-bucket-with-staged-templates

 keyring:
@@ -316,6 +316,12 @@
    <url-pattern>/_dr/task/resaveAllEppResources</url-pattern>
  </servlet-mapping>

+  <!--  Dataflow pipeline to re-save all EPP resources. -->
+  <servlet-mapping>
+    <servlet-name>backend-servlet</servlet-name>
+    <url-pattern>/_dr/task/resaveAllEppResourcesPipeline</url-pattern>
+  </servlet-mapping>
+
  <!--  Reread all Registrar RDAP Base Urls from the ICANN endpoint. -->
  <servlet-mapping>
    <servlet-name>backend-servlet</servlet-name>
@@ -1,16 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<blacklistentries>
-
-  <!-- Example IPv4 CIDR Subnet
-  <blacklist>
-    <subnet>1.2.3.4/24</subnet>
-    <description>An IPv4 subnet</description>
-  </blacklist> -->
-
-  <!-- Example IPv6 CIDR Subnet
-  <blacklist>
-    <subnet>abcd::123:4567/48</subnet>
-    <description>An IPv6 subnet</description>
-  </blacklist> -->
-
-</blacklistentries>
@@ -1,16 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<blacklistentries>
-
-  <!-- Example IPv4 CIDR Subnet
-  <blacklist>
-    <subnet>1.2.3.4/24</subnet>
-    <description>An IPv4 subnet</description>
-  </blacklist> -->
-
-  <!-- Example IPv6 CIDR Subnet
-  <blacklist>
-    <subnet>abcd::123:4567/48</subnet>
-    <description>An IPv6 subnet</description>
-  </blacklist> -->
-
-</blacklistentries>
@@ -36,19 +36,6 @@
    <target>backend</target>
  </cron>

-  <cron>
-    <url>/_dr/task/rdeStaging?beam=true</url>
-    <description>
-      This job generates a full RDE escrow deposit as a single gigantic XML
-      document using the Beam pipeline regardless of the current TM
-      configuration and streams it to cloud storage. It does not trigger the
-      subsequent upload tasks and is meant to run parallel with the main cron
-      job in order to compare the results from both runs.
-    </description>
-    <schedule>every 8 hours from 00:07 to 20:00</schedule>
-    <target>backend</target>
-  </cron>
-
  <cron>
    <url><![CDATA[/_dr/cron/fanout?queue=rde-upload&endpoint=/_dr/task/rdeUpload&forEachRealTld]]></url>
    <description>
@@ -31,7 +31,7 @@ import static google.registry.monitoring.whitebox.CheckApiMetric.Status.UNKNOWN_
 import static google.registry.monitoring.whitebox.CheckApiMetric.Tier.PREMIUM;
 import static google.registry.monitoring.whitebox.CheckApiMetric.Tier.STANDARD;
 import static google.registry.pricing.PricingEngineProxy.isDomainPremium;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static org.json.simple.JSONValue.toJSONString;

 import com.google.common.collect.ImmutableList;
@@ -105,7 +105,7 @@ public class CheckApiAction implements Runnable {
    String domainString;
    InternetDomainName domainName;
    try {
-      domainString = canonicalizeDomainName(nullToEmpty(domain));
+      domainString = canonicalizeHostname(nullToEmpty(domain));
      domainName = validateDomainName(domainString);
    } catch (IllegalArgumentException | EppException e) {
      metricBuilder.status(INVALID_NAME);
@@ -25,6 +25,7 @@ import static google.registry.model.ResourceTransferUtils.handlePendingTransferO
 import static google.registry.model.eppoutput.Result.Code.SUCCESS;
 import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
 import static google.registry.model.transfer.TransferStatus.SERVER_CANCELLED;
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertAsyncActionsAreAllowed;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

 import com.google.common.collect.ImmutableSet;
@@ -94,6 +95,7 @@ public final class ContactDeleteFlow implements TransactionalFlow {
    extensionManager.register(MetadataExtension.class);
    validateRegistrarIsLoggedIn(registrarId);
    extensionManager.validate();
+    assertAsyncActionsAreAllowed();
    DateTime now = tm().getTransactionTime();
    checkLinkedDomains(targetId, now, ContactResource.class, DomainBase::getReferencedContacts);
    ContactResource existingContact = loadAndVerifyExistence(ContactResource.class, targetId, now);
@@ -22,6 +22,7 @@ import static google.registry.flows.ResourceFlowUtils.verifyResourceOwnership;
 import static google.registry.flows.host.HostFlowUtils.validateHostName;
 import static google.registry.model.eppoutput.Result.Code.SUCCESS;
 import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertAsyncActionsAreAllowed;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

 import com.google.common.collect.ImmutableSet;
@@ -96,6 +97,7 @@ public final class HostDeleteFlow implements TransactionalFlow {
    extensionManager.register(MetadataExtension.class);
    validateRegistrarIsLoggedIn(registrarId);
    extensionManager.validate();
+    assertAsyncActionsAreAllowed();
    DateTime now = tm().getTransactionTime();
    validateHostName(targetId);
    checkLinkedDomains(targetId, now, HostResource.class, DomainBase::getNameservers);
@@ -28,6 +28,7 @@ import static google.registry.flows.host.HostFlowUtils.verifySuperordinateDomain
 import static google.registry.flows.host.HostFlowUtils.verifySuperordinateDomainOwnership;
 import static google.registry.model.index.ForeignKeyIndex.loadAndGetKey;
 import static google.registry.model.reporting.HistoryEntry.Type.HOST_UPDATE;
+import static google.registry.persistence.transaction.TransactionManagerFactory.assertAsyncActionsAreAllowed;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.CollectionUtils.isNullOrEmpty;

@@ -136,6 +137,9 @@ public final class HostUpdateFlow implements TransactionalFlow {
    validateHostName(targetId);
    HostResource existingHost = loadAndVerifyExistence(HostResource.class, targetId, now);
    boolean isHostRename = suppliedNewHostName != null;
+    if (isHostRename) {
+      assertAsyncActionsAreAllowed();
+    }
    String oldHostName = targetId;
    String newHostName = firstNonNull(suppliedNewHostName, oldHostName);
    DomainBase oldSuperordinateDomain =
@@ -17,10 +17,10 @@ package google.registry.model;
 import static com.google.common.base.Suppliers.memoizeWithExpiration;
 import static google.registry.config.RegistryConfig.getSingletonCacheRefreshDuration;
 import static java.util.concurrent.TimeUnit.MILLISECONDS;
-import static org.joda.time.Duration.ZERO;

+import com.github.benmanes.caffeine.cache.Caffeine;
 import com.google.common.base.Supplier;
-import org.joda.time.Duration;
+import java.time.Duration;

 /** Utility methods related to caching Datastore entities. */
 public class CacheUtils {
@@ -41,8 +41,36 @@ public class CacheUtils {
   */
  public static <T> Supplier<T> tryMemoizeWithExpiration(
      Duration expiration, Supplier<T> original) {
-    return expiration.isEqual(ZERO)
+    return expiration.isZero()
        ? original
-        : memoizeWithExpiration(original, expiration.getMillis(), MILLISECONDS);
+        : memoizeWithExpiration(original, expiration.toMillis(), MILLISECONDS);
+  }
+
+  /** Creates and returns a new {@link Caffeine} builder. */
+  public static Caffeine<Object, Object> newCacheBuilder() {
+    return Caffeine.newBuilder();
+  }
+
+  /**
+   * Creates and returns a new {@link Caffeine} builder with the specified cache expiration.
+   *
+   * <p>This also sets the refresh duration to half of the cache expiration. The resultant behavior
+   * is that a cache entry is eligible to be asynchronously refreshed after access once more than
+   * half of its cache duration has elapsed, and then it is synchronously refreshed (blocking the
+   * read) once its full cache duration has elapsed. So you will never get data older than the cache
+   * expiration, but for frequently accessed keys it will be refreshed more often than that and the
+   * cost of the load will never be incurred during the read.
+   */
+  public static Caffeine<Object, Object> newCacheBuilder(Duration expireAfterWrite) {
+    Duration refreshAfterWrite = expireAfterWrite.dividedBy(2);
+    Caffeine<Object, Object> caffeine = Caffeine.newBuilder().expireAfterWrite(expireAfterWrite);
+    // In tests, the cache duration is usually set to 0, which means the cache load synchronously
+    // blocks every time it is called anyway because of the expireAfterWrite() above. Thus, setting
+    // the refreshAfterWrite won't do anything, plus it's not legal to call it with a zero value
+    // anyway (Caffeine allows expireAfterWrite to be zero but not refreshAfterWrite).
+    if (!refreshAfterWrite.isZero()) {
+      caffeine = caffeine.refreshAfterWrite(refreshAfterWrite);
+    }
+    return caffeine;
  }
 }
@@ -21,11 +21,9 @@ import static com.google.common.collect.Maps.transformValues;
 import static java.util.stream.Collectors.toCollection;
 import static java.util.stream.Collectors.toList;

+import com.github.benmanes.caffeine.cache.LoadingCache;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Predicate;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedMap;
@@ -59,36 +57,33 @@ public class ModelUtils {

  /** Caches all instance fields on an object, including non-public and inherited fields. */
  private static final LoadingCache<Class<?>, ImmutableMap<String, Field>> ALL_FIELDS_CACHE =
-      CacheBuilder.newBuilder()
+      CacheUtils.newCacheBuilder()
          .build(
-              new CacheLoader<Class<?>, ImmutableMap<String, Field>>() {
-                @Override
-                public ImmutableMap<String, Field> load(Class<?> clazz) {
-                  Deque<Class<?>> hierarchy = new ArrayDeque<>();
-                  // Walk the hierarchy up to but not including ImmutableObject (to ignore
-                  // hashCode).
-                  for (; clazz != ImmutableObject.class; clazz = clazz.getSuperclass()) {
-                    // Add to the front, so that shadowed fields show up later in the list.
-                    // This will mean that getFieldValues will show the most derived value.
-                    hierarchy.addFirst(clazz);
-                  }
-                  Map<String, Field> fields = new LinkedHashMap<>();
-                  for (Class<?> hierarchyClass : hierarchy) {
-                    // Don't use hierarchyClass.getFields() because it only picks up public fields.
-                    for (Field field : hierarchyClass.getDeclaredFields()) {
-                      if (!Modifier.isStatic(field.getModifiers())) {
-                        field.setAccessible(true);
-                        fields.put(field.getName(), field);
-                      }
+              clazz -> {
+                Deque<Class<?>> hierarchy = new ArrayDeque<>();
+                // Walk the hierarchy up to but not including ImmutableObject (to ignore
+                // hashCode).
+                for (; clazz != ImmutableObject.class; clazz = clazz.getSuperclass()) {
+                  // Add to the front, so that shadowed fields show up later in the list.
+                  // This will mean that getFieldValues will show the most derived value.
+                  hierarchy.addFirst(clazz);
+                }
+                Map<String, Field> fields = new LinkedHashMap<>();
+                for (Class<?> hierarchyClass : hierarchy) {
+                  // Don't use hierarchyClass.getFields() because it only picks up public fields.
+                  for (Field field : hierarchyClass.getDeclaredFields()) {
+                    if (!Modifier.isStatic(field.getModifiers())) {
+                      field.setAccessible(true);
+                      fields.put(field.getName(), field);
                    }
                  }
-                  return ImmutableMap.copyOf(fields);
                }
+                return ImmutableMap.copyOf(fields);
              });

  /** Lists all instance fields on an object, including non-public and inherited fields. */
  public static Map<String, Field> getAllFields(Class<?> clazz) {
-    return ALL_FIELDS_CACHE.getUnchecked(clazz);
+    return ALL_FIELDS_CACHE.get(clazz);
  }

  /** Return a string representing the persisted schema of a type or enum. */
@@ -38,6 +38,7 @@ import google.registry.model.registrar.RegistrarAddress;
 import google.registry.model.registrar.RegistrarContact;
 import google.registry.model.tld.Registry;
 import google.registry.model.tld.Registry.TldState;
+import google.registry.model.tld.Registry.TldType;
 import google.registry.model.tld.label.PremiumList;
 import google.registry.model.tld.label.PremiumListDao;
 import google.registry.persistence.VKey;
@@ -111,6 +112,17 @@ public final class OteAccountBuilder {
          DateTime.parse("2030-03-01T00:00:00Z"),
          Money.of(CurrencyUnit.USD, 0));

+  /**
+   * The default billing account map applied to all OT&amp;E registrars.
+   *
+   * <p>This contains dummy values for USD and JPY so that OT&amp;E registrars can be granted access
+   * to all existing TLDs in sandbox. Note that OT&amp;E is only on sandbox and thus these dummy
+   * values will never be used in production (the only environment where real invoicing takes
+   * place).
+   */
+  public static final ImmutableMap<CurrencyUnit, String> DEFAULT_BILLING_ACCOUNT_MAP =
+      ImmutableMap.of(CurrencyUnit.USD, "123", CurrencyUnit.JPY, "456");
+
  private final ImmutableMap<String, String> registrarIdToTld;
  private final Registry sunriseTld;
  private final Registry gaTld;
@@ -305,6 +317,7 @@ public final class OteAccountBuilder {
            .setTldStateTransitions(ImmutableSortedMap.of(START_OF_TIME, initialTldState))
            .setDnsWriters(ImmutableSet.of("VoidDnsWriter"))
            .setPremiumList(premiumList.get())
+            .setTldType(TldType.TEST)
            .setRoidSuffix(
                String.format(
                    "%S%X",
@@ -334,6 +347,7 @@ public final class OteAccountBuilder {
        .setPhoneNumber("+1.2125550100")
        .setIcannReferralEmail("nightmare@registrar.test")
        .setState(Registrar.State.ACTIVE)
+        .setBillingAccountMap(DEFAULT_BILLING_ACCOUNT_MAP)
        .build();
  }

@@ -126,7 +126,40 @@ public abstract class BillingEvent extends ImmutableObject
     * This flag will be added to any {@link OneTime} events that are created via, e.g., an
     * automated process to expand {@link Recurring} events.
     */
-    SYNTHETIC
+    SYNTHETIC;
+  }
+
+  /**
+   * Sets of renewal price behaviors that can be applied to billing recurrences.
+   *
+   * <p>When a client renews a domain, they could be charged differently, depending on factors such
+   * as the client type and the domain itself.
+   */
+  public enum RenewalPriceBehavior {
+    /**
+     * This indicates the renewal price is the default price.
+     *
+     * <p>By default, if the domain is premium, then premium price will be used. Otherwise, the
+     * standard price of the TLD will be used.
+     */
+    DEFAULT,
+    /**
+     * This indicates the domain will be renewed at standard price even if it's a premium domain.
+     *
+     * <p>We chose to name this "NONPREMIUM" rather than simply "STANDARD" to avoid confusion
+     * between "STANDARD" and "DEFAULT".
+     *
+     * <p>This price behavior is used with anchor tenants.
+     */
+    NONPREMIUM,
+    /**
+     * This indicates that the renewalPrice in {@link BillingEvent.Recurring} will be used for
+     * domain renewal.
+     *
+     * <p>The renewalPrice has a non-null value iff the price behavior is set to "SPECIFIED". This
+     * behavior is used with internal registrations.
+     */
+    SPECIFIED;
  }

  /** Entity id. */
@@ -555,6 +588,22 @@ public abstract class BillingEvent extends ImmutableObject
    })
    TimeOfYear recurrenceTimeOfYear;

+    /**
+     * The renewal price for domain renewal if and only if it's specified.
+     *
+     * <p>This price column remains null except when the renewal price behavior of the billing is
+     * SPECIFIED. This column is used for internal registrations.
+     */
+    @Nullable
+    @Type(type = JodaMoneyType.TYPE_NAME)
+    @Columns(
+        columns = {@Column(name = "renewalPriceAmount"), @Column(name = "renewalPriceCurrency")})
+    Money renewalPrice;
+
+    @Enumerated(EnumType.STRING)
+    @Column(name = "renewalPriceBehavior", nullable = false)
+    RenewalPriceBehavior renewalPriceBehavior = RenewalPriceBehavior.DEFAULT;
+
    public DateTime getRecurrenceEndTime() {
      return recurrenceEndTime;
    }
@@ -563,6 +612,14 @@ public abstract class BillingEvent extends ImmutableObject
      return recurrenceTimeOfYear;
    }

+    public RenewalPriceBehavior getRenewalPriceBehavior() {
+      return renewalPriceBehavior;
+    }
+
+    public Optional<Money> getRenewalPrice() {
+      return Optional.ofNullable(renewalPrice);
+    }
+
    @Override
    public VKey<Recurring> createVKey() {
      return VKey.create(Recurring.class, getId(), Key.create(this));
@@ -591,11 +648,26 @@ public abstract class BillingEvent extends ImmutableObject
        return this;
      }

+      public Builder setRenewalPriceBehavior(RenewalPriceBehavior renewalPriceBehavior) {
+        getInstance().renewalPriceBehavior = renewalPriceBehavior;
+        return this;
+      }
+
+      public Builder setRenewalPrice(@Nullable Money renewalPrice) {
+        getInstance().renewalPrice = renewalPrice;
+        return this;
+      }
+
      @Override
      public Recurring build() {
        Recurring instance = getInstance();
        checkNotNull(instance.eventTime);
        checkNotNull(instance.reason);
+        checkArgument(
+            (instance.renewalPriceBehavior == RenewalPriceBehavior.SPECIFIED)
+                ^ (instance.renewalPrice == null),
+            "Renewal price can have a value if and only if the renewal price behavior is"
+                + " SPECIFIED");
        instance.recurrenceTimeOfYear = TimeOfYear.fromDateTime(instance.eventTime);
        instance.recurrenceEndTime =
            Optional.ofNullable(instance.recurrenceEndTime).orElse(END_OF_TIME);
@@ -18,18 +18,18 @@ import static com.google.common.base.Preconditions.checkArgument;
 import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;

+import com.github.benmanes.caffeine.cache.LoadingCache;
 import com.google.common.annotations.VisibleForTesting;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
 import com.google.common.collect.ImmutableMultimap;
 import com.google.common.collect.ImmutableSortedMap;
 import com.google.common.flogger.FluentLogger;
 import google.registry.config.RegistryEnvironment;
+import google.registry.model.CacheUtils;
 import google.registry.model.annotations.DeleteAfterMigration;
 import google.registry.model.common.TimedTransitionProperty.TimedTransition;
 import google.registry.model.replay.SqlOnlyEntity;
 import java.time.Duration;
+import java.util.Arrays;
 import javax.persistence.Entity;
 import javax.persistence.PersistenceException;
 import org.joda.time.DateTime;
@@ -62,11 +62,28 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
   * not the phase is read-only.
   */
  public enum MigrationState {
+    /** Datastore is the only DB being used. */
    DATASTORE_ONLY(PrimaryDatabase.DATASTORE, false, ReplayDirection.NO_REPLAY),
+
+    /** Datastore is the primary DB, with changes replicated to Cloud SQL. */
    DATASTORE_PRIMARY(PrimaryDatabase.DATASTORE, false, ReplayDirection.DATASTORE_TO_SQL),
+
+    /** Datastore is the primary DB, with replication, and async actions are disallowed. */
+    DATASTORE_PRIMARY_NO_ASYNC(PrimaryDatabase.DATASTORE, false, ReplayDirection.DATASTORE_TO_SQL),
+
+    /** Datastore is the primary DB, with replication, and all mutating actions are disallowed. */
    DATASTORE_PRIMARY_READ_ONLY(PrimaryDatabase.DATASTORE, true, ReplayDirection.DATASTORE_TO_SQL),
+
+    /**
+     * Cloud SQL is the primary DB, with replication back to Datastore, and all mutating actions are
+     * disallowed.
+     */
    SQL_PRIMARY_READ_ONLY(PrimaryDatabase.CLOUD_SQL, true, ReplayDirection.SQL_TO_DATASTORE),
+
+    /** Cloud SQL is the primary DB, with changes replicated to Datastore. */
    SQL_PRIMARY(PrimaryDatabase.CLOUD_SQL, false, ReplayDirection.SQL_TO_DATASTORE),
+
+    /** Cloud SQL is the only DB being used. */
    SQL_ONLY(PrimaryDatabase.CLOUD_SQL, false, ReplayDirection.NO_REPLAY);

    private final PrimaryDatabase primaryDatabase;
@@ -117,18 +134,8 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
          TimedTransitionProperty<MigrationState, MigrationStateTransition>>
      // Each instance should cache the migration schedule for five minutes before reloading
      CACHE =
-          CacheBuilder.newBuilder()
-              .expireAfterWrite(Duration.ofMinutes(5))
-              .build(
-                  new CacheLoader<
-                      Class<DatabaseMigrationStateSchedule>,
-                      TimedTransitionProperty<MigrationState, MigrationStateTransition>>() {
-                    @Override
-                    public TimedTransitionProperty<MigrationState, MigrationStateTransition> load(
-                        Class<DatabaseMigrationStateSchedule> unused) {
-                      return DatabaseMigrationStateSchedule.getUncached();
-                    }
-                  });
+          CacheUtils.newCacheBuilder(Duration.ofMinutes(5))
+              .build(singletonClazz -> DatabaseMigrationStateSchedule.getUncached());

  // Restrictions on the state transitions, e.g. no going from DATASTORE_ONLY to SQL_ONLY
  private static final ImmutableMultimap<MigrationState, MigrationState> VALID_STATE_TRANSITIONS =
@@ -146,11 +153,17 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
            .putAll(
                MigrationState.DATASTORE_PRIMARY,
                MigrationState.DATASTORE_ONLY,
+                MigrationState.DATASTORE_PRIMARY_NO_ASYNC)
+            .putAll(
+                MigrationState.DATASTORE_PRIMARY_NO_ASYNC,
+                MigrationState.DATASTORE_ONLY,
+                MigrationState.DATASTORE_PRIMARY,
                MigrationState.DATASTORE_PRIMARY_READ_ONLY)
            .putAll(
                MigrationState.DATASTORE_PRIMARY_READ_ONLY,
                MigrationState.DATASTORE_ONLY,
                MigrationState.DATASTORE_PRIMARY,
+                MigrationState.DATASTORE_PRIMARY_NO_ASYNC,
                MigrationState.SQL_PRIMARY_READ_ONLY,
                MigrationState.SQL_PRIMARY)
            .putAll(
@@ -165,10 +178,9 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
                MigrationState.SQL_ONLY,
                MigrationState.SQL_PRIMARY_READ_ONLY,
                MigrationState.SQL_PRIMARY);
+
    // In addition, we can always transition from a state to itself (useful when updating the map).
-    for (MigrationState migrationState : MigrationState.values()) {
-      builder.put(migrationState, migrationState);
-    }
+    Arrays.stream(MigrationState.values()).forEach(state -> builder.put(state, state));
    return builder.build();
  }

@@ -212,7 +224,7 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements

  /** Loads the currently-set migration schedule from the cache, or the default if none exists. */
  public static TimedTransitionProperty<MigrationState, MigrationStateTransition> get() {
-    return CACHE.getUnchecked(DatabaseMigrationStateSchedule.class);
+    return CACHE.get(DatabaseMigrationStateSchedule.class);
  }

  /** Returns the database migration status at the given time. */
@@ -246,7 +258,7 @@ public class DatabaseMigrationStateSchedule extends CrossTldSingleton implements
   * A provided map of transitions may be valid by itself (i.e. it shifts states properly, doesn't
   * skip states, and doesn't backtrack incorrectly) while still being invalid. In addition to the
   * transitions in the map being valid, the single transition from the current map at the current
-   * time to the new map at the current time time must also be valid.
+   * time to the new map at the current time must also be valid.
   */
  private static void validateTransitionAtCurrentTime(
      TimedTransitionProperty<MigrationState, MigrationStateTransition> newTransitions) {
@@ -175,7 +175,7 @@ public class DomainBase extends DomainContent

  @Override
  public void beforeSqlSaveOnReplay() {
-    fullyQualifiedDomainName = DomainNameUtils.canonicalizeDomainName(fullyQualifiedDomainName);
+    fullyQualifiedDomainName = DomainNameUtils.canonicalizeHostname(fullyQualifiedDomainName);
    dsData =
        dsData.stream()
            .filter(datum -> datum.getDigest() != null && datum.getDigest().length > 0)
@@ -34,7 +34,7 @@ import static google.registry.util.DateTimeUtils.END_OF_TIME;
 import static google.registry.util.DateTimeUtils.earliestOf;
 import static google.registry.util.DateTimeUtils.isBeforeOrAt;
 import static google.registry.util.DateTimeUtils.leapSafeAddYears;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static google.registry.util.DomainNameUtils.getTldFromDomainName;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

@@ -890,7 +890,7 @@ public class DomainContent extends EppResource

    public B setDomainName(String domainName) {
      checkArgument(
-          domainName.equals(canonicalizeDomainName(domainName)),
+          domainName.equals(canonicalizeHostname(domainName)),
          "Domain name %s not in puny-coded, lower-case form",
          domainName);
      getInstance().fullyQualifiedDomainName = domainName;
@@ -309,7 +309,7 @@ public class DomainHistory extends HistoryEntry implements SqlEntity {
    if (domainContent == null) {
      domainContent = jpaTm().getEntityManager().find(DomainBase.class, getDomainRepoId());
      domainContent.fullyQualifiedDomainName =
-          DomainNameUtils.canonicalizeDomainName(domainContent.fullyQualifiedDomainName);
+          DomainNameUtils.canonicalizeHostname(domainContent.fullyQualifiedDomainName);
      fillAuxiliaryFieldsFromDomain(this);
    }
  }
@@ -43,6 +43,7 @@ import google.registry.model.BackupGroupRoot;
 import google.registry.model.Buildable;
 import google.registry.model.CreateAutoTimestamp;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.model.billing.BillingEvent.RenewalPriceBehavior;
 import google.registry.model.common.TimedTransitionProperty;
 import google.registry.model.common.TimedTransitionProperty.TimeMapper;
 import google.registry.model.common.TimedTransitionProperty.TimedTransition;
@@ -151,6 +152,10 @@ public class AllocationToken extends BackupGroupRoot implements Buildable, Datas
  @Enumerated(EnumType.STRING)
  TokenType tokenType;

+  @Enumerated(EnumType.STRING)
+  @Column(name = "renewalPriceBehavior", nullable = false)
+  RenewalPriceBehavior renewalPriceBehavior = RenewalPriceBehavior.DEFAULT;
+
  // TODO: Remove onLoad once all allocation tokens are migrated to have a discountYears of 1.
  @OnLoad
  void onLoad() {
@@ -240,6 +245,10 @@ public class AllocationToken extends BackupGroupRoot implements Buildable, Datas
    return tokenStatusTransitions;
  }

+  public RenewalPriceBehavior getRenewalPriceBehavior() {
+    return renewalPriceBehavior;
+  }
+
  public VKey<AllocationToken> createVKey() {
    return VKey.create(AllocationToken.class, getToken(), Key.create(this));
  }
@@ -362,5 +371,10 @@ public class AllocationToken extends BackupGroupRoot implements Buildable, Datas
              "tokenStatusTransitions must start with NOT_STARTED");
      return this;
    }
+
+    public Builder setRenewalPriceBehavior(RenewalPriceBehavior renewalPriceBehavior) {
+      getInstance().renewalPriceBehavior = renewalPriceBehavior;
+      return this;
+    }
  }
 }
@@ -19,7 +19,7 @@ import static com.google.common.collect.Sets.difference;
 import static com.google.common.collect.Sets.union;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;

 import com.google.common.collect.ImmutableSet;
 import com.googlecode.objectify.annotation.IgnoreSave;
@@ -195,7 +195,7 @@ public class HostBase extends EppResource {

    public B setHostName(String hostName) {
      checkArgument(
-          hostName.equals(canonicalizeDomainName(hostName)),
+          hostName.equals(canonicalizeHostname(hostName)),
          "Host name %s not in puny-coded, lower-case form",
          hostName);
      getInstance().fullyQualifiedHostName = hostName;
@@ -143,7 +143,7 @@ public class HostHistory extends HistoryEntry implements SqlEntity, UnsafeSerial
    if (hostBase == null) {
      hostBase = jpaTm().getEntityManager().find(HostResource.class, getHostRepoId());
      hostBase.fullyQualifiedHostName =
-          DomainNameUtils.canonicalizeDomainName(hostBase.fullyQualifiedHostName);
+          DomainNameUtils.canonicalizeHostname(hostBase.fullyQualifiedHostName);
    }
  }

@@ -73,7 +73,7 @@ public class HostResource extends HostBase

  @Override
  public void beforeSqlSaveOnReplay() {
-    fullyQualifiedHostName = DomainNameUtils.canonicalizeDomainName(fullyQualifiedHostName);
+    fullyQualifiedHostName = DomainNameUtils.canonicalizeHostname(fullyQualifiedHostName);
  }

  @Override
@@ -69,7 +69,7 @@ public class CommitLogMutation extends ImmutableObject implements DatastoreOnlyE
   * converted to a raw Datastore Entity, serialized to bytes, and stored within the mutation.
   */
  public static CommitLogMutation create(Key<CommitLogManifest> parent, Object entity) {
-    return createFromRaw(parent, auditedOfy().save().toEntity(entity));
+    return createFromRaw(parent, auditedOfy().saveIgnoringReadOnlyWithBackup().toEntity(entity));
  }

  /**
@@ -156,7 +156,7 @@ public class CommitLoggedWork<R> implements Runnable {
            .map(entity -> (ImmutableObject) CommitLogMutation.create(manifestKey, entity))
            .collect(toImmutableSet());
    auditedOfy()
-        .saveWithoutBackup()
+        .saveIgnoringReadOnlyWithoutBackup()
        .entities(
            new ImmutableSet.Builder<>()
                .add(manifest)
@@ -81,6 +81,7 @@ import google.registry.model.common.EntityGroupRoot;
 import google.registry.model.registrar.Registrar.BillingAccountEntry.CurrencyMapper;
 import google.registry.model.replay.DatastoreAndSqlEntity;
 import google.registry.model.tld.Registry;
+import google.registry.model.tld.Registry.TldType;
 import google.registry.persistence.VKey;
 import google.registry.util.CidrAddressBlock;
 import java.security.cert.CertificateParsingException;
@@ -798,13 +799,9 @@ public class Registrar extends ImmutableObject
    }

    public Builder setBillingAccountMap(@Nullable Map<CurrencyUnit, String> billingAccountMap) {
-      if (billingAccountMap == null) {
-        getInstance().billingAccountMap = null;
-      } else {
-        getInstance().billingAccountMap =
-            billingAccountMap.entrySet().stream()
-                .collect(toImmutableMap(Map.Entry::getKey, BillingAccountEntry::new));
-      }
+      getInstance().billingAccountMap =
+          nullToEmptyImmutableCopy(billingAccountMap).entrySet().stream()
+              .collect(toImmutableMap(Map.Entry::getKey, BillingAccountEntry::new));
      return this;
    }

@@ -1015,6 +1012,20 @@ public class Registrar extends ImmutableObject
          String.format(
              "Supplied IANA ID is not valid for %s registrar type: %s",
              getInstance().type, getInstance().ianaIdentifier));
+
+      // In order to grant access to real TLDs, the registrar must have a corresponding billing
+      // account ID for that TLD's billing currency.
+      ImmutableSet<String> nonBillableTlds =
+          Registry.get(getInstance().getAllowedTlds()).stream()
+              .filter(r -> r.getTldType() == TldType.REAL)
+              .filter(r -> !getInstance().getBillingAccountMap().containsKey(r.getCurrency()))
+              .map(Registry::getTldStr)
+              .collect(toImmutableSet());
+      checkArgument(
+          nonBillableTlds.isEmpty(),
+          "Cannot set these allowed, real TLDs because their currency is missing "
+              + "from the billing account map: %s",
+          nonBillableTlds);
      return cloneEmptyToNull(super.build());
    }
  }
@@ -16,15 +16,14 @@ package google.registry.model.server;

 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

+import com.github.benmanes.caffeine.cache.LoadingCache;
 import com.google.common.annotations.VisibleForTesting;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
 import com.google.common.primitives.Longs;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.OnLoad;
 import com.googlecode.objectify.annotation.Unindex;
+import google.registry.model.CacheUtils;
 import google.registry.model.annotations.NotBackedUp;
 import google.registry.model.annotations.NotBackedUp.Reason;
 import google.registry.model.common.CrossTldSingleton;
@@ -32,7 +31,6 @@ import google.registry.model.replay.NonReplicatedEntity;
 import java.nio.ByteBuffer;
 import java.util.Optional;
 import java.util.UUID;
-import java.util.concurrent.ExecutionException;
 import javax.persistence.Column;
 import javax.persistence.PostLoad;
 import javax.persistence.Transient;
@@ -52,14 +50,7 @@ public class ServerSecret extends CrossTldSingleton implements NonReplicatedEnti
   * Supplier that can be reset for testing purposes.
   */
  private static final LoadingCache<Class<ServerSecret>, ServerSecret> CACHE =
-      CacheBuilder.newBuilder()
-          .build(
-              new CacheLoader<Class<ServerSecret>, ServerSecret>() {
-                @Override
-                public ServerSecret load(Class<ServerSecret> unused) {
-                  return retrieveAndSaveSecret();
-                }
-              });
+      CacheUtils.newCacheBuilder().build(singletonClazz -> retrieveAndSaveSecret());

  private static ServerSecret retrieveAndSaveSecret() {
    if (tm().isOfy()) {
@@ -84,11 +75,7 @@ public class ServerSecret extends CrossTldSingleton implements NonReplicatedEnti

  /** Returns the global ServerSecret instance, creating it if one isn't already in Datastore. */
  public static ServerSecret get() {
-    try {
-      return CACHE.get(ServerSecret.class);
-    } catch (ExecutionException e) {
-      throw new RuntimeException(e);
-    }
+    return CACHE.get(ServerSecret.class);
  }

  /** Most significant 8 bytes of the UUID value (stored separately for legacy purposes). */
@@ -72,7 +72,7 @@ public final class Registries {
                                .stream()
                                .map(Key::getName)
                                .collect(toImmutableSet());
-                        return Registry.getAll(tlds).stream()
+                        return Registry.get(tlds).stream()
                            .map(e -> Maps.immutableEntry(e.getTldStr(), e.getTldType()))
                            .collect(entriesToImmutableMap());
                      } else {
@@ -105,7 +105,7 @@ public final class Registries {

  /** Returns the Registry entities themselves of the given type loaded fresh from Datastore. */
  public static ImmutableSet<Registry> getTldEntitiesOfType(TldType type) {
-    return Registry.getAll(filterValues(cache.get(), equalTo(type)).keySet());
+    return Registry.get(filterValues(cache.get(), equalTo(type)).keySet());
  }

  /** Pass-through check that the specified TLD exists, otherwise throw an IAE. */
@@ -142,10 +142,16 @@ public class Registry extends ImmutableObject

  /** The type of TLD, which determines things like backups and escrow policy. */
  public enum TldType {
-    /** A real, official TLD. */
+    /**
+     * A real, official TLD (but not necessarily only on production).
+     *
+     * <p>Note that, to avoid unnecessary costly DB writes, {@link
+     * google.registry.model.reporting.DomainTransactionRecord}s are only written out for REAL TLDs
+     * (these transaction records are only used for ICANN reporting purposes).
+     */
    REAL,

-    /** A test TLD, for the prober. */
+    /** A test TLD, for the prober, OT&amp;E, and other testing purposes. */
    TEST
  }

@@ -232,7 +238,7 @@ public class Registry extends ImmutableObject
  }

  /** Returns the registry entities for the given TLD strings, throwing if any don't exist. */
-  static ImmutableSet<Registry> getAll(Set<String> tlds) {
+  public static ImmutableSet<Registry> get(Set<String> tlds) {
    try {
      ImmutableMap<String, Optional<Registry>> registries = CACHE.getAll(tlds);
      ImmutableSet<String> missingRegistries =
@@ -264,8 +270,7 @@ public class Registry extends ImmutableObject
  /** A cache that loads the {@link Registry} for a given tld. */
  private static final LoadingCache<String, Optional<Registry>> CACHE =
      CacheBuilder.newBuilder()
-          .expireAfterWrite(
-              java.time.Duration.ofMillis(getSingletonCacheRefreshDuration().getMillis()))
+          .expireAfterWrite(getSingletonCacheRefreshDuration())
          .build(
              new CacheLoader<String, Optional<Registry>>() {
                @Override
@@ -1001,7 +1006,7 @@ public class Registry extends ImmutableObject
      instance.renewBillingCostTransitions.checkValidity();
      instance.eapFeeSchedule.checkValidity();
      // All costs must be in the expected currency.
-      // TODO(b/21854155): When we move PremiumList into Datastore, verify its currency too.
+      checkArgumentNotNull(instance.getCurrency(), "Currency must be set");
      checkArgument(
          instance.getStandardCreateCost().getCurrencyUnit().equals(instance.currency),
          "Create cost must be in the registry's currency");
@@ -16,7 +16,7 @@ package google.registry.model.tld.label;

 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Strings.emptyToNull;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

 import com.google.common.net.InternetDomainName;
@@ -77,7 +77,7 @@ public abstract class DomainLabelEntry<T extends Comparable<?>, D extends Domain
    public T build() {
      checkArgumentNotNull(emptyToNull(getInstance().domainLabel), "Label must be specified");
      checkArgument(
-          getInstance().domainLabel.equals(canonicalizeDomainName(getInstance().domainLabel)),
+          getInstance().domainLabel.equals(canonicalizeHostname(getInstance().domainLabel)),
          "Label '%s' must be in puny-coded, lower-case form",
          getInstance().domainLabel);
      checkArgumentNotNull(getInstance().getValue(), "Value must be specified");
@@ -70,8 +70,7 @@ public class ClaimsList extends ImmutableObject implements SqlOnlyEntity {
   *
   * <p>Note that the value of this field is parsed from the claims list file(See this <a
   * href="https://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.1">RFC</>), it is
-   * the DNL List creation datetime from the rfc. Since this field has been used by Datastore, we
-   * cannot change its name until we finish the migration.
+   * the DNL List creation datetime from the rfc.
   *
   * <p>TODO(b/177567432): Rename this field to tmdbGenerationTime.
   */
@@ -47,7 +47,7 @@ public final class TmchCrl extends CrossTldSingleton implements SqlOnlyEntity {
   * Change the singleton to a new ASCII-armored X.509 CRL.
   *
   * <p>Please do not call this function unless your CRL is properly formatted, signed by the root,
-   * and actually newer than the one currently in Datastore.
+   * and actually newer than the one currently in the database.
   */
  public static void set(final String crl, final String url) {
    jpaTm()
@@ -44,6 +44,7 @@ import google.registry.request.Modules.DatastoreServiceModule;
 import google.registry.request.Modules.Jackson2Module;
 import google.registry.request.Modules.NetHttpTransportModule;
 import google.registry.request.Modules.UrlConnectionServiceModule;
+import google.registry.request.Modules.UrlFetchServiceModule;
 import google.registry.request.Modules.UrlFetchTransportModule;
 import google.registry.request.Modules.UserServiceModule;
 import google.registry.request.auth.AuthModule;
@@ -81,6 +82,7 @@ import javax.inject.Singleton;
      SheetsServiceModule.class,
      StackdriverModule.class,
      UrlConnectionServiceModule.class,
+      UrlFetchServiceModule.class,
      UrlFetchTransportModule.class,
      UserServiceModule.class,
      VoidDnsWriterModule.class,
@@ -31,6 +31,7 @@ import google.registry.batch.ExpandRecurringBillingEventsAction;
 import google.registry.batch.RefreshDnsOnHostRenameAction;
 import google.registry.batch.RelockDomainAction;
 import google.registry.batch.ResaveAllEppResourcesAction;
+import google.registry.batch.ResaveAllEppResourcesPipelineAction;
 import google.registry.batch.ResaveEntityAction;
 import google.registry.batch.SendExpiringCertificateNotificationEmailAction;
 import google.registry.batch.WipeOutCloudSqlAction;
@@ -196,6 +197,8 @@ interface BackendRequestComponent {

  ResaveAllEppResourcesAction resaveAllEppResourcesAction();

+  ResaveAllEppResourcesPipelineAction resaveAllEppResourcesPipelineAction();
+
  ResaveEntityAction resaveEntityAction();

  SendExpiringCertificateNotificationEmailAction sendExpiringCertificateNotificationEmailAction();
@@ -21,6 +21,7 @@ import java.sql.PreparedStatement;
 import java.sql.ResultSet;
 import java.sql.SQLException;
 import java.util.Objects;
+import javax.annotation.Nullable;
 import org.hibernate.HibernateException;
 import org.hibernate.engine.spi.SharedSessionContractImplementor;
 import org.hibernate.type.StandardBasicTypes;
@@ -116,20 +117,28 @@ public class JodaMoneyType implements CompositeUserType {
    return Objects.hashCode(x);
  }

+  @Nullable
  @Override
  public Object nullSafeGet(
      ResultSet rs, String[] names, SharedSessionContractImplementor session, Object owner)
      throws HibernateException, SQLException {
    BigDecimal amount = StandardBasicTypes.BIG_DECIMAL.nullSafeGet(rs, names[AMOUNT_ID], session);
-    CurrencyUnit currencyUnit =
-        CurrencyUnit.of(StandardBasicTypes.STRING.nullSafeGet(rs, names[CURRENCY_ID], session));
-    if (amount != null && currencyUnit != null) {
-      return Money.of(currencyUnit, amount.stripTrailingZeros());
-    }
-    if (amount == null && currencyUnit == null) {
+    String currencyUnitString =
+        StandardBasicTypes.STRING.nullSafeGet(rs, names[CURRENCY_ID], session);
+    // It is allowable for a Money object to be null, but only if both the currency unit and the
+    // amount are null
+    if (amount == null && currencyUnitString == null) {
      return null;
+    } else if (amount != null && currencyUnitString != null) {
+      // CurrencyUnit.of() throws an IllegalCurrencyException for unknown currency, which means the
+      // currency is valid if it returns a value
+      return Money.of(CurrencyUnit.of(currencyUnitString), amount.stripTrailingZeros());
+    } else {
+      throw new HibernateException(
+          String.format(
+              "Mismatching null state between currency '%s' and amount '%s'",
+              currencyUnitString, amount));
    }
-    throw new HibernateException("Mismatching null state between currency and amount.");
  }

  @Override
@@ -140,7 +149,7 @@ public class JodaMoneyType implements CompositeUserType {
    String currencyUnit = value == null ? null : ((Money) value).getCurrencyUnit().getCode();

    if ((amount == null && currencyUnit != null) || (amount != null && currencyUnit == null)) {
-      throw new HibernateException("Mismatching null state between currency and amount.");
+      throw new HibernateException("Mismatching null state between currency and amount");
    }
    StandardBasicTypes.BIG_DECIMAL.nullSafeSet(st, amount, index, session);
    StandardBasicTypes.STRING.nullSafeSet(st, currencyUnit, index + 1, session);
@@ -33,8 +33,10 @@ import java.io.ByteArrayInputStream;
 import java.io.ByteArrayOutputStream;
 import java.io.EOFException;
 import java.io.IOException;
+import java.io.InputStream;
 import java.io.ObjectInputStream;
 import java.io.ObjectOutputStream;
+import java.io.ObjectStreamClass;

 /**
 * A SQL transaction that can be serialized and stored in its own table.
@@ -105,7 +107,8 @@ public class Transaction extends ImmutableObject implements Buildable {
  }

  public static Transaction deserialize(byte[] serializedTransaction) throws IOException {
-    ObjectInputStream in = new ObjectInputStream(new ByteArrayInputStream(serializedTransaction));
+    ObjectInputStream in =
+        new LenientObjectInputStream(new ByteArrayInputStream(serializedTransaction));

    // Verify that the data is what we expect.
    int version = in.readInt();
@@ -304,4 +307,35 @@ public class Transaction extends ImmutableObject implements Buildable {
      }
    }
  }
+
+  /**
+   * ObjectInputStream that ignores the UIDs of serialized objects.
+   *
+   * <p>We only really need to deserialize VKeys. However, VKeys have a class object associated with
+   * them, and if the class is changed and we haven't defined a serialVersionUID for it, we get an
+   * exception during deserialization.
+   *
+   * <p>It's safe for us to ignore this condition: we only care about attaching the correct local
+   * class object to the VKey. So this class effectively does so by replacing the class descriptor
+   * if it's version UID doesn't match that of the local class.
+   */
+  private static class LenientObjectInputStream extends ObjectInputStream {
+
+    public LenientObjectInputStream(InputStream in) throws IOException {
+      super(in);
+    }
+
+    @Override
+    protected ObjectStreamClass readClassDescriptor() throws IOException, ClassNotFoundException {
+      ObjectStreamClass persistedDescriptor = super.readClassDescriptor();
+      Class localClass = Class.forName(persistedDescriptor.getName());
+      ObjectStreamClass localDescriptor = ObjectStreamClass.lookup(localClass);
+      if (localDescriptor != null) {
+        if (persistedDescriptor.getSerialVersionUID() != localDescriptor.getSerialVersionUID()) {
+          return localDescriptor;
+        }
+      }
+      return persistedDescriptor;
+    }
+  }
 }
@@ -15,23 +15,25 @@
 package google.registry.persistence.transaction;

 import static com.google.common.base.Preconditions.checkState;
+import static google.registry.model.common.DatabaseMigrationStateSchedule.MigrationState.DATASTORE_PRIMARY_NO_ASYNC;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
-import static org.joda.time.DateTimeZone.UTC;

 import com.google.appengine.api.utils.SystemProperty;
 import com.google.appengine.api.utils.SystemProperty.Environment.Value;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Suppliers;
 import google.registry.config.RegistryEnvironment;
+import google.registry.model.annotations.DeleteAfterMigration;
 import google.registry.model.common.DatabaseMigrationStateSchedule;
 import google.registry.model.common.DatabaseMigrationStateSchedule.PrimaryDatabase;
 import google.registry.model.ofy.DatastoreTransactionManager;
 import google.registry.persistence.DaggerPersistenceComponent;
 import google.registry.tools.RegistryToolEnvironment;
+import google.registry.util.Clock;
 import google.registry.util.NonFinalForTesting;
+import google.registry.util.SystemClock;
 import java.util.Optional;
 import java.util.function.Supplier;
-import org.joda.time.DateTime;

 /** Factory class to create {@link TransactionManager} instance. */
 // TODO: Rename this to PersistenceFactory and move to persistence package.
@@ -42,6 +44,9 @@ public final class TransactionManagerFactory {
  /** Optional override to manually set the transaction manager per-test. */
  private static Optional<TransactionManager> tmForTest = Optional.empty();

+  /** The current clock (defined as a variable so we can override it in tests) */
+  private static Clock clock = new SystemClock();
+
  /** Supplier for jpaTm so that it is initialized only once, upon first usage. */
  @NonFinalForTesting
  private static Supplier<JpaTransactionManager> jpaTm =
@@ -103,7 +108,7 @@ public final class TransactionManagerFactory {
    if (onBeam) {
      return jpaTm();
    }
-    return DatabaseMigrationStateSchedule.getValueAtTime(DateTime.now(UTC))
+    return DatabaseMigrationStateSchedule.getValueAtTime(clock.nowUtc())
            .getPrimaryDatabase()
            .equals(PrimaryDatabase.DATASTORE)
        ? ofyTm()
@@ -193,11 +198,33 @@ public final class TransactionManagerFactory {
  }

  public static void assertNotReadOnlyMode() {
-    if (DatabaseMigrationStateSchedule.getValueAtTime(DateTime.now(UTC)).isReadOnly()) {
+    if (DatabaseMigrationStateSchedule.getValueAtTime(clock.nowUtc()).isReadOnly()) {
      throw new ReadOnlyModeException();
    }
  }

+  /**
+   * Asserts that async actions (contact/host deletes and host renames) are allowed.
+   *
+   * <p>These are allowed at all times except during the {@link
+   * DatabaseMigrationStateSchedule.MigrationState#DATASTORE_PRIMARY_NO_ASYNC} stage. Note that
+   * {@link ReadOnlyModeException} may well be thrown during other read-only stages inside the
+   * transaction manager; this method specifically checks only async actions.
+   */
+  @DeleteAfterMigration
+  public static void assertAsyncActionsAreAllowed() {
+    if (DatabaseMigrationStateSchedule.getValueAtTime(clock.nowUtc())
+        .equals(DATASTORE_PRIMARY_NO_ASYNC)) {
+      throw new ReadOnlyModeException();
+    }
+  }
+
+  /** Allows us to set the clock used by the factory in unit tests. */
+  @VisibleForTesting
+  public static void setClockForTesting(Clock clock) {
+    TransactionManagerFactory.clock = clock;
+  }
+
  /** Registry is currently undergoing maintenance and is in read-only mode. */
  public static class ReadOnlyModeException extends IllegalStateException {
    public ReadOnlyModeException() {
@@ -18,7 +18,7 @@ import static com.google.common.base.Charsets.UTF_8;
 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.net.HttpHeaders.ACCESS_CONTROL_ALLOW_ORIGIN;
 import static google.registry.request.Actions.getPathForAction;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static javax.servlet.http.HttpServletResponse.SC_BAD_REQUEST;
 import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
 import static javax.servlet.http.HttpServletResponse.SC_OK;
@@ -245,7 +245,7 @@ public abstract class RdapActionBase implements Runnable {
  }

  String canonicalizeName(String name) {
-    name = canonicalizeDomainName(name);
+    name = canonicalizeHostname(name);
    if (name.endsWith(".")) {
      name = name.substring(0, name.length() - 1);
    }
@@ -14,24 +14,25 @@

 package google.registry.rde;

-import static google.registry.request.UrlConnectionUtils.getResponseBytes;
-import static google.registry.request.UrlConnectionUtils.setBasicAuth;
-import static google.registry.request.UrlConnectionUtils.setPayload;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static com.google.appengine.api.urlfetch.FetchOptions.Builder.validateCertificate;
+import static com.google.appengine.api.urlfetch.HTTPMethod.PUT;
+import static com.google.common.io.BaseEncoding.base64;
+import static com.google.common.net.HttpHeaders.AUTHORIZATION;
+import static com.google.common.net.HttpHeaders.CONTENT_TYPE;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static javax.servlet.http.HttpServletResponse.SC_BAD_REQUEST;
 import static javax.servlet.http.HttpServletResponse.SC_OK;

-import com.google.api.client.http.HttpMethods;
+import com.google.appengine.api.urlfetch.HTTPHeader;
+import com.google.appengine.api.urlfetch.HTTPRequest;
 import com.google.appengine.api.urlfetch.HTTPResponse;
+import com.google.appengine.api.urlfetch.URLFetchService;
 import com.google.common.flogger.FluentLogger;
-import com.google.common.net.MediaType;
 import google.registry.config.RegistryConfig.Config;
 import google.registry.keyring.api.KeyModule.Key;
 import google.registry.request.HttpException.InternalServerErrorException;
-import google.registry.request.UrlConnectionService;
 import google.registry.util.Retrier;
-import google.registry.util.UrlConnectionException;
 import google.registry.xjc.XjcXmlTransformer;
 import google.registry.xjc.iirdea.XjcIirdeaResponseElement;
 import google.registry.xjc.iirdea.XjcIirdeaResult;
@@ -39,7 +40,6 @@ import google.registry.xjc.rdeheader.XjcRdeHeader;
 import google.registry.xjc.rdereport.XjcRdeReportReport;
 import google.registry.xml.XmlException;
 import java.io.ByteArrayInputStream;
-import java.net.HttpURLConnection;
 import java.net.MalformedURLException;
 import java.net.SocketTimeoutException;
 import java.net.URL;
@@ -58,10 +58,10 @@ public class RdeReporter {
   * @see <a href="http://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-05#section-4">
   *     ICANN Registry Interfaces - Interface details</a>
   */
-  private static final MediaType MEDIA_TYPE = MediaType.XML_UTF_8;
+  private static final String REPORT_MIME = "text/xml";

  @Inject Retrier retrier;
-  @Inject UrlConnectionService urlConnectionService;
+  @Inject URLFetchService urlFetchService;

  @Inject @Config("rdeReportUrlPrefix") String reportUrlPrefix;
  @Inject @Key("icannReportingPassword") String password;
@@ -76,24 +76,29 @@ public class RdeReporter {
    // Send a PUT request to ICANN's HTTPS server.
    URL url = makeReportUrl(header.getTld(), report.getId());
    String username = header.getTld() + "_ry";
+    String token = base64().encode(String.format("%s:%s", username, password).getBytes(UTF_8));
+    final HTTPRequest req = new HTTPRequest(url, PUT, validateCertificate().setDeadline(60d));
+    req.addHeader(new HTTPHeader(CONTENT_TYPE, REPORT_MIME));
+    req.addHeader(new HTTPHeader(AUTHORIZATION, "Basic " + token));
+    req.setPayload(reportBytes);
    logger.atInfo().log("Sending report:\n%s", new String(reportBytes, UTF_8));
-    byte[] responseBytes =
+    HTTPResponse rsp =
        retrier.callWithRetry(
            () -> {
-              HttpURLConnection connection = urlConnectionService.createConnection(url);
-              connection.setRequestMethod(HttpMethods.PUT);
-              setBasicAuth(connection, username, password);
-              setPayload(connection, reportBytes, MEDIA_TYPE.toString());
-              int responseCode = connection.getResponseCode();
-              if (responseCode == SC_OK || responseCode == SC_BAD_REQUEST) {
-                return getResponseBytes(connection);
+              HTTPResponse rsp1 = urlFetchService.fetch(req);
+              switch (rsp1.getResponseCode()) {
+                case SC_OK:
+                case SC_BAD_REQUEST:
+                  break;
+                default:
+                  throw new RuntimeException("PUT failed");
              }
-              throw new UrlConnectionException("PUT failed", connection);
+              return rsp1;
            },
            SocketTimeoutException.class);

    // Ensure the XML response is valid.
-    XjcIirdeaResult result = parseResult(responseBytes);
+    XjcIirdeaResult result = parseResult(rsp.getContent());
    if (result.getCode().getValue() != 1000) {
      logger.atWarning().log(
          "PUT rejected: %d %s\n%s",
@@ -118,7 +123,7 @@ public class RdeReporter {

  private URL makeReportUrl(String tld, String id) {
    try {
-      return new URL(String.format("%s/%s/%s", reportUrlPrefix, canonicalizeDomainName(tld), id));
+      return new URL(String.format("%s/%s/%s", reportUrlPrefix, canonicalizeHostname(tld), id));
    } catch (MalformedURLException e) {
      throw new RuntimeException(e);
    }
@@ -261,6 +261,10 @@ public final class RdeStagingAction implements Runnable {
  @Config("highPerformanceMachineType")
  String machineType;

+  @Inject
+  @Config("initialWorkerCount")
+  int numWorkers;
+
  @Inject @Config("transactionCooldown") Duration transactionCooldown;
  @Inject @Config("beamStagingBucketUrl") String stagingBucketUrl;
  @Inject @Config("rdeBucket") String rdeBucket;
@@ -341,6 +345,7 @@ public final class RdeStagingAction implements Runnable {
                                          .encode(stagingKeyBytes))
                                  .put("registryEnvironment", RegistryEnvironment.get().name())
                                  .put("workerMachineType", machineType)
+                                  .put("numWorkers", String.valueOf(numWorkers))
                                  .put(
                                      "jpaTransactionManagerType",
                                      JpaTransactionManagerType.READ_ONLY_REPLICA.toString())
@@ -32,6 +32,7 @@ import static java.util.Arrays.asList;
 import com.google.cloud.storage.BlobId;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Ordering;
 import com.google.common.flogger.FluentLogger;
 import com.google.common.io.ByteStreams;
 import com.jcraft.jsch.JSch;
@@ -122,7 +123,6 @@ public final class RdeUploadAction implements Runnable, EscrowTask {
  @Inject @Key("rdeSigningKey") PGPKeyPair signingKey;
  @Inject @Key("rdeStagingDecryptionKey") PGPPrivateKey stagingDecryptionKey;
  @Inject RdeUploadAction() {}
-
  @Override
  public void run() {
    logger.atInfo().log("Attempting to acquire RDE upload lock for TLD '%s'.", tld);
@@ -140,6 +140,27 @@ public final class RdeUploadAction implements Runnable, EscrowTask {

  @Override
  public void runWithLock(final DateTime watermark) throws Exception {
+    // If a prefix is not provided, but we are in SQL mode, try to determine the prefix. This should
+    // only happen when the RDE upload cron job runs to catch up any un-retried (i. e. expected)
+    // RDE failures.
+    if (!prefix.isPresent() && !tm().isOfy()) {
+      // The prefix is always in the format of: rde-2022-02-21t00-00-00z-2022-02-21t00-07-33z, where
+      // the first datetime is the watermark and the second one is the time when the RDE beam job
+      // launched. We search for the latest folder that starts with "rde-[watermark]".
+      String partialPrefix =
+          String.format("rde-%s", watermark.toString("yyyy-MM-dd't'HH-mm-ss'z'"));
+      String latestFilenameSuffix =
+          gcsUtils.listFolderObjects(bucket, partialPrefix).stream()
+              .max(Ordering.natural())
+              .orElse(null);
+      if (latestFilenameSuffix == null) {
+        throw new NoContentException(
+            String.format("RDE deposit for TLD %s on %s does not exist", tld, watermark));
+      }
+      int firstSlashPosition = latestFilenameSuffix.indexOf('/');
+      prefix =
+          Optional.of(partialPrefix + latestFilenameSuffix.substring(0, firstSlashPosition + 1));
+    }
    logger.atInfo().log("Verifying readiness to upload the RDE deposit.");
    Optional<Cursor> cursor =
        transactIfJpaTm(() -> tm().loadByKeyIfPresent(Cursor.createVKey(RDE_STAGING, tld)));
@@ -241,9 +262,9 @@ public final class RdeUploadAction implements Runnable, EscrowTask {
                    .setSignatureOutput(sigOut, signingKey)
                    .setFileMetadata(nameWithoutPrefix, xmlLength, watermark)
                    .build()) {
-            long bytesCopied = ByteStreams.copy(ghostrydeDecoder, rydeEncoder);
+          long bytesCopied = ByteStreams.copy(ghostrydeDecoder, rydeEncoder);
          logger.atInfo().log("Uploaded %,d bytes to path '%s'.", bytesCopied, rydeFilename);
-          }
+        }
        String sigFilename = nameWithoutPrefix + ".sig";
        BlobId sigGcsFilename = BlobId.of(bucket, name + ".sig");
        byte[] signature = sigOut.toByteArray();
@@ -23,6 +23,8 @@ import com.google.api.client.http.javanet.NetHttpTransport;
 import com.google.api.client.json.JsonFactory;
 import com.google.api.client.json.jackson2.JacksonFactory;
 import com.google.appengine.api.datastore.DatastoreService;
+import com.google.appengine.api.urlfetch.URLFetchService;
+import com.google.appengine.api.urlfetch.URLFetchServiceFactory;
 import com.google.appengine.api.users.UserService;
 import com.google.appengine.api.users.UserServiceFactory;
 import dagger.Module;
@@ -53,6 +55,17 @@ public final class Modules {
    }
  }

+  /** Dagger module for {@link URLFetchService}. */
+  @Module
+  public static final class UrlFetchServiceModule {
+    private static final URLFetchService fetchService = URLFetchServiceFactory.getURLFetchService();
+
+    @Provides
+    static URLFetchService provideUrlFetchService() {
+      return fetchService;
+    }
+  }
+
  /** Dagger module for {@link UserService}. */
  @Module
  public static final class UserServiceModule {
@@ -17,9 +17,10 @@ package google.registry.request;
 import java.io.IOException;
 import java.net.HttpURLConnection;
 import java.net.URL;
+import java.security.GeneralSecurityException;

 /** Functional interface for opening a connection from a URL, injectable for testing. */
 public interface UrlConnectionService {

-  HttpURLConnection createConnection(URL url) throws IOException;
+  HttpURLConnection createConnection(URL url) throws IOException, GeneralSecurityException;
 }
@@ -32,6 +32,7 @@ import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.net.HttpURLConnection;
 import java.net.URL;
+import java.security.GeneralSecurityException;
 import java.security.Security;
 import java.security.SignatureException;
 import java.util.Arrays;
@@ -110,7 +111,8 @@ public final class Marksdb {
    }
  }

-  byte[] fetch(URL url, Optional<String> loginAndPassword) throws IOException {
+  byte[] fetch(URL url, Optional<String> loginAndPassword)
+      throws IOException, GeneralSecurityException {
    HttpURLConnection connection = urlConnectionService.createConnection(url);
    loginAndPassword.ifPresent(auth -> setBasicAuth(connection, auth));
    try {
@@ -124,7 +126,7 @@ public final class Marksdb {
  }

  List<String> fetchSignedCsv(Optional<String> loginAndPassword, String csvPath, String sigPath)
-      throws IOException, SignatureException, PGPException {
+      throws IOException, GeneralSecurityException, PGPException {
    checkArgument(
        loginAndPassword.isPresent(), "Cannot fetch from MarksDB without login credentials");

@@ -54,6 +54,7 @@ import google.registry.util.UrlConnectionException;
 import java.io.IOException;
 import java.net.HttpURLConnection;
 import java.net.URL;
+import java.security.GeneralSecurityException;
 import java.security.SecureRandom;
 import java.util.List;
 import java.util.Random;
@@ -116,7 +117,7 @@ public final class NordnUploadAction implements Runnable {
  public void run() {
    try {
      processLordnTasks();
-    } catch (IOException e) {
+    } catch (IOException | GeneralSecurityException e) {
      throw new RuntimeException(e);
    }
  }
@@ -161,7 +162,7 @@ public final class NordnUploadAction implements Runnable {
    }
  }

-  private void processLordnTasks() throws IOException {
+  private void processLordnTasks() throws IOException, GeneralSecurityException {
    checkArgument(phase.equals(PARAM_LORDN_PHASE_SUNRISE)
        || phase.equals(PARAM_LORDN_PHASE_CLAIMS),
        "Invalid phase specified to Nordn servlet: %s.", phase);
@@ -194,7 +195,8 @@ public final class NordnUploadAction implements Runnable {
   * @see <a href="http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.3">TMCH
   *     functional specifications - LORDN File</a>
   */
-  private void uploadCsvToLordn(String urlPath, String csvData) throws IOException {
+  private void uploadCsvToLordn(String urlPath, String csvData)
+      throws IOException, GeneralSecurityException {
    String url = tmchMarksdbUrl + urlPath;
    logger.atInfo().log(
        "LORDN upload task %s: Sending to URL: %s ; data: %s", actionLogId, url, csvData);
@@ -34,6 +34,7 @@ import google.registry.util.UrlConnectionException;
 import java.io.IOException;
 import java.net.HttpURLConnection;
 import java.net.URL;
+import java.security.GeneralSecurityException;
 import java.util.Map.Entry;
 import javax.inject.Inject;

@@ -77,7 +78,7 @@ public final class NordnVerifyAction implements Runnable {
  public void run() {
    try {
      verify();
-    } catch (IOException e) {
+    } catch (IOException | GeneralSecurityException e) {
      throw new RuntimeException(e);
    }
  }
@@ -89,11 +90,11 @@ public final class NordnVerifyAction implements Runnable {
   * available.
   *
   * @throws ConflictException if MarksDB has not yet finished processing the LORDN upload
-   * @see <a href="http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.3.1">
-   *     TMCH functional specifications LORDN Log File</a>
+   * @see <a href="http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.3.1">TMCH
+   *     functional specifications LORDN Log File</a>
   */
  @VisibleForTesting
-  LordnLog verify() throws IOException {
+  LordnLog verify() throws IOException, GeneralSecurityException {
    logger.atInfo().log("LORDN verify task %s: Sending request to URL %s", actionLogId, url);
    HttpURLConnection connection = urlConnectionService.createConnection(url);
    lordnRequestInitializer.initialize(connection, tld);
@@ -17,11 +17,11 @@ package google.registry.tmch;
 import static google.registry.config.RegistryConfig.ConfigModule.TmchCaMode.PILOT;
 import static google.registry.config.RegistryConfig.ConfigModule.TmchCaMode.PRODUCTION;
 import static google.registry.config.RegistryConfig.getSingletonCacheRefreshDuration;
+import static google.registry.model.CacheUtils.newCacheBuilder;
 import static google.registry.util.ResourceUtils.readResourceUtf8;

-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
+import com.github.benmanes.caffeine.cache.CacheLoader;
+import com.github.benmanes.caffeine.cache.LoadingCache;
 import com.google.common.collect.ImmutableMap;
 import google.registry.config.RegistryConfig.Config;
 import google.registry.config.RegistryConfig.ConfigModule.TmchCaMode;
@@ -41,8 +41,8 @@ import javax.inject.Inject;
 * Helper methods for accessing ICANN's TMCH root certificate and revocation list.
 *
 * <p>There are two CRLs, a real one for the production environment and a pilot one for
- * non-production environments. The Datastore singleton {@link TmchCrl} entity is used to cache this
- * CRL once loaded and will always contain the proper one corresponding to the environment.
+ * non-production environments. The singleton {@link TmchCrl} entity is used to cache this CRL once
+ * loaded and will always contain the proper one corresponding to the environment.
 *
 * <p>The CRTs do not change and are included as files in the codebase that are not refreshed. They
 * were downloaded from https://ca.icann.org/tmch.crt and https://ca.icann.org/tmch_pilot.crt
@@ -66,7 +66,7 @@ public final class TmchCertificateAuthority {
  }

  /**
-   * A cached supplier that loads the CRL from Datastore or chooses a default value.
+   * A cached supplier that loads the CRL from the database or chooses a default value.
   *
   * <p>We keep the cache here rather than caching TmchCrl in the model, because loading the CRL
   * string into an X509CRL instance is expensive and should itself be cached.
@@ -76,9 +76,7 @@ public final class TmchCertificateAuthority {
   * persist the correct one for this given environment.
   */
  private static final LoadingCache<TmchCaMode, X509CRL> CRL_CACHE =
-      CacheBuilder.newBuilder()
-          .expireAfterWrite(
-              java.time.Duration.ofMillis(getSingletonCacheRefreshDuration().getMillis()))
+      newCacheBuilder(getSingletonCacheRefreshDuration())
          .build(
              new CacheLoader<TmchCaMode, X509CRL>() {
                @Override
@@ -132,7 +130,7 @@ public final class TmchCertificateAuthority {
  }

  /**
-   * Update to the latest TMCH X.509 certificate revocation list and save it to Datastore.
+   * Update to the latest TMCH X.509 certificate revocation list and save it to the database.
   *
   * <p>Your ASCII-armored CRL must be signed by the current ICANN root certificate.
   *
@@ -40,7 +40,7 @@ public final class TmchCrlAction implements Runnable {
  @Inject TmchCertificateAuthority tmchCertificateAuthority;
  @Inject TmchCrlAction() {}

-  /** Synchronously fetches latest ICANN TMCH CRL and saves it to Datastore. */
+  /** Synchronously fetches latest ICANN TMCH CRL and saves it to the database. */
  @Override
  public void run() {
    try {
@@ -23,7 +23,7 @@ import google.registry.model.tmch.ClaimsListDao;
 import google.registry.request.Action;
 import google.registry.request.auth.Auth;
 import java.io.IOException;
-import java.security.SignatureException;
+import java.security.GeneralSecurityException;
 import java.util.List;
 import java.util.Optional;
 import javax.inject.Inject;
@@ -52,7 +52,7 @@ public final class TmchDnlAction implements Runnable {
    List<String> lines;
    try {
      lines = marksdb.fetchSignedCsv(marksdbDnlLoginAndPassword, DNL_CSV_PATH, DNL_SIG_PATH);
-    } catch (SignatureException | IOException | PGPException e) {
+    } catch (GeneralSecurityException | IOException | PGPException e) {
      throw new RuntimeException(e);
    }
    ClaimsList claims = ClaimsListParser.parse(lines);
@@ -22,7 +22,7 @@ import google.registry.model.smd.SignedMarkRevocationList;
 import google.registry.request.Action;
 import google.registry.request.auth.Auth;
 import java.io.IOException;
-import java.security.SignatureException;
+import java.security.GeneralSecurityException;
 import java.util.List;
 import java.util.Optional;
 import javax.inject.Inject;
@@ -51,7 +51,7 @@ public final class TmchSmdrlAction implements Runnable {
    List<String> lines;
    try {
      lines = marksdb.fetchSignedCsv(marksdbSmdrlLoginAndPassword, SMDRL_CSV_PATH, SMDRL_SIG_PATH);
-    } catch (SignatureException | IOException | PGPException e) {
+    } catch (GeneralSecurityException | IOException | PGPException e) {
      throw new RuntimeException(e);
    }
    SignedMarkRevocationList smdrl = SmdrlCsvParser.parse(lines);
@@ -14,7 +14,7 @@

 package google.registry.tools;

-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static java.nio.charset.StandardCharsets.UTF_8;

 import com.beust.jcommander.Parameter;
@@ -80,7 +80,7 @@ final class CanonicalizeLabelsCommand implements Command {

  private String canonicalize(String rawLabel) {
    try {
-      return canonicalizeDomainName(rawLabel.replaceAll(" ", ""));
+      return canonicalizeHostname(rawLabel.replaceAll(" ", ""));
    } catch (Exception e) {
      System.err.printf("Error canonicalizing %s: %s\n", rawLabel, e.getMessage());
      return "";
@@ -15,7 +15,7 @@
 package google.registry.tools;

 import static google.registry.util.DomainNameUtils.ACE_PREFIX;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;

 import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
@@ -38,7 +38,7 @@ final class ConvertIdnCommand implements Command {
      if (label.startsWith(ACE_PREFIX)) {
        System.out.println(Idn.toUnicode(Ascii.toLowerCase(label)));
      } else {
-        System.out.println(canonicalizeDomainName(label));
+        System.out.println(canonicalizeHostname(label));
      }
    }
  }
@@ -22,6 +22,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.net.InetAddresses;
 import com.google.template.soy.data.SoyMapData;
 import google.registry.tools.soy.HostCreateSoyInfo;
+import google.registry.util.DomainNameUtils;
 import java.net.Inet4Address;
 import java.net.Inet6Address;
 import java.net.InetAddress;
@@ -68,7 +69,7 @@ final class CreateHostCommand extends MutatingEppToolCommand {
    addSoyRecord(
        clientId,
        new SoyMapData(
-            "hostname", hostName,
+            "hostname", DomainNameUtils.canonicalizeHostname(hostName),
            "ipv4addresses", ipv4Addresses.build(),
            "ipv6addresses", ipv6Addresses.build()));
  }
@@ -18,20 +18,17 @@ import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Predicates.isNull;
 import static com.google.common.base.Strings.isNullOrEmpty;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;
 import static google.registry.util.RegistrarUtils.normalizeRegistrarName;
 import static java.nio.charset.StandardCharsets.US_ASCII;
 import static org.joda.time.DateTimeZone.UTC;

 import com.beust.jcommander.Parameter;
-import com.google.common.base.Joiner;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
-import com.google.common.collect.Sets;
 import google.registry.flows.certs.CertificateChecker;
 import google.registry.model.registrar.Registrar;
 import google.registry.model.registrar.RegistrarAddress;
-import google.registry.model.tld.Registry;
 import google.registry.tools.params.KeyValueMapParameter.CurrencyUnitToStringMap;
 import google.registry.tools.params.OptionalLongParameter;
 import google.registry.tools.params.OptionalPhoneNumberParameter;
@@ -46,7 +43,6 @@ import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
-import java.util.Set;
 import javax.annotation.Nullable;
 import javax.inject.Inject;
 import org.joda.money.CurrencyUnit;
@@ -319,7 +315,7 @@ abstract class CreateOrUpdateRegistrarCommand extends MutatingCommand {
            addAllowedTlds.isEmpty(), "Can't specify both --allowedTlds and --addAllowedTlds");
        ImmutableSet.Builder<String> allowedTldsBuilder = new ImmutableSet.Builder<>();
        for (String allowedTld : allowedTlds) {
-          allowedTldsBuilder.add(canonicalizeDomainName(allowedTld));
+          allowedTldsBuilder.add(canonicalizeHostname(allowedTld));
        }
        builder.setAllowedTlds(allowedTldsBuilder.build());
      }
@@ -329,7 +325,7 @@ abstract class CreateOrUpdateRegistrarCommand extends MutatingCommand {
          allowedTldsBuilder.addAll(oldRegistrar.getAllowedTlds());
        }
        for (String allowedTld : addAllowedTlds) {
-          allowedTldsBuilder.add(canonicalizeDomainName(allowedTld));
+          allowedTldsBuilder.add(canonicalizeHostname(allowedTld));
        }
        builder.setAllowedTlds(allowedTldsBuilder.build());
      }
@@ -433,22 +429,6 @@ abstract class CreateOrUpdateRegistrarCommand extends MutatingCommand {
        // Require a phone passcode.
        checkArgument(
            newRegistrar.getPhonePasscode() != null, "--passcode is required for REAL registrars.");
-        // Check if registrar has billing account IDs for the currency of the TLDs that it is
-        // allowed to register.
-        ImmutableSet<CurrencyUnit> tldCurrencies =
-            newRegistrar
-                .getAllowedTlds()
-                .stream()
-                .map(tld -> Registry.get(tld).getCurrency())
-                .collect(toImmutableSet());
-        Set<CurrencyUnit> currenciesWithoutBillingAccountId =
-            newRegistrar.getBillingAccountMap() == null
-                ? tldCurrencies
-                : Sets.difference(tldCurrencies, newRegistrar.getBillingAccountMap().keySet());
-        checkArgument(
-            currenciesWithoutBillingAccountId.isEmpty(),
-            "Need billing account map entries for currencies: %s",
-            Joiner.on(' ').join(currenciesWithoutBillingAccountId));
      }

      stageEntityChange(oldRegistrar, newRegistrar);
@@ -16,7 +16,7 @@ package google.registry.tools;

 import static com.google.common.base.Preconditions.checkArgument;
 import static google.registry.util.CollectionUtils.findDuplicates;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;

 import com.beust.jcommander.Parameter;
 import com.google.common.base.Joiner;
@@ -263,10 +263,10 @@ abstract class CreateOrUpdateTldCommand extends MutatingCommand {
        "Can't update roid suffixes on multiple TLDs simultaneously");
    for (String tld : tlds) {
      checkArgument(
-          tld.equals(canonicalizeDomainName(tld)),
+          tld.equals(canonicalizeHostname(tld)),
          "TLD '%s' should be given in the canonical form '%s'",
          tld,
-          canonicalizeDomainName(tld));
+          canonicalizeHostname(tld));
      checkArgument(
          !Character.isDigit(tld.charAt(0)),
          "TLDs cannot begin with a number");
@@ -18,6 +18,7 @@ import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
 import com.google.template.soy.data.SoyMapData;
 import google.registry.tools.soy.HostDeleteSoyInfo;
+import google.registry.util.DomainNameUtils;

 /** A command to delete a host via EPP. */
@Parameters(separators = " =", commandDescription = "Delete host")
@@ -50,9 +51,11 @@ final class DeleteHostCommand extends MutatingEppToolCommand {
  @Override
  protected void initMutatingEppToolCommand() {
    setSoyTemplate(HostDeleteSoyInfo.getInstance(), HostDeleteSoyInfo.DELETEHOST);
-    addSoyRecord(clientId, new SoyMapData(
-        "hostName", hostName,
-        "reason", reason,
-        "requestedByRegistrar", requestedByRegistrar));
+    addSoyRecord(
+        clientId,
+        new SoyMapData(
+            "hostName", DomainNameUtils.canonicalizeHostname(hostName),
+            "reason", reason,
+            "requestedByRegistrar", requestedByRegistrar));
  }
 }
@@ -14,7 +14,7 @@

 package google.registry.tools;

-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.canonicalizeHostname;

 import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
@@ -61,6 +61,6 @@ class EncryptEscrowDepositCommand implements CommandWithRemoteApi {

  @Override
  public final void run() throws Exception {
-    encryptor.encrypt(mode, canonicalizeDomainName(tld), revision, input, outdir);
+    encryptor.encrypt(mode, canonicalizeHostname(tld), revision, input, outdir);
  }
 }
@@ -30,6 +30,7 @@ import google.registry.model.domain.DomainHistory;
 import google.registry.model.poll.PollMessage;
 import google.registry.model.registrar.Registrar;
 import google.registry.model.reporting.HistoryEntry;
+import google.registry.util.DomainNameUtils;
 import java.util.List;
 import java.util.Optional;
 import javax.inject.Inject;
@@ -79,6 +80,7 @@ class EnqueuePollMessageCommand extends MutatingCommand {

  @Override
  protected final void init() {
+    domainName = DomainNameUtils.canonicalizeHostname(domainName);
    checkArgument(
        !sendToAll || isNullOrEmpty(clientIds), "Cannot specify both --all and --clients");
    tm().transact(
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Michael MullerandGitHub	f273783894	Ignore version UIDs during txn deserialization (#1607 ) * Ignore version UIDs during txn deserialization When deserializing transactions for replay to datastore, ignore class version UIDs that don't match those of the local classes and just use the local class descriptors instead. This is a simple solution for the problem of persisted VKeys containing references to classes where the class has been updated and the serial version UID has changed. Also add a "replay_txns" command that replays the transactions from a given start point so we can verify all transactions are deserializable. TESTED: Ran replay_txns against all transactions on sandbox beginning with transaction id 1828385, which includes Recurring billing events containing both the old and the new serial version UIDs.	2022-04-27 15:40:27 -04:00
Ben McIlwainandGitHub	0dfabe1c64	Convert more Guava caches to Caffeine (#1603 ) * Convert more Guava caches to Caffeine	2022-04-26 11:26:51 -04:00
Weimin YuandGitHub	60a011c593	Remove stray file that slipped in the repo (#1604 ) * Remove stray file that slipped in the repo	2022-04-25 17:08:58 -04:00
Michael MullerandGitHub	7716eebfff	Change check for root directory during rollback (#1602 ) * Change check for root directory during rollback `rollback_tool` tries to infer the root of the nomulus tree by checking for a directory named "nomulus". This is potentially problematic (and, indeed, was for me) since there is no guarantee what that directory will be named. There are a number of features that characterize the root directory. Check for the presence of the `rollback_tool` wrapper script, as this is both at root level and tightly coupled to the python code, so hopefully we won't move it without testing that the script still works.	2022-04-25 12:39:16 -04:00
Ben McIlwainandGitHub	1e76eeed37	Validate that a registrar has billing accounts for all its allowed TLDs (#1601 ) This will require edits to a substantial number of registrars on sandbox (nearly all of them) because almost all of them have access to at least one TLD, but almost none of them have any billing accounts set. Until this is set, any updates to the existing registrars that aren't adding the billing accounts will cause failures. Unfortunately, there wasn't any less invasive foolproof way to implement this change, and we already had one attempt to implement it on create registrar command that wasn't working (because allowed TLDs tend not to be added on initial registrar creation, but rather, afterwards as an update).	2022-04-22 16:33:18 -04:00
sarahcaseybotandGitHub	147d133aef	Don't fail invoicing on missing PAK (#1595 ) * Don't fail invoicing on missing PAK * Skip line if missing PAK * Add log check in test	2022-04-22 13:00:50 -04:00
Ben McIlwainandGitHub	c2e1f2e640	Downgrade Caffeine to 2.9.3 (#1600 ) * Downgrade Caffeine to 2.9.3 Apparently Caffeine >=3.* requires Java 11, and we're still stuck on Java 8 because of App Engine Standard. Fortunately this doesn't affect the exposed interface we're using, so we can simply go back to the newest Caffeine version once Registry 3.0 Phase 3 (GKE migration) is completed.	2022-04-20 14:05:37 -04:00
Lai JiangandGitHub	5d2639834a	Remove the BEAM RDE pipeline side job (#1599 ) Now that SQL is the default, we do not need this side job to run alongside the main one. Its purpose was to validate the BEAM pipeline while Datastore was primary. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1599) <!-- Reviewable:end -->	2022-04-20 12:12:23 -04:00
Rachel GuanandGitHub	7912576e3d	Add rachelguan@ to CONTRIBUTORS (#1598 )	2022-04-19 19:18:44 -04:00
Rachel GuanandGitHub	8424c85258	Fix build warning for unused variable (#1594 )	2022-04-19 14:11:23 -04:00
Lai JiangandGitHub	e72dd73ed8	Fix build (#1597 )	2022-04-19 08:27:06 -04:00
Rachel GuanandGitHub	508d221b94	Replace .get()).isEqualTo() with hasValue() (#1589 ) * Replace .get()).isEqualTo() with hasValue() * Use containsExactly for list comparison * Fix spacing	2022-04-18 18:35:46 -04:00
gbrodmanandGitHub	073d0a416a	Create a Dataflow pipeline to resave EPP resources (#1553 ) * Create a Dataflow pipeline to resave EPP resources This has two modes. If `fast` is false, then we will just load all EPP resources, project them to the current time, and save them. If `fast` is true, we will attempt to intelligently load and save only resources that we expect to have changes applied when we project them to the current time. This means resources with pending transfers that have expired, domains with expired grace periods, and non-deleted domains that have expired (we expect that they autorenewed).	2022-04-15 15:46:35 -04:00
Rachel GuanandGitHub	f2ead5a0e3	Make code change for AllocationToken schema change (#1581 ) * Make code change for AllocationToken schema change	2022-04-15 15:28:39 -04:00
Lai JiangandGitHub	212dbbe520	Fix a in issue with RDE (#1593 ) For some inexplicable reason, the RDE beam pipeline in both sandbox and production has been broken for the past week or so. Our investigations revealed that during the CoGropuByKey stage, some repo ID -> revision ID pairs were duplicated. This may be a problem with the Dataflow runtime which somehow introduced the duplicate during reshuffling. This PR attempts to fix the symptom only by deduping the revision IDs. We will do some more investigation and possibly follow up with the Dataflow team if we determine it is an upstream issue. TESTED=deployed the pipeline and successfully run sandbox RDE with it.	2022-04-15 10:32:44 -04:00
Ben McIlwainandGitHub	8594a61fd4	Begin migration from Guava Cache to Caffeine (#1590 ) * Begin migration from Guava Cache to Caffeine Caffeine is apparently strictly superior to the older Guava Cache (and is even recommended in lieu of Guava Cache on Guava Cache's own documentation). This adds the relevant dependencies and switch over just a single call site to use the new Caffeine cache. It also implements a new pattern, asynchronously refreshing the cache value starting from half of our configuration time. For frequently accessed entities this will allow us to NEVER block on a load, as it will be asynchronously refreshed in the background long before it ever expires synchronously during a read operation.	2022-04-14 13:38:53 -04:00
Rachel GuanandGitHub	36837eb3e6	Change to hasSize() for assertions (#1588 ) * Change to hasSize() for assertions	2022-04-13 18:20:34 -04:00
Rachel GuanandGitHub	3a9a8c6557	Add new columns to BillingEvent (#1573 ) * Add new columns to BillingEvent.java * Improve PR and modifyJodaMoneyType to handle null currency in override * Add test cases for edge cases of nullSafeGet in JodaMoneyType * Improve assertions	2022-04-11 20:09:26 -04:00
Weimin YuandGitHub	65c2570b8f	Remove dos.xml from the configs (#1587 ) * Remove dos.xml from the configs We don't have dos config right now, and applying dos from "gcloud app deploy" is deprecated and has started causing problems. If we add dos configs, it should be using "gcloud app firewall-rules".	2022-04-11 15:22:42 -04:00
Weimin YuandGitHub	86acaa1b31	Build Java8-compatible release (#1586 ) * Build Java8-compatible release Use the new options.release Gradle property to make sure builds are compatible with Java 8, which is the runtime on Appengine. This new property replaces sourceCompatibility, targetCompatibility, and bootclasspath (wasn't previously set, which is the reason why we couldn't detect Java9 api usage when building).	2022-04-11 11:00:00 -04:00
Weimin YuandGitHub	436cc03be9	Remove Optional.isEmpty() in code (#1585 ) * Remove Optional.isEmpty() in code	2022-04-08 21:30:22 +00:00
Ben McIlwainandGitHub	e110ddd412	Canonicalize domain/host names in nomulus tool commands (#1583 ) * Canonicalize domain/host names in nomulus tool commands This helps prevent some common user errors.	2022-04-06 18:35:38 -04:00
Michael MullerandGitHub	214b23e99c	Ignore read-only when saving commit logs (#1584 ) * Ignore read-only when saving commit logs Ignore read-only when saving commit logs and commit log mutations so that we can safely replicate in read-only mode. This should be safe, as we only ever to the situation of saving commit logs and mutations when something has already actually been modified in a transaction, meaning that we should have hit the "read only" sentinel already. This also introduces the ability to set the Clock in the TransactionManagerFactory so that we can test this functionality. * Changes per review * Fix issues affecting tests - Restore clobbered async phase in testNoInMigrationState_doesNothing - Restore system clock to TransactionManagerFactory to avoid affecting other tests.	2022-04-06 13:06:08 -04:00
Rachel GuanandGitHub	743dea9ca2	Add renewal price behavior to AllocationToken (#1580 )	2022-04-04 18:51:49 -04:00
sarahcaseybotandGitHub	41f9f1ef7d	Change use of BillingIdentifier to BillingAccountMap in invoicing pipeline (#1577 ) * Change billingIdentifier to BillingAccountMap in invoicing pipeline * Add a default for billing account map * Throw error on missing PAK * Add unit test	2022-04-04 16:16:43 -04:00
Michael MullerandGitHub	44ede2b022	Check for error suggesting another nomulus running (#1582 ) Check for a PSQLException referencing a failed connection to "google:5433", which likely indicates that there is another nomulus tool instance running. It's worth giving this hint because in cases like this it's not at all obvious that the other instance of nomulus is problematic.	2022-04-04 11:14:43 -04:00
Ben McIlwainandGitHub	e4312322dc	Add a no-async actions DB migration phase (#1579 ) * Add a no-async actions DB migration phase This needs to be set several hours prior to entering the READONLY stage. This is not a read-only stage; all synchronous actions under Datastore (such as domain creates) will continue to succeed. The only thing that will fail is host deletes, host renames, and contact deletes, as these three actions require a mapreduce to run before they are complete, and we don't want mapreduces hanging around and executing during what is supposed to be a short duration READONLY period.	2022-04-01 16:55:51 -04:00
gbrodmanandGitHub	24dfaf6406	Use UrlFetch for RDE and default TLS (1.2) for other URL connections (#1578 ) * Use UrlFetch for RDE and default TLS (1.2) for other URL connections This removes the TLS 1.3-settings in the module providers and, essentially, reverts the changes in #1535 only to the RdeReporter and RdeReportActionTest	2022-03-31 14:08:28 -04:00
Rachel GuanandGitHub	7afb8fa343	Add default value to renewal_price_behavior (#1575 ) * Add default value to renewal_price_behavior * Change DEFAULT_PRICE to DEFAULT	2022-03-31 12:27:32 -04:00
Michael MullerandGitHub	02b3f7b505	Fix a few references to "Datastore" in comments (#1576 ) * Fix a few references to "Datastore" in comments Fix references to Datastore in the comments of classes that are now SQL-only.	2022-03-30 15:17:38 -04:00
Lai JiangandGitHub	25342aa480	Make a best effort guess on the RDE folder name (prefix) when not provided. (#1574 ) We have a cron job that runs the RDE upload action every 4 hours for all TLD. Normally this should be a no-op beacuse a RDE upload is scheduled after RDE staging is completed, and when it fails with non-2XX status it will retry. However if for some reason it failed due to 20X status (like waiting for the SFTP cursor), it will not retry but rely on the cron job to catch up. With the BEAM RDE pipeline every staging job saves all its deposits in a uniquely named folder to avoid the need to use a lock, which is not practical in BEAM. However the cron job has no way of knowing what the prefixes are for each TLD so it will fail in SQL mode. In this PR we implemented a logic to guess what the prefix should be and use it, if we are in SQL mode and a prefix is not provided. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1574) <!-- Reviewable:end -->	2022-03-30 11:36:24 -04:00
Rachel GuanandGitHub	3ef1e6c6a4	Add renewal columns in BillingRecurrence (#1568 ) * Add renewal columns in BillingRecurrence * Change from event to recurrence in file name	2022-03-28 17:42:01 -04:00
Lai JiangandGitHub	9363b30b3e	Set the initial worker count for the RDE beam pipeline at 24 (#1572 ) * Set the initial worker count for the RDE beam pipeline at 24 This likely will speed up the pipeline by skipping the initially slow process of spinning up instances.	2022-03-27 22:51:58 -04:00
Weimin YuandGitHub	342f7d72a2	Fix sporadic SQL Snapshot failure (#1571 ) * Fix sporadic SQL Snapshot failure The Postgresql set-snapshot statement (called in JpaTransactionManager.setDatabaseSnapshot() method) must be the first statement in the SQL transaction. Currenty the JpaTransaction.transact() method may insert a query for DatabaseMigrationStateSchedule before the user query when the cache is empty or the cached value expires. This PR proactively preloads the cache in RegistryJpaIO to prevent cache loading inside the transaction. This PR also changes some DatabaseSnapshotTest tests to be retrying, in case they run just after the cache expires. (This has happened before in CI).	2022-03-25 15:55:00 -04:00
Michael MullerandGitHub	3592877210	Terminate with exit code of 1 on format check fail (#1570 ) The format check script currently outputs "true" if there were files that need reformatting and "false" if not, which is useful for gradle but less so for other applications (notably commit hooks). Terminate with an exit code of 1 if the format check fails. TESTED: Tried this from both a pre-commit hook and from the gradle build.	2022-03-25 12:50:27 -04:00
Michael MullerandGitHub	3006ca39ca	Add a "list_txns" to dump Transaction table (#1569 ) * Add a "list_txns" to dump Transaction table Add the list_txns command which can dump the entire contents of the Transaction table, either in csv format or as human readable transactions. The CSV format is useful for storing the transaction table at a specific point in time for later reference without requiring us to repeatedly hit the replica. Creating this without tests because this command has a very short shelf-life and is really only intended to be run by developers. Tested all features locally. * Reformatted	2022-03-25 12:41:10 -04:00
gbrodmanandGitHub	63adfa77ed	Use TLS v1.3 explicitly in RDE reporting (#1564 ) * Use TLS v1.3 explicitly in RDE reporting The default Java 1.8 TLS version is 1.2 which isn't supported by the ICANN upload site.	2022-03-25 12:09:46 -04:00