Temporarily disable spec 11 pipeline deployment in GCB (#755 )

The current setup causes the GCB job to fail validation and not run because it uses backticks in the configuration yaml, which is not allowed -- there is no shell to perform backtick substitution. See the error message here: https://spinnaker.endpoints.domain-registry-dev.cloud.goog/gate/pipelines/01EF5GRMD625613H6Z033DBD3Z In the future please make sure to test the GCB pipeline as instructed in the comments at the beginning of each file before committing. I tried to work around it by downloading the nomulus tool jar file instead (running the nomulus-tool docker image inside a docker image is not advisable). However the "nomulus deploy_spec11_pipeline" command still fails. I'm not sure why. Has the command itself been tested locally? The error message is shown below: ``` Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions Step #2: WARNING: --region not set; will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory tryCreateDefaultBucket Step #2: INFO: No tempLocation specified, attempting to use default bucket: dataflow-staging-us-central1-937378958468 Step #2: Aug 09, 2020 3:11:47 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse Step #2: WARNING: Request failed with code 409, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://www.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=domain-registry-alpha. Step #2: Exception in thread "main" Step #2: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) Step #2: at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) Step #2: Step #2: at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) Step #2: at org.apache.beam.sdk.Pipeline.create(Pipeline.java:147) Step #2: Step #2: at google.registry.beam.spec11.Spec11Pipeline.deploy(Spec11Pipeline.java:157) Step #2: at google.registry.tools.DeploySpec11PipelineCommand.run(DeploySpec11PipelineCommand.java:80) Step #2: at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:257) Step #2: at google.registry.tools.RegistryCli.run(RegistryCli.java:182) Step #2: at google.registry.tools.RegistryTool.main(RegistryTool.java:129) Step #2: Caused by: java.lang.reflect.InvocationTargetException Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Step #2: Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Step #2: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Step #2: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) Step #2: ... 8 more Step #2: Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@5cb0d902, only URLClassLoaders are supported. Step #2: at org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:58) Step #2: Step #2: at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:285) Step #2: Step #2: ... 13 more ``` Lastly the "--project" flag refers to the KMS project. While I'm not sure which project is that, I don't think we can use the PROJECT_ID variable as this is a GCB-substituted variable which refers to the project that the GCB job runs in, which in our cases means domain-registry-dev. We shouldn't use that project for KMS. I've changed it to the same project as the one we are deploying to, but please note that we have a separate project ${project_id}-keys that is used for all KMS purposes. This is specified in the config file so if that's what you meant to use, there is no need to specify it in the command line. Actually if you meant use the project to be deployed to for KMS, it also shouldn't be necessary to specify it separately as this information is already known when you specified "nomulus -e ENV". https://team.git.corp.google.com/domain-registry-eng/nomulus-internal/+/refs/heads/master/core/src/main/java/google/registry/config/files/nomulus-config-production.yaml#168 Can you add more description on what the KMS project is supposed to be? I don't think we specify a project for KMS purpose in any other commands. Given that there are several unresolved issues, I've commented out my proposed solution so that deployment can proceed.
Disable tests in RC builds (#752 )
2026-07-18 14:02:35 +00:00 · 2020-08-09 22:41:31 -04:00 · 2020-08-07 17:51:34 -04:00 · 2020-08-07 15:38:53 -04:00 · 2020-08-07 10:53:46 -04:00 · 2020-08-06 21:13:27 -04:00
492 changed files with 8047 additions and 3804 deletions
@@ -97,8 +97,9 @@ nomulus.iws
 ######################################################################
 # Gradle Ignores

-# We don't want to ignore the gradle jar files
+# We don't want to ignore the gradle or google-java-format jar files
 !/gradle/wrapper/**/*.jar
+!/java-format/*.jar
 .gradle/
 **/build
 cloudbuild-caches/
@@ -2,3 +2,5 @@ extraction:
  java:
    prepare:
      packages: "npm"
+    index:
+      java_version: "11"
@@ -35,9 +35,6 @@ plugins {
  // Java static analysis plugins. Keep versions consistent with
  // ./buildSrc/build.gradle
  id 'nebula.lint' version '16.0.2'
-  // TODO(weiminyu): consider remove net.ltgt.apt. Gradle 5.2+
-  //  has similar functionalities.
-  id 'net.ltgt.apt' version '0.19' apply false
  id 'net.ltgt.errorprone' version '0.6.1'
  id 'checkstyle'
  id 'com.github.johnrengelman.shadow' version '5.1.0'
@@ -290,12 +287,6 @@ subprojects {

  project.tasks.test.dependsOn runPresubmits

-// Path to code generated with annotation processors. Note that this path is
-// chosen by the 'net.ltgt.apt' plugin, and may change if IDE-specific plugins
-// are applied, e.g., 'idea' or 'eclipse'
-  def aptGeneratedDir = "${project.buildDir}/generated/source/apt/main"
-  def aptGeneratedTestDir = "${project.buildDir}/generated/source/apt/test"
-
  def commonlyExcludedResources = ['**/*.java', '**/BUILD']

  project.ext.javaDir = "${project.projectDir}/src/main/java"
@@ -304,18 +295,12 @@ subprojects {

  sourceSets {
    main {
-      java {
-        srcDirs += aptGeneratedDir
-      }
      resources {
        srcDirs += project.ext.javaDir
        exclude commonlyExcludedResources
      }
    }
    test {
-      java {
-        srcDirs += aptGeneratedTestDir
-      }
      resources {
        srcDirs += project.ext.javaTestDir
        exclude commonlyExcludedResources
@@ -323,9 +308,14 @@ subprojects {
    }
  }

-  javadocSource << project.sourceSets.main.allJava
-  javadocClasspath << project.sourceSets.main.compileClasspath
-  javadocDependentTasks << project.tasks.compileJava
+  // No need to produce javadoc for the docs subproject, which has no APIs to
+  // expose to users.
+  if (project.name != 'docs') {
+    javadocSource << project.sourceSets.main.allJava
+    javadocClasspath << project.sourceSets.main.compileClasspath
+    javadocClasspath << "${buildDir}/generated/sources/annotationProcessor/java/main"
+    javadocDependentTasks << project.tasks.compileJava
+  }
 }

 // If "-P verboseTestOutput=true" is passed in, configure all subprojects to dump all of their
@@ -26,9 +26,6 @@ buildscript {
 plugins {
  // Java static analysis plugins. Keep versions consistent with ../build.gradle
  id 'nebula.lint' version '16.0.2'
-  // Config helper for annotation processors such as AutoValue and Dagger.
-  // Ensures that source code is generated at an appropriate location.
-  id 'net.ltgt.apt' version '0.19' apply false
  id 'net.ltgt.errorprone' version '0.6.1'
  id 'checkstyle'
  id 'com.diffplug.gradle.spotless' version '3.25.0'
@@ -83,10 +80,8 @@ dependencies {
  annotationProcessor deps['com.google.auto.value:auto-value']
  testCompile deps['com.google.truth:truth']
  testCompile deps['com.google.truth.extensions:truth-java8-extension']
-  testCompile deps['junit:junit']
  testCompile deps['org.junit.jupiter:junit-jupiter-api']
  testCompile deps['org.junit.jupiter:junit-jupiter-engine']
-  testCompile deps['org.junit.vintage:junit-vintage-engine']
  testCompile deps['org.mockito:mockito-core']
 }

@@ -19,9 +19,7 @@ commons-lang:commons-lang:2.6
 javax.inject:javax.inject:1
 junit:junit:4.12
 nebula.lint:nebula.lint.gradle.plugin:16.0.2
-net.ltgt.apt:net.ltgt.apt.gradle.plugin:0.19
 net.ltgt.errorprone:net.ltgt.errorprone.gradle.plugin:0.6.1
-net.ltgt.gradle:gradle-apt-plugin:0.19
 net.ltgt.gradle:gradle-errorprone-plugin:0.6.1
 org.apache.commons:commons-lang3:3.8.1
 org.apache.maven:maven-artifact:3.6.2
@@ -49,7 +49,7 @@ javax.annotation:jsr250-api:1.0
 javax.inject:javax.inject:1
 javax.validation:validation-api:1.0.0.GA
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 net.bytebuddy:byte-buddy-agent:1.10.5
 net.bytebuddy:byte-buddy:1.10.5
 org.apache.commons:commons-lang3:3.8.1
@@ -65,7 +65,6 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
@@ -49,7 +49,7 @@ javax.annotation:jsr250-api:1.0
 javax.inject:javax.inject:1
 javax.validation:validation-api:1.0.0.GA
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 net.bytebuddy:byte-buddy-agent:1.10.5
 net.bytebuddy:byte-buddy:1.10.5
 org.apache.commons:commons-lang3:3.8.1
@@ -65,7 +65,6 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
@@ -49,7 +49,7 @@ javax.annotation:jsr250-api:1.0
 javax.inject:javax.inject:1
 javax.validation:validation-api:1.0.0.GA
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 net.bytebuddy:byte-buddy-agent:1.10.5
 net.bytebuddy:byte-buddy:1.10.5
 org.apache.commons:commons-lang3:3.8.1
@@ -65,7 +65,6 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
@@ -49,7 +49,7 @@ javax.annotation:jsr250-api:1.0
 javax.inject:javax.inject:1
 javax.validation:validation-api:1.0.0.GA
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 net.bytebuddy:byte-buddy-agent:1.10.5
 net.bytebuddy:byte-buddy:1.10.5
 org.apache.commons:commons-lang3:3.8.1
@@ -65,7 +65,6 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
@@ -57,6 +57,10 @@ final class GcsPluginUtils {
    return file.toPath().toAbsolutePath().normalize();
  }

+  static Path toNormalizedPath(Path file) {
+    return file.toAbsolutePath().normalize();
+  }
+
  static String getContentType(String fileName) {
    return EXTENSION_TO_CONTENT_TYPE.getOrDefault(
        Files.getFileExtension(fileName), DEFAULT_CONTENT_TYPE);
@@ -26,13 +26,10 @@ import com.google.common.collect.ImmutableSet;
 import google.registry.gradle.plugin.ProjectData.TaskData;
 import java.io.File;
 import java.nio.file.Paths;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
+import org.junit.jupiter.api.Test;

 /** Tests for {@link CoverPageGenerator} */
-@RunWith(JUnit4.class)
-public final class CoverPageGeneratorTest {
+final class CoverPageGeneratorTest {

  private static final ProjectData EMPTY_PROJECT =
      ProjectData.builder()
@@ -88,14 +85,14 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testGetFilesToUpload_entryPoint_isIndexHtml() {
+  void testGetFilesToUpload_entryPoint_isIndexHtml() {
    CoverPageGenerator coverPageGenerator = new CoverPageGenerator(EMPTY_PROJECT);
    assertThat(coverPageGenerator.getFilesToUpload().entryPoint())
        .isEqualTo(Paths.get("index.html"));
  }

  @Test
-  public void testGetFilesToUpload_containsEntryFile() {
+  void testGetFilesToUpload_containsEntryFile() {
    String content = getContentOfGeneratedFile(EMPTY_PROJECT, "index.html");
    assertThat(content)
        .contains(
@@ -103,7 +100,7 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCoverPage_showsFailedTask() {
+  void testCoverPage_showsFailedTask() {
    String content = getCoverPage(EMPTY_PROJECT.toBuilder().addTask(EMPTY_TASK_FAILURE).build());
    assertThat(content).contains("task-failure");
    assertThat(content).contains("<p>FAILURE</p>");
@@ -112,7 +109,7 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCoverPage_showsSuccessfulTask() {
+  void testCoverPage_showsSuccessfulTask() {
    String content = getCoverPage(EMPTY_PROJECT.toBuilder().addTask(EMPTY_TASK_SUCCESS).build());
    assertThat(content).contains("task-success");
    assertThat(content).doesNotContain("<p>FAILURE</p>");
@@ -121,7 +118,7 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCoverPage_showsUpToDateTask() {
+  void testCoverPage_showsUpToDateTask() {
    String content = getCoverPage(EMPTY_PROJECT.toBuilder().addTask(EMPTY_TASK_UP_TO_DATE).build());
    assertThat(content).contains("task-up-to-date");
    assertThat(content).doesNotContain("<p>FAILURE</p>");
@@ -130,11 +127,10 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCoverPage_failedAreFirst() {
+  void testCoverPage_failedAreFirst() {
    String content =
        getCoverPage(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(EMPTY_TASK_UP_TO_DATE)
                .addTask(EMPTY_TASK_FAILURE)
                .addTask(EMPTY_TASK_SUCCESS)
@@ -149,11 +145,10 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCoverPage_failingTask_statusIsFailure() {
+  void testCoverPage_failingTask_statusIsFailure() {
    String content =
        getCoverPage(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(EMPTY_TASK_UP_TO_DATE)
                .addTask(EMPTY_TASK_FAILURE)
                .addTask(EMPTY_TASK_SUCCESS)
@@ -162,11 +157,10 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCoverPage_noFailingTask_statusIsSuccess() {
+  void testCoverPage_noFailingTask_statusIsSuccess() {
    String content =
        getCoverPage(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(EMPTY_TASK_UP_TO_DATE)
                .addTask(EMPTY_TASK_SUCCESS)
                .build());
@@ -174,7 +168,7 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testGetFilesToUpload_containsCssFile() {
+  void testGetFilesToUpload_containsCssFile() {
    ImmutableMap<String, String> files = getGeneratedFiles(EMPTY_PROJECT);
    assertThat(files).containsKey(filenameJoiner.join("css", "style.css"));
    assertThat(files.get(filenameJoiner.join("css", "style.css"))).contains("body {");
@@ -183,14 +177,12 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCreateReportFiles_taskWithLog() {
+  void testCreateReportFiles_taskWithLog() {
    ImmutableMap<String, String> files =
        getGeneratedFiles(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(
-                    EMPTY_TASK_SUCCESS
-                        .toBuilder()
+                    EMPTY_TASK_SUCCESS.toBuilder()
                        .setUniqueName("my:name")
                        .setLog(toByteArraySupplier("my log data"))
                        .build())
@@ -200,11 +192,10 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCreateReportFiles_taskWithoutLog() {
+  void testCreateReportFiles_taskWithoutLog() {
    ImmutableMap<String, String> files =
        getGeneratedFiles(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(EMPTY_TASK_SUCCESS.toBuilder().setUniqueName("my:name").build())
                .build());
    assertThat(files).doesNotContainKey("logs/my-name.log");
@@ -212,14 +203,12 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCreateReportFiles_taskWithFilledReport() {
+  void testCreateReportFiles_taskWithFilledReport() {
    ImmutableMap<String, String> files =
        getGeneratedFiles(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(
-                    EMPTY_TASK_SUCCESS
-                        .toBuilder()
+                    EMPTY_TASK_SUCCESS.toBuilder()
                        .putReport(
                            "someReport",
                            FilesWithEntryPoint.create(
@@ -234,14 +223,12 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCreateReportFiles_taskWithEmptyReport() {
+  void testCreateReportFiles_taskWithEmptyReport() {
    ImmutableMap<String, String> files =
        getGeneratedFiles(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(
-                    EMPTY_TASK_SUCCESS
-                        .toBuilder()
+                    EMPTY_TASK_SUCCESS.toBuilder()
                        .putReport(
                            "someReport",
                            FilesWithEntryPoint.create(
@@ -254,14 +241,12 @@ public final class CoverPageGeneratorTest {
  }

  @Test
-  public void testCreateReportFiles_taskWithLogAndMultipleReports() {
+  void testCreateReportFiles_taskWithLogAndMultipleReports() {
    ImmutableMap<String, String> files =
        getGeneratedFiles(
-            EMPTY_PROJECT
-                .toBuilder()
+            EMPTY_PROJECT.toBuilder()
                .addTask(
-                    EMPTY_TASK_SUCCESS
-                        .toBuilder()
+                    EMPTY_TASK_SUCCESS.toBuilder()
                        .setUniqueName("my:name")
                        .setLog(toByteArraySupplier("log data"))
                        .putReport(
@@ -23,6 +23,9 @@ import static google.registry.gradle.plugin.GcsPluginUtils.toNormalizedPath;
 import static google.registry.gradle.plugin.GcsPluginUtils.uploadFileToGcs;
 import static google.registry.gradle.plugin.GcsPluginUtils.uploadFilesToGcsMultithread;
 import static java.nio.charset.StandardCharsets.UTF_8;
+import static java.nio.file.Files.createDirectories;
+import static java.nio.file.Files.createDirectory;
+import static java.nio.file.Files.createFile;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.verifyNoMoreInteractions;
@@ -36,22 +39,20 @@ import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.Optional;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;

 /** Tests for {@link GcsPluginUtilsTest} */
-@RunWith(JUnit4.class)
-public final class GcsPluginUtilsTest {
+final class GcsPluginUtilsTest {

  private static final Joiner filenameJoiner = Joiner.on(File.separator);

-  @Rule public final TemporaryFolder folder = new TemporaryFolder();
+  @SuppressWarnings("WeakerAccess")
+  @TempDir
+  Path tmpDir;

  @Test
-  public void testGetContentType_knownTypes() {
+  void testGetContentType_knownTypes() {
    assertThat(getContentType("path/to/file.html")).isEqualTo("text/html");
    assertThat(getContentType("path/to/file.htm")).isEqualTo("text/html");
    assertThat(getContentType("path/to/file.log")).isEqualTo("text/plain");
@@ -63,12 +64,12 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testGetContentType_unknownTypes() {
+  void testGetContentType_unknownTypes() {
    assertThat(getContentType("path/to/file.unknown")).isEqualTo("application/octet-stream");
  }

  @Test
-  public void testUploadFileToGcs() {
+  void testUploadFileToGcs() {
    Storage storage = mock(Storage.class);
    uploadFileToGcs(
        storage, "my-bucket", Paths.get("my", "filename.txt"), toByteArraySupplier("my data"));
@@ -82,7 +83,7 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testUploadFilesToGcsMultithread() {
+  void testUploadFilesToGcsMultithread() {
    Storage storage = mock(Storage.class);
    uploadFilesToGcsMultithread(
        storage,
@@ -121,21 +122,21 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testToByteArraySupplier_string() {
+  void testToByteArraySupplier_string() {
    assertThat(toByteArraySupplier("my string").get()).isEqualTo("my string".getBytes(UTF_8));
  }

  @Test
-  public void testToByteArraySupplier_stringSupplier() {
+  void testToByteArraySupplier_stringSupplier() {
    assertThat(toByteArraySupplier(() -> "my string").get()).isEqualTo("my string".getBytes(UTF_8));
  }

  @Test
-  public void testToByteArraySupplier_file() throws Exception {
-    folder.newFolder("arbitrary");
-    File file = folder.newFile("arbitrary/file.txt");
-    Files.write(file.toPath(), "some data".getBytes(UTF_8));
-    assertThat(toByteArraySupplier(file).get()).isEqualTo("some data".getBytes(UTF_8));
+  void testToByteArraySupplier_file() throws Exception {
+    Path dir = createDirectory(tmpDir.resolve("arbitrary"));
+    Path file = createFile(dir.resolve("file.txt"));
+    Files.write(file, "some data".getBytes(UTF_8));
+    assertThat(toByteArraySupplier(file.toFile()).get()).isEqualTo("some data".getBytes(UTF_8));
  }

  private ImmutableMap<String, String> readAllFiles(FilesWithEntryPoint reportFiles) {
@@ -147,16 +148,16 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testCreateReportFiles_destinationIsFile() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
-    folder.newFolder("my", "root", "some", "path");
-    File destination = folder.newFile("my/root/some/path/file.txt");
-    Files.write(destination.toPath(), "some data".getBytes(UTF_8));
+  void testCreateReportFiles_destinationIsFile() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
+    Path somePath = createDirectories(root.resolve("some/path"));
+    Path destination = createFile(somePath.resolve("file.txt"));
+    Files.write(destination, "some data".getBytes(UTF_8));
    // Since the entry point is obvious here - any hint given is just ignored.
-    File ignoredHint = folder.newFile("my/root/ignored.txt");
+    File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();

    FilesWithEntryPoint files =
-        readFilesWithEntryPoint(destination, Optional.of(ignoredHint), root);
+        readFilesWithEntryPoint(destination.toFile(), Optional.of(ignoredHint), root);

    assertThat(files.entryPoint().toString())
        .isEqualTo(filenameJoiner.join("some", "path", "file.txt"));
@@ -165,13 +166,13 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testCreateReportFiles_destinationDoesntExist() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
+  void testCreateReportFiles_destinationDoesntExist() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
    File destination = root.resolve("non/existing.txt").toFile();
    assertThat(destination.isFile()).isFalse();
    assertThat(destination.isDirectory()).isFalse();
-    // Since there are not files, any hint given is obvioulsy wrong and will be ignored.
-    File ignoredHint = folder.newFile("my/root/ignored.txt");
+    // Since there are no files, any hint given is obviously wrong and will be ignored.
+    File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();

    FilesWithEntryPoint files =
        readFilesWithEntryPoint(destination, Optional.of(ignoredHint), root);
@@ -181,34 +182,33 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testCreateReportFiles_noFiles() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
-    File destination = folder.newFolder("my", "root", "some", "path");
-    folder.newFolder("my", "root", "some", "path", "a", "b");
-    folder.newFolder("my", "root", "some", "path", "c");
-    // Since there are not files, any hint given is obvioulsy wrong and will be ignored.
-    File ignoredHint = folder.newFile("my/root/ignored.txt");
+  void testCreateReportFiles_noFiles() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
+    Path destination = createDirectories(root.resolve("some/path"));
+    createDirectories(destination.resolve("a/b"));
+    createDirectory(destination.resolve("c"));
+    // Since there are not files, any hint given is obviously wrong and will be ignored.
+    File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();

    FilesWithEntryPoint files =
-        readFilesWithEntryPoint(destination, Optional.of(ignoredHint), root);
+        readFilesWithEntryPoint(destination.toFile(), Optional.of(ignoredHint), root);

    assertThat(files.entryPoint().toString()).isEqualTo(filenameJoiner.join("some", "path"));
    assertThat(files.files()).isEmpty();
  }

  @Test
-  public void testCreateReportFiles_oneFile() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
-    File destination = folder.newFolder("my", "root", "some", "path");
-    folder.newFolder("my", "root", "some", "path", "a", "b");
-    folder.newFolder("my", "root", "some", "path", "c");
-    Files.write(
-        folder.newFile("my/root/some/path/a/file.txt").toPath(), "some data".getBytes(UTF_8));
+  void testCreateReportFiles_oneFile() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
+    Path destination = createDirectories(root.resolve("some/path"));
+    createDirectories(destination.resolve("a/b"));
+    createDirectory(destination.resolve("c"));
+    Files.write(createFile(destination.resolve("a/file.txt")), "some data".getBytes(UTF_8));
    // Since the entry point is obvious here - any hint given is just ignored.
-    File ignoredHint = folder.newFile("my/root/ignored.txt");
+    File ignoredHint = createFile(root.resolve("ignored.txt")).toFile();

    FilesWithEntryPoint files =
-        readFilesWithEntryPoint(destination, Optional.of(ignoredHint), root);
+        readFilesWithEntryPoint(destination.toFile(), Optional.of(ignoredHint), root);

    assertThat(files.entryPoint().toString())
        .isEqualTo(filenameJoiner.join("some", "path", "a", "file.txt"));
@@ -222,22 +222,19 @@ public final class GcsPluginUtilsTest {
   * <p>TODO(guyben): switch to checking zip file instead.
   */
  @Test
-  public void testCreateReportFiles_multipleFiles_noHint() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
-    File destination = folder.newFolder("my", "root", "some", "path");
-    folder.newFolder("my", "root", "some", "path", "a", "b");
-    folder.newFolder("my", "root", "some", "path", "c");
+  void testCreateReportFiles_multipleFiles_noHint() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
+    Path destination = createDirectories(root.resolve("some/path"));
+    createDirectories(destination.resolve("a/b"));
+    createDirectory(destination.resolve("c"));

-    Files.write(
-        folder.newFile("my/root/some/path/index.html").toPath(), "some data".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/a/index.html").toPath(), "wrong index".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/c/style.css").toPath(), "css file".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/my_image.png").toPath(), "images".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("index.html")), "some data".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("a/index.html")), "wrong index".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("c/style.css")), "css file".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("my_image.png")), "images".getBytes(UTF_8));

-    FilesWithEntryPoint files = readFilesWithEntryPoint(destination, Optional.empty(), root);
+    FilesWithEntryPoint files =
+        readFilesWithEntryPoint(destination.toFile(), Optional.empty(), root);

    assertThat(files.entryPoint().toString())
        .isEqualTo(filenameJoiner.join("some", "path", "path.zip"));
@@ -251,24 +248,20 @@ public final class GcsPluginUtilsTest {
   * <p>TODO(guyben): switch to checking zip file instead.
   */
  @Test
-  public void testCreateReportFiles_multipleFiles_withBadHint() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
-    File destination = folder.newFolder("my", "root", "some", "path");
+  void testCreateReportFiles_multipleFiles_withBadHint() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
+    Path destination = createDirectories(root.resolve("some/path"));
    // This entry point points to a directory, which isn't an appropriate entry point
-    File badEntryPoint = folder.newFolder("my", "root", "some", "path", "a", "b");
-    folder.newFolder("my", "root", "some", "path", "c");
+    File badEntryPoint = createDirectories(destination.resolve("a/b")).toFile();
+    createDirectory(destination.resolve("c"));

-    Files.write(
-        folder.newFile("my/root/some/path/index.html").toPath(), "some data".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/a/index.html").toPath(), "wrong index".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/c/style.css").toPath(), "css file".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/my_image.png").toPath(), "images".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("index.html")), "some data".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("a/index.html")), "wrong index".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("c/style.css")), "css file".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("my_image.png")), "images".getBytes(UTF_8));

    FilesWithEntryPoint files =
-        readFilesWithEntryPoint(destination, Optional.of(badEntryPoint), root);
+        readFilesWithEntryPoint(destination.toFile(), Optional.of(badEntryPoint), root);

    assertThat(files.entryPoint().toString())
        .isEqualTo(filenameJoiner.join("some", "path", "path.zip"));
@@ -277,24 +270,21 @@ public final class GcsPluginUtilsTest {
  }

  @Test
-  public void testCreateReportFiles_multipleFiles_withGoodHint() throws Exception {
-    Path root = toNormalizedPath(folder.newFolder("my", "root"));
-    File destination = folder.newFolder("my", "root", "some", "path");
-    folder.newFolder("my", "root", "some", "path", "a", "b");
-    folder.newFolder("my", "root", "some", "path", "c");
+  void testCreateReportFiles_multipleFiles_withGoodHint() throws Exception {
+    Path root = toNormalizedPath(createDirectories(tmpDir.resolve("my/root")).toAbsolutePath());
+    Path destination = createDirectories(root.resolve("some/path"));
+    createDirectories(destination.resolve("a/b"));
+    createDirectory(destination.resolve("c"));
    // The hint is an actual file nested in the destination directory!
-    File goodEntryPoint = folder.newFile("my/root/some/path/index.html");
+    Path goodEntryPoint = createFile(destination.resolve("index.html"));

-    Files.write(goodEntryPoint.toPath(), "some data".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/a/index.html").toPath(), "wrong index".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/c/style.css").toPath(), "css file".getBytes(UTF_8));
-    Files.write(
-        folder.newFile("my/root/some/path/my_image.png").toPath(), "images".getBytes(UTF_8));
+    Files.write(goodEntryPoint, "some data".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("a/index.html")), "wrong index".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("c/style.css")), "css file".getBytes(UTF_8));
+    Files.write(createFile(destination.resolve("my_image.png")), "images".getBytes(UTF_8));

    FilesWithEntryPoint files =
-        readFilesWithEntryPoint(destination, Optional.of(goodEntryPoint), root);
+        readFilesWithEntryPoint(destination.toFile(), Optional.of(goodEntryPoint.toFile()), root);

    assertThat(files.entryPoint().toString())
        .isEqualTo(filenameJoiner.join("some", "path", "index.html"));
@@ -61,8 +61,6 @@ dependencies {
  testingCompile deps['com.google.truth:truth']
  testingCompile deps['io.github.java-diff-utils:java-diff-utils']

-  testCompile deps['junit:junit']
  testCompile deps['org.junit.jupiter:junit-jupiter-api']
  testCompile deps['org.junit.jupiter:junit-jupiter-engine']
-  testCompile deps['org.junit.vintage:junit-vintage-engine']
 }
@@ -1,4 +1,3 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
-com.google.errorprone:javac:9+181-r4173-1
@@ -14,7 +14,7 @@ com.googlecode.java-diff-utils:diffutils:1.3.0
 io.github.java-diff-utils:java-diff-utils:4.0
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -24,6 +24,5 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -14,7 +14,7 @@ com.googlecode.java-diff-utils:diffutils:1.3.0
 io.github.java-diff-utils:java-diff-utils:4.0
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -24,6 +24,5 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -15,7 +15,7 @@ com.googlecode.java-diff-utils:diffutils:1.3.0
 io.github.java-diff-utils:java-diff-utils:4.0
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -25,6 +25,5 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -15,7 +15,7 @@ com.googlecode.java-diff-utils:diffutils:1.3.0
 io.github.java-diff-utils:java-diff-utils:4.0
 javax.inject:javax.inject:1
 joda-time:joda-time:2.9.2
-junit:junit:4.13
+junit:junit:4.12
 org.apiguardian:apiguardian-api:1.1.0
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -25,6 +25,5 @@ org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
-org.junit.vintage:junit-vintage-engine:5.6.2
 org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -17,7 +17,7 @@ package google.registry.testing.truth;
 import static com.google.common.io.Resources.getResource;
 import static com.google.common.truth.Truth.assertThat;
 import static google.registry.testing.truth.TextDiffSubject.assertThat;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.common.base.Joiner;
 import com.google.common.collect.ImmutableList;
@@ -25,13 +25,10 @@ import com.google.common.io.Resources;
 import google.registry.testing.truth.TextDiffSubject.DiffFormat;
 import java.io.IOException;
 import java.nio.charset.StandardCharsets;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
+import org.junit.jupiter.api.Test;

 /** Unit tests for {@link TextDiffSubject}. */
-@RunWith(JUnit4.class)
-public class TextDiffSubjectTest {
+class TextDiffSubjectTest {

  private static final String RESOURCE_FOLDER = "google/registry/testing/truth/";
  // Resources for input data.
@@ -44,21 +41,21 @@ public class TextDiffSubjectTest {
      RESOURCE_FOLDER + "text-sidebyside-diff.txt";

  @Test
-  public void unifiedDiff_equal() throws IOException {
+  void unifiedDiff_equal() throws IOException {
    assertThat(getResource(ACTUAL_RESOURCE))
        .withDiffFormat(DiffFormat.UNIFIED_DIFF)
        .hasSameContentAs(getResource(ACTUAL_RESOURCE));
  }

  @Test
-  public void sideBySideDiff_equal() throws IOException {
+  void sideBySideDiff_equal() throws IOException {
    assertThat(getResource(ACTUAL_RESOURCE))
        .withDiffFormat(DiffFormat.SIDE_BY_SIDE_MARKDOWN)
        .hasSameContentAs(getResource(ACTUAL_RESOURCE));
  }

  @Test
-  public void unifedDiff_notEqual() throws IOException {
+  void unifedDiff_notEqual() throws IOException {
    assertThrows(
        AssertionError.class,
        () ->
@@ -68,7 +65,7 @@ public class TextDiffSubjectTest {
  }

  @Test
-  public void sideBySideDiff_notEqual() throws IOException {
+  void sideBySideDiff_notEqual() throws IOException {
    assertThrows(
        AssertionError.class,
        () ->
@@ -78,13 +75,13 @@ public class TextDiffSubjectTest {
  }

  @Test
-  public void displayed_unifiedDiff_noDiff() throws IOException {
+  void displayed_unifiedDiff_noDiff() throws IOException {
    ImmutableList<String> actual = readAllLinesFromResource(ACTUAL_RESOURCE);
    assertThat(TextDiffSubject.generateUnifiedDiff(actual, actual)).isEqualTo("");
  }

  @Test
-  public void displayed_unifiedDiff_hasDiff() throws IOException {
+  void displayed_unifiedDiff_hasDiff() throws IOException {
    ImmutableList<String> actual = readAllLinesFromResource(ACTUAL_RESOURCE);
    ImmutableList<String> expected = readAllLinesFromResource(EXPECTED_RESOURCE);
    String diff = Joiner.on('\n').join(readAllLinesFromResource(UNIFIED_DIFF_RESOURCE));
@@ -92,7 +89,7 @@ public class TextDiffSubjectTest {
  }

  @Test
-  public void displayed_sideBySideDiff_hasDiff() throws IOException {
+  void displayed_sideBySideDiff_hasDiff() throws IOException {
    ImmutableList<String> actual = readAllLinesFromResource(ACTUAL_RESOURCE);
    ImmutableList<String> expected = readAllLinesFromResource(EXPECTED_RESOURCE);
    String diff = Joiner.on('\n').join(readAllLinesFromResource(SIDE_BY_SIDE_DIFF_RESOURCE));
@@ -22,7 +22,7 @@ plugins {

 // Path to code generated by ad hoc tasks in this project. A separate path is
 // used for easy inspection.
-def generatedDir = "${project.buildDir}/generated/source/custom/main"
+def generatedDir = "${project.buildDir}/generated/sources/custom/java/main"
 def resourcesDir = "${project.buildDir}/resources/main"
 def screenshotsDir = "${project.buildDir}/screenshots"
 def screenshotsForGoldensDir = "${project.buildDir}/screenshots_for_goldens"
@@ -63,6 +63,8 @@ def dockerIncompatibleTestPatterns = [
    // methods, so we exclude the whole test class.
    "google/registry/tools/params/PathParameterTest.*",
    "google/registry/persistence/PersistenceModuleTest.*",
+    // This test is failing in docker when using Java 11. The cause is unclear.
+    "google/registry/tools/DomainLockUtilsTest.*",
 ]

 // Tests that conflict with members of both the main test suite and the
@@ -258,7 +260,7 @@ dependencies {
  compile deps['org.joda:joda-money']
  compile deps['org.json:json']
  testCompile deps['org.mortbay.jetty:jetty']
-  runtimeOnly deps['org.postgresql:postgresql']
+  compile deps['org.postgresql:postgresql']
  testCompile deps['org.seleniumhq.selenium:selenium-api']
  testCompile deps['org.seleniumhq.selenium:selenium-chrome-driver']
  testCompile deps['org.seleniumhq.selenium:selenium-java']
@@ -404,6 +406,9 @@ task jaxbToJava {
        }
      }
    }
+    execInBash(
+        'find . -name *.java -exec sed -i /\\*\\ \\<p\\>\\$/d {} +',
+        generatedDir)
  }
 }

@@ -702,6 +707,10 @@ task fragileTest(type: FilteringTest) {

  // Run every test class in a freshly started process.
  forkEvery 1
+
+  doFirst {
+    new File(screenshotsDir).deleteDir()
+  }
 }

 task outcastTest(type: FilteringTest) {
@@ -819,6 +828,30 @@ createToolTask(
    'google.registry.tools.DevTool',
    sourceSets.nonprod)

+project.tasks.create('initSqlPipeline', JavaExec) {
+  main = 'google.registry.beam.initsql.InitSqlPipeline'
+
+  doFirst {
+    getToolArgsList().ifPresent {
+      args it
+    }
+
+    def isDirectRunner =
+        args.contains('DirectRunner') || args.contains('--runner=DirectRunner')
+    // The dependency containing DirectRunner is intentionally excluded from the
+    // production binary, so that it won't be chosen by mistake: we definitely do
+    // not want to use it for the real jobs, yet DirectRunner is the default if
+    // the user forgets to override it.
+    // DirectRunner is required for tests and is already on testRuntimeClasspath.
+    // For simplicity, we add testRuntimeClasspath to this task's classpath instead
+    // of defining a new configuration just for the DirectRunner dependency.
+    classpath =
+        isDirectRunner
+            ? sourceSets.main.runtimeClasspath.plus(sourceSets.test.runtimeClasspath)
+            : sourceSets.main.runtimeClasspath
+  }
+}
+
 project.tasks.create('generateSqlSchema', JavaExec) {
  classpath = sourceSets.nonprod.runtimeClasspath
  main = 'google.registry.tools.DevTool'
@@ -870,10 +903,6 @@ task standardTest(type: FilteringTest) {

  systemProperty 'test.projectRoot', rootProject.projectRootDir
  systemProperty 'test.resourcesDir', resourcesDir
-
-  doFirst {
-    new File(screenshotsDir).deleteDir()
-  }
 }

 test {
@@ -228,6 +228,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
+org.postgresql:postgresql:42.2.14
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -223,6 +223,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
+org.postgresql:postgresql:42.2.14
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -1,4 +1,3 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
-com.google.errorprone:javac:9+181-r4173-1
@@ -228,6 +228,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
+org.postgresql:postgresql:42.2.14
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -226,6 +226,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
+org.postgresql:postgresql:42.2.14
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -266,6 +266,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
+org.postgresql:postgresql:42.2.14
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -264,6 +264,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
+org.postgresql:postgresql:42.2.14
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -39,7 +39,7 @@ module.exports = function(config) {
        included: false
      },
      {
-        pattern: 'core/build/generated/source/custom/main/**/*.soy.js',
+        pattern: 'core/build/generated/sources/custom/java/main/**/*.soy.js',
        included: false
      },
      {
@@ -65,7 +65,7 @@ module.exports = function(config) {
      'node_modules/google-closure-library/closure/goog/base.js': ['closure'],
      'node_modules/google-closure-library/closure/**/*.js': ['closure'],
      'core/src/*/javascript/**/*.js': ['closure'],
-      'core/build/generated/source/custom/main/**/*.soy.js': ['closure'],
+      'core/build/generated/sources/custom/java/main/**/*.soy.js': ['closure'],
    },
    proxies: {
      "/assets/": "/base/core/build/resources/main/google/registry/ui/assets/"
@@ -32,7 +32,7 @@ public final class BackupPaths {
  private BackupPaths() {}

  private static final String WILDCARD_CHAR = "*";
-  private static final String EXPORT_PATTERN_TEMPLATE = "%s/all_namespaces/kind_%s/input-%s";
+  private static final String EXPORT_PATTERN_TEMPLATE = "%s/all_namespaces/kind_%s/output-%s";

  public static final String COMMIT_LOG_NAME_PREFIX = "commit_diff_until_";
  private static final String COMMIT_LOG_PATTERN_TEMPLATE = "%s/" + COMMIT_LOG_NAME_PREFIX + "*";
@@ -40,6 +40,7 @@ import java.util.Collection;
 import java.util.Optional;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.Wait;
@@ -234,4 +235,10 @@ public class InitSqlPipeline implements Serializable {
      return entityClasses.stream().map(Key::getKind).collect(ImmutableList.toImmutableList());
    }
  }
+
+  public static void main(String[] args) {
+    InitSqlPipelineOptions options =
+        PipelineOptionsFactory.fromArgs(args).withValidation().as(InitSqlPipelineOptions.class);
+    new InitSqlPipeline(options).run();
+  }
 }
@@ -19,9 +19,15 @@ import static google.registry.beam.BeamUtils.getQueryFromFile;

 import com.google.auth.oauth2.GoogleCredentials;
 import com.google.auto.value.AutoValue;
+import com.google.common.collect.ImmutableSet;
+import google.registry.backup.AppEngineEnvironment;
+import google.registry.beam.initsql.Transforms.SerializableSupplier;
 import google.registry.beam.spec11.SafeBrowsingTransforms.EvaluateSafeBrowsingFn;
 import google.registry.config.CredentialModule.LocalCredential;
 import google.registry.config.RegistryConfig.Config;
+import google.registry.model.reporting.Spec11ThreatMatch;
+import google.registry.model.reporting.Spec11ThreatMatch.ThreatType;
+import google.registry.persistence.transaction.JpaTransactionManager;
 import google.registry.util.GoogleCredentialsBundle;
 import google.registry.util.Retrier;
 import google.registry.util.SqlTemplate;
@@ -37,6 +43,7 @@ import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.ParDo;
@@ -46,6 +53,7 @@ import org.apache.beam.sdk.values.TypeDescriptor;
 import org.apache.beam.sdk.values.TypeDescriptors;
 import org.joda.time.LocalDate;
 import org.joda.time.YearMonth;
+import org.joda.time.format.ISODateTimeFormat;
 import org.json.JSONArray;
 import org.json.JSONException;
 import org.json.JSONObject;
@@ -86,6 +94,7 @@ public class Spec11Pipeline implements Serializable {
  private final String reportingBucketUrl;
  private final GoogleCredentials googleCredentials;
  private final Retrier retrier;
+  private final SerializableSupplier<JpaTransactionManager> jpaSupplierFactory;

  @Inject
  public Spec11Pipeline(
@@ -93,12 +102,14 @@ public class Spec11Pipeline implements Serializable {
      @Config("beamStagingUrl") String beamStagingUrl,
      @Config("spec11TemplateUrl") String spec11TemplateUrl,
      @Config("reportingBucketUrl") String reportingBucketUrl,
+      SerializableSupplier<JpaTransactionManager> jpaSupplierFactory,
      @LocalCredential GoogleCredentialsBundle googleCredentialsBundle,
      Retrier retrier) {
    this.projectId = projectId;
    this.beamStagingUrl = beamStagingUrl;
    this.spec11TemplateUrl = spec11TemplateUrl;
    this.reportingBucketUrl = reportingBucketUrl;
+    this.jpaSupplierFactory = jpaSupplierFactory;
    this.googleCredentials = googleCredentialsBundle.getGoogleCredentials();
    this.retrier = retrier;
  }
@@ -177,12 +188,40 @@ public class Spec11Pipeline implements Serializable {
      EvaluateSafeBrowsingFn evaluateSafeBrowsingFn,
      ValueProvider<String> dateProvider) {

+    PCollection<KV<Subdomain, ThreatMatch>> subdomainsSql =
+        domains.apply("Run through SafeBrowsing API", ParDo.of(evaluateSafeBrowsingFn));
+    /* Store ThreatMatch objects in SQL. */
+    subdomainsSql.apply(
+        ParDo.of(
+            new DoFn<KV<Subdomain, ThreatMatch>, Void>() {
+              @ProcessElement
+              public void processElement(ProcessContext context) {
+                // create the Spec11ThreatMatch from Subdomain and ThreatMatch
+                try (AppEngineEnvironment env = new AppEngineEnvironment()) {
+                  Subdomain subdomain = context.element().getKey();
+                  Spec11ThreatMatch threatMatch =
+                      new Spec11ThreatMatch.Builder()
+                          .setThreatTypes(
+                              ImmutableSet.of(
+                                  ThreatType.valueOf(context.element().getValue().threatType())))
+                          .setCheckDate(
+                              LocalDate.parse(dateProvider.get(), ISODateTimeFormat.date()))
+                          .setDomainName(subdomain.domainName())
+                          .setDomainRepoId(subdomain.domainRepoId())
+                          .setRegistrarId(subdomain.registrarId())
+                          .build();
+                  JpaTransactionManager jpaTransactionManager = jpaSupplierFactory.get();
+                  jpaTransactionManager.transact(() -> jpaTransactionManager.saveNew(threatMatch));
+                }
+              }
+            }));
+
    /* Store ThreatMatch objects in JSON. */
    PCollection<KV<Subdomain, ThreatMatch>> subdomainsJson =
        domains.apply("Run through SafeBrowsingAPI", ParDo.of(evaluateSafeBrowsingFn));
    subdomainsJson
        .apply(
-            "Map registrar client ID to email/ThreatMatch pair",
+            "Map registrar ID to email/ThreatMatch pair",
            MapElements.into(
                    TypeDescriptors.kvs(
                        TypeDescriptors.strings(), TypeDescriptor.of(EmailAndThreatMatch.class)))
@@ -42,6 +42,7 @@ import google.registry.model.EppResource.ForeignKeyedEppResource;
 import google.registry.model.EppResource.ResourceWithTransferData;
 import google.registry.model.contact.ContactResource;
 import google.registry.model.domain.DomainBase;
+import google.registry.model.domain.DomainContent;
 import google.registry.model.domain.Period;
 import google.registry.model.domain.rgp.GracePeriodStatus;
 import google.registry.model.eppcommon.AuthInfo;
@@ -255,7 +256,7 @@ public final class ResourceFlowUtils {
   * @param domain is the domain already projected at approvalTime
   */
  public static DateTime computeExDateForApprovalTime(
-      DomainBase domain, DateTime approvalTime, Period period) {
+      DomainContent domain, DateTime approvalTime, Period period) {
    boolean inAutoRenew = domain.getGracePeriodStatuses().contains(GracePeriodStatus.AUTO_RENEW);
    // inAutoRenew is set to false if the period is zero because a zero-period transfer should not
    // subsume an autorenew.
@@ -360,7 +360,8 @@ public class DomainCreateFlow implements TransactionalFlow {
                    command.getNameservers().stream().collect(toImmutableSet()))
            .setStatusValues(statuses.build())
            .setContacts(command.getContacts())
-            .addGracePeriod(GracePeriod.forBillingEvent(GracePeriodStatus.ADD, createBillingEvent))
+            .addGracePeriod(
+                GracePeriod.forBillingEvent(GracePeriodStatus.ADD, repoId, createBillingEvent))
            .build();
    entitiesToSave.add(
        newDomain,
@@ -186,14 +186,18 @@ public final class DomainDeleteFlow implements TransactionalFlow {
      DateTime redemptionTime = now.plus(redemptionGracePeriodLength);
      asyncTaskEnqueuer.enqueueAsyncResave(
          existingDomain, now, ImmutableSortedSet.of(redemptionTime, deletionTime));
-      builder.setDeletionTime(deletionTime)
+      builder
+          .setDeletionTime(deletionTime)
          .setStatusValues(ImmutableSet.of(StatusValue.PENDING_DELETE))
          // Clear out all old grace periods and add REDEMPTION, which does not include a key to a
          // billing event because there isn't one for a domain delete.
-          .setGracePeriods(ImmutableSet.of(GracePeriod.createWithoutBillingEvent(
-              GracePeriodStatus.REDEMPTION,
-              redemptionTime,
-              clientId)));
+          .setGracePeriods(
+              ImmutableSet.of(
+                  GracePeriod.createWithoutBillingEvent(
+                      GracePeriodStatus.REDEMPTION,
+                      existingDomain.getRepoId(),
+                      redemptionTime,
+                      clientId)));
      // Note: The expiration time is unchanged, so if it's before the new deletion time, there will
      // be a "phantom autorenew" where the expiration time advances. No poll message will be
      // produced (since we are ending the autorenew recurrences at "now" below) and the billing
@@ -60,7 +60,7 @@ public final class DomainPricingLogic {
   * <p>If {@code allocationToken} is present and the domain is non-premium, that discount will be
   * applied to the first year.
   */
-  public FeesAndCredits getCreatePrice(
+  FeesAndCredits getCreatePrice(
      Registry registry,
      String domainName,
      DateTime dateTime,
@@ -104,8 +104,8 @@ public final class DomainPricingLogic {

  /** Returns a new renew price for the pricer. */
  @SuppressWarnings("unused")
-  public FeesAndCredits getRenewPrice(
-      Registry registry, String domainName, DateTime dateTime, int years) throws EppException {
+  FeesAndCredits getRenewPrice(Registry registry, String domainName, DateTime dateTime, int years)
+      throws EppException {
    DomainPrices domainPrices = getPricesForDomainName(domainName, dateTime);
    BigDecimal renewCost = domainPrices.getRenewCost().multipliedBy(years).getAmount();
    return customLogic.customizeRenewPrice(
@@ -123,7 +123,7 @@ public final class DomainPricingLogic {
  }

  /** Returns a new restore price for the pricer. */
-  public FeesAndCredits getRestorePrice(
+  FeesAndCredits getRestorePrice(
      Registry registry, String domainName, DateTime dateTime, boolean isExpired)
      throws EppException {
    DomainPrices domainPrices = getPricesForDomainName(domainName, dateTime);
@@ -147,7 +147,7 @@ public final class DomainPricingLogic {
  }

  /** Returns a new transfer price for the pricer. */
-  public FeesAndCredits getTransferPrice(Registry registry, String domainName, DateTime dateTime)
+  FeesAndCredits getTransferPrice(Registry registry, String domainName, DateTime dateTime)
      throws EppException {
    DomainPrices domainPrices = getPricesForDomainName(domainName, dateTime);
    return customLogic.customizeTransferPrice(
@@ -168,7 +168,7 @@ public final class DomainPricingLogic {
  }

  /** Returns a new update price for the pricer. */
-  public FeesAndCredits getUpdatePrice(Registry registry, String domainName, DateTime dateTime)
+  FeesAndCredits getUpdatePrice(Registry registry, String domainName, DateTime dateTime)
      throws EppException {
    CurrencyUnit currency = registry.getCurrency();
    BaseFee feeOrCredit = Fee.create(zeroInCurrency(currency), FeeType.UPDATE, false);
@@ -191,16 +191,20 @@ public final class DomainPricingLogic {
      throws EppException {
    if (allocationToken.isPresent()
        && allocationToken.get().getDiscountFraction() != 0.0
-        && domainPrices.isPremium()) {
+        && domainPrices.isPremium()
+        && !allocationToken.get().shouldDiscountPremiums()) {
      throw new AllocationTokenInvalidForPremiumNameException();
    }
    Money oneYearCreateCost = domainPrices.getCreateCost();
    Money totalDomainCreateCost = oneYearCreateCost.multipliedBy(years);
-    // If a discount is applicable, apply it only to the first year
+
+    // Apply the allocation token discount, if applicable.
    if (allocationToken.isPresent()) {
+      int discountedYears = Math.min(years, allocationToken.get().getDiscountYears());
      Money discount =
          oneYearCreateCost.multipliedBy(
-              allocationToken.get().getDiscountFraction(), RoundingMode.HALF_UP);
+              discountedYears * allocationToken.get().getDiscountFraction(),
+              RoundingMode.HALF_EVEN);
      totalDomainCreateCost = totalDomainCreateCost.minus(discount);
    }
    return totalDomainCreateCost;
@@ -209,7 +213,7 @@ public final class DomainPricingLogic {
  /** An allocation token was provided that is invalid for premium domains. */
  public static class AllocationTokenInvalidForPremiumNameException
      extends CommandUseErrorException {
-    public AllocationTokenInvalidForPremiumNameException() {
+    AllocationTokenInvalidForPremiumNameException() {
      super("A nonzero discount code cannot be applied to premium domains");
    }
  }
@@ -183,7 +183,8 @@ public final class DomainRenewFlow implements TransactionalFlow {
            .setAutorenewBillingEvent(newAutorenewEvent.createVKey())
            .setAutorenewPollMessage(newAutorenewPollMessage.createVKey())
            .addGracePeriod(
-                GracePeriod.forBillingEvent(GracePeriodStatus.RENEW, explicitRenewEvent))
+                GracePeriod.forBillingEvent(
+                    GracePeriodStatus.RENEW, existingDomain.getRepoId(), explicitRenewEvent))
            .build();
    EntityChanges entityChanges =
        flowCustomLogic.beforeSave(
@@ -192,7 +192,10 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
            .setGracePeriods(
                billingEvent.isPresent()
                    ? ImmutableSet.of(
-                        GracePeriod.forBillingEvent(GracePeriodStatus.TRANSFER, billingEvent.get()))
+                        GracePeriod.forBillingEvent(
+                            GracePeriodStatus.TRANSFER,
+                            existingDomain.getRepoId(),
+                            billingEvent.get()))
                    : ImmutableSet.of())
            .setLastEppUpdateTime(now)
            .setLastEppUpdateClientId(clientId)
@@ -14,7 +14,6 @@

 package google.registry.model;

-import com.google.common.annotations.VisibleForTesting;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
 import javax.persistence.MappedSuperclass;
@@ -30,6 +29,7 @@ import javax.xml.bind.annotation.XmlTransient;
 */
@MappedSuperclass
 public abstract class BackupGroupRoot extends ImmutableObject {
+
  /**
   * An automatically managed timestamp of when this object was last written to Datastore.
   *
@@ -40,7 +40,6 @@ public abstract class BackupGroupRoot extends ImmutableObject {
  // Prevents subclasses from unexpectedly accessing as property (e.g., HostResource), which would
  // require an unnecessary non-private setter method.
  @Access(AccessType.FIELD)
-  @VisibleForTesting
  UpdateAutoTimestamp updateTimestamp = UpdateAutoTimestamp.create(null);

  /** Get the {@link UpdateAutoTimestamp} for this entity. */
@@ -347,27 +347,11 @@ public abstract class BillingEvent extends ImmutableObject

    @Override
    public VKey<OneTime> createVKey() {
-      return VKey.create(
-          getClass(),
-          parent.getParent().getName() + "/" + parent.getId() + "/" + getId(),
-          Key.create(this));
+      return VKey.create(getClass(), getId(), Key.create(this));
    }

    public static VKey<OneTime> createVKey(Key<OneTime> key) {
-      // TODO(b/159207551): As it stands, the SQL key generated here doesn't mesh with the primary
-      // key type for the table, which is a single long integer.
-      if (key == null) {
-        return null;
-      }
-      Key parent = key.getParent();
-      Key grandparent = (parent != null) ? parent.getParent() : null;
-      String path =
-          (grandparent != null ? grandparent.getName() : "")
-              + "/"
-              + (parent != null ? parent.getId() : "")
-              + "/"
-              + key.getId();
-      return VKey.create(OneTime.class, path, key);
+      return VKey.create(OneTime.class, key.getId(), key);
    }

    @Override
@@ -505,21 +489,11 @@ public abstract class BillingEvent extends ImmutableObject

    @Override
    public VKey<Recurring> createVKey() {
-      return VKey.create(
-          getClass(),
-          parent.getParent().getName() + "/" + parent.getId() + "/" + getId(),
-          Key.create(this));
+      return VKey.create(getClass(), getId(), Key.create(this));
    }

    public static VKey<Recurring> createVKey(Key<Recurring> key) {
-      // TODO(b/159207551): As it stands, the SQL key generated here doesn't mesh with the primary
-      // key type for the table, which is a single long integer.
-      if (key == null) {
-        return null;
-      }
-      String path =
-          key.getParent().getParent().getName() + "/" + key.getParent().getId() + "/" + key.getId();
-      return VKey.create(Recurring.class, path, key);
+      return VKey.create(Recurring.class, key.getId(), key);
    }

    @Override
@@ -639,21 +613,11 @@ public abstract class BillingEvent extends ImmutableObject

    @Override
    public VKey<Cancellation> createVKey() {
-      return VKey.create(
-          getClass(),
-          parent.getParent().getName() + "/" + parent.getId() + "/" + getId(),
-          Key.create(this));
+      return VKey.create(getClass(), getId(), Key.create(this));
    }

    public static VKey<Cancellation> createVKey(Key<Cancellation> key) {
-      // TODO(b/159207551): As it stands, the SQL key generated here doesn't mesh with the primary
-      // key type for the table, which is a single long integer.
-      if (key == null) {
-        return null;
-      }
-      String path =
-          key.getParent().getParent().getName() + "/" + key.getParent().getId() + "/" + key.getId();
-      return VKey.create(Cancellation.class, path, key);
+      return VKey.create(Cancellation.class, key.getId(), key);
    }

    @Override
@@ -733,21 +697,11 @@ public abstract class BillingEvent extends ImmutableObject

    @Override
    public VKey<Modification> createVKey() {
-      return VKey.create(
-          getClass(),
-          parent.getParent().getName() + "/" + parent.getId() + "/" + getId(),
-          Key.create(this));
+      return VKey.create(getClass(), getId(), Key.create(this));
    }

    public static VKey<Modification> createVKey(Key<Modification> key) {
-      // TODO(b/159207551): As it stands, the SQL key generated here doesn't mesh with the primary
-      // key type for the table, which is a single long integer.
-      if (key == null) {
-        return null;
-      }
-      String path =
-          key.getParent().getParent().getName() + "/" + key.getParent().getId() + "/" + key.getId();
-      return VKey.create(Modification.class, path, key);
+      return VKey.create(Modification.class, key.getId(), key);
    }

    /**
@@ -44,10 +44,11 @@ import org.joda.time.DateTime;
 /**
 * A persistable contact resource including mutable and non-mutable fields.
 *
+ * <p>This class deliberately does not include an {@link javax.persistence.Id} so that any
+ * foreign-keyed fields can refer to the proper parent entity's ID, whether we're storing this in
+ * the DB itself or as part of another entity
+ *
 * @see <a href="https://tools.ietf.org/html/rfc5733">RFC 5733</a>
- *     <p>This class deliberately does not include an {@link javax.persistence.Id} so that any
- *     foreign-keyed fields can refer to the proper parent entity's ID, whether we're storing this
- *     in the DB itself or as part of another entity
 */
@MappedSuperclass
@Embeddable
@@ -25,8 +25,8 @@ import javax.persistence.Entity;
 * A persisted history entry representing an EPP modification to a contact.
 *
 * <p>In addition to the general history fields (e.g. action time, registrar ID) we also persist a
- * copy of the host entity at this point in time. We persist a raw {@link ContactBase} so that the
- * foreign-keyed fields in that class can refer to this object.
+ * copy of the contact entity at this point in time. We persist a raw {@link ContactBase} so that
+ * the foreign-keyed fields in that class can refer to this object.
 */
@Entity
@javax.persistence.Table(
@@ -14,79 +14,28 @@

 package google.registry.model.domain;

-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Strings.emptyToNull;
-import static com.google.common.collect.ImmutableSet.toImmutableSet;
-import static com.google.common.collect.ImmutableSortedSet.toImmutableSortedSet;
-import static com.google.common.collect.Sets.difference;
-import static com.google.common.collect.Sets.intersection;
-import static com.google.common.collect.Sets.union;
-import static google.registry.model.EppResourceUtils.projectResourceOntoBuilderAtTime;
-import static google.registry.model.EppResourceUtils.setAutomaticTransferSuccessProperties;
-import static google.registry.model.ofy.ObjectifyService.ofy;
-import static google.registry.util.CollectionUtils.forceEmptyToNull;
-import static google.registry.util.CollectionUtils.nullToEmpty;
-import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
-import static google.registry.util.CollectionUtils.union;
-import static google.registry.util.DateTimeUtils.END_OF_TIME;
-import static google.registry.util.DateTimeUtils.earliestOf;
-import static google.registry.util.DateTimeUtils.isBeforeOrAt;
-import static google.registry.util.DateTimeUtils.leapSafeAddYears;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
-import static google.registry.util.DomainNameUtils.getTldFromDomainName;
-import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

-import com.google.common.collect.ImmutableSet;
-import com.google.common.collect.ImmutableSortedSet;
-import com.google.common.collect.Ordering;
-import com.google.common.collect.Streams;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
-import com.googlecode.objectify.annotation.Ignore;
-import com.googlecode.objectify.annotation.IgnoreSave;
-import com.googlecode.objectify.annotation.Index;
-import com.googlecode.objectify.annotation.OnLoad;
-import com.googlecode.objectify.condition.IfNull;
-import google.registry.flows.ResourceFlowUtils;
 import google.registry.model.EppResource;
 import google.registry.model.EppResource.ForeignKeyedEppResource;
-import google.registry.model.EppResource.ResourceWithTransferData;
 import google.registry.model.annotations.ExternalMessagingName;
 import google.registry.model.annotations.ReportedOn;
-import google.registry.model.billing.BillingEvent;
-import google.registry.model.contact.ContactResource;
-import google.registry.model.domain.DesignatedContact.Type;
-import google.registry.model.domain.launch.LaunchNotice;
-import google.registry.model.domain.rgp.GracePeriodStatus;
-import google.registry.model.domain.secdns.DelegationSignerData;
-import google.registry.model.eppcommon.StatusValue;
 import google.registry.model.host.HostResource;
-import google.registry.model.poll.PollMessage;
-import google.registry.model.registry.Registry;
-import google.registry.model.transfer.DomainTransferData;
-import google.registry.model.transfer.TransferStatus;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithStringVKey;
 import google.registry.schema.replay.DatastoreAndSqlEntity;
-import google.registry.util.CollectionUtils;
-import java.util.HashSet;
-import java.util.Objects;
-import java.util.Optional;
 import java.util.Set;
-import java.util.function.Predicate;
-import javax.annotation.Nullable;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
-import javax.persistence.AttributeOverride;
-import javax.persistence.AttributeOverrides;
+import javax.persistence.CascadeType;
 import javax.persistence.Column;
 import javax.persistence.ElementCollection;
-import javax.persistence.Embedded;
+import javax.persistence.FetchType;
+import javax.persistence.JoinColumn;
 import javax.persistence.JoinTable;
-import javax.persistence.PostLoad;
-import javax.persistence.Transient;
+import javax.persistence.OneToMany;
 import org.joda.time.DateTime;
-import org.joda.time.Interval;

 /**
 * A persistable domain resource including mutable and non-mutable fields.
@@ -112,197 +61,8 @@ import org.joda.time.Interval;
@WithStringVKey
@ExternalMessagingName("domain")
@Access(AccessType.FIELD)
-public class DomainBase extends EppResource
-    implements DatastoreAndSqlEntity,
-        ForeignKeyedEppResource,
-        ResourceWithTransferData<DomainTransferData> {
-
-  /** The max number of years that a domain can be registered for, as set by ICANN policy. */
-  public static final int MAX_REGISTRATION_YEARS = 10;
-
-  /** Status values which prohibit DNS information from being published. */
-  private static final ImmutableSet<StatusValue> DNS_PUBLISHING_PROHIBITED_STATUSES =
-      ImmutableSet.of(
-          StatusValue.CLIENT_HOLD,
-          StatusValue.INACTIVE,
-          StatusValue.PENDING_DELETE,
-          StatusValue.SERVER_HOLD);
-
-  /**
-   * Fully qualified domain name (puny-coded), which serves as the foreign key for this domain.
-   *
-   * <p>This is only unique in the sense that for any given lifetime specified as the time range
-   * from (creationTime, deletionTime) there can only be one domain in Datastore with this name.
-   * However, there can be many domains with the same name and non-overlapping lifetimes.
-   *
-   * @invariant fullyQualifiedDomainName == fullyQualifiedDomainName.toLowerCase(Locale.ENGLISH)
-   */
-  // TODO(b/158858642): Rename this to domainName when we are off Datastore
-  @Column(name = "domainName")
-  @Index
-  String fullyQualifiedDomainName;
-
-  /** The top level domain this is under, dernormalized from {@link #fullyQualifiedDomainName}. */
-  @Index
-  String tld;
-
-  /** References to hosts that are the nameservers for the domain. */
-  @Index
-  @ElementCollection
-  @JoinTable(name = "DomainHost")
-  Set<VKey<HostResource>> nsHosts;
-
-  /**
-   * The union of the contacts visible via {@link #getContacts} and {@link #getRegistrant}.
-   *
-   * <p>These are stored in one field so that we can query across all contacts at once.
-   */
-  @Transient Set<DesignatedContact> allContacts;
-
-  /**
-   * Contacts as they are stored in cloud SQL.
-   *
-   * <p>This information is duplicated in allContacts, and must be kept in sync with it.
-   */
-  @Ignore VKey<ContactResource> adminContact;
-
-  @Ignore VKey<ContactResource> billingContact;
-  @Ignore VKey<ContactResource> techContact;
-  @Ignore VKey<ContactResource> registrantContact;
-
-  /** Authorization info (aka transfer secret) of the domain. */
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "pw.value", column = @Column(name = "auth_info_value")),
-    @AttributeOverride(name = "pw.repoId", column = @Column(name = "auth_info_repo_id")),
-  })
-  DomainAuthInfo authInfo;
-
-  /**
-   * Data used to construct DS records for this domain.
-   *
-   * <p>This is {@literal @}XmlTransient because it needs to be returned under the "extension" tag
-   * of an info response rather than inside the "infData" tag.
-   */
-  @Transient Set<DelegationSignerData> dsData;
-
-  /**
-   * The claims notice supplied when this application or domain was created, if there was one. It's
-   * {@literal @}XmlTransient because it's not returned in an info response.
-   */
-  @IgnoreSave(IfNull.class)
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "noticeId.tcnId", column = @Column(name = "launch_notice_tcn_id")),
-    @AttributeOverride(
-        name = "noticeId.validatorId",
-        column = @Column(name = "launch_notice_validator_id")),
-    @AttributeOverride(
-        name = "expirationTime",
-        column = @Column(name = "launch_notice_expiration_time")),
-    @AttributeOverride(
-        name = "acceptedTime",
-        column = @Column(name = "launch_notice_accepted_time")),
-  })
-  LaunchNotice launchNotice;
-
-  /**
-   * Name of first IDN table associated with TLD that matched the characters in this domain label.
-   *
-   * @see google.registry.tldconfig.idn.IdnLabelValidator#findValidIdnTableForTld
-   */
-  @IgnoreSave(IfNull.class)
-  String idnTableName;
-
-  /** Fully qualified host names of this domain's active subordinate hosts. */
-  Set<String> subordinateHosts;
-
-  /** When this domain's registration will expire. */
-  DateTime registrationExpirationTime;
-
-  /**
-   * The poll message associated with this domain being deleted.
-   *
-   * <p>This field should be null if the domain is not in pending delete. If it is, the field should
-   * refer to a {@link PollMessage} timed to when the domain is fully deleted. If the domain is
-   * restored, the message should be deleted.
-   */
-  @Column(name = "deletion_poll_message_id")
-  VKey<PollMessage.OneTime> deletePollMessage;
-
-  /**
-   * The recurring billing event associated with this domain's autorenewals.
-   *
-   * <p>The recurrence should be open ended unless the domain is in pending delete or fully deleted,
-   * in which case it should be closed at the time the delete was requested. Whenever the domain's
-   * {@link #registrationExpirationTime} is changed the recurrence should be closed, a new one
-   * should be created, and this field should be updated to point to the new one.
-   */
-  @Column(name = "billing_recurrence_id")
-  VKey<BillingEvent.Recurring> autorenewBillingEvent;
-
-  /**
-   * The recurring poll message associated with this domain's autorenewals.
-   *
-   * <p>The recurrence should be open ended unless the domain is in pending delete or fully deleted,
-   * in which case it should be closed at the time the delete was requested. Whenever the domain's
-   * {@link #registrationExpirationTime} is changed the recurrence should be closed, a new one
-   * should be created, and this field should be updated to point to the new one.
-   */
-  @Column(name = "autorenew_poll_message_id")
-  VKey<PollMessage.Autorenew> autorenewPollMessage;
-
-  /** The unexpired grace periods for this domain (some of which may not be active yet). */
-  @Transient @ElementCollection Set<GracePeriod> gracePeriods;
-
-  /**
-   * The id of the signed mark that was used to create this domain in sunrise.
-   *
-   * <p>Will only be populated for domains created in sunrise.
-   */
-  @IgnoreSave(IfNull.class)
-  String smdId;
-
-  /** Data about any pending or past transfers on this domain. */
-  DomainTransferData transferData;
-
-  /**
-   * The time that this resource was last transferred.
-   *
-   * <p>Can be null if the resource has never been transferred.
-   */
-  DateTime lastTransferTime;
-
-  @OnLoad
-  void load() {
-    // Reconstitute all of the contacts so that they have VKeys.
-    allContacts =
-        allContacts.stream().map(contact -> contact.reconstitute()).collect(toImmutableSet());
-    setContactFields(allContacts, true);
-  }
-
-  @PostLoad
-  void postLoad() {
-    // Reconstitute the contact list.
-    ImmutableSet.Builder<DesignatedContact> contactsBuilder =
-        new ImmutableSet.Builder<DesignatedContact>();
-
-    if (registrantContact != null) {
-      contactsBuilder.add(
-          DesignatedContact.create(DesignatedContact.Type.REGISTRANT, registrantContact));
-    }
-    if (billingContact != null) {
-      contactsBuilder.add(DesignatedContact.create(DesignatedContact.Type.BILLING, billingContact));
-    }
-    if (techContact != null) {
-      contactsBuilder.add(DesignatedContact.create(DesignatedContact.Type.TECH, techContact));
-    }
-    if (adminContact != null) {
-      contactsBuilder.add(DesignatedContact.create(DesignatedContact.Type.ADMIN, adminContact));
-    }
-
-    allContacts = contactsBuilder.build();
-  }
+public class DomainBase extends DomainContent
+    implements DatastoreAndSqlEntity, ForeignKeyedEppResource {

  @Override
  @javax.persistence.Id
@@ -311,326 +71,23 @@ public class DomainBase extends EppResource
    return super.getRepoId();
  }

-  public ImmutableSet<String> getSubordinateHosts() {
-    return nullToEmptyImmutableCopy(subordinateHosts);
+  @ElementCollection
+  @JoinTable(name = "DomainHost")
+  @Access(AccessType.PROPERTY)
+  @Column(name = "host_repo_id")
+  public Set<VKey<HostResource>> getNsHosts() {
+    return super.nsHosts;
  }

-  public DateTime getRegistrationExpirationTime() {
-    return registrationExpirationTime;
-  }
-
-  public VKey<PollMessage.OneTime> getDeletePollMessage() {
-    return deletePollMessage;
-  }
-
-  public VKey<BillingEvent.Recurring> getAutorenewBillingEvent() {
-    return autorenewBillingEvent;
-  }
-
-  public VKey<PollMessage.Autorenew> getAutorenewPollMessage() {
-    return autorenewPollMessage;
-  }
-
-  public ImmutableSet<GracePeriod> getGracePeriods() {
-    return nullToEmptyImmutableCopy(gracePeriods);
-  }
-
-  public String getSmdId() {
-    return smdId;
-  }
-
-  @Override
-  public DomainTransferData getTransferData() {
-    return Optional.ofNullable(transferData).orElse(DomainTransferData.EMPTY);
-  }
-
-  @Override
-  public DateTime getLastTransferTime() {
-    return lastTransferTime;
-  }
-
-  @Override
-  public String getForeignKey() {
-    return fullyQualifiedDomainName;
-  }
-
-  public String getDomainName() {
-    return fullyQualifiedDomainName;
-  }
-
-  public ImmutableSet<DelegationSignerData> getDsData() {
-    return nullToEmptyImmutableCopy(dsData);
-  }
-
-  public LaunchNotice getLaunchNotice() {
-    return launchNotice;
-  }
-
-  public String getIdnTableName() {
-    return idnTableName;
-  }
-
-  public ImmutableSet<VKey<HostResource>> getNameservers() {
-    return nullToEmptyImmutableCopy(nsHosts);
-  }
-
-  public final String getCurrentSponsorClientId() {
-    return getPersistedCurrentSponsorClientId();
-  }
-
-  /** Returns true if DNS information should be published for the given domain. */
-  public boolean shouldPublishToDns() {
-    return intersection(getStatusValues(), DNS_PUBLISHING_PROHIBITED_STATUSES).isEmpty();
-  }
-
-  /**
-   * Returns the Registry Grace Period Statuses for this domain.
-   *
-   * <p>This collects all statuses from the domain's {@link GracePeriod} entries and also adds the
-   * PENDING_DELETE status if needed.
-   */
-  public ImmutableSet<GracePeriodStatus> getGracePeriodStatuses() {
-    Set<GracePeriodStatus> gracePeriodStatuses = new HashSet<>();
-    for (GracePeriod gracePeriod : getGracePeriods()) {
-      gracePeriodStatuses.add(gracePeriod.getType());
-    }
-    if (getStatusValues().contains(StatusValue.PENDING_DELETE)
-        && !gracePeriodStatuses.contains(GracePeriodStatus.REDEMPTION)) {
-      gracePeriodStatuses.add(GracePeriodStatus.PENDING_DELETE);
-    }
-    return ImmutableSet.copyOf(gracePeriodStatuses);
-  }
-
-  /** Returns the subset of grace periods having the specified type. */
-  public ImmutableSet<GracePeriod> getGracePeriodsOfType(GracePeriodStatus gracePeriodType) {
-    ImmutableSet.Builder<GracePeriod> builder = new ImmutableSet.Builder<>();
-    for (GracePeriod gracePeriod : getGracePeriods()) {
-      if (gracePeriod.getType() == gracePeriodType) {
-        builder.add(gracePeriod);
-      }
-    }
-    return builder.build();
-  }
-
-  /**
-   * The logic in this method, which handles implicit server approval of transfers, very closely
-   * parallels the logic in {@code DomainTransferApproveFlow} which handles explicit client
-   * approvals.
-   */
-  @Override
-  public DomainBase cloneProjectedAtTime(final DateTime now) {
-
-    DomainTransferData transferData = getTransferData();
-    DateTime transferExpirationTime = transferData.getPendingTransferExpirationTime();
-
-    // If there's a pending transfer that has expired, handle it.
-    if (TransferStatus.PENDING.equals(transferData.getTransferStatus())
-        && isBeforeOrAt(transferExpirationTime, now)) {
-      // Project until just before the transfer time. This will handle the case of an autorenew
-      // before the transfer was even requested or during the request period.
-      // If the transfer time is precisely the moment that the domain expires, there will not be an
-      // autorenew billing event (since we end the recurrence at transfer time and recurrences are
-      // exclusive of their ending), and we can just proceed with the transfer.
-      DomainBase domainAtTransferTime =
-          cloneProjectedAtTime(transferExpirationTime.minusMillis(1));
-
-      DateTime expirationDate = transferData.getTransferredRegistrationExpirationTime();
-      if (expirationDate == null) {
-        // Extend the registration by the correct number of years from the expiration time
-        // that was current on the domain right before the transfer, capped at 10 years from
-        // the moment of the transfer.
-        expirationDate =
-            ResourceFlowUtils.computeExDateForApprovalTime(
-                domainAtTransferTime, transferExpirationTime, transferData.getTransferPeriod());
-      }
-      // If we are within an autorenew grace period, the transfer will subsume the autorenew. There
-      // will already be a cancellation written in advance by the transfer request flow, so we don't
-      // need to worry about billing, but we do need to cancel out the expiration time increase.
-      // The transfer period saved in the transfer data will be one year, unless the superuser
-      // extension set the transfer period to zero.
-      // Set the expiration, autorenew events, and grace period for the transfer. (Transfer ends
-      // all other graces).
-      Builder builder =
-          domainAtTransferTime
-              .asBuilder()
-              .setRegistrationExpirationTime(expirationDate)
-              // Set the speculatively-written new autorenew events as the domain's autorenew
-              // events.
-              .setAutorenewBillingEvent(transferData.getServerApproveAutorenewEvent())
-              .setAutorenewPollMessage(transferData.getServerApproveAutorenewPollMessage());
-      if (transferData.getTransferPeriod().getValue() == 1) {
-        // Set the grace period using a key to the prescheduled transfer billing event.  Not using
-        // GracePeriod.forBillingEvent() here in order to avoid the actual Datastore fetch.
-        builder.setGracePeriods(
-            ImmutableSet.of(
-                GracePeriod.create(
-                    GracePeriodStatus.TRANSFER,
-                    transferExpirationTime.plus(
-                        Registry.get(getTld()).getTransferGracePeriodLength()),
-                    transferData.getGainingClientId(),
-                    transferData.getServerApproveBillingEvent())));
-      } else {
-        // There won't be a billing event, so we don't need a grace period
-        builder.setGracePeriods(ImmutableSet.of());
-      }
-      // Set all remaining transfer properties.
-      setAutomaticTransferSuccessProperties(builder, transferData);
-      builder
-          .setLastEppUpdateTime(transferExpirationTime)
-          .setLastEppUpdateClientId(transferData.getGainingClientId());
-      // Finish projecting to now.
-      return builder.build().cloneProjectedAtTime(now);
-    }
-
-    Optional<DateTime> newLastEppUpdateTime = Optional.empty();
-
-    // There is no transfer. Do any necessary autorenews for active domains.
-
-    Builder builder = asBuilder();
-    if (isBeforeOrAt(registrationExpirationTime, now) && END_OF_TIME.equals(getDeletionTime())) {
-      // Autorenew by the number of years between the old expiration time and now.
-      DateTime lastAutorenewTime = leapSafeAddYears(
-          registrationExpirationTime,
-          new Interval(registrationExpirationTime, now).toPeriod().getYears());
-      DateTime newExpirationTime  = lastAutorenewTime.plusYears(1);
-      builder
-          .setRegistrationExpirationTime(newExpirationTime)
-          .addGracePeriod(
-              GracePeriod.createForRecurring(
-                  GracePeriodStatus.AUTO_RENEW,
-                  lastAutorenewTime.plus(Registry.get(getTld()).getAutoRenewGracePeriodLength()),
-                  getCurrentSponsorClientId(),
-                  autorenewBillingEvent));
-      newLastEppUpdateTime = Optional.of(lastAutorenewTime);
-    }
-
-    // Remove any grace periods that have expired.
-    DomainBase almostBuilt = builder.build();
-    builder = almostBuilt.asBuilder();
-    for (GracePeriod gracePeriod : almostBuilt.getGracePeriods()) {
-      if (isBeforeOrAt(gracePeriod.getExpirationTime(), now)) {
-        builder.removeGracePeriod(gracePeriod);
-        if (!newLastEppUpdateTime.isPresent()
-            || isBeforeOrAt(newLastEppUpdateTime.get(), gracePeriod.getExpirationTime())) {
-          newLastEppUpdateTime = Optional.of(gracePeriod.getExpirationTime());
-        }
-      }
-    }
-
-    // It is possible that the lastEppUpdateClientId is different from current sponsor client
-    // id, so we have to do the comparison instead of having one variable just storing the most
-    // recent time.
-    if (newLastEppUpdateTime.isPresent()) {
-      if (getLastEppUpdateTime() == null
-          || newLastEppUpdateTime.get().isAfter(getLastEppUpdateTime())) {
-        builder
-            .setLastEppUpdateTime(newLastEppUpdateTime.get())
-            .setLastEppUpdateClientId(getCurrentSponsorClientId());
-      }
-    }
-
-    // Handle common properties like setting or unsetting linked status. This also handles the
-    // general case of pending transfers for other resource types, but since we've always handled
-    // a pending transfer by this point that's a no-op for domains.
-    projectResourceOntoBuilderAtTime(almostBuilt, builder, now);
-    return builder.build();
-  }
-
-  /** Return what the expiration time would be if the given number of years were added to it. */
-  public static DateTime extendRegistrationWithCap(
-      DateTime now,
-      DateTime currentExpirationTime,
-      @Nullable Integer extendedRegistrationYears) {
-    // We must cap registration at the max years (aka 10), even if that truncates the last year.
-    return earliestOf(
-        leapSafeAddYears(
-            currentExpirationTime,
-            Optional.ofNullable(extendedRegistrationYears).orElse(0)),
-        leapSafeAddYears(now, MAX_REGISTRATION_YEARS));
-  }
-
-  /** Loads and returns the fully qualified host names of all linked nameservers. */
-  public ImmutableSortedSet<String> loadNameserverHostNames() {
-    return ofy()
-        .load()
-        .keys(getNameservers().stream().map(VKey::getOfyKey).collect(toImmutableSet()))
-        .values()
-        .stream()
-        .map(HostResource::getHostName)
-        .collect(toImmutableSortedSet(Ordering.natural()));
-  }
-
-  /** A key to the registrant who registered this domain. */
-  public VKey<ContactResource> getRegistrant() {
-    return registrantContact;
-  }
-
-  public VKey<ContactResource> getAdminContact() {
-    return adminContact;
-  }
-
-  public VKey<ContactResource> getBillingContact() {
-    return billingContact;
-  }
-
-  public VKey<ContactResource> getTechContact() {
-    return techContact;
-  }
-
-  /** Associated contacts for the domain (other than registrant). */
-  public ImmutableSet<DesignatedContact> getContacts() {
-    return nullToEmpty(allContacts)
-        .stream()
-        .filter(IS_REGISTRANT.negate())
-        .collect(toImmutableSet());
-  }
-
-  public DomainAuthInfo getAuthInfo() {
-    return authInfo;
-  }
-
-  /** Returns all referenced contacts from this domain or application. */
-  public ImmutableSet<VKey<ContactResource>> getReferencedContacts() {
-    return nullToEmptyImmutableCopy(allContacts)
-        .stream()
-        .map(DesignatedContact::getContactKey)
-        .filter(Objects::nonNull)
-        .collect(toImmutableSet());
-  }
-
-  public String getTld() {
-    return tld;
-  }
-
-  /**
-   * Sets the individual contact fields from {@code contacts}.
-   *
-   * <p>The registrant field is only set if {@code includeRegistrant} is true, as this field needs
-   * to be set in some circumstances but not in others.
-   */
-  private void setContactFields(Set<DesignatedContact> contacts, boolean includeRegistrant) {
-
-    // Set the individual contact fields.
-    for (DesignatedContact contact : contacts) {
-      switch (contact.getType()) {
-        case BILLING:
-          billingContact = contact.getContactKey();
-          break;
-        case TECH:
-          techContact = contact.getContactKey();
-          break;
-        case ADMIN:
-          adminContact = contact.getContactKey();
-          break;
-        case REGISTRANT:
-          if (includeRegistrant) {
-            registrantContact = contact.getContactKey();
-          }
-          break;
-        default:
-          throw new IllegalArgumentException("Unknown contact resource type: " + contact.getType());
-      }
-    }
+  @Access(AccessType.PROPERTY)
+  @OneToMany(
+      cascade = {CascadeType.ALL},
+      fetch = FetchType.EAGER,
+      orphanRemoval = true)
+  @JoinColumn(name = "domainRepoId", referencedColumnName = "repoId")
+  @SuppressWarnings("UnusedMethod")
+  private Set<GracePeriod> getInternalGracePeriods() {
+    return gracePeriods;
  }

  @Override
@@ -638,9 +95,14 @@ public class DomainBase extends EppResource
    return VKey.create(DomainBase.class, getRepoId(), Key.create(this));
  }

-  /** Predicate to determine if a given {@link DesignatedContact} is the registrant. */
-  private static final Predicate<DesignatedContact> IS_REGISTRANT =
-      (DesignatedContact contact) -> DesignatedContact.Type.REGISTRANT.equals(contact.type);
+  @Override
+  public DomainBase cloneProjectedAtTime(final DateTime now) {
+    return cloneDomainProjectedAtTime(this, now);
+  }
+
+  public static VKey<DomainBase> createVKey(Key key) {
+    return VKey.create(DomainBase.class, key.getName(), key);
+  }

  /** An override of {@link EppResource#asBuilder} with tighter typing. */
  @Override
@@ -649,199 +111,12 @@ public class DomainBase extends EppResource
  }

  /** A builder for constructing {@link DomainBase}, since it is immutable. */
-  public static class Builder extends EppResource.Builder<DomainBase, Builder>
-      implements BuilderWithTransferData<DomainTransferData, Builder> {
+  public static class Builder extends DomainContent.Builder<DomainBase, Builder> {

    public Builder() {}

    Builder(DomainBase instance) {
      super(instance);
    }
-
-    @Override
-    public DomainBase build() {
-      DomainBase instance = getInstance();
-      // If TransferData is totally empty, set it to null.
-      if (DomainTransferData.EMPTY.equals(getInstance().transferData)) {
-        setTransferData(null);
-      }
-      // A DomainBase has status INACTIVE if there are no nameservers.
-      if (getInstance().getNameservers().isEmpty()) {
-        addStatusValue(StatusValue.INACTIVE);
-      } else {  // There are nameservers, so make sure INACTIVE isn't there.
-        removeStatusValue(StatusValue.INACTIVE);
-      }
-
-      checkArgumentNotNull(emptyToNull(instance.fullyQualifiedDomainName), "Missing domainName");
-      if (instance.getRegistrant() == null
-          && instance.allContacts.stream().anyMatch(IS_REGISTRANT)) {
-        throw new IllegalArgumentException("registrant is null but is in allContacts");
-      }
-      checkArgumentNotNull(instance.getRegistrant(), "Missing registrant");
-      instance.tld = getTldFromDomainName(instance.fullyQualifiedDomainName);
-      return super.build();
-    }
-
-    public Builder setDomainName(String domainName) {
-      checkArgument(
-          domainName.equals(canonicalizeDomainName(domainName)),
-          "Domain name must be in puny-coded, lower-case form");
-      getInstance().fullyQualifiedDomainName = domainName;
-      return thisCastToDerived();
-    }
-
-    public Builder setDsData(ImmutableSet<DelegationSignerData> dsData) {
-      getInstance().dsData = dsData;
-      return thisCastToDerived();
-    }
-
-    public Builder setRegistrant(VKey<ContactResource> registrant) {
-      // Replace the registrant contact inside allContacts.
-      getInstance().allContacts = union(
-          getInstance().getContacts(),
-          DesignatedContact.create(Type.REGISTRANT, checkArgumentNotNull(registrant)));
-
-      // Set the registrant field specifically.
-      getInstance().registrantContact = registrant;
-      return thisCastToDerived();
-    }
-
-    public Builder setAuthInfo(DomainAuthInfo authInfo) {
-      getInstance().authInfo = authInfo;
-      return thisCastToDerived();
-    }
-
-    public Builder setNameservers(VKey<HostResource> nameserver) {
-      getInstance().nsHosts = ImmutableSet.of(nameserver);
-      return thisCastToDerived();
-    }
-
-    public Builder setNameservers(ImmutableSet<VKey<HostResource>> nameservers) {
-      getInstance().nsHosts = forceEmptyToNull(nameservers);
-      return thisCastToDerived();
-    }
-
-    public Builder addNameserver(VKey<HostResource> nameserver) {
-      return addNameservers(ImmutableSet.of(nameserver));
-    }
-
-    public Builder addNameservers(ImmutableSet<VKey<HostResource>> nameservers) {
-      return setNameservers(
-          ImmutableSet.copyOf(union(getInstance().getNameservers(), nameservers)));
-    }
-
-    public Builder removeNameserver(VKey<HostResource> nameserver) {
-      return removeNameservers(ImmutableSet.of(nameserver));
-    }
-
-    public Builder removeNameservers(ImmutableSet<VKey<HostResource>> nameservers) {
-      return setNameservers(
-          ImmutableSet.copyOf(difference(getInstance().getNameservers(), nameservers)));
-    }
-
-    public Builder setContacts(DesignatedContact contact) {
-      return setContacts(ImmutableSet.of(contact));
-    }
-
-    public Builder setContacts(ImmutableSet<DesignatedContact> contacts) {
-      checkArgument(contacts.stream().noneMatch(IS_REGISTRANT), "Registrant cannot be a contact");
-
-      // Replace the non-registrant contacts inside allContacts.
-      getInstance().allContacts =
-          Streams.concat(
-                  nullToEmpty(getInstance().allContacts).stream().filter(IS_REGISTRANT),
-                  contacts.stream())
-              .collect(toImmutableSet());
-
-      // Set the individual fields.
-      getInstance().setContactFields(contacts, false);
-      return thisCastToDerived();
-    }
-
-    public Builder addContacts(ImmutableSet<DesignatedContact> contacts) {
-      return setContacts(ImmutableSet.copyOf(union(getInstance().getContacts(), contacts)));
-    }
-
-    public Builder removeContacts(ImmutableSet<DesignatedContact> contacts) {
-      return setContacts(ImmutableSet.copyOf(difference(getInstance().getContacts(), contacts)));
-    }
-
-    public Builder setLaunchNotice(LaunchNotice launchNotice) {
-      getInstance().launchNotice = launchNotice;
-      return thisCastToDerived();
-    }
-
-    public Builder setIdnTableName(String idnTableName) {
-      getInstance().idnTableName = idnTableName;
-      return thisCastToDerived();
-    }
-
-    public Builder setSubordinateHosts(ImmutableSet<String> subordinateHosts) {
-      getInstance().subordinateHosts = subordinateHosts;
-      return thisCastToDerived();
-    }
-
-    public Builder addSubordinateHost(String hostToAdd) {
-      return setSubordinateHosts(ImmutableSet.copyOf(
-          union(getInstance().getSubordinateHosts(), hostToAdd)));
-    }
-
-    public Builder removeSubordinateHost(String hostToRemove) {
-      return setSubordinateHosts(ImmutableSet.copyOf(
-          CollectionUtils.difference(getInstance().getSubordinateHosts(), hostToRemove)));
-    }
-
-    public Builder setRegistrationExpirationTime(DateTime registrationExpirationTime) {
-      getInstance().registrationExpirationTime = registrationExpirationTime;
-      return this;
-    }
-
-    public Builder setDeletePollMessage(VKey<PollMessage.OneTime> deletePollMessage) {
-      getInstance().deletePollMessage = deletePollMessage;
-      return this;
-    }
-
-    public Builder setAutorenewBillingEvent(VKey<BillingEvent.Recurring> autorenewBillingEvent) {
-      getInstance().autorenewBillingEvent = autorenewBillingEvent;
-      return this;
-    }
-
-    public Builder setAutorenewPollMessage(VKey<PollMessage.Autorenew> autorenewPollMessage) {
-      getInstance().autorenewPollMessage = autorenewPollMessage;
-      return this;
-    }
-
-    public Builder setSmdId(String smdId) {
-      getInstance().smdId = smdId;
-      return this;
-    }
-
-    public Builder setGracePeriods(ImmutableSet<GracePeriod> gracePeriods) {
-      getInstance().gracePeriods = gracePeriods;
-      return this;
-    }
-
-    public Builder addGracePeriod(GracePeriod gracePeriod) {
-      getInstance().gracePeriods = union(getInstance().getGracePeriods(), gracePeriod);
-      return this;
-    }
-
-    public Builder removeGracePeriod(GracePeriod gracePeriod) {
-      getInstance().gracePeriods = CollectionUtils
-          .difference(getInstance().getGracePeriods(), gracePeriod);
-      return this;
-    }
-
-    @Override
-    public Builder setTransferData(DomainTransferData transferData) {
-      getInstance().transferData = transferData;
-      return thisCastToDerived();
-    }
-
-    @Override
-    public Builder setLastTransferTime(DateTime lastTransferTime) {
-      getInstance().lastTransferTime = lastTransferTime;
-      return thisCastToDerived();
-    }
  }
 }
@@ -0,0 +1,853 @@
+// Copyright 2017 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.domain;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Strings.emptyToNull;
+import static com.google.common.collect.ImmutableSet.toImmutableSet;
+import static com.google.common.collect.ImmutableSortedSet.toImmutableSortedSet;
+import static com.google.common.collect.Sets.difference;
+import static com.google.common.collect.Sets.intersection;
+import static google.registry.model.EppResourceUtils.projectResourceOntoBuilderAtTime;
+import static google.registry.model.EppResourceUtils.setAutomaticTransferSuccessProperties;
+import static google.registry.model.ofy.ObjectifyService.ofy;
+import static google.registry.util.CollectionUtils.forceEmptyToNull;
+import static google.registry.util.CollectionUtils.nullToEmpty;
+import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
+import static google.registry.util.CollectionUtils.union;
+import static google.registry.util.DateTimeUtils.END_OF_TIME;
+import static google.registry.util.DateTimeUtils.earliestOf;
+import static google.registry.util.DateTimeUtils.isBeforeOrAt;
+import static google.registry.util.DateTimeUtils.leapSafeAddYears;
+import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
+import static google.registry.util.DomainNameUtils.getTldFromDomainName;
+import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedSet;
+import com.google.common.collect.Ordering;
+import com.google.common.collect.Sets;
+import com.google.common.collect.Streams;
+import com.googlecode.objectify.Key;
+import com.googlecode.objectify.annotation.Ignore;
+import com.googlecode.objectify.annotation.IgnoreSave;
+import com.googlecode.objectify.annotation.Index;
+import com.googlecode.objectify.annotation.OnLoad;
+import com.googlecode.objectify.condition.IfNull;
+import google.registry.flows.ResourceFlowUtils;
+import google.registry.model.EppResource;
+import google.registry.model.EppResource.ResourceWithTransferData;
+import google.registry.model.billing.BillingEvent;
+import google.registry.model.contact.ContactResource;
+import google.registry.model.domain.launch.LaunchNotice;
+import google.registry.model.domain.rgp.GracePeriodStatus;
+import google.registry.model.domain.secdns.DelegationSignerData;
+import google.registry.model.eppcommon.StatusValue;
+import google.registry.model.host.HostResource;
+import google.registry.model.poll.PollMessage;
+import google.registry.model.registry.Registry;
+import google.registry.model.transfer.DomainTransferData;
+import google.registry.model.transfer.TransferStatus;
+import google.registry.persistence.VKey;
+import google.registry.util.CollectionUtils;
+import java.util.HashSet;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.function.Predicate;
+import javax.annotation.Nullable;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.AttributeOverride;
+import javax.persistence.AttributeOverrides;
+import javax.persistence.Column;
+import javax.persistence.Embeddable;
+import javax.persistence.Embedded;
+import javax.persistence.MappedSuperclass;
+import javax.persistence.PostLoad;
+import javax.persistence.Transient;
+import org.joda.time.DateTime;
+import org.joda.time.Interval;
+
+/**
+ * A persistable domain resource including mutable and non-mutable fields.
+ *
+ * <p>This class deliberately does not include an {@link javax.persistence.Id} so that any
+ * foreign-keyed fields can refer to the proper parent entity's ID, whether we're storing this in
+ * the DB itself or as part of another entity.
+ *
+ * <p>For historical reasons, the name of this class is "DomainContent". Ideally it would be
+ * "DomainBase" for parallelism with the other {@link EppResource} entity classes, but because that
+ * name is already taken by {@link DomainBase} (also for historical reasons), we can't use it. Once
+ * we are no longer on Datastore, we can rename the classes.
+ *
+ * @see <a href="https://tools.ietf.org/html/rfc5731">RFC 5731</a>
+ */
+@MappedSuperclass
+@Embeddable
+@Access(AccessType.FIELD)
+public class DomainContent extends EppResource
+    implements ResourceWithTransferData<DomainTransferData> {
+
+  /** The max number of years that a domain can be registered for, as set by ICANN policy. */
+  public static final int MAX_REGISTRATION_YEARS = 10;
+
+  /** Status values which prohibit DNS information from being published. */
+  private static final ImmutableSet<StatusValue> DNS_PUBLISHING_PROHIBITED_STATUSES =
+      ImmutableSet.of(
+          StatusValue.CLIENT_HOLD,
+          StatusValue.INACTIVE,
+          StatusValue.PENDING_DELETE,
+          StatusValue.SERVER_HOLD);
+
+  /**
+   * Fully qualified domain name (puny-coded), which serves as the foreign key for this domain.
+   *
+   * <p>This is only unique in the sense that for any given lifetime specified as the time range
+   * from (creationTime, deletionTime) there can only be one domain in Datastore with this name.
+   * However, there can be many domains with the same name and non-overlapping lifetimes.
+   *
+   * @invariant fullyQualifiedDomainName == fullyQualifiedDomainName.toLowerCase(Locale.ENGLISH)
+   */
+  // TODO(b/158858642): Rename this to domainName when we are off Datastore
+  @Column(name = "domainName")
+  @Index
+  String fullyQualifiedDomainName;
+
+  /** The top level domain this is under, dernormalized from {@link #fullyQualifiedDomainName}. */
+  @Index String tld;
+
+  /** References to hosts that are the nameservers for the domain. */
+  @Index @Transient Set<VKey<HostResource>> nsHosts;
+
+  /**
+   * The union of the contacts visible via {@link #getContacts} and {@link #getRegistrant}.
+   *
+   * <p>These are stored in one field so that we can query across all contacts at once.
+   */
+  @Transient Set<DesignatedContact> allContacts;
+
+  /**
+   * Contacts as they are stored in cloud SQL.
+   *
+   * <p>This information is duplicated in allContacts, and must be kept in sync with it.
+   */
+  @Ignore VKey<ContactResource> adminContact;
+
+  @Ignore VKey<ContactResource> billingContact;
+  @Ignore VKey<ContactResource> techContact;
+  @Ignore VKey<ContactResource> registrantContact;
+
+  /** Authorization info (aka transfer secret) of the domain. */
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "pw.value", column = @Column(name = "auth_info_value")),
+    @AttributeOverride(name = "pw.repoId", column = @Column(name = "auth_info_repo_id")),
+  })
+  DomainAuthInfo authInfo;
+
+  /**
+   * Data used to construct DS records for this domain.
+   *
+   * <p>This is {@literal @}XmlTransient because it needs to be returned under the "extension" tag
+   * of an info response rather than inside the "infData" tag.
+   */
+  @Transient Set<DelegationSignerData> dsData;
+
+  /**
+   * The claims notice supplied when this application or domain was created, if there was one. It's
+   * {@literal @}XmlTransient because it's not returned in an info response.
+   */
+  @IgnoreSave(IfNull.class)
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "noticeId.tcnId", column = @Column(name = "launch_notice_tcn_id")),
+    @AttributeOverride(
+        name = "noticeId.validatorId",
+        column = @Column(name = "launch_notice_validator_id")),
+    @AttributeOverride(
+        name = "expirationTime",
+        column = @Column(name = "launch_notice_expiration_time")),
+    @AttributeOverride(
+        name = "acceptedTime",
+        column = @Column(name = "launch_notice_accepted_time")),
+  })
+  LaunchNotice launchNotice;
+
+  /**
+   * Name of first IDN table associated with TLD that matched the characters in this domain label.
+   *
+   * @see google.registry.tldconfig.idn.IdnLabelValidator#findValidIdnTableForTld
+   */
+  @IgnoreSave(IfNull.class)
+  String idnTableName;
+
+  /** Fully qualified host names of this domain's active subordinate hosts. */
+  Set<String> subordinateHosts;
+
+  /** When this domain's registration will expire. */
+  DateTime registrationExpirationTime;
+
+  /**
+   * The poll message associated with this domain being deleted.
+   *
+   * <p>This field should be null if the domain is not in pending delete. If it is, the field should
+   * refer to a {@link PollMessage} timed to when the domain is fully deleted. If the domain is
+   * restored, the message should be deleted.
+   */
+  @Column(name = "deletion_poll_message_id")
+  VKey<PollMessage.OneTime> deletePollMessage;
+
+  /**
+   * The recurring billing event associated with this domain's autorenewals.
+   *
+   * <p>The recurrence should be open ended unless the domain is in pending delete or fully deleted,
+   * in which case it should be closed at the time the delete was requested. Whenever the domain's
+   * {@link #registrationExpirationTime} is changed the recurrence should be closed, a new one
+   * should be created, and this field should be updated to point to the new one.
+   */
+  @Column(name = "billing_recurrence_id")
+  VKey<BillingEvent.Recurring> autorenewBillingEvent;
+
+  /**
+   * The recurring poll message associated with this domain's autorenewals.
+   *
+   * <p>The recurrence should be open ended unless the domain is in pending delete or fully deleted,
+   * in which case it should be closed at the time the delete was requested. Whenever the domain's
+   * {@link #registrationExpirationTime} is changed the recurrence should be closed, a new one
+   * should be created, and this field should be updated to point to the new one.
+   */
+  @Column(name = "autorenew_poll_message_id")
+  VKey<PollMessage.Autorenew> autorenewPollMessage;
+
+  /** The unexpired grace periods for this domain (some of which may not be active yet). */
+  @Transient Set<GracePeriod> gracePeriods;
+
+  /**
+   * The id of the signed mark that was used to create this domain in sunrise.
+   *
+   * <p>Will only be populated for domains created in sunrise.
+   */
+  @IgnoreSave(IfNull.class)
+  String smdId;
+
+  /** Data about any pending or past transfers on this domain. */
+  DomainTransferData transferData;
+
+  /**
+   * The time that this resource was last transferred.
+   *
+   * <p>Can be null if the resource has never been transferred.
+   */
+  DateTime lastTransferTime;
+
+  @OnLoad
+  void load() {
+    // Reconstitute all of the contacts so that they have VKeys.
+    allContacts =
+        allContacts.stream().map(contact -> contact.reconstitute()).collect(toImmutableSet());
+    setContactFields(allContacts, true);
+
+    // We have to return the cloned object here because the original object's
+    // hashcode is not correct due to the change to its domainRepoId. The cloned
+    // object will have a null hashcode so that it can get a recalculated hashcode
+    // when its hashCode() is invoked.
+    // TODO(b/162739503): Remove this after fully migrating to Cloud SQL.
+    if (gracePeriods != null) {
+      gracePeriods =
+          gracePeriods.stream()
+              .map(gracePeriod -> gracePeriod.cloneWithDomainRepoId(getRepoId()))
+              .collect(toImmutableSet());
+    }
+  }
+
+  @PostLoad
+  void postLoad() {
+    // Reconstitute the contact list.
+    ImmutableSet.Builder<DesignatedContact> contactsBuilder =
+        new ImmutableSet.Builder<DesignatedContact>();
+
+    if (registrantContact != null) {
+      contactsBuilder.add(
+          DesignatedContact.create(DesignatedContact.Type.REGISTRANT, registrantContact));
+    }
+    if (billingContact != null) {
+      contactsBuilder.add(DesignatedContact.create(DesignatedContact.Type.BILLING, billingContact));
+    }
+    if (techContact != null) {
+      contactsBuilder.add(DesignatedContact.create(DesignatedContact.Type.TECH, techContact));
+    }
+    if (adminContact != null) {
+      contactsBuilder.add(DesignatedContact.create(DesignatedContact.Type.ADMIN, adminContact));
+    }
+
+    allContacts = contactsBuilder.build();
+  }
+
+  public ImmutableSet<String> getSubordinateHosts() {
+    return nullToEmptyImmutableCopy(subordinateHosts);
+  }
+
+  public DateTime getRegistrationExpirationTime() {
+    return registrationExpirationTime;
+  }
+
+  public VKey<PollMessage.OneTime> getDeletePollMessage() {
+    return deletePollMessage;
+  }
+
+  public VKey<BillingEvent.Recurring> getAutorenewBillingEvent() {
+    return autorenewBillingEvent;
+  }
+
+  public VKey<PollMessage.Autorenew> getAutorenewPollMessage() {
+    return autorenewPollMessage;
+  }
+
+  public ImmutableSet<GracePeriod> getGracePeriods() {
+    return nullToEmptyImmutableCopy(gracePeriods);
+  }
+
+  public String getSmdId() {
+    return smdId;
+  }
+
+  @Override
+  public DomainTransferData getTransferData() {
+    return Optional.ofNullable(transferData).orElse(DomainTransferData.EMPTY);
+  }
+
+  @Override
+  public DateTime getLastTransferTime() {
+    return lastTransferTime;
+  }
+
+  @Override
+  public String getForeignKey() {
+    return fullyQualifiedDomainName;
+  }
+
+  public String getDomainName() {
+    return fullyQualifiedDomainName;
+  }
+
+  public ImmutableSet<DelegationSignerData> getDsData() {
+    return nullToEmptyImmutableCopy(dsData);
+  }
+
+  public LaunchNotice getLaunchNotice() {
+    return launchNotice;
+  }
+
+  public String getIdnTableName() {
+    return idnTableName;
+  }
+
+  public ImmutableSet<VKey<HostResource>> getNameservers() {
+    return nullToEmptyImmutableCopy(nsHosts);
+  }
+
+  // Hibernate needs this in order to populate nsHosts but no one else should ever use it
+  @SuppressWarnings("UnusedMethod")
+  private void setNsHosts(Set<VKey<HostResource>> nsHosts) {
+    this.nsHosts = nsHosts;
+  }
+
+  // Hibernate needs this in order to populate gracePeriods but no one else should ever use it
+  @SuppressWarnings("UnusedMethod")
+  private void setInternalGracePeriods(Set<GracePeriod> gracePeriods) {
+    this.gracePeriods = gracePeriods;
+  }
+
+  public final String getCurrentSponsorClientId() {
+    return getPersistedCurrentSponsorClientId();
+  }
+
+  /** Returns true if DNS information should be published for the given domain. */
+  public boolean shouldPublishToDns() {
+    return intersection(getStatusValues(), DNS_PUBLISHING_PROHIBITED_STATUSES).isEmpty();
+  }
+
+  /**
+   * Returns the Registry Grace Period Statuses for this domain.
+   *
+   * <p>This collects all statuses from the domain's {@link GracePeriod} entries and also adds the
+   * PENDING_DELETE status if needed.
+   */
+  public ImmutableSet<GracePeriodStatus> getGracePeriodStatuses() {
+    Set<GracePeriodStatus> gracePeriodStatuses = new HashSet<>();
+    for (GracePeriod gracePeriod : getGracePeriods()) {
+      gracePeriodStatuses.add(gracePeriod.getType());
+    }
+    if (getStatusValues().contains(StatusValue.PENDING_DELETE)
+        && !gracePeriodStatuses.contains(GracePeriodStatus.REDEMPTION)) {
+      gracePeriodStatuses.add(GracePeriodStatus.PENDING_DELETE);
+    }
+    return ImmutableSet.copyOf(gracePeriodStatuses);
+  }
+
+  /** Returns the subset of grace periods having the specified type. */
+  public ImmutableSet<GracePeriod> getGracePeriodsOfType(GracePeriodStatus gracePeriodType) {
+    ImmutableSet.Builder<GracePeriod> builder = new ImmutableSet.Builder<>();
+    for (GracePeriod gracePeriod : getGracePeriods()) {
+      if (gracePeriod.getType() == gracePeriodType) {
+        builder.add(gracePeriod);
+      }
+    }
+    return builder.build();
+  }
+
+  @Override
+  public DomainContent cloneProjectedAtTime(final DateTime now) {
+    return cloneDomainProjectedAtTime(this, now);
+  }
+
+  /**
+   * The logic in this method, which handles implicit server approval of transfers, very closely
+   * parallels the logic in {@code DomainTransferApproveFlow} which handles explicit client
+   * approvals.
+   */
+  protected static <T extends DomainContent> T cloneDomainProjectedAtTime(T domain, DateTime now) {
+    DomainTransferData transferData = domain.getTransferData();
+    DateTime transferExpirationTime = transferData.getPendingTransferExpirationTime();
+
+    // If there's a pending transfer that has expired, handle it.
+    if (TransferStatus.PENDING.equals(transferData.getTransferStatus())
+        && isBeforeOrAt(transferExpirationTime, now)) {
+      // Project until just before the transfer time. This will handle the case of an autorenew
+      // before the transfer was even requested or during the request period.
+      // If the transfer time is precisely the moment that the domain expires, there will not be an
+      // autorenew billing event (since we end the recurrence at transfer time and recurrences are
+      // exclusive of their ending), and we can just proceed with the transfer.
+      T domainAtTransferTime =
+          cloneDomainProjectedAtTime(domain, transferExpirationTime.minusMillis(1));
+
+      DateTime expirationDate = transferData.getTransferredRegistrationExpirationTime();
+      if (expirationDate == null) {
+        // Extend the registration by the correct number of years from the expiration time
+        // that was current on the domain right before the transfer, capped at 10 years from
+        // the moment of the transfer.
+        expirationDate =
+            ResourceFlowUtils.computeExDateForApprovalTime(
+                domainAtTransferTime, transferExpirationTime, transferData.getTransferPeriod());
+      }
+      // If we are within an autorenew grace period, the transfer will subsume the autorenew. There
+      // will already be a cancellation written in advance by the transfer request flow, so we don't
+      // need to worry about billing, but we do need to cancel out the expiration time increase.
+      // The transfer period saved in the transfer data will be one year, unless the superuser
+      // extension set the transfer period to zero.
+      // Set the expiration, autorenew events, and grace period for the transfer. (Transfer ends
+      // all other graces).
+      Builder builder =
+          domainAtTransferTime
+              .asBuilder()
+              .setRegistrationExpirationTime(expirationDate)
+              // Set the speculatively-written new autorenew events as the domain's autorenew
+              // events.
+              .setAutorenewBillingEvent(transferData.getServerApproveAutorenewEvent())
+              .setAutorenewPollMessage(transferData.getServerApproveAutorenewPollMessage());
+      if (transferData.getTransferPeriod().getValue() == 1) {
+        // Set the grace period using a key to the prescheduled transfer billing event.  Not using
+        // GracePeriod.forBillingEvent() here in order to avoid the actual Datastore fetch.
+        builder.setGracePeriods(
+            ImmutableSet.of(
+                GracePeriod.create(
+                    GracePeriodStatus.TRANSFER,
+                    domain.getRepoId(),
+                    transferExpirationTime.plus(
+                        Registry.get(domain.getTld()).getTransferGracePeriodLength()),
+                    transferData.getGainingClientId(),
+                    transferData.getServerApproveBillingEvent())));
+      } else {
+        // There won't be a billing event, so we don't need a grace period
+        builder.setGracePeriods(ImmutableSet.of());
+      }
+      // Set all remaining transfer properties.
+      setAutomaticTransferSuccessProperties(builder, transferData);
+      builder
+          .setLastEppUpdateTime(transferExpirationTime)
+          .setLastEppUpdateClientId(transferData.getGainingClientId());
+      // Finish projecting to now.
+      return (T) builder.build().cloneProjectedAtTime(now);
+    }
+
+    Optional<DateTime> newLastEppUpdateTime = Optional.empty();
+
+    // There is no transfer. Do any necessary autorenews for active domains.
+
+    Builder builder = domain.asBuilder();
+    if (isBeforeOrAt(domain.getRegistrationExpirationTime(), now)
+        && END_OF_TIME.equals(domain.getDeletionTime())) {
+      // Autorenew by the number of years between the old expiration time and now.
+      DateTime lastAutorenewTime =
+          leapSafeAddYears(
+              domain.getRegistrationExpirationTime(),
+              new Interval(domain.getRegistrationExpirationTime(), now).toPeriod().getYears());
+      DateTime newExpirationTime = lastAutorenewTime.plusYears(1);
+      builder
+          .setRegistrationExpirationTime(newExpirationTime)
+          .addGracePeriod(
+              GracePeriod.createForRecurring(
+                  GracePeriodStatus.AUTO_RENEW,
+                  domain.getRepoId(),
+                  lastAutorenewTime.plus(
+                      Registry.get(domain.getTld()).getAutoRenewGracePeriodLength()),
+                  domain.getCurrentSponsorClientId(),
+                  domain.getAutorenewBillingEvent()));
+      newLastEppUpdateTime = Optional.of(lastAutorenewTime);
+    }
+
+    // Remove any grace periods that have expired.
+    T almostBuilt = (T) builder.build();
+    builder = almostBuilt.asBuilder();
+    for (GracePeriod gracePeriod : almostBuilt.getGracePeriods()) {
+      if (isBeforeOrAt(gracePeriod.getExpirationTime(), now)) {
+        builder.removeGracePeriod(gracePeriod);
+        if (!newLastEppUpdateTime.isPresent()
+            || isBeforeOrAt(newLastEppUpdateTime.get(), gracePeriod.getExpirationTime())) {
+          newLastEppUpdateTime = Optional.of(gracePeriod.getExpirationTime());
+        }
+      }
+    }
+
+    // It is possible that the lastEppUpdateClientId is different from current sponsor client
+    // id, so we have to do the comparison instead of having one variable just storing the most
+    // recent time.
+    if (newLastEppUpdateTime.isPresent()) {
+      if (domain.getLastEppUpdateTime() == null
+          || newLastEppUpdateTime.get().isAfter(domain.getLastEppUpdateTime())) {
+        builder
+            .setLastEppUpdateTime(newLastEppUpdateTime.get())
+            .setLastEppUpdateClientId(domain.getCurrentSponsorClientId());
+      }
+    }
+
+    // Handle common properties like setting or unsetting linked status. This also handles the
+    // general case of pending transfers for other resource types, but since we've always handled
+    // a pending transfer by this point that's a no-op for domains.
+    projectResourceOntoBuilderAtTime(almostBuilt, builder, now);
+    return (T) builder.build();
+  }
+
+  /** Return what the expiration time would be if the given number of years were added to it. */
+  public static DateTime extendRegistrationWithCap(
+      DateTime now, DateTime currentExpirationTime, @Nullable Integer extendedRegistrationYears) {
+    // We must cap registration at the max years (aka 10), even if that truncates the last year.
+    return earliestOf(
+        leapSafeAddYears(
+            currentExpirationTime, Optional.ofNullable(extendedRegistrationYears).orElse(0)),
+        leapSafeAddYears(now, MAX_REGISTRATION_YEARS));
+  }
+
+  /** Loads and returns the fully qualified host names of all linked nameservers. */
+  public ImmutableSortedSet<String> loadNameserverHostNames() {
+    return ofy()
+        .load()
+        .keys(getNameservers().stream().map(VKey::getOfyKey).collect(toImmutableSet()))
+        .values()
+        .stream()
+        .map(HostResource::getHostName)
+        .collect(toImmutableSortedSet(Ordering.natural()));
+  }
+
+  /** A key to the registrant who registered this domain. */
+  public VKey<ContactResource> getRegistrant() {
+    return registrantContact;
+  }
+
+  public VKey<ContactResource> getAdminContact() {
+    return adminContact;
+  }
+
+  public VKey<ContactResource> getBillingContact() {
+    return billingContact;
+  }
+
+  public VKey<ContactResource> getTechContact() {
+    return techContact;
+  }
+
+  /** Associated contacts for the domain (other than registrant). */
+  public ImmutableSet<DesignatedContact> getContacts() {
+    return nullToEmpty(allContacts).stream()
+        .filter(IS_REGISTRANT.negate())
+        .collect(toImmutableSet());
+  }
+
+  public DomainAuthInfo getAuthInfo() {
+    return authInfo;
+  }
+
+  /** Returns all referenced contacts from this domain or application. */
+  public ImmutableSet<VKey<ContactResource>> getReferencedContacts() {
+    return nullToEmptyImmutableCopy(allContacts).stream()
+        .map(DesignatedContact::getContactKey)
+        .filter(Objects::nonNull)
+        .collect(toImmutableSet());
+  }
+
+  public String getTld() {
+    return tld;
+  }
+
+  /**
+   * Sets the individual contact fields from {@code contacts}.
+   *
+   * <p>The registrant field is only set if {@code includeRegistrant} is true, as this field needs
+   * to be set in some circumstances but not in others.
+   */
+  protected void setContactFields(Set<DesignatedContact> contacts, boolean includeRegistrant) {
+    // Set the individual contact fields.
+    for (DesignatedContact contact : contacts) {
+      switch (contact.getType()) {
+        case BILLING:
+          billingContact = contact.getContactKey();
+          break;
+        case TECH:
+          techContact = contact.getContactKey();
+          break;
+        case ADMIN:
+          adminContact = contact.getContactKey();
+          break;
+        case REGISTRANT:
+          if (includeRegistrant) {
+            registrantContact = contact.getContactKey();
+          }
+          break;
+        default:
+          throw new IllegalArgumentException("Unknown contact resource type: " + contact.getType());
+      }
+    }
+  }
+
+  @Override
+  public VKey<DomainBase> createVKey() {
+    return VKey.create(DomainBase.class, getRepoId(), Key.create(this));
+  }
+
+  public static VKey<DomainBase> createVKey(Key key) {
+    return VKey.create(DomainBase.class, key.getName(), key);
+  }
+
+  /** Predicate to determine if a given {@link DesignatedContact} is the registrant. */
+  protected static final Predicate<DesignatedContact> IS_REGISTRANT =
+      (DesignatedContact contact) -> DesignatedContact.Type.REGISTRANT.equals(contact.type);
+
+  /** An override of {@link EppResource#asBuilder} with tighter typing. */
+  @Override
+  public Builder asBuilder() {
+    return new Builder<>(clone(this));
+  }
+
+  /** A builder for constructing {@link DomainBase}, since it is immutable. */
+  public static class Builder<T extends DomainContent, B extends Builder<T, B>>
+      extends EppResource.Builder<T, B> implements BuilderWithTransferData<DomainTransferData, B> {
+
+    public Builder() {}
+
+    Builder(T instance) {
+      super(instance);
+    }
+
+    @Override
+    public T build() {
+      T instance = getInstance();
+      // If TransferData is totally empty, set it to null.
+      if (DomainTransferData.EMPTY.equals(getInstance().transferData)) {
+        setTransferData(null);
+      }
+      // A DomainBase has status INACTIVE if there are no nameservers.
+      if (getInstance().getNameservers().isEmpty()) {
+        addStatusValue(StatusValue.INACTIVE);
+      } else { // There are nameservers, so make sure INACTIVE isn't there.
+        removeStatusValue(StatusValue.INACTIVE);
+      }
+
+      checkArgumentNotNull(emptyToNull(instance.fullyQualifiedDomainName), "Missing domainName");
+      if (instance.getRegistrant() == null
+          && instance.allContacts.stream().anyMatch(IS_REGISTRANT)) {
+        throw new IllegalArgumentException("registrant is null but is in allContacts");
+      }
+      checkArgumentNotNull(instance.getRegistrant(), "Missing registrant");
+      instance.tld = getTldFromDomainName(instance.fullyQualifiedDomainName);
+      return super.build();
+    }
+
+    public B setDomainName(String domainName) {
+      checkArgument(
+          domainName.equals(canonicalizeDomainName(domainName)),
+          "Domain name must be in puny-coded, lower-case form");
+      getInstance().fullyQualifiedDomainName = domainName;
+      return thisCastToDerived();
+    }
+
+    public B setDsData(ImmutableSet<DelegationSignerData> dsData) {
+      getInstance().dsData = dsData;
+      return thisCastToDerived();
+    }
+
+    public B setRegistrant(VKey<ContactResource> registrant) {
+      // Replace the registrant contact inside allContacts.
+      getInstance().allContacts =
+          union(
+              getInstance().getContacts(),
+              DesignatedContact.create(
+                  DesignatedContact.Type.REGISTRANT, checkArgumentNotNull(registrant)));
+
+      // Set the registrant field specifically.
+      getInstance().registrantContact = registrant;
+      return thisCastToDerived();
+    }
+
+    public B setAuthInfo(DomainAuthInfo authInfo) {
+      getInstance().authInfo = authInfo;
+      return thisCastToDerived();
+    }
+
+    public B setNameservers(VKey<HostResource> nameserver) {
+      getInstance().nsHosts = ImmutableSet.of(nameserver);
+      return thisCastToDerived();
+    }
+
+    public B setNameservers(ImmutableSet<VKey<HostResource>> nameservers) {
+      getInstance().nsHosts = forceEmptyToNull(nameservers);
+      return thisCastToDerived();
+    }
+
+    public B addNameserver(VKey<HostResource> nameserver) {
+      return addNameservers(ImmutableSet.of(nameserver));
+    }
+
+    public B addNameservers(ImmutableSet<VKey<HostResource>> nameservers) {
+      return setNameservers(
+          ImmutableSet.copyOf(Sets.union(getInstance().getNameservers(), nameservers)));
+    }
+
+    public B removeNameserver(VKey<HostResource> nameserver) {
+      return removeNameservers(ImmutableSet.of(nameserver));
+    }
+
+    public B removeNameservers(ImmutableSet<VKey<HostResource>> nameservers) {
+      return setNameservers(
+          ImmutableSet.copyOf(difference(getInstance().getNameservers(), nameservers)));
+    }
+
+    public B setContacts(DesignatedContact contact) {
+      return setContacts(ImmutableSet.of(contact));
+    }
+
+    public B setContacts(ImmutableSet<DesignatedContact> contacts) {
+      checkArgument(contacts.stream().noneMatch(IS_REGISTRANT), "Registrant cannot be a contact");
+
+      // Replace the non-registrant contacts inside allContacts.
+      getInstance().allContacts =
+          Streams.concat(
+                  nullToEmpty(getInstance().allContacts).stream().filter(IS_REGISTRANT),
+                  contacts.stream())
+              .collect(toImmutableSet());
+
+      // Set the individual fields.
+      getInstance().setContactFields(contacts, false);
+      return thisCastToDerived();
+    }
+
+    public B addContacts(ImmutableSet<DesignatedContact> contacts) {
+      return setContacts(ImmutableSet.copyOf(Sets.union(getInstance().getContacts(), contacts)));
+    }
+
+    public B removeContacts(ImmutableSet<DesignatedContact> contacts) {
+      return setContacts(ImmutableSet.copyOf(difference(getInstance().getContacts(), contacts)));
+    }
+
+    public B setLaunchNotice(LaunchNotice launchNotice) {
+      getInstance().launchNotice = launchNotice;
+      return thisCastToDerived();
+    }
+
+    public B setIdnTableName(String idnTableName) {
+      getInstance().idnTableName = idnTableName;
+      return thisCastToDerived();
+    }
+
+    public B setSubordinateHosts(ImmutableSet<String> subordinateHosts) {
+      getInstance().subordinateHosts = subordinateHosts;
+      return thisCastToDerived();
+    }
+
+    public B addSubordinateHost(String hostToAdd) {
+      return setSubordinateHosts(
+          ImmutableSet.copyOf(union(getInstance().getSubordinateHosts(), hostToAdd)));
+    }
+
+    public B removeSubordinateHost(String hostToRemove) {
+      return setSubordinateHosts(
+          ImmutableSet.copyOf(
+              CollectionUtils.difference(getInstance().getSubordinateHosts(), hostToRemove)));
+    }
+
+    public B setRegistrationExpirationTime(DateTime registrationExpirationTime) {
+      getInstance().registrationExpirationTime = registrationExpirationTime;
+      return thisCastToDerived();
+    }
+
+    public B setDeletePollMessage(VKey<PollMessage.OneTime> deletePollMessage) {
+      getInstance().deletePollMessage = deletePollMessage;
+      return thisCastToDerived();
+    }
+
+    public B setAutorenewBillingEvent(VKey<BillingEvent.Recurring> autorenewBillingEvent) {
+      getInstance().autorenewBillingEvent = autorenewBillingEvent;
+      return thisCastToDerived();
+    }
+
+    public B setAutorenewPollMessage(VKey<PollMessage.Autorenew> autorenewPollMessage) {
+      getInstance().autorenewPollMessage = autorenewPollMessage;
+      return thisCastToDerived();
+    }
+
+    public B setSmdId(String smdId) {
+      getInstance().smdId = smdId;
+      return thisCastToDerived();
+    }
+
+    public B setGracePeriods(ImmutableSet<GracePeriod> gracePeriods) {
+      getInstance().gracePeriods = gracePeriods;
+      return thisCastToDerived();
+    }
+
+    public B addGracePeriod(GracePeriod gracePeriod) {
+      getInstance().gracePeriods = union(getInstance().getGracePeriods(), gracePeriod);
+      return thisCastToDerived();
+    }
+
+    public B removeGracePeriod(GracePeriod gracePeriod) {
+      getInstance().gracePeriods =
+          CollectionUtils.difference(getInstance().getGracePeriods(), gracePeriod);
+      return thisCastToDerived();
+    }
+
+    @Override
+    public B setTransferData(DomainTransferData transferData) {
+      getInstance().transferData = transferData;
+      return thisCastToDerived();
+    }
+
+    @Override
+    public B setLastTransferTime(DateTime lastTransferTime) {
+      getInstance().lastTransferTime = lastTransferTime;
+      return thisCastToDerived();
+    }
+  }
+}
@@ -0,0 +1,111 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.domain;
+
+import com.googlecode.objectify.Key;
+import google.registry.model.EppResource;
+import google.registry.model.contact.ContactResource;
+import google.registry.model.host.HostResource;
+import google.registry.model.reporting.HistoryEntry;
+import google.registry.persistence.VKey;
+import java.util.Set;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.Column;
+import javax.persistence.ElementCollection;
+import javax.persistence.Entity;
+import javax.persistence.JoinTable;
+
+/**
+ * A persisted history entry representing an EPP modification to a domain.
+ *
+ * <p>In addition to the general history fields (e.g. action time, registrar ID) we also persist a
+ * copy of the domain entity at this point in time. We persist a raw {@link DomainContent} so that
+ * the foreign-keyed fields in that class can refer to this object.
+ */
+@Entity
+@javax.persistence.Table(
+    indexes = {
+      @javax.persistence.Index(columnList = "creationTime"),
+      @javax.persistence.Index(columnList = "historyRegistrarId"),
+      @javax.persistence.Index(columnList = "historyType"),
+      @javax.persistence.Index(columnList = "historyModificationTime")
+    })
+public class DomainHistory extends HistoryEntry {
+  // Store DomainContent instead of DomainBase so we don't pick up its @Id
+  DomainContent domainContent;
+
+  @Column(nullable = false)
+  VKey<DomainBase> domainRepoId;
+
+  @ElementCollection
+  @JoinTable(name = "DomainHistoryHost")
+  @Access(AccessType.PROPERTY)
+  @Column(name = "host_repo_id")
+  public Set<VKey<HostResource>> getNsHosts() {
+    return domainContent.nsHosts;
+  }
+
+  /** The state of the {@link DomainContent} object at this point in time. */
+  public DomainContent getDomainContent() {
+    return domainContent;
+  }
+
+  /** The key to the {@link ContactResource} this is based off of. */
+  public VKey<DomainBase> getDomainRepoId() {
+    return domainRepoId;
+  }
+
+  // Hibernate needs this in order to populate nsHosts but no one else should ever use it
+  @SuppressWarnings("UnusedMethod")
+  private void setNsHosts(Set<VKey<HostResource>> nsHosts) {
+    if (domainContent != null) {
+      domainContent.nsHosts = nsHosts;
+    }
+  }
+
+  @Override
+  public Builder asBuilder() {
+    return new Builder(clone(this));
+  }
+
+  public static class Builder extends HistoryEntry.Builder<DomainHistory, DomainHistory.Builder> {
+
+    public Builder() {}
+
+    public Builder(DomainHistory instance) {
+      super(instance);
+    }
+
+    public Builder setDomainContent(DomainContent domainContent) {
+      getInstance().domainContent = domainContent;
+      return this;
+    }
+
+    public Builder setDomainRepoId(VKey<DomainBase> domainRepoId) {
+      getInstance().domainRepoId = domainRepoId;
+      domainRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
+      return this;
+    }
+
+    // We can remove this once all HistoryEntries are converted to History objects
+    @Override
+    public Builder setParent(Key<? extends EppResource> parent) {
+      super.setParent(parent);
+      getInstance().domainRepoId = VKey.create(DomainBase.class, parent.getName(), parent);
+      return this;
+    }
+  }
+}
@@ -18,16 +18,14 @@ import static com.google.common.base.Preconditions.checkArgument;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

 import com.googlecode.objectify.annotation.Embed;
-import com.googlecode.objectify.annotation.Ignore;
-import google.registry.model.ImmutableObject;
 import google.registry.model.billing.BillingEvent;
 import google.registry.model.billing.BillingEvent.Recurring;
 import google.registry.model.domain.rgp.GracePeriodStatus;
 import google.registry.persistence.VKey;
 import javax.annotation.Nullable;
-import javax.persistence.Column;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
+import javax.persistence.Entity;
+import javax.persistence.Index;
+import javax.persistence.Table;
 import org.joda.time.DateTime;

 /**
@@ -37,75 +35,13 @@ import org.joda.time.DateTime;
 * the resource is loaded from Datastore.
 */
@Embed
-@javax.persistence.Entity
-public class GracePeriod extends ImmutableObject {
-
-  @javax.persistence.Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
-  @Ignore
-  /** Unique id required for hibernate representation. */
-  long id;
-
-  /** The type of grace period. */
-  GracePeriodStatus type;
-
-  /** When the grace period ends. */
-  DateTime expirationTime;
-
-  /** The registrar to bill. */
-  @Column(name = "registrarId")
-  String clientId;
-
-  /**
-   * The one-time billing event corresponding to the action that triggered this grace period, or
-   * null if not applicable. Not set for autorenew grace periods (which instead use the field {@code
-   * billingEventRecurring}) or for redemption grace periods (since deletes have no cost).
-   */
-  // NB: Would @IgnoreSave(IfNull.class), but not allowed for @Embed collections.
-  VKey<BillingEvent.OneTime> billingEventOneTime = null;
-
-  /**
-   * The recurring billing event corresponding to the action that triggered this grace period, if
-   * applicable - i.e. if the action was an autorenew - or null in all other cases.
-   */
-  // NB: Would @IgnoreSave(IfNull.class), but not allowed for @Embed collections.
-  VKey<BillingEvent.Recurring> billingEventRecurring = null;
-
-  public GracePeriodStatus getType() {
-    return type;
-  }
-
-  public DateTime getExpirationTime() {
-    return expirationTime;
-  }
-
-  public String getClientId() {
-    return clientId;
-  }
-
-  /** Returns true if this GracePeriod has an associated BillingEvent; i.e. if it's refundable. */
-  public boolean hasBillingEvent() {
-    return billingEventOneTime != null || billingEventRecurring != null;
-  }
-
-  /**
-   * Returns the one time billing event. The value will only be non-null if the type of this grace
-   * period is not AUTO_RENEW.
-   */
-  public VKey<BillingEvent.OneTime> getOneTimeBillingEvent() {
-    return billingEventOneTime;
-  }
-
-  /**
-   * Returns the recurring billing event. The value will only be non-null if the type of this grace
-   * period is AUTO_RENEW.
-   */
-  public VKey<BillingEvent.Recurring> getRecurringBillingEvent() {
-    return billingEventRecurring;
-  }
+@Entity
+@Table(indexes = @Index(columnList = "domainRepoId"))
+public class GracePeriod extends GracePeriodBase {

  private static GracePeriod createInternal(
      GracePeriodStatus type,
+      String domainRepoId,
      DateTime expirationTime,
      String clientId,
      @Nullable VKey<BillingEvent.OneTime> billingEventOneTime,
@@ -117,6 +53,7 @@ public class GracePeriod extends ImmutableObject {
        "Recurring billing events must be present on (and only on) autorenew grace periods");
    GracePeriod instance = new GracePeriod();
    instance.type = checkArgumentNotNull(type);
+    instance.domainRepoId = checkArgumentNotNull(domainRepoId);
    instance.expirationTime = checkArgumentNotNull(expirationTime);
    instance.clientId = checkArgumentNotNull(clientId);
    instance.billingEventOneTime = billingEventOneTime;
@@ -133,32 +70,51 @@ public class GracePeriod extends ImmutableObject {
   */
  public static GracePeriod create(
      GracePeriodStatus type,
+      String domainRepoId,
      DateTime expirationTime,
      String clientId,
      @Nullable VKey<BillingEvent.OneTime> billingEventOneTime) {
-    return createInternal(type, expirationTime, clientId, billingEventOneTime, null);
+    return createInternal(type, domainRepoId, expirationTime, clientId, billingEventOneTime, null);
  }

  /** Creates a GracePeriod for a Recurring billing event. */
  public static GracePeriod createForRecurring(
      GracePeriodStatus type,
+      String domainRepoId,
      DateTime expirationTime,
      String clientId,
      VKey<Recurring> billingEventRecurring) {
    checkArgumentNotNull(billingEventRecurring, "billingEventRecurring cannot be null");
-    return createInternal(type, expirationTime, clientId, null, billingEventRecurring);
+    return createInternal(
+        type, domainRepoId, expirationTime, clientId, null, billingEventRecurring);
  }

  /** Creates a GracePeriod with no billing event. */
  public static GracePeriod createWithoutBillingEvent(
-      GracePeriodStatus type, DateTime expirationTime, String clientId) {
-    return createInternal(type, expirationTime, clientId, null, null);
+      GracePeriodStatus type, String domainRepoId, DateTime expirationTime, String clientId) {
+    return createInternal(type, domainRepoId, expirationTime, clientId, null, null);
  }

  /** Constructs a GracePeriod of the given type from the provided one-time BillingEvent. */
  public static GracePeriod forBillingEvent(
-      GracePeriodStatus type, BillingEvent.OneTime billingEvent) {
+      GracePeriodStatus type, String domainRepoId, BillingEvent.OneTime billingEvent) {
    return create(
-        type, billingEvent.getBillingTime(), billingEvent.getClientId(), billingEvent.createVKey());
+        type,
+        domainRepoId,
+        billingEvent.getBillingTime(),
+        billingEvent.getClientId(),
+        billingEvent.createVKey());
  }
+
+  /**
+   * Returns a clone of this {@link GracePeriod} with {@link #domainRepoId} set to the given value.
+   *
+   * <p>TODO(b/162739503): Remove this function after fully migrating to Cloud SQL.
+   */
+  public GracePeriod cloneWithDomainRepoId(String domainRepoId) {
+    GracePeriod clone = clone(this);
+    clone.domainRepoId = checkArgumentNotNull(domainRepoId);
+    return clone;
+  }
+
 }
@@ -0,0 +1,114 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.domain;
+
+import com.googlecode.objectify.annotation.Embed;
+import com.googlecode.objectify.annotation.Ignore;
+import google.registry.model.ImmutableObject;
+import google.registry.model.billing.BillingEvent;
+import google.registry.model.billing.BillingEvent.OneTime;
+import google.registry.model.domain.rgp.GracePeriodStatus;
+import google.registry.persistence.VKey;
+import javax.persistence.Column;
+import javax.persistence.EnumType;
+import javax.persistence.Enumerated;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.MappedSuperclass;
+import org.joda.time.DateTime;
+
+/** Base class containing common fields and methods for {@link GracePeriod}. */
+@Embed
+@MappedSuperclass
+public class GracePeriodBase extends ImmutableObject {
+
+  /** Unique id required for hibernate representation. */
+  @javax.persistence.Id
+  @GeneratedValue(strategy = GenerationType.IDENTITY)
+  @Ignore
+  Long id;
+
+  /** Repository id for the domain which this grace period belongs to. */
+  @Ignore
+  @Column(nullable = false)
+  String domainRepoId;
+
+  /** The type of grace period. */
+  @Column(nullable = false)
+  @Enumerated(EnumType.STRING)
+  GracePeriodStatus type;
+
+  /** When the grace period ends. */
+  @Column(nullable = false)
+  DateTime expirationTime;
+
+  /** The registrar to bill. */
+  @Column(name = "registrarId", nullable = false)
+  String clientId;
+
+  /**
+   * The one-time billing event corresponding to the action that triggered this grace period, or
+   * null if not applicable. Not set for autorenew grace periods (which instead use the field {@code
+   * billingEventRecurring}) or for redemption grace periods (since deletes have no cost).
+   */
+  // NB: Would @IgnoreSave(IfNull.class), but not allowed for @Embed collections.
+  @Column(name = "billing_event_id")
+  VKey<OneTime> billingEventOneTime = null;
+
+  /**
+   * The recurring billing event corresponding to the action that triggered this grace period, if
+   * applicable - i.e. if the action was an autorenew - or null in all other cases.
+   */
+  // NB: Would @IgnoreSave(IfNull.class), but not allowed for @Embed collections.
+  @Column(name = "billing_recurrence_id")
+  VKey<BillingEvent.Recurring> billingEventRecurring = null;
+
+  public GracePeriodStatus getType() {
+    return type;
+  }
+
+  public String getDomainRepoId() {
+    return domainRepoId;
+  }
+
+  public DateTime getExpirationTime() {
+    return expirationTime;
+  }
+
+  public String getClientId() {
+    return clientId;
+  }
+
+  /** Returns true if this GracePeriod has an associated BillingEvent; i.e. if it's refundable. */
+  public boolean hasBillingEvent() {
+    return billingEventOneTime != null || billingEventRecurring != null;
+  }
+
+  /**
+   * Returns the one time billing event. The value will only be non-null if the type of this grace
+   * period is not AUTO_RENEW.
+   */
+  public VKey<BillingEvent.OneTime> getOneTimeBillingEvent() {
+    return billingEventOneTime;
+  }
+
+  /**
+   * Returns the recurring billing event. The value will only be non-null if the type of this grace
+   * period is AUTO_RENEW.
+   */
+  public VKey<BillingEvent.Recurring> getRecurringBillingEvent() {
+    return billingEventRecurring;
+  }
+}
@@ -29,12 +29,14 @@ import com.google.common.base.Strings;
 import com.google.common.collect.ImmutableMultimap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedMap;
+import com.google.common.collect.Range;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Embed;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Index;
 import com.googlecode.objectify.annotation.Mapify;
+import com.googlecode.objectify.annotation.OnLoad;
 import google.registry.flows.EppException;
 import google.registry.flows.domain.DomainFlowUtils;
 import google.registry.model.BackupGroupRoot;
@@ -108,9 +110,22 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
   */
  double discountFraction;

+  /** Whether the discount fraction (if any) also applies to premium names. Defaults to false. */
+  boolean discountPremiums;
+
+  /** Up to how many years of initial creation receive the discount (if any). Defaults to 1. */
+  int discountYears = 1;
+
  /** The type of the token, either single-use or unlimited-use. */
-  // TODO(b/130301183): this should not be nullable, we can remove this once we're sure it isn't
-  @Nullable TokenType tokenType;
+  TokenType tokenType;
+
+  // TODO: Remove onLoad once all allocation tokens are migrated to have a discountYears of 1.
+  @OnLoad
+  void onLoad() {
+    if (discountYears == 0) {
+      discountYears = 1;
+    }
+  }

  /**
   * Promotional token validity periods.
@@ -146,8 +161,8 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
    return token;
  }

-  public Key<HistoryEntry> getRedemptionHistoryEntry() {
-    return redemptionHistoryEntry;
+  public Optional<Key<HistoryEntry>> getRedemptionHistoryEntry() {
+    return Optional.ofNullable(redemptionHistoryEntry);
  }

  public boolean isRedeemed() {
@@ -174,6 +189,16 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
    return discountFraction;
  }

+  public boolean shouldDiscountPremiums() {
+    return discountPremiums;
+  }
+
+  public int getDiscountYears() {
+    // Allocation tokens created prior to the addition of the discountYears field will have a value
+    // of 0 for it, but it should be the default value of 1 to retain the previous behavior.
+    return Math.max(1, discountYears);
+  }
+
  public TokenType getTokenType() {
    return tokenType;
  }
@@ -193,6 +218,7 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {

  /** A builder for constructing {@link AllocationToken} objects, since they are immutable. */
  public static class Builder extends Buildable.Builder<AllocationToken> {
+
    public Builder() {}

    private Builder(AllocationToken instance) {
@@ -210,6 +236,12 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
          getInstance().redemptionHistoryEntry == null
              || TokenType.SINGLE_USE.equals(getInstance().tokenType),
          "Redemption history entry can only be specified for SINGLE_USE tokens");
+      checkArgument(
+          getInstance().discountFraction > 0 || !getInstance().discountPremiums,
+          "Discount premiums can only be specified along with a discount fraction");
+      checkArgument(
+          getInstance().discountFraction > 0 || getInstance().discountYears == 1,
+          "Discount years can only be specified along with a discount fraction");
      if (getInstance().domainName != null) {
        try {
          DomainFlowUtils.validateDomainName(getInstance().domainName);
@@ -258,10 +290,26 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
    }

    public Builder setDiscountFraction(double discountFraction) {
+      checkArgument(
+          Range.closed(0.0d, 1.0d).contains(discountFraction),
+          "Discount fraction must be between 0 and 1 inclusive");
      getInstance().discountFraction = discountFraction;
      return this;
    }

+    public Builder setDiscountPremiums(boolean discountPremiums) {
+      getInstance().discountPremiums = discountPremiums;
+      return this;
+    }
+
+    public Builder setDiscountYears(int discountYears) {
+      checkArgument(
+          Range.closed(1, 10).contains(discountYears),
+          "Discount years must be between 1 and 10 inclusive");
+      getInstance().discountYears = discountYears;
+      return this;
+    }
+
    public Builder setTokenType(TokenType tokenType) {
      checkState(getInstance().tokenType == null, "Token type can only be set once");
      getInstance().tokenType = tokenType;
@@ -155,15 +155,7 @@ public abstract class PollMessage extends ImmutableObject
  public abstract VKey<? extends PollMessage> createVKey();

  public static VKey<PollMessage> createVKey(Key<PollMessage> key) {
-    // TODO(b/159207551): As it stands, the SQL key generated here doesn't mesh with the primary key
-    // type for the table, which is a single long integer.  Also note that the class id is not
-    // correct here and as such the resulting key will not be loadable from SQL.
-    if (key == null) {
-      return null;
-    }
-    String path =
-        key.getParent().getParent().getName() + "/" + key.getParent().getId() + key.getId();
-    return VKey.create(PollMessage.class, path, key);
+    return VKey.create(PollMessage.class, key.getId(), key);
  }

  /** Override Buildable.asBuilder() to give this method stronger typing. */
@@ -140,6 +140,16 @@ public class Spec11ThreatMatch extends ImmutableObject implements Buildable, Sql
      return super.build();
    }

+    /**
+     * Manually set the ID for testing or other special circumstances.
+     *
+     * <p>In general the ID is generated by SQL and there should be no need to set it manually.
+     */
+    public Builder setId(Long id) {
+      getInstance().id = id;
+      return this;
+    }
+
    public Builder setDomainName(String domainName) {
      getInstance().domainName = domainName;
      getInstance().tld = DomainNameUtils.getTldFromDomainName(domainName);
@@ -19,6 +19,7 @@ import com.google.common.collect.ImmutableSet;
 import java.sql.SQLException;
 import java.util.function.Predicate;
 import javax.persistence.OptimisticLockException;
+import org.hibernate.exception.JDBCConnectionException;

 /** Helpers for identifying retriable database operations. */
 public final class JpaRetries {
@@ -40,6 +41,13 @@ public final class JpaRetries {
              e instanceof SQLException
                  && RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));

+  private static final Predicate<Throwable> RETRIABLE_QUERY_PREDICATE =
+      Predicates.or(
+          JDBCConnectionException.class::isInstance,
+          e ->
+              e instanceof SQLException
+                  && RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
+
  public static boolean isFailedTxnRetriable(Throwable throwable) {
    Throwable t = throwable;
    while (t != null) {
@@ -53,6 +61,13 @@ public final class JpaRetries {

  public static boolean isFailedQueryRetriable(Throwable throwable) {
    // TODO(weiminyu): check for more error codes.
-    return isFailedTxnRetriable(throwable);
+    Throwable t = throwable;
+    while (t != null) {
+      if (RETRIABLE_QUERY_PREDICATE.test(t)) {
+        return true;
+      }
+      t = t.getCause();
+    }
+    return false;
  }
 }
@@ -13,6 +13,7 @@
 // limitations under the License.
 package google.registry.persistence;

+import google.registry.persistence.converter.IntervalDescriptor;
 import google.registry.persistence.converter.StringCollectionDescriptor;
 import google.registry.persistence.converter.StringMapDescriptor;
 import java.sql.Types;
@@ -30,6 +31,7 @@ public class NomulusPostgreSQLDialect extends PostgreSQL95Dialect {
    registerColumnType(StringMapDescriptor.COLUMN_TYPE, StringMapDescriptor.COLUMN_NAME);
    registerColumnType(
        StringCollectionDescriptor.COLUMN_TYPE, StringCollectionDescriptor.COLUMN_DDL_NAME);
+    registerColumnType(IntervalDescriptor.COLUMN_TYPE, IntervalDescriptor.COLUMN_NAME);
  }

  @Override
@@ -40,5 +42,7 @@ public class NomulusPostgreSQLDialect extends PostgreSQL95Dialect {
    typeContributions.contributeSqlTypeDescriptor(StringCollectionDescriptor.getInstance());
    typeContributions.contributeJavaTypeDescriptor(StringMapDescriptor.getInstance());
    typeContributions.contributeSqlTypeDescriptor(StringMapDescriptor.getInstance());
+    typeContributions.contributeJavaTypeDescriptor(IntervalDescriptor.getInstance());
+    typeContributions.contributeSqlTypeDescriptor(IntervalDescriptor.getInstance());
  }
 }
@@ -14,24 +14,67 @@

 package google.registry.persistence.converter;

+import java.sql.SQLException;
 import javax.annotation.Nullable;
 import javax.persistence.AttributeConverter;
 import javax.persistence.Converter;
 import org.joda.time.Duration;
+import org.joda.time.Period;
+import org.postgresql.util.PGInterval;

-/** JPA converter to for storing/retrieving {@link org.joda.time.DateTime} objects. */
+/**
+ * JPA converter to for storing/retrieving {@link org.joda.time.Duration} objects.
+ *
+ * <p>The Joda Time Duration is simply a number of milliseconds representing a length of time. This
+ * can be converted into a PGInterval, but only for the fields that have a standard number of
+ * milliseconds. Therefore, there is no way to populate the months or years field of a PGInterval
+ * and be confident that it is representing the exact number of milliseconds it was intended to
+ * represent.
+ */
@Converter(autoApply = true)
-public class DurationConverter implements AttributeConverter<Duration, Long> {
+public class DurationConverter implements AttributeConverter<Duration, PGInterval> {

  @Override
  @Nullable
-  public Long convertToDatabaseColumn(@Nullable Duration duration) {
-    return duration == null ? null : duration.getMillis();
+  public PGInterval convertToDatabaseColumn(@Nullable Duration duration) {
+    if (duration == null) {
+      return new PGInterval();
+    }
+    PGInterval interval = new PGInterval();
+    Period period = new Period(duration);
+    // For some reason when the period is created from the duration, it does not set days, but
+    // instead just a total number of hours. Years and months are not created because those can
+    // differ in length of milliseconds.
+    interval.setDays(period.getHours() / 24);
+    interval.setHours(period.getHours() % 24);
+    interval.setMinutes(period.getMinutes());
+    double millis = (double) period.getMillis() / 1000;
+    interval.setSeconds(period.getSeconds() + millis);
+    return interval;
  }

  @Override
  @Nullable
-  public Duration convertToEntityAttribute(@Nullable Long dbData) {
-    return dbData == null ? null : new Duration(dbData);
+  public Duration convertToEntityAttribute(@Nullable PGInterval dbData) {
+    if (dbData == null) {
+      return null;
+    }
+    PGInterval interval = null;
+    try {
+      interval = new PGInterval(dbData.toString());
+    } catch (SQLException e) {
+      throw new RuntimeException(e);
+    }
+
+    if (interval.equals(new PGInterval())) {
+      return null;
+    }
+
+    final int days = interval.getDays();
+    final int hours = interval.getHours();
+    final int mins = interval.getMinutes();
+    final int secs = (int) interval.getSeconds();
+    final int millis = interval.getMicroSeconds() / 1000;
+    return new Period(0, 0, 0, days, hours, mins, secs, millis).toStandardDuration();
  }
 }
@@ -0,0 +1,139 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.converter;
+
+import java.sql.CallableStatement;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Types;
+import org.hibernate.type.descriptor.ValueBinder;
+import org.hibernate.type.descriptor.ValueExtractor;
+import org.hibernate.type.descriptor.WrapperOptions;
+import org.hibernate.type.descriptor.java.AbstractTypeDescriptor;
+import org.hibernate.type.descriptor.java.JavaTypeDescriptor;
+import org.hibernate.type.descriptor.spi.JdbcRecommendedSqlTypeMappingContext;
+import org.hibernate.type.descriptor.sql.BasicBinder;
+import org.hibernate.type.descriptor.sql.BasicExtractor;
+import org.hibernate.type.descriptor.sql.SqlTypeDescriptor;
+import org.postgresql.util.PGInterval;
+
+/**
+ * The {@link JavaTypeDescriptor} and {@link SqlTypeDescriptor} for {@link PGInterval}.
+ *
+ * @see <a
+ *     href="https://docs.jboss.org/hibernate/orm/current/userguide/html_single/Hibernate_User_Guide.html#basic-jpa-convert">JPA
+ *     2.1 AttributeConverters</a>
+ */
+public class IntervalDescriptor extends AbstractTypeDescriptor<PGInterval>
+    implements SqlTypeDescriptor {
+  public static final int COLUMN_TYPE = Types.JAVA_OBJECT;
+  public static final String COLUMN_NAME = "interval";
+  private static final IntervalDescriptor INSTANCE = new IntervalDescriptor();
+
+  private IntervalDescriptor() {
+    super(PGInterval.class);
+  }
+
+  public static IntervalDescriptor getInstance() {
+    return INSTANCE;
+  }
+
+  @Override
+  public PGInterval fromString(String string) {
+    throw new UnsupportedOperationException(
+        "Constructing IntervalDescriptor from string is not allowed");
+  }
+
+  @Override
+  public <X> X unwrap(PGInterval value, Class<X> type, WrapperOptions options) {
+    if (value == null) {
+      return null;
+    }
+    if (PGInterval.class.isAssignableFrom(type)) {
+      return (X) value;
+    }
+    throw unknownUnwrap(type);
+  }
+
+  @Override
+  public <X> PGInterval wrap(X value, WrapperOptions options) {
+    if (value == null) {
+      return null;
+    }
+    if (value instanceof PGInterval) {
+      try {
+        return new PGInterval(value.toString());
+      } catch (SQLException e) {
+        throw new RuntimeException(e);
+      }
+    }
+    throw unknownWrap(value.getClass());
+  }
+
+  @Override
+  public int getSqlType() {
+    return COLUMN_TYPE;
+  }
+
+  @Override
+  public SqlTypeDescriptor getJdbcRecommendedSqlType(JdbcRecommendedSqlTypeMappingContext context) {
+    return this;
+  }
+
+  @Override
+  public boolean canBeRemapped() {
+    return false;
+  }
+
+  @Override
+  public <X> ValueBinder<X> getBinder(JavaTypeDescriptor<X> javaTypeDescriptor) {
+    return new BasicBinder<X>(javaTypeDescriptor, this) {
+      @Override
+      protected void doBind(PreparedStatement st, X value, int index, WrapperOptions options)
+          throws SQLException {
+        st.setObject(index, new PGInterval(value.toString()));
+      }
+
+      @Override
+      protected void doBind(CallableStatement st, X value, String name, WrapperOptions options)
+          throws SQLException {
+        st.setObject(name, new PGInterval(value.toString()));
+      }
+    };
+  }
+
+  @Override
+  public <X> ValueExtractor<X> getExtractor(JavaTypeDescriptor<X> javaTypeDescriptor) {
+    return new BasicExtractor<X>(javaTypeDescriptor, this) {
+      @Override
+      protected X doExtract(ResultSet rs, String name, WrapperOptions options) throws SQLException {
+        return javaTypeDescriptor.wrap(rs.getObject(name), options);
+      }
+
+      @Override
+      protected X doExtract(CallableStatement statement, int index, WrapperOptions options)
+          throws SQLException {
+        return javaTypeDescriptor.wrap(statement.getObject(index), options);
+      }
+
+      @Override
+      protected X doExtract(CallableStatement statement, String name, WrapperOptions options)
+          throws SQLException {
+        return javaTypeDescriptor.wrap(statement.getObject(name), options);
+      }
+    };
+  }
+}
@@ -15,6 +15,7 @@
 package google.registry.persistence.transaction;

 import google.registry.persistence.VKey;
+import java.util.function.Supplier;
 import javax.persistence.EntityManager;

 /** Sub-interface of {@link TransactionManager} which defines JPA related methods. */
@@ -23,6 +24,12 @@ public interface JpaTransactionManager extends TransactionManager {
  /** Returns the {@link EntityManager} for the current request. */
  EntityManager getEntityManager();

+  /** Executes the work in a transaction with no retries and returns the result. */
+  <T> T transactNoRetry(Supplier<T> work);
+
+  /** Executes the work in a transaction with no retries. */
+  void transactNoRetry(Runnable work);
+
  /** Deletes the entity by its id, throws exception if the entity is not deleted. */
  public abstract <T> void assertDelete(VKey<T> key);

@@ -27,8 +27,11 @@ import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.flogger.FluentLogger;
 import google.registry.config.RegistryConfig;
+import google.registry.persistence.JpaRetries;
 import google.registry.persistence.VKey;
 import google.registry.util.Clock;
+import google.registry.util.Retrier;
+import google.registry.util.SystemSleeper;
 import java.lang.reflect.Field;
 import java.util.Map.Entry;
 import java.util.NoSuchElementException;
@@ -49,6 +52,7 @@ import org.joda.time.DateTime;
 public class JpaTransactionManagerImpl implements JpaTransactionManager {

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+  private static final Retrier retrier = new Retrier(new SystemSleeper(), 3);

  // EntityManagerFactory is thread safe.
  private final EntityManagerFactory emf;
@@ -94,6 +98,40 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  public <T> T transact(Supplier<T> work) {
    // TODO(shicong): Investigate removing transactNew functionality after migration as it may
    //  be same as this one.
+    return retrier.callWithRetry(
+        () -> {
+          if (inTransaction()) {
+            return work.get();
+          }
+          TransactionInfo txnInfo = transactionInfo.get();
+          txnInfo.entityManager = emf.createEntityManager();
+          EntityTransaction txn = txnInfo.entityManager.getTransaction();
+          try {
+            txn.begin();
+            txnInfo.start(clock);
+            T result = work.get();
+            txnInfo.recordTransaction();
+            txn.commit();
+            return result;
+          } catch (RuntimeException | Error e) {
+            // Error is unchecked!
+            try {
+              txn.rollback();
+              logger.atWarning().log("Error during transaction; transaction rolled back");
+            } catch (Throwable rollbackException) {
+              logger.atSevere().withCause(rollbackException).log(
+                  "Rollback failed; suppressing error");
+            }
+            throw e;
+          } finally {
+            txnInfo.clear();
+          }
+        },
+        JpaRetries::isFailedTxnRetriable);
+  }
+
+  @Override
+  public <T> T transactNoRetry(Supplier<T> work) {
    if (inTransaction()) {
      return work.get();
    }
@@ -130,6 +168,15 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
        });
  }

+  @Override
+  public void transactNoRetry(Runnable work) {
+    transactNoRetry(
+        () -> {
+          work.run();
+          return null;
+        });
+  }
+
  @Override
  public <T> T transactNew(Supplier<T> work) {
    return transact(work);
@@ -142,11 +189,14 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {

  @Override
  public <T> T transactNewReadOnly(Supplier<T> work) {
-    return transact(
-        () -> {
-          getEntityManager().createNativeQuery("SET TRANSACTION READ ONLY").executeUpdate();
-          return work.get();
-        });
+    return retrier.callWithRetry(
+        () ->
+            transact(
+                () -> {
+                  getEntityManager().createNativeQuery("SET TRANSACTION READ ONLY").executeUpdate();
+                  return work.get();
+                }),
+        JpaRetries::isFailedQueryRetriable);
  }

  @Override
@@ -160,7 +210,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {

  @Override
  public <T> T doTransactionless(Supplier<T> work) {
-    return transact(work);
+    return retrier.callWithRetry(() -> transact(work), JpaRetries::isFailedQueryRetriable);
  }

  @Override
@@ -15,20 +15,11 @@
 package google.registry.tools;

 import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.collect.ImmutableList.toImmutableList;
 import static google.registry.util.CollectionUtils.findDuplicates;

-import com.beust.jcommander.IStringConverter;
 import com.beust.jcommander.Parameter;
-import com.google.auto.value.AutoValue;
-import com.google.common.base.Ascii;
-import com.google.common.base.CharMatcher;
 import com.google.common.base.Joiner;
-import com.google.common.base.Splitter;
 import com.google.common.collect.ImmutableSet;
-import com.google.common.io.BaseEncoding;
-import com.google.template.soy.data.SoyListData;
-import com.google.template.soy.data.SoyMapData;
 import google.registry.tools.params.NameserversParameter;
 import java.util.ArrayList;
 import java.util.HashSet;
@@ -80,77 +71,15 @@ abstract class CreateOrUpdateDomainCommand extends MutatingEppToolCommand {
  String password;

  @Parameter(
-    names = "--ds_records",
-    description =
-        "Comma-separated list of DS records. Each DS record is given as "
-        + "<keyTag> <alg> <digestType> <digest>, in order, as it appears in the Zonefile.",
-    converter = DsRecordConverter.class
-  )
+      names = "--ds_records",
+      description =
+          "Comma-separated list of DS records. Each DS record is given as "
+              + "<keyTag> <alg> <digestType> <digest>, in order, as it appears in the Zonefile.",
+      converter = DsRecord.Converter.class)
  List<DsRecord> dsRecords = new ArrayList<>();

  Set<String> domains;

-  @AutoValue
-  abstract static class DsRecord {
-    private static final Splitter SPLITTER =
-        Splitter.on(CharMatcher.whitespace()).omitEmptyStrings();
-
-    public abstract int keyTag();
-    public abstract int alg();
-    public abstract int digestType();
-    public abstract String digest();
-
-    private static DsRecord create(int keyTag, int alg, int digestType, String digest) {
-      digest = Ascii.toUpperCase(digest);
-      checkArgument(
-          BaseEncoding.base16().canDecode(digest),
-          "digest should be even-lengthed hex, but is %s (length %s)",
-          digest,
-          digest.length());
-      return new AutoValue_CreateOrUpdateDomainCommand_DsRecord(keyTag, alg, digestType, digest);
-    }
-
-    /**
-     * Parses a string representation of the DS record.
-     *
-     * <p>The string format accepted is "[keyTag] [alg] [digestType] [digest]" (i.e., the 4
-     * arguments separated by any number of spaces, as it appears in the Zone file)
-     */
-    public static DsRecord parse(String dsRecord) {
-      List<String> elements = SPLITTER.splitToList(dsRecord);
-      checkArgument(
-          elements.size() == 4,
-          "dsRecord %s should have 4 parts, but has %s",
-          dsRecord,
-          elements.size());
-      return DsRecord.create(
-          Integer.parseUnsignedInt(elements.get(0)),
-          Integer.parseUnsignedInt(elements.get(1)),
-          Integer.parseUnsignedInt(elements.get(2)),
-          elements.get(3));
-    }
-
-    public SoyMapData toSoyData() {
-      return new SoyMapData(
-          "keyTag", keyTag(),
-          "alg", alg(),
-          "digestType", digestType(),
-          "digest", digest());
-    }
-
-    public static SoyListData convertToSoy(List<DsRecord> dsRecords) {
-      return new SoyListData(
-          dsRecords.stream().map(DsRecord::toSoyData).collect(toImmutableList()));
-    }
-  }
-
-  public static class DsRecordConverter implements IStringConverter<DsRecord> {
-    @Override
-    public DsRecord convert(String dsRecord) {
-      return DsRecord.parse(dsRecord);
-    }
-  }
-
  @Override
  protected void initEppToolCommand() throws Exception {
    checkArgument(nameservers.size() <= 13, "There can be at most 13 nameservers.");
@@ -14,7 +14,10 @@

 package google.registry.tools;

+import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
+import google.registry.beam.initsql.BeamJpaModule.JpaTransactionManagerComponent;
+import google.registry.beam.initsql.JpaSupplierFactory;
 import google.registry.beam.spec11.Spec11Pipeline;
 import google.registry.config.CredentialModule.LocalCredential;
 import google.registry.config.RegistryConfig.Config;
@@ -31,6 +34,12 @@ public class DeploySpec11PipelineCommand implements Command {
  @Config("projectId")
  String projectId;

+  @Parameter(
+      names = {"-p", "--project"},
+      description = "Cloud KMS project ID",
+      required = true)
+  String cloudKmsProjectId;
+
  @Inject
  @Config("beamStagingUrl")
  String beamStagingUrl;
@@ -53,12 +62,19 @@ public class DeploySpec11PipelineCommand implements Command {

  @Override
  public void run() {
+    JpaSupplierFactory jpaSupplierFactory =
+        new JpaSupplierFactory(
+            sqlAccessInfoFile,
+            cloudKmsProjectId,
+            JpaTransactionManagerComponent::cloudSqlJpaTransactionManager);
+
    Spec11Pipeline pipeline =
        new Spec11Pipeline(
            projectId,
            beamStagingUrl,
            spec11TemplateUrl,
            reportingBucketUrl,
+            jpaSupplierFactory,
            googleCredentialsBundle,
            retrier);
    pipeline.deploy();
@@ -0,0 +1,90 @@
+// Copyright 2017 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.tools;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.collect.ImmutableList.toImmutableList;
+
+import com.beust.jcommander.IStringConverter;
+import com.google.auto.value.AutoValue;
+import com.google.common.base.Ascii;
+import com.google.common.base.CharMatcher;
+import com.google.common.base.Splitter;
+import com.google.common.io.BaseEncoding;
+import com.google.template.soy.data.SoyListData;
+import com.google.template.soy.data.SoyMapData;
+import java.util.List;
+
+@AutoValue
+abstract class DsRecord {
+  private static final Splitter SPLITTER = Splitter.on(CharMatcher.whitespace()).omitEmptyStrings();
+
+  public abstract int keyTag();
+
+  public abstract int alg();
+
+  public abstract int digestType();
+
+  public abstract String digest();
+
+  private static DsRecord create(int keyTag, int alg, int digestType, String digest) {
+    digest = Ascii.toUpperCase(digest);
+    checkArgument(
+        BaseEncoding.base16().canDecode(digest),
+        "digest should be even-lengthed hex, but is %s (length %s)",
+        digest,
+        digest.length());
+    return new AutoValue_DsRecord(keyTag, alg, digestType, digest);
+  }
+
+  /**
+   * Parses a string representation of the DS record.
+   *
+   * <p>The string format accepted is "[keyTag] [alg] [digestType] [digest]" (i.e., the 4 arguments
+   * separated by any number of spaces, as it appears in the Zone file)
+   */
+  public static DsRecord parse(String dsRecord) {
+    List<String> elements = SPLITTER.splitToList(dsRecord);
+    checkArgument(
+        elements.size() == 4,
+        "dsRecord %s should have 4 parts, but has %s",
+        dsRecord,
+        elements.size());
+    return DsRecord.create(
+        Integer.parseUnsignedInt(elements.get(0)),
+        Integer.parseUnsignedInt(elements.get(1)),
+        Integer.parseUnsignedInt(elements.get(2)),
+        elements.get(3));
+  }
+
+  public SoyMapData toSoyData() {
+    return new SoyMapData(
+        "keyTag", keyTag(),
+        "alg", alg(),
+        "digestType", digestType(),
+        "digest", digest());
+  }
+
+  public static SoyListData convertToSoy(List<DsRecord> dsRecords) {
+    return new SoyListData(dsRecords.stream().map(DsRecord::toSoyData).collect(toImmutableList()));
+  }
+
+  public static class Converter implements IStringConverter<DsRecord> {
+    @Override
+    public DsRecord convert(String dsRecord) {
+      return DsRecord.parse(dsRecord);
+    }
+  }
+}
@@ -115,7 +115,20 @@ class GenerateAllocationTokensCommand implements CommandWithRemoteApi {
      description =
          "A discount off the base price for the first year between 0.0 and 1.0. Default is 0.0,"
              + " i.e. no discount.")
-  private double discountFraction;
+  private Double discountFraction;
+
+  @Parameter(
+      names = {"--discount_premiums"},
+      description =
+          "Whether the discount is valid for premium names in addition to standard ones. Default"
+              + " is false.",
+      arity = 1)
+  private Boolean discountPremiums;
+
+  @Parameter(
+      names = {"--discount_years"},
+      description = "The number of years the discount applies for. Default is 1, max value is 10.")
+  private Integer discountYears;

  @Parameter(
      names = "--token_status_transitions",
@@ -170,8 +183,10 @@ class GenerateAllocationTokensCommand implements CommandWithRemoteApi {
                            .setToken(t)
                            .setTokenType(tokenType == null ? SINGLE_USE : tokenType)
                            .setAllowedClientIds(ImmutableSet.copyOf(nullToEmpty(allowedClientIds)))
-                            .setAllowedTlds(ImmutableSet.copyOf(nullToEmpty(allowedTlds)))
-                            .setDiscountFraction(discountFraction);
+                            .setAllowedTlds(ImmutableSet.copyOf(nullToEmpty(allowedTlds)));
+                    Optional.ofNullable(discountFraction).ifPresent(token::setDiscountFraction);
+                    Optional.ofNullable(discountPremiums).ifPresent(token::setDiscountPremiums);
+                    Optional.ofNullable(discountYears).ifPresent(token::setDiscountYears);
                    Optional.ofNullable(tokenStatusTransitions)
                        .ifPresent(token::setTokenStatusTransitions);
                    Optional.ofNullable(domainNames)
@@ -27,7 +27,7 @@ import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.token.AllocationToken;
 import java.util.Collection;
 import java.util.List;
-import java.util.Objects;
+import java.util.Optional;

 /** Command to show allocation tokens. */
@Parameters(separators = " =", commandDescription = "Show allocation token(s)")
@@ -55,11 +55,11 @@ final class GetAllocationTokenCommand implements CommandWithRemoteApi {
      if (loadedTokens.containsKey(token)) {
        AllocationToken loadedToken = loadedTokens.get(token);
        System.out.println(loadedToken.toString());
-        if (loadedToken.getRedemptionHistoryEntry() == null) {
+        if (!loadedToken.getRedemptionHistoryEntry().isPresent()) {
          System.out.printf("Token %s was not redeemed.\n", token);
        } else {
          DomainBase domain =
-              domains.get(loadedToken.getRedemptionHistoryEntry().<DomainBase>getParent());
+              domains.get(loadedToken.getRedemptionHistoryEntry().get().<DomainBase>getParent());
          if (domain == null) {
            System.out.printf("ERROR: Token %s was redeemed but domain can't be loaded.\n", token);
          } else {
@@ -80,7 +80,8 @@ final class GetAllocationTokenCommand implements CommandWithRemoteApi {
    ImmutableList<Key<DomainBase>> domainKeys =
        tokens.stream()
            .map(AllocationToken::getRedemptionHistoryEntry)
-            .filter(Objects::nonNull)
+            .filter(Optional::isPresent)
+            .map(Optional::get)
            .map(Key::<DomainBase>getParent)
            .collect(toImmutableList());
    ImmutableMap.Builder<Key<DomainBase>, DomainBase> domainsBuilder = new ImmutableMap.Builder<>();
@@ -29,7 +29,6 @@ import com.google.appengine.tools.remoteapi.RemoteApiOptions;
 import com.google.common.base.Throwables;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Iterables;
-import google.registry.backup.AppEngineEnvironment;
 import google.registry.config.RegistryConfig;
 import google.registry.model.ofy.ObjectifyService;
 import google.registry.persistence.transaction.TransactionManagerFactory;
@@ -179,7 +178,7 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
            Iterables.getOnlyElement(jcommander.getCommands().get(parsedCommand).getObjects());
    loggingParams.configureLogging();  // Must be called after parameters are parsed.

-    try (AppEngineEnvironment env = new AppEngineEnvironment()) {
+    try {
      runCommand(command);
    } catch (RuntimeException ex) {
      if (Throwables.getRootCause(ex) instanceof LoginRequiredException) {
@@ -15,7 +15,7 @@
 package google.registry.tools;

 import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Strings.nullToEmpty;
+import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.collect.Sets.difference;
 import static google.registry.model.EppResourceUtils.checkResourcesExist;
 import static google.registry.model.EppResourceUtils.loadByForeignKey;
@@ -26,9 +26,11 @@ import static org.joda.time.DateTimeZone.UTC;
 import com.beust.jcommander.Parameter;
 import com.beust.jcommander.Parameters;
 import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedSet;
+import com.google.template.soy.data.SoyListData;
 import com.google.template.soy.data.SoyMapData;
 import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.secdns.DelegationSignerData;
@@ -37,14 +39,10 @@ import google.registry.model.host.HostResource;
 import google.registry.tools.soy.UniformRapidSuspensionSoyInfo;
 import java.util.ArrayList;
 import java.util.List;
-import java.util.Map;
 import java.util.Optional;
 import java.util.Set;
 import javax.xml.bind.annotation.adapters.HexBinaryAdapter;
 import org.joda.time.DateTime;
-import org.json.simple.JSONArray;
-import org.json.simple.JSONValue;
-import org.json.simple.parser.ParseException;

 /** A command to suspend a domain for the Uniform Rapid Suspension process. */
@Parameters(separators = " =",
@@ -59,9 +57,6 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {
  /** Client id that made this change. Only recorded in the history entry. **/
  private static final String CLIENT_ID = "CharlestonRoad";

-  private static final ImmutableSet<String> DSDATA_FIELDS =
-      ImmutableSet.of("keyTag", "alg", "digestType", "digest");
-
  @Parameter(
      names = {"-n", "--domain_name"},
      description = "Domain to suspend.",
@@ -76,10 +71,12 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {

  @Parameter(
      names = {"-s", "--dsdata"},
-      description = "Comma-delimited set of dsdata to replace the current dsdata on the domain, "
-          + "where each dsdata is represented as a JSON object with fields 'keyTag', 'alg', "
-          + "'digestType' and 'digest'.")
-  private String newDsData;
+      description =
+          "Comma-delimited set of dsdata to replace the current dsdata on the domain, "
+              + "Each DS record is given as <keyTag> <alg> <digestType> <digest>, in order, as it "
+              + "appears in the Zonefile.",
+      converter = DsRecord.Converter.class)
+  private List<DsRecord> newDsData;

  @Parameter(
      names = {"-p", "--locks_to_preserve"},
@@ -88,6 +85,13 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {
              + "locks: serverDeleteProhibited, serverTransferProhibited, serverUpdateProhibited")
  private List<String> locksToPreserve = new ArrayList<>();

+  @Parameter(
+      names = {"--restore_client_hold"},
+      description =
+          "Restores a CLIENT_HOLD status that was previously removed for a URS suspension (only "
+              + "valid with --undo).")
+  private boolean restoreClientHold;
+
  @Parameter(
      names = {"--undo"},
      description = "Flag indicating that is is an undo command, which removes locks.")
@@ -100,28 +104,16 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {
  ImmutableSortedSet<String> existingNameservers;

  /** Set of existing dsdata jsons that need to be restored during undo, sorted for nicer output. */
-  ImmutableSortedSet<String> existingDsData;
+  ImmutableList<ImmutableMap<String, Object>> existingDsData;
+
+  /** Set of status values to remove. */
+  ImmutableSet<String> removeStatuses;

  @Override
  protected void initMutatingEppToolCommand() {
    superuser = true;
    DateTime now = DateTime.now(UTC);
    ImmutableSet<String> newHostsSet = ImmutableSet.copyOf(newHosts);
-    ImmutableSet.Builder<Map<String, Object>> newDsDataBuilder = new ImmutableSet.Builder<>();
-    try {
-      // Add brackets around newDsData to convert it to a parsable JSON array.
-      String jsonArrayString = String.format("[%s]", nullToEmpty(newDsData));
-      for (Object dsData : (JSONArray) JSONValue.parseWithException(jsonArrayString)) {
-        @SuppressWarnings("unchecked")
-        Map<String, Object> dsDataJson = (Map<String, Object>) dsData;
-        checkArgument(
-            dsDataJson.keySet().equals(DSDATA_FIELDS),
-            "Incorrect fields on --dsdata JSON: " + JSONValue.toJSONString(dsDataJson));
-        newDsDataBuilder.add(dsDataJson);
-      }
-    } catch (ClassCastException | ParseException e) {
-      throw new IllegalArgumentException("Invalid --dsdata JSON", e);
-    }
    Optional<DomainBase> domain = loadByForeignKey(DomainBase.class, domainName, now);
    checkArgumentPresent(domain, "Domain '%s' does not exist or is deleted", domainName);
    Set<String> missingHosts =
@@ -133,18 +125,39 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {
    existingNameservers = getExistingNameservers(domain.get());
    existingLocks = getExistingLocks(domain.get());
    existingDsData = getExistingDsData(domain.get());
+    removeStatuses =
+        (hasClientHold(domain.get()) && !undo)
+            ? ImmutableSet.of(StatusValue.CLIENT_HOLD.getXmlName())
+            : ImmutableSet.of();
+    ImmutableSet<String> statusesToApply;
+    if (undo) {
+      statusesToApply =
+          restoreClientHold
+              ? ImmutableSet.of(StatusValue.CLIENT_HOLD.getXmlName())
+              : ImmutableSet.of();
+    } else {
+      statusesToApply = URS_LOCKS;
+    }
    setSoyTemplate(
        UniformRapidSuspensionSoyInfo.getInstance(),
        UniformRapidSuspensionSoyInfo.UNIFORMRAPIDSUSPENSION);
-    addSoyRecord(CLIENT_ID, new SoyMapData(
-        "domainName", domainName,
-        "hostsToAdd", difference(newHostsSet, existingNameservers),
-        "hostsToRemove", difference(existingNameservers, newHostsSet),
-        "locksToApply", undo ? ImmutableSet.of() : URS_LOCKS,
-        "locksToRemove",
-            undo ? difference(URS_LOCKS, ImmutableSet.copyOf(locksToPreserve)) : ImmutableSet.of(),
-        "newDsData", newDsDataBuilder.build(),
-        "reason", (undo ? "Undo " : "") + "Uniform Rapid Suspension"));
+    addSoyRecord(
+        CLIENT_ID,
+        new SoyMapData(
+            "domainName",
+            domainName,
+            "hostsToAdd",
+            difference(newHostsSet, existingNameservers),
+            "hostsToRemove",
+            difference(existingNameservers, newHostsSet),
+            "statusesToApply",
+            statusesToApply,
+            "statusesToRemove",
+            undo ? difference(URS_LOCKS, ImmutableSet.copyOf(locksToPreserve)) : removeStatuses,
+            "newDsData",
+            newDsData != null ? DsRecord.convertToSoy(newDsData) : new SoyListData(),
+            "reason",
+            (undo ? "Undo " : "") + "Uniform Rapid Suspension"));
  }

  private ImmutableSortedSet<String> getExistingNameservers(DomainBase domain) {
@@ -165,15 +178,25 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {
    return locks.build();
  }

-  private ImmutableSortedSet<String> getExistingDsData(DomainBase domain) {
-    ImmutableSortedSet.Builder<String> dsDataJsons = ImmutableSortedSet.naturalOrder();
+  private boolean hasClientHold(DomainBase domain) {
+    for (StatusValue status : domain.getStatusValues()) {
+      if (status == StatusValue.CLIENT_HOLD) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  private ImmutableList<ImmutableMap<String, Object>> getExistingDsData(DomainBase domain) {
+    ImmutableList.Builder<ImmutableMap<String, Object>> dsDataJsons = new ImmutableList.Builder();
    HexBinaryAdapter hexBinaryAdapter = new HexBinaryAdapter();
    for (DelegationSignerData dsData : domain.getDsData()) {
-      dsDataJsons.add(JSONValue.toJSONString(ImmutableMap.of(
-          "keyTag", dsData.getKeyTag(),
-          "algorithm", dsData.getAlgorithm(),
-          "digestType", dsData.getDigestType(),
-          "digest", hexBinaryAdapter.marshal(dsData.getDigest()))));
+      dsDataJsons.add(
+          ImmutableMap.of(
+              "keyTag", dsData.getKeyTag(),
+              "algorithm", dsData.getAlgorithm(),
+              "digestType", dsData.getDigestType(),
+              "digest", hexBinaryAdapter.marshal(dsData.getDigest())));
    }
    return dsDataJsons.build();
  }
@@ -194,8 +217,23 @@ final class UniformRapidSuspensionCommand extends MutatingEppToolCommand {
    if (!existingLocks.isEmpty()) {
      undoBuilder.append(" --locks_to_preserve ").append(Joiner.on(',').join(existingLocks));
    }
+    if (removeStatuses.contains(StatusValue.CLIENT_HOLD.getXmlName())) {
+      undoBuilder.append(" --restore_client_hold");
+    }
    if (!existingDsData.isEmpty()) {
-      undoBuilder.append(" --dsdata ").append(Joiner.on(',').join(existingDsData));
+      ImmutableList<String> formattedDsRecords =
+          existingDsData.stream()
+              .map(
+                  rec ->
+                      String.format(
+                          "%s %s %s %s",
+                          rec.get("keyTag"),
+                          rec.get("algorithm"),
+                          rec.get("digestType"),
+                          rec.get("digest")))
+              .sorted()
+              .collect(toImmutableList());
+      undoBuilder.append(" --dsdata ").append(Joiner.on(',').join(formattedDsRecords));
    }
    return undoBuilder.toString();
  }
@@ -66,6 +66,19 @@ final class UpdateAllocationTokensCommand extends UpdateOrDeleteAllocationTokens
              + "i.e. no discount.")
  private Double discountFraction;

+  @Parameter(
+      names = {"--discount_premiums"},
+      description =
+          "Whether the discount is valid for premium names in addition to standard ones. Default"
+              + " is false.",
+      arity = 1)
+  private Boolean discountPremiums;
+
+  @Parameter(
+      names = {"--discount_years"},
+      description = "The number of years the discount applies for. Default is 1, max value is 10.")
+  private Integer discountYears;
+
  @Parameter(
      names = "--token_status_transitions",
      converter = TokenStatusTransitions.class,
@@ -122,6 +135,8 @@ final class UpdateAllocationTokensCommand extends UpdateOrDeleteAllocationTokens
    Optional.ofNullable(allowedTlds)
        .ifPresent(tlds -> builder.setAllowedTlds(ImmutableSet.copyOf(tlds)));
    Optional.ofNullable(discountFraction).ifPresent(builder::setDiscountFraction);
+    Optional.ofNullable(discountPremiums).ifPresent(builder::setDiscountPremiums);
+    Optional.ofNullable(discountYears).ifPresent(builder::setDiscountYears);
    Optional.ofNullable(tokenStatusTransitions).ifPresent(builder::setTokenStatusTransitions);
    return builder.build();
  }
@@ -76,10 +76,10 @@ final class UpdateDomainCommand extends CreateOrUpdateDomainCommand {
  private List<String> addStatuses = new ArrayList<>();

  @Parameter(
-    names = "--add_ds_records",
-    description = "DS records to add. Cannot be set if --ds_records or --clear_ds_records is set.",
-    converter = DsRecordConverter.class
-  )
+      names = "--add_ds_records",
+      description =
+          "DS records to add. Cannot be set if --ds_records or --clear_ds_records is set.",
+      converter = DsRecord.Converter.class)
  private List<DsRecord> addDsRecords = new ArrayList<>();

  @Parameter(
@@ -110,11 +110,10 @@ final class UpdateDomainCommand extends CreateOrUpdateDomainCommand {
  private List<String> removeStatuses = new ArrayList<>();

  @Parameter(
-    names = "--remove_ds_records",
-    description =
-        "DS records to remove. Cannot be set if --ds_records or --clear_ds_records is set.",
-    converter = DsRecordConverter.class
-  )
+      names = "--remove_ds_records",
+      description =
+          "DS records to remove. Cannot be set if --ds_records or --clear_ds_records is set.",
+      converter = DsRecord.Converter.class)
  private List<DsRecord> removeDsRecords = new ArrayList<>();

  @Parameter(
@@ -25,6 +25,7 @@
    <class>google.registry.model.contact.ContactHistory</class>
    <class>google.registry.model.contact.ContactResource</class>
    <class>google.registry.model.domain.DomainBase</class>
+    <class>google.registry.model.domain.DomainHistory</class>
    <class>google.registry.model.host.HostHistory</class>
    <class>google.registry.model.host.HostResource</class>
    <class>google.registry.model.registrar.Registrar</class>
@@ -21,8 +21,8 @@
 {@param domainName: string}
 {@param hostsToAdd: list<string>}
 {@param hostsToRemove: list<string>}
-{@param locksToApply: list<string>}
-{@param locksToRemove: list<string>}
+{@param statusesToApply: list<string>}
+{@param statusesToRemove: list<string>}
 {@param newDsData: list<[keyTag:int, alg:int, digestType:int, digest:string]>}
 {@param reason: string}
 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
@@ -39,7 +39,7 @@
          {/for}
          </domain:ns>
        {/if}
-        {for $la in $locksToApply}
+        {for $la in $statusesToApply}
          <domain:status s="{$la}" />
        {/for}
        </domain:add>
@@ -51,7 +51,7 @@
          {/for}
          </domain:ns>
        {/if}
-        {for $lr in $locksToRemove}
+        {for $lr in $statusesToRemove}
          <domain:status s="{$lr}" />
        {/for}
        </domain:rem>
@@ -21,8 +21,8 @@ import static google.registry.backup.BackupUtils.GcsMetadataKeys.LOWER_BOUND_CHE
 import static google.registry.backup.ExportCommitLogDiffAction.DIFF_FILE_PREFIX;
 import static java.lang.reflect.Proxy.newProxyInstance;
 import static org.joda.time.DateTimeZone.UTC;
-import static org.junit.Assert.assertThrows;
-import static org.junit.Assert.fail;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;

 import com.google.appengine.tools.cloudstorage.GcsFileMetadata;
 import com.google.appengine.tools.cloudstorage.GcsFileOptions;
@@ -31,7 +31,7 @@ import static google.registry.testing.TestLogHandlerUtils.assertLogMessage;
 import static org.joda.time.Duration.standardDays;
 import static org.joda.time.Duration.standardHours;
 import static org.joda.time.Duration.standardSeconds;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.when;

 import com.google.common.collect.ImmutableSortedSet;
@@ -30,7 +30,7 @@ import static google.registry.testing.TaskQueueHelper.assertDnsTasksEnqueued;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
 import static org.joda.time.DateTimeZone.UTC;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.common.collect.ImmutableSet;
 import com.googlecode.objectify.Key;
@@ -31,7 +31,7 @@ import static google.registry.testing.DatastoreHelper.persistResource;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
 import static org.joda.money.CurrencyUnit.USD;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedMap;
@@ -117,9 +117,10 @@ public class ResaveEntityActionTest {

  @Test
  void test_domainPendingDeletion_isResavedAndReenqueued() {
+    DomainBase newDomain = newDomainBase("domain.tld");
    DomainBase domain =
        persistResource(
-            newDomainBase("domain.tld")
+            newDomain
                .asBuilder()
                .setDeletionTime(clock.nowUtc().plusDays(35))
                .setStatusValues(ImmutableSet.of(StatusValue.PENDING_DELETE))
@@ -127,6 +128,7 @@ public class ResaveEntityActionTest {
                    ImmutableSet.of(
                        GracePeriod.createWithoutBillingEvent(
                            GracePeriodStatus.REDEMPTION,
+                            newDomain.getRepoId(),
                            clock.nowUtc().plusDays(30),
                            "TheRegistrar")))
                .build());
@@ -15,7 +15,7 @@
 package google.registry.beam;

 import static com.google.common.truth.Truth.assertThat;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.common.collect.ImmutableList;
 import org.apache.avro.Schema;
@@ -27,9 +27,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
 import com.google.common.base.Strings;
 import com.google.common.collect.Maps;
 import java.io.IOException;
-import java.lang.annotation.Annotation;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
@@ -59,7 +57,6 @@ import org.apache.beam.sdk.testing.TestPipelineOptions;
 import org.apache.beam.sdk.testing.ValidatesRunner;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.junit.experimental.categories.Category;
 import org.junit.jupiter.api.extension.AfterEachCallback;
 import org.junit.jupiter.api.extension.BeforeEachCallback;
 import org.junit.jupiter.api.extension.ExtensionContext;
@@ -89,11 +86,10 @@ import org.junit.jupiter.api.extension.ExtensionContext;
 * remote execution modes. For example:
 *
 * <pre><code>
- * {@literal @Rule}
- *  public final transient TestPipeline p = TestPipeline.create();
+ * {@literal @RegisterExtension}
+ *  final transient TestPipeline p = TestPipeline.create();
 *
 * {@literal @Test}
- * {@literal @Category}(NeedsRunner.class)
 *  public void myPipelineTest() throws Exception {
 *    final PCollection&lt;String&gt; pCollection = pipeline.apply(...)
 *    PAssert.that(pCollection).containsInAnyOrder(...);
@@ -119,17 +115,17 @@ public class TestPipelineExtension extends Pipeline

    protected final Pipeline pipeline;

-    protected boolean runAttempted;
+    boolean runAttempted;

    private PipelineRunEnforcement(final Pipeline pipeline) {
      this.pipeline = pipeline;
    }

-    protected void enableAutoRunIfMissing(final boolean enable) {
+    void enableAutoRunIfMissing(final boolean enable) {
      enableAutoRunIfMissing = enable;
    }

-    protected void beforePipelineExecution() {
+    void beforePipelineExecution() {
      runAttempted = true;
    }

@@ -248,9 +244,9 @@ public class TestPipelineExtension extends Pipeline
  }

  /** System property used to set {@link TestPipelineOptions}. */
-  public static final String PROPERTY_BEAM_TEST_PIPELINE_OPTIONS = "beamTestPipelineOptions";
+  private static final String PROPERTY_BEAM_TEST_PIPELINE_OPTIONS = "beamTestPipelineOptions";

-  static final String PROPERTY_USE_DEFAULT_DUMMY_RUNNER = "beamUseDummyRunner";
+  private static final String PROPERTY_USE_DEFAULT_DUMMY_RUNNER = "beamUseDummyRunner";

  private static final ObjectMapper MAPPER =
      new ObjectMapper()
@@ -332,8 +328,9 @@ public class TestPipelineExtension extends Pipeline
  public PipelineResult run(PipelineOptions options) {
    checkState(
        enforcement.isPresent(),
-        "Is your TestPipeline declaration missing a @Rule annotation? Usage: "
-            + "@Rule public final transient TestPipeline pipeline = TestPipeline.create();");
+        "Is your TestPipeline declaration missing a @RegisterExtension annotation? Usage:"
+            + " @RegisterExtension final transient TestPipelineExtension pipeline ="
+            + " TestPipeline.create();");

    final PipelineResult pipelineResult;
    try {
@@ -434,7 +431,7 @@ public class TestPipelineExtension extends Pipeline
  }

  /** Creates {@link PipelineOptions} for testing. */
-  public static PipelineOptions testingPipelineOptions() {
+  private static PipelineOptions testingPipelineOptions() {
    try {
      @Nullable
      String beamTestPipelineOptions = System.getProperty(PROPERTY_BEAM_TEST_PIPELINE_OPTIONS);
@@ -474,7 +471,7 @@ public class TestPipelineExtension extends Pipeline
   * <p>Note this only runs for runners which support Metrics. Runners which do not should verify
   * this in some other way. See: https://issues.apache.org/jira/browse/BEAM-2001
   */
-  public static void verifyPAssertsSucceeded(Pipeline pipeline, PipelineResult pipelineResult) {
+  private static void verifyPAssertsSucceeded(Pipeline pipeline, PipelineResult pipelineResult) {
    if (MetricsEnvironment.isMetricsSupported()) {
      long expectedNumberOfAssertions = (long) PAssert.countAsserts(pipeline);

@@ -514,29 +511,4 @@ public class TestPipelineExtension extends Pipeline
      empty = false;
    }
  }
-
-  /**
-   * A utility class for querying annotations.
-   *
-   * <p>NOTE: This was copied from the Apache Beam project from a separate file only for visibility
-   * reasons (it's package-private there).
-   */
-  static class Annotations {
-
-    /** Annotation predicates. */
-    static class Predicates {
-
-      static Predicate<Annotation> isAnnotationOfType(final Class<? extends Annotation> clazz) {
-        return annotation ->
-            annotation.annotationType() != null && annotation.annotationType().equals(clazz);
-      }
-
-      static Predicate<Annotation> isCategoryOf(final Class<?> value, final boolean allowDerived) {
-        return category ->
-            Arrays.stream(((Category) category).value())
-                .anyMatch(
-                    aClass -> allowDerived ? value.isAssignableFrom(aClass) : value.equals(aClass));
-      }
-    }
-  }
 }
@@ -109,9 +109,9 @@ public class BackupTestStoreTest {
            .map(string -> string.substring(exportFolder.getAbsolutePath().length()))) {
      assertThat(files)
          .containsExactly(
-              "/all_namespaces/kind_Registry/input-0",
-              "/all_namespaces/kind_DomainBase/input-0",
-              "/all_namespaces/kind_ContactResource/input-0");
+              "/all_namespaces/kind_Registry/output-0",
+              "/all_namespaces/kind_DomainBase/output-0",
+              "/all_namespaces/kind_ContactResource/output-0");
    }
  }

@@ -132,16 +132,16 @@ public class BackupTestStoreTest {
    String exportRootPath = tempDir.getAbsolutePath();
    File exportFolder = export(exportRootPath, Collections.EMPTY_SET);
    ImmutableList<Object> loadedRegistries =
-        loadExportedEntities(new File(exportFolder, "/all_namespaces/kind_Registry/input-0"));
+        loadExportedEntities(new File(exportFolder, "/all_namespaces/kind_Registry/output-0"));
    assertThat(loadedRegistries).containsExactly(registry);

    ImmutableList<Object> loadedDomains =
-        loadExportedEntities(new File(exportFolder, "/all_namespaces/kind_DomainBase/input-0"));
+        loadExportedEntities(new File(exportFolder, "/all_namespaces/kind_DomainBase/output-0"));
    assertThat(loadedDomains).containsExactly(domain);

    ImmutableList<Object> loadedContacts =
        loadExportedEntities(
-            new File(exportFolder, "/all_namespaces/kind_ContactResource/input-0"));
+            new File(exportFolder, "/all_namespaces/kind_ContactResource/output-0"));
    assertThat(loadedContacts).containsExactly(contact);
  }

@@ -156,7 +156,7 @@ public class BackupTestStoreTest {
        export(
            exportRootPath, ImmutableSet.of(Key.create(getCrossTldKey(), Registry.class, "tld1")));
    ImmutableList<Object> loadedRegistries =
-        loadExportedEntities(new File(exportFolder, "/all_namespaces/kind_Registry/input-0"));
+        loadExportedEntities(new File(exportFolder, "/all_namespaces/kind_Registry/output-0"));
    assertThat(loadedRegistries).containsExactly(newRegistry);
  }

@@ -21,7 +21,7 @@ import static google.registry.testing.DatastoreHelper.cloneAndSetAutoTimestamps;
 import static google.registry.testing.DatastoreHelper.createTld;
 import static google.registry.testing.DatastoreHelper.persistResource;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.appengine.api.datastore.Entity;
 import com.google.common.collect.ImmutableSet;
@@ -161,6 +161,7 @@ public class DomainBaseUtilTest {
                    .addGracePeriod(
                        GracePeriod.create(
                            GracePeriodStatus.ADD,
+                            "4-COM",
                            fakeClock.nowUtc().plusDays(1),
                            "registrar",
                            null))
@@ -119,9 +119,9 @@ class ExportloadingTransformsTest implements Serializable {

    ImmutableList<String> expectedPatterns =
        ImmutableList.of(
-            exportDir.getAbsolutePath() + "/all_namespaces/kind_Registry/input-*",
-            exportDir.getAbsolutePath() + "/all_namespaces/kind_DomainBase/input-*",
-            exportDir.getAbsolutePath() + "/all_namespaces/kind_ContactResource/input-*");
+            exportDir.getAbsolutePath() + "/all_namespaces/kind_Registry/output-*",
+            exportDir.getAbsolutePath() + "/all_namespaces/kind_DomainBase/output-*",
+            exportDir.getAbsolutePath() + "/all_namespaces/kind_ContactResource/output-*");

    PAssert.that(patterns).containsInAnyOrder(expectedPatterns);

@@ -135,10 +135,10 @@ class ExportloadingTransformsTest implements Serializable {
            .apply(
                "File patterns to metadata",
                Create.of(
-                        exportDir.getAbsolutePath() + "/all_namespaces/kind_Registry/input-*",
-                        exportDir.getAbsolutePath() + "/all_namespaces/kind_DomainBase/input-*",
+                        exportDir.getAbsolutePath() + "/all_namespaces/kind_Registry/output-*",
+                        exportDir.getAbsolutePath() + "/all_namespaces/kind_DomainBase/output-*",
                        exportDir.getAbsolutePath()
-                            + "/all_namespaces/kind_ContactResource/input-*")
+                            + "/all_namespaces/kind_ContactResource/output-*")
                    .withCoder(StringUtf8Coder.of()))
            .apply(Transforms.getFilesByPatterns());

@@ -157,9 +157,9 @@ class ExportloadingTransformsTest implements Serializable {

    ImmutableList<String> expectedFilenames =
        ImmutableList.of(
-            exportDir.getAbsolutePath() + "/all_namespaces/kind_Registry/input-0",
-            exportDir.getAbsolutePath() + "/all_namespaces/kind_DomainBase/input-0",
-            exportDir.getAbsolutePath() + "/all_namespaces/kind_ContactResource/input-0");
+            exportDir.getAbsolutePath() + "/all_namespaces/kind_Registry/output-0",
+            exportDir.getAbsolutePath() + "/all_namespaces/kind_DomainBase/output-0",
+            exportDir.getAbsolutePath() + "/all_namespaces/kind_ContactResource/output-0");

    PAssert.that(fileNames).containsInAnyOrder(expectedFilenames);

@@ -214,7 +214,11 @@ class InitSqlPipelineTest {
                  .setSmdId("smdid")
                  .addGracePeriod(
                      GracePeriod.create(
-                          GracePeriodStatus.ADD, fakeClock.nowUtc().plusDays(1), "registrar", null))
+                          GracePeriodStatus.ADD,
+                          "4-COM",
+                          fakeClock.nowUtc().plusDays(1),
+                          "registrar",
+                          null))
                  .build());
      exportDir = store.export(exportRootDir.getAbsolutePath(), ALL_KINDS, ImmutableSet.of());
      commitLogDir = Files.createDirectory(tmpDir.resolve("commits")).toFile();
@@ -15,7 +15,7 @@
 package google.registry.beam.invoicing;

 import static com.google.common.truth.Truth.assertThat;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import google.registry.beam.invoicing.BillingEvent.InvoiceGroupingKey;
 import google.registry.beam.invoicing.BillingEvent.InvoiceGroupingKey.InvoiceGroupingKeyCoder;
@@ -17,15 +17,22 @@ package google.registry.beam.spec11;
 import static com.google.common.truth.Truth.assertThat;
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
 import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
 import static org.mockito.Mockito.when;
 import static org.mockito.Mockito.withSettings;

 import com.google.auth.oauth2.GoogleCredentials;
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
 import com.google.common.io.CharStreams;
 import google.registry.beam.TestPipelineExtension;
 import google.registry.beam.spec11.SafeBrowsingTransforms.EvaluateSafeBrowsingFn;
+import google.registry.model.reporting.Spec11ThreatMatch;
+import google.registry.model.reporting.Spec11ThreatMatch.ThreatType;
+import google.registry.persistence.transaction.JpaTransactionManager;
 import google.registry.testing.FakeClock;
 import google.registry.testing.FakeSleeper;
 import google.registry.util.GoogleCredentialsBundle;
@@ -55,22 +62,42 @@ import org.apache.http.entity.BasicHttpEntity;
 import org.apache.http.impl.client.CloseableHttpClient;
 import org.apache.http.message.BasicStatusLine;
 import org.joda.time.DateTime;
+import org.joda.time.LocalDate;
+import org.joda.time.format.ISODateTimeFormat;
 import org.json.JSONArray;
 import org.json.JSONException;
 import org.json.JSONObject;
 import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
 import org.junit.jupiter.api.extension.RegisterExtension;
 import org.junit.jupiter.api.io.TempDir;
+import org.mockito.Mock;
 import org.mockito.invocation.InvocationOnMock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
 import org.mockito.stubbing.Answer;

 /** Unit tests for {@link Spec11Pipeline}. */
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.LENIENT)
 class Spec11PipelineTest {
+  private static class SaveNewThreatMatchAnswer implements Answer<Void>, Serializable {
+    @Override
+    public Void answer(InvocationOnMock invocation) {
+      Runnable runnable = invocation.getArgument(0, Runnable.class);
+      runnable.run();
+      return null;
+    }
+  }

  private static PipelineOptions pipelineOptions;

+  @Mock(serializable = true)
+  private static JpaTransactionManager mockJpaTm;
+
  @BeforeAll
  static void beforeAll() {
    pipelineOptions = PipelineOptionsFactory.create();
@@ -93,20 +120,23 @@ class Spec11PipelineTest {
  void beforeEach() throws IOException {
    String beamTempFolder =
        Files.createDirectory(tmpDir.resolve("beam_temp")).toAbsolutePath().toString();
+
    spec11Pipeline =
        new Spec11Pipeline(
            "test-project",
            beamTempFolder + "/staging",
            beamTempFolder + "/templates/invoicing",
            tmpDir.toAbsolutePath().toString(),
+            () -> mockJpaTm,
            GoogleCredentialsBundle.create(GoogleCredentials.create(null)),
            retrier);
  }

  private static final ImmutableList<String> BAD_DOMAINS =
-      ImmutableList.of("111.com", "222.com", "444.com", "no-email.com");
+      ImmutableList.of(
+          "111.com", "222.com", "444.com", "no-email.com", "testThreatMatchToSqlBad.com");

-  private ImmutableList<Subdomain> getInputDomains() {
+  private ImmutableList<Subdomain> getInputDomainsJson() {
    ImmutableList.Builder<Subdomain> subdomainsBuilder = new ImmutableList.Builder<>();
    // Put in at least 2 batches worth (x > 490) to guarantee multiple executions.
    // Put in half for theRegistrar and half for someRegistrar
@@ -134,17 +164,18 @@ class Spec11PipelineTest {
  @SuppressWarnings("unchecked")
  void testEndToEndPipeline_generatesExpectedFiles() throws Exception {
    // Establish mocks for testing
-    ImmutableList<Subdomain> inputRows = getInputDomains();
-    CloseableHttpClient httpClient = mock(CloseableHttpClient.class, withSettings().serializable());
+    ImmutableList<Subdomain> inputRows = getInputDomainsJson();
+    CloseableHttpClient mockHttpClient =
+        mock(CloseableHttpClient.class, withSettings().serializable());

    // Return a mock HttpResponse that returns a JSON response based on the request.
-    when(httpClient.execute(any(HttpPost.class))).thenAnswer(new HttpResponder());
+    when(mockHttpClient.execute(any(HttpPost.class))).thenAnswer(new HttpResponder());

    EvaluateSafeBrowsingFn evalFn =
        new EvaluateSafeBrowsingFn(
            StaticValueProvider.of("apikey"),
            new Retrier(new FakeSleeper(new FakeClock()), 3),
-            (Serializable & Supplier) () -> httpClient);
+            (Serializable & Supplier) () -> mockHttpClient);

    // Apply input and evaluation transforms
    PCollection<Subdomain> input = testPipeline.apply(Create.of(inputRows));
@@ -207,6 +238,56 @@ class Spec11PipelineTest {
                .toString());
  }

+  @Test
+  @SuppressWarnings("unchecked")
+  public void testSpec11ThreatMatchToSql() throws Exception {
+    doAnswer(new SaveNewThreatMatchAnswer()).when(mockJpaTm).transact(any(Runnable.class));
+
+    // Create one bad and one good Subdomain to test with evaluateUrlHealth. Only the bad one should
+    // be detected and persisted.
+    Subdomain badDomain =
+        Subdomain.create(
+            "testThreatMatchToSqlBad.com", "theDomain", "theRegistrar", "fake@theRegistrar.com");
+    Subdomain goodDomain =
+        Subdomain.create(
+            "testThreatMatchToSqlGood.com",
+            "someDomain",
+            "someRegistrar",
+            "fake@someRegistrar.com");
+
+    // Establish a mock HttpResponse that returns a JSON response based on the request.
+    CloseableHttpClient mockHttpClient =
+        mock(CloseableHttpClient.class, withSettings().serializable());
+    when(mockHttpClient.execute(any(HttpPost.class))).thenAnswer(new HttpResponder());
+
+    EvaluateSafeBrowsingFn evalFn =
+        new EvaluateSafeBrowsingFn(
+            StaticValueProvider.of("apikey"),
+            new Retrier(new FakeSleeper(new FakeClock()), 3),
+            (Serializable & Supplier) () -> mockHttpClient);
+
+    // Apply input and evaluation transforms
+    PCollection<Subdomain> input = testPipeline.apply(Create.of(badDomain, goodDomain));
+    spec11Pipeline.evaluateUrlHealth(input, evalFn, StaticValueProvider.of("2020-06-10"));
+    testPipeline.run();
+
+    // Verify that the expected threat created from the bad Subdomain and the persisted
+    // Spec11TThreatMatch are equal.
+    Spec11ThreatMatch expected =
+        new Spec11ThreatMatch()
+            .asBuilder()
+            .setThreatTypes(ImmutableSet.of(ThreatType.MALWARE))
+            .setCheckDate(LocalDate.parse("2020-06-10", ISODateTimeFormat.date()))
+            .setDomainName(badDomain.domainName())
+            .setDomainRepoId(badDomain.domainRepoId())
+            .setRegistrarId(badDomain.registrarId())
+            .build();
+
+    verify(mockJpaTm).transact(any(Runnable.class));
+    verify(mockJpaTm).saveNew(expected);
+    verifyNoMoreInteractions(mockJpaTm);
+  }
+
  /**
   * A serializable {@link Answer} that returns a mock HTTP response based on the HTTP request's
   * content.
@@ -21,7 +21,7 @@ import static google.registry.bigquery.BigqueryUtils.toBigqueryTimestampString;
 import static google.registry.bigquery.BigqueryUtils.toJobReferenceString;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.api.services.bigquery.model.JobReference;
 import java.util.concurrent.TimeUnit;
@@ -21,7 +21,7 @@ import static google.registry.testing.DatastoreHelper.createTlds;
 import static google.registry.testing.DatastoreHelper.persistResource;
 import static google.registry.testing.TaskQueueHelper.assertNoTasksEnqueued;
 import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.appengine.api.taskqueue.dev.QueueStateInfo.TaskStateInfo;
 import com.google.appengine.tools.development.testing.LocalTaskQueueTestConfig;
@@ -20,7 +20,7 @@ import static google.registry.testing.DatastoreHelper.persistActiveDomain;
 import static google.registry.testing.DatastoreHelper.persistActiveSubordinateHost;
 import static google.registry.testing.TaskQueueHelper.assertDnsTasksEnqueued;
 import static google.registry.testing.TaskQueueHelper.assertNoDnsTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;

@@ -18,7 +18,7 @@ import static com.google.common.truth.Truth.assertThat;
 import static google.registry.testing.DatastoreHelper.createTld;
 import static google.registry.testing.TaskQueueHelper.assertNoTasksEnqueued;
 import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import google.registry.testing.AppEngineExtension;
 import google.registry.testing.FakeClock;
@@ -19,7 +19,7 @@ import static google.registry.testing.DatastoreHelper.createTld;
 import static google.registry.testing.DatastoreHelper.persistActiveDomain;
 import static google.registry.testing.DatastoreHelper.persistActiveSubordinateHost;
 import static google.registry.testing.DatastoreHelper.persistResource;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.Mockito.doThrow;
 import static org.mockito.Mockito.mock;
@@ -19,7 +19,7 @@ import static google.registry.testing.DatastoreHelper.createTld;
 import static google.registry.testing.DatastoreHelper.persistActiveDomain;
 import static google.registry.testing.DatastoreHelper.persistActiveHost;
 import static google.registry.testing.DatastoreHelper.persistActiveSubordinateHost;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.verifyNoMoreInteractions;
@@ -15,7 +15,7 @@
 package google.registry.dns.writer;

 import static com.google.common.truth.Truth.assertThat;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import org.junit.jupiter.api.Test;

@@ -16,7 +16,7 @@ package google.registry.dns.writer.dnsupdate;

 import static com.google.common.io.BaseEncoding.base16;
 import static com.google.common.truth.Truth.assertThat;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
@@ -26,7 +26,7 @@ import static google.registry.testing.DatastoreHelper.persistActiveSubordinateHo
 import static google.registry.testing.DatastoreHelper.persistDeletedDomain;
 import static google.registry.testing.DatastoreHelper.persistDeletedHost;
 import static google.registry.testing.DatastoreHelper.persistResource;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.verifyNoInteractions;
@@ -23,7 +23,7 @@ import static google.registry.testing.TestLogHandlerUtils.assertLogMessage;
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static java.util.logging.Level.INFO;
 import static java.util.logging.Level.SEVERE;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;

@@ -19,7 +19,7 @@ import static google.registry.export.CheckBackupAction.CHECK_BACKUP_KINDS_TO_LOA
 import static google.registry.export.CheckBackupAction.CHECK_BACKUP_NAME_PARAM;
 import static google.registry.testing.TaskQueueHelper.assertNoTasksEnqueued;
 import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.anyString;
 import static org.mockito.Mockito.when;

@@ -24,7 +24,7 @@ import static google.registry.testing.DatastoreHelper.persistDeletedDomain;
 import static google.registry.testing.DatastoreHelper.persistResource;
 import static google.registry.testing.GcsTestingUtils.readGcsFile;
 import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.eq;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
@@ -25,7 +25,7 @@ import static google.registry.testing.DatastoreHelper.persistResource;
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
 import static javax.servlet.http.HttpServletResponse.SC_OK;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.anyString;
 import static org.mockito.ArgumentMatchers.eq;
@@ -23,7 +23,7 @@ import static google.registry.testing.DatastoreHelper.persistResource;
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
 import static javax.servlet.http.HttpServletResponse.SC_OK;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.anyString;
 import static org.mockito.Mockito.mock;
@@ -23,7 +23,7 @@ import static google.registry.export.UpdateSnapshotViewAction.UPDATE_SNAPSHOT_TA
 import static google.registry.export.UpdateSnapshotViewAction.UPDATE_SNAPSHOT_VIEWNAME_PARAM;
 import static google.registry.export.UpdateSnapshotViewAction.createViewUpdateTask;
 import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.anyString;
 import static org.mockito.ArgumentMatchers.eq;
@@ -26,7 +26,7 @@ import static google.registry.export.UploadDatastoreBackupAction.UPLOAD_BACKUP_K
 import static google.registry.export.UploadDatastoreBackupAction.enqueueUploadBackupTask;
 import static google.registry.export.UploadDatastoreBackupAction.getBackupInfoFileForKind;
 import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.eq;
 import static org.mockito.Mockito.mock;
@@ -15,7 +15,7 @@
 package google.registry.export.datastore;

 import static com.google.common.truth.Truth.assertThat;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.api.client.json.JsonFactory;
 import com.google.api.client.json.jackson2.JacksonFactory;
@@ -121,15 +121,10 @@ class EppXmlSanitizerTest {
  void testSanitize_utf16_encodingPreserved() {
    // Test data should specify an endian-specific UTF-16 scheme for easy assertion. If 'UTF-16' is
    // used, the XMLEventReader in sanitizer may resolve it to an endian-specific one.
-    String inputXml = "<?xml version=\"1.0\" encoding=\"UTF-16LE\"?><p>\u03bc</p>\n";
+    String inputXml =
+        "<?xml version=\"1.0\" encoding=\"UTF-16LE\" standalone=\"no\"?>" + "<p>\u03bc</p>\n";
    String sanitizedXml = sanitizeEppXml(inputXml.getBytes(UTF_16LE));

-    // As of Java 9, standalone is always written to the XML header and is defaulted to "no" if not
-    // otherwise specified. We don't care about that for this specific test, and since we want to be
-    // compatible with both Java 8 and Java 11 (for now), we strip the standalone="no" attribute if
-    // it is present prior to doing the string comparison.
-    // TODO(java11): Remove this stripping and do a full assertion once we no longer care about
-    //               maintaining Java 8 build compatibility.
-    assertThat(sanitizedXml.replace(" standalone=\"no\"", "")).isEqualTo(inputXml);
+    assertThat(sanitizedXml).isEqualTo(inputXml);
  }
 }
@@ -16,7 +16,7 @@ package google.registry.flows;

 import static com.google.common.truth.Truth.assertThat;
 import static google.registry.testing.EppExceptionSubject.assertAboutEppExceptions;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
@@ -16,15 +16,12 @@ package google.registry.flows;

 import static com.google.common.base.MoreObjects.firstNonNull;
 import static com.google.common.base.Preconditions.checkNotNull;
-import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.collect.Sets.difference;
 import static com.google.common.truth.Truth.assertThat;
 import static com.google.common.truth.Truth.assertWithMessage;
 import static google.registry.model.eppcommon.EppXmlTransformer.marshal;
 import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
-import static google.registry.testing.DatastoreHelper.POLL_MESSAGE_ID_STRIPPER;
-import static google.registry.testing.DatastoreHelper.getPollMessages;
 import static google.registry.testing.DatastoreHelper.stripBillingEventId;
 import static google.registry.xml.XmlTestUtils.assertXmlEquals;
 import static java.nio.charset.StandardCharsets.UTF_8;
@@ -34,7 +31,6 @@ import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Maps;
 import com.google.common.collect.ObjectArrays;
-import com.google.common.collect.Streams;
 import google.registry.config.RegistryConfig.ConfigModule.TmchCaMode;
 import google.registry.flows.EppTestComponent.FakesAndMocksModule;
 import google.registry.flows.picker.FlowPicker;
@@ -44,7 +40,6 @@ import google.registry.model.eppcommon.ProtocolDefinition;
 import google.registry.model.eppinput.EppInput;
 import google.registry.model.eppoutput.EppOutput;
 import google.registry.model.ofy.Ofy;
-import google.registry.model.poll.PollMessage;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.model.tmch.ClaimsListShard.ClaimsListSingleton;
 import google.registry.monitoring.whitebox.EppMetric;
@@ -185,6 +180,7 @@ public abstract class FlowTestCase<F extends Flow> {
      builder.put(
          GracePeriod.create(
              entry.getKey().getType(),
+              entry.getKey().getDomainRepoId(),
              entry.getKey().getExpirationTime(),
              entry.getKey().getClientId(),
              null),
@@ -213,24 +209,6 @@ public abstract class FlowTestCase<F extends Flow> {
        .isEqualTo(canonicalizeGracePeriods(expected));
  }

-  protected void assertPollMessages(String clientId, PollMessage... expected) {
-    assertPollMessagesHelper(getPollMessages(clientId), expected);
-  }
-
-  protected void assertPollMessages(PollMessage... expected) {
-    assertPollMessagesHelper(getPollMessages(), expected);
-  }
-
-  /** Assert that the list matches all the poll messages in the fake Datastore. */
-  private void assertPollMessagesHelper(
-      Iterable<PollMessage> pollMessages, PollMessage... expected) {
-    // Ordering is irrelevant but duplicates should be considered independently.
-    assertThat(
-            Streams.stream(pollMessages).map(POLL_MESSAGE_ID_STRIPPER).collect(toImmutableList()))
-        .containsExactlyElementsIn(
-            Arrays.stream(expected).map(POLL_MESSAGE_ID_STRIPPER).collect(toImmutableList()));
-  }
-
  private EppOutput runFlowInternal(CommitMode commitMode, UserPrivileges userPrivileges)
      throws Exception {
    eppMetricBuilder = EppMetric.builderForRequest(clock);
@@ -22,7 +22,7 @@ import static google.registry.model.tmch.ClaimsListShardTest.createTestClaimsLis
 import static google.registry.testing.EppExceptionSubject.assertAboutEppExceptions;
 import static google.registry.testing.LogsSubject.assertAboutLogs;
 import static google.registry.testing.TaskQueueHelper.assertTasksEnqueued;
-import static org.junit.Assert.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertThrows;

 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Lai Jiang	ab4cecba22	Temporarily disable spec 11 pipeline deployment in GCB (#755 ) The current setup causes the GCB job to fail validation and not run because it uses backticks in the configuration yaml, which is not allowed -- there is no shell to perform backtick substitution. See the error message here: https://spinnaker.endpoints.domain-registry-dev.cloud.goog/gate/pipelines/01EF5GRMD625613H6Z033DBD3Z In the future please make sure to test the GCB pipeline as instructed in the comments at the beginning of each file before committing. I tried to work around it by downloading the nomulus tool jar file instead (running the nomulus-tool docker image inside a docker image is not advisable). However the "nomulus deploy_spec11_pipeline" command still fails. I'm not sure why. Has the command itself been tested locally? The error message is shown below: ``` Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions Step #2: WARNING: --region not set; will default to us-central1. Future releases of Beam will require the user to set the region explicitly. https://cloud.google.com/compute/docs/regions-zones/regions-zones Step #2: Aug 09, 2020 3:11:46 AM org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory tryCreateDefaultBucket Step #2: INFO: No tempLocation specified, attempting to use default bucket: dataflow-staging-us-central1-937378958468 Step #2: Aug 09, 2020 3:11:47 AM org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler handleResponse Step #2: WARNING: Request failed with code 409, performed 0 retries due to IOExceptions, performed 0 retries due to unsuccessful status codes, HTTP framework says request can be retried, (caller responsible for retrying): https://www.googleapis.com/storage/v1/b?predefinedAcl=projectPrivate&predefinedDefaultObjectAcl=projectPrivate&project=domain-registry-alpha. Step #2: Exception in thread "main" Step #2: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) Step #2: at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) Step #2: Step #2: at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) Step #2: at org.apache.beam.sdk.Pipeline.create(Pipeline.java:147) Step #2: Step #2: at google.registry.beam.spec11.Spec11Pipeline.deploy(Spec11Pipeline.java:157) Step #2: at google.registry.tools.DeploySpec11PipelineCommand.run(DeploySpec11PipelineCommand.java:80) Step #2: at google.registry.tools.RegistryCli.runCommand(RegistryCli.java:257) Step #2: at google.registry.tools.RegistryCli.run(RegistryCli.java:182) Step #2: at google.registry.tools.RegistryTool.main(RegistryTool.java:129) Step #2: Caused by: java.lang.reflect.InvocationTargetException Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) Step #2: Step #2: at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) Step #2: at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Step #2: at java.base/java.lang.reflect.Method.invoke(Method.java:566) Step #2: at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) Step #2: ... 8 more Step #2: Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@5cb0d902, only URLClassLoaders are supported. Step #2: at org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:58) Step #2: Step #2: at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:285) Step #2: Step #2: ... 13 more ``` Lastly the "--project" flag refers to the KMS project. While I'm not sure which project is that, I don't think we can use the PROJECT_ID variable as this is a GCB-substituted variable which refers to the project that the GCB job runs in, which in our cases means domain-registry-dev. We shouldn't use that project for KMS. I've changed it to the same project as the one we are deploying to, but please note that we have a separate project ${project_id}-keys that is used for all KMS purposes. This is specified in the config file so if that's what you meant to use, there is no need to specify it in the command line. Actually if you meant use the project to be deployed to for KMS, it also shouldn't be necessary to specify it separately as this information is already known when you specified "nomulus -e ENV". https://team.git.corp.google.com/domain-registry-eng/nomulus-internal/+/refs/heads/master/core/src/main/java/google/registry/config/files/nomulus-config-production.yaml#168 Can you add more description on what the KMS project is supposed to be? I don't think we specify a project for KMS purpose in any other commands. Given that there are several unresolved issues, I've commented out my proposed solution so that deployment can proceed.	2020-08-09 22:41:31 -04:00
Lai Jiang	00941ddc3d	Disable tests in RC builds (#752 ) For reasons unclear at the moment the tests are not passing. Disabling them for now so that release candidates can be built. We have CI runs after each merge so we should be pretty confident if the build is broken or not.	2020-08-07 17:51:34 -04:00
Shicong Huang	7b2b49dd53	Fix vkey construction in PollMessage and BillingEvent (#747 ) * Fix vkey construction in PollMessage and BillingEvent * Remove null check	2020-08-07 15:38:53 -04:00
Lai Jiang	bc8df8f34e	Exclude a test that fails in GCB build using Java 11 (#748 )	2020-08-07 10:53:46 -04:00
Ben McIlwain	74b4424509	Improve and consolidate testing of poll messages/billing events (#745 ) * Improve and consolidate testing of poll messages/billing events	2020-08-06 21:13:27 -04:00
Shicong Huang	486bf32353	Add SQL schema for GracePeriod (#709 ) * Add SQL schema for GracePeriod * Remove the join table * Add a domainRepoId in GracePeriod * Move the clone logic to GracePeriod * Rebase on HEAD	2020-08-06 10:26:19 -04:00
Ben McIlwain	b2a78b5d68	Allow allocation token discounts on premiums and for multiple years (#744 ) * Allow allocation token discounts on premiums and for multiple years * Add domain check flow tests * Address code review comments * Update schema file	2020-08-05 17:54:47 -04:00
Lai Jiang	95f4ae0e3a	Use nodesource to install node (#742 ) The node installed by nvm gives errors when running "npm install". Also installs Python as it is need. Presumbly the system provided npm version has python as a dependency so it was installed when npm was installed.	2020-08-05 14:56:40 -04:00
Lai Jiang	915405b735	Remove extra space after bashslash (#743 )	2020-08-04 20:00:46 -04:00
sarahcaseybot	d7fb6097ba	Add retrier to JpaTransactionManager (#693 ) * Add retrier to retry exceptions we've seen so far * Check for nested exceptions * fix formatting * check for nested transactions * add JDBCConnectionException to isFailedQueryRetriable * Add retrier to methods with suppliers	2020-08-04 12:22:08 -04:00
Michael Muller	d224d96924	Improve the uniform_rapid_suspension command (#739 ) - Reuse DS record format processing from the create/update domain commands (BIND format, commonly used in URS requests) - Remove the CLIENT_HOLD status from domains that have it (this blocks us from serving the new nameservers and DS record)	2020-08-04 11:06:02 -04:00
Lai Jiang	6d110c77ac	Use the latest version of node in the builder image (#741 ) The default node version from the base image (Ubuntu 18.04) is too older and karma is not happy about it.	2020-08-03 17:40:50 -04:00
Legina Chen	a79940e822	Persist ThreatMatches into Spec11ThreatMatch (#723 ) * Replace jpaTm with a JpaSupplierFactory * Style * Style * Pipeline takes in a SerializableSupplier instead * Change the ordering of imports * Test a good domain in addition to a bad one * Rename and check good domain for Transact Answer * Use standard Mockito verify * Verify transact call and no more interactions * Remove Answer comment * Naming chsnges * Deploy Spec 11 pipeline correctly * Fix formatting of deploy file * Use a file to persist state across Cloud Build steps Co-authored-by: Gus Brodman <gbrodman@google.com>	2020-08-03 14:40:00 -07:00
gbrodman	d8ec6294c3	Refactor DomainBase into DomainContent and create DomainHistory (#668 ) * Refactor DomainBase into DomainContent and create DomainHistory This is similar to #587 and #634, but for domains. One caveat is that we refactor some of the Domain* instance methods to be static so that they can be called either on DomainBase or DomainContent, returning the appropriate type each time. Note that we set DomainHistory to use the same revision ID sequence as HostHistory and ContactHistory. In addition, we refactor the tests to the History objects a bit to reduce duplicate code and because we cannot guarantee yet that the SQL-stored VKeys are symmetrical -- the ofy keys are not persisted at the moment. In addition, rename the DomainHost table to the default Domain_nsHosts so that it automatically creates two separate nsHosts tables for us -- one foreign-keyed on the domain repo ID, and one foreign-keyed on the history revision ID * Use access hackery to allow manual names for nsHosts tables * Clean up post merge artifacts * Add unused setters that Hibernate requires * Fix the tests and semantic merge conflicts * Change ns_hosts to ns_host everywhere * Rename ns_host to host_repo_id * V42 -> V44	2020-08-03 17:36:12 -04:00
Lai Jiang	9304e2f421	Enable Java 11 features (#737 ) * Enable Java 11 features As of this commit Java 11 must be used to build. The generated bytecode is still at Java 8 due to App Engine task queue limit. Also fixed a bug where the included google-java-format jar file is not used, requiring the user to install it separately. See: https://cloud.google.com/appengine/docs/standard/java/taskqueue/push	2020-08-03 16:08:29 -04:00
Lai Jiang	5f2be914a1	Use Java 11 in GCB to build release candidates (#736 )	2020-08-03 13:13:08 -04:00
sarahcaseybot	4ad7f9734d	Add IntervalDescriptor and change DurationConverter to use Interval datatype (#653 ) * Add use of interval data type * Add support for Millis * Use Java-object type * Change column type for relock_duration * add years and months * Add tests for hours, minutes, and seconds * Add javadoc describing how joda duration is stored * Add test for lots of days	2020-08-03 12:42:34 -04:00
Lai Jiang	90d080d42f	Delete screenshots folder in fragile test first (#735 ) It is not clear to me why this folder is deleted in standardTest first, presumably this is done when the webdriver tests were part of standardTest? Also not clear to me is why screenshots are only deleted in presubmits but not locally. At least on my workstation standardTest runs after fragileTest, just like in kokoro, so it should have deleted the screenshots folder, but it didn't. Regardless, the correct thing to do is to delete this folder first in fragileTest so that it doesn't interfere with the results.	2020-08-03 11:46:34 -04:00
Ben McIlwain	e95ce30fa6	Upgrade prober & proxy packages from JUnit 4 to JUnit 5 (#734 ) * Upgrade prober & proxy packages from JUnit 4 to JUnit 5	2020-08-01 11:34:58 -04:00
Lai Jiang	090c233592	Some more changes to make it possible to use Java 11 to compile (#732 ) Add the class paths of the source files generated by annotation processors to the javadoc task's class path so that it doesn't complain about missing Dagger classes. Also remove empty <p> tags in all generated source files, because jaxb genrerates files in multiple locations. Lastly, for unkown reasons when the source level is set to > 8, the core subproject throws a warning about a Gradle internal annotator processor that only supports up to Java 8 and cause the Java compliation to fail because we set -Werror on all java compliation tasks. I don't think there is a strong reason that we set -Werror anyway, so this commit removes it.	2020-07-31 17:21:28 -04:00
Ben McIlwain	16a31e460c	Get rid of all remaining JUnit 4 usages except in prober & proxy (#731 ) * Get rid of all remaining JUnit 4 usages except in prober & proxy subprojects Caveat: Test suites aren't yet implemented in JUnit 5 so we still use the ones from JUnit 5 in the core subproject. * Fix some build errors	2020-07-30 20:29:00 -04:00
Lai Jiang	a02b67caf5	Migrate the documentation package to Java 11 (#729 ) * Migrate the documentation package to Java 11 The old Doclet API is deprected and removed in Java 12. This commit changes the documentation package to use the new recommended API. However it is not a drop-in replacement and there are non-idiomatic usages all over the place. I think it is eaiser to keep the current code logic and kind of shoehorn in the new API than starting afresh as the return on investment of a do-over is not great. Also note that the docs package is disabled as of this commit because we are still using Java 8 to compile which lacks the new API. Once we switch our toolchains to Java 11 (but still compiling Java 8 bytecode) we can re-enable this package. TESTED=ran `./gradlew :docs:test` locally with the documentation package enabled.	2020-07-30 17:12:33 -04:00
Lai Jiang	bf20a8ef96	Remove empty <p> tag from generated javadoc (#730 )	2020-07-30 16:31:25 -04:00
Lai Jiang	8750c07fef	Clean up generated sources configuration (#724 ) 1. The Gradle apt plugin is no longer needed to process annotations. 2. Without the apt plugin, Gralde puts the source files generated by annotation processors in build/generated/sources/annotationProcessor. 3. Change the location of custom generated files to be consistent. 4. Fix a javadoc formatting error.	2020-07-30 15:43:42 -04:00
Weimin Yu	7821de67f8	Run InitSqlPipeline (#727 ) * Run InitSqlPipeline Added the main() method to InitSqlPipeline. Added a Gradle task to run InitSqlPipeline from command line. This task is meant for testing and experiments. Corrected the file name prefix of Datastore export files. Should be 'output-', defined as 'input-'.	2020-07-30 14:28:53 -04:00
Shicong Huang	1022817384	Fix broken RegistryCli (#728 )	2020-07-30 14:10:09 -04:00