Integrate transaction persistence into JpaTM (#717 )

* Integrate transaction persistence into JpaTM Store the serialized transaction whenever we commit from the JPA transaction manager. This change also adds: - The Transaction table. - The TransactionEntity which is stored in it. - Changes to the test infrastructure to register the TransactionEntity for tests where we don't load the nomulus schema. - A new configuration variable to allow us to turn the transaction persistence functionality on and off (default is "off"). * Changes for review. * Incremented sequence number of flyway file
Upgrade App Engine and webserver tests from JUnit 4 to 5 (#720 )
2026-06-09 16:33:02 +00:00 · 2020-07-28 19:23:44 -04:00 · 2020-07-28 14:18:16 -04:00 · 2020-07-27 21:04:52 -04:00 · 2020-07-27 18:33:16 -04:00 · 2020-07-27 18:16:24 -04:00
720 changed files with 16612 additions and 13355 deletions
@@ -79,6 +79,10 @@ nomulus.iml
 nomulus.ipr
 nomulus.iws

+# Auto-generated java classes by Intellij
+*/src/main/generated/
+*/src/test/generated_tests/
+
 # VScode
 .vscode

@@ -25,7 +25,7 @@ buildscript {

  dependencies {
    classpath 'com.google.cloud.tools:appengine-gradle-plugin:2.0.1'
-    classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.6.1"
+    classpath 'net.ltgt.gradle:gradle-errorprone-plugin:0.6.1'
    classpath 'org.sonatype.aether:aether-api:1.13.1'
    classpath 'org.sonatype.aether:aether-impl:1.13.1'
  }
@@ -49,6 +49,7 @@ plugins {
  id 'com.diffplug.gradle.spotless' version '3.25.0'

  id 'jacoco'
+  id 'com.dorongold.task-tree' version '1.5'
 }

 wrapper {
@@ -443,6 +444,7 @@ task javaIncrementalFormatDryRun {
    println("${invokeJavaDiffFormatScript("show")}")
  }
 }
+tasks.build.dependsOn(tasks.javaIncrementalFormatCheck)

 // Checks if modified lines in Java source files need reformatting.
 // Note that this task processes modified Java files in the entire repository.
@@ -452,8 +454,6 @@ task javaIncrementalFormatApply {
  }
 }

-tasks.build.dependsOn(tasks.javaIncrementalFormatCheck)
-
 task javadoc(type: Javadoc) {
  source javadocSource
  classpath = files(javadocClasspath)
@@ -468,4 +468,16 @@ task javadoc(type: Javadoc) {

 tasks.build.dependsOn(tasks.javadoc)

+// Task for doing development on core Nomulus.
+// This fixes code formatting automatically as necessary, builds and tests the
+// core Nomulus codebase, and runs all presubmits.
+task coreDev {
+  dependsOn 'javaIncrementalFormatApply'
+  dependsOn 'javadoc'
+  dependsOn 'checkDependenciesDotGradle'
+  dependsOn 'checkLicense'
+  dependsOn ':core:check'
+  dependsOn 'assemble'
+}
+
 javadocDependentTasks.each { tasks.javadoc.dependsOn(it) }
@@ -61,12 +61,12 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.hamcrest:hamcrest-core:1.3
 org.json:json:20160212
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
 org.opentest4j:opentest4j:1.2.0
@@ -61,12 +61,12 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.hamcrest:hamcrest-core:1.3
 org.json:json:20160212
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
 org.opentest4j:opentest4j:1.2.0
@@ -61,12 +61,12 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.hamcrest:hamcrest-core:1.3
 org.json:json:20160212
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
 org.opentest4j:opentest4j:1.2.0
@@ -61,12 +61,12 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.hamcrest:hamcrest-core:1.3
 org.json:json:20160212
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.mockito:mockito-core:3.3.3
 org.objenesis:objenesis:2.6
 org.opentest4j:opentest4j:1.2.0
@@ -20,10 +20,10 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
 org.hamcrest:hamcrest-core:1.3
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -20,10 +20,10 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
 org.hamcrest:hamcrest-core:1.3
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -21,10 +21,10 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
 org.hamcrest:hamcrest-core:1.3
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -21,10 +21,10 @@ org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
 org.eclipse.jgit:org.eclipse.jgit:4.4.1.201607150455-r
 org.hamcrest:hamcrest-core:1.3
-org.junit.jupiter:junit-jupiter-api:5.6.1
-org.junit.jupiter:junit-jupiter-engine:5.6.1
-org.junit.platform:junit-platform-commons:1.6.1
-org.junit.platform:junit-platform-engine:1.6.1
-org.junit.vintage:junit-vintage-engine:5.6.1
-org.junit:junit-bom:5.6.1
+org.junit.jupiter:junit-jupiter-api:5.6.2
+org.junit.jupiter:junit-jupiter-engine:5.6.2
+org.junit.platform:junit-platform-commons:1.6.2
+org.junit.platform:junit-platform-engine:1.6.2
+org.junit.vintage:junit-vintage-engine:5.6.2
+org.junit:junit-bom:5.6.2
 org.opentest4j:opentest4j:1.2.0
@@ -16,6 +16,7 @@ package google.registry.testing.truth;

 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.truth.Truth.assertAbout;
+import static com.google.common.truth.Truth.assertWithMessage;
 import static java.nio.charset.StandardCharsets.UTF_8;

 import com.github.difflib.DiffUtils;
@@ -31,6 +32,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.io.Resources;
 import com.google.common.truth.Fact;
 import com.google.common.truth.FailureMetadata;
+import com.google.common.truth.SimpleSubjectBuilder;
 import com.google.common.truth.Subject;
 import java.io.IOException;
 import java.net.URL;
@@ -68,6 +70,15 @@ public class TextDiffSubject extends Subject {
    this.actual = ImmutableList.copyOf(actual);
  }

+  protected TextDiffSubject(FailureMetadata metadata, URL actual) {
+    super(metadata, actual);
+    try {
+      this.actual = ImmutableList.copyOf(Resources.asCharSource(actual, UTF_8).readLines());
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
  public TextDiffSubject withDiffFormat(DiffFormat format) {
    this.diffFormat = format;
    return this;
@@ -100,6 +111,11 @@ public class TextDiffSubject extends Subject {
    return assertThat(Resources.asCharSource(resourceUrl, UTF_8).readLines());
  }

+  public static SimpleSubjectBuilder<TextDiffSubject, URL> assertWithMessageAboutUrlSource(
+      String format, Object... params) {
+    return assertWithMessage(format, params).about(urlFactory());
+  }
+
  private static final Subject.Factory<TextDiffSubject, ImmutableList<String>>
      TEXT_DIFF_SUBJECT_TEXT_FACTORY = TextDiffSubject::new;

@@ -107,6 +123,13 @@ public class TextDiffSubject extends Subject {
    return TEXT_DIFF_SUBJECT_TEXT_FACTORY;
  }

+  private static final Subject.Factory<TextDiffSubject, URL> TEXT_DIFF_SUBJECT_URL_FACTORY =
+      TextDiffSubject::new;
+
+  public static Subject.Factory<TextDiffSubject, URL> urlFactory() {
+    return TEXT_DIFF_SUBJECT_URL_FACTORY;
+  }
+
  static String generateUnifiedDiff(
      ImmutableList<String> expectedContent, ImmutableList<String> actualContent) {
    Patch<String> diff;
@@ -19,6 +19,7 @@ import argparse
 import attr
 import io
 import os
+import shutil
 import subprocess
 import sys
 from typing import List, Union
@@ -49,15 +50,30 @@ PROPERTIES_HEADER = """\
 # This file defines properties used by the gradle build.  It must be kept in
 # sync with config/nom_build.py.
 #
-# To regenerate, run config/nom_build.py --generate-gradle-properties
+# To regenerate, run ./nom_build --generate-gradle-properties
 #
 # To view property descriptions (which are command line flags for
-# nom_build), run config/nom_build.py --help.
+# nom_build), run ./nom_build --help.
 #
 # DO NOT EDIT THIS FILE BY HAND
 org.gradle.jvmargs=-Xmx1024m
 """

+# Help text to be displayed (in addition to the synopsis and flag help, which
+# are displayed automatically).
+HELP_TEXT = """\
+A wrapper around the gradle build that provides the following features:
+
+-   Converts properties into flags to guard against property name spelling errors
+    and to provide help descriptions for all properties.
+-   Provides pseudo-commands (with the ":nom:" prefix) that encapsulate common
+    actions that are difficult to implement in gradle.
+
+Pseudo-commands:
+    :nom:generate_golden_file - regenerates the golden file from the current
+        set of flyway files.
+"""
+
 # Define all of our special gradle properties here.
 PROPERTIES = [
    Property('mavenUrl',
@@ -114,6 +130,11 @@ PROPERTIES = [
    Property('nomulus_version',
             'The version of nomulus to test against in a database '
             'integration test.'),
+    Property('dot_path',
+             'The path to "dot", part of the graphviz package that converts '
+             'a BEAM pipeline to image. Setting this property to empty string '
+             'will disable image generation.',
+             '/usr/bin/dot'),
 ]

 GRADLE_FLAGS = [
@@ -251,8 +272,42 @@ def get_root() -> str:
    return cur_dir


-def main(args):
-    parser = argparse.ArgumentParser('nom_build')
+class Abort(Exception):
+    """Raised to terminate the process with a non-zero error code.
+
+    Parameters are ignored.
+    """
+
+
+def do_pseudo_task(task: str) -> None:
+    root = get_root()
+    if task == ':nom:generate_golden_file':
+        if not subprocess.call([f'{root}/gradlew', ':db:test']):
+            print('\033[33mWARNING:\033[0m Golden schema appears to be '
+                  'up-to-date.  If you are making schema changes,  be sure to '
+                  'add a flyway file for them.')
+            return
+        print('\033[33mWARNING:\033[0m Ignore the above failure, it is '
+              'expected.')
+
+        # Copy the new schema into place.
+        shutil.copy(f'{root}/db/build/resources/test/testcontainer/'
+                    'mount/dump.txt',
+                    f'{root}/db/src/main/resources/sql/schema/'
+                    'nomulus.golden.sql')
+
+        if subprocess.call([f'{root}/gradlew', ':db:test']):
+            print('\033[31mERROR:\033[0m Golden file test failed after '
+                  'copying schema.  Please check your flyway files.')
+            raise Abort()
+    else:
+        print(f'\033[31mERROR:\033[0m Unknown task {task}')
+        raise Abort()
+
+
+def main(args) -> int:
+    parser = argparse.ArgumentParser('nom_build', description=HELP_TEXT,
+                                     formatter_class=argparse.RawTextHelpFormatter)
    for prop in PROPERTIES:
        parser.add_argument('--' + prop.name, default=prop.default,
                            help=prop.desc)
@@ -291,7 +346,7 @@ def main(args):
    if args.generate_gradle_properties:
        with open(f'{root}/gradle.properties', 'w') as dst:
            dst.write(gradle_properties)
-        return
+        return 0

    # Verify that the gradle properties file is what we expect it to be.
    with open(f'{root}/gradle.properties') as src:
@@ -316,12 +371,39 @@ def main(args):
            if flag.has_arg:
                gradle_command.append(arg_val)

+    # See if there are any special ":nom:" pseudo-tasks specified.
+    got_non_pseudo_tasks = False
+    for arg in args.non_flag_args[1:]:
+        if arg.startswith(':nom:'):
+            if got_non_pseudo_tasks:
+                # We can't currently deal with the situation of gradle tasks
+                # before pseudo-tasks.  This could be implemented by invoking
+                # gradle for only the set of gradle tasks before the pseudo
+                # task, but that's overkill for now.
+                print(f'\033[31mERROR:\033[0m Pseudo task ({arg}) must be '
+                      'specified prior to all actual gradle tasks.  Aborting.')
+                return 1
+            do_pseudo_task(arg)
+        else:
+            got_non_pseudo_tasks = True
+    non_flag_args = [
+        arg for arg in args.non_flag_args[1:] if not arg.startswith(':nom:')]
+
+    if not non_flag_args:
+        if not got_non_pseudo_tasks:
+            print('\033[33mWARNING:\033[0m No tasks specified.  Not '
+                  'doing anything')
+        return 0
+
    # Add the non-flag args (we exclude the first, which is the command name
    # itself) and run.
-    gradle_command.extend(args.non_flag_args[1:])
-    subprocess.call(gradle_command)
+    gradle_command.extend(non_flag_args)
+    return subprocess.call(gradle_command)


 if __name__ == '__main__':
-    main(sys.argv)
+    try:
+        sys.exit(main(sys.argv))
+    except Abort as ex:
+        sys.exit(1)

@@ -14,6 +14,7 @@

 import io
 import os
+import shutil
 import unittest
 from unittest import mock
 import nom_build
@@ -67,6 +68,7 @@ class MyTest(unittest.TestCase):
            mock.patch.object(nom_build, 'print', self.print_fake).start())

        self.call_mock = mock.patch.object(subprocess, 'call').start()
+        self.copy_mock = mock.patch.object(shutil, 'copy').start()

        self.file_contents = {
            # Prefil with the actual file contents.
@@ -92,17 +94,32 @@ class MyTest(unittest.TestCase):

    def test_no_args(self):
        nom_build.main(['nom_build'])
-        self.assertEqual(self.printed, [])
-        self.call_mock.assert_called_with([GRADLEW])
+        self.assertEqual(self.printed,
+                         ['\x1b[33mWARNING:\x1b[0m No tasks specified.  Not '
+                         'doing anything'])

    def test_property_calls(self):
-        nom_build.main(['nom_build', '--testFilter=foo'])
-        self.call_mock.assert_called_with([GRADLEW, '-P', 'testFilter=foo'])
+        nom_build.main(['nom_build', 'task-name', '--testFilter=foo'])
+        self.call_mock.assert_called_with([GRADLEW, '-P', 'testFilter=foo',
+                                          'task-name'])

    def test_gradle_flags(self):
-        nom_build.main(['nom_build', '-d', '-b', 'foo'])
+        nom_build.main(['nom_build', 'task-name', '-d', '-b', 'foo'])
        self.call_mock.assert_called_with([GRADLEW, '--build-file', 'foo',
-                                          '--debug'])
+                                          '--debug', 'task-name'])
+
+    def test_generate_golden_file(self):
+        self.call_mock.side_effect = [1, 0]
+        nom_build.main(['nom_build', ':nom:generate_golden_file'])
+        self.call_mock.assert_has_calls([
+            mock.call([GRADLEW, ':db:test']),
+            mock.call([GRADLEW, ':db:test'])
+        ])
+
+    def test_generate_golden_file_nofail(self):
+        self.call_mock.return_value = 0
+        nom_build.main(['nom_build', ':nom:generate_golden_file'])
+        self.call_mock.assert_has_calls([mock.call([GRADLEW, ':db:test'])])

 unittest.main()

@@ -22,7 +22,7 @@ import sys
 import re

 # We should never analyze any generated files
-UNIVERSALLY_SKIPPED_PATTERNS = {"/build/", "cloudbuild-caches", "/out/"}
+UNIVERSALLY_SKIPPED_PATTERNS = {"/build/", "cloudbuild-caches", "/out/", ".git/"}
 # We can't rely on CI to have the Enum package installed so we do this instead.
 FORBIDDEN = 1
 REQUIRED = 2
@@ -77,9 +77,9 @@ PRESUBMITS = {
    PresubmitCheck(
        r".*Copyright 20\d{2} The Nomulus Authors\. All Rights Reserved\.",
        ("java", "js", "soy", "sql", "py", "sh", "gradle"), {
-            ".git", "/build/", "/generated/", "node_modules/",
-            "JUnitBackports.java", "registrar_bin.", "registrar_dbg.",
-            "google-java-format-diff.py",
+            ".git", "/build/", "/generated/", "/generated_tests/",
+            "node_modules/", "JUnitBackports.java", "registrar_bin.",
+            "registrar_dbg.", "google-java-format-diff.py",
            "nomulus.golden.sql", "soyutils_usegoog.js"
        }, REQUIRED):
        "File did not include the license header.",
@@ -93,20 +93,20 @@ PRESUBMITS = {
    PresubmitCheck(
        r".*\bSystem\.(out|err)\.print", "java", {
            "StackdriverDashboardBuilder.java", "/tools/", "/example/",
-            "RegistryTestServerMain.java", "TestServerRule.java",
+            "RegistryTestServerMain.java", "TestServerExtension.java",
            "FlowDocumentationTool.java"
        }):
        "System.(out|err).println is only allowed in tools/ packages. Please "
        "use a logger instead.",

-    # ObjectifyService.register is restricted to main/ or AppEngineRule.
+    # ObjectifyService.register is restricted to main/ or AppEngineExtension.
    PresubmitCheck(
        r".*\bObjectifyService\.register", "java", {
            "/build/", "/generated/", "node_modules/", "src/main/",
-            "AppEngineRule.java"
+            "AppEngineExtension.java"
        }):
-      "ObjectifyService.register is not allowed in tests. Please use "
-      "AppengineRule.register instead.",
+      "ObjectifyService.register(...) is not allowed in tests. Please use "
+      "AppEngineExtension.register(...) instead.",

    # PostgreSQLContainer instantiation must specify docker tag
    PresubmitCheck(
@@ -238,6 +238,7 @@ dependencies {
  compile deps['jline:jline']
  compile deps['joda-time:joda-time']
  compile deps['org.apache.avro:avro']
+  testCompile deps['org.apache.beam:beam-runners-core-construction-java']
  testCompile deps['org.apache.beam:beam-runners-direct-java']
  compile deps['org.apache.beam:beam-runners-google-cloud-dataflow-java']
  compile deps['org.apache.beam:beam-sdks-java-core']
@@ -256,6 +257,7 @@ dependencies {
  compile deps['org.bouncycastle:bcpg-jdk15on']
  testCompile deps['org.bouncycastle:bcpkix-jdk15on']
  compile deps['org.bouncycastle:bcprov-jdk15on']
+  testCompile deps['com.fasterxml.jackson.core:jackson-databind']
  runtime deps['org.glassfish.jaxb:jaxb-runtime']
  compile deps['org.hibernate:hibernate-core']
  compile deps['org.joda:joda-money']
@@ -312,6 +314,8 @@ dependencies {
  testCompile deps['org.junit.jupiter:junit-jupiter-api']
  testCompile deps['org.junit.jupiter:junit-jupiter-engine']
  testCompile deps['org.junit.jupiter:junit-jupiter-migrationsupport']
+  testCompile deps['org.junit.jupiter:junit-jupiter-params']
+  testCompile deps['org.junit-pioneer:junit-pioneer']
  testCompile deps['org.junit.platform:junit-platform-runner']
  testCompile deps['org.junit.platform:junit-platform-suite-api']
  testCompile deps['org.junit.vintage:junit-vintage-engine']
@@ -966,6 +970,49 @@ task buildToolImage(dependsOn: nomulus, type: Exec) {
  commandLine 'docker', 'build', '-t', 'nomulus-tool', '.'
 }

+task generateInitSqlPipelineGraph(type: Test) {
+  include "**/InitSqlPipelineGraphTest.*"
+  testNameIncludePatterns = ["**createPipeline_compareGraph"]
+  ignoreFailures = true
+}
+
+task updateInitSqlPipelineGraph(type: Copy) {
+  def graphRelativePath = 'google/registry/beam/initsql/'
+  from ("${projectDir}/build/resources/test/${graphRelativePath}") {
+    include 'pipeline_curr.dot'
+    rename 'curr', 'golden'
+  }
+  into "src/test/resources/${graphRelativePath}"
+
+  dependsOn generateInitSqlPipelineGraph
+
+  doLast {
+    if (com.google.common.base.Strings.isNullOrEmpty(project.dot_path)) {
+      getLogger().info('Property dot_path is null. Not creating image for pipeline graph.')
+    }
+    def dotPath = project.dot_path
+    if (!new File(dotPath).exists()) {
+      throw new RuntimeException(
+          """\
+              ${dotPath} not found. Make sure graphviz is installed
+              and the dot_path property is set correctly."""
+              .stripIndent())
+    }
+    def goldenGraph = "src/test/resources/${graphRelativePath}/pipeline_golden.dot"
+    def goldenImage = "src/test/resources/${graphRelativePath}/pipeline_golden.png"
+    def cmd = "${dotPath} -Tpng -o \"${goldenImage}\" \"${goldenGraph}\""
+    try {
+      rootProject.ext.execInBash(cmd, projectDir)
+    } catch (Throwable throwable) {
+      throw new RuntimeException(
+          """\
+              Failed to generate golden image with command ${cmd}
+              Error: ${throwable.getMessage()}
+          """)
+    }
+  }
+}
+
 // Build the devtool jar.
 createUberJar(
    'devtool',
@@ -243,9 +243,11 @@ org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.jupiter:junit-jupiter-migrationsupport:5.6.2
+org.junit.jupiter:junit-jupiter-params:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
 org.junit.platform:junit-platform-launcher:1.6.2
@@ -241,9 +241,11 @@ org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.jupiter:junit-jupiter-migrationsupport:5.6.2
+org.junit.jupiter:junit-jupiter-params:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
 org.junit.platform:junit-platform-launcher:1.6.2
@@ -246,9 +246,11 @@ org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.jupiter:junit-jupiter-migrationsupport:5.6.2
+org.junit.jupiter:junit-jupiter-params:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
 org.junit.platform:junit-platform-launcher:1.6.2
@@ -246,9 +246,11 @@ org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
 org.junit.jupiter:junit-jupiter-migrationsupport:5.6.2
+org.junit.jupiter:junit-jupiter-params:5.6.2
 org.junit.platform:junit-platform-commons:1.6.2
 org.junit.platform:junit-platform-engine:1.6.2
 org.junit.platform:junit-platform-launcher:1.6.2
@@ -22,20 +22,35 @@ import java.lang.reflect.Method;
 import java.lang.reflect.Proxy;

 /**
- * Sets up a placeholder {@link Environment} on a non-AppEngine platform so that Datastore Entities
- * can be deserialized. See {@code DatastoreEntityExtension} in test source for more information.
+ * Sets up a fake {@link Environment} so that the following operations can be performed without the
+ * Datastore service:
+ *
+ * <ul>
+ *   <li>Create Objectify {@code Keys}.
+ *   <li>Instantiate Objectify objects.
+ *   <li>Convert Datastore {@code Entities} to their corresponding Objectify objects.
+ * </ul>
+ *
+ * <p>User has the option to specify their desired {@code appId} string, which forms part of an
+ * Objectify {@code Key} and is included in the equality check. This feature makes it easy to
+ * compare a migrated object in SQL with the original in Objectify.
+ *
+ * <p>Note that conversion from Objectify objects to Datastore {@code Entities} still requires the
+ * Datastore service.
 */
 public class AppEngineEnvironment implements Closeable {

-  private static final Environment PLACEHOLDER_ENV = createAppEngineEnvironment();
-
  private boolean isPlaceHolderNeeded;

  public AppEngineEnvironment() {
+    this("PlaceholderAppId");
+  }
+
+  public AppEngineEnvironment(String appId) {
    isPlaceHolderNeeded = ApiProxy.getCurrentEnvironment() == null;
    // isPlaceHolderNeeded may be true when we are invoked in a test with AppEngineRule.
    if (isPlaceHolderNeeded) {
-      ApiProxy.setEnvironmentForCurrentThread(PLACEHOLDER_ENV);
+      ApiProxy.setEnvironmentForCurrentThread(createAppEngineEnvironment(appId));
    }
  }

@@ -47,7 +62,7 @@ public class AppEngineEnvironment implements Closeable {
  }

  /** Returns a placeholder {@link Environment} that can return hardcoded AppId and Attributes. */
-  private static Environment createAppEngineEnvironment() {
+  private static Environment createAppEngineEnvironment(String appId) {
    return (Environment)
        Proxy.newProxyInstance(
            Environment.class.getClassLoader(),
@@ -55,7 +70,7 @@ public class AppEngineEnvironment implements Closeable {
            (Object proxy, Method method, Object[] args) -> {
              switch (method.getName()) {
                case "getAppId":
-                  return "PlaceholderAppId";
+                  return appId;
                case "getAttributes":
                  return ImmutableMap.<String, Object>of();
                default:
@@ -169,10 +169,12 @@ public final class AsyncTaskEnqueuer {
        lock.getRelockDuration().isPresent(),
        "Lock with ID %s not configured for relock",
        lock.getRevisionId());
+    String backendHostname = appEngineServiceUtils.getServiceHostname("backend");
    addTaskToQueueWithRetry(
        asyncActionsPushQueue,
        TaskOptions.Builder.withUrl(RelockDomainAction.PATH)
            .method(Method.POST)
+            .header("Host", backendHostname)
            .param(
                RelockDomainAction.OLD_UNLOCK_REVISION_ID_PARAM,
                String.valueOf(lock.getRevisionId()))
@@ -532,7 +532,7 @@ public class DeleteContactsAndHostsAction implements Runnable {
          resource.getClass().getSimpleName());
      return new AutoValue_DeleteContactsAndHostsAction_DeletionRequest.Builder()
          .setKey(resourceKey)
-          .setLastUpdateTime(resource.getUpdateAutoTimestamp().getTimestamp())
+          .setLastUpdateTime(resource.getUpdateTimestamp().getTimestamp())
          .setRequestingClientId(
              checkNotNull(
                  params.get(PARAM_REQUESTING_CLIENT_ID), "Requesting client id not specified"))
@@ -319,13 +319,13 @@ public class RefreshDnsOnHostRenameAction implements Runnable {
      HostResource host =
          checkNotNull(ofy().load().key(hostKey).now(), "Host to refresh doesn't exist");
      boolean isHostDeleted =
-          isDeleted(host, latestOf(now, host.getUpdateAutoTimestamp().getTimestamp()));
+          isDeleted(host, latestOf(now, host.getUpdateTimestamp().getTimestamp()));
      if (isHostDeleted) {
        logger.atInfo().log("Host %s is already deleted, not refreshing DNS.", hostKey);
      }
      return new AutoValue_RefreshDnsOnHostRenameAction_DnsRefreshRequest.Builder()
          .setHostKey(hostKey)
-          .setLastUpdateTime(host.getUpdateAutoTimestamp().getTimestamp())
+          .setLastUpdateTime(host.getUpdateTimestamp().getTimestamp())
          .setRequestedTime(
              DateTime.parse(
                  checkNotNull(params.get(PARAM_REQUESTED_TIME), "Requested time not specified")))
@@ -19,31 +19,26 @@ import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;

 import com.google.common.base.Splitter;
-import com.google.common.collect.ImmutableList;
-import dagger.Binds;
 import dagger.Component;
 import dagger.Lazy;
 import dagger.Module;
 import dagger.Provides;
-import google.registry.beam.initsql.BeamJpaModule.BindModule;
 import google.registry.config.CredentialModule;
 import google.registry.config.RegistryConfig.Config;
+import google.registry.config.RegistryConfig.ConfigModule;
 import google.registry.keyring.kms.KmsModule;
 import google.registry.persistence.PersistenceModule;
 import google.registry.persistence.PersistenceModule.JdbcJpaTm;
 import google.registry.persistence.PersistenceModule.SocketFactoryJpaTm;
 import google.registry.persistence.transaction.JpaTransactionManager;
-import google.registry.util.Clock;
-import google.registry.util.Sleeper;
-import google.registry.util.SystemClock;
-import google.registry.util.SystemSleeper;
+import google.registry.util.UtilsModule;
 import java.io.BufferedReader;
 import java.io.IOException;
 import java.io.InputStreamReader;
 import java.nio.channels.Channels;
 import java.nio.charset.StandardCharsets;
 import java.util.List;
-import javax.inject.Named;
+import javax.annotation.Nullable;
 import javax.inject.Singleton;
 import org.apache.beam.sdk.io.FileSystems;
 import org.apache.beam.sdk.io.fs.ResourceId;
@@ -53,38 +48,45 @@ import org.apache.beam.sdk.io.fs.ResourceId;
 *
 * <p>This module is intended for use in BEAM pipelines, and uses a BEAM utility to access GCS like
 * a regular file system.
- *
- * <p>Note that {@link google.registry.config.RegistryConfig.ConfigModule} cannot be used here,
- * since many bindings, especially KMS-related ones, are different.
 */
-@Module(includes = {BindModule.class})
-class BeamJpaModule {
+@Module
+public class BeamJpaModule {

  private static final String GCS_SCHEME = "gs://";

-  private final String credentialFilePath;
+  @Nullable private final String sqlAccessInfoFile;
+  @Nullable private final String cloudKmsProjectId;

  /**
   * Constructs a new instance of {@link BeamJpaModule}.
   *
-   * @param credentialFilePath the path to a Cloud SQL credential file. This must refer to either a
+   * <p>Note: it is an unfortunately necessary antipattern to check for the validity of
+   * sqlAccessInfoFile in {@link #provideCloudSqlAccessInfo} rather than in the constructor.
+   * Unfortunately, this is a restriction imposed upon us by Dagger. Specifically, because we use
+   * this in at least one 1 {@link google.registry.tools.RegistryTool} command(s), it must be
+   * instantiated in {@code google.registry.tools.RegistryToolComponent} for all possible commands;
+   * Dagger doesn't permit it to ever be null. For the vast majority of commands, it will never be
+   * used (so a null credential file path is fine in those cases).
+   *
+   * @param sqlAccessInfoFile the path to a Cloud SQL credential file. This must refer to either a
   *     real encrypted file on GCS as returned by {@link
   *     BackupPaths#getCloudSQLCredentialFilePatterns} or an unencrypted file on local filesystem
   *     with credentials to a test database.
   */
-  BeamJpaModule(String credentialFilePath) {
-    checkArgument(!isNullOrEmpty(credentialFilePath), "Null or empty credentialFilePath");
-    this.credentialFilePath = credentialFilePath;
+  public BeamJpaModule(@Nullable String sqlAccessInfoFile, @Nullable String cloudKmsProjectId) {
+    this.sqlAccessInfoFile = sqlAccessInfoFile;
+    this.cloudKmsProjectId = cloudKmsProjectId;
  }

  /** Returns true if the credential file is on GCS (and therefore expected to be encrypted). */
  private boolean isCloudSqlCredential() {
-    return credentialFilePath.startsWith(GCS_SCHEME);
+    return sqlAccessInfoFile.startsWith(GCS_SCHEME);
  }

  @Provides
  @Singleton
  SqlAccessInfo provideCloudSqlAccessInfo(Lazy<CloudSqlCredentialDecryptor> lazyDecryptor) {
+    checkArgument(!isNullOrEmpty(sqlAccessInfoFile), "Null or empty credentialFilePath");
    String line = readOnlyLineFromCredentialFile();
    if (isCloudSqlCredential()) {
      line = lazyDecryptor.get().decrypt(line);
@@ -101,7 +103,7 @@ class BeamJpaModule {

  String readOnlyLineFromCredentialFile() {
    try {
-      ResourceId resourceId = FileSystems.matchSingleFileSpec(credentialFilePath).resourceId();
+      ResourceId resourceId = FileSystems.matchSingleFileSpec(sqlAccessInfoFile).resourceId();
      try (BufferedReader reader =
          new BufferedReader(
              new InputStreamReader(
@@ -114,13 +116,13 @@ class BeamJpaModule {
  }

  @Provides
-  @Config("cloudSqlJdbcUrl")
+  @Config("beamCloudSqlJdbcUrl")
  String provideJdbcUrl(SqlAccessInfo sqlAccessInfo) {
    return sqlAccessInfo.jdbcUrl();
  }

  @Provides
-  @Config("cloudSqlInstanceConnectionName")
+  @Config("beamCloudSqlInstanceConnectionName")
  String provideSqlInstanceName(SqlAccessInfo sqlAccessInfo) {
    return sqlAccessInfo
        .cloudSqlInstanceName()
@@ -128,58 +130,45 @@ class BeamJpaModule {
  }

  @Provides
-  @Config("cloudSqlUsername")
+  @Config("beamCloudSqlUsername")
  String provideSqlUsername(SqlAccessInfo sqlAccessInfo) {
    return sqlAccessInfo.user();
  }

  @Provides
-  @Config("cloudSqlPassword")
+  @Config("beamCloudSqlPassword")
  String provideSqlPassword(SqlAccessInfo sqlAccessInfo) {
    return sqlAccessInfo.password();
  }

  @Provides
-  @Config("cloudKmsProjectId")
-  static String kmsProjectId() {
-    return "domain-registry-dev";
+  @Config("beamCloudKmsProjectId")
+  String kmsProjectId() {
+    return cloudKmsProjectId;
  }

  @Provides
-  @Config("cloudKmsKeyRing")
+  @Config("beamCloudKmsKeyRing")
  static String keyRingName() {
    return "nomulus-tool-keyring";
  }

  @Provides
-  @Config("defaultCredentialOauthScopes")
-  static ImmutableList<String> defaultCredentialOauthScopes() {
-    return ImmutableList.of("https://www.googleapis.com/auth/cloud-platform");
-  }
-
-  @Provides
-  @Named("transientFailureRetries")
-  static int transientFailureRetries() {
-    return 12;
-  }
-
-  @Module
-  interface BindModule {
-
-    @Binds
-    Sleeper sleeper(SystemSleeper sleeper);
-
-    @Binds
-    Clock clock(SystemClock clock);
+  @Config("beamHibernateHikariMaximumPoolSize")
+  static int getBeamHibernateHikariMaximumPoolSize() {
+    // TODO(weiminyu): make this configurable. Should be equal to number of cores.
+    return 4;
  }

  @Singleton
  @Component(
      modules = {
+        ConfigModule.class,
        CredentialModule.class,
        BeamJpaModule.class,
        KmsModule.class,
-        PersistenceModule.class
+        PersistenceModule.class,
+        UtilsModule.class
      })
  public interface JpaTransactionManagerComponent {
    @SocketFactoryJpaTm
@@ -18,6 +18,7 @@ import static com.google.common.base.Preconditions.checkArgument;

 import com.google.api.services.cloudkms.v1.model.DecryptRequest;
 import com.google.common.base.Strings;
+import google.registry.config.RegistryConfig.Config;
 import google.registry.keyring.kms.KmsConnection;
 import java.nio.charset.StandardCharsets;
 import java.util.Base64;
@@ -34,7 +35,7 @@ public class CloudSqlCredentialDecryptor {
  private final KmsConnection kmsConnection;

  @Inject
-  CloudSqlCredentialDecryptor(KmsConnection kmsConnection) {
+  CloudSqlCredentialDecryptor(@Config("beamKmsConnection") KmsConnection kmsConnection) {
    this.kmsConnection = kmsConnection;
  }

@@ -0,0 +1,75 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.beam.initsql;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.appengine.api.datastore.Entity;
+import java.util.Objects;
+
+/** Helper for manipulating {@code DomainBase} when migrating from Datastore to SQL database */
+final class DomainBaseUtil {
+
+  private DomainBaseUtil() {}
+
+  /**
+   * Removes {@link google.registry.model.billing.BillingEvent.Recurring}, {@link
+   * google.registry.model.poll.PollMessage PollMessages} and {@link
+   * google.registry.model.host.HostResource name servers} from a Datastore {@link Entity} that
+   * represents an Ofy {@link google.registry.model.domain.DomainBase}. This breaks the cycle of
+   * foreign key constraints between these entity kinds, allowing {@code DomainBases} to be inserted
+   * into the SQL database. See {@link InitSqlPipeline} for a use case, where the full {@code
+   * DomainBases} are written again during the last stage of the pipeline.
+   *
+   * <p>The returned object may be in bad state. Specifically, {@link
+   * google.registry.model.eppcommon.StatusValue#INACTIVE} is not added after name servers are
+   * removed. This only impacts tests.
+   *
+   * <p>This operation is performed on an Datastore {@link Entity} instead of Ofy Java object
+   * because Objectify requires access to a Datastore service when converting an Ofy object to a
+   * Datastore {@code Entity}. If we insist on working with Objectify objects, we face a few
+   * unsatisfactory options:
+   *
+   * <ul>
+   *   <li>Connect to our production Datastore, which incurs unnecessary security and code health
+   *       risk.
+   *   <li>Connect to a separate real Datastore instance, which is a waster and overkill.
+   *   <li>Use an in-memory test Datastore, which is a project health risk in that the test
+   *       Datastore would be added to Nomulus' production binary unless we create a separate
+   *       project for this pipeline.
+   * </ul>
+   *
+   * <p>Given our use case, operating on Datastore entities is the best option.
+   *
+   * @throws IllegalArgumentException if input does not represent a DomainBase
+   */
+  static Entity removeBillingAndPollAndHosts(Entity domainBase) {
+    checkNotNull(domainBase, "domainBase");
+    checkArgument(
+        Objects.equals(domainBase.getKind(), "DomainBase"),
+        "Expecting DomainBase, got %s",
+        domainBase.getKind());
+    Entity clone = domainBase.clone();
+    clone.removeProperty("autorenewBillingEvent");
+    clone.removeProperty("autorenewPollMessage");
+    clone.removeProperty("deletePollMessage");
+    clone.removeProperty("nsHosts");
+    domainBase.getProperties().keySet().stream()
+        .filter(s -> s.startsWith("transferData."))
+        .forEach(s -> clone.removeProperty(s));
+    return clone;
+  }
+}
@@ -0,0 +1,237 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.beam.initsql;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.googlecode.objectify.Key;
+import google.registry.backup.AppEngineEnvironment;
+import google.registry.backup.VersionedEntity;
+import google.registry.beam.initsql.BeamJpaModule.JpaTransactionManagerComponent;
+import google.registry.beam.initsql.Transforms.RemoveDomainBaseForeignKeys;
+import google.registry.model.billing.BillingEvent;
+import google.registry.model.contact.ContactResource;
+import google.registry.model.domain.DomainBase;
+import google.registry.model.domain.token.AllocationToken;
+import google.registry.model.host.HostResource;
+import google.registry.model.poll.PollMessage;
+import google.registry.model.registrar.Registrar;
+import google.registry.model.registrar.RegistrarContact;
+import google.registry.model.registry.Registry;
+import google.registry.model.reporting.HistoryEntry;
+import google.registry.persistence.transaction.JpaTransactionManager;
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Optional;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.Wait;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.joda.time.DateTime;
+
+/**
+ * A BEAM pipeline that populates a SQL database with data from a Datastore backup.
+ *
+ * <p>This pipeline migrates EPP resources and related entities that cross-reference each other. To
+ * avoid violating foreign key constraints, writes to SQL are ordered by entity kinds. In addition,
+ * the {@link DomainBase} kind is written twice (see details below). The write order is presented
+ * below. Although some kinds can be written concurrently, e.g. {@code ContactResource} and {@code
+ * RegistrarContact}, we do not expect any performance benefit since the limiting resource is the
+ * number of JDBC connections. Google internal users may refer to <a
+ * href="http://go/registry-r3-init-sql">the design doc</a> for more information.
+ *
+ * <ol>
+ *   <li>{@link Registry}: Assumes that {@code PremiumList} and {@code ReservedList} have been set
+ *       up in the SQL database.
+ *   <li>{@link Registrar}: Logically depends on {@code Registry}, Foreign key not modeled yet.
+ *   <li>{@link ContactResource}: references {@code Registrar}
+ *   <li>{@link RegistrarContact}: references {@code Registrar}.
+ *   <li>Cleansed {@link DomainBase}: with references to {@code BillingEvent}, {@code Recurring},
+ *       {@code Cancellation} and {@code HostResource} removed, still references {@code Registrar}
+ *       and {@code ContactResource}. The removal breaks circular Foreign Key references.
+ *   <li>{@link HostResource}: references {@code DomainBase}.
+ *   <li>{@link HistoryEntry}: maps to one of three SQL entity types and may reference {@code
+ *       Registrar}, {@code ContactResource}, {@code HostResource}, and {@code DomainBase}.
+ *   <li>{@link AllocationToken}: references {@code HistoryEntry}.
+ *   <li>{@link BillingEvent.Recurring}: references {@code Registrar}, {@code DomainBase} and {@code
+ *       HistoryEntry}.
+ *   <li>{@link BillingEvent.OneTime}: references {@code Registrar}, {@code DomainBase}, {@code
+ *       BillingEvent.Recurring}, {@code HistoryEntry} and {@code AllocationToken}.
+ *   <li>{@link BillingEvent.Modification}: SQL model TBD. Will reference {@code Registrar}, {@code
+ *       DomainBase} and {@code BillingEvent.OneTime}.
+ *   <li>{@link BillingEvent.Cancellation}: references {@code Registrar}, {@code DomainBase}, {@code
+ *       BillingEvent.Recurring}, {@code BillingEvent.OneTime}, and {@code HistoryEntry}.
+ *   <li>{@link PollMessage}: references {@code Registrar}, {@code DomainBase}, {@code
+ *       ContactResource}, {@code HostResource}, and {@code HistoryEntry}.
+ *   <li>{@link DomainBase}, original copy from Datastore.
+ * </ol>
+ */
+public class InitSqlPipeline implements Serializable {
+
+  /**
+   * Datastore kinds to be written to the SQL database before the cleansed version of {@link
+   * DomainBase}.
+   */
+  // TODO(weiminyu): include Registry.class when it is modeled in JPA.
+  private static final ImmutableList<Class<?>> PHASE_ONE_ORDERED =
+      ImmutableList.of(Registrar.class, ContactResource.class);
+
+  /**
+   * Datastore kinds to be written to the SQL database after the cleansed version of {@link
+   * DomainBase}.
+   *
+   * <p>The following entities are missing from the list:
+   *
+   * <ul>
+   *   <li>Those not modeled in JPA yet, e.g., {@code BillingEvent.Modification}.
+   *   <li>Those waiting for sanitation, e.g., {@code HistoryEntry}, which would have duplicate keys
+   *       after converting to SQL model.
+   *   <li>Those that have foreign key constraints on the above.
+   * </ul>
+   */
+  // TODO(weiminyu): add more entities when available.
+  private static final ImmutableList<Class<?>> PHASE_TWO_ORDERED =
+      ImmutableList.of(HostResource.class);
+
+  private final InitSqlPipelineOptions options;
+
+  private final Pipeline pipeline;
+
+  private final SerializableFunction<JpaTransactionManagerComponent, JpaTransactionManager>
+      jpaGetter;
+
+  InitSqlPipeline(InitSqlPipelineOptions options) {
+    this.options = options;
+    pipeline = Pipeline.create(options);
+    jpaGetter = JpaTransactionManagerComponent::cloudSqlJpaTransactionManager;
+  }
+
+  @VisibleForTesting
+  InitSqlPipeline(InitSqlPipelineOptions options, Pipeline pipeline) {
+    this.options = options;
+    this.pipeline = pipeline;
+    jpaGetter = JpaTransactionManagerComponent::localDbJpaTransactionManager;
+  }
+
+  public PipelineResult run() {
+    setupPipeline();
+    return pipeline.run();
+  }
+
+  @VisibleForTesting
+  void setupPipeline() {
+    PCollectionTuple datastoreSnapshot =
+        pipeline.apply(
+            "Load Datastore snapshot",
+            Transforms.loadDatastoreSnapshot(
+                options.getDatastoreExportDir(),
+                options.getCommitLogDir(),
+                DateTime.parse(options.getCommitLogStartTimestamp()),
+                DateTime.parse(options.getCommitLogEndTimestamp()),
+                ImmutableSet.<String>builder()
+                    .add("DomainBase")
+                    .addAll(toKindStrings(PHASE_ONE_ORDERED))
+                    .addAll(toKindStrings(PHASE_TWO_ORDERED))
+                    .build()));
+
+    // Set up the pipeline to write entity kinds from PHASE_ONE_ORDERED to SQL. Return a object
+    // that signals the completion of the phase.
+    PCollection<Void> blocker =
+        scheduleOnePhaseWrites(datastoreSnapshot, PHASE_ONE_ORDERED, Optional.empty(), null);
+    blocker =
+        writeToSql(
+            "DomainBase without circular foreign keys",
+            removeDomainBaseForeignKeys(datastoreSnapshot)
+                .apply("Wait on phase one", Wait.on(blocker)));
+    // Set up the pipeline to write entity kinds from PHASE_TWO_ORDERED to SQL. This phase won't
+    // start until all cleansed DomainBases have been written (started by line above).
+    scheduleOnePhaseWrites(
+        datastoreSnapshot, PHASE_TWO_ORDERED, Optional.of(blocker), "DomainBaseNoFkeys");
+  }
+
+  private PCollection<VersionedEntity> removeDomainBaseForeignKeys(
+      PCollectionTuple datastoreSnapshot) {
+    PCollection<VersionedEntity> domainBases =
+        datastoreSnapshot.get(Transforms.createTagForKind("DomainBase"));
+    return domainBases.apply(
+        "Remove circular foreign keys from DomainBase",
+        ParDo.of(new RemoveDomainBaseForeignKeys()));
+  }
+
+  /**
+   * Sets up the pipeline to write entities in {@code entityClasses} to SQL. Entities are written
+   * one kind at a time based on each kind's position in {@code entityClasses}. Concurrency exists
+   * within each kind.
+   *
+   * @param datastoreSnapshot the Datastore snapshot of all data to be migrated to SQL
+   * @param entityClasses the entity types in write order
+   * @param blockingPCollection the pipeline stage that blocks this phase
+   * @param blockingTag description of the stage (if exists) that blocks this phase. Needed for
+   *     generating unique transform ids
+   * @return the output {@code PCollection} from the writing of the last entity kind. Other parts of
+   *     the pipeline can {@link Wait} on this object
+   */
+  private PCollection<Void> scheduleOnePhaseWrites(
+      PCollectionTuple datastoreSnapshot,
+      Collection<Class<?>> entityClasses,
+      Optional<PCollection<Void>> blockingPCollection,
+      String blockingTag) {
+    checkArgument(!entityClasses.isEmpty(), "Each phase must have at least one kind.");
+    ImmutableList<TupleTag<VersionedEntity>> tags =
+        toKindStrings(entityClasses).stream()
+            .map(Transforms::createTagForKind)
+            .collect(ImmutableList.toImmutableList());
+
+    PCollection<Void> prev = blockingPCollection.orElse(null);
+    String prevTag = blockingTag;
+    for (TupleTag<VersionedEntity> tag : tags) {
+      PCollection<VersionedEntity> curr = datastoreSnapshot.get(tag);
+      if (prev != null) {
+        curr = curr.apply("Wait on " + prevTag, Wait.on(prev));
+      }
+      prev = writeToSql(tag.getId(), curr);
+      prevTag = tag.getId();
+    }
+    return prev;
+  }
+
+  private PCollection<Void> writeToSql(String transformId, PCollection<VersionedEntity> data) {
+    String credentialFileUrl =
+        options.getSqlCredentialUrlOverride() != null
+            ? options.getSqlCredentialUrlOverride()
+            : BackupPaths.getCloudSQLCredentialFilePatterns(options.getEnvironment()).get(0);
+
+    return data.apply(
+        "Write to sql: " + transformId,
+        Transforms.writeToSql(
+            transformId,
+            options.getMaxConcurrentSqlWriters(),
+            options.getSqlWriteBatchSize(),
+            new JpaSupplierFactory(credentialFileUrl, options.getCloudKmsProjectId(), jpaGetter)));
+  }
+
+  private static ImmutableList<String> toKindStrings(Collection<Class<?>> entityClasses) {
+    try (AppEngineEnvironment env = new AppEngineEnvironment()) {
+      return entityClasses.stream().map(Key::getKind).collect(ImmutableList.toImmutableList());
+    }
+  }
+}
@@ -0,0 +1,91 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.beam.initsql;
+
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.Validation;
+
+/** Pipeline options for {@link InitSqlPipeline} */
+public interface InitSqlPipelineOptions extends GcpOptions {
+
+  @Description(
+      "Overrides the URL to the SQL credential file. " + "Required if environment is not provided.")
+  @Nullable
+  String getSqlCredentialUrlOverride();
+
+  void setSqlCredentialUrlOverride(String credentialUrlOverride);
+
+  @Description("The root directory of the export to load.")
+  String getDatastoreExportDir();
+
+  void setDatastoreExportDir(String datastoreExportDir);
+
+  @Description("The directory that contains all CommitLog files.")
+  String getCommitLogDir();
+
+  void setCommitLogDir(String commitLogDir);
+
+  @Description("The earliest CommitLogs to load, in ISO8601 format.")
+  @Validation.Required
+  String getCommitLogStartTimestamp();
+
+  void setCommitLogStartTimestamp(String commitLogStartTimestamp);
+
+  @Description("The latest CommitLogs to load, in ISO8601 format.")
+  @Validation.Required
+  String getCommitLogEndTimestamp();
+
+  void setCommitLogEndTimestamp(String commitLogEndTimestamp);
+
+  @Description(
+      "The deployed environment, alpha, crash, sandbox, or production. "
+          + "Not required only if sqlCredentialUrlOverride is provided.")
+  @Nullable
+  String getEnvironment();
+
+  void setEnvironment(String environment);
+
+  @Description(
+      "The GCP project that contains the keyring used for decrypting the " + "SQL credential file.")
+  @Nullable
+  String getCloudKmsProjectId();
+
+  void setCloudKmsProjectId(String cloudKmsProjectId);
+
+  @Description(
+      "The maximum JDBC connection pool size on a VM. "
+          + "This value should be equal to or greater than the number of cores on the VM.")
+  @Default.Integer(4)
+  int getJdbcMaxPoolSize();
+
+  void setJdbcMaxPoolSize(int jdbcMaxPoolSize);
+
+  @Description(
+      "A hint to the pipeline runner of the maximum number of concurrent SQL writers to create. "
+          + "Note that multiple writers may run on the same VM and share the connection pool.")
+  @Default.Integer(4)
+  int getMaxConcurrentSqlWriters();
+
+  void setMaxConcurrentSqlWriters(int maxConcurrentSqlWriters);
+
+  @Description("The number of entities to be written to the SQL database in one transaction.")
+  @Default.Integer(20)
+  int getSqlWriteBatchSize();
+
+  void setSqlWriteBatchSize(int sqlWriteBatchSize);
+}
@@ -0,0 +1,48 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.beam.initsql;
+
+import google.registry.beam.initsql.BeamJpaModule.JpaTransactionManagerComponent;
+import google.registry.beam.initsql.Transforms.SerializableSupplier;
+import google.registry.persistence.transaction.JpaTransactionManager;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+
+public class JpaSupplierFactory implements SerializableSupplier<JpaTransactionManager> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final String credentialFileUrl;
+  @Nullable private final String cloudKmsProjectId;
+  private final SerializableFunction<JpaTransactionManagerComponent, JpaTransactionManager>
+      jpaGetter;
+
+  public JpaSupplierFactory(
+      String credentialFileUrl,
+      @Nullable String cloudKmsProjectId,
+      SerializableFunction<JpaTransactionManagerComponent, JpaTransactionManager> jpaGetter) {
+    this.credentialFileUrl = credentialFileUrl;
+    this.cloudKmsProjectId = cloudKmsProjectId;
+    this.jpaGetter = jpaGetter;
+  }
+
+  @Override
+  public JpaTransactionManager get() {
+    return jpaGetter.apply(
+        DaggerBeamJpaModule_JpaTransactionManagerComponent.builder()
+            .beamJpaModule(new BeamJpaModule(credentialFileUrl, cloudKmsProjectId))
+            .build());
+  }
+}
@@ -1,3 +1,17 @@
 ## Summary

-This package contains a BEAM pipeline that populates a Cloud SQL database from a Datastore backup.
+This package contains a BEAM pipeline that populates a Cloud SQL database from a
+Datastore backup. The pipeline uses an unsynchronized Datastore export and
+overlapping CommitLogs generated by the Nomulus server to recreate a consistent
+Datastore snapshot, and writes the data to a Cloud SQL instance.
+
+## Pipeline Visualization
+
+The golden flow graph of the InitSqlPipeline is saved both as a text-base
+[DOT file](../../../../../../test/resources/google/registry/beam/initsql/pipeline_golden.dot)
+and a
+[.png file](../../../../../../test/resources/google/registry/beam/initsql/pipeline_golden.png).
+A test compares the flow graph of the current pipeline with the golden graph,
+and will fail if changes are detected. When this happens, run the Gradle task
+':core:updateInitSqlPipelineGraph' to update the golden files and review the
+changes.
@@ -16,18 +16,43 @@ package google.registry.beam.initsql;

 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
+import static com.google.common.base.Preconditions.checkState;
+import static com.google.common.base.Throwables.throwIfUnchecked;
 import static google.registry.beam.initsql.BackupPaths.getCommitLogTimestamp;
 import static google.registry.beam.initsql.BackupPaths.getExportFilePatterns;
+import static google.registry.persistence.JpaRetries.isFailedTxnRetriable;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.persistence.transaction.TransactionManagerFactory.setJpaTm;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
 import static google.registry.util.DateTimeUtils.isBeforeOrAt;
+import static java.util.Comparator.comparing;
+import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;

 import avro.shaded.com.google.common.collect.Iterators;
+import com.google.appengine.api.datastore.Entity;
+import com.google.appengine.api.datastore.EntityTranslator;
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Streams;
+import google.registry.backup.AppEngineEnvironment;
 import google.registry.backup.CommitLogImports;
 import google.registry.backup.VersionedEntity;
+import google.registry.model.domain.DomainBase;
+import google.registry.model.ofy.ObjectifyService;
+import google.registry.model.ofy.Ofy;
+import google.registry.persistence.transaction.JpaTransactionManager;
 import google.registry.tools.LevelDbLogReader;
+import google.registry.util.SystemSleeper;
+import java.io.Serializable;
 import java.util.Collection;
 import java.util.Iterator;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.Supplier;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.io.Compression;
 import org.apache.beam.sdk.io.FileIO;
@@ -36,12 +61,23 @@ import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
 import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.GroupIntoBatches;
+import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.ProcessFunction;
+import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
 import org.joda.time.DateTime;
+import org.joda.time.Duration;

 /**
 * {@link PTransform Pipeline transforms} used in pipelines that load from both Datastore export
@@ -57,6 +93,110 @@ public final class Transforms {
   */
  @VisibleForTesting static final long EXPORT_ENTITY_TIME_STAMP = START_OF_TIME.getMillis();

+  /**
+   * Returns a {@link TupleTag} that can be used to retrieve entities of the given {@code kind} from
+   * the Datastore snapshot returned by {@link #loadDatastoreSnapshot}.
+   */
+  public static TupleTag<VersionedEntity> createTagForKind(String kind) {
+    // When used with PCollectionTuple the result must retain generic type information.
+    // Both the Generic param and the empty bracket below are important.
+    return new TupleTag<VersionedEntity>(Transforms.class.getSimpleName() + ":" + kind) {};
+  }
+
+  /**
+   * Composite {@link PTransform transform} that loads the Datastore snapshot at {@code
+   * commitLogToTime} for caller specified {@code kinds}.
+   *
+   * <p>Caller must provide the location of a Datastore export that started AFTER {@code
+   * commitLogFromTime} and completed BEFORE {@code commitLogToTime}, as well as the root directory
+   * of all CommitLog files.
+   *
+   * <p>Selection of {@code commitLogFromTime} and {@code commitLogToTime} should follow the
+   * guidelines below to ensure that all incremental changes concurrent with the export are covered:
+   *
+   * <ul>
+   *   <li>Two or more CommitLogs should exist between {@code commitLogFromTime} and the starting
+   *       time of the Datastore export. This ensures that the earlier CommitLog file was complete
+   *       before the export started.
+   *   <li>Two or more CommitLogs should exit between the export completion time and {@code
+   *       commitLogToTime}.
+   * </ul>
+   *
+   * <p>The output from the returned transform is a {@link PCollectionTuple} consisting of {@link
+   * VersionedEntity VersionedEntities} grouped into {@link PCollection PCollections} by {@code
+   * kind}.
+   */
+  public static PTransform<PBegin, PCollectionTuple> loadDatastoreSnapshot(
+      String exportDir,
+      String commitLogDir,
+      DateTime commitLogFromTime,
+      DateTime commitLogToTime,
+      Set<String> kinds) {
+    checkArgument(kinds != null && !kinds.isEmpty(), "At least one kind is expected.");
+
+    // Create tags to collect entities by kind in final step.
+    final ImmutableMap<String, TupleTag<VersionedEntity>> outputTags =
+        kinds.stream()
+            .collect(ImmutableMap.toImmutableMap(kind -> kind, Transforms::createTagForKind));
+    // Arbitrarily select one tag as mainOutTag and put the remaining ones in a TupleTagList.
+    // This separation is required by ParDo's config API.
+    Iterator<TupleTag<VersionedEntity>> tagsIt = outputTags.values().iterator();
+    final TupleTag<VersionedEntity> mainOutputTag = tagsIt.next();
+    final TupleTagList additionalTags = TupleTagList.of(ImmutableList.copyOf(tagsIt));
+
+    return new PTransform<PBegin, PCollectionTuple>() {
+      @Override
+      public PCollectionTuple expand(PBegin input) {
+        PCollection<VersionedEntity> exportedEntities =
+            input
+                .apply("Get export file patterns", getDatastoreExportFilePatterns(exportDir, kinds))
+                .apply("Find export files", getFilesByPatterns())
+                .apply("Load export data", loadExportDataFromFiles());
+        PCollection<VersionedEntity> commitLogEntities =
+            input
+                .apply("Get commitlog file patterns", getCommitLogFilePatterns(commitLogDir))
+                .apply("Find commitlog files", getFilesByPatterns())
+                .apply(
+                    "Filter commitLog by time",
+                    filterCommitLogsByTime(commitLogFromTime, commitLogToTime))
+                .apply("Load commitlog data", loadCommitLogsFromFiles(kinds));
+        return PCollectionList.of(exportedEntities)
+            .and(commitLogEntities)
+            .apply("Merge exports and CommitLogs", Flatten.pCollections())
+            .apply(
+                "Key entities by Datastore Keys",
+                // Converting to KV<String, VE> instead of KV<Key, VE> b/c default coder for Key
+                // (SerializableCoder) is not deterministic and cannot be used with GroupBy.
+                MapElements.into(kvs(strings(), TypeDescriptor.of(VersionedEntity.class)))
+                    .via((VersionedEntity e) -> KV.of(e.key().toString(), e)))
+            .apply("Gather entities by key", GroupByKey.create())
+            .apply(
+                "Output latest version per entity",
+                ParDo.of(
+                        new DoFn<KV<String, Iterable<VersionedEntity>>, VersionedEntity>() {
+                          @ProcessElement
+                          public void processElement(
+                              @Element KV<String, Iterable<VersionedEntity>> kv,
+                              MultiOutputReceiver out) {
+                            Optional<VersionedEntity> latest =
+                                Streams.stream(kv.getValue())
+                                    .sorted(comparing(VersionedEntity::commitTimeMills).reversed())
+                                    .findFirst();
+                            // Throw to abort (after default retries). Investigate, fix, and rerun.
+                            checkState(
+                                latest.isPresent(), "Unexpected key with no data", kv.getKey());
+                            if (latest.get().isDelete()) {
+                              return;
+                            }
+                            String kind = latest.get().getEntity().get().getKind();
+                            out.get(outputTags.get(kind)).output(latest.get());
+                          }
+                        })
+                    .withOutputTags(mainOutputTag, additionalTags));
+      }
+    };
+  }
+
  /**
   * Returns a {@link PTransform transform} that can generate a collection of patterns that match
   * all Datastore CommitLog files.
@@ -87,7 +227,7 @@ public final class Transforms {
    return new PTransform<PCollection<String>, PCollection<Metadata>>() {
      @Override
      public PCollection<Metadata> expand(PCollection<String> input) {
-        return input.apply(FileIO.matchAll().withEmptyMatchTreatment(EmptyMatchTreatment.DISALLOW));
+        return input.apply(FileIO.matchAll().withEmptyMatchTreatment(EmptyMatchTreatment.ALLOW));
      }
    };
  }
@@ -96,7 +236,7 @@ public final class Transforms {
   * Returns CommitLog files with timestamps between {@code fromTime} (inclusive) and {@code
   * endTime} (exclusive).
   */
-  public static PTransform<PCollection<? extends String>, PCollection<String>>
+  public static PTransform<PCollection<? extends Metadata>, PCollection<Metadata>>
      filterCommitLogsByTime(DateTime fromTime, DateTime toTime) {
    return ParDo.of(new FilterCommitLogFileByTime(fromTime, toTime));
  }
@@ -114,11 +254,51 @@ public final class Transforms {

  /** Returns a {@link PTransform} from file {@link Metadata} to {@link VersionedEntity}. */
  public static PTransform<PCollection<Metadata>, PCollection<VersionedEntity>>
-      loadCommitLogsFromFiles() {
+      loadCommitLogsFromFiles(Set<String> kinds) {
    return processFiles(
-        new BackupFileReader(file -> CommitLogImports.loadEntities(file.open()).iterator()));
+        new BackupFileReader(
+            file ->
+                CommitLogImports.loadEntities(file.open()).stream()
+                    .filter(e -> kinds.contains(e.key().getKind()))
+                    .iterator()));
  }

+  /**
+   * Returns a {@link PTransform} that writes a {@link PCollection} of entities to a SQL database.
+   * and outputs an empty {@code PCollection<Void>}. This allows other operations to {@link
+   * org.apache.beam.sdk.transforms.Wait wait} for the completion of this transform.
+   *
+   * <p>Errors are handled according to the pipeline runner's default policy. As part of a one-time
+   * job, we will not add features unless proven necessary.
+   *
+   * @param transformId a unique ID for an instance of the returned transform
+   * @param maxWriters the max number of concurrent writes to SQL, which also determines the max
+   *     number of connection pools created
+   * @param batchSize the number of entities to write in each operation
+   * @param jpaSupplier supplier of a {@link JpaTransactionManager}
+   */
+  public static PTransform<PCollection<VersionedEntity>, PCollection<Void>> writeToSql(
+      String transformId,
+      int maxWriters,
+      int batchSize,
+      SerializableSupplier<JpaTransactionManager> jpaSupplier) {
+    return new PTransform<PCollection<VersionedEntity>, PCollection<Void>>() {
+      @Override
+      public PCollection<Void> expand(PCollection<VersionedEntity> input) {
+        return input
+            .apply(
+                "Shard data for " + transformId,
+                MapElements.into(kvs(integers(), TypeDescriptor.of(VersionedEntity.class)))
+                    .via(ve -> KV.of(ThreadLocalRandom.current().nextInt(maxWriters), ve)))
+            .apply("Batch output by shard " + transformId, GroupIntoBatches.ofSize(batchSize))
+            .apply("Write in batch for " + transformId, ParDo.of(new SqlBatchWriter(jpaSupplier)));
+      }
+    };
+  }
+
+  /** Interface for serializable {@link Supplier suppliers}. */
+  public interface SerializableSupplier<T> extends Supplier<T>, Serializable {}
+
  /**
   * Returns a {@link PTransform} that produces a {@link PCollection} containing all elements in the
   * given {@link Iterable}.
@@ -139,12 +319,11 @@ public final class Transforms {
        return input
            .apply(FileIO.readMatches().withCompression(Compression.UNCOMPRESSED))
            .apply(transformer.getClass().getSimpleName(), ParDo.of(transformer));
-        // TODO(weiminyu): reshuffle to enable dynamic work rebalance per beam dev guide
      }
    };
  }

-  private static class FilterCommitLogFileByTime extends DoFn<String, String> {
+  private static class FilterCommitLogFileByTime extends DoFn<Metadata, Metadata> {
    private final DateTime fromTime;
    private final DateTime toTime;

@@ -161,10 +340,10 @@ public final class Transforms {
    }

    @ProcessElement
-    public void processElement(@Element String fileName, OutputReceiver<String> out) {
-      DateTime timestamp = getCommitLogTimestamp(fileName);
+    public void processElement(@Element Metadata fileMeta, OutputReceiver<Metadata> out) {
+      DateTime timestamp = getCommitLogTimestamp(fileMeta.resourceId().toString());
      if (isBeforeOrAt(fromTime, timestamp) && timestamp.isBefore(toTime)) {
-        out.output(fileName);
+        out.output(fileMeta);
      }
    }
  }
@@ -197,4 +376,116 @@ public final class Transforms {
      }
    }
  }
+
+  /**
+   * Writes a batch of entities to a SQL database.
+   *
+   * <p>Note that an arbitrary number of instances of this class may be created and freed in
+   * arbitrary order in a single JVM. Due to the tech debt that forced us to use a static variable
+   * to hold the {@code JpaTransactionManager} instance, we must ensure that JpaTransactionManager
+   * is not changed or torn down while being used by some instance.
+   */
+  private static class SqlBatchWriter extends DoFn<KV<Integer, Iterable<VersionedEntity>>, Void> {
+
+    private static int instanceCount = 0;
+    private static JpaTransactionManager originalJpa;
+
+    private final SerializableSupplier<JpaTransactionManager> jpaSupplier;
+
+    private transient Ofy ofy;
+    private transient SystemSleeper sleeper;
+
+    SqlBatchWriter(SerializableSupplier<JpaTransactionManager> jpaSupplier) {
+      this.jpaSupplier = jpaSupplier;
+    }
+
+    @Setup
+    public void setup() {
+      sleeper = new SystemSleeper();
+
+      try (AppEngineEnvironment env = new AppEngineEnvironment()) {
+        ObjectifyService.initOfy();
+        ofy = ObjectifyService.ofy();
+      }
+
+      synchronized (SqlBatchWriter.class) {
+        if (instanceCount == 0) {
+          originalJpa = jpaTm();
+          setJpaTm(jpaSupplier);
+        }
+        instanceCount++;
+      }
+    }
+
+    @Teardown
+    public void teardown() {
+      synchronized (SqlBatchWriter.class) {
+        instanceCount--;
+        if (instanceCount == 0) {
+          jpaTm().teardown();
+          setJpaTm(() -> originalJpa);
+        }
+      }
+    }
+
+    @ProcessElement
+    public void processElement(@Element KV<Integer, Iterable<VersionedEntity>> kv) {
+      try (AppEngineEnvironment env = new AppEngineEnvironment()) {
+        ImmutableList<Object> ofyEntities =
+            Streams.stream(kv.getValue())
+                .map(VersionedEntity::getEntity)
+                .map(Optional::get)
+                .map(ofy::toPojo)
+                .collect(ImmutableList.toImmutableList());
+        retry(() -> jpaTm().transact(() -> jpaTm().saveNewOrUpdateAll(ofyEntities)));
+      }
+    }
+
+    // TODO(b/160632289): Enhance Retrier and use it here.
+    private void retry(Runnable runnable) {
+      int maxAttempts = 5;
+      int initialDelayMillis = 100;
+      double jitterRatio = 0.2;
+
+      for (int attempt = 0; attempt < maxAttempts; attempt++) {
+        try {
+          runnable.run();
+          return;
+        } catch (Throwable throwable) {
+          if (!isFailedTxnRetriable(throwable)) {
+            throwIfUnchecked(throwable);
+            throw new RuntimeException(throwable);
+          }
+          int sleepMillis = (1 << attempt) * initialDelayMillis;
+          int jitter =
+              ThreadLocalRandom.current().nextInt((int) (sleepMillis * jitterRatio))
+                  - (int) (sleepMillis * jitterRatio / 2);
+          sleeper.sleepUninterruptibly(Duration.millis(sleepMillis + jitter));
+        }
+      }
+    }
+  }
+
+  /**
+   * Removes BillingEvents, {@link google.registry.model.poll.PollMessage PollMessages} and {@link
+   * google.registry.model.host.HostResource} from a {@link DomainBase}. These are circular foreign
+   * key constraints that prevent migration of {@code DomainBase} to SQL databases.
+   *
+   * <p>See {@link InitSqlPipeline} for more information.
+   */
+  static class RemoveDomainBaseForeignKeys extends DoFn<VersionedEntity, VersionedEntity> {
+
+    @ProcessElement
+    public void processElement(
+        @Element VersionedEntity domainBase, OutputReceiver<VersionedEntity> out) {
+      checkArgument(
+          domainBase.getEntity().isPresent(), "Unexpected delete entity %s", domainBase.key());
+      Entity outputEntity =
+          DomainBaseUtil.removeBillingAndPollAndHosts(domainBase.getEntity().get());
+      out.output(
+          VersionedEntity.from(
+              domainBase.commitTimeMills(),
+              EntityTranslator.convertToPb(outputEntity).toByteArray()));
+    }
+  }
 }
@@ -235,6 +235,7 @@ public abstract class BillingEvent implements Serializable {
                DATE_TIME_FORMATTER.format(eventTime()),
                registrarId(),
                billingId(),
+                poNumber(),
                tld(),
                action(),
                domain(),
@@ -141,7 +141,7 @@ public class SafeBrowsingTransforms {
    @ProcessElement
    public void processElement(ProcessContext context) {
      Subdomain subdomain = context.element();
-      subdomainBuffer.put(subdomain.fullyQualifiedDomainName(), subdomain);
+      subdomainBuffer.put(subdomain.domainName(), subdomain);
      if (subdomainBuffer.size() >= BATCH_SIZE) {
        ImmutableSet<KV<Subdomain, ThreatMatch>> results = evaluateAndFlush();
        results.forEach(context::output);
@@ -239,7 +239,7 @@ public class SafeBrowsingTransforms {
            String url = match.getJSONObject("threat").getString("url");
            Subdomain subdomain = subdomainBuffer.get(url);
            resultBuilder.add(
-                KV.of(subdomain, ThreatMatch.create(match, subdomain.fullyQualifiedDomainName())));
+                KV.of(subdomain, ThreatMatch.create(match, subdomain.domainName())));
          }
        }
      }
@@ -77,7 +77,7 @@ public class Spec11Pipeline implements Serializable {
  public static final String REGISTRAR_EMAIL_FIELD = "registrarEmailAddress";
  /** The JSON object field into which we put the registrar's name for Spec11 reports. */
  public static final String REGISTRAR_CLIENT_ID_FIELD = "registrarClientId";
-  /** The JSON object field we put the threat match array for Spec11 reports. */
+  /** The JSON object field into which we put the threat match array for Spec11 reports. */
  public static final String THREAT_MATCHES_FIELD = "threatMatches";

  private final String projectId;
@@ -94,8 +94,7 @@ public class Spec11Pipeline implements Serializable {
      @Config("spec11TemplateUrl") String spec11TemplateUrl,
      @Config("reportingBucketUrl") String reportingBucketUrl,
      @LocalCredential GoogleCredentialsBundle googleCredentialsBundle,
-      Retrier retrier
-  ) {
+      Retrier retrier) {
    this.projectId = projectId;
    this.beamStagingUrl = beamStagingUrl;
    this.spec11TemplateUrl = spec11TemplateUrl;
@@ -177,9 +176,11 @@ public class Spec11Pipeline implements Serializable {
      PCollection<Subdomain> domains,
      EvaluateSafeBrowsingFn evaluateSafeBrowsingFn,
      ValueProvider<String> dateProvider) {
-    PCollection<KV<Subdomain, ThreatMatch>> subdomains =
+
+    /* Store ThreatMatch objects in JSON. */
+    PCollection<KV<Subdomain, ThreatMatch>> subdomainsJson =
        domains.apply("Run through SafeBrowsingAPI", ParDo.of(evaluateSafeBrowsingFn));
-    subdomains
+    subdomainsJson
        .apply(
            "Map registrar client ID to email/ThreatMatch pair",
            MapElements.into(
@@ -188,7 +189,7 @@ public class Spec11Pipeline implements Serializable {
                .via(
                    (KV<Subdomain, ThreatMatch> kv) ->
                        KV.of(
-                            kv.getKey().registrarClientId(),
+                            kv.getKey().registrarId(),
                            EmailAndThreatMatch.create(
                                kv.getKey().registrarEmailAddress(), kv.getValue()))))
        .apply("Group by registrar client ID", GroupByKey.create())
@@ -36,12 +36,14 @@ import org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord;
 public abstract class Subdomain implements Serializable {

  private static final ImmutableList<String> FIELD_NAMES =
-      ImmutableList.of("fullyQualifiedDomainName", "registrarClientId", "registrarEmailAddress");
+      ImmutableList.of("domainName", "domainRepoId", "registrarId", "registrarEmailAddress");

  /** Returns the fully qualified domain name. */
-  abstract String fullyQualifiedDomainName();
-  /** Returns the client ID of the associated registrar for this domain. */
-  abstract String registrarClientId();
+  abstract String domainName();
+  /** Returns the domain repo ID (the primary key of the domain table). */
+  abstract String domainRepoId();
+  /** Returns the registrar ID of the associated registrar for this domain. */
+  abstract String registrarId();
  /** Returns the email address of the registrar associated with this domain. */
  abstract String registrarEmailAddress();

@@ -56,8 +58,9 @@ public abstract class Subdomain implements Serializable {
    checkFieldsNotNull(FIELD_NAMES, schemaAndRecord);
    GenericRecord record = schemaAndRecord.getRecord();
    return create(
-        extractField(record, "fullyQualifiedDomainName"),
-        extractField(record, "registrarClientId"),
+        extractField(record, "domainName"),
+        extractField(record, "domainRepoId"),
+        extractField(record, "registrarId"),
        extractField(record, "registrarEmailAddress"));
  }

@@ -69,9 +72,11 @@ public abstract class Subdomain implements Serializable {
   */
  @VisibleForTesting
  static Subdomain create(
-      String fullyQualifiedDomainName, String registrarClientId, String registrarEmailAddress) {
+      String domainName,
+      String domainRepoId,
+      String registrarId,
+      String registrarEmailAddress) {
    return new AutoValue_Subdomain(
-        fullyQualifiedDomainName, registrarClientId, registrarEmailAddress);
+        domainName, domainRepoId, registrarId, registrarEmailAddress);
  }
 }
-
@@ -19,11 +19,13 @@
  -- email address.

 SELECT
-  domain.fullyQualifiedDomainName AS fullyQualifiedDomainName,
-  registrar.clientId AS registrarClientId,
+  domain.fullyQualifiedDomainName AS domainName,
+  domain.__key__.name AS domainRepoId,
+  registrar.clientId AS registrarId,
  COALESCE(registrar.emailAddress, '') AS registrarEmailAddress
 FROM ( (
    SELECT
+      __key__,
      fullyQualifiedDomainName,
      currentSponsorClientId,
      creationTime
@@ -1527,6 +1527,21 @@ public final class RegistryConfig {
    return CONFIG_SETTINGS.get().hibernate.hikariIdleTimeout;
  }

+  /**
+   * Returns whether to replicate cloud SQL transactions to datastore.
+   *
+   * <p>If true, all cloud SQL transactions will be persisted as TransactionEntity objects in the
+   * Transaction table and replayed against datastore in a cron job.
+   */
+  public static boolean getCloudSqlReplicateTransactions() {
+    return CONFIG_SETTINGS.get().cloudSql.replicateTransactions;
+  }
+
+  @VisibleForTesting
+  public static void overrideCloudSqlReplicateTransactions(boolean replicateTransactions) {
+    CONFIG_SETTINGS.get().cloudSql.replicateTransactions = replicateTransactions;
+  }
+
  /** Returns the roid suffix to be used for the roids of all contacts and hosts. */
  public static String getContactAndHostRoidSuffix() {
    return CONFIG_SETTINGS.get().registryPolicy.contactAndHostRoidSuffix;
@@ -122,6 +122,7 @@ public class RegistryConfigSettings {
    public String jdbcUrl;
    public String username;
    public String instanceConnectionName;
+    public boolean replicateTransactions;
  }

  /** Configuration for Apache Beam (Cloud Dataflow). */
@@ -230,6 +230,9 @@ cloudSql:
  username: username
  # This name is used by Cloud SQL when connecting to the database.
  instanceConnectionName: project-id:region:instance-id
+  # Set this to true to replicate cloud SQL transactions to datastore in the
+  # background.
+  replicateTransactions: false

 cloudDns:
  # Set both properties to null in Production.
@@ -367,6 +367,12 @@
    <url-pattern>/_dr/task/linkRdeHosts</url-pattern>
  </servlet-mapping>

+  <!-- Action to automatically re-lock a domain after unlocking it -->
+  <servlet-mapping>
+    <servlet-name>backend-servlet</servlet-name>
+    <url-pattern>/_dr/task/relockDomain</url-pattern>
+  </servlet-mapping>
+
  <!-- Security config -->
  <security-constraint>
    <web-resource-collection>
@@ -7,7 +7,7 @@
  <sessions-enabled>true</sessions-enabled>
  <instance-class>B4_1G</instance-class>
  <manual-scaling>
-    <instances>10</instances>
+    <instances>20</instances>
  </manual-scaling>

  <system-properties>
@@ -18,7 +18,14 @@
      and streams it to cloud storage. When this job has finished successfully, it'll
      launch a separate task that uploads the deposit file to Iron Mountain via SFTP.
    </description>
-    <schedule>every day 00:07</schedule>
+    <!--
+      This only needs to run once per day, but we launch additional jobs in case the
+      cursor is lagging behind, so it'll catch up to the current date eventually.
+
+      See <a href="../../../production/default/WEB-INF/cron.xml">production config</a> for an
+      explanation of job starting times.
+    -->
+    <schedule>every 12 hours from 00:07 to 12:07</schedule>
    <target>backend</target>
  </cron>

@@ -347,8 +347,8 @@ public class DomainCreateFlow implements TransactionalFlow {
            .setRepoId(repoId)
            .setIdnTableName(validateDomainNameWithIdnTables(domainName))
            .setRegistrationExpirationTime(registrationExpirationTime)
-            .setAutorenewBillingEvent(Key.create(autorenewBillingEvent))
-            .setAutorenewPollMessage(Key.create(autorenewPollMessage))
+            .setAutorenewBillingEvent(autorenewBillingEvent.createVKey())
+            .setAutorenewPollMessage(autorenewPollMessage.createVKey())
            .setLaunchNotice(hasClaimsNotice ? launchCreate.get().getNotice() : null)
            .setSmdId(signedMarkId)
            .setDsData(secDnsCreate.isPresent() ? secDnsCreate.get().getDsData() : null)
@@ -31,7 +31,6 @@ import static google.registry.model.ResourceTransferUtils.handlePendingTransferO
 import static google.registry.model.ResourceTransferUtils.updateForeignKeyIndexDeletionTime;
 import static google.registry.model.eppoutput.Result.Code.SUCCESS;
 import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
-import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.model.reporting.DomainTransactionRecord.TransactionReportField.ADD_FIELDS;
 import static google.registry.model.reporting.DomainTransactionRecord.TransactionReportField.RENEW_FIELDS;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
@@ -209,7 +208,7 @@ public final class DomainDeleteFlow implements TransactionalFlow {
      PollMessage.OneTime deletePollMessage =
          createDeletePollMessage(existingDomain, historyEntry, deletionTime);
      entitiesToSave.add(deletePollMessage);
-      builder.setDeletePollMessage(Key.create(deletePollMessage));
+      builder.setDeletePollMessage(deletePollMessage.createVKey());
    }

    // Cancel any grace periods that were still active, and set the expiration time accordingly.
@@ -222,8 +221,7 @@ public final class DomainDeleteFlow implements TransactionalFlow {
        if (gracePeriod.getOneTimeBillingEvent() != null) {
          // Take the amount of amount of registration time being refunded off the expiration time.
          // This can be either add grace periods or renew grace periods.
-          BillingEvent.OneTime oneTime =
-              ofy().load().key(gracePeriod.getOneTimeBillingEvent()).now();
+          BillingEvent.OneTime oneTime = tm().load(gracePeriod.getOneTimeBillingEvent());
          newExpirationTime = newExpirationTime.minusYears(oneTime.getPeriodYears());
        } else if (gracePeriod.getRecurringBillingEvent() != null) {
          // Take 1 year off the registration if in the autorenew grace period (no need to load the
@@ -370,12 +368,12 @@ public final class DomainDeleteFlow implements TransactionalFlow {
  private Money getGracePeriodCost(GracePeriod gracePeriod, DateTime now) {
    if (gracePeriod.getType() == GracePeriodStatus.AUTO_RENEW) {
      DateTime autoRenewTime =
-          ofy().load().key(checkNotNull(gracePeriod.getRecurringBillingEvent())).now()
+          tm().load(checkNotNull(gracePeriod.getRecurringBillingEvent()))
              .getRecurrenceTimeOfYear()
-                  .getLastInstanceBeforeOrAt(now);
+              .getLastInstanceBeforeOrAt(now);
      return getDomainRenewCost(targetId, autoRenewTime, 1);
    }
-    return ofy().load().key(checkNotNull(gracePeriod.getOneTimeBillingEvent())).now().getCost();
+    return tm().load(checkNotNull(gracePeriod.getOneTimeBillingEvent())).getCost();
  }

  @Nullable
@@ -517,14 +517,14 @@ public class DomainFlowUtils {
   */
  public static void updateAutorenewRecurrenceEndTime(DomainBase domain, DateTime newEndTime) {
    Optional<PollMessage.Autorenew> autorenewPollMessage =
-        Optional.ofNullable(ofy().load().key(domain.getAutorenewPollMessage()).now());
+        tm().maybeLoad(domain.getAutorenewPollMessage());

    // Construct an updated autorenew poll message. If the autorenew poll message no longer exists,
    // create a new one at the same id. This can happen if a transfer was requested on a domain
    // where all autorenew poll messages had already been delivered (this would cause the poll
    // message to be deleted), and then subsequently the transfer was canceled, rejected, or deleted
    // (which would cause the poll message to be recreated here).
-    Key<PollMessage.Autorenew> existingAutorenewKey = domain.getAutorenewPollMessage();
+    Key<PollMessage.Autorenew> existingAutorenewKey = domain.getAutorenewPollMessage().getOfyKey();
    PollMessage.Autorenew updatedAutorenewPollMessage =
        autorenewPollMessage.isPresent()
            ? autorenewPollMessage.get().asBuilder().setAutorenewEndTime(newEndTime).build()
@@ -542,7 +542,7 @@ public class DomainFlowUtils {
      ofy().save().entity(updatedAutorenewPollMessage);
    }

-    Recurring recurring = ofy().load().key(domain.getAutorenewBillingEvent()).now();
+    Recurring recurring = tm().load(domain.getAutorenewBillingEvent());
    ofy().save().entity(recurring.asBuilder().setRecurrenceEndTime(newEndTime).build());
  }

@@ -33,7 +33,6 @@ import static google.registry.util.DateTimeUtils.leapSafeAddYears;

 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
-import com.googlecode.objectify.Key;
 import google.registry.flows.EppException;
 import google.registry.flows.EppException.ParameterValueRangeErrorException;
 import google.registry.flows.ExtensionManager;
@@ -181,8 +180,8 @@ public final class DomainRenewFlow implements TransactionalFlow {
            .setLastEppUpdateTime(now)
            .setLastEppUpdateClientId(clientId)
            .setRegistrationExpirationTime(newExpirationTime)
-            .setAutorenewBillingEvent(Key.create(newAutorenewEvent))
-            .setAutorenewPollMessage(Key.create(newAutorenewPollMessage))
+            .setAutorenewBillingEvent(newAutorenewEvent.createVKey())
+            .setAutorenewPollMessage(newAutorenewPollMessage.createVKey())
            .addGracePeriod(
                GracePeriod.forBillingEvent(GracePeriodStatus.RENEW, explicitRenewEvent))
            .build();
@@ -26,7 +26,6 @@ import static google.registry.flows.domain.DomainFlowUtils.verifyNotReserved;
 import static google.registry.flows.domain.DomainFlowUtils.verifyPremiumNameIsNotBlocked;
 import static google.registry.flows.domain.DomainFlowUtils.verifyRegistrarIsActive;
 import static google.registry.model.ResourceTransferUtils.updateForeignKeyIndexDeletionTime;
-import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;

@@ -174,8 +173,8 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow  {
            existingDomain, newExpirationTime, autorenewEvent, autorenewPollMessage, now, clientId);
    updateForeignKeyIndexDeletionTime(newDomain);
    entitiesToSave.add(newDomain, historyEntry, autorenewEvent, autorenewPollMessage);
-    ofy().save().entities(entitiesToSave.build());
-    ofy().delete().key(existingDomain.getDeletePollMessage());
+    tm().saveNewOrUpdateAll(entitiesToSave.build());
+    tm().delete(existingDomain.getDeletePollMessage());
    dnsQueue.addDomainRefreshTask(existingDomain.getDomainName());
    return responseBuilder
        .setExtensions(createResponseExtensions(feesAndCredits, feeUpdate, isExpired))
@@ -232,8 +231,8 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow  {
        .setStatusValues(null)
        .setGracePeriods(null)
        .setDeletePollMessage(null)
-        .setAutorenewBillingEvent(Key.create(autorenewEvent))
-        .setAutorenewPollMessage(Key.create(autorenewPollMessage))
+        .setAutorenewBillingEvent(autorenewEvent.createVKey())
+        .setAutorenewPollMessage(autorenewPollMessage.createVKey())
        .setLastEppUpdateTime(now)
        .setLastEppUpdateClientId(clientId)
        .build();
@@ -186,8 +186,8 @@ public final class DomainTransferApproveFlow implements TransactionalFlow {
                    .setTransferredRegistrationExpirationTime(newExpirationTime)
                    .build())
            .setRegistrationExpirationTime(newExpirationTime)
-            .setAutorenewBillingEvent(Key.create(autorenewEvent))
-            .setAutorenewPollMessage(Key.create(gainingClientAutorenewPollMessage))
+            .setAutorenewBillingEvent(autorenewEvent.createVKey())
+            .setAutorenewPollMessage(gainingClientAutorenewPollMessage.createVKey())
            // Remove all the old grace periods and add a new one for the transfer.
            .setGracePeriods(
                billingEvent.isPresent()
@@ -14,8 +14,8 @@

 package google.registry.flows.poll;

-import static com.google.common.base.Preconditions.checkState;
 import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
+import static google.registry.flows.poll.PollFlowUtils.ackPollMessage;
 import static google.registry.flows.poll.PollFlowUtils.getPollMessagesQuery;
 import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_NO_MESSAGES;
 import static google.registry.model.ofy.ObjectifyService.ofy;
@@ -100,27 +100,8 @@ public class PollAckFlow implements TransactionalFlow {
    // This keeps track of whether we should include the current acked message in the updated
    // message count that's returned to the user. The only case where we do so is if an autorenew
    // poll message is acked, but its next event is already ready to be delivered.
-    boolean includeAckedMessageInCount = false;
-    if (pollMessage instanceof PollMessage.OneTime) {
-      // One-time poll messages are deleted once acked.
-      ofy().delete().entity(pollMessage);
-    } else {
-      checkState(pollMessage instanceof PollMessage.Autorenew, "Unknown poll message type");
-      PollMessage.Autorenew autorenewPollMessage = (PollMessage.Autorenew) pollMessage;
+    boolean includeAckedMessageInCount = ackPollMessage(pollMessage);

-      // Move the eventTime of this autorenew poll message forward by a year.
-      DateTime nextEventTime = autorenewPollMessage.getEventTime().plusYears(1);
-
-      // If the next event falls within the bounds of the end time, then just update the eventTime
-      // and re-save it for future autorenew poll messages to be delivered. Otherwise, this
-      // autorenew poll message has no more events to deliver and should be deleted.
-      if (nextEventTime.isBefore(autorenewPollMessage.getAutorenewEndTime())) {
-        ofy().save().entity(autorenewPollMessage.asBuilder().setEventTime(nextEventTime).build());
-        includeAckedMessageInCount = isBeforeOrAt(nextEventTime, now);
-      } else {
-        ofy().delete().entity(autorenewPollMessage);
-      }
-    }
    // We need to return the new queue length. If this was the last message in the queue being
    // acked, then we return a special status code indicating that. Note that the query will
    // include the message being acked.
@@ -14,7 +14,10 @@

 package google.registry.flows.poll;

+import static com.google.common.base.Preconditions.checkArgument;
 import static google.registry.model.ofy.ObjectifyService.ofy;
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
+import static google.registry.util.DateTimeUtils.isBeforeOrAt;

 import com.googlecode.objectify.cmd.Query;
 import google.registry.model.poll.PollMessage;
@@ -33,4 +36,42 @@ public final class PollFlowUtils {
        .filter("eventTime <=", now.toDate())
        .order("eventTime");
  }
+
+  /**
+   * Acknowledges the given {@link PollMessage} and returns whether we should include the current
+   * acked message in the updated message count that's returned to the user.
+   *
+   * <p>The only case where we do so is if an autorenew poll message is acked, but its next event is
+   * already ready to be delivered.
+   */
+  public static boolean ackPollMessage(PollMessage pollMessage) {
+    checkArgument(
+        isBeforeOrAt(pollMessage.getEventTime(), tm().getTransactionTime()),
+        "Cannot ACK poll message with ID %s because its event time is in the future: %s",
+        pollMessage.getId(),
+        pollMessage.getEventTime());
+    boolean includeAckedMessageInCount = false;
+    if (pollMessage instanceof PollMessage.OneTime) {
+      // One-time poll messages are deleted once acked.
+      tm().delete(pollMessage.createVKey());
+    } else if (pollMessage instanceof PollMessage.Autorenew) {
+      PollMessage.Autorenew autorenewPollMessage = (PollMessage.Autorenew) pollMessage;
+
+      // Move the eventTime of this autorenew poll message forward by a year.
+      DateTime nextEventTime = autorenewPollMessage.getEventTime().plusYears(1);
+
+      // If the next event falls within the bounds of the end time, then just update the eventTime
+      // and re-save it for future autorenew poll messages to be delivered. Otherwise, this
+      // autorenew poll message has no more events to deliver and should be deleted.
+      if (nextEventTime.isBefore(autorenewPollMessage.getAutorenewEndTime())) {
+        tm().saveNewOrUpdate(autorenewPollMessage.asBuilder().setEventTime(nextEventTime).build());
+        includeAckedMessageInCount = isBeforeOrAt(nextEventTime, tm().getTransactionTime());
+      } else {
+        tm().delete(autorenewPollMessage.createVKey());
+      }
+    } else {
+      throw new IllegalArgumentException("Unknown poll message type: " + pollMessage.getClass());
+    }
+    return includeAckedMessageInCount;
+  }
 }
@@ -25,11 +25,9 @@ import com.google.api.services.cloudkms.v1.model.DecryptRequest;
 import com.google.api.services.cloudkms.v1.model.EncryptRequest;
 import com.google.api.services.cloudkms.v1.model.KeyRing;
 import com.google.api.services.cloudkms.v1.model.UpdateCryptoKeyPrimaryVersionRequest;
-import google.registry.config.RegistryConfig.Config;
 import google.registry.keyring.api.KeyringException;
 import google.registry.util.Retrier;
 import java.io.IOException;
-import javax.inject.Inject;

 /** The {@link KmsConnection} which talks to Cloud KMS. */
 class KmsConnectionImpl implements KmsConnection {
@@ -44,12 +42,7 @@ class KmsConnectionImpl implements KmsConnection {
  private final String projectId;
  private final Retrier retrier;

-  @Inject
-  KmsConnectionImpl(
-      @Config("cloudKmsProjectId") String projectId,
-      @Config("cloudKmsKeyRing") String kmsKeyRingName,
-      Retrier retrier,
-      CloudKMS kms) {
+  KmsConnectionImpl(String projectId, String kmsKeyRingName, Retrier retrier, CloudKMS kms) {
    this.projectId = projectId;
    this.kmsKeyRingName = kmsKeyRingName;
    this.retrier = retrier;
@@ -21,6 +21,7 @@ import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
 import static google.registry.model.ofy.ObjectifyService.ofy;

 import com.googlecode.objectify.Key;
+import google.registry.config.RegistryConfig.Config;
 import google.registry.keyring.api.KeySerializer;
 import google.registry.keyring.api.Keyring;
 import google.registry.keyring.api.KeyringException;
@@ -86,7 +87,7 @@ public class KmsKeyring implements Keyring {
  private final KmsConnection kmsConnection;

  @Inject
-  KmsKeyring(KmsConnection kmsConnection) {
+  KmsKeyring(@Config("defaultKmsConnection") KmsConnection kmsConnection) {
    this.kmsConnection = kmsConnection;
  }

@@ -24,6 +24,7 @@ import google.registry.config.CredentialModule.DefaultCredential;
 import google.registry.config.RegistryConfig.Config;
 import google.registry.keyring.api.Keyring;
 import google.registry.util.GoogleCredentialsBundle;
+import google.registry.util.Retrier;

 /** Dagger module for Cloud KMS. */
@Module
@@ -32,9 +33,22 @@ public abstract class KmsModule {
  public static final String NAME = "KMS";

  @Provides
+  @Config("defaultKms")
  static CloudKMS provideKms(
      @DefaultCredential GoogleCredentialsBundle credentialsBundle,
      @Config("cloudKmsProjectId") String projectId) {
+    return createKms(credentialsBundle, projectId);
+  }
+
+  @Provides
+  @Config("beamKms")
+  static CloudKMS provideBeamKms(
+      @DefaultCredential GoogleCredentialsBundle credentialsBundle,
+      @Config("beamCloudKmsProjectId") String projectId) {
+    return createKms(credentialsBundle, projectId);
+  }
+
+  private static CloudKMS createKms(GoogleCredentialsBundle credentialsBundle, String projectId) {
    return new CloudKMS.Builder(
            credentialsBundle.getHttpTransport(),
            credentialsBundle.getJsonFactory(),
@@ -43,11 +57,28 @@ public abstract class KmsModule {
        .build();
  }

+  @Provides
+  @Config("defaultKmsConnection")
+  static KmsConnection provideKmsConnection(
+      @Config("cloudKmsProjectId") String projectId,
+      @Config("cloudKmsKeyRing") String keyringName,
+      Retrier retrier,
+      @Config("defaultKms") CloudKMS defaultKms) {
+    return new KmsConnectionImpl(projectId, keyringName, retrier, defaultKms);
+  }
+
+  @Provides
+  @Config("beamKmsConnection")
+  static KmsConnection provideBeamKmsConnection(
+      @Config("beamCloudKmsProjectId") String projectId,
+      @Config("beamCloudKmsKeyRing") String keyringName,
+      Retrier retrier,
+      @Config("beamKms") CloudKMS defaultKms) {
+    return new KmsConnectionImpl(projectId, keyringName, retrier, defaultKms);
+  }
+
  @Binds
  @IntoMap
  @StringKey(NAME)
  abstract Keyring provideKeyring(KmsKeyring keyring);
-
-  @Binds
-  abstract KmsConnection provideKmsConnection(KmsConnectionImpl kmsConnectionImpl);
 }
@@ -39,6 +39,7 @@ import static google.registry.persistence.transaction.TransactionManagerFactory.
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

 import com.google.common.collect.ImmutableMap;
+import google.registry.config.RegistryConfig.Config;
 import google.registry.keyring.api.KeySerializer;
 import google.registry.keyring.kms.KmsKeyring.PrivateKeyLabel;
 import google.registry.keyring.kms.KmsKeyring.PublicKeyLabel;
@@ -64,7 +65,7 @@ public final class KmsUpdater {
  private final HashMap<String, byte[]> secretValues;

  @Inject
-  public KmsUpdater(KmsConnection kmsConnection) {
+  public KmsUpdater(@Config("defaultKmsConnection") KmsConnection kmsConnection) {
    this.kmsConnection = kmsConnection;

    // Use LinkedHashMap to preserve insertion order on update() to simplify testing and debugging
@@ -14,16 +14,21 @@

 package google.registry.model;

+import com.google.common.annotations.VisibleForTesting;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.MappedSuperclass;
 import javax.xml.bind.annotation.XmlTransient;

 /**
 * Base class for entities that are the root of a Registry 2.0 entity group that gets enrolled in
 * commit logs for backup purposes.
 *
- * <p>The commit log system needs to preserve the ordering of closely timed mutations to entities
- * in a single entity group. We require an {@link UpdateAutoTimestamp} field on the root of a group
- * so that we can enforce strictly increasing timestamps.
+ * <p>The commit log system needs to preserve the ordering of closely timed mutations to entities in
+ * a single entity group. We require an {@link UpdateAutoTimestamp} field on the root of a group so
+ * that we can enforce strictly increasing timestamps.
 */
+@MappedSuperclass
 public abstract class BackupGroupRoot extends ImmutableObject {
  /**
   * An automatically managed timestamp of when this object was last written to Datastore.
@@ -32,10 +37,14 @@ public abstract class BackupGroupRoot extends ImmutableObject {
   * that this is updated on every save, rather than only in response to an {@code <update>} command
   */
  @XmlTransient
+  // Prevents subclasses from unexpectedly accessing as property (e.g., HostResource), which would
+  // require an unnecessary non-private setter method.
+  @Access(AccessType.FIELD)
+  @VisibleForTesting
  UpdateAutoTimestamp updateTimestamp = UpdateAutoTimestamp.create(null);

  /** Get the {@link UpdateAutoTimestamp} for this entity. */
-  public final UpdateAutoTimestamp getUpdateAutoTimestamp() {
+  public UpdateAutoTimestamp getUpdateTimestamp() {
    return updateTimestamp;
  }
 }
@@ -155,7 +155,7 @@ public final class EppResourceUtils {
    // time for writes.
    return Optional.of(
        cloneProjectedAtTime(
-            resource, latestOf(now, resource.getUpdateAutoTimestamp().getTimestamp())));
+            resource, latestOf(now, resource.getUpdateTimestamp().getTimestamp())));
  }

  /**
@@ -298,7 +298,7 @@ public final class EppResourceUtils {
    // and returns it projected forward to exactly the desired timestamp, or null if the resource is
    // deleted at that timestamp.
    final Result<T> loadResult =
-        isAtOrAfter(timestamp, resource.getUpdateAutoTimestamp().getTimestamp())
+        isAtOrAfter(timestamp, resource.getUpdateTimestamp().getTimestamp())
            ? new ResultNow<>(resource)
            : loadMostRecentRevisionAtTime(resource, timestamp);
    return () -> {
@@ -630,9 +630,9 @@ public abstract class BillingEvent extends ImmutableObject
          .setParent(historyEntry);
      // Set the grace period's billing event using the appropriate Cancellation builder method.
      if (gracePeriod.getOneTimeBillingEvent() != null) {
-        builder.setOneTimeEventKey(VKey.from(gracePeriod.getOneTimeBillingEvent()));
+        builder.setOneTimeEventKey(gracePeriod.getOneTimeBillingEvent());
      } else if (gracePeriod.getRecurringBillingEvent() != null) {
-        builder.setRecurringEventKey(VKey.from(gracePeriod.getRecurringBillingEvent()));
+        builder.setRecurringEventKey(gracePeriod.getRecurringBillingEvent());
      }
      return builder.build();
    }
@@ -0,0 +1,398 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.contact;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static google.registry.model.EppResourceUtils.projectResourceOntoBuilderAtTime;
+
+import com.google.common.collect.ImmutableList;
+import com.googlecode.objectify.Key;
+import com.googlecode.objectify.annotation.IgnoreSave;
+import com.googlecode.objectify.annotation.Index;
+import com.googlecode.objectify.condition.IfNull;
+import google.registry.model.EppResource;
+import google.registry.model.EppResource.ResourceWithTransferData;
+import google.registry.model.transfer.ContactTransferData;
+import google.registry.persistence.VKey;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Stream;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.AttributeOverride;
+import javax.persistence.AttributeOverrides;
+import javax.persistence.Column;
+import javax.persistence.Embeddable;
+import javax.persistence.Embedded;
+import javax.persistence.MappedSuperclass;
+import javax.xml.bind.annotation.XmlElement;
+import org.joda.time.DateTime;
+
+/**
+ * A persistable contact resource including mutable and non-mutable fields.
+ *
+ * @see <a href="https://tools.ietf.org/html/rfc5733">RFC 5733</a>
+ *     <p>This class deliberately does not include an {@link javax.persistence.Id} so that any
+ *     foreign-keyed fields can refer to the proper parent entity's ID, whether we're storing this
+ *     in the DB itself or as part of another entity
+ */
+@MappedSuperclass
+@Embeddable
+@Access(AccessType.FIELD)
+public class ContactBase extends EppResource implements ResourceWithTransferData {
+
+  /**
+   * Unique identifier for this contact.
+   *
+   * <p>This is only unique in the sense that for any given lifetime specified as the time range
+   * from (creationTime, deletionTime) there can only be one contact in Datastore with this id.
+   * However, there can be many contacts with the same id and non-overlapping lifetimes.
+   */
+  String contactId;
+
+  /**
+   * Localized postal info for the contact. All contained values must be representable in the 7-bit
+   * US-ASCII character set. Personal info; cleared by {@link ContactResource.Builder#wipeOut}.
+   */
+  @IgnoreSave(IfNull.class)
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "name", column = @Column(name = "addr_local_name")),
+    @AttributeOverride(name = "org", column = @Column(name = "addr_local_org")),
+    @AttributeOverride(name = "type", column = @Column(name = "addr_local_type")),
+    @AttributeOverride(
+        name = "address.streetLine1",
+        column = @Column(name = "addr_local_street_line1")),
+    @AttributeOverride(
+        name = "address.streetLine2",
+        column = @Column(name = "addr_local_street_line2")),
+    @AttributeOverride(
+        name = "address.streetLine3",
+        column = @Column(name = "addr_local_street_line3")),
+    @AttributeOverride(name = "address.city", column = @Column(name = "addr_local_city")),
+    @AttributeOverride(name = "address.state", column = @Column(name = "addr_local_state")),
+    @AttributeOverride(name = "address.zip", column = @Column(name = "addr_local_zip")),
+    @AttributeOverride(
+        name = "address.countryCode",
+        column = @Column(name = "addr_local_country_code"))
+  })
+  PostalInfo localizedPostalInfo;
+
+  /**
+   * Internationalized postal info for the contact. Personal info; cleared by {@link
+   * ContactResource.Builder#wipeOut}.
+   */
+  @IgnoreSave(IfNull.class)
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "name", column = @Column(name = "addr_i18n_name")),
+    @AttributeOverride(name = "org", column = @Column(name = "addr_i18n_org")),
+    @AttributeOverride(name = "type", column = @Column(name = "addr_i18n_type")),
+    @AttributeOverride(
+        name = "address.streetLine1",
+        column = @Column(name = "addr_i18n_street_line1")),
+    @AttributeOverride(
+        name = "address.streetLine2",
+        column = @Column(name = "addr_i18n_street_line2")),
+    @AttributeOverride(
+        name = "address.streetLine3",
+        column = @Column(name = "addr_i18n_street_line3")),
+    @AttributeOverride(name = "address.city", column = @Column(name = "addr_i18n_city")),
+    @AttributeOverride(name = "address.state", column = @Column(name = "addr_i18n_state")),
+    @AttributeOverride(name = "address.zip", column = @Column(name = "addr_i18n_zip")),
+    @AttributeOverride(
+        name = "address.countryCode",
+        column = @Column(name = "addr_i18n_country_code"))
+  })
+  PostalInfo internationalizedPostalInfo;
+
+  /**
+   * Contact name used for name searches. This is set automatically to be the internationalized
+   * postal name, or if null, the localized postal name, or if that is null as well, null. Personal
+   * info; cleared by {@link ContactResource.Builder#wipeOut}.
+   */
+  @Index String searchName;
+
+  /** Contact’s voice number. Personal info; cleared by {@link ContactResource.Builder#wipeOut}. */
+  @IgnoreSave(IfNull.class)
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "phoneNumber", column = @Column(name = "voice_phone_number")),
+    @AttributeOverride(name = "extension", column = @Column(name = "voice_phone_extension")),
+  })
+  ContactPhoneNumber voice;
+
+  /** Contact’s fax number. Personal info; cleared by {@link ContactResource.Builder#wipeOut}. */
+  @IgnoreSave(IfNull.class)
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "phoneNumber", column = @Column(name = "fax_phone_number")),
+    @AttributeOverride(name = "extension", column = @Column(name = "fax_phone_extension")),
+  })
+  ContactPhoneNumber fax;
+
+  /** Contact’s email address. Personal info; cleared by {@link ContactResource.Builder#wipeOut}. */
+  @IgnoreSave(IfNull.class)
+  String email;
+
+  /** Authorization info (aka transfer secret) of the contact. */
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "pw.value", column = @Column(name = "auth_info_value")),
+    @AttributeOverride(name = "pw.repoId", column = @Column(name = "auth_info_repo_id")),
+  })
+  ContactAuthInfo authInfo;
+
+  /** Data about any pending or past transfers on this contact. */
+  ContactTransferData transferData;
+
+  /**
+   * The time that this resource was last transferred.
+   *
+   * <p>Can be null if the resource has never been transferred.
+   */
+  DateTime lastTransferTime;
+
+  // If any new fields are added which contain personal information, make sure they are cleared by
+  // the wipeOut() function, so that data is not kept around for deleted contacts.
+
+  /** Disclosure policy. */
+  @Embedded
+  @AttributeOverrides({
+    @AttributeOverride(name = "name", column = @Column(name = "disclose_types_name")),
+    @AttributeOverride(name = "org", column = @Column(name = "disclose_types_org")),
+    @AttributeOverride(name = "addr", column = @Column(name = "disclose_types_addr")),
+    @AttributeOverride(name = "flag", column = @Column(name = "disclose_mode_flag")),
+    @AttributeOverride(name = "voice.marked", column = @Column(name = "disclose_show_voice")),
+    @AttributeOverride(name = "fax.marked", column = @Column(name = "disclose_show_fax")),
+    @AttributeOverride(name = "email.marked", column = @Column(name = "disclose_show_email"))
+  })
+  Disclose disclose;
+
+  @Override
+  public VKey<? extends ContactBase> createVKey() {
+    // TODO(mmuller): create symmetric keys if we can ever reload both sides.
+    return VKey.create(ContactBase.class, getRepoId(), Key.create(this));
+  }
+
+  public String getContactId() {
+    return contactId;
+  }
+
+  public PostalInfo getLocalizedPostalInfo() {
+    return localizedPostalInfo;
+  }
+
+  public PostalInfo getInternationalizedPostalInfo() {
+    return internationalizedPostalInfo;
+  }
+
+  public String getSearchName() {
+    return searchName;
+  }
+
+  public ContactPhoneNumber getVoiceNumber() {
+    return voice;
+  }
+
+  public ContactPhoneNumber getFaxNumber() {
+    return fax;
+  }
+
+  public String getEmailAddress() {
+    return email;
+  }
+
+  public ContactAuthInfo getAuthInfo() {
+    return authInfo;
+  }
+
+  public Disclose getDisclose() {
+    return disclose;
+  }
+
+  public String getCurrentSponsorClientId() {
+    return getPersistedCurrentSponsorClientId();
+  }
+
+  @Override
+  public ContactTransferData getTransferData() {
+    return Optional.ofNullable(transferData).orElse(ContactTransferData.EMPTY);
+  }
+
+  @Override
+  public DateTime getLastTransferTime() {
+    return lastTransferTime;
+  }
+
+  @Override
+  public String getForeignKey() {
+    return contactId;
+  }
+
+  /**
+   * Postal info for the contact.
+   *
+   * <p>The XML marshalling expects the {@link PostalInfo} objects in a list, but we can't actually
+   * persist them to Datastore that way because Objectify can't handle collections of embedded
+   * objects that themselves contain collections, and there's a list of streets inside. This method
+   * transforms the persisted format to the XML format for marshalling.
+   */
+  @XmlElement(name = "postalInfo")
+  public ImmutableList<PostalInfo> getPostalInfosAsList() {
+    return Stream.of(localizedPostalInfo, internationalizedPostalInfo)
+        .filter(Objects::nonNull)
+        .collect(toImmutableList());
+  }
+
+  @Override
+  public ContactBase cloneProjectedAtTime(DateTime now) {
+    return cloneContactProjectedAtTime(this, now);
+  }
+
+  /**
+   * Clones the contact (or subclass). A separate static method so that we can pass in and return a
+   * T without the compiler complaining.
+   */
+  protected static <T extends ContactBase> T cloneContactProjectedAtTime(T contact, DateTime now) {
+    Builder builder = contact.asBuilder();
+    projectResourceOntoBuilderAtTime(contact, builder, now);
+    return (T) builder.build();
+  }
+
+  @Override
+  public Builder asBuilder() {
+    return new Builder<>(clone(this));
+  }
+
+  /** A builder for constructing {@link ContactResource}, since it is immutable. */
+  public static class Builder<T extends ContactBase, B extends Builder<T, B>>
+      extends EppResource.Builder<T, B> implements BuilderWithTransferData<ContactTransferData, B> {
+
+    public Builder() {}
+
+    protected Builder(T instance) {
+      super(instance);
+    }
+
+    public B setContactId(String contactId) {
+      getInstance().contactId = contactId;
+      return thisCastToDerived();
+    }
+
+    public B setLocalizedPostalInfo(PostalInfo localizedPostalInfo) {
+      checkArgument(
+          localizedPostalInfo == null
+              || PostalInfo.Type.LOCALIZED.equals(localizedPostalInfo.getType()));
+      getInstance().localizedPostalInfo = localizedPostalInfo;
+      return thisCastToDerived();
+    }
+
+    public B setInternationalizedPostalInfo(PostalInfo internationalizedPostalInfo) {
+      checkArgument(
+          internationalizedPostalInfo == null
+              || PostalInfo.Type.INTERNATIONALIZED.equals(internationalizedPostalInfo.getType()));
+      getInstance().internationalizedPostalInfo = internationalizedPostalInfo;
+      return thisCastToDerived();
+    }
+
+    public B overlayLocalizedPostalInfo(PostalInfo localizedPostalInfo) {
+      return setLocalizedPostalInfo(
+          getInstance().localizedPostalInfo == null
+              ? localizedPostalInfo
+              : getInstance().localizedPostalInfo.overlay(localizedPostalInfo));
+    }
+
+    public B overlayInternationalizedPostalInfo(PostalInfo internationalizedPostalInfo) {
+      return setInternationalizedPostalInfo(
+          getInstance().internationalizedPostalInfo == null
+              ? internationalizedPostalInfo
+              : getInstance().internationalizedPostalInfo.overlay(internationalizedPostalInfo));
+    }
+
+    public B setVoiceNumber(ContactPhoneNumber voiceNumber) {
+      getInstance().voice = voiceNumber;
+      return thisCastToDerived();
+    }
+
+    public B setFaxNumber(ContactPhoneNumber faxNumber) {
+      getInstance().fax = faxNumber;
+      return thisCastToDerived();
+    }
+
+    public B setEmailAddress(String emailAddress) {
+      getInstance().email = emailAddress;
+      return thisCastToDerived();
+    }
+
+    public B setAuthInfo(ContactAuthInfo authInfo) {
+      getInstance().authInfo = authInfo;
+      return thisCastToDerived();
+    }
+
+    public B setDisclose(Disclose disclose) {
+      getInstance().disclose = disclose;
+      return thisCastToDerived();
+    }
+
+    @Override
+    public B setTransferData(ContactTransferData transferData) {
+      getInstance().transferData = transferData;
+      return thisCastToDerived();
+    }
+
+    @Override
+    public B setLastTransferTime(DateTime lastTransferTime) {
+      getInstance().lastTransferTime = lastTransferTime;
+      return thisCastToDerived();
+    }
+
+    /**
+     * Remove all personally identifying information about a contact.
+     *
+     * <p>This should be used when deleting a contact so that the soft-deleted entity doesn't
+     * contain information that the registrant requested to be deleted.
+     */
+    public B wipeOut() {
+      setEmailAddress(null);
+      setFaxNumber(null);
+      setInternationalizedPostalInfo(null);
+      setLocalizedPostalInfo(null);
+      setVoiceNumber(null);
+      return thisCastToDerived();
+    }
+
+    @Override
+    public T build() {
+      T instance = getInstance();
+      // If TransferData is totally empty, set it to null.
+      if (ContactTransferData.EMPTY.equals(instance.transferData)) {
+        setTransferData(null);
+      }
+      // Set the searchName using the internationalized and localized postal info names.
+      if ((instance.internationalizedPostalInfo != null)
+          && (instance.internationalizedPostalInfo.getName() != null)) {
+        instance.searchName = instance.internationalizedPostalInfo.getName();
+      } else if ((instance.localizedPostalInfo != null)
+          && (instance.localizedPostalInfo.getName() != null)) {
+        instance.searchName = instance.localizedPostalInfo.getName();
+      } else {
+        instance.searchName = null;
+      }
+      return super.build();
+    }
+  }
+}
@@ -0,0 +1,88 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.contact;
+
+import com.googlecode.objectify.Key;
+import google.registry.model.EppResource;
+import google.registry.model.reporting.HistoryEntry;
+import google.registry.persistence.VKey;
+import javax.persistence.Column;
+import javax.persistence.Entity;
+
+/**
+ * A persisted history entry representing an EPP modification to a contact.
+ *
+ * <p>In addition to the general history fields (e.g. action time, registrar ID) we also persist a
+ * copy of the host entity at this point in time. We persist a raw {@link ContactBase} so that the
+ * foreign-keyed fields in that class can refer to this object.
+ */
+@Entity
+@javax.persistence.Table(
+    indexes = {
+      @javax.persistence.Index(columnList = "creationTime"),
+      @javax.persistence.Index(columnList = "historyRegistrarId"),
+      @javax.persistence.Index(columnList = "historyType"),
+      @javax.persistence.Index(columnList = "historyModificationTime")
+    })
+public class ContactHistory extends HistoryEntry {
+  // Store ContactBase instead of ContactResource so we don't pick up its @Id
+  ContactBase contactBase;
+
+  @Column(nullable = false)
+  VKey<ContactResource> contactRepoId;
+
+  /** The state of the {@link ContactBase} object at this point in time. */
+  public ContactBase getContactBase() {
+    return contactBase;
+  }
+
+  /** The key to the {@link ContactResource} this is based off of. */
+  public VKey<ContactResource> getContactRepoId() {
+    return contactRepoId;
+  }
+
+  @Override
+  public Builder asBuilder() {
+    return new Builder(clone(this));
+  }
+
+  public static class Builder extends HistoryEntry.Builder<ContactHistory, ContactHistory.Builder> {
+
+    public Builder() {}
+
+    public Builder(ContactHistory instance) {
+      super(instance);
+    }
+
+    public Builder setContactBase(ContactBase contactBase) {
+      getInstance().contactBase = contactBase;
+      return this;
+    }
+
+    public Builder setContactRepoId(VKey<ContactResource> contactRepoId) {
+      getInstance().contactRepoId = contactRepoId;
+      contactRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
+      return this;
+    }
+
+    // We can remove this once all HistoryEntries are converted to History objects
+    @Override
+    public Builder setParent(Key<? extends EppResource> parent) {
+      super.setParent(parent);
+      getInstance().contactRepoId = VKey.create(ContactResource.class, parent.getName(), parent);
+      return this;
+    }
+  }
+}
@@ -14,36 +14,16 @@

 package google.registry.model.contact;

-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.collect.ImmutableList.toImmutableList;
-import static google.registry.model.EppResourceUtils.projectResourceOntoBuilderAtTime;
-
-import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
-import com.googlecode.objectify.annotation.IgnoreSave;
-import com.googlecode.objectify.annotation.Index;
-import com.googlecode.objectify.condition.IfNull;
-import google.registry.model.EppResource;
 import google.registry.model.EppResource.ForeignKeyedEppResource;
-import google.registry.model.EppResource.ResourceWithTransferData;
 import google.registry.model.annotations.ExternalMessagingName;
 import google.registry.model.annotations.ReportedOn;
-import google.registry.model.contact.PostalInfo.Type;
-import google.registry.model.transfer.ContactTransferData;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithStringVKey;
 import google.registry.schema.replay.DatastoreAndSqlEntity;
-import java.util.Objects;
-import java.util.Optional;
-import java.util.stream.Stream;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
-import javax.persistence.AttributeOverride;
-import javax.persistence.AttributeOverrides;
-import javax.persistence.Column;
-import javax.persistence.Embedded;
-import javax.xml.bind.annotation.XmlElement;
 import org.joda.time.DateTime;

 /**
@@ -66,140 +46,12 @@ import org.joda.time.DateTime;
@ExternalMessagingName("contact")
@WithStringVKey
@Access(AccessType.FIELD)
-public class ContactResource extends EppResource
-    implements DatastoreAndSqlEntity, ForeignKeyedEppResource, ResourceWithTransferData {
-
-  /**
-   * Unique identifier for this contact.
-   *
-   * <p>This is only unique in the sense that for any given lifetime specified as the time range
-   * from (creationTime, deletionTime) there can only be one contact in Datastore with this id.
-   * However, there can be many contacts with the same id and non-overlapping lifetimes.
-   */
-  String contactId;
-
-  /**
-   * Localized postal info for the contact. All contained values must be representable in the 7-bit
-   * US-ASCII character set. Personal info; cleared by {@link Builder#wipeOut}.
-   */
-  @IgnoreSave(IfNull.class)
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "name", column = @Column(name = "addr_local_name")),
-    @AttributeOverride(name = "org", column = @Column(name = "addr_local_org")),
-    @AttributeOverride(name = "type", column = @Column(name = "addr_local_type")),
-    @AttributeOverride(
-        name = "address.streetLine1",
-        column = @Column(name = "addr_local_street_line1")),
-    @AttributeOverride(
-        name = "address.streetLine2",
-        column = @Column(name = "addr_local_street_line2")),
-    @AttributeOverride(
-        name = "address.streetLine3",
-        column = @Column(name = "addr_local_street_line3")),
-    @AttributeOverride(name = "address.city", column = @Column(name = "addr_local_city")),
-    @AttributeOverride(name = "address.state", column = @Column(name = "addr_local_state")),
-    @AttributeOverride(name = "address.zip", column = @Column(name = "addr_local_zip")),
-    @AttributeOverride(
-        name = "address.countryCode",
-        column = @Column(name = "addr_local_country_code"))
-  })
-  PostalInfo localizedPostalInfo;
-
-  /**
-   * Internationalized postal info for the contact. Personal info; cleared by {@link
-   * Builder#wipeOut}.
-   */
-  @IgnoreSave(IfNull.class)
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "name", column = @Column(name = "addr_i18n_name")),
-    @AttributeOverride(name = "org", column = @Column(name = "addr_i18n_org")),
-    @AttributeOverride(name = "type", column = @Column(name = "addr_i18n_type")),
-    @AttributeOverride(
-        name = "address.streetLine1",
-        column = @Column(name = "addr_i18n_street_line1")),
-    @AttributeOverride(
-        name = "address.streetLine2",
-        column = @Column(name = "addr_i18n_street_line2")),
-    @AttributeOverride(
-        name = "address.streetLine3",
-        column = @Column(name = "addr_i18n_street_line3")),
-    @AttributeOverride(name = "address.city", column = @Column(name = "addr_i18n_city")),
-    @AttributeOverride(name = "address.state", column = @Column(name = "addr_i18n_state")),
-    @AttributeOverride(name = "address.zip", column = @Column(name = "addr_i18n_zip")),
-    @AttributeOverride(
-        name = "address.countryCode",
-        column = @Column(name = "addr_i18n_country_code"))
-  })
-  PostalInfo internationalizedPostalInfo;
-
-  /**
-   * Contact name used for name searches. This is set automatically to be the internationalized
-   * postal name, or if null, the localized postal name, or if that is null as well, null. Personal
-   * info; cleared by {@link Builder#wipeOut}.
-   */
-  @Index
-  String searchName;
-
-  /** Contact’s voice number. Personal info; cleared by {@link Builder#wipeOut}. */
-  @IgnoreSave(IfNull.class)
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "phoneNumber", column = @Column(name = "voice_phone_number")),
-    @AttributeOverride(name = "extension", column = @Column(name = "voice_phone_extension")),
-  })
-  ContactPhoneNumber voice;
-
-  /** Contact’s fax number. Personal info; cleared by {@link Builder#wipeOut}. */
-  @IgnoreSave(IfNull.class)
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "phoneNumber", column = @Column(name = "fax_phone_number")),
-    @AttributeOverride(name = "extension", column = @Column(name = "fax_phone_extension")),
-  })
-  ContactPhoneNumber fax;
-
-  /** Contact’s email address. Personal info; cleared by {@link Builder#wipeOut}. */
-  @IgnoreSave(IfNull.class)
-  String email;
-
-  /** Authorization info (aka transfer secret) of the contact. */
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "pw.value", column = @Column(name = "auth_info_value")),
-    @AttributeOverride(name = "pw.repoId", column = @Column(name = "auth_info_repo_id")),
-  })
-  ContactAuthInfo authInfo;
-
-  /** Data about any pending or past transfers on this contact. */
-  ContactTransferData transferData;
-
-  /**
-   * The time that this resource was last transferred.
-   *
-   * <p>Can be null if the resource has never been transferred.
-   */
-  DateTime lastTransferTime;
-
-  // If any new fields are added which contain personal information, make sure they are cleared by
-  // the wipeOut() function, so that data is not kept around for deleted contacts.
-
-  /** Disclosure policy. */
-  @Embedded
-  @AttributeOverrides({
-    @AttributeOverride(name = "name", column = @Column(name = "disclose_types_name")),
-    @AttributeOverride(name = "org", column = @Column(name = "disclose_types_org")),
-    @AttributeOverride(name = "addr", column = @Column(name = "disclose_types_addr")),
-    @AttributeOverride(name = "flag", column = @Column(name = "disclose_mode_flag")),
-    @AttributeOverride(name = "voice.marked", column = @Column(name = "disclose_show_voice")),
-    @AttributeOverride(name = "fax.marked", column = @Column(name = "disclose_show_fax")),
-    @AttributeOverride(name = "email.marked", column = @Column(name = "disclose_show_email"))
-  })
-  Disclose disclose;
+public class ContactResource extends ContactBase
+    implements DatastoreAndSqlEntity, ForeignKeyedEppResource {

  @Override
  public VKey<ContactResource> createVKey() {
+    // TODO(mmuller): create symmetric keys if we can ever reload both sides.
    return VKey.create(ContactResource.class, getRepoId(), Key.create(this));
  }

@@ -210,81 +62,9 @@ public class ContactResource extends EppResource
    return super.getRepoId();
  }

-  public String getContactId() {
-    return contactId;
-  }
-
-  public PostalInfo getLocalizedPostalInfo() {
-    return localizedPostalInfo;
-  }
-
-  public PostalInfo getInternationalizedPostalInfo() {
-    return internationalizedPostalInfo;
-  }
-
-  public String getSearchName() {
-    return searchName;
-  }
-
-  public ContactPhoneNumber getVoiceNumber() {
-    return voice;
-  }
-
-  public ContactPhoneNumber getFaxNumber() {
-    return fax;
-  }
-
-  public String getEmailAddress() {
-    return email;
-  }
-
-  public ContactAuthInfo getAuthInfo() {
-    return authInfo;
-  }
-
-  public Disclose getDisclose() {
-    return disclose;
-  }
-
-  public final String getCurrentSponsorClientId() {
-    return getPersistedCurrentSponsorClientId();
-  }
-
-  @Override
-  public ContactTransferData getTransferData() {
-    return Optional.ofNullable(transferData).orElse(ContactTransferData.EMPTY);
-  }
-
-  @Override
-  public DateTime getLastTransferTime() {
-    return lastTransferTime;
-  }
-
-  @Override
-  public String getForeignKey() {
-    return contactId;
-  }
-
-  /**
-   * Postal info for the contact.
-   *
-   * <p>The XML marshalling expects the {@link PostalInfo} objects in a list, but we can't actually
-   * persist them to Datastore that way because Objectify can't handle collections of embedded
-   * objects that themselves contain collections, and there's a list of streets inside. This method
-   * transforms the persisted format to the XML format for marshalling.
-   */
-  @XmlElement(name = "postalInfo")
-  public ImmutableList<PostalInfo> getPostalInfosAsList() {
-    return Stream.of(localizedPostalInfo, internationalizedPostalInfo)
-        .filter(Objects::nonNull)
-        .collect(toImmutableList());
-  }
-
  @Override
  public ContactResource cloneProjectedAtTime(DateTime now) {
-    Builder builder = this.asBuilder();
-    projectResourceOntoBuilderAtTime(this, builder, now);
-    return builder.build();
+    return ContactBase.cloneContactProjectedAtTime(this, now);
  }

  @Override
@@ -293,116 +73,12 @@ public class ContactResource extends EppResource
  }

  /** A builder for constructing {@link ContactResource}, since it is immutable. */
-  public static class Builder extends EppResource.Builder<ContactResource, Builder>
-      implements BuilderWithTransferData<ContactTransferData, Builder> {
+  public static class Builder extends ContactBase.Builder<ContactResource, Builder> {

    public Builder() {}

    private Builder(ContactResource instance) {
      super(instance);
    }
-
-    public Builder setContactId(String contactId) {
-      getInstance().contactId = contactId;
-      return this;
-    }
-
-    public Builder setLocalizedPostalInfo(PostalInfo localizedPostalInfo) {
-      checkArgument(localizedPostalInfo == null
-          || Type.LOCALIZED.equals(localizedPostalInfo.getType()));
-      getInstance().localizedPostalInfo = localizedPostalInfo;
-      return this;
-    }
-
-    public Builder setInternationalizedPostalInfo(PostalInfo internationalizedPostalInfo) {
-      checkArgument(internationalizedPostalInfo == null
-          || Type.INTERNATIONALIZED.equals(internationalizedPostalInfo.getType()));
-      getInstance().internationalizedPostalInfo = internationalizedPostalInfo;
-      return this;
-    }
-
-    public Builder overlayLocalizedPostalInfo(PostalInfo localizedPostalInfo) {
-      return setLocalizedPostalInfo(getInstance().localizedPostalInfo == null
-          ? localizedPostalInfo
-          : getInstance().localizedPostalInfo.overlay(localizedPostalInfo));
-    }
-
-    public Builder overlayInternationalizedPostalInfo(PostalInfo internationalizedPostalInfo) {
-      return setInternationalizedPostalInfo(getInstance().internationalizedPostalInfo == null
-          ? internationalizedPostalInfo
-          : getInstance().internationalizedPostalInfo.overlay(internationalizedPostalInfo));
-    }
-
-    public Builder setVoiceNumber(ContactPhoneNumber voiceNumber) {
-      getInstance().voice = voiceNumber;
-      return this;
-    }
-
-    public Builder setFaxNumber(ContactPhoneNumber faxNumber) {
-      getInstance().fax = faxNumber;
-      return this;
-    }
-
-    public Builder setEmailAddress(String emailAddress) {
-      getInstance().email = emailAddress;
-      return this;
-    }
-
-    public Builder setAuthInfo(ContactAuthInfo authInfo) {
-      getInstance().authInfo = authInfo;
-      return this;
-    }
-
-    public Builder setDisclose(Disclose disclose) {
-      getInstance().disclose = disclose;
-      return this;
-    }
-
-    @Override
-    public Builder setTransferData(ContactTransferData transferData) {
-      getInstance().transferData = transferData;
-      return this;
-    }
-
-    @Override
-    public Builder setLastTransferTime(DateTime lastTransferTime) {
-      getInstance().lastTransferTime = lastTransferTime;
-      return thisCastToDerived();
-    }
-
-    /**
-     * Remove all personally identifying information about a contact.
-     *
-     * <p>This should be used when deleting a contact so that the soft-deleted entity doesn't
-     * contain information that the registrant requested to be deleted.
-     */
-    public Builder wipeOut() {
-      setEmailAddress(null);
-      setFaxNumber(null);
-      setInternationalizedPostalInfo(null);
-      setLocalizedPostalInfo(null);
-      setVoiceNumber(null);
-      return this;
-    }
-
-    @Override
-    public ContactResource build() {
-      ContactResource instance = getInstance();
-      // If TransferData is totally empty, set it to null.
-      if (ContactTransferData.EMPTY.equals(instance.transferData)) {
-        setTransferData(null);
-      }
-      // Set the searchName using the internationalized and localized postal info names.
-      if ((instance.internationalizedPostalInfo != null)
-          && (instance.internationalizedPostalInfo.getName() != null)) {
-        instance.searchName = instance.internationalizedPostalInfo.getName();
-      } else if ((instance.localizedPostalInfo != null)
-          && (instance.localizedPostalInfo.getName() != null)) {
-        instance.searchName = instance.localizedPostalInfo.getName();
-      } else {
-        instance.searchName = null;
-      }
-      return super.build();
-    }
  }
 }
@@ -227,7 +227,8 @@ public class DomainBase extends EppResource
   * refer to a {@link PollMessage} timed to when the domain is fully deleted. If the domain is
   * restored, the message should be deleted.
   */
-  @Transient Key<PollMessage.OneTime> deletePollMessage;
+  @Column(name = "deletion_poll_message_id")
+  VKey<PollMessage.OneTime> deletePollMessage;

  /**
   * The recurring billing event associated with this domain's autorenewals.
@@ -237,7 +238,8 @@ public class DomainBase extends EppResource
   * {@link #registrationExpirationTime} is changed the recurrence should be closed, a new one
   * should be created, and this field should be updated to point to the new one.
   */
-  @Transient Key<BillingEvent.Recurring> autorenewBillingEvent;
+  @Column(name = "billing_recurrence_id")
+  VKey<BillingEvent.Recurring> autorenewBillingEvent;

  /**
   * The recurring poll message associated with this domain's autorenewals.
@@ -247,7 +249,8 @@ public class DomainBase extends EppResource
   * {@link #registrationExpirationTime} is changed the recurrence should be closed, a new one
   * should be created, and this field should be updated to point to the new one.
   */
-  @Transient Key<PollMessage.Autorenew> autorenewPollMessage;
+  @Column(name = "autorenew_poll_message_id")
+  VKey<PollMessage.Autorenew> autorenewPollMessage;

  /** The unexpired grace periods for this domain (some of which may not be active yet). */
  @Transient @ElementCollection Set<GracePeriod> gracePeriods;
@@ -316,15 +319,15 @@ public class DomainBase extends EppResource
    return registrationExpirationTime;
  }

-  public Key<PollMessage.OneTime> getDeletePollMessage() {
+  public VKey<PollMessage.OneTime> getDeletePollMessage() {
    return deletePollMessage;
  }

-  public Key<BillingEvent.Recurring> getAutorenewBillingEvent() {
+  public VKey<BillingEvent.Recurring> getAutorenewBillingEvent() {
    return autorenewBillingEvent;
  }

-  public Key<PollMessage.Autorenew> getAutorenewPollMessage() {
+  public VKey<PollMessage.Autorenew> getAutorenewPollMessage() {
    return autorenewPollMessage;
  }

@@ -453,14 +456,8 @@ public class DomainBase extends EppResource
              .setRegistrationExpirationTime(expirationDate)
              // Set the speculatively-written new autorenew events as the domain's autorenew
              // events.
-              .setAutorenewBillingEvent(
-                  transferData.getServerApproveAutorenewEvent() == null
-                      ? null
-                      : transferData.getServerApproveAutorenewEvent().getOfyKey())
-              .setAutorenewPollMessage(
-                  transferData.getServerApproveAutorenewPollMessage() == null
-                      ? null
-                      : transferData.getServerApproveAutorenewPollMessage().getOfyKey());
+              .setAutorenewBillingEvent(transferData.getServerApproveAutorenewEvent())
+              .setAutorenewPollMessage(transferData.getServerApproveAutorenewPollMessage());
      if (transferData.getTransferPeriod().getValue() == 1) {
        // Set the grace period using a key to the prescheduled transfer billing event.  Not using
        // GracePeriod.forBillingEvent() here in order to avoid the actual Datastore fetch.
@@ -471,9 +468,7 @@ public class DomainBase extends EppResource
                    transferExpirationTime.plus(
                        Registry.get(getTld()).getTransferGracePeriodLength()),
                    transferData.getGainingClientId(),
-                    transferData.getServerApproveBillingEvent() == null
-                        ? null
-                        : transferData.getServerApproveBillingEvent().getOfyKey())));
+                    transferData.getServerApproveBillingEvent())));
      } else {
        // There won't be a billing event, so we don't need a grace period
        builder.setGracePeriods(ImmutableSet.of());
@@ -801,19 +796,17 @@ public class DomainBase extends EppResource
      return this;
    }

-    public Builder setDeletePollMessage(Key<PollMessage.OneTime> deletePollMessage) {
+    public Builder setDeletePollMessage(VKey<PollMessage.OneTime> deletePollMessage) {
      getInstance().deletePollMessage = deletePollMessage;
      return this;
    }

-    public Builder setAutorenewBillingEvent(
-        Key<BillingEvent.Recurring> autorenewBillingEvent) {
+    public Builder setAutorenewBillingEvent(VKey<BillingEvent.Recurring> autorenewBillingEvent) {
      getInstance().autorenewBillingEvent = autorenewBillingEvent;
      return this;
    }

-    public Builder setAutorenewPollMessage(
-        Key<PollMessage.Autorenew> autorenewPollMessage) {
+    public Builder setAutorenewPollMessage(VKey<PollMessage.Autorenew> autorenewPollMessage) {
      getInstance().autorenewPollMessage = autorenewPollMessage;
      return this;
    }
@@ -17,12 +17,13 @@ package google.registry.model.domain;
 import static com.google.common.base.Preconditions.checkArgument;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

-import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Embed;
 import com.googlecode.objectify.annotation.Ignore;
 import google.registry.model.ImmutableObject;
 import google.registry.model.billing.BillingEvent;
+import google.registry.model.billing.BillingEvent.Recurring;
 import google.registry.model.domain.rgp.GracePeriodStatus;
+import google.registry.persistence.VKey;
 import javax.annotation.Nullable;
 import javax.persistence.Column;
 import javax.persistence.GeneratedValue;
@@ -57,18 +58,18 @@ public class GracePeriod extends ImmutableObject {

  /**
   * The one-time billing event corresponding to the action that triggered this grace period, or
-   * null if not applicable.  Not set for autorenew grace periods (which instead use the field
-   * {@code billingEventRecurring}) or for redemption grace periods (since deletes have no cost).
+   * null if not applicable. Not set for autorenew grace periods (which instead use the field {@code
+   * billingEventRecurring}) or for redemption grace periods (since deletes have no cost).
   */
  // NB: Would @IgnoreSave(IfNull.class), but not allowed for @Embed collections.
-  Key<BillingEvent.OneTime> billingEventOneTime = null;
+  VKey<BillingEvent.OneTime> billingEventOneTime = null;

  /**
   * The recurring billing event corresponding to the action that triggered this grace period, if
   * applicable - i.e. if the action was an autorenew - or null in all other cases.
   */
  // NB: Would @IgnoreSave(IfNull.class), but not allowed for @Embed collections.
-  Key<BillingEvent.Recurring> billingEventRecurring = null;
+  VKey<BillingEvent.Recurring> billingEventRecurring = null;

  public GracePeriodStatus getType() {
    return type;
@@ -91,8 +92,7 @@ public class GracePeriod extends ImmutableObject {
   * Returns the one time billing event. The value will only be non-null if the type of this grace
   * period is not AUTO_RENEW.
   */
-
-  public Key<BillingEvent.OneTime> getOneTimeBillingEvent() {
+  public VKey<BillingEvent.OneTime> getOneTimeBillingEvent() {
    return billingEventOneTime;
  }

@@ -100,16 +100,16 @@ public class GracePeriod extends ImmutableObject {
   * Returns the recurring billing event. The value will only be non-null if the type of this grace
   * period is AUTO_RENEW.
   */
-  public Key<BillingEvent.Recurring> getRecurringBillingEvent() {
+  public VKey<BillingEvent.Recurring> getRecurringBillingEvent() {
    return billingEventRecurring;
  }

  private static GracePeriod createInternal(
-       GracePeriodStatus type,
-       DateTime expirationTime,
-       String clientId,
-       @Nullable Key<BillingEvent.OneTime> billingEventOneTime,
-       @Nullable Key<BillingEvent.Recurring> billingEventRecurring) {
+      GracePeriodStatus type,
+      DateTime expirationTime,
+      String clientId,
+      @Nullable VKey<BillingEvent.OneTime> billingEventOneTime,
+      @Nullable VKey<BillingEvent.Recurring> billingEventRecurring) {
    checkArgument((billingEventOneTime == null) || (billingEventRecurring == null),
        "A grace period can have at most one billing event");
    checkArgument(
@@ -127,15 +127,15 @@ public class GracePeriod extends ImmutableObject {
  /**
   * Creates a GracePeriod for an (optional) OneTime billing event.
   *
-   * <p>Normal callers should always use {@link #forBillingEvent} instead, assuming they do not
-   * need to avoid loading the BillingEvent from Datastore.  This method should typically be
-   * called only from test code to explicitly construct GracePeriods.
+   * <p>Normal callers should always use {@link #forBillingEvent} instead, assuming they do not need
+   * to avoid loading the BillingEvent from Datastore. This method should typically be called only
+   * from test code to explicitly construct GracePeriods.
   */
  public static GracePeriod create(
      GracePeriodStatus type,
      DateTime expirationTime,
      String clientId,
-      @Nullable Key<BillingEvent.OneTime> billingEventOneTime) {
+      @Nullable VKey<BillingEvent.OneTime> billingEventOneTime) {
    return createInternal(type, expirationTime, clientId, billingEventOneTime, null);
  }

@@ -144,7 +144,7 @@ public class GracePeriod extends ImmutableObject {
      GracePeriodStatus type,
      DateTime expirationTime,
      String clientId,
-      Key<BillingEvent.Recurring> billingEventRecurring) {
+      VKey<Recurring> billingEventRecurring) {
    checkArgumentNotNull(billingEventRecurring, "billingEventRecurring cannot be null");
    return createInternal(type, expirationTime, clientId, null, billingEventRecurring);
  }
@@ -159,6 +159,6 @@ public class GracePeriod extends ImmutableObject {
  public static GracePeriod forBillingEvent(
      GracePeriodStatus type, BillingEvent.OneTime billingEvent) {
    return create(
-        type, billingEvent.getBillingTime(), billingEvent.getClientId(), Key.create(billingEvent));
+        type, billingEvent.getBillingTime(), billingEvent.getClientId(), billingEvent.createVKey());
  }
 }
@@ -125,7 +125,7 @@ public class HostBase extends EppResource {
  }

  @Override
-  public VKey<? extends EppResource> createVKey() {
+  public VKey<? extends HostBase> createVKey() {
    return VKey.create(HostBase.class, getRepoId(), Key.create(this));
  }

@@ -73,7 +73,7 @@ public class HostHistory extends HistoryEntry {
      return this;
    }

-    public Builder setHostResourceId(VKey<HostResource> hostRepoId) {
+    public Builder setHostRepoId(VKey<HostResource> hostRepoId) {
      getInstance().hostRepoId = hostRepoId;
      hostRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
      return this;
@@ -83,8 +83,7 @@ public class HostHistory extends HistoryEntry {
    @Override
    public Builder setParent(Key<? extends EppResource> parent) {
      super.setParent(parent);
-      getInstance().hostRepoId =
-          VKey.create(HostResource.class, parent.getName(), (Key<HostResource>) parent);
+      getInstance().hostRepoId = VKey.create(HostResource.class, parent.getName(), parent);
      return this;
    }
  }
@@ -168,7 +168,7 @@ class CommitLoggedWork<R> implements Runnable {
      DateTime transactionTime, Set<Entry<Key<BackupGroupRoot>, BackupGroupRoot>> bgrEntries) {
    ImmutableMap.Builder<Key<BackupGroupRoot>, DateTime> builder = new ImmutableMap.Builder<>();
    for (Entry<Key<BackupGroupRoot>, BackupGroupRoot> entry : bgrEntries) {
-      DateTime updateTime = entry.getValue().getUpdateAutoTimestamp().getTimestamp();
+      DateTime updateTime = entry.getValue().getUpdateTimestamp().getTimestamp();
      if (!updateTime.isBefore(transactionTime)) {
        builder.put(entry.getKey(), updateTime);
      }
@@ -48,6 +48,10 @@ public final class RdeRevision extends ImmutableObject {
   */
  int revision;

+  public int getRevision() {
+    return revision;
+  }
+
  /**
   * Returns next revision ID to use when staging a new deposit file for the given triplet.
   *
@@ -15,6 +15,7 @@
 package google.registry.model.registry.label;

 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
@@ -30,6 +31,7 @@ import com.google.common.collect.Multiset;
 import com.google.common.util.concurrent.UncheckedExecutionException;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Id;
+import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
@@ -42,6 +44,11 @@ import java.util.Map;
 import java.util.Optional;
 import java.util.concurrent.ExecutionException;
 import javax.annotation.Nullable;
+import javax.persistence.Column;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.MappedSuperclass;
+import javax.persistence.Transient;
 import org.joda.time.DateTime;

 /**
@@ -49,25 +56,46 @@ import org.joda.time.DateTime;
 *
 * @param <T> The type of the root value being listed, e.g. {@link ReservationType}.
 * @param <R> The type of domain label entry being listed, e.g. {@link ReservedListEntry} (note,
- *            must subclass {@link DomainLabelEntry}.
+ *     must subclass {@link DomainLabelEntry}.
 */
+@MappedSuperclass
 public abstract class BaseDomainLabelList<T extends Comparable<?>, R extends DomainLabelEntry<T, ?>>
    extends ImmutableObject implements Buildable {

+  @Ignore
+  @javax.persistence.Id
+  @GeneratedValue(strategy = GenerationType.IDENTITY)
+  Long revisionId;
+
  @Id
+  @Column(nullable = false)
  String name;

-  @Parent
-  Key<EntityGroupRoot> parent = getCrossTldKey();
+  @Parent @Transient Key<EntityGroupRoot> parent = getCrossTldKey();

-  DateTime creationTime;
+  @Transient DateTime creationTime;

+  // The list in Cloud SQL is immutable, we only have a creation_timestamp field and it should be
+  // set to the timestamp when the list is created. In Datastore, we have two fields and the
+  // lastUpdateTime is set to the current timestamp when creating and updating a list. So, we use
+  // lastUpdateTime as the creation_timestamp column during the dual-write phase for compatibility.
+  @Column(name = "creation_timestamp", nullable = false)
  DateTime lastUpdateTime;

+  /** Returns the ID of this revision, or throws if null. */
+  public long getRevisionId() {
+    checkState(
+        revisionId != null,
+        "revisionId is null because this object has not been persisted to the database yet");
+    return revisionId;
+  }
+
+  /** Returns the name of the reserved list. */
  public String getName() {
    return name;
  }

+  /** Returns the creation time of this revision of the reserved list. */
  public DateTime getCreationTime() {
    return creationTime;
  }
@@ -183,6 +211,9 @@ public abstract class BaseDomainLabelList<T extends Comparable<?>, R extends Dom
    @Override
    public T build() {
      checkArgument(!isNullOrEmpty(getInstance().name), "List must have a name");
+      // The list is immutable in Cloud SQL, so make sure the revision id is not set when the
+      // builder object is created from a list object
+      getInstance().revisionId = null;
      return super.build();
    }
  }
@@ -23,16 +23,20 @@ import com.google.common.net.InternetDomainName;
 import com.googlecode.objectify.annotation.Id;
 import google.registry.model.Buildable.GenericBuilder;
 import google.registry.model.ImmutableObject;
+import javax.persistence.Column;
+import javax.persistence.MappedSuperclass;

 /**
 * Represents a label entry parsed from a line in a reserved/premium list txt file.
 *
 * @param <T> The type of the value stored for the domain label, e.g. {@link ReservationType}.
 */
+@MappedSuperclass
 public abstract class DomainLabelEntry<T extends Comparable<?>, D extends DomainLabelEntry<?, ?>>
    extends ImmutableObject implements Comparable<D> {

  @Id
+  @Column(name = "domain_label", insertable = false, updatable = false)
  String label;

  String comment;
@@ -14,7 +14,9 @@

 package google.registry.model.registry.label;

+import static com.google.common.base.Charsets.US_ASCII;
 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.hash.Funnels.stringFunnel;
 import static com.google.common.hash.Funnels.unencodedCharsFunnel;
 import static google.registry.config.RegistryConfig.getDomainLabelListCacheDuration;
 import static google.registry.config.RegistryConfig.getSingletonCachePersistDuration;
@@ -32,43 +34,82 @@ import com.google.common.cache.CacheLoader;
 import com.google.common.cache.CacheLoader.InvalidCacheLoadException;
 import com.google.common.cache.LoadingCache;
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
 import com.google.common.hash.BloomFilter;
 import com.google.common.util.concurrent.UncheckedExecutionException;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Id;
+import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ReportedOn;
 import google.registry.model.registry.Registry;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import google.registry.schema.replay.DatastoreEntity;
 import google.registry.schema.replay.SqlEntity;
+import google.registry.schema.tld.PremiumListDao;
 import google.registry.util.NonFinalForTesting;
 import java.io.ByteArrayOutputStream;
 import java.io.IOException;
+import java.math.BigDecimal;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.ExecutionException;
 import javax.annotation.Nullable;
+import javax.persistence.CollectionTable;
+import javax.persistence.Column;
+import javax.persistence.ElementCollection;
+import javax.persistence.Index;
+import javax.persistence.JoinColumn;
+import javax.persistence.MapKeyColumn;
+import javax.persistence.PostLoad;
+import javax.persistence.PrePersist;
+import javax.persistence.Table;
+import javax.persistence.Transient;
+import org.hibernate.LazyInitializationException;
+import org.joda.money.CurrencyUnit;
 import org.joda.money.Money;
 import org.joda.time.Duration;

-/** A premium list entity, persisted to Datastore, that is used to check domain label prices. */
+/**
+ * A premium list entity that is used to check domain label prices.
+ *
+ * <p>Note that the primary key of this entity is {@link #revisionId}, which is auto-generated by
+ * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
+ * succeeds, we will end up with having two exact same premium lists that differ only by revisionId.
+ * This is fine though, because we only use the list with the highest revisionId.
+ */
@ReportedOn
@Entity
+@javax.persistence.Entity
+@Table(indexes = {@Index(columnList = "name", name = "premiumlist_name_idx")})
 public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.PremiumListEntry>
-    implements DatastoreEntity {
+    implements DatastoreAndSqlEntity {

  /** Stores the revision key for the set of currently used premium list entry entities. */
-  Key<PremiumListRevision> revisionKey;
+  @Transient Key<PremiumListRevision> revisionKey;

-  @Override
-  public ImmutableList<SqlEntity> toSqlEntities() {
-    return ImmutableList.of(); // PremiumList is dual-written
-  }
+  @Ignore
+  @Column(nullable = false)
+  CurrencyUnit currency;
+
+  @Ignore
+  @ElementCollection
+  @CollectionTable(
+      name = "PremiumEntry",
+      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
+  @MapKeyColumn(name = "domainLabel")
+  @Column(name = "price", nullable = false)
+  Map<String, BigDecimal> labelsToPrices;
+
+  @Ignore
+  @Column(nullable = false)
+  BloomFilter<String> bloomFilter;

  /** Virtual parent entity for premium list entry entities associated with a single revision. */
  @ReportedOn
@@ -247,6 +288,35 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
    return Optional.ofNullable(loadPremiumList(name));
  }

+  /** Returns the {@link CurrencyUnit} used for this list. */
+  public CurrencyUnit getCurrency() {
+    return currency;
+  }
+
+  /**
+   * Returns a {@link Map} of domain labels to prices.
+   *
+   * <p>Note that this is lazily loaded and thus will throw a {@link LazyInitializationException} if
+   * used outside the transaction in which the given entity was loaded. You generally should not be
+   * using this anyway as it's inefficient to load all of the PremiumEntry rows if you don't need
+   * them. To check prices, use {@link PremiumListDao#getPremiumPrice} instead.
+   */
+  @Nullable
+  public ImmutableMap<String, BigDecimal> getLabelsToPrices() {
+    return labelsToPrices == null ? null : ImmutableMap.copyOf(labelsToPrices);
+  }
+
+  /**
+   * Returns a Bloom filter to determine whether a label might be premium, or is definitely not.
+   *
+   * <p>If the domain label might be premium, then the next step is to check for the existence of a
+   * corresponding row in the PremiumListEntry table. Otherwise, we know for sure it's not premium,
+   * and no DB load is required.
+   */
+  public BloomFilter<String> getBloomFilter() {
+    return bloomFilter;
+  }
+
  /**
   * A premium list entry entity, persisted to Datastore. Each instance represents the price of a
   * single label on a given TLD.
@@ -339,9 +409,39 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
      return this;
    }

+    public Builder setCurrency(CurrencyUnit currency) {
+      getInstance().currency = currency;
+      return this;
+    }
+
+    public Builder setLabelsToPrices(Map<String, BigDecimal> labelsToPrices) {
+      getInstance().labelsToPrices = ImmutableMap.copyOf(labelsToPrices);
+      return this;
+    }
+
    @Override
    public PremiumList build() {
+      if (getInstance().labelsToPrices != null) {
+        // ASCII is used for the charset because all premium list domain labels are stored
+        // punycoded.
+        getInstance().bloomFilter =
+            BloomFilter.create(stringFunnel(US_ASCII), getInstance().labelsToPrices.size());
+        getInstance()
+            .labelsToPrices
+            .keySet()
+            .forEach(label -> getInstance().bloomFilter.put(label));
+      }
      return super.build();
    }
  }
+
+  @PrePersist
+  void prePersist() {
+    lastUpdateTime = creationTime;
+  }
+
+  @PostLoad
+  void postLoad() {
+    creationTime = lastUpdateTime;
+  }
 }
@@ -16,11 +16,8 @@ package google.registry.model.registry.label;

 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
-import static com.google.common.collect.ImmutableMap.toImmutableMap;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.config.RegistryConfig.getDomainLabelListCacheDuration;
-import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
-import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.model.registry.label.ReservationType.FULLY_BLOCKED;
 import static google.registry.util.CollectionUtils.nullToEmpty;
 import static java.util.concurrent.TimeUnit.MILLISECONDS;
@@ -30,12 +27,8 @@ import com.google.common.base.Splitter;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
 import com.google.common.cache.LoadingCache;
-import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
-import com.google.common.collect.MapDifference;
-import com.google.common.collect.Maps;
-import com.google.common.flogger.FluentLogger;
 import com.google.common.util.concurrent.UncheckedExecutionException;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Embed;
@@ -45,45 +38,58 @@ import com.googlecode.objectify.mapper.Mapper;
 import google.registry.model.Buildable;
 import google.registry.model.registry.Registry;
 import google.registry.model.registry.label.DomainLabelMetrics.MetricsReservedListMatch;
-import google.registry.schema.replay.DatastoreEntity;
-import google.registry.schema.replay.SqlEntity;
-import google.registry.schema.tld.ReservedList.ReservedEntry;
-import google.registry.schema.tld.ReservedListDao;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
 import java.util.concurrent.ExecutionException;
 import javax.annotation.Nullable;
+import javax.persistence.CollectionTable;
+import javax.persistence.Column;
+import javax.persistence.ElementCollection;
+import javax.persistence.Embeddable;
+import javax.persistence.Index;
+import javax.persistence.JoinColumn;
+import javax.persistence.MapKeyColumn;
+import javax.persistence.Table;
 import org.joda.time.DateTime;

 /**
- * A reserved list entity, persisted to Datastore, that is used to check domain label reservations.
+ * A list of reserved domain labels that are blocked from being registered for various reasons.
+ *
+ * <p>Note that the primary key of this entity is {@link #revisionId}, which is auto-generated by
+ * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
+ * succeeds, we will end up with having two exact same reserved lists that differ only by
+ * revisionId. This is fine though, because we only use the list with the highest revisionId.
 */
@Entity
+@javax.persistence.Entity
+@Table(indexes = {@Index(columnList = "name", name = "reservedlist_name_idx")})
 public final class ReservedList
    extends BaseDomainLabelList<ReservationType, ReservedList.ReservedListEntry>
-    implements DatastoreEntity {
-
-  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+    implements DatastoreAndSqlEntity {

  @Mapify(ReservedListEntry.LabelMapper.class)
+  @ElementCollection
+  @CollectionTable(
+      name = "ReservedEntry",
+      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
+  @MapKeyColumn(name = "domain_label")
  Map<String, ReservedListEntry> reservedListMap;

+  @Column(nullable = false)
  boolean shouldPublish = true;

-  @Override
-  public ImmutableList<SqlEntity> toSqlEntities() {
-    return ImmutableList.of(); // ReservedList is dual-written
-  }
-
  /**
   * A reserved list entry entity, persisted to Datastore, that represents a single label and its
   * reservation type.
   */
  @Embed
-  public static class ReservedListEntry
-      extends DomainLabelEntry<ReservationType, ReservedListEntry> implements Buildable {
+  @Embeddable
+  public static class ReservedListEntry extends DomainLabelEntry<ReservationType, ReservedListEntry>
+      implements Buildable {

+    @Column(nullable = false)
    ReservationType reservationType;

    /** Mapper for use with @Mapify */
@@ -150,6 +156,7 @@ public final class ReservedList
    return shouldPublish;
  }

+  /** Returns a {@link Map} of domain labels to {@link ReservedListEntry}. */
  public ImmutableMap<String, ReservedListEntry> getReservedListEntries() {
    return ImmutableMap.copyOf(nullToEmpty(reservedListMap));
  }
@@ -239,65 +246,10 @@ public final class ReservedList
              new CacheLoader<String, ReservedList>() {
                @Override
                public ReservedList load(String listName) {
-                  ReservedList datastoreList =
-                      ofy()
-                          .load()
-                          .type(ReservedList.class)
-                          .parent(getCrossTldKey())
-                          .id(listName)
-                          .now();
-                  // Also load the list from Cloud SQL, compare the two lists, and log if different.
-                  try {
-                    loadAndCompareCloudSqlList(datastoreList);
-                  } catch (Throwable t) {
-                    logger.atSevere().withCause(t).log("Error comparing reserved lists.");
-                  }
-                  return datastoreList;
+                  return ReservedListDualWriteDao.getLatestRevision(listName).orElse(null);
                }
              });

-  private static final void loadAndCompareCloudSqlList(ReservedList datastoreList) {
-    Optional<google.registry.schema.tld.ReservedList> maybeCloudSqlList =
-        ReservedListDao.getLatestRevision(datastoreList.getName());
-    if (maybeCloudSqlList.isPresent()) {
-      Map<String, ReservedEntry> datastoreLabelsToReservations =
-          datastoreList.reservedListMap.entrySet().parallelStream()
-              .collect(
-                  toImmutableMap(
-                      Map.Entry::getKey,
-                      entry ->
-                          ReservedEntry.create(
-                              entry.getValue().reservationType, entry.getValue().comment)));
-
-      google.registry.schema.tld.ReservedList cloudSqlList = maybeCloudSqlList.get();
-      MapDifference<String, ReservedEntry> diff =
-          Maps.difference(datastoreLabelsToReservations, cloudSqlList.getLabelsToReservations());
-      if (!diff.areEqual()) {
-        if (diff.entriesDiffering().size() > 10) {
-          logger.atWarning().log(
-              String.format(
-                  "Unequal reserved lists detected, Cloud SQL list with revision"
-                      + " id %d has %d different records than the current"
-                      + " Datastore list.",
-                  cloudSqlList.getRevisionId(), diff.entriesDiffering().size()));
-        } else {
-          StringBuilder diffMessage = new StringBuilder("Unequal reserved lists detected:\n");
-          diff.entriesDiffering()
-              .forEach(
-                  (label, valueDiff) ->
-                      diffMessage.append(
-                          String.format(
-                              "Domain label %s has entry %s in Datastore and entry"
-                                  + " %s in Cloud SQL.\n",
-                              label, valueDiff.leftValue(), valueDiff.rightValue())));
-          logger.atWarning().log(diffMessage.toString());
-        }
-      }
-    } else {
-      logger.atWarning().log("Reserved list in Cloud SQL is empty.");
-    }
-  }
-
  /**
   * Gets the {@link ReservationType} of a label in a single ReservedList, or returns an absent
   * Optional if none exists in the list.
@@ -0,0 +1,124 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.registry.label;
+
+import static com.google.common.collect.ImmutableMap.toImmutableMap;
+import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
+import static google.registry.persistence.transaction.TransactionManagerFactory.ofyTm;
+
+import com.google.common.collect.MapDifference;
+import com.google.common.collect.MapDifference.ValueDifference;
+import com.google.common.collect.Maps;
+import com.google.common.flogger.FluentLogger;
+import com.googlecode.objectify.Key;
+import google.registry.model.registry.label.ReservedList.ReservedListEntry;
+import google.registry.persistence.VKey;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * A {@link ReservedList} DAO that does dual-write and dual-read against Datastore and Cloud SQL. It
+ * still uses Datastore as the primary storage and suppresses any exception thrown by Cloud SQL.
+ *
+ * <p>TODO(b/160993806): Delete this DAO and switch to use the SQL only DAO after migrating to Cloud
+ * SQL.
+ */
+public class ReservedListDualWriteDao {
+
+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
+  private ReservedListDualWriteDao() {}
+
+  /** Persist a new reserved list to Cloud SQL. */
+  public static void save(ReservedList reservedList) {
+    ofyTm().transact(() -> ofyTm().saveNewOrUpdate(reservedList));
+    try {
+      logger.atInfo().log("Saving reserved list %s to Cloud SQL", reservedList.getName());
+      ReservedListSqlDao.save(reservedList);
+      logger.atInfo().log(
+          "Saved reserved list %s with %d entries to Cloud SQL",
+          reservedList.getName(), reservedList.getReservedListEntries().size());
+    } catch (Throwable t) {
+      logger.atSevere().withCause(t).log("Error saving the reserved list to Cloud SQL.");
+    }
+  }
+
+  /**
+   * Returns the most recent revision of the {@link ReservedList} with the specified name, if it
+   * exists.
+   */
+  public static Optional<ReservedList> getLatestRevision(String reservedListName) {
+    Optional<ReservedList> maybeDatastoreList =
+        ofyTm()
+            .maybeLoad(
+                VKey.createOfy(
+                    ReservedList.class,
+                    Key.create(getCrossTldKey(), ReservedList.class, reservedListName)));
+    try {
+      // Also load the list from Cloud SQL, compare the two lists, and log if different.
+      maybeDatastoreList.ifPresent(ReservedListDualWriteDao::loadAndCompareCloudSqlList);
+    } catch (Throwable t) {
+      logger.atSevere().withCause(t).log("Error comparing reserved lists.");
+    }
+    return maybeDatastoreList;
+  }
+
+  private static void loadAndCompareCloudSqlList(ReservedList datastoreList) {
+    Optional<ReservedList> maybeCloudSqlList =
+        ReservedListSqlDao.getLatestRevision(datastoreList.getName());
+    if (maybeCloudSqlList.isPresent()) {
+      Map<String, ReservedListEntry> datastoreLabelsToReservations =
+          datastoreList.reservedListMap.entrySet().parallelStream()
+              .collect(
+                  toImmutableMap(
+                      Map.Entry::getKey,
+                      entry ->
+                          ReservedListEntry.create(
+                              entry.getKey(),
+                              entry.getValue().reservationType,
+                              entry.getValue().comment)));
+
+      ReservedList cloudSqlList = maybeCloudSqlList.get();
+      MapDifference<String, ReservedListEntry> diff =
+          Maps.difference(datastoreLabelsToReservations, cloudSqlList.reservedListMap);
+      if (!diff.areEqual()) {
+        if (diff.entriesDiffering().size() > 10) {
+          logger.atWarning().log(
+              String.format(
+                  "Unequal reserved lists detected, Cloud SQL list with revision"
+                      + " id %d has %d different records than the current"
+                      + " Datastore list.",
+                  cloudSqlList.getRevisionId(), diff.entriesDiffering().size()));
+        } else {
+          StringBuilder diffMessage = new StringBuilder("Unequal reserved lists detected:\n");
+          diff.entriesDiffering().entrySet().stream()
+              .forEach(
+                  entry -> {
+                    String label = entry.getKey();
+                    ValueDifference<ReservedListEntry> valueDiff = entry.getValue();
+                    diffMessage.append(
+                        String.format(
+                            "Domain label %s has entry %s in Datastore and entry"
+                                + " %s in Cloud SQL.\n",
+                            label, valueDiff.leftValue(), valueDiff.rightValue()));
+                  });
+          logger.atWarning().log(diffMessage.toString());
+        }
+      }
+    } else {
+      logger.atWarning().log("Reserved list in Cloud SQL is empty.");
+    }
+  }
+}
@@ -12,20 +12,46 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-package google.registry.schema.tld;
+package google.registry.model.registry.label;

 import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

-import com.google.common.util.concurrent.UncheckedExecutionException;
 import java.util.Optional;
-import java.util.concurrent.ExecutionException;

-/** Data access object class for {@link ReservedList} */
-public class ReservedListDao {
+/**
+ * A {@link ReservedList} DAO for Cloud SQL.
+ *
+ * <p>TODO(b/160993806): Rename this class to ReservedListDao after migrating to Cloud SQL.
+ */
+public class ReservedListSqlDao {
+
+  private ReservedListSqlDao() {}

  /** Persist a new reserved list to Cloud SQL. */
  public static void save(ReservedList reservedList) {
-    jpaTm().transact(() -> jpaTm().getEntityManager().persist(reservedList));
+    checkArgumentNotNull(reservedList, "Must specify reservedList");
+    jpaTm().transact(() -> jpaTm().saveNew(reservedList));
+  }
+
+  /**
+   * Returns the most recent revision of the {@link ReservedList} with the specified name, if it
+   * exists.
+   */
+  public static Optional<ReservedList> getLatestRevision(String reservedListName) {
+    return jpaTm()
+        .transact(
+            () ->
+                jpaTm()
+                    .getEntityManager()
+                    .createQuery(
+                        "FROM ReservedList rl LEFT JOIN FETCH rl.reservedListMap WHERE"
+                            + " rl.revisionId IN (SELECT MAX(revisionId) FROM ReservedList subrl"
+                            + " WHERE subrl.name = :name)",
+                        ReservedList.class)
+                    .setParameter("name", reservedListName)
+                    .getResultStream()
+                    .findFirst());
  }

  /**
@@ -46,39 +72,4 @@ public class ReservedListDao {
                        .size()
                    > 0);
  }
-
-  /**
-   * Returns the most recent revision of the {@link ReservedList} with the specified name, if it
-   * exists. TODO(shicong): Change this method to package level access after dual-read phase.
-   */
-  public static Optional<ReservedList> getLatestRevision(String reservedListName) {
-    return jpaTm()
-        .transact(
-            () ->
-                jpaTm()
-                    .getEntityManager()
-                    .createQuery(
-                        "FROM ReservedList rl LEFT JOIN FETCH rl.labelsToReservations WHERE"
-                            + " rl.revisionId IN (SELECT MAX(revisionId) FROM ReservedList subrl"
-                            + " WHERE subrl.name = :name)",
-                        ReservedList.class)
-                    .setParameter("name", reservedListName)
-                    .getResultStream()
-                    .findFirst());
-  }
-
-  /**
-   * Returns the most recent revision of the {@link ReservedList} with the specified name, from
-   * cache.
-   */
-  public static Optional<ReservedList> getLatestRevisionCached(String reservedListName) {
-    try {
-      return ReservedListCache.cacheReservedLists.get(reservedListName);
-    } catch (ExecutionException e) {
-      throw new UncheckedExecutionException(
-          "Could not retrieve reserved list named " + reservedListName, e);
-    }
-  }
-
-  private ReservedListDao() {}
 }
@@ -185,6 +185,10 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
  @Transient // domain-specific
  Set<DomainTransactionRecord> domainTransactionRecords;

+  public Long getId() {
+    return id;
+  }
+
  public Key<? extends EppResource> getParent() {
    return parent;
  }
@@ -0,0 +1,58 @@
+// Copyright 2019 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence;
+
+import com.google.common.base.Predicates;
+import com.google.common.collect.ImmutableSet;
+import java.sql.SQLException;
+import java.util.function.Predicate;
+import javax.persistence.OptimisticLockException;
+
+/** Helpers for identifying retriable database operations. */
+public final class JpaRetries {
+
+  private JpaRetries() {}
+
+  private static final ImmutableSet<String> RETRIABLE_TXN_SQL_STATE =
+      ImmutableSet.of(
+          "40001", // serialization_failure
+          "40P01", // deadlock_detected, PSQL-specific
+          "55006", // object_in_use, PSQL and DB2
+          "55P03" // lock_not_available, PSQL-specific
+          );
+
+  private static final Predicate<Throwable> RETRIABLE_TXN_PREDICATE =
+      Predicates.or(
+          OptimisticLockException.class::isInstance,
+          e ->
+              e instanceof SQLException
+                  && RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
+
+  public static boolean isFailedTxnRetriable(Throwable throwable) {
+    Throwable t = throwable;
+    while (t != null) {
+      if (RETRIABLE_TXN_PREDICATE.test(t)) {
+        return true;
+      }
+      t = t.getCause();
+    }
+    return false;
+  }
+
+  public static boolean isFailedQueryRetriable(Throwable throwable) {
+    // TODO(weiminyu): check for more error codes.
+    return isFailedTxnRetriable(throwable);
+  }
+}
@@ -94,6 +94,21 @@ public class PersistenceModule {
      @Config("cloudSqlJdbcUrl") String jdbcUrl,
      @Config("cloudSqlInstanceConnectionName") String instanceConnectionName,
      @DefaultHibernateConfigs ImmutableMap<String, String> defaultConfigs) {
+    return createPartialSqlConfigs(jdbcUrl, instanceConnectionName, defaultConfigs);
+  }
+
+  @Provides
+  @Singleton
+  @BeamPipelineCloudSqlConfigs
+  static ImmutableMap<String, String> provideBeamPipelineCloudSqlConfigs(
+      @Config("beamCloudSqlJdbcUrl") String jdbcUrl,
+      @Config("beamCloudSqlInstanceConnectionName") String instanceConnectionName,
+      @DefaultHibernateConfigs ImmutableMap<String, String> defaultConfigs) {
+    return createPartialSqlConfigs(jdbcUrl, instanceConnectionName, defaultConfigs);
+  }
+
+  private static ImmutableMap<String, String> createPartialSqlConfigs(
+      String jdbcUrl, String instanceConnectionName, ImmutableMap<String, String> defaultConfigs) {
    HashMap<String, String> overrides = Maps.newHashMap(defaultConfigs);
    overrides.put(Environment.URL, jdbcUrl);
    overrides.put(HIKARI_DS_SOCKET_FACTORY, "com.google.cloud.sql.postgres.SocketFactory");
@@ -135,13 +150,15 @@ public class PersistenceModule {
  @Singleton
  @SocketFactoryJpaTm
  static JpaTransactionManager provideSocketFactoryJpaTm(
-      @Config("cloudSqlUsername") String username,
-      @Config("cloudSqlPassword") String password,
-      @PartialCloudSqlConfigs ImmutableMap<String, String> cloudSqlConfigs,
+      @Config("beamCloudSqlUsername") String username,
+      @Config("beamCloudSqlPassword") String password,
+      @Config("beamHibernateHikariMaximumPoolSize") int hikariMaximumPoolSize,
+      @BeamPipelineCloudSqlConfigs ImmutableMap<String, String> cloudSqlConfigs,
      Clock clock) {
    HashMap<String, String> overrides = Maps.newHashMap(cloudSqlConfigs);
    overrides.put(Environment.USER, username);
    overrides.put(Environment.PASS, password);
+    overrides.put(HIKARI_MAXIMUM_POOL_SIZE, String.valueOf(hikariMaximumPoolSize));
    return new JpaTransactionManagerImpl(create(overrides), clock);
  }

@@ -149,9 +166,9 @@ public class PersistenceModule {
  @Singleton
  @JdbcJpaTm
  static JpaTransactionManager provideLocalJpaTm(
-      @Config("cloudSqlJdbcUrl") String jdbcUrl,
-      @Config("cloudSqlUsername") String username,
-      @Config("cloudSqlPassword") String password,
+      @Config("beamCloudSqlJdbcUrl") String jdbcUrl,
+      @Config("beamCloudSqlUsername") String username,
+      @Config("beamCloudSqlPassword") String password,
      @DefaultHibernateConfigs ImmutableMap<String, String> defaultConfigs,
      Clock clock) {
    HashMap<String, String> overrides = Maps.newHashMap(defaultConfigs);
@@ -218,6 +235,11 @@ public class PersistenceModule {
  @Documented
  @interface PartialCloudSqlConfigs {}

+  /** Dagger qualifier for the Cloud SQL configs used by Beam pipelines. */
+  @Qualifier
+  @Documented
+  @interface BeamPipelineCloudSqlConfigs {}
+
  /** Dagger qualifier for the default Hibernate configurations. */
  // TODO(shicong): Change annotations in this class to none public or put them in a top level
  //  package
@@ -58,6 +58,31 @@ public class VKey<T> extends ImmutableObject implements Serializable {
    return new VKey(kind, null, sqlKey);
  }

+  /** Creates a {@link VKey} which only contains the ofy primary key. */
+  public static <T> VKey<T> createOfy(
+      Class<? extends T> kind, com.googlecode.objectify.Key<? extends T> ofyKey) {
+    checkArgumentNotNull(kind, "kind must not be null");
+    checkArgumentNotNull(ofyKey, "ofyKey must not be null");
+    return new VKey(kind, ofyKey, null);
+  }
+
+  /**
+   * Creates a {@link VKey} which only contains the ofy primary key by specifying the id of the
+   * {@link Key}.
+   */
+  public static <T> VKey<T> createOfy(Class<? extends T> kind, long id) {
+    return createOfy(kind, Key.create(kind, id));
+  }
+
+  /**
+   * Creates a {@link VKey} which only contains the ofy primary key by specifying the name of the
+   * {@link Key}.
+   */
+  public static <T> VKey<T> createOfy(Class<? extends T> kind, String name) {
+    checkArgumentNotNull(kind, "name must not be null");
+    return createOfy(kind, Key.create(kind, name));
+  }
+
  /** Creates a {@link VKey} which only contains both sql and ofy primary key. */
  public static <T> VKey<T> create(
      Class<? extends T> kind, Object sqlKey, com.googlecode.objectify.Key ofyKey) {
@@ -25,4 +25,11 @@ public interface JpaTransactionManager extends TransactionManager {

  /** Deletes the entity by its id, throws exception if the entity is not deleted. */
  public abstract <T> void assertDelete(VKey<T> key);
+
+  /**
+   * Releases all resources and shuts down.
+   *
+   * <p>The errorprone check forbids injection of {@link java.io.Closeable} resources.
+   */
+  void teardown();
 }
@@ -26,6 +26,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.flogger.FluentLogger;
+import google.registry.config.RegistryConfig;
 import google.registry.persistence.VKey;
 import google.registry.util.Clock;
 import java.lang.reflect.Field;
@@ -62,6 +63,11 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    this.clock = clock;
  }

+  @Override
+  public void teardown() {
+    emf.close();
+  }
+
  @Override
  public EntityManager getEntityManager() {
    if (transactionInfo.get().entityManager == null) {
@@ -96,12 +102,13 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    EntityTransaction txn = txnInfo.entityManager.getTransaction();
    try {
      txn.begin();
-      txnInfo.inTransaction = true;
-      txnInfo.transactionTime = clock.nowUtc();
+      txnInfo.start(clock);
      T result = work.get();
+      txnInfo.recordTransaction();
      txn.commit();
      return result;
-    } catch (RuntimeException e) {
+    } catch (RuntimeException | Error e) {
+      // Error is unchecked!
      try {
        txn.rollback();
        logger.atWarning().log("Error during transaction; transaction rolled back");
@@ -171,6 +178,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    checkArgumentNotNull(entity, "entity must be specified");
    assertInTransaction();
    getEntityManager().persist(entity);
+    transactionInfo.get().addUpdate(entity);
  }

  @Override
@@ -185,6 +193,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    checkArgumentNotNull(entity, "entity must be specified");
    assertInTransaction();
    getEntityManager().merge(entity);
+    transactionInfo.get().addUpdate(entity);
  }

  @Override
@@ -200,6 +209,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    assertInTransaction();
    checkArgument(checkExists(entity), "Given entity does not exist");
    getEntityManager().merge(entity);
+    transactionInfo.get().addUpdate(entity);
  }

  @Override
@@ -291,6 +301,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
        String.format("DELETE FROM %s WHERE %s", entityType.getName(), getAndClause(entityIds));
    Query query = getEntityManager().createQuery(sql);
    entityIds.forEach(entityId -> query.setParameter(entityId.name, entityId.value));
+    transactionInfo.get().addDelete(key);
    return query.executeUpdate();
  }

@@ -381,9 +392,23 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    boolean inTransaction = false;
    DateTime transactionTime;

+    // Serializable representation of the transaction to be persisted in the Transaction table.
+    Transaction.Builder contentsBuilder;
+
+    /** Start a new transaction. */
+    private void start(Clock clock) {
+      checkArgumentNotNull(clock);
+      inTransaction = true;
+      transactionTime = clock.nowUtc();
+      if (RegistryConfig.getCloudSqlReplicateTransactions()) {
+        contentsBuilder = new Transaction.Builder();
+      }
+    }
+
    private void clear() {
      inTransaction = false;
      transactionTime = null;
+      contentsBuilder = null;
      if (entityManager != null) {
        // Close this EntityManager just let the connection pool be able to reuse it, it doesn't
        // close the underlying database connection.
@@ -391,5 +416,26 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
        entityManager = null;
      }
    }
+
+    private void addUpdate(Object entity) {
+      if (contentsBuilder != null) {
+        contentsBuilder.addUpdate(entity);
+      }
+    }
+
+    private void addDelete(VKey<?> key) {
+      if (contentsBuilder != null) {
+        contentsBuilder.addDelete(key);
+      }
+    }
+
+    private void recordTransaction() {
+      if (contentsBuilder != null) {
+        Transaction persistedTxn = contentsBuilder.build();
+        if (!persistedTxn.isEmpty()) {
+          entityManager.persist(persistedTxn.toEntity());
+        }
+      }
+    }
  }
 }
@@ -109,11 +109,20 @@ public class Transaction extends ImmutableObject implements Buildable {
    return builder.build();
  }

+  /** Returns true if the transaction contains no mutations. */
+  public boolean isEmpty() {
+    return mutations.isEmpty();
+  }
+
  @Override
  public Builder asBuilder() {
    return new Builder(clone(this));
  }

+  public final TransactionEntity toEntity() {
+    return new TransactionEntity(serialize());
+  }
+
  public static class Builder extends GenericBuilder<Transaction, Builder> {

    ImmutableList.Builder listBuilder = new ImmutableList.Builder();
@@ -0,0 +1,43 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.transaction;
+
+import javax.persistence.Entity;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.Id;
+import javax.persistence.Table;
+
+/**
+ * Object to be stored in the transaction table.
+ *
+ * <p>This consists of a sequential identifier and a serialized {@code Tranaction} object.
+ */
+@Entity
+@Table(name = "Transaction")
+public class TransactionEntity {
+
+  @Id
+  @GeneratedValue(strategy = GenerationType.IDENTITY)
+  long id;
+
+  byte[] contents;
+
+  TransactionEntity() {}
+
+  TransactionEntity(byte[] contents) {
+    this.contents = contents;
+  }
+}
@@ -75,7 +75,12 @@ public class TransactionManagerFactory {
    return tm;
  }

-  /** Returns {@link JpaTransactionManager} instance. */
+  /**
+   * Returns {@link JpaTransactionManager} instance.
+   *
+   * <p>Between invocations of {@link TransactionManagerFactory#setJpaTm} every call to this method
+   * returns the same instance.
+   */
  public static JpaTransactionManager jpaTm() {
    return jpaTm.get();
  }
@@ -93,7 +98,7 @@ public class TransactionManagerFactory {
        RegistryEnvironment.get().equals(RegistryEnvironment.UNITTEST)
            || RegistryToolEnvironment.get() != null,
        "setJpamTm() should only be called by tools and tests.");
-    jpaTm = jpaTmSupplier;
+    jpaTm = Suppliers.memoize(jpaTmSupplier::get);
  }

  /** Sets the return of {@link #tm()} to the given instance of {@link TransactionManager}. */
@@ -89,7 +89,7 @@ public abstract class RdeModule {
  @Provides
  @Parameter(PARAM_LENIENT)
  static boolean provideLenient(HttpServletRequest req) {
-    return extractBooleanParameter(req, PARAM_REVISION);
+    return extractBooleanParameter(req, PARAM_LENIENT);
  }

  @Provides
@@ -29,7 +29,7 @@ public enum RdeResourceType {
  DOMAIN("urn:ietf:params:xml:ns:rdeDomain-1.0", EnumSet.of(FULL, THIN)),
  HOST("urn:ietf:params:xml:ns:rdeHost-1.0", EnumSet.of(FULL)),
  REGISTRAR("urn:ietf:params:xml:ns:rdeRegistrar-1.0", EnumSet.of(FULL, THIN)),
-  IDN("urn:ietf:params:xml:ns:rdeIDN-1.0", EnumSet.of(FULL, THIN)),
+  IDN("urn:ietf:params:xml:ns:rdeIDN-1.0", EnumSet.of(FULL)),
  HEADER("urn:ietf:params:xml:ns:rdeHeader-1.0", EnumSet.of(FULL, THIN));

  private final String uri;
@@ -77,7 +77,7 @@ public final class RdeStagingReducer extends Reducer<PendingDeposit, DepositFrag
  private final byte[] stagingKeyBytes;
  private final RdeMarshaller marshaller;

-  private RdeStagingReducer(
+  RdeStagingReducer(
      TaskQueueUtils taskQueueUtils,
      LockHandler lockHandler,
      int gcsBufferSize,
@@ -125,7 +125,7 @@ public final class RdeStagingReducer extends Reducer<PendingDeposit, DepositFrag
    final DateTime watermark = key.watermark();
    final int revision =
        Optional.ofNullable(key.revision())
-            .orElse(RdeRevision.getNextRevision(tld, watermark, mode));
+            .orElseGet(() -> RdeRevision.getNextRevision(tld, watermark, mode));
    String id = RdeUtil.timestampToId(watermark);
    String prefix = RdeNamingUtils.makeRydeFilename(tld, watermark, mode, 1, revision);
    if (key.manual()) {
@@ -168,9 +168,13 @@ public final class RdeStagingReducer extends Reducer<PendingDeposit, DepositFrag
          logger.atSevere().log("Fragment error: %s", fragment.error());
        }
      }
-      for (IdnTableEnum idn : IdnTableEnum.values()) {
-        output.write(marshaller.marshalIdn(idn.getTable()));
-        counter.increment(RdeResourceType.IDN);
+
+      // Don't write the IDN elements for BRDA.
+      if (mode == RdeMode.FULL) {
+        for (IdnTableEnum idn : IdnTableEnum.values()) {
+          output.write(marshaller.marshalIdn(idn.getTable()));
+          counter.increment(RdeResourceType.IDN);
+        }
      }

      // Output XML that says how many resources were emitted.
@@ -16,6 +16,7 @@ package google.registry.schema.tld;

 import com.google.common.collect.ImmutableList;
 import google.registry.model.ImmutableObject;
+import google.registry.model.registry.label.PremiumList;
 import google.registry.schema.replay.DatastoreEntity;
 import google.registry.schema.replay.SqlEntity;
 import java.io.Serializable;
@@ -1,151 +0,0 @@
-// Copyright 2019 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.schema.tld;
-
-import static com.google.common.base.Charsets.US_ASCII;
-import static com.google.common.base.Preconditions.checkState;
-import static com.google.common.hash.Funnels.stringFunnel;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.hash.BloomFilter;
-import google.registry.model.CreateAutoTimestamp;
-import google.registry.schema.replay.DatastoreEntity;
-import google.registry.schema.replay.SqlEntity;
-import java.math.BigDecimal;
-import java.util.Map;
-import javax.annotation.Nullable;
-import javax.persistence.CollectionTable;
-import javax.persistence.Column;
-import javax.persistence.ElementCollection;
-import javax.persistence.Entity;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
-import javax.persistence.Id;
-import javax.persistence.Index;
-import javax.persistence.JoinColumn;
-import javax.persistence.MapKeyColumn;
-import javax.persistence.Table;
-import org.hibernate.LazyInitializationException;
-import org.joda.money.CurrencyUnit;
-import org.joda.time.DateTime;
-
-/**
- * A list of premium prices for domain names.
- *
- * <p>Note that the primary key of this entity is {@link #revisionId}, which is auto-generated by
- * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
- * succeeds, we will end up with having two exact same premium lists that differ only by revisionId.
- * This is fine though, because we only use the list with the highest revisionId.
- */
-@Entity
-@Table(indexes = {@Index(columnList = "name", name = "premiumlist_name_idx")})
-public class PremiumList implements SqlEntity {
-
-  @Column(nullable = false)
-  private String name;
-
-  @Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
-  @Column(nullable = false)
-  private Long revisionId;
-
-  @Column(nullable = false)
-  private CreateAutoTimestamp creationTimestamp = CreateAutoTimestamp.create(null);
-
-  @Column(nullable = false)
-  private CurrencyUnit currency;
-
-  @ElementCollection
-  @CollectionTable(
-      name = "PremiumEntry",
-      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
-  @MapKeyColumn(name = "domainLabel")
-  @Column(name = "price", nullable = false)
-  private Map<String, BigDecimal> labelsToPrices;
-
-  @Column(nullable = false)
-  private BloomFilter<String> bloomFilter;
-
-  private PremiumList(String name, CurrencyUnit currency, Map<String, BigDecimal> labelsToPrices) {
-    this.name = name;
-    this.currency = currency;
-    this.labelsToPrices = labelsToPrices;
-    // ASCII is used for the charset because all premium list domain labels are stored punycoded.
-    this.bloomFilter = BloomFilter.create(stringFunnel(US_ASCII), labelsToPrices.size());
-    labelsToPrices.keySet().forEach(this.bloomFilter::put);
-  }
-
-  // Hibernate requires this default constructor.
-  private PremiumList() {}
-
-  /** Constructs a {@link PremiumList} object. */
-  public static PremiumList create(
-      String name, CurrencyUnit currency, Map<String, BigDecimal> labelsToPrices) {
-    return new PremiumList(name, currency, labelsToPrices);
-  }
-
-  /** Returns the name of the premium list, which is usually also a TLD string. */
-  public String getName() {
-    return name;
-  }
-
-  /** Returns the {@link CurrencyUnit} used for this list. */
-  public CurrencyUnit getCurrency() {
-    return currency;
-  }
-
-  /** Returns the ID of this revision, or throws if null. */
-  public Long getRevisionId() {
-    checkState(
-        revisionId != null,
-        "revisionId is null because this object has not yet been persisted to the DB");
-    return revisionId;
-  }
-
-  /** Returns the creation time of this revision of the premium list. */
-  public DateTime getCreationTimestamp() {
-    return creationTimestamp.getTimestamp();
-  }
-
-  /**
-   * Returns a {@link Map} of domain labels to prices.
-   *
-   * <p>Note that this is lazily loaded and thus will throw a {@link LazyInitializationException} if
-   * used outside the transaction in which the given entity was loaded. You generally should not be
-   * using this anyway as it's inefficient to load all of the PremiumEntry rows if you don't need
-   * them. To check prices, use {@link PremiumListDao#getPremiumPrice} instead.
-   */
-  @Nullable
-  public ImmutableMap<String, BigDecimal> getLabelsToPrices() {
-    return labelsToPrices == null ? null : ImmutableMap.copyOf(labelsToPrices);
-  }
-
-  /**
-   * Returns a Bloom filter to determine whether a label might be premium, or is definitely not.
-   *
-   * <p>If the domain label might be premium, then the next step is to check for the existence of a
-   * corresponding row in the PremiumListEntry table. Otherwise, we know for sure it's not premium,
-   * and no DB load is required.
-   */
-  public BloomFilter<String> getBloomFilter() {
-    return bloomFilter;
-  }
-
-  @Override
-  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
-    return ImmutableList.of(); // PremiumList is dual-written
-  }
-}
@@ -25,6 +25,7 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
 import com.google.common.cache.LoadingCache;
+import google.registry.model.registry.label.PremiumList;
 import google.registry.util.NonFinalForTesting;
 import java.math.BigDecimal;
 import java.util.Optional;
@@ -20,6 +20,7 @@ import static google.registry.persistence.transaction.TransactionManagerFactory.
 import com.google.common.cache.CacheLoader.InvalidCacheLoadException;
 import com.google.common.util.concurrent.UncheckedExecutionException;
 import google.registry.model.registry.Registry;
+import google.registry.model.registry.label.PremiumList;
 import google.registry.schema.tld.PremiumListCache.RevisionIdAndLabel;
 import java.math.BigDecimal;
 import java.util.Optional;
@@ -16,6 +16,7 @@ package google.registry.schema.tld;

 import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
+import static org.joda.time.DateTimeZone.UTC;

 import com.google.common.base.Splitter;
 import com.google.common.collect.ImmutableMap;
@@ -23,11 +24,13 @@ import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedSet;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Maps;
+import google.registry.model.registry.label.PremiumList;
 import google.registry.model.registry.label.PremiumList.PremiumListEntry;
 import java.math.BigDecimal;
 import java.util.List;
 import java.util.Map;
 import org.joda.money.CurrencyUnit;
+import org.joda.time.DateTime;

 /** Static utility methods for {@link PremiumList}. */
 public class PremiumListUtils {
@@ -37,10 +40,7 @@ public class PremiumListUtils {
        Splitter.on('\n').omitEmptyStrings().splitToList(inputData);

    ImmutableMap<String, PremiumListEntry> prices =
-        new google.registry.model.registry.label.PremiumList.Builder()
-            .setName(name)
-            .build()
-            .parse(inputDataPreProcessed);
+        new PremiumList.Builder().setName(name).build().parse(inputDataPreProcessed);
    ImmutableSet<CurrencyUnit> currencies =
        prices.values().stream()
            .map(e -> e.getValue().getCurrencyUnit())
@@ -54,7 +54,12 @@ public class PremiumListUtils {

    Map<String, BigDecimal> priceAmounts =
        Maps.transformValues(prices, ple -> ple.getValue().getAmount());
-    return PremiumList.create(name, currency, priceAmounts);
+    return new PremiumList.Builder()
+        .setName(name)
+        .setCurrency(currency)
+        .setLabelsToPrices(priceAmounts)
+        .setCreationTime(DateTime.now(UTC))
+        .build();
  }

  private PremiumListUtils() {}
@@ -1,177 +0,0 @@
-// Copyright 2019 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.schema.tld;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkState;
-import static com.google.common.collect.ImmutableList.sortedCopyOf;
-import static com.google.common.collect.ImmutableList.toImmutableList;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
-
-import com.google.common.base.Joiner;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableMap;
-import google.registry.model.CreateAutoTimestamp;
-import google.registry.model.ImmutableObject;
-import google.registry.model.registry.label.ReservationType;
-import google.registry.schema.replay.DatastoreEntity;
-import google.registry.schema.replay.SqlEntity;
-import java.util.Map;
-import java.util.stream.Stream;
-import javax.annotation.Nullable;
-import javax.persistence.CollectionTable;
-import javax.persistence.Column;
-import javax.persistence.ElementCollection;
-import javax.persistence.Embeddable;
-import javax.persistence.Entity;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
-import javax.persistence.Id;
-import javax.persistence.Index;
-import javax.persistence.JoinColumn;
-import javax.persistence.MapKeyColumn;
-import javax.persistence.Table;
-import org.joda.time.DateTime;
-
-/**
- * A list of reserved domain labels that are blocked from being registered for various reasons.
- *
- * <p>Note that the primary key of this entity is {@link #revisionId}, which is auto-generated by
- * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
- * succeeds, we will end up with having two exact same reserved lists that differ only by
- * revisionId. This is fine though, because we only use the list with the highest revisionId.
- */
-@Entity
-@Table(indexes = {@Index(columnList = "name", name = "reservedlist_name_idx")})
-public class ReservedList extends ImmutableObject implements SqlEntity {
-
-  @Column(nullable = false)
-  private String name;
-
-  @Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
-  @Column(nullable = false)
-  private Long revisionId;
-
-  @Column(nullable = false)
-  private CreateAutoTimestamp creationTimestamp = CreateAutoTimestamp.create(null);
-
-  @Column(nullable = false)
-  private Boolean shouldPublish;
-
-  @ElementCollection
-  @CollectionTable(
-      name = "ReservedEntry",
-      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
-  @MapKeyColumn(name = "domainLabel")
-  private Map<String, ReservedEntry> labelsToReservations;
-
-  @Override
-  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
-    return ImmutableList.of(); // ReservedList is dual-written\
-  }
-
-  @Embeddable
-  public static class ReservedEntry extends ImmutableObject {
-    @Column(nullable = false)
-    private ReservationType reservationType;
-
-    @Column(nullable = true)
-    private String comment;
-
-    private ReservedEntry(ReservationType reservationType, @Nullable String comment) {
-      this.reservationType = reservationType;
-      this.comment = comment;
-    }
-
-    // Hibernate requires this default constructor.
-    private ReservedEntry() {}
-
-    /** Constructs a {@link ReservedEntry} object. */
-    public static ReservedEntry create(ReservationType reservationType, @Nullable String comment) {
-      return new ReservedEntry(reservationType, comment);
-    }
-
-    /** Returns the reservation type for this entry. */
-    public ReservationType getReservationType() {
-      return reservationType;
-    }
-
-    /** Returns the comment for this entry. Retruns null if there is no comment. */
-    public String getComment() {
-      return comment;
-    }
-  }
-
-  private ReservedList(
-      String name, Boolean shouldPublish, Map<String, ReservedEntry> labelsToReservations) {
-    this.name = name;
-    this.shouldPublish = shouldPublish;
-    this.labelsToReservations = labelsToReservations;
-  }
-
-  // Hibernate requires this default constructor.
-  private ReservedList() {}
-
-  /** Constructs a {@link ReservedList} object. */
-  public static ReservedList create(
-      String name, Boolean shouldPublish, Map<String, ReservedEntry> labelsToReservations) {
-    ImmutableList<String> invalidLabels =
-        labelsToReservations.entrySet().parallelStream()
-            .flatMap(
-                entry -> {
-                  String label = entry.getKey();
-                  if (label.equals(canonicalizeDomainName(label))) {
-                    return Stream.empty();
-                  } else {
-                    return Stream.of(label);
-                  }
-                })
-            .collect(toImmutableList());
-    checkArgument(
-        invalidLabels.isEmpty(),
-        "Label(s) [%s] must be in puny-coded, lower-case form",
-        Joiner.on(",").join(sortedCopyOf(invalidLabels)));
-    return new ReservedList(name, shouldPublish, labelsToReservations);
-  }
-
-  /** Returns the name of the reserved list. */
-  public String getName() {
-    return name;
-  }
-
-  /** Returns the ID of this revision, or throws if null. */
-  public Long getRevisionId() {
-    checkState(
-        revisionId != null,
-        "revisionId is null because this object has not been persisted to the database yet");
-    return revisionId;
-  }
-
-  /** Returns the creation time of this revision of the reserved list. */
-  public DateTime getCreationTimestamp() {
-    return creationTimestamp.getTimestamp();
-  }
-
-  /** Returns a {@link Map} of domain labels to {@link ReservedEntry}. */
-  public ImmutableMap<String, ReservedEntry> getLabelsToReservations() {
-    return ImmutableMap.copyOf(labelsToReservations);
-  }
-
-  /** Returns true if the reserved list should be published. */
-  public Boolean getShouldPublish() {
-    return shouldPublish;
-  }
-}
@@ -1,54 +0,0 @@
-// Copyright 2019 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.schema.tld;
-
-import static google.registry.config.RegistryConfig.getDomainLabelListCacheDuration;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
-import google.registry.util.NonFinalForTesting;
-import java.util.Optional;
-import org.joda.time.Duration;
-
-/** Caching utils for {@link ReservedList} */
-public class ReservedListCache {
-
-  /**
-   * In-memory cache for reserved lists.
-   *
-   * <p>This is cached for a shorter duration because we need to periodically reload from the DB to
-   * check if a new revision has been published, and if so, then use that.
-   */
-  @NonFinalForTesting
-  static LoadingCache<String, Optional<ReservedList>> cacheReservedLists =
-      createCacheReservedLists(getDomainLabelListCacheDuration());
-
-  @VisibleForTesting
-  static LoadingCache<String, Optional<ReservedList>> createCacheReservedLists(
-      Duration cachePersistDuration) {
-    return CacheBuilder.newBuilder()
-        .expireAfterWrite(cachePersistDuration.getMillis(), MILLISECONDS)
-        .build(
-            new CacheLoader<String, Optional<ReservedList>>() {
-              @Override
-              public Optional<ReservedList> load(String reservedListName) {
-                return ReservedListDao.getLatestRevision(reservedListName);
-              }
-            });
-  }
-}
@@ -28,6 +28,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Iterables;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.cmd.QueryKeys;
+import google.registry.flows.poll.PollFlowUtils;
 import google.registry.model.poll.PollMessage;
 import google.registry.model.poll.PollMessage.Autorenew;
 import google.registry.model.poll.PollMessage.OneTime;
@@ -49,8 +50,7 @@ import javax.inject.Inject;
 * delete confirmations) and it is desired that the registrar be able to ACK the rest of the poll
 * messages in-band.
 *
- * <p>This command only ACKs {@link OneTime} poll messages because that's our only use case so far,
- * but it could be extended to ACK {@link Autorenew} poll messages as well if needed. The main
+ * <p>This command ACKs both {@link OneTime} and {@link Autorenew} poll messages. The main
 * difference is that one-time poll messages are deleted when ACKed whereas Autorenews are sometimes
 * modified and re-saved instead, if the corresponding domain is still active.
 *
@@ -85,18 +85,17 @@ final class AckPollMessagesCommand implements CommandWithRemoteApi {
  @Override
  public void run() {
    QueryKeys<PollMessage> query = getPollMessagesQuery(clientId, clock.nowUtc()).keys();
+    // TODO(b/160325686): Remove the batch logic after db migration.
    for (List<Key<PollMessage>> keys : Iterables.partition(query, BATCH_SIZE)) {
-      tm()
-          .transact(
+      tm().transact(
              () -> {
                // Load poll messages and filter to just those of interest.
                ImmutableList<PollMessage> pollMessages =
                    ofy().load().keys(keys).values().stream()
-                        .filter(pm -> pm instanceof OneTime)
                        .filter(pm -> isNullOrEmpty(message) || pm.getMsg().contains(message))
                        .collect(toImmutableList());
                if (!dryRun) {
-                  ofy().delete().entities(pollMessages).now();
+                  pollMessages.forEach(PollFlowUtils::ackPollMessage);
                }
                pollMessages.forEach(
                    pm ->
@@ -50,7 +50,6 @@ import java.lang.annotation.RetentionPolicy;
 import java.nio.file.Files;
 import java.nio.file.Paths;
 import javax.annotation.Nullable;
-import javax.inject.Named;
 import javax.inject.Qualifier;
 import javax.inject.Singleton;

@@ -151,10 +150,10 @@ public class AuthModule {
  public static String provideLocalCredentialJson(
      Lazy<GoogleClientSecrets> clientSecrets,
      @StoredCredential Lazy<Credential> credential,
-      @Nullable @Named("credentialFileName") String credentialFilename) {
+      @Nullable @Config("credentialFilePath") String credentialFilePath) {
    try {
-      if (credentialFilename != null) {
-        return new String(Files.readAllBytes(Paths.get(credentialFilename)), UTF_8);
+      if (credentialFilePath != null) {
+        return new String(Files.readAllBytes(Paths.get(credentialFilePath)), UTF_8);
      } else {
        return new Gson()
            .toJson(
@@ -14,23 +14,12 @@

 package google.registry.tools;

-import static com.google.common.base.Preconditions.checkArgument;
-import static google.registry.model.registry.label.BaseDomainLabelList.splitOnComment;
-import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
-
 import com.beust.jcommander.Parameter;
-import com.google.common.base.Splitter;
-import com.google.common.collect.HashMultiset;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Multiset;
 import com.google.common.flogger.FluentLogger;
-import google.registry.model.registry.label.ReservationType;
-import google.registry.schema.tld.ReservedList.ReservedEntry;
+import google.registry.model.registry.label.ReservedList;
+import google.registry.model.registry.label.ReservedListDualWriteDao;
 import google.registry.tools.params.PathParameter;
 import java.nio.file.Path;
-import java.util.List;
-import java.util.Map;
 import javax.annotation.Nullable;

 /**
@@ -62,69 +51,22 @@ public abstract class CreateOrUpdateReservedListCommand extends MutatingCommand
      arity = 1)
  Boolean shouldPublish;

-  google.registry.schema.tld.ReservedList cloudSqlReservedList;
-
-  abstract void saveToCloudSql();
+  ReservedList reservedList;

  @Override
-  protected String execute() throws Exception {
-    // Save the list to Datastore and output its response.
-    String output = super.execute();
-    logger.atInfo().log(output);
-
-    String cloudSqlMessage =
+  protected String execute() {
+    String message =
        String.format(
            "Saved reserved list %s with %d entries",
-            name, cloudSqlReservedList.getLabelsToReservations().size());
+            name, reservedList.getReservedListEntries().size());
    try {
-      logger.atInfo().log("Saving reserved list to Cloud SQL for TLD %s", name);
-      saveToCloudSql();
-      logger.atInfo().log(cloudSqlMessage);
+      logger.atInfo().log("Saving reserved list for TLD %s", name);
+      ReservedListDualWriteDao.save(reservedList);
+      logger.atInfo().log(message);
    } catch (Throwable e) {
-      cloudSqlMessage =
-          "Unexpected error saving reserved list to Cloud SQL from nomulus tool command";
-      logger.atSevere().withCause(e).log(cloudSqlMessage);
+      message = "Unexpected error saving reserved list from nomulus tool command";
+      logger.atSevere().withCause(e).log(message);
    }
-    return cloudSqlMessage;
-  }
-
-  /** Turns the list CSV data into a map of labels to {@link ReservedEntry}. */
-  static ImmutableMap<String, ReservedEntry> parseToReservationsByLabels(Iterable<String> lines) {
-    Map<String, ReservedEntry> labelsToEntries = Maps.newHashMap();
-    Multiset<String> duplicateLabels = HashMultiset.create();
-    for (String originalLine : lines) {
-      List<String> lineAndComment = splitOnComment(originalLine);
-      if (lineAndComment.isEmpty()) {
-        continue;
-      }
-      String line = lineAndComment.get(0);
-      String comment = lineAndComment.get(1);
-      List<String> parts = Splitter.on(',').trimResults().splitToList(line);
-      checkArgument(
-          parts.size() == 2 || parts.size() == 3,
-          "Could not parse line in reserved list: %s",
-          originalLine);
-      String label = parts.get(0);
-      checkArgument(
-          label.equals(canonicalizeDomainName(label)),
-          "Label '%s' must be in puny-coded, lower-case form",
-          label);
-      ReservationType reservationType = ReservationType.valueOf(parts.get(1));
-      ReservedEntry reservedEntry = ReservedEntry.create(reservationType, comment);
-      // Check if the label was already processed for this list (which is an error), and if so,
-      // accumulate it so that a list of all duplicates can be thrown.
-      if (labelsToEntries.containsKey(label)) {
-        duplicateLabels.add(label, duplicateLabels.contains(label) ? 1 : 2);
-      } else {
-        labelsToEntries.put(label, reservedEntry);
-      }
-    }
-    if (!duplicateLabels.isEmpty()) {
-      throw new IllegalStateException(
-          String.format(
-              "Reserved list cannot contain duplicate labels. Dupes (with counts) were: %s",
-              duplicateLabels));
-    }
-    return ImmutableMap.copyOf(labelsToEntries);
+    return message;
  }
 }
@@ -16,7 +16,6 @@ package google.registry.tools;

 import static com.google.common.base.Preconditions.checkArgument;
 import static google.registry.model.registry.Registries.assertTldExists;
-import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
 import static google.registry.util.ListNamingUtils.convertFilePathToName;
 import static java.nio.charset.StandardCharsets.UTF_8;
 import static org.joda.time.DateTimeZone.UTC;
@@ -27,7 +26,6 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Splitter;
 import com.google.common.base.Strings;
 import google.registry.model.registry.label.ReservedList;
-import google.registry.schema.tld.ReservedListDao;
 import java.nio.file.Files;
 import java.util.List;
 import org.joda.time.DateTime;
@@ -50,15 +48,14 @@ final class CreateReservedListCommand extends CreateOrUpdateReservedListCommand
  protected void init() throws Exception {
    name = Strings.isNullOrEmpty(name) ? convertFilePathToName(input) : name;
    checkArgument(
-        !ReservedList.get(name).isPresent(),
-        "A reserved list already exists by this name");
+        !ReservedList.get(name).isPresent(), "A reserved list already exists by this name");
    if (!override) {
      validateListName(name);
    }
    DateTime now = DateTime.now(UTC);
    List<String> allLines = Files.readAllLines(input, UTF_8);
    boolean shouldPublish = this.shouldPublish == null || this.shouldPublish;
-    ReservedList reservedList =
+    reservedList =
        new ReservedList.Builder()
            .setName(name)
            .setReservedListMapFromLines(allLines)
@@ -66,23 +63,6 @@ final class CreateReservedListCommand extends CreateOrUpdateReservedListCommand
            .setCreationTime(now)
            .setLastUpdateTime(now)
            .build();
-    stageEntityChange(null, reservedList);
-    cloudSqlReservedList =
-        google.registry.schema.tld.ReservedList.create(
-            name, shouldPublish, parseToReservationsByLabels(allLines));
-  }
-
-  @Override
-  void saveToCloudSql() {
-    jpaTm()
-        .transact(
-            () -> {
-              checkArgument(
-                  !ReservedListDao.checkExists(cloudSqlReservedList.getName()),
-                  "A reserved list of this name already exists: %s.",
-                  cloudSqlReservedList.getName());
-              ReservedListDao.save(cloudSqlReservedList);
-            });
  }

  private static void validateListName(String name) {
@@ -16,17 +16,51 @@ package google.registry.tools;

 import com.beust.jcommander.Parameters;
 import google.registry.beam.spec11.Spec11Pipeline;
+import google.registry.config.CredentialModule.LocalCredential;
+import google.registry.config.RegistryConfig.Config;
+import google.registry.util.GoogleCredentialsBundle;
+import google.registry.util.Retrier;
+import javax.annotation.Nullable;
 import javax.inject.Inject;

 /** Nomulus command that deploys the {@link Spec11Pipeline} template. */
@Parameters(commandDescription = "Deploy the Spec11 pipeline to GCS.")
 public class DeploySpec11PipelineCommand implements Command {

-  @Inject Spec11Pipeline spec11Pipeline;
+  @Inject
+  @Config("projectId")
+  String projectId;
+
+  @Inject
+  @Config("beamStagingUrl")
+  String beamStagingUrl;
+
+  @Inject
+  @Config("spec11TemplateUrl")
+  String spec11TemplateUrl;
+
+  @Inject
+  @Config("reportingBucketUrl")
+  String reportingBucketUrl;
+
+  @Inject @LocalCredential GoogleCredentialsBundle googleCredentialsBundle;
+  @Inject Retrier retrier;
+
+  @Inject
+  @Nullable
+  @Config("sqlAccessInfoFile")
+  String sqlAccessInfoFile;

  @Override
  public void run() {
-    spec11Pipeline.deploy();
+    Spec11Pipeline pipeline =
+        new Spec11Pipeline(
+            projectId,
+            beamStagingUrl,
+            spec11TemplateUrl,
+            reportingBucketUrl,
+            googleCredentialsBundle,
+            retrier);
+    pipeline.deploy();
  }
 }
-
@@ -25,6 +25,7 @@ import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Sets;
 import com.googlecode.objectify.Key;
 import google.registry.batch.AsyncTaskEnqueuer;
+import google.registry.config.RegistryConfig.Config;
 import google.registry.model.billing.BillingEvent;
 import google.registry.model.billing.BillingEvent.Reason;
 import google.registry.model.domain.DomainBase;
@@ -52,13 +53,16 @@ public final class DomainLockUtils {
  private static final int VERIFICATION_CODE_LENGTH = 32;

  private final StringGenerator stringGenerator;
+  private final String registryAdminRegistrarId;
  private final AsyncTaskEnqueuer asyncTaskEnqueuer;

  @Inject
  public DomainLockUtils(
      @Named("base58StringGenerator") StringGenerator stringGenerator,
+      @Config("registryAdminClientId") String registryAdminRegistrarId,
      AsyncTaskEnqueuer asyncTaskEnqueuer) {
    this.stringGenerator = stringGenerator;
+    this.registryAdminRegistrarId = registryAdminRegistrarId;
    this.asyncTaskEnqueuer = asyncTaskEnqueuer;
  }

@@ -113,7 +117,7 @@ public final class DomainLockUtils {
              RegistryLock newLock =
                  RegistryLockDao.save(lock.asBuilder().setLockCompletionTimestamp(now).build());
              setAsRelock(newLock);
-              tm().transact(() -> applyLockStatuses(newLock, now));
+              tm().transact(() -> applyLockStatuses(newLock, now, isAdmin));
              return newLock;
            });
  }
@@ -167,7 +171,7 @@ public final class DomainLockUtils {
                      createLockBuilder(domainName, registrarId, registrarPocId, isAdmin)
                          .setLockCompletionTimestamp(now)
                          .build());
-              tm().transact(() -> applyLockStatuses(newLock, now));
+              tm().transact(() -> applyLockStatuses(newLock, now, isAdmin));
              setAsRelock(newLock);
              return newLock;
            });
@@ -217,19 +221,19 @@ public final class DomainLockUtils {
  private RegistryLock.Builder createLockBuilder(
      String domainName, String registrarId, @Nullable String registrarPocId, boolean isAdmin) {
    DateTime now = jpaTm().getTransactionTime();
-    DomainBase domainBase = getDomain(domainName, now);
-    verifyDomainNotLocked(domainBase);
+    DomainBase domainBase = getDomain(domainName, registrarId, now);
+    verifyDomainNotLocked(domainBase, isAdmin);

-    // Multiple pending actions are not allowed
+    // Multiple pending actions are not allowed for non-admins
    RegistryLockDao.getMostRecentByRepoId(domainBase.getRepoId())
        .ifPresent(
            previousLock ->
                checkArgument(
                    previousLock.isLockRequestExpired(now)
-                        || previousLock.getUnlockCompletionTimestamp().isPresent(),
+                        || previousLock.getUnlockCompletionTimestamp().isPresent()
+                        || isAdmin,
                    "A pending or completed lock action already exists for %s",
                    previousLock.getDomainName()));
-
    return new RegistryLock.Builder()
        .setVerificationCode(stringGenerator.createString(VERIFICATION_CODE_LENGTH))
        .setDomainName(domainName)
@@ -242,10 +246,12 @@ public final class DomainLockUtils {
  private RegistryLock.Builder createUnlockBuilder(
      String domainName, String registrarId, boolean isAdmin, Optional<Duration> relockDuration) {
    DateTime now = jpaTm().getTransactionTime();
-    DomainBase domainBase = getDomain(domainName, now);
+    DomainBase domainBase = getDomain(domainName, registrarId, now);
    Optional<RegistryLock> lockOptional =
        RegistryLockDao.getMostRecentVerifiedLockByRepoId(domainBase.getRepoId());

+    verifyDomainLocked(domainBase, isAdmin);
+
    RegistryLock.Builder newLockBuilder;
    if (isAdmin) {
      // Admins should always be able to unlock domains in case we get in a bad state
@@ -261,7 +267,6 @@ public final class DomainLockUtils {
                      .setLockCompletionTimestamp(now)
                      .setRegistrarId(registrarId));
    } else {
-      verifyDomainLocked(domainBase);
      RegistryLock lock =
          lockOptional.orElseThrow(
              () ->
@@ -289,24 +294,33 @@ public final class DomainLockUtils {
        .setRegistrarId(registrarId);
  }

-  private static void verifyDomainNotLocked(DomainBase domainBase) {
+  private static void verifyDomainNotLocked(DomainBase domainBase, boolean isAdmin) {
    checkArgument(
-        !domainBase.getStatusValues().containsAll(REGISTRY_LOCK_STATUSES),
+        isAdmin || !domainBase.getStatusValues().containsAll(REGISTRY_LOCK_STATUSES),
        "Domain %s is already locked",
        domainBase.getDomainName());
  }

-  private static void verifyDomainLocked(DomainBase domainBase) {
+  private static void verifyDomainLocked(DomainBase domainBase, boolean isAdmin) {
    checkArgument(
-        !Sets.intersection(domainBase.getStatusValues(), REGISTRY_LOCK_STATUSES).isEmpty(),
+        isAdmin
+            || !Sets.intersection(domainBase.getStatusValues(), REGISTRY_LOCK_STATUSES).isEmpty(),
        "Domain %s is already unlocked",
        domainBase.getDomainName());
  }

-  private static DomainBase getDomain(String domainName, DateTime now) {
-    return loadByForeignKeyCached(DomainBase.class, domainName, now)
-        .orElseThrow(
-            () -> new IllegalArgumentException(String.format("Unknown domain %s", domainName)));
+  private DomainBase getDomain(String domainName, String registrarId, DateTime now) {
+    DomainBase domain =
+        loadByForeignKeyCached(DomainBase.class, domainName, now)
+            .orElseThrow(() -> new IllegalArgumentException("Domain doesn't exist"));
+    // The user must have specified either the correct registrar ID or the admin registrar ID
+    checkArgument(
+        registryAdminRegistrarId.equals(registrarId)
+            || domain.getCurrentSponsorClientId().equals(registrarId),
+        "Domain %s is not owned by registrar %s",
+        domainName,
+        registrarId);
+    return domain;
  }

  private static RegistryLock getByVerificationCode(String verificationCode) {
@@ -317,9 +331,9 @@ public final class DomainLockUtils {
                    String.format("Invalid verification code %s", verificationCode)));
  }

-  private static void applyLockStatuses(RegistryLock lock, DateTime lockTime) {
-    DomainBase domain = getDomain(lock.getDomainName(), lockTime);
-    verifyDomainNotLocked(domain);
+  private void applyLockStatuses(RegistryLock lock, DateTime lockTime, boolean isAdmin) {
+    DomainBase domain = getDomain(lock.getDomainName(), lock.getRegistrarId(), lockTime);
+    verifyDomainNotLocked(domain, isAdmin);

    DomainBase newDomain =
        domain
@@ -330,11 +344,9 @@ public final class DomainLockUtils {
    saveEntities(newDomain, lock, lockTime, true);
  }

-  private static void removeLockStatuses(RegistryLock lock, boolean isAdmin, DateTime unlockTime) {
-    DomainBase domain = getDomain(lock.getDomainName(), unlockTime);
-    if (!isAdmin) {
-      verifyDomainLocked(domain);
-    }
+  private void removeLockStatuses(RegistryLock lock, boolean isAdmin, DateTime unlockTime) {
+    DomainBase domain = getDomain(lock.getDomainName(), lock.getRegistrarId(), unlockTime);
+    verifyDomainLocked(domain, isAdmin);

    DomainBase newDomain =
        domain
@@ -14,15 +14,7 @@

 package google.registry.tools;

-import static google.registry.model.EppResourceUtils.loadByForeignKey;
-
 import com.beust.jcommander.Parameters;
-import com.google.common.collect.ImmutableSet;
-import com.google.common.collect.Sets;
-import com.google.common.flogger.FluentLogger;
-import google.registry.model.domain.DomainBase;
-import google.registry.model.eppcommon.StatusValue;
-import org.joda.time.DateTime;

 /**
 * A command to registry lock domain names.
@@ -32,25 +24,6 @@ import org.joda.time.DateTime;
@Parameters(separators = " =", commandDescription = "Registry lock a domain via EPP.")
 public class LockDomainCommand extends LockOrUnlockDomainCommand {

-  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
-
-  @Override
-  protected boolean shouldApplyToDomain(String domain, DateTime now) {
-    DomainBase domainBase =
-        loadByForeignKey(DomainBase.class, domain, now)
-            .orElseThrow(
-                () ->
-                    new IllegalArgumentException(
-                        String.format("Domain '%s' does not exist or is deleted", domain)));
-    ImmutableSet<StatusValue> statusesToAdd =
-        Sets.difference(REGISTRY_LOCK_STATUSES, domainBase.getStatusValues()).immutableCopy();
-    if (statusesToAdd.isEmpty()) {
-      logger.atInfo().log("Domain '%s' is already locked and needs no updates.", domain);
-      return false;
-    }
-    return true;
-  }
-
  @Override
  protected void createAndApplyRequest(String domain) {
    domainLockUtils.administrativelyApplyLock(domain, clientId, null, true);
@@ -15,22 +15,24 @@
 package google.registry.tools;

 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static com.google.common.collect.Iterables.partition;
 import static google.registry.model.eppcommon.StatusValue.SERVER_DELETE_PROHIBITED;
 import static google.registry.model.eppcommon.StatusValue.SERVER_TRANSFER_PROHIBITED;
 import static google.registry.model.eppcommon.StatusValue.SERVER_UPDATE_PROHIBITED;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.CollectionUtils.findDuplicates;

 import com.beust.jcommander.Parameter;
 import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.flogger.FluentLogger;
 import google.registry.config.RegistryConfig.Config;
 import google.registry.model.eppcommon.StatusValue;
 import java.util.List;
 import javax.inject.Inject;
-import org.joda.time.DateTime;

 /** Shared base class for commands to registry lock or unlock a domain via EPP. */
 public abstract class LockOrUnlockDomainCommand extends ConfirmingCommand
@@ -78,37 +80,37 @@ public abstract class LockOrUnlockDomainCommand extends ConfirmingCommand
  @Override
  protected String execute() {
    ImmutableSet.Builder<String> successfulDomainsBuilder = new ImmutableSet.Builder<>();
-    ImmutableSet.Builder<String> skippedDomainsBuilder = new ImmutableSet.Builder<>();
-    ImmutableSet.Builder<String> failedDomainsBuilder = new ImmutableSet.Builder<>();
+    ImmutableMap.Builder<String, String> failedDomainsToReasons = new ImmutableMap.Builder<>();
    partition(getDomains(), BATCH_SIZE)
        .forEach(
            batch ->
-                tm().transact(
-                        () -> {
-                          for (String domain : batch) {
-                            if (shouldApplyToDomain(domain, tm().getTransactionTime())) {
-                              try {
-                                createAndApplyRequest(domain);
-                              } catch (Throwable t) {
-                                logger.atSevere().withCause(t).log(
-                                    "Error when (un)locking domain %s.", domain);
-                                failedDomainsBuilder.add(domain);
-                              }
-                              successfulDomainsBuilder.add(domain);
-                            } else {
-                              skippedDomainsBuilder.add(domain);
-                            }
-                          }
-                        }));
+                // we require that the jpaTm is the outer transaction in DomainLockUtils
+                jpaTm()
+                    .transact(
+                        () ->
+                            tm().transact(
+                                    () -> {
+                                      for (String domain : batch) {
+                                        try {
+                                          createAndApplyRequest(domain);
+                                        } catch (Throwable t) {
+                                          logger.atSevere().withCause(t).log(
+                                              "Error when (un)locking domain %s.", domain);
+                                          failedDomainsToReasons.put(domain, t.getMessage());
+                                          continue;
+                                        }
+                                        successfulDomainsBuilder.add(domain);
+                                      }
+                                    })));
    ImmutableSet<String> successfulDomains = successfulDomainsBuilder.build();
-    ImmutableSet<String> skippedDomains = skippedDomainsBuilder.build();
-    ImmutableSet<String> failedDomains = failedDomainsBuilder.build();
+    ImmutableSet<String> failedDomains =
+        failedDomainsToReasons.build().entrySet().stream()
+            .map(entry -> String.format("%s (%s)", entry.getKey(), entry.getValue()))
+            .collect(toImmutableSet());
    return String.format(
-        "Successfully locked/unlocked domains:\n%s\nSkipped domains:\n%s\nFailed domains:\n%s",
-        successfulDomains, skippedDomains, failedDomains);
+        "Successfully locked/unlocked domains:\n%s\nFailed domains:\n%s",
+        successfulDomains, failedDomains);
  }

-  protected abstract boolean shouldApplyToDomain(String domain, DateTime now);
-
  protected abstract void createAndApplyRequest(String domain);
 }
@@ -29,6 +29,7 @@ import com.google.appengine.tools.remoteapi.RemoteApiOptions;
 import com.google.common.base.Throwables;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Iterables;
+import google.registry.backup.AppEngineEnvironment;
 import google.registry.config.RegistryConfig;
 import google.registry.model.ofy.ObjectifyService;
 import google.registry.persistence.transaction.TransactionManagerFactory;
@@ -65,6 +66,13 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
              + "If not set, credentials saved by running `nomulus login' will be used.")
  private String credentialJson = null;

+  @Parameter(
+      names = {"--sql_access_info"},
+      description =
+          "Name of a file containing space-separated SQL access info used when deploying "
+              + "Beam pipelines")
+  private String sqlAccessInfoFile = null;
+
  // Do not make this final - compile-time constant inlining may interfere with JCommander.
  @ParametersDelegate
  private LoggingParameters loggingParams = new LoggingParameters();
@@ -153,11 +161,15 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
      return;
    }

-    checkState(RegistryToolEnvironment.get() == environment,
+    checkState(
+        RegistryToolEnvironment.get() == environment,
        "RegistryToolEnvironment argument pre-processing kludge failed.");

    component =
-        DaggerRegistryToolComponent.builder().credentialFilename(credentialJson).build();
+        DaggerRegistryToolComponent.builder()
+            .credentialFilePath(credentialJson)
+            .sqlAccessInfoFile(sqlAccessInfoFile)
+            .build();

    // JCommander stores sub-commands as nested JCommander objects containing a list of user objects
    // to be populated.  Extract the subcommand by getting the JCommander wrapper and then
@@ -167,7 +179,7 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
            Iterables.getOnlyElement(jcommander.getCommands().get(parsedCommand).getObjects());
    loggingParams.configureLogging();  // Must be called after parameters are parsed.

-    try {
+    try (AppEngineEnvironment env = new AppEngineEnvironment()) {
      runCommand(command);
    } catch (RuntimeException ex) {
      if (Throwables.getRootCause(ex) instanceof LoginRequiredException) {
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Michael Muller	f55270c46f	Integrate transaction persistence into JpaTM (#717 ) * Integrate transaction persistence into JpaTM Store the serialized transaction whenever we commit from the JPA transaction manager. This change also adds: - The Transaction table. - The TransactionEntity which is stored in it. - Changes to the test infrastructure to register the TransactionEntity for tests where we don't load the nomulus schema. - A new configuration variable to allow us to turn the transaction persistence functionality on and off (default is "off"). * Changes for review. * Incremented sequence number of flyway file	2020-07-28 19:23:44 -04:00
Ben McIlwain	d6d9874da1	Upgrade App Engine and webserver tests from JUnit 4 to 5 (#720 ) * Upgrade App Engine and webserver tests from JUnit 4 to 5 * Fix most errors * Merge branch 'master' into junit5ification * Fix test server by extracting non-test setup/tear-down * Merge branch 'master' into junit5ification * Fix backup tests * Don't createFile(); asCharSink does it * Increase the timeout for all WebDriver tests to 60s (helps w/ flakiness)	2020-07-28 14:18:16 -04:00
gbrodman	e0d04cec4f	Set up deployment of the Spec11 pipeline with JPA TM (#716 ) * Set up deployment of the Spec11 pipeline with JPA TM * Remove unnecessarily pipeline options setting * Use enviroment name in BeamJpaModuleTest * Fix checkstyle error	2020-07-27 21:04:52 -04:00
Michael Muller	0ce431212e	Add the :nom:generate_golden_schema pseudo-task (#718 ) Add a "pseudo-task" in nom_build to do the three step process of generating the golden schema. In the course of this, add support for pseudo-tasks in general, improve the database directory readme and make nom_build not call gradlew if there are no tasks.	2020-07-27 18:33:16 -04:00
gbrodman	32868b3ab8	Run the (Un)lockDomainCommand in an outer JPA txn (#688 ) * Run the (Un)lockDomainCommand in an outer JPA txn There are a couple things going on here in this commit. First, we add an external JPA transaction in the LockOrUnlockDomainCommand class. This doesn't appear to do much, but it avoids a situation similar to deadlock if an error occurs in Datastore when saving the domain object. Specifically, DomainLockUtils relies on the fact that any error in Datastore will be re-thrown in the JPA transaction, meaning that any Datastore error will back out of the SQL transaction as well. However, this is no longer true if we are already in a Datastore transaction when calling DomainLockUtils (unless, again, we are also in a JPA transaction). Basically, we require that the outer transaction is the JPA one. Secondly, this just allows for more breakglass operations in the lock or unlock domain commands -- in a situation where things possibly go haywire, we should allow admins to make sure with certainty that a domain is locked or unlocked. * Add more robustness and tests for admins locking locked domains * Fix expected exception message in tests	2020-07-27 18:16:24 -04:00
Shicong Huang	0ecc20b48c	Rename a V40 flyway file to V41 to resolve conflict (#719 )	2020-07-27 15:16:01 -04:00
Shicong Huang	c65af4b480	Add remaining columns to Domain's SQL schema (#702 )	2020-07-27 13:32:39 -04:00
Legina Chen	3a15a8bdc7	Drop foreign key constraint for Registrar table (#715 )	2020-07-27 09:05:30 -07:00
Weimin Yu	9806fab880	Use rearranged sql credentials in flyway task (#712 ) * Use rearranged sql credentials in flyway task Let the flyway tasks use the sql credential files set up for BEAM pipelines. Credential files have been created for each environment in GCS at gs://${project}-beam/cloudsql/admin_credential.enc. All project editors have access to this file, including the Dataflow control service account. Alpha and crash use the 'nomulus-tools-key' in their own project to decrypt the credential file. Sandbox and production use the 'nomulus-tools-key' in domain-registry-dev to decrypt the credential file. Note that this setup is temporary. It will become obsolete once we migrate to Cloud Secret Manager for secret storage.	2020-07-24 15:32:01 -04:00
Weimin Yu	6591e0672a	End-to-end Datastore to SQL pipeline (#707 ) * End-to-end Datastore to SQL pipeline Defined InitSqlPipeline that performs end-to-end migration from a Datastore backup to a SQL database. Also fixed/refined multiple tests related to this migration.	2020-07-24 09:57:43 -04:00
Ben McIlwain	91b7d92cf8	Upgrade TestPipeline extension from JUnit 4 to 5	2020-07-23 21:21:58 -04:00
Ben McIlwain	33910613da	Get presubmits passing This involves Guava -> Java 8 util migrations and fixing the license header.	2020-07-23 21:21:58 -04:00
Ben McIlwain	1fde678250	Copy TestPipeline rule from Apache Beam project into our codebase This is copied in here with the absolute minimum # of modifications required (just a rename to JUnit 5 format and some small fixes required to enable compilation to be successful). This is in preparation for the next commit where I'll convert this Rule into a JUnit 5 extension, which is the entire goal here. But I wanted to get the code from Apache Beam in with the maximum possible fidelity so that my changes will be in a separate commit and will thus be obvious. Note that we do unfortunately need to modify/rewrite the Rule itself; merely wrapping it in some manner isn't possible.	2020-07-23 21:21:58 -04:00
gbrodman	8d56577653	Don't run presubmits over the .git folder (#711 )	2020-07-23 18:12:34 -04:00
Ben McIlwain	3891d411de	Upgrade most of remaining tests from JUnit 4 to JUnit 5 (#708 )	2020-07-23 15:43:59 -04:00
gbrodman	cadecb15d8	Rename the email field in UI and include rlock email if it exists (#697 ) * Rename the email field in UI and include rlock email if it exists * Change the capitalization of fields and titles and add a description	2020-07-23 14:30:12 -04:00
gbrodman	9b7f6ce500	Fix some SQL credential issues identified when deploying Beam pipelines (#706 ) * Fix some SQL credential issues identified when deploying Beam pipelines There are two issues fixed here. 1. Without calling `FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create()), the Nomulus tool doesn't know how to handle gs:// scheme files. Thus, if you try to deploy (for instance) the Spec11 pipeline using a GCS credential file, it fails. 2. There was a misunderstanding before about what the credential file actually refers to -- there is a credential file in JSON format that is used for gcloud authorization, and there is a space-delimited SQL access info file that has the instance name, username, and password. These are separate options and should have separate command-line params. * Actually we don't need this for remote deployment	2020-07-22 16:52:31 -04:00
Ben McIlwain	cd23748fe8	Upgrade rest of tools test classes to JUnit 5 (#705 )	2020-07-22 11:09:21 -04:00
Ben McIlwain	cf41f5d354	Upgrade all remaining flows tests to JUnit 5 (#704 )	2020-07-21 19:52:33 -04:00
Ben McIlwain	9a5ba249db	Upgrade converters/TMCH/RDAP to JUnit 5 (#703 ) Also renames some existing Rules to Extensions (and removes JUnit 4 features from them entirely if no longer being used).	2020-07-21 18:48:41 -04:00
Shicong Huang	f5186f8476	Merge two PremiumList entities (#690 )	2020-07-21 18:18:52 -04:00
Lai Jiang	4e0ca19d2e	Remove IDN elements from BRDA (#670 ) Also added unit tests for RdeStagingReducer.	2020-07-21 15:29:32 -04:00
Ben McIlwain	c812807ab3	Upgrade mapreduce and DNS tests from JUnit 4 to JUnit 5 (#701 ) * Upgrade mapreduce and DNS tests from JUnit 4 to JUnit 5 * Merge branch 'master' into junit5-batch-and-dns	2020-07-20 21:33:24 -04:00
Ben McIlwain	9edb43f3e4	Upgrade command test classes from JUnit 4 to JUnit 5 (#700 ) * Convert first batch of command tests to JUnit 5 * Upgrade rest of command tests to JUnit 5 * Migrate the last few test classes	2020-07-20 20:45:52 -04:00
gbrodman	b721533759	Create an ImmutableObjectSubject for comparing SQL objects (#695 ) * Create an ImmutableObjectSubject for comparing SQL objects Many times, when comparing objects that are loaded in from / saved to SQL in tests, there are some fields we don't care about. Specifically, we might not care about the last update time, revision ID, or other things like that that are autoassigned by the DB. If we use this, we can ignore those fields while still comparing the other ones. * Create an ImmutableObject Correspondence for more flexible usage	2020-07-20 13:14:09 -04:00
gbrodman	ce35f6bc93	Include the user's registry lock email in the lock/unlock modal (#696 ) * Include the user's registry lock email in the lock/unlock modal	2020-07-20 12:01:34 -04:00
gbrodman	f7a67b7676	Add a 'Host' parameter to the relock action enqueuer (#699 ) * Add a 'Host' parameter to the relock action enqueuer I believe this is why we are seeing 404s currently -- we should be specifying the backend host as the target like we do for the resave-entity async action.	2020-07-17 15:35:44 -04:00
gbrodman	4438944900	Validate potentially-invalid domain names when (un)locking domains (#698 ) * Validate potentially-invalid domain names when (un)locking domains	2020-07-17 12:05:19 -04:00
Legina Chen	a22998e1bc	Change clientId to registrarId to resolve the bug wrt the mismatch of the variable names (#692 )	2020-07-15 11:03:52 -07:00
Weimin Yu	03d02ab299	Fix JpaIntegrationRule in JUnit4 (#687 ) * Fix JpaIntegrationRule in JUnit4 Made DatastoreExtension a JUnit4 Rule. Nomulus model objects need Datastore API when manipulating Ofy keys. As a result, JpaIntegrationTestRule must be used with AppEngineRule or DatastoreExtension. Also fixed WriteToSqlTest, which is the only JUnit4 test that uses JpaIntegrationTestRule.	2020-07-15 10:42:33 -04:00
Lai Jiang	47f65f70ab	Fix a typo (#689 )	2020-07-15 10:39:11 -04:00
Weimin Yu	1aa1f351bf	Run rdeStaging twice daily in Sandbox (#684 ) * Run rdeStaging twice daily in Sandbox This will allow the cursor to catch up to current date if it somehow falls behind.	2020-07-14 14:54:34 -04:00
Weimin Yu	94c8c6b9f3	Add lastUpdateTime column to epp resources (#683 ) * Add lastUpdateTime column to epp resources Property was inadvertently left out. Renamed getter and setter to match the property name. Added a test helper to compare EppResources while ignoring lastUpdateTime, which changes every time an instance is persisted.	2020-07-14 14:53:05 -04:00
gbrodman	e74a9e6f02	Allow overrides of ContactBase methods (#681 ) Hibernate might (will?) need to override these, so they shouldn't be final.	2020-07-14 14:47:47 -04:00
gbrodman	37d3cc44b4	Fix small naming issue in a test (#685 )	2020-07-14 13:57:44 -04:00
Lai Jiang	c844c8e9b1	Add the ability to parse PKCS#8 private key in PEM file (#682 )	2020-07-14 11:20:00 -04:00
gbrodman	f747610533	Include the relock action in the web.xml routing file (#680 )	2020-07-13 21:57:35 -04:00
Shicong Huang	e1db357fc3	Merge two reserved list entities (#616 ) * Merge reserved list * Replace INSTANCE with getInstance() * Fix broken test * Rebase on master * Simplify class	2020-07-13 13:40:34 -04:00
Weimin Yu	ba1915e271	Write one PCollection to SQL (#664 ) * Write one PCollection to SQL Defined a transform that writes a PCollection of entities to SQL using JPA. Allows configuring parallelism level and batch size.	2020-07-13 13:34:01 -04:00
Shicong Huang	58618a274e	Add two folders of auto-generated Java classes to .gitignore (#679 )	2020-07-13 10:09:56 -04:00
Lai Jiang	e4d0571125	Increase the maximum number of nodes in a nood pool to 15 (#672 )	2020-07-10 21:54:18 -04:00
Ben McIlwain	4cb88ab6e7	Convert RDE tests (and some test rules) from JUnit 4 to JUnit 5 (#677 ) * Add JUnit Params and start using it * Convert rest of RDE tests * Don't check headers for generated tests * Expand visibility to fix build breakage * Bump JUnit versions to 5.6.2	2020-07-10 21:32:36 -04:00
gbrodman	987f390ff7	Run 'npm audit fix' to fix low-severity vulnerabilities in packages (#676 )	2020-07-10 15:57:59 -04:00
Ben McIlwain	ca756e14e6	Migrate all model tests from JUnit 4 to JUnit 5 (#675 ) * Make first handful of tests JUnit 5 * Migrate rest of model package to JUnit 5	2020-07-10 14:56:28 -04:00
Ben McIlwain	caa0cd9d61	Add a "coreDev" gradle target (#667 ) * Add a "buildFmt" gradle target This does the same thing as the automatic Java build target, except instead of failing if the code formatting isn't correct, it just automatically reformats as necessary and continues on. * Remove unnecessary mustRunAfters * Make it run tests too, and add :taskTree task * Rename task to coreDev and remove run afters * Add task tree dependency * Actually that may not be necessary	2020-07-10 10:03:59 -04:00
Legina Chen	7806cc7edb	Add domainRepoId to Subdomain class (#674 ) * Change Subdomain class to contain domainRepoId * Remove jpaTm from Spec11PipelineTest and change clientId -> registrarId * Remove 'client' from a comment * Include changes to Spec11Pipeline * add SafeBrowsingTransforms * Run style	2020-07-09 16:26:35 -07:00
Lai Jiang	0964fdf1dc	Upgrade to Gradle 6.5.1 (#673 )	2020-07-09 14:04:22 -04:00
gbrodman	d17ec1fcb1	Use an enum instead of boolean in EntityTestCase constructor (#669 ) * Use an enum instead of boolean in EntityTestCase constructor It's more clear to use an enum rather than just a simple boolean * Add Javadoc and make the enum name more verbose	2020-07-09 12:54:32 -04:00
Ben McIlwain	fac5987c13	Double the # of pubapi instances to better handle traffic spikes (#671 ) * Double the # of pubapi instances to better handle traffic spikes We may also consider switching to an automatic scaling mode soon, on the hope that it's working better than the last time we tried it (it would help to keep resource costs down at least).	2020-07-09 11:52:15 -04:00
Ben McIlwain	a3319e0026	Upgrade flow test classes to JUnit 5 (#666 ) Most of the diffs are visibility changes. Also deletes ShardableTestCase, which was only necessary because of Blaze (and possible Bazel) limitations.	2020-07-08 14:08:05 -04:00
Weimin Yu	5578464e06	Make sure uncommitted txn is rolled back (#665 ) * Make sure uncommit txn is rolled back The try block around commit that catches RuntimeException should also catch Error, which is also unchecked.	2020-07-06 17:39:13 -04:00
gbrodman	c24a61f813	Refactor ContactResource into ContactBase and create ContactHistory (#634 ) * Create ContactHistory class + table This is similar to #587, but with contacts instead of hosts. This also includes a couple cleanups for HostHistoryTest and RegistryLockDaoTest, just making code more proper (we shouldn't be referencing constant revision IDs when using a sequence that is used by multiple classes, and RLDT can extend EntityTest) Note as well that we set ContactHistory to use the same revision ID sequence as HostHistory. * Move ContactResource -> ContactBase * Alter ContactBase and ContactResource	2020-07-06 12:52:16 -04:00
gbrodman	806f3b2456	Verify that the RegistryLock input has the correct registrar ID (#661 ) * Verify that the RegistryLock input has the correct registrar ID We already verify (correctly) that the user has access to the registrar they specify, but nowhere did we verify that the registrar ID they used is actually the current sponsor ID for the domain in question. This is an oversight caused by the fact that our testing framework only uses admin accounts, which by the nature of things have access to all registrars and domains. In addition, rename "clientId" to "registrarId" in the RLPA object * Change the wording on the incorrect-registrar message	2020-07-05 22:31:14 -04:00
gbrodman	333170a724	Allow users the option of seeing their registry lock password (#663 ) * Allow users the option of seeing their registry lock password Only when entering it for the first time, of course.	2020-07-05 20:08:22 -04:00
Lai Jiang	47eeb8c4e4	Output PO number in detailed report (#659 ) * Output PO number in detailed report The PO number header was added during the beam migration but we forgot to print the actual data in the corresponding column. This resulted in a misalignment of columns in the detailed report. This PR fixes it. Note that we cannot drop PO number from the header (as is not useful in the detailed report) because the header represents all fields that are to be parsed from the SQL query results, and PO number is needed when generating the invoice itself. By dual-purposing the header (both as the required fields in the parser and the first line in the detailed report) we have to include the value of PO number in the detailed report CSV as well.	2020-07-01 19:09:05 -04:00
Shicong Huang	391929b518	Expand AckPollMessagesCommand to ack PollMessage.Autorenew (#647 ) * Expand AckPollMessagesCommand to ack PollMessage.Autorenew * Rebase on master and address comment * Resolve comments	2020-07-01 15:06:35 -04:00
gbrodman	7f62b7a89c	Include the registry lock email in the JS object as a sensitive field (#658 ) * Include the registry lock email in the JS object as a sensitive field * Change wording of exceptions to be more consistent	2020-07-01 13:05:21 -04:00
gbrodman	a1da32bfde	Disambiguate injected Cloud SQL parameter names (#657 ) * Disambiguate injected Cloud SQL parameter names This allows us to also inject the BeamJpaModule into RegistryTool, which allows us to use the SocketJpaTransactionManager in Beam pipelines. Some side effects of this include: - duplication of KMS connections -- one standard, one Beam - duplication of the creation of the partial Hibernate SQL configs - removal of ambiguity between credentialFileName, credentialFilename, and credentialFilePath -- we now use the latter. - Performing the credential null check when instantiating the SQL connection rather than when instantiating the module object. See the code comments for more details on this. I verified that this compiles and the tests run successfully when injecting a @SocketFactoryJpaTm into a Beam pipeline. * Remove two unnecessary config points and change the name of two params * Use @Config instead of @Named and change the pool size * Replace non-visible link with code	2020-07-01 11:55:21 -04:00
Weimin Yu	1961a5759d	Load Datastore snapshot from backup files (#660 ) * Load Datastore snapshot from backup files Defined a composite transform that loads from a Datastore export and concurrent CommitLog files, identify entities that still exist at the end of the time window, and resolve their latest states in the window.	2020-07-01 09:58:42 -04:00