Replace ofyOrJpaTm with tm().isOfy() conditionals (#870 )

* Replace ofyOrJpaTm with tm().isOfy() conditionals Replace existing ofyOrJpaTm() calls with conditionals (either "if" statements or ternary expressions) gated on tm().isOfy().
Maintain only one row in the TmchCrl table at all times (#869 )
2026-08-04 06:16:12 +00:00 · 2020-11-11 07:40:43 -05:00 · 2020-11-10 17:09:26 -05:00 · 2020-11-10 17:09:18 -05:00 · 2020-11-10 11:12:09 -05:00 · 2020-11-10 10:03:28 -05:00
294 changed files with 31712 additions and 2356 deletions
@@ -1,25 +1,28 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto.value:auto-value:1.6.3
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -24,6 +24,7 @@ import java.time.ZonedDateTime;
 import java.util.TimeZone;
 import org.joda.time.DateTime;
 import org.joda.time.DateTimeZone;
+import org.joda.time.LocalDate;

 /** Utilities methods and constants related to Joda {@link DateTime} objects. */
 public class DateTimeUtils {
@@ -108,4 +109,12 @@ public class DateTimeUtils {
        zonedDateTime.toInstant().toEpochMilli(),
        DateTimeZone.forTimeZone(TimeZone.getTimeZone(zonedDateTime.getZone())));
  }
+
+  public static java.sql.Date toSqlDate(LocalDate localDate) {
+    return new java.sql.Date(localDate.toDateTimeAtStartOfDay().getMillis());
+  }
+
+  public static LocalDate toLocalDate(java.sql.Date date) {
+    return new LocalDate(date.getTime(), DateTimeZone.UTC);
+  }
 }
@@ -18,7 +18,6 @@ import static com.google.common.base.Preconditions.checkArgument;

 import com.google.common.util.concurrent.Uninterruptibles;
 import java.io.Serializable;
-import java.util.concurrent.TimeUnit;
 import javax.annotation.concurrent.ThreadSafe;
 import javax.inject.Inject;
 import org.joda.time.ReadableDuration;
@@ -41,6 +40,6 @@ public final class SystemSleeper implements Sleeper, Serializable {
  @Override
  public void sleepUninterruptibly(ReadableDuration duration) {
    checkArgument(duration.getMillis() >= 0);
-    Uninterruptibles.sleepUninterruptibly(duration.getMillis(), TimeUnit.MILLISECONDS);
+    Uninterruptibles.sleepUninterruptibly(java.time.Duration.ofMillis(duration.getMillis()));
  }
 }
@@ -16,7 +16,7 @@
 """

 import argparse
-import attr
+import dataclasses
 import io
 import os
 import shutil
@@ -25,7 +25,7 @@ import sys
 from typing import List, Union


-@attr.s(auto_attribs=True)
+@dataclasses.dataclass
 class Property:
    name : str = ''
    desc : str = ''
@@ -39,7 +39,7 @@ class Property:
                raise ValidationError('value of {self.name} must be "true" or '
                                      '"false".')

-@attr.s(auto_attribs=True)
+@dataclasses.dataclass
 class GradleFlag:
    flags : Union[str, List[str]]
    desc : str
@@ -300,9 +300,15 @@ def do_pseudo_task(task: str) -> None:
                    f'{root}/db/src/main/resources/sql/schema/'
                    'nomulus.golden.sql')

-        if subprocess.call([f'{root}/gradlew', ':db:test']):
-            print('\033[31mERROR:\033[0m Golden file test failed after '
-                  'copying schema.  Please check your flyway files.')
+        # Rerun :db:test and regenerate the ER diagram (at "warning" log
+        # level so it doesn't generate pages of messaging)
+        if subprocess.call([f'{root}/gradlew', ':db:test', 'devTool',
+                            '--args=-e localhost --log_level=WARNING '
+                            'generate_sql_er_diagram -o '
+                            f'{root}/db/src/main/resources/sql/er_diagram']):
+            print('\033[31mERROR:\033[0m Golden file test or ER diagram '
+                  'generation failed after copying schema.  Please check your '
+                  'flyway files.')
            raise Abort()
    else:
        print(f'\033[31mERROR:\033[0m Unknown task {task}')
@@ -377,6 +383,7 @@ def main(args) -> int:

    # See if there are any special ":nom:" pseudo-tasks specified.
    got_non_pseudo_tasks = False
+    got_pseudo_tasks = False
    for arg in args.non_flag_args[1:]:
        if arg.startswith(':nom:'):
            if got_non_pseudo_tasks:
@@ -388,13 +395,14 @@ def main(args) -> int:
                      'specified prior to all actual gradle tasks.  Aborting.')
                return 1
            do_pseudo_task(arg)
+            got_pseudo_tasks = True
        else:
            got_non_pseudo_tasks = True
    non_flag_args = [
        arg for arg in args.non_flag_args[1:] if not arg.startswith(':nom:')]

    if not non_flag_args:
-        if not got_non_pseudo_tasks:
+        if not got_pseudo_tasks:
            print('\033[33mWARNING:\033[0m No tasks specified.  Not '
                  'doing anything')
        return 0
@@ -113,7 +113,10 @@ class MyTest(unittest.TestCase):
        nom_build.main(['nom_build', ':nom:generate_golden_file'])
        self.call_mock.assert_has_calls([
            mock.call([GRADLEW, ':db:test']),
-            mock.call([GRADLEW, ':db:test'])
+            mock.call([GRADLEW, ':db:test', 'devTool',
+                       '--args=-e localhost --log_level=WARNING '
+                       'generate_sql_er_diagram -o '
+                       '/tmp/rootdir/db/src/main/resources/sql/er_diagram'])
        ])

    def test_generate_golden_file_nofail(self):
@@ -122,5 +125,3 @@ class MyTest(unittest.TestCase):
        self.call_mock.assert_has_calls([mock.call([GRADLEW, ':db:test'])])

 unittest.main()
-
-
@@ -220,7 +220,9 @@ dependencies {
  compile deps['com.googlecode.json-simple:json-simple']
  compile deps['com.jcraft:jsch']
  testCompile deps['com.thoughtworks.qdox:qdox']
+  compile deps['com.zaxxer:HikariCP']
  compile deps['dnsjava:dnsjava']
+  runtime deps['guru.nidi:graphviz-java-all-j2v8']
  testCompile deps['io.github.classgraph:classgraph']
  testRuntime deps['io.github.java-diff-utils:java-diff-utils']
  testCompile deps['javax.annotation:javax.annotation-api']
@@ -256,8 +258,10 @@ dependencies {
  testCompile deps['com.fasterxml.jackson.core:jackson-databind']
  runtime deps['org.glassfish.jaxb:jaxb-runtime']
  compile deps['org.hibernate:hibernate-core']
+  compile deps['org.hibernate:hibernate-hikaricp']
  compile deps['org.joda:joda-money']
  compile deps['org.json:json']
+  compile deps['org.jsoup:jsoup']
  testCompile deps['org.mortbay.jetty:jetty']
  compile deps['org.postgresql:postgresql']
  testCompile deps['org.seleniumhq.selenium:selenium-api']
@@ -270,6 +274,10 @@ dependencies {
  compile deps['org.testcontainers:postgresql']
  testCompile deps['org.testcontainers:selenium']
  testCompile deps['org.testcontainers:testcontainers']
+  compile deps['us.fatehi:schemacrawler']
+  compile deps['us.fatehi:schemacrawler-api']
+  compile deps['us.fatehi:schemacrawler-diagram']
+  compile deps['us.fatehi:schemacrawler-tools']
  compile deps['xerces:xmlParserAPIs']
  compile deps['xpp3:xpp3']
  // This dependency must come after javax.mail:mail as it would otherwise
@@ -304,7 +312,6 @@ dependencies {
  testCompile deps['org.hamcrest:hamcrest-all']
  testCompile deps['org.hamcrest:hamcrest-core']
  testCompile deps['org.hamcrest:hamcrest-library']
-  compile deps['org.hibernate:hibernate-hikaricp']
  testCompile deps['junit:junit']
  testCompile deps['org.junit.jupiter:junit-jupiter-api']
  testCompile deps['org.junit.jupiter:junit-jupiter-engine']
@@ -1,8 +1,8 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto.value:auto-value:1.6.3
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
@@ -11,11 +11,11 @@ com.google.dagger:dagger-compiler:2.28
 com.google.dagger:dagger-producers:2.28
 com.google.dagger:dagger-spi:2.28
 com.google.dagger:dagger:2.28
-com.google.errorprone:error_prone_annotation:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
 com.google.errorprone:error_prone_annotations:2.3.4
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.errorprone:javac-shaded:9-dev-r4023-3
 com.google.googlejavaformat:google-java-format:1.5
 com.google.guava:failureaccess:1.0.1
@@ -30,11 +30,14 @@ javax.inject:javax.inject:1
 javax.persistence:javax.persistence-api:2.2
 net.ltgt.gradle.incap:incap:0.2
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:2.11.1
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.jetbrains.kotlin:kotlin-stdlib-common:1.3.61
 org.jetbrains.kotlin:kotlin-stdlib:1.3.61
 org.jetbrains.kotlinx:kotlinx-metadata-jvm:0.1.0
 org.jetbrains:annotations:13.0
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -125,7 +125,7 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
@@ -176,7 +176,7 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -202,9 +202,10 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -215,16 +216,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -234,7 +236,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -124,7 +124,7 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
@@ -174,7 +174,7 @@ javax.validation:validation-api:1.0.0.GA
 javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -200,6 +200,7 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -212,16 +213,17 @@ org.easymock:easymock:3.0
 org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -231,7 +233,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -246,5 +248,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.10.2
 com.fasterxml.jackson.core:jackson-core:2.10.2
 com.fasterxml.jackson.core:jackson-databind:2.10.2
@@ -126,10 +130,13 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.grpc:grpc-all:1.27.2
@@ -177,7 +184,8 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.arnx:nashorn-promise:0.1.1
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -198,15 +206,17 @@ org.apache.beam:beam-vendor-grpc-1_26_0:0.3
 org.apache.beam:beam-vendor-guava-26_0-jre:0.1
 org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
 org.apache.commons:commons-compress:1.20
+org.apache.commons:commons-exec:1.3
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.13.3
 org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -217,16 +227,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -236,11 +247,13 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
 org.scijava:native-lib-loader:2.0.2
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.slf4j:slf4j-jdk14:1.7.28
 org.testcontainers:database-commons:1.14.3
@@ -250,7 +263,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.10.2
 com.fasterxml.jackson.core:jackson-core:2.10.2
 com.fasterxml.jackson.core:jackson-databind:2.10.2
@@ -126,10 +130,13 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.grpc:grpc-all:1.27.2
@@ -176,7 +183,8 @@ javax.validation:validation-api:1.0.0.GA
 javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.arnx:nashorn-promise:0.1.1
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -197,15 +205,17 @@ org.apache.beam:beam-vendor-grpc-1_26_0:0.3
 org.apache.beam:beam-vendor-guava-26_0-jre:0.1
 org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
 org.apache.commons:commons-compress:1.20
+org.apache.commons:commons-exec:1.3
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.13.3
 org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -215,16 +225,17 @@ org.easymock:easymock:3.0
 org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -234,11 +245,13 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
 org.scijava:native-lib-loader:2.0.2
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.slf4j:slf4j-jdk14:1.7.28
 org.testcontainers:database-commons:1.14.3
@@ -248,7 +261,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -1,24 +1,27 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
 com.google.code.findbugs:jsr305:3.0.2
-com.google.errorprone:error_prone_annotation:2.3.3
-com.google.errorprone:error_prone_annotations:2.3.3
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
+com.google.errorprone:error_prone_annotations:2.3.4
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.guava:failureaccess:1.0.1
 com.google.guava:guava:27.0.1-jre
 com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
 com.google.j2objc:j2objc-annotations:1.1
 com.google.protobuf:protobuf-java:3.4.0
 com.googlecode.java-diff-utils:diffutils:1.3.0
-org.checkerframework:checker-qual:2.5.3
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.codehaus.mojo:animal-sniffer-annotations:1.17
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -125,7 +125,7 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
@@ -176,7 +176,7 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -202,9 +202,10 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -215,16 +216,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -234,7 +236,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -124,7 +124,7 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
@@ -175,7 +175,7 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -201,6 +201,7 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -214,16 +215,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -233,7 +235,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -248,5 +250,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.10.2
 com.fasterxml.jackson.core:jackson-core:2.10.2
 com.fasterxml.jackson.core:jackson-databind:2.10.2
@@ -125,10 +129,13 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.grpc:grpc-all:1.27.2
@@ -176,7 +183,8 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.arnx:nashorn-promise:0.1.1
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -197,15 +205,17 @@ org.apache.beam:beam-vendor-grpc-1_26_0:0.3
 org.apache.beam:beam-vendor-guava-26_0-jre:0.1
 org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
 org.apache.commons:commons-compress:1.20
+org.apache.commons:commons-exec:1.3
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.13.3
 org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -216,16 +226,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -235,11 +246,13 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
 org.scijava:native-lib-loader:2.0.2
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.testcontainers:database-commons:1.14.3
 org.testcontainers:jdbc:1.14.3
@@ -248,7 +261,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.10.2
 com.fasterxml.jackson.core:jackson-core:2.10.2
 com.fasterxml.jackson.core:jackson-databind:2.10.2
@@ -125,10 +129,13 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.grpc:grpc-all:1.27.2
@@ -176,7 +183,8 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.arnx:nashorn-promise:0.1.1
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -197,15 +205,17 @@ org.apache.beam:beam-vendor-grpc-1_26_0:0.3
 org.apache.beam:beam-vendor-guava-26_0-jre:0.1
 org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
 org.apache.commons:commons-compress:1.20
+org.apache.commons:commons-exec:1.3
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.13.3
 org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -216,16 +226,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -235,11 +246,13 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
 org.scijava:native-lib-loader:2.0.2
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.testcontainers:database-commons:1.14.3
 org.testcontainers:jdbc:1.14.3
@@ -248,7 +261,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.10.2
 com.fasterxml.jackson.core:jackson-core:2.10.2
 com.fasterxml.jackson.core:jackson-databind:2.10.2
@@ -125,10 +129,13 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.grpc:grpc-all:1.27.2
@@ -176,7 +183,8 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.12
-net.bytebuddy:byte-buddy:1.10.10
+net.arnx:nashorn-promise:0.1.1
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -197,15 +205,17 @@ org.apache.beam:beam-vendor-grpc-1_26_0:0.3
 org.apache.beam:beam-vendor-guava-26_0-jre:0.1
 org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
 org.apache.commons:commons-compress:1.20
+org.apache.commons:commons-exec:1.3
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.13.3
 org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -216,16 +226,17 @@ org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-core:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -235,11 +246,13 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
 org.scijava:native-lib-loader:2.0.2
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.testcontainers:database-commons:1.14.3
 org.testcontainers:jdbc:1.14.3
@@ -248,7 +261,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.10.2
 com.fasterxml.jackson.core:jackson-core:2.10.2
 com.fasterxml.jackson.core:jackson-databind:2.10.2
@@ -126,10 +130,13 @@ com.squareup.okio:okio:1.13.0
 com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.grpc:grpc-all:1.27.2
@@ -176,7 +183,8 @@ javax.validation:validation-api:1.0.0.GA
 javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.arnx:nashorn-promise:0.1.1
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -197,15 +205,17 @@ org.apache.beam:beam-vendor-grpc-1_26_0:0.3
 org.apache.beam:beam-vendor-guava-26_0-jre:0.1
 org.apache.beam:beam-vendor-sdks-java-extensions-protobuf:2.23.0
 org.apache.commons:commons-compress:1.20
+org.apache.commons:commons-exec:1.3
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.13.3
 org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -215,16 +225,17 @@ org.easymock:easymock:3.0
 org.flywaydb:flyway-core:5.2.4
 org.glassfish.jaxb:jaxb-runtime:2.3.1
 org.glassfish.jaxb:txw2:2.3.1
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.jvnet.staxex:stax-ex:1.8
 org.mortbay.jetty:jetty-util:6.1.26
 org.mortbay.jetty:jetty:6.1.26
@@ -234,11 +245,13 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
 org.scijava:native-lib-loader:2.0.2
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.slf4j:slf4j-jdk14:1.7.28
 org.testcontainers:database-commons:1.14.3
@@ -248,7 +261,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -1,8 +1,8 @@
 # This is a Gradle generated file for dependency locking.
 # Manual edits can break the build and are not advised.
 # This file is expected to be part of source control.
+com.github.ben-manes.caffeine:caffeine:2.7.0
 com.github.kevinstern:software-and-algorithms:1.0
-com.github.stephenc.jcip:jcip-annotations:1.0-1
 com.google.auto.value:auto-value:1.6.3
 com.google.auto:auto-common:0.10
 com.google.code.findbugs:jFormatString:3.0.0
@@ -11,11 +11,11 @@ com.google.dagger:dagger-compiler:2.28
 com.google.dagger:dagger-producers:2.28
 com.google.dagger:dagger-spi:2.28
 com.google.dagger:dagger:2.28
-com.google.errorprone:error_prone_annotation:2.3.3
+com.google.errorprone:error_prone_annotation:2.3.4
 com.google.errorprone:error_prone_annotations:2.3.4
-com.google.errorprone:error_prone_check_api:2.3.3
-com.google.errorprone:error_prone_core:2.3.3
-com.google.errorprone:error_prone_type_annotations:2.3.3
+com.google.errorprone:error_prone_check_api:2.3.4
+com.google.errorprone:error_prone_core:2.3.4
+com.google.errorprone:error_prone_type_annotations:2.3.4
 com.google.errorprone:javac-shaded:9-dev-r4023-3
 com.google.googlejavaformat:google-java-format:1.5
 com.google.guava:failureaccess:1.0.1
@@ -30,11 +30,14 @@ javax.inject:javax.inject:1
 javax.persistence:javax.persistence-api:2.2
 net.ltgt.gradle.incap:incap:0.2
 org.checkerframework:checker-compat-qual:2.5.3
-org.checkerframework:checker-qual:2.11.1
-org.checkerframework:dataflow:2.5.3
-org.checkerframework:javacutil:2.5.3
+org.checkerframework:checker-qual:3.0.0
+org.checkerframework:dataflow:3.0.0
+org.checkerframework:javacutil:3.0.0
 org.jetbrains.kotlin:kotlin-stdlib-common:1.3.61
 org.jetbrains.kotlin:kotlin-stdlib:1.3.61
 org.jetbrains.kotlinx:kotlinx-metadata-jvm:0.1.0
 org.jetbrains:annotations:13.0
 org.pcollections:pcollections:2.1.2
+org.plumelib:plume-util:1.0.6
+org.plumelib:reflection-util:0.0.2
+org.plumelib:require-javadoc:0.1.0
@@ -133,7 +133,7 @@ com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
 com.thoughtworks.qdox:qdox:1.12.1
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
@@ -185,7 +185,7 @@ jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.13
 net.bytebuddy:byte-buddy-agent:1.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -225,7 +225,7 @@ org.bouncycastle:bcpg-jdk15on:1.61
 org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -238,16 +238,17 @@ org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-all:1.3
 org.hamcrest:hamcrest-core:1.3
 org.hamcrest:hamcrest-library:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -271,7 +272,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -298,5 +299,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -132,7 +132,7 @@ com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
 com.thoughtworks.qdox:qdox:1.12.1
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
@@ -184,7 +184,7 @@ jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.13
 net.bytebuddy:byte-buddy-agent:1.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -237,16 +237,17 @@ org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-all:1.3
 org.hamcrest:hamcrest-core:1.3
 org.hamcrest:hamcrest-library:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -270,7 +271,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -297,5 +298,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.11.2
 com.fasterxml.jackson.core:jackson-core:2.11.2
 com.fasterxml.jackson.core:jackson-databind:2.11.2
@@ -134,10 +138,13 @@ com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
 com.thoughtworks.qdox:qdox:1.12.1
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.github.java-diff-utils:java-diff-utils:4.0
@@ -186,8 +193,9 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.13
+net.arnx:nashorn-promise:0.1.1
 net.bytebuddy:byte-buddy-agent:1.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -228,7 +236,7 @@ org.bouncycastle:bcpg-jdk15on:1.61
 org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -242,16 +250,17 @@ org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-all:1.3
 org.hamcrest:hamcrest-core:1.3
 org.hamcrest:hamcrest-library:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -275,7 +284,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -290,6 +299,8 @@ org.seleniumhq.selenium:selenium-opera-driver:3.141.59
 org.seleniumhq.selenium:selenium-remote-driver:3.141.59
 org.seleniumhq.selenium:selenium-safari-driver:3.141.59
 org.seleniumhq.selenium:selenium-support:3.141.59
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.testcontainers:database-commons:1.14.3
 org.testcontainers:jdbc:1.14.3
@@ -300,7 +311,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,6 +6,10 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
+com.eclipsesource.j2v8:j2v8_linux_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_macosx_x86_64:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86:4.6.0
+com.eclipsesource.j2v8:j2v8_win32_x86_64:4.6.0
 com.fasterxml.jackson.core:jackson-annotations:2.11.2
 com.fasterxml.jackson.core:jackson-core:2.11.2
 com.fasterxml.jackson.core:jackson-databind:2.11.2
@@ -134,10 +138,13 @@ com.sun.istack:istack-commons-runtime:3.0.7
 com.sun.xml.fastinfoset:FastInfoset:1.2.15
 com.thoughtworks.paranamer:paranamer:2.7
 com.thoughtworks.qdox:qdox:1.12.1
-com.zaxxer:HikariCP:3.2.0
+com.zaxxer:HikariCP:3.4.5
 commons-codec:commons-codec:1.13
 commons-logging:commons-logging:1.2
 dnsjava:dnsjava:2.1.7
+guru.nidi.com.kitfox:svgSalamander:1.1.3
+guru.nidi:graphviz-java-all-j2v8:0.17.0
+guru.nidi:graphviz-java:0.17.0
 io.dropwizard.metrics:metrics-core:3.2.6
 io.github.classgraph:classgraph:4.8.65
 io.github.java-diff-utils:java-diff-utils:4.0
@@ -186,8 +193,9 @@ javax.xml.bind:jaxb-api:2.3.1
 jline:jline:1.0
 joda-time:joda-time:2.10.5
 junit:junit:4.13
+net.arnx:nashorn-promise:0.1.1
 net.bytebuddy:byte-buddy-agent:1.10.5
-net.bytebuddy:byte-buddy:1.10.10
+net.bytebuddy:byte-buddy:1.10.17
 net.java.dev.jna:jna-platform:5.5.0
 net.java.dev.jna:jna:5.5.0
 org.apache.avro:avro:1.8.2
@@ -228,7 +236,7 @@ org.bouncycastle:bcpg-jdk15on:1.61
 org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
-org.checkerframework:checker-qual:2.11.1
+org.checkerframework:checker-qual:3.5.0
 org.codehaus.jackson:jackson-core-asl:1.9.13
 org.codehaus.jackson:jackson-mapper-asl:1.9.13
 org.codehaus.mojo:animal-sniffer-annotations:1.18
@@ -242,16 +250,17 @@ org.glassfish.jaxb:txw2:2.3.1
 org.hamcrest:hamcrest-all:1.3
 org.hamcrest:hamcrest-core:1.3
 org.hamcrest:hamcrest-library:1.3
-org.hibernate.common:hibernate-commons-annotations:5.1.0.Final
-org.hibernate:hibernate-core:5.4.17.Final
-org.hibernate:hibernate-hikaricp:5.4.17.Final
+org.hibernate.common:hibernate-commons-annotations:5.1.2.Final
+org.hibernate:hibernate-core:5.4.23.Final
+org.hibernate:hibernate-hikaricp:5.4.23.Final
 org.javassist:javassist:3.24.0-GA
-org.jboss.logging:jboss-logging:3.3.2.Final
+org.jboss.logging:jboss-logging:3.4.1.Final
 org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:1.1.1.Final
 org.jboss:jandex:2.1.3.Final
 org.jetbrains:annotations:19.0.0
 org.joda:joda-money:1.0.1
 org.json:json:20160810
+org.jsoup:jsoup:1.13.1
 org.junit-pioneer:junit-pioneer:0.7.0
 org.junit.jupiter:junit-jupiter-api:5.6.2
 org.junit.jupiter:junit-jupiter-engine:5.6.2
@@ -275,7 +284,7 @@ org.ow2.asm:asm-commons:7.1
 org.ow2.asm:asm-tree:8.0.1
 org.ow2.asm:asm-util:8.0.1
 org.ow2.asm:asm:8.0.1
-org.postgresql:postgresql:42.2.14
+org.postgresql:postgresql:42.2.18
 org.rnorth.duct-tape:duct-tape:1.0.8
 org.rnorth.visible-assertions:visible-assertions:2.1.2
 org.rnorth:tcp-unix-socket-proxy:1.0.2
@@ -290,6 +299,8 @@ org.seleniumhq.selenium:selenium-opera-driver:3.141.59
 org.seleniumhq.selenium:selenium-remote-driver:3.141.59
 org.seleniumhq.selenium:selenium-safari-driver:3.141.59
 org.seleniumhq.selenium:selenium-support:3.141.59
+org.slf4j:jcl-over-slf4j:1.7.30
+org.slf4j:jul-to-slf4j:1.7.30
 org.slf4j:slf4j-api:1.7.30
 org.slf4j:slf4j-jdk14:1.7.28
 org.testcontainers:database-commons:1.14.3
@@ -301,7 +312,13 @@ org.testcontainers:testcontainers:1.14.3
 org.threeten:threetenbp:1.4.1
 org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
+org.webjars.npm:viz.js-for-graphviz-java:2.1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -92,9 +92,8 @@ public class InitSqlPipeline implements Serializable {
   * Datastore kinds to be written to the SQL database before the cleansed version of {@link
   * DomainBase}.
   */
-  // TODO(weiminyu): include Registry.class when it is modeled in JPA.
  private static final ImmutableList<Class<?>> PHASE_ONE_ORDERED =
-      ImmutableList.of(Registrar.class, ContactResource.class);
+      ImmutableList.of(Registry.class, Registrar.class, ContactResource.class);

  /**
   * Datastore kinds to be written to the SQL database after the cleansed version of {@link
@@ -55,7 +55,7 @@ FROM (
    FROM
      `%PROJECT_ID%.%DATASTORE_EXPORT_DATA_SET%.%REGISTRY_TABLE%`
    WHERE
-      enableInvoicing IS TRUE) ) AS BillingEvent
+      invoicingEnabled IS TRUE) ) AS BillingEvent
  -- Gather billing ID from registrar table
  -- This is a 'JOIN' as opposed to 'LEFT JOIN' to filter out
  -- non-billable registrars
@@ -16,9 +16,12 @@ package google.registry.config;

 import static com.google.common.base.Suppliers.memoize;
 import static com.google.common.collect.ImmutableList.toImmutableList;
+import static com.google.common.collect.ImmutableSortedMap.toImmutableSortedMap;
 import static google.registry.config.ConfigUtils.makeUrl;
+import static google.registry.util.DateTimeUtils.START_OF_TIME;
 import static google.registry.util.ResourceUtils.readResourceUtf8;
 import static java.lang.annotation.RetentionPolicy.RUNTIME;
+import static java.util.Comparator.naturalOrder;

 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Ascii;
@@ -27,6 +30,7 @@ import com.google.common.base.Strings;
 import com.google.common.base.Supplier;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedMap;
 import dagger.Module;
 import dagger.Provides;
 import google.registry.util.TaskQueueUtils;
@@ -46,6 +50,7 @@ import javax.inject.Qualifier;
 import javax.inject.Singleton;
 import javax.mail.internet.AddressException;
 import javax.mail.internet.InternetAddress;
+import org.joda.time.DateTime;
 import org.joda.time.DateTimeConstants;
 import org.joda.time.Duration;

@@ -1345,6 +1350,39 @@ public final class RegistryConfig {
    public static String provideRdapTosStaticUrl(RegistryConfigSettings config) {
      return config.registryPolicy.rdapTosStaticUrl;
    }
+
+    @Provides
+    @Config("maxValidityDaysSchedule")
+    public static ImmutableSortedMap<DateTime, Integer> provideValidityDaysMap(
+        RegistryConfigSettings config) {
+      return config.sslCertificateValidation.maxValidityDaysSchedule.entrySet().stream()
+          .collect(
+              toImmutableSortedMap(
+                  naturalOrder(),
+                  e ->
+                      e.getKey().equals("START_OF_TIME")
+                          ? START_OF_TIME
+                          : DateTime.parse(e.getKey()),
+                  e -> e.getValue()));
+    }
+
+    @Provides
+    @Config("expirationWarningDays")
+    public static int provideDaysToExpiration(RegistryConfigSettings config) {
+      return config.sslCertificateValidation.expirationWarningDays;
+    }
+
+    @Provides
+    @Config("minimumRsaKeyLength")
+    public static int provideMinimumRsaKeyLength(RegistryConfigSettings config) {
+      return config.sslCertificateValidation.minimumRsaKeyLength;
+    }
+
+    @Provides
+    @Config("allowedEcdsaCurves")
+    public static ImmutableSet<String> provideAllowedEcdsaCurves(RegistryConfigSettings config) {
+      return ImmutableSet.copyOf(config.sslCertificateValidation.allowedEcdsaCurves);
+    }
  }

  /** Returns the App Engine project ID, which is based off the environment name. */
@@ -15,6 +15,8 @@
 package google.registry.config;

 import java.util.List;
+import java.util.Map;
+import java.util.Set;

 /** The POJO that YAML config files are deserialized into. */
 public class RegistryConfigSettings {
@@ -38,6 +40,7 @@ public class RegistryConfigSettings {
  public Beam beam;
  public Keyring keyring;
  public RegistryTool registryTool;
+  public SslCertificateValidation sslCertificateValidation;

  /** Configuration options that apply to the entire App Engine project. */
  public static class AppEngine {
@@ -218,4 +221,12 @@ public class RegistryConfigSettings {
    public String clientSecret;
    public String username;
  }
+
+  /** Configuration for the certificate checker. */
+  public static class SslCertificateValidation {
+    public Map<String, Integer> maxValidityDaysSchedule;
+    public int expirationWarningDays;
+    public int minimumRsaKeyLength;
+    public Set<String> allowedEcdsaCurves;
+  }
 }
@@ -446,3 +446,21 @@ registryTool:
  # OAuth client secret used by the tool.
  clientSecret: YOUR_CLIENT_SECRET
  username: toolusername
+
+# Configuration options for checking SSL certificates.
+sslCertificateValidation:
+  # A map specifying the maximum amount of days the certificate can be valid.
+  # The entry key is the date closest before the date the certificate was issued
+  # and the entry value is the applicable maximum validity days for that certificate.
+  maxValidityDaysSchedule:
+    "START_OF_TIME": 825
+    "2020-09-01T00:00:00Z": 398
+  # The number of days before a certificate expires that indicates the
+  # certificate is nearing expiration and warnings should be sent.
+  expirationWarningDays: 30
+  # The minimum number of bits an RSA key must contain.
+  minimumRsaKeyLength: 2048
+  # The ECDSA curves that are allowed for public keys.
+  allowedEcdsaCurves:
+    - secp256r1
+    - secp384r1
@@ -0,0 +1,261 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.flows.certs;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static google.registry.util.DateTimeUtils.START_OF_TIME;
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedMap;
+import google.registry.config.RegistryConfig.Config;
+import google.registry.util.Clock;
+import google.registry.util.DateTimeUtils;
+import java.io.ByteArrayInputStream;
+import java.security.PublicKey;
+import java.security.cert.CertificateException;
+import java.security.cert.CertificateFactory;
+import java.security.cert.X509Certificate;
+import java.security.interfaces.ECPublicKey;
+import java.security.interfaces.RSAPublicKey;
+import java.util.Date;
+import java.util.stream.Collectors;
+import javax.inject.Inject;
+import org.bouncycastle.jcajce.provider.asymmetric.util.EC5Util;
+import org.bouncycastle.jce.ECNamedCurveTable;
+import org.bouncycastle.jce.spec.ECNamedCurveParameterSpec;
+import org.joda.time.DateTime;
+import org.joda.time.Days;
+
+/** An utility to check that a given certificate meets our requirements */
+public class CertificateChecker {
+
+  private final ImmutableSortedMap<DateTime, Integer> maxValidityLengthSchedule;
+  private final int daysToExpiration;
+  private final int minimumRsaKeyLength;
+  private final Clock clock;
+  private final ImmutableSet<String> allowedEcdsaCurves;
+
+  /**
+   * Constructs a CertificateChecker instance with the specified configuration parameters.
+   *
+   * <p>The max validity length schedule is a sorted map of {@link DateTime} to {@link Integer}
+   * entries representing a maximum validity period for certificates issued on or after that date.
+   * The first entry must have a key of {@link DateTimeUtils#START_OF_TIME}, such that every
+   * possible date has an applicable max validity period. Since security requirements tighten over
+   * time, the max validity periods will be decreasing as the date increases.
+   *
+   * <p>The validity length schedule used by all major Web browsers as of 2020Q4 would be
+   * represented as:
+   *
+   * <pre>
+   *   ImmutableSortedMap.of(
+   *     START_OF_TIME, 825,
+   *     DateTime.parse("2020-09-01T00:00:00Z"), 398
+   *   );
+   * </pre>
+   */
+  @Inject
+  public CertificateChecker(
+      @Config("maxValidityDaysSchedule")
+          ImmutableSortedMap<DateTime, Integer> maxValidityDaysSchedule,
+      @Config("expirationWarningDays") int expirationWarningDays,
+      @Config("minimumRsaKeyLength") int minimumRsaKeyLength,
+      @Config("allowedEcdsaCurves") ImmutableSet<String> allowedEcdsaCurves,
+      Clock clock) {
+    checkArgument(
+        maxValidityDaysSchedule.containsKey(START_OF_TIME),
+        "Max validity length schedule must contain an entry for START_OF_TIME");
+    this.maxValidityLengthSchedule = maxValidityDaysSchedule;
+    this.daysToExpiration = expirationWarningDays;
+    this.minimumRsaKeyLength = minimumRsaKeyLength;
+    this.allowedEcdsaCurves = allowedEcdsaCurves;
+    this.clock = clock;
+  }
+
+  /**
+   * Checks the given certificate string for violations and throws an exception if any violations
+   * exist.
+   */
+  public void validateCertificate(String certificateString) {
+    ImmutableSet<CertificateViolation> violations = checkCertificate(certificateString);
+    if (!violations.isEmpty()) {
+      String displayMessages =
+          violations.stream()
+              .map(violation -> getViolationDisplayMessage(violation))
+              .collect(Collectors.joining("\n"));
+      throw new IllegalArgumentException(displayMessages);
+    }
+  }
+
+  /**
+   * Checks a given certificate for violations and returns a list of all the violations the
+   * certificate has.
+   */
+  public ImmutableSet<CertificateViolation> checkCertificate(X509Certificate certificate) {
+    ImmutableSet.Builder<CertificateViolation> violations = new ImmutableSet.Builder<>();
+
+    // Check if currently in validity period
+    Date now = clock.nowUtc().toDate();
+    if (certificate.getNotAfter().before(now)) {
+      violations.add(CertificateViolation.EXPIRED);
+    } else if (certificate.getNotBefore().after(now)) {
+      violations.add(CertificateViolation.NOT_YET_VALID);
+    }
+
+    // Check validity period length
+    int maxValidityDays =
+        maxValidityLengthSchedule.floorEntry(new DateTime(certificate.getNotBefore())).getValue();
+    if (getValidityLengthInDays(certificate) > maxValidityDays) {
+      violations.add(CertificateViolation.VALIDITY_LENGTH_TOO_LONG);
+    }
+
+    // Check key strengths
+    PublicKey key = certificate.getPublicKey();
+    if (key.getAlgorithm().equals("RSA")) {
+      RSAPublicKey rsaPublicKey = (RSAPublicKey) key;
+      if (rsaPublicKey.getModulus().bitLength() < minimumRsaKeyLength) {
+        violations.add(CertificateViolation.RSA_KEY_LENGTH_TOO_SHORT);
+      }
+    } else if (key.getAlgorithm().equals("EC")) {
+      if (!checkCurveName(key, allowedEcdsaCurves)) {
+        violations.add(CertificateViolation.INVALID_ECDSA_CURVE);
+      }
+    } else {
+      violations.add(CertificateViolation.ALGORITHM_CONSTRAINED);
+    }
+    return violations.build();
+  }
+
+  /**
+   * Converts a given string to a certificate and checks it for violations, returning a list of all
+   * the violations the certificate has.
+   */
+  public ImmutableSet<CertificateViolation> checkCertificate(String certificateString) {
+    X509Certificate certificate;
+
+    try {
+      certificate =
+          (X509Certificate)
+              CertificateFactory.getInstance("X509")
+                  .generateCertificate(new ByteArrayInputStream(certificateString.getBytes(UTF_8)));
+    } catch (CertificateException e) {
+      throw new IllegalArgumentException("Unable to read given certificate.");
+    }
+
+    return checkCertificate(certificate);
+  }
+
+  /**
+   * Returns whether the certificate is nearing expiration.
+   *
+   * <p>Note that this is <i>all</i> that it checks. The certificate itself may well be expired or
+   * not yet valid and this message will still return false. So you definitely want to pair a call
+   * to this method with a call to {@link #checkCertificate} to determine other issues with the
+   * certificate that may be occurring.
+   */
+  public boolean isNearingExpiration(X509Certificate certificate) {
+    Date nearingExpirationDate =
+        DateTime.parse(certificate.getNotAfter().toInstant().toString())
+            .minusDays(daysToExpiration)
+            .toDate();
+    return clock.nowUtc().toDate().after(nearingExpirationDate);
+  }
+
+  private static int getValidityLengthInDays(X509Certificate certificate) {
+    DateTime start = DateTime.parse(certificate.getNotBefore().toInstant().toString());
+    DateTime end = DateTime.parse(certificate.getNotAfter().toInstant().toString());
+    return Days.daysBetween(start.withTimeAtStartOfDay(), end.withTimeAtStartOfDay()).getDays();
+  }
+
+  /** Checks if the curve used for a public key is in the list of acceptable curves. */
+  private static boolean checkCurveName(PublicKey key, ImmutableSet<String> allowedEcdsaCurves) {
+    org.bouncycastle.jce.spec.ECParameterSpec params;
+    // These 2 different instances of PublicKey need to be handled separately since their OIDs are
+    // encoded differently. More details on this can be found at
+    // https://stackoverflow.com/questions/49895713/how-to-find-the-matching-curve-name-from-an-ecpublickey.
+    if (key instanceof ECPublicKey) {
+      ECPublicKey ecKey = (ECPublicKey) key;
+      params = EC5Util.convertSpec(ecKey.getParams(), false);
+    } else if (key instanceof org.bouncycastle.jce.interfaces.ECPublicKey) {
+      org.bouncycastle.jce.interfaces.ECPublicKey ecKey =
+          (org.bouncycastle.jce.interfaces.ECPublicKey) key;
+      params = ecKey.getParameters();
+    } else {
+      throw new IllegalArgumentException("Unrecognized instance of PublicKey.");
+    }
+    return allowedEcdsaCurves.stream()
+        .anyMatch(
+            curve -> {
+              ECNamedCurveParameterSpec cParams = ECNamedCurveTable.getParameterSpec(curve);
+              return cParams.getN().equals(params.getN())
+                  && cParams.getH().equals(params.getH())
+                  && cParams.getCurve().equals(params.getCurve())
+                  && cParams.getG().equals(params.getG());
+            });
+  }
+
+  private String getViolationDisplayMessage(CertificateViolation certificateViolation) {
+    // Yes, we'd rather do this as an instance method on the CertificateViolation enum itself, but
+    // we can't because we need access to configuration (injected as instance variables) which you
+    // can't get in a static enum context.
+    switch (certificateViolation) {
+      case EXPIRED:
+        return "Certificate is expired.";
+      case NOT_YET_VALID:
+        return "Certificate start date is in the future.";
+      case ALGORITHM_CONSTRAINED:
+        return "Certificate key algorithm must be RSA or ECDSA.";
+      case RSA_KEY_LENGTH_TOO_SHORT:
+        return String.format(
+            "RSA key length is too short; the minimum allowed length is %d bits.",
+            this.minimumRsaKeyLength);
+      case VALIDITY_LENGTH_TOO_LONG:
+        return String.format(
+            "Certificate validity period is too long; it must be less than or equal to %d days.",
+            this.maxValidityLengthSchedule.lastEntry().getValue());
+      case INVALID_ECDSA_CURVE:
+        return String.format(
+            "The ECDSA key must use one of these algorithms: %s", allowedEcdsaCurves);
+      default:
+        throw new IllegalArgumentException(
+            String.format(
+                "Unknown CertificateViolation enum value: %s", certificateViolation.name()));
+    }
+  }
+
+  /**
+   * The type of violation a certificate has based on the certificate requirements
+   * (go/registry-proxy-security).
+   */
+  public enum CertificateViolation {
+    EXPIRED,
+    NOT_YET_VALID,
+    VALIDITY_LENGTH_TOO_LONG,
+    RSA_KEY_LENGTH_TOO_SHORT,
+    ALGORITHM_CONSTRAINED,
+    INVALID_ECDSA_CURVE;
+
+    /**
+     * Gets a suitable end-user-facing display message for this particular certificate violation.
+     *
+     * <p>Note that the {@link CertificateChecker} instance must be passed in because it contains
+     * configuration values (e.g. minimum RSA key length) that go into the error message text.
+     */
+    public String getDisplayMessage(CertificateChecker certificateChecker) {
+      return certificateChecker.getViolationDisplayMessage(this);
+    }
+  }
+}
@@ -67,6 +67,7 @@ import google.registry.model.domain.DomainCommand.Update.AddRemove;
 import google.registry.model.domain.DomainCommand.Update.Change;
 import google.registry.model.domain.fee.FeeUpdateCommandExtension;
 import google.registry.model.domain.metadata.MetadataExtension;
+import google.registry.model.domain.secdns.DelegationSignerData;
 import google.registry.model.domain.secdns.SecDnsUpdateExtension;
 import google.registry.model.domain.superuser.DomainUpdateSuperuserExtension;
 import google.registry.model.eppcommon.AuthInfo;
@@ -238,10 +239,16 @@ public final class DomainUpdateFlow implements TransactionalFlow {
    DomainBase.Builder domainBuilder =
        domain
            .asBuilder()
-            // Handle the secDNS extension.
+            // Handle the secDNS extension. As dsData in secDnsUpdate is read from EPP input and
+            // does not have domainRepoId set, we create a copy of the existing dsData without
+            // domainRepoId for comparison.
            .setDsData(
                secDnsUpdate.isPresent()
-                    ? updateDsData(domain.getDsData(), secDnsUpdate.get())
+                    ? updateDsData(
+                        domain.getDsData().stream()
+                            .map(DelegationSignerData::cloneWithoutDomainRepoId)
+                            .collect(toImmutableSet()),
+                        secDnsUpdate.get())
                    : domain.getDsData())
            .setLastEppUpdateTime(now)
            .setLastEppUpdateClientId(clientId)
@@ -24,7 +24,6 @@ import static google.registry.persistence.transaction.TransactionManagerFactory.
 import static google.registry.util.CollectionUtils.nullToEmpty;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;

 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.cache.CacheBuilder;
@@ -387,7 +386,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
  private static LoadingCache<VKey<? extends EppResource>, EppResource> createEppResourcesCache(
      Duration expiry) {
    return CacheBuilder.newBuilder()
-        .expireAfterWrite(expiry.getMillis(), MILLISECONDS)
+        .expireAfterWrite(java.time.Duration.ofMillis(expiry.getMillis()))
        .maximumSize(getEppResourceMaxCachedEntries())
        .build(CACHE_LOADER);
  }
@@ -61,7 +61,17 @@ public abstract class ImmutableObject implements Cloneable {
  private boolean equalsImmutableObject(ImmutableObject other) {
    return getClass().equals(other.getClass())
        && hashCode() == other.hashCode()
-        && ModelUtils.getFieldValues(this).equals(ModelUtils.getFieldValues(other));
+        && getSignificantFields().equals(other.getSignificantFields());
+  }
+
+  /**
+   * Returns the map of significant fields (fields that we care about for purposes of comparison and
+   * display).
+   *
+   * <p>Isolated into a method so that derived classes can override it.
+   */
+  protected Map<Field, Object> getSignificantFields() {
+    return ModelUtils.getFieldValues(this);
  }

  @Override
@@ -72,7 +82,7 @@ public abstract class ImmutableObject implements Cloneable {
  @Override
  public int hashCode() {
    if (hashCode == null) {
-      hashCode = Arrays.hashCode(ModelUtils.getFieldValues(this).values().toArray());
+      hashCode = Arrays.hashCode(getSignificantFields().values().toArray());
    }
    return hashCode;
  }
@@ -111,7 +121,7 @@ public abstract class ImmutableObject implements Cloneable {
  @Override
  public String toString() {
    NavigableMap<String, Object> sortedFields = new TreeMap<>();
-    for (Entry<Field, Object> entry : ModelUtils.getFieldValues(this).entrySet()) {
+    for (Entry<Field, Object> entry : getSignificantFields().entrySet()) {
      sortedFields.put(entry.getKey().getName(), entry.getValue());
    }
    return toStringHelper(sortedFields);
@@ -121,7 +131,7 @@ public abstract class ImmutableObject implements Cloneable {
  public String toHydratedString() {
    // We can't use ImmutableSortedMap because we need to allow null values.
    NavigableMap<String, Object> sortedFields = new TreeMap<>();
-    for (Entry<Field, Object> entry : ModelUtils.getFieldValues(this).entrySet()) {
+    for (Entry<Field, Object> entry : getSignificantFields().entrySet()) {
      Field field = entry.getKey();
      Object value = entry.getValue();
      sortedFields.put(
@@ -161,7 +171,7 @@ public abstract class ImmutableObject implements Cloneable {
      // LinkedHashMap to preserve field ordering and because ImmutableMap forbids null
      // values.
      Map<String, Object> result = new LinkedHashMap<>();
-      for (Entry<Field, Object> entry : ModelUtils.getFieldValues(o).entrySet()) {
+      for (Entry<Field, Object> entry : ((ImmutableObject) o).getSignificantFields().entrySet()) {
        Field field = entry.getKey();
        if (!field.isAnnotationPresent(IgnoredInDiffableMap.class)) {
          result.put(field.getName(), toMapRecursive(entry.getValue()));
@@ -194,7 +194,7 @@ public class ModelUtils {
   * returned map in its implementation of {@link ImmutableObject#toString} and {@link
   * ImmutableObject#equals}, which work by comparing and printing these maps.
   */
-  static Map<Field, Object> getFieldValues(Object instance) {
+  public static Map<Field, Object> getFieldValues(Object instance) {
    // Don't make this ImmutableMap because field values can be null.
    Map<Field, Object> values = new LinkedHashMap<>();
    for (Field field : getAllFields(instance.getClass()).values()) {
@@ -252,7 +252,8 @@ public final class OteAccountBuilder {
  private void saveAllEntities() {
    tm().assertInTransaction();

-    ImmutableList<Registry> registries = ImmutableList.of(sunriseTld, gaTld, eapTld);
+    // use ImmutableObject instead of Registry so that the Key generation doesn't break
+    ImmutableList<ImmutableObject> registries = ImmutableList.of(sunriseTld, gaTld, eapTld);
    ImmutableList<RegistrarContact> contacts = contactsBuilder.build();

    if (!replaceExisting) {
@@ -59,25 +59,11 @@ public class OteStats {

  private OteStats() {}

-  private static final Predicate<EppInput> HAS_CLAIMS_NOTICE =
-      eppInput -> {
-        Optional<LaunchCreateExtension> launchCreate =
-            eppInput.getSingleExtension(LaunchCreateExtension.class);
-        return launchCreate.isPresent() && launchCreate.get().getNotice() != null;
-      };
-
  private static final Predicate<EppInput> HAS_SEC_DNS =
      eppInput ->
          eppInput.getSingleExtension(SecDnsCreateExtension.class).isPresent()
              || eppInput.getSingleExtension(SecDnsUpdateExtension.class).isPresent();

-  private static final Predicate<EppInput> IS_SUNRISE =
-      eppInput -> {
-        Optional<LaunchCreateExtension> launchCreate =
-            eppInput.getSingleExtension(LaunchCreateExtension.class);
-        return launchCreate.isPresent() && !isNullOrEmpty(launchCreate.get().getSignedMarks());
-      };
-
  private static final Predicate<EppInput> IS_IDN =
      eppInput ->
          ((DomainCommand.Create)
@@ -94,6 +80,18 @@ public class OteStats {
                          .getResourceCommand())
                  .getInetAddresses());

+  private static boolean hasClaimsNotice(EppInput eppInput) {
+    Optional<LaunchCreateExtension> launchCreate =
+        eppInput.getSingleExtension(LaunchCreateExtension.class);
+    return launchCreate.isPresent() && launchCreate.get().getNotice() != null;
+  }
+
+  private static boolean isSunrise(EppInput eppInput) {
+    Optional<LaunchCreateExtension> launchCreate =
+        eppInput.getSingleExtension(LaunchCreateExtension.class);
+    return launchCreate.isPresent() && !isNullOrEmpty(launchCreate.get().getSignedMarks());
+  }
+
  /** Enum defining the distinct statistics (types of registrar actions) to record. */
  public enum StatType {
    CONTACT_CREATES(0, equalTo(Type.CONTACT_CREATE)),
@@ -107,8 +105,8 @@ public class OteStats {
    DOMAIN_CREATES(0, equalTo(Type.DOMAIN_CREATE)),
    DOMAIN_CREATES_ASCII(1, equalTo(Type.DOMAIN_CREATE), IS_IDN.negate()),
    DOMAIN_CREATES_IDN(1, equalTo(Type.DOMAIN_CREATE), IS_IDN),
-    DOMAIN_CREATES_START_DATE_SUNRISE(1, equalTo(Type.DOMAIN_CREATE), IS_SUNRISE),
-    DOMAIN_CREATES_WITH_CLAIMS_NOTICE(1, equalTo(Type.DOMAIN_CREATE), HAS_CLAIMS_NOTICE),
+    DOMAIN_CREATES_START_DATE_SUNRISE(1, equalTo(Type.DOMAIN_CREATE), OteStats::isSunrise),
+    DOMAIN_CREATES_WITH_CLAIMS_NOTICE(1, equalTo(Type.DOMAIN_CREATE), OteStats::hasClaimsNotice),
    DOMAIN_CREATES_WITH_FEE(
        1,
        equalTo(Type.DOMAIN_CREATE),
@@ -24,6 +24,7 @@ import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Sets;
@@ -49,6 +50,8 @@ import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithLongVKey;
 import google.registry.schema.replay.DatastoreAndSqlEntity;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
@@ -67,7 +70,6 @@ import org.joda.time.DateTime;

 /** A billable event in a domain's lifecycle. */
@MappedSuperclass
-@WithLongVKey
 public abstract class BillingEvent extends ImmutableObject
    implements Buildable, TransferServerApproveEntity {

@@ -119,13 +121,11 @@ public abstract class BillingEvent extends ImmutableObject
  String clientId;

  /** Revision id of the entry in DomainHistory table that ths bill belongs to. */
-  // TODO(shicong): Add foreign key constraint when DomainHistory table is generated
  @Ignore
  @Column(nullable = false)
  Long domainHistoryRevisionId;

  /** ID of the EPP resource that the bill is for. */
-  // TODO(shicong): Add foreign key constraint when we expand DatastoreHelp for Postgresql
  @Ignore
  @Column(nullable = false)
  String domainRepoId;
@@ -289,9 +289,10 @@ public abstract class BillingEvent extends ImmutableObject
        @javax.persistence.Index(columnList = "eventTime"),
        @javax.persistence.Index(columnList = "billingTime"),
        @javax.persistence.Index(columnList = "syntheticCreationTime"),
-        @javax.persistence.Index(columnList = "allocation_token_id")
+        @javax.persistence.Index(columnList = "allocationToken")
      })
  @AttributeOverride(name = "id", column = @Column(name = "billing_event_id"))
+  @WithLongVKey
  public static class OneTime extends BillingEvent implements DatastoreAndSqlEntity {

    /** The billable value. */
@@ -327,14 +328,11 @@ public abstract class BillingEvent extends ImmutableObject
     * Cancellation}s.
     */
    @Column(name = "cancellation_matching_billing_recurrence_id")
-    VKey<? extends BillingEvent> cancellationMatchingBillingEvent;
+    VKey<Recurring> cancellationMatchingBillingEvent;

    /**
     * The {@link AllocationToken} used in the creation of this event, or null if one was not used.
-     *
-     * <p>TODO(shicong): Add foreign key constraint when AllocationToken schema is generated
     */
-    @Column(name = "allocation_token_id")
    @Index
    @Nullable
    VKey<AllocationToken> allocationToken;
@@ -409,7 +407,7 @@ public abstract class BillingEvent extends ImmutableObject
      }

      public Builder setCancellationMatchingBillingEvent(
-          VKey<? extends BillingEvent> cancellationMatchingBillingEvent) {
+          VKey<Recurring> cancellationMatchingBillingEvent) {
        getInstance().cancellationMatchingBillingEvent = cancellationMatchingBillingEvent;
        return this;
      }
@@ -468,6 +466,7 @@ public abstract class BillingEvent extends ImmutableObject
        @javax.persistence.Index(columnList = "recurrence_time_of_year")
      })
  @AttributeOverride(name = "id", column = @Column(name = "billing_recurrence_id"))
+  @WithLongVKey
  public static class Recurring extends BillingEvent implements DatastoreAndSqlEntity {

    /**
@@ -562,6 +561,7 @@ public abstract class BillingEvent extends ImmutableObject
        @javax.persistence.Index(columnList = "billingTime")
      })
  @AttributeOverride(name = "id", column = @Column(name = "billing_cancellation_id"))
+  @WithLongVKey
  public static class Cancellation extends BillingEvent implements DatastoreAndSqlEntity {

    /** The billing time of the charge that is being cancelled. */
@@ -682,7 +682,8 @@ public abstract class BillingEvent extends ImmutableObject
  /** An event representing a modification of an existing one-time billing event. */
  @ReportedOn
  @Entity
-  public static class Modification extends BillingEvent implements DatastoreAndSqlEntity {
+  @WithLongVKey
+  public static class Modification extends BillingEvent implements DatastoreEntity {

    /** The change in cost that should be applied to the original billing event. */
    Money cost;
@@ -744,6 +745,11 @@ public abstract class BillingEvent extends ImmutableObject
          .build();
    }

+    @Override
+    public ImmutableList<SqlEntity> toSqlEntities() {
+      return ImmutableList.of(); // not persisted in SQL
+    }
+
    /** A builder for {@link Modification} since it is immutable. */
    public static class Builder extends BillingEvent.Builder<Modification, Builder> {

@@ -20,15 +20,16 @@ import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.ImmutableObject;
+import javax.persistence.MappedSuperclass;
+import javax.persistence.Transient;

 /** A singleton entity in Datastore. */
+@MappedSuperclass
 public abstract class CrossTldSingleton extends ImmutableObject {

-  public static final long SINGLETON_ID = 1;  // There is always exactly one of these.
+  public static final long SINGLETON_ID = 1; // There is always exactly one of these.

-  @Id
-  long id = SINGLETON_ID;
+  @Id @Transient long id = SINGLETON_ID;

-  @Parent
-  Key<EntityGroupRoot> parent = getCrossTldKey();
+  @Transient @Parent Key<EntityGroupRoot> parent = getCrossTldKey();
 }
@@ -14,20 +14,24 @@

 package google.registry.model.contact;

+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EntitySubclass;
-import google.registry.model.EppResource;
+import google.registry.model.ImmutableObject;
+import google.registry.model.contact.ContactHistory.ContactHistoryId;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import java.io.Serializable;
 import java.util.Optional;
 import javax.annotation.Nullable;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
 import javax.persistence.Column;
 import javax.persistence.Entity;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.Id;
+import javax.persistence.IdClass;
 import javax.persistence.PostLoad;

 /**
@@ -47,16 +51,27 @@ import javax.persistence.PostLoad;
    })
@EntitySubclass
@Access(AccessType.FIELD)
-public class ContactHistory extends HistoryEntry {
+@IdClass(ContactHistoryId.class)
+public class ContactHistory extends HistoryEntry implements SqlEntity {

  // Store ContactBase instead of ContactResource so we don't pick up its @Id
  @Nullable ContactBase contactBase;

-  @Column(nullable = false)
-  VKey<ContactResource> contactRepoId;
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getContactRepoId() {
+    // We need to handle null case here because Hibernate sometimes accesses this method before
+    // parent gets initialized
+    return parent == null ? null : parent.getName();
+  }
+
+  /** This method is private because it is only used by Hibernate. */
+  @SuppressWarnings("unused")
+  private void setContactRepoId(String contactRepoId) {
+    parent = Key.create(ContactResource.class, contactRepoId);
+  }

  @Id
-  @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "TempHistorySequenceGenerator")
  @Column(name = "historyRevisionId")
  @Access(AccessType.PROPERTY)
  @Override
@@ -75,8 +90,14 @@ public class ContactHistory extends HistoryEntry {
  }

  /** The key to the {@link ContactResource} this is based off of. */
-  public VKey<ContactResource> getContactRepoId() {
-    return contactRepoId;
+  public VKey<ContactResource> getParentVKey() {
+    return VKey.create(ContactResource.class, getContactRepoId());
+  }
+
+  /** Creates a {@link VKey} instance for this entity. */
+  public VKey<ContactHistory> createVKey() {
+    return VKey.create(
+        ContactHistory.class, new ContactHistoryId(getContactRepoId(), getId()), Key.create(this));
  }

  @PostLoad
@@ -86,11 +107,70 @@ public class ContactHistory extends HistoryEntry {
    if (contactBase != null && contactBase.getContactId() == null) {
      contactBase = null;
    }
-    // Fill in the full, symmetric, parent repo ID key
-    Key<ContactResource> parentKey =
-        Key.create(ContactResource.class, (String) contactRepoId.getSqlKey());
-    parent = parentKey;
-    contactRepoId = VKey.create(ContactResource.class, contactRepoId.getSqlKey(), parentKey);
+  }
+
+  // In Datastore, save as a HistoryEntry object regardless of this object's type
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(asHistoryEntry());
+  }
+
+  /** Class to represent the composite primary key of {@link ContactHistory} entity. */
+  static class ContactHistoryId extends ImmutableObject implements Serializable {
+
+    private String contactRepoId;
+
+    private Long id;
+
+    /** Hibernate requires this default constructor. */
+    private ContactHistoryId() {}
+
+    ContactHistoryId(String contactRepoId, long id) {
+      this.contactRepoId = contactRepoId;
+      this.id = id;
+    }
+
+    /**
+     * Returns the contact repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getContactRepoId() {
+      return contactRepoId;
+    }
+
+    /**
+     * Returns the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private long getId() {
+      return id;
+    }
+
+    /**
+     * Sets the contact repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setContactRepoId(String contactRepoId) {
+      this.contactRepoId = contactRepoId;
+    }
+
+    /**
+     * Sets the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setId(long id) {
+      this.id = id;
+    }
  }

  @Override
@@ -111,18 +191,8 @@ public class ContactHistory extends HistoryEntry {
      return this;
    }

-    public Builder setContactRepoId(VKey<ContactResource> contactRepoId) {
-      getInstance().contactRepoId = contactRepoId;
-      contactRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
-      return this;
-    }
-
-    // We can remove this once all HistoryEntries are converted to History objects
-    @Override
-    public Builder setParent(Key<? extends EppResource> parent) {
-      super.setParent(parent);
-      getInstance().contactRepoId =
-          VKey.create(ContactResource.class, parent.getName(), (Key<ContactResource>) parent);
+    public Builder setContactRepoId(String contactRepoId) {
+      getInstance().parent = Key.create(ContactResource.class, contactRepoId);
      return this;
    }
  }
@@ -51,7 +51,6 @@ public class ContactResource extends ContactBase

  @Override
  public VKey<ContactResource> createVKey() {
-    // TODO(mmuller): create symmetric keys if we can ever reload both sides.
    return VKey.create(ContactResource.class, getRepoId(), Key.create(this));
  }

@@ -19,6 +19,7 @@ import google.registry.model.EppResource;
 import google.registry.model.EppResource.ForeignKeyedEppResource;
 import google.registry.model.annotations.ExternalMessagingName;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.model.domain.secdns.DelegationSignerData;
 import google.registry.model.host.HostResource;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithStringVKey;
@@ -106,6 +107,32 @@ public class DomainBase extends DomainContent
    return gracePeriods;
  }

+  /**
+   * Returns the set of {@link DelegationSignerData} associated with the domain.
+   *
+   * <p>This is the getter method specific for Hibernate to access the field so it is set to
+   * private. The caller can use the public {@link #getDsData()} to get the DS data.
+   *
+   * <p>Note that we need to set `insertable = false, updatable = false` for @JoinColumn, otherwise
+   * Hibernate would try to set the foreign key to null(through an UPDATE TABLE sql) instead of
+   * deleting the whole entry from the table when the {@link DelegationSignerData} is removed from
+   * the set.
+   */
+  @Access(AccessType.PROPERTY)
+  @OneToMany(
+      cascade = {CascadeType.ALL},
+      fetch = FetchType.EAGER,
+      orphanRemoval = true)
+  @JoinColumn(
+      name = "domainRepoId",
+      referencedColumnName = "repoId",
+      insertable = false,
+      updatable = false)
+  @SuppressWarnings("UnusedMethod")
+  private Set<DelegationSignerData> getInternalDelegationSignerData() {
+    return dsData;
+  }
+
  @Override
  public VKey<DomainBase> createVKey() {
    return VKey.create(DomainBase.class, getRepoId(), Key.create(this));
@@ -35,11 +35,13 @@ import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
 import static google.registry.util.DomainNameUtils.getTldFromDomainName;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

+import com.google.common.base.Predicate;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedSet;
 import com.google.common.collect.Ordering;
 import com.google.common.collect.Sets;
 import com.google.common.collect.Streams;
+import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.IgnoreSave;
 import com.googlecode.objectify.annotation.Index;
@@ -49,6 +51,7 @@ import google.registry.flows.ResourceFlowUtils;
 import google.registry.model.EppResource;
 import google.registry.model.EppResource.ResourceWithTransferData;
 import google.registry.model.billing.BillingEvent;
+import google.registry.model.common.EntityGroupRoot;
 import google.registry.model.contact.ContactResource;
 import google.registry.model.domain.launch.LaunchNotice;
 import google.registry.model.domain.rgp.GracePeriodStatus;
@@ -57,6 +60,7 @@ import google.registry.model.eppcommon.StatusValue;
 import google.registry.model.host.HostResource;
 import google.registry.model.poll.PollMessage;
 import google.registry.model.registry.Registry;
+import google.registry.model.reporting.HistoryEntry;
 import google.registry.model.transfer.DomainTransferData;
 import google.registry.model.transfer.TransferStatus;
 import google.registry.persistence.VKey;
@@ -65,7 +69,6 @@ import java.util.HashSet;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
-import java.util.function.Predicate;
 import javax.annotation.Nullable;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
@@ -209,6 +212,15 @@ public class DomainContent extends EppResource
  @Column(name = "deletion_poll_message_id")
  VKey<PollMessage.OneTime> deletePollMessage;

+  /**
+   * History record for the delete poll message.
+   *
+   * <p>Here so we can restore the original ofy key from sql.
+   */
+  @Column(name = "deletion_poll_message_history_id")
+  @Ignore
+  Long deletePollMessageHistoryId;
+
  /**
   * The recurring billing event associated with this domain's autorenewals.
   *
@@ -220,6 +232,15 @@ public class DomainContent extends EppResource
  @Column(name = "billing_recurrence_id")
  VKey<BillingEvent.Recurring> autorenewBillingEvent;

+  /**
+   * History record for the autorenew billing event.
+   *
+   * <p>Here so we can restore the original ofy key from sql.
+   */
+  @Column(name = "billing_recurrence_history_id")
+  @Ignore
+  Long autorenewBillingEventHistoryId;
+
  /**
   * The recurring poll message associated with this domain's autorenewals.
   *
@@ -231,6 +252,13 @@ public class DomainContent extends EppResource
  @Column(name = "autorenew_poll_message_id")
  VKey<PollMessage.Autorenew> autorenewPollMessage;

+  /**
+   * History record for the autorenew poll message.
+   *
+   * <p>Here so we can restore the original ofy key from sql.
+   */
+  @Ignore Long autorenewPollMessageHistoryId;
+
  /** The unexpired grace periods for this domain (some of which may not be active yet). */
  @Transient Set<GracePeriod> gracePeriods;

@@ -275,19 +303,28 @@ public class DomainContent extends EppResource
        allContacts.stream().map(DesignatedContact::reconstitute).collect(toImmutableSet());
    setContactFields(allContacts, true);

-    // We have to return the cloned object here because the original object's
-    // hashcode is not correct due to the change to its domainRepoId. The cloned
-    // object will have a null hashcode so that it can get a recalculated hashcode
-    // when its hashCode() is invoked.
+    // We have to return the cloned object here because the original object's hashcode is not
+    // correct due to the change to its domainRepoId and history ids. The cloned object will have a
+    // null hashcode so that it can get a recalculated hashcode when its hashCode() is invoked.
    // TODO(b/162739503): Remove this after fully migrating to Cloud SQL.
    gracePeriods =
        nullToEmptyImmutableCopy(gracePeriods).stream()
-            .map(gracePeriod -> gracePeriod.cloneWithDomainRepoId(getRepoId()))
+            .map(gracePeriod -> gracePeriod.cloneAfterOfyLoad(getRepoId()))
+            .collect(toImmutableSet());
+
+    // Restore history record ids.
+    autorenewPollMessageHistoryId = getHistoryId(autorenewPollMessage);
+    autorenewBillingEventHistoryId = getHistoryId(autorenewBillingEvent);
+    deletePollMessageHistoryId = getHistoryId(deletePollMessage);
+    dsData =
+        nullToEmptyImmutableCopy(dsData).stream()
+            .map(dsData -> dsData.cloneWithDomainRepoId(getRepoId()))
            .collect(toImmutableSet());
  }

  @PostLoad
-  void postLoad() {
+  @SuppressWarnings("UnusedMethod")
+  private final void postLoad() {
    // Reconstitute the contact list.
    ImmutableSet.Builder<DesignatedContact> contactsBuilder = new ImmutableSet.Builder<>();

@@ -306,6 +343,27 @@ public class DomainContent extends EppResource
    }

    allContacts = contactsBuilder.build();
+
+    // Reconstitute the composite ofy keys from the SQL data.
+    Key<DomainBase> myKey = Key.create(DomainBase.class, getRepoId());
+    deletePollMessage = restoreOfyFrom(myKey, deletePollMessage, deletePollMessageHistoryId);
+    autorenewBillingEvent =
+        restoreOfyFrom(myKey, autorenewBillingEvent, autorenewBillingEventHistoryId);
+    autorenewPollMessage =
+        restoreOfyFrom(myKey, autorenewPollMessage, autorenewPollMessageHistoryId);
+
+    if (transferData != null) {
+      transferData.restoreOfyKeys(myKey);
+    }
+  }
+
+  public static <T> VKey<T> restoreOfyFrom(Key<DomainBase> domainKey, VKey<T> key, Long historyId) {
+    if (historyId == null) {
+      // This is a legacy key (or a null key, in which case this works too)
+      return VKey.restoreOfyFrom(key, EntityGroupRoot.class, "per-tld");
+    } else {
+      return VKey.restoreOfyFrom(key, domainKey, HistoryEntry.class, historyId);
+    }
  }

  public ImmutableSet<String> getSubordinateHosts() {
@@ -387,6 +445,12 @@ public class DomainContent extends EppResource
    this.gracePeriods = gracePeriods;
  }

+  // Hibernate needs this in order to populate dsData but no one else should ever use it
+  @SuppressWarnings("UnusedMethod")
+  private void setInternalDelegationSignerData(Set<DelegationSignerData> dsData) {
+    this.dsData = dsData;
+  }
+
  public final String getCurrentSponsorClientId() {
    return getPersistedCurrentSponsorClientId();
  }
@@ -655,13 +719,31 @@ public class DomainContent extends EppResource
            + " use DomainBase instead");
  }

+  /**
+   * Obtains a history id from the given key.
+   *
+   * <p>The key must be a composite key either of the form domain-key/history-key/long-event-key or
+   * EntityGroupRoot/long-event-key (for legacy keys). In the latter case or for a null key returns
+   * a history id of null.
+   */
+  public static Long getHistoryId(VKey<?> key) {
+    if (key == null) {
+      return null;
+    }
+    Key<?> parent = key.getOfyKey().getParent();
+    if (parent == null || parent.getKind().equals("EntityGroupRoot")) {
+      return null;
+    }
+    return parent.getId();
+  }
+
  /** Predicate to determine if a given {@link DesignatedContact} is the registrant. */
  static final Predicate<DesignatedContact> IS_REGISTRANT =
      (DesignatedContact contact) -> DesignatedContact.Type.REGISTRANT.equals(contact.type);

  /** An override of {@link EppResource#asBuilder} with tighter typing. */
  @Override
-  public Builder asBuilder() {
+  public Builder<? extends DomainContent, ?> asBuilder() {
    return new Builder<>(clone(this));
  }

@@ -698,10 +780,16 @@ public class DomainContent extends EppResource
      instance.tld = getTldFromDomainName(instance.fullyQualifiedDomainName);

      T newDomain = super.build();
-      // Hibernate throws exception if gracePeriods is null because we enabled all cascadable
-      // operations and orphan removal.
+      // Hibernate throws exception if gracePeriods or dsData is null because we enabled all
+      // cascadable operations and orphan removal.
      newDomain.gracePeriods =
          newDomain.gracePeriods == null ? ImmutableSet.of() : newDomain.gracePeriods;
+      newDomain.dsData =
+          newDomain.dsData == null
+              ? ImmutableSet.of()
+              : newDomain.dsData.stream()
+                  .map(ds -> ds.cloneWithDomainRepoId(instance.getRepoId()))
+                  .collect(toImmutableSet());
      return newDomain;
    }

@@ -824,16 +912,19 @@ public class DomainContent extends EppResource

    public B setDeletePollMessage(VKey<PollMessage.OneTime> deletePollMessage) {
      getInstance().deletePollMessage = deletePollMessage;
+      getInstance().deletePollMessageHistoryId = getHistoryId(deletePollMessage);
      return thisCastToDerived();
    }

    public B setAutorenewBillingEvent(VKey<BillingEvent.Recurring> autorenewBillingEvent) {
      getInstance().autorenewBillingEvent = autorenewBillingEvent;
+      getInstance().autorenewBillingEventHistoryId = getHistoryId(autorenewBillingEvent);
      return thisCastToDerived();
    }

    public B setAutorenewPollMessage(VKey<PollMessage.Autorenew> autorenewPollMessage) {
      getInstance().autorenewPollMessage = autorenewPollMessage;
+      getInstance().autorenewPollMessageHistoryId = getHistoryId(autorenewPollMessage);
      return thisCastToDerived();
    }

@@ -14,18 +14,23 @@

 package google.registry.model.domain;

+import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;

+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EntitySubclass;
 import com.googlecode.objectify.annotation.Ignore;
-import google.registry.model.EppResource;
 import google.registry.model.ImmutableObject;
 import google.registry.model.domain.DomainHistory.DomainHistoryId;
+import google.registry.model.domain.secdns.DomainDsDataHistory;
 import google.registry.model.host.HostResource;
 import google.registry.model.reporting.DomainTransactionRecord;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
 import java.io.Serializable;
 import java.util.Optional;
 import java.util.Set;
@@ -38,12 +43,12 @@ import javax.persistence.CascadeType;
 import javax.persistence.Column;
 import javax.persistence.ElementCollection;
 import javax.persistence.Entity;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
+import javax.persistence.FetchType;
 import javax.persistence.Id;
 import javax.persistence.IdClass;
 import javax.persistence.Index;
 import javax.persistence.JoinColumn;
+import javax.persistence.JoinColumns;
 import javax.persistence.JoinTable;
 import javax.persistence.OneToMany;
 import javax.persistence.PostLoad;
@@ -67,12 +72,24 @@ import javax.persistence.Table;
@EntitySubclass
@Access(AccessType.FIELD)
@IdClass(DomainHistoryId.class)
-public class DomainHistory extends HistoryEntry {
+public class DomainHistory extends HistoryEntry implements SqlEntity {

  // Store DomainContent instead of DomainBase so we don't pick up its @Id
  @Nullable DomainContent domainContent;

-  @Id String domainRepoId;
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getDomainRepoId() {
+    // We need to handle null case here because Hibernate sometimes accesses this method before
+    // parent gets initialized
+    return parent == null ? null : parent.getName();
+  }
+
+  /** This method is private because it is only used by Hibernate. */
+  @SuppressWarnings("unused")
+  private void setDomainRepoId(String domainRepoId) {
+    parent = Key.create(DomainBase.class, domainRepoId);
+  }

  // We could have reused domainContent.nsHosts here, but Hibernate throws a weird exception after
  // we change to use a composite primary key.
@@ -83,6 +100,24 @@ public class DomainHistory extends HistoryEntry {
  @Column(name = "host_repo_id")
  Set<VKey<HostResource>> nsHosts;

+  @OneToMany(
+      cascade = {CascadeType.ALL},
+      fetch = FetchType.EAGER,
+      orphanRemoval = true)
+  @JoinColumns({
+    @JoinColumn(
+        name = "domainHistoryRevisionId",
+        referencedColumnName = "historyRevisionId",
+        insertable = false,
+        updatable = false),
+    @JoinColumn(
+        name = "domainRepoId",
+        referencedColumnName = "domainRepoId",
+        insertable = false,
+        updatable = false)
+  })
+  Set<DomainDsDataHistory> dsDataHistories;
+
  @Override
  @Nullable
  @Access(AccessType.PROPERTY)
@@ -117,18 +152,27 @@ public class DomainHistory extends HistoryEntry {
   *
   * <p>This will be empty for any DomainHistory/HistoryEntry generated before this field was added,
   * mid-2017, as well as any action that does not generate billable events (e.g. updates).
+   *
+   * <p>This method is dedicated for Hibernate, external caller should use {@link
+   * #getDomainTransactionRecords()}.
   */
  @Access(AccessType.PROPERTY)
  @OneToMany(cascade = {CascadeType.ALL})
  @JoinColumn(name = "historyRevisionId", referencedColumnName = "historyRevisionId")
  @JoinColumn(name = "domainRepoId", referencedColumnName = "domainRepoId")
-  @Override
-  public Set<DomainTransactionRecord> getDomainTransactionRecords() {
-    return super.getDomainTransactionRecords();
+  @SuppressWarnings("unused")
+  private Set<DomainTransactionRecord> getInternalDomainTransactionRecords() {
+    return domainTransactionRecords;
+  }
+
+  /** Sets the domain transaction records. This method is dedicated for Hibernate. */
+  @SuppressWarnings("unused")
+  private void setInternalDomainTransactionRecords(
+      Set<DomainTransactionRecord> domainTransactionRecords) {
+    this.domainTransactionRecords = domainTransactionRecords;
  }

  @Id
-  @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "TempHistorySequenceGenerator")
  @Column(name = "historyRevisionId")
  @Access(AccessType.PROPERTY)
  @Override
@@ -141,6 +185,11 @@ public class DomainHistory extends HistoryEntry {
    return nsHosts;
  }

+  /** Returns the collection of {@link DomainDsDataHistory} instances. */
+  public ImmutableSet<DomainDsDataHistory> getDsDataHistories() {
+    return nullToEmptyImmutableCopy(dsDataHistories);
+  }
+
  /**
   * The values of all the fields on the {@link DomainContent} object after the action represented
   * by this history object was executed.
@@ -152,12 +201,14 @@ public class DomainHistory extends HistoryEntry {
  }

  /** The key to the {@link DomainBase} this is based off of. */
-  public VKey<DomainBase> getDomainRepoId() {
-    return VKey.create(DomainBase.class, domainRepoId, Key.create(DomainBase.class, domainRepoId));
+  public VKey<DomainBase> getParentVKey() {
+    return VKey.create(DomainBase.class, getDomainRepoId());
  }

+  /** Creates a {@link VKey} instance for this entity. */
  public VKey<DomainHistory> createVKey() {
-    return VKey.createSql(DomainHistory.class, new DomainHistoryId(domainRepoId, getId()));
+    return VKey.create(
+        DomainHistory.class, new DomainHistoryId(getDomainRepoId(), getId()), Key.create(this));
  }

  @PostLoad
@@ -168,9 +219,18 @@ public class DomainHistory extends HistoryEntry {
      // domainContent with a null object. Unfortunately, the updateTimestamp is never null in SQL.
      if (domainContent.getDomainName() == null) {
        domainContent = null;
+      } else {
+        if (domainContent.getRepoId() == null) {
+          domainContent = domainContent.asBuilder().setRepoId(parent.getName()).build();
+        }
      }
    }
-    parent = Key.create(DomainBase.class, domainRepoId);
+  }
+
+  // In Datastore, save as a HistoryEntry object regardless of this object's type
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(asHistoryEntry());
  }

  /** Class to represent the composite primary key of {@link DomainHistory} entity. */
@@ -188,19 +248,45 @@ public class DomainHistory extends HistoryEntry {
      this.id = id;
    }

-    String getDomainRepoId() {
+    /**
+     * Returns the domain repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getDomainRepoId() {
      return domainRepoId;
    }

-    void setDomainRepoId(String domainRepoId) {
-      this.domainRepoId = domainRepoId;
-    }
-
-    long getId() {
+    /**
+     * Returns the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private long getId() {
      return id;
    }

-    void setId(long id) {
+    /**
+     * Sets the domain repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setDomainRepoId(String domainRepoId) {
+      this.domainRepoId = domainRepoId;
+    }
+
+    /**
+     * Sets the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setId(long id) {
      this.id = id;
    }
  }
@@ -220,24 +306,29 @@ public class DomainHistory extends HistoryEntry {

    public Builder setDomainContent(DomainContent domainContent) {
      getInstance().domainContent = domainContent;
-      if (domainContent != null) {
-        getInstance().nsHosts = nullToEmptyImmutableCopy(domainContent.nsHosts);
-      }
      return this;
    }

    public Builder setDomainRepoId(String domainRepoId) {
-      getInstance().domainRepoId = domainRepoId;
      getInstance().parent = Key.create(DomainBase.class, domainRepoId);
      return this;
    }

-    // We can remove this once all HistoryEntries are converted to History objects
    @Override
-    public Builder setParent(Key<? extends EppResource> parent) {
-      super.setParent(parent);
-      getInstance().domainRepoId = parent.getName();
-      return this;
+    public DomainHistory build() {
+      DomainHistory instance = super.build();
+      // TODO(b/171990736): Assert instance.domainContent is not null after database migration.
+      // Note that we cannot assert that instance.domainContent is not null here because this
+      // builder is also used to convert legacy HistoryEntry objects to DomainHistory, when
+      // domainContent is not available.
+      if (instance.domainContent != null) {
+        instance.nsHosts = nullToEmptyImmutableCopy(instance.domainContent.nsHosts);
+        instance.dsDataHistories =
+            nullToEmptyImmutableCopy(instance.domainContent.getDsData()).stream()
+                .map(dsData -> DomainDsDataHistory.createFrom(instance.id, dsData))
+                .collect(toImmutableSet());
+      }
+      return instance;
    }
  }
 }
@@ -21,7 +21,9 @@ import com.googlecode.objectify.annotation.Embed;
 import google.registry.model.billing.BillingEvent;
 import google.registry.model.billing.BillingEvent.Recurring;
 import google.registry.model.domain.rgp.GracePeriodStatus;
+import google.registry.model.ofy.ObjectifyService;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import javax.annotation.Nullable;
 import javax.persistence.Entity;
 import javax.persistence.Index;
@@ -37,7 +39,7 @@ import org.joda.time.DateTime;
@Embed
@Entity
@Table(indexes = @Index(columnList = "domainRepoId"))
-public class GracePeriod extends GracePeriodBase {
+public class GracePeriod extends GracePeriodBase implements DatastoreAndSqlEntity {

  private static GracePeriod createInternal(
      GracePeriodStatus type,
@@ -52,12 +54,15 @@ public class GracePeriod extends GracePeriodBase {
        (billingEventRecurring != null) == GracePeriodStatus.AUTO_RENEW.equals(type),
        "Recurring billing events must be present on (and only on) autorenew grace periods");
    GracePeriod instance = new GracePeriod();
+    instance.id = ObjectifyService.allocateId();
    instance.type = checkArgumentNotNull(type);
    instance.domainRepoId = checkArgumentNotNull(domainRepoId);
    instance.expirationTime = checkArgumentNotNull(expirationTime);
    instance.clientId = checkArgumentNotNull(clientId);
    instance.billingEventOneTime = billingEventOneTime;
+    instance.billingEventOneTimeHistoryId = DomainBase.getHistoryId(billingEventOneTime);
    instance.billingEventRecurring = billingEventRecurring;
+    instance.billingEventRecurringHistoryId = DomainBase.getHistoryId(billingEventRecurring);
    return instance;
  }

@@ -107,14 +112,28 @@ public class GracePeriod extends GracePeriodBase {
  }

  /**
-   * Returns a clone of this {@link GracePeriod} with {@link #domainRepoId} set to the given value.
+   * Returns a clone of this {@link GracePeriod} with {@link #domainRepoId} set to the given value
+   * and reconstructed history ids.
   *
   * <p>TODO(b/162739503): Remove this function after fully migrating to Cloud SQL.
   */
-  public GracePeriod cloneWithDomainRepoId(String domainRepoId) {
+  public GracePeriod cloneAfterOfyLoad(String domainRepoId) {
    GracePeriod clone = clone(this);
+    clone.id = ObjectifyService.allocateId();
    clone.domainRepoId = checkArgumentNotNull(domainRepoId);
+    clone.restoreHistoryIds();
    return clone;
  }

+  /**
+   * Returns a clone of this {@link GracePeriod} with {@link #billingEventRecurring} set to the
+   * given value.
+   *
+   * <p>TODO(b/162231099): Remove this function after duplicate id issue is solved.
+   */
+  public GracePeriod cloneWithRecurringBillingEvent(VKey<BillingEvent.Recurring> recurring) {
+    GracePeriod clone = clone(this);
+    clone.billingEventRecurring = recurring;
+    return clone;
+  }
 }
@@ -14,18 +14,21 @@

 package google.registry.model.domain;

+import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Embed;
 import com.googlecode.objectify.annotation.Ignore;
 import google.registry.model.ImmutableObject;
+import google.registry.model.ModelUtils;
 import google.registry.model.billing.BillingEvent;
 import google.registry.model.billing.BillingEvent.OneTime;
 import google.registry.model.domain.rgp.GracePeriodStatus;
 import google.registry.persistence.VKey;
+import java.lang.reflect.Field;
+import java.util.LinkedHashMap;
+import java.util.Map;
 import javax.persistence.Column;
 import javax.persistence.EnumType;
 import javax.persistence.Enumerated;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.MappedSuperclass;
 import org.joda.time.DateTime;

@@ -36,7 +39,6 @@ public class GracePeriodBase extends ImmutableObject {

  /** Unique id required for hibernate representation. */
  @javax.persistence.Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
  @Ignore
  Long id;

@@ -67,6 +69,10 @@ public class GracePeriodBase extends ImmutableObject {
  @Column(name = "billing_event_id")
  VKey<OneTime> billingEventOneTime = null;

+  @Ignore
+  @Column(name = "billing_event_history_id")
+  Long billingEventOneTimeHistoryId;
+
  /**
   * The recurring billing event corresponding to the action that triggered this grace period, if
   * applicable - i.e. if the action was an autorenew - or null in all other cases.
@@ -75,6 +81,14 @@ public class GracePeriodBase extends ImmutableObject {
  @Column(name = "billing_recurrence_id")
  VKey<BillingEvent.Recurring> billingEventRecurring = null;

+  @Ignore
+  @Column(name = "billing_recurrence_history_id")
+  Long billingEventRecurringHistoryId;
+
+  public long getId() {
+    return id;
+  }
+
  public GracePeriodStatus getType() {
    return type;
  }
@@ -101,6 +115,7 @@ public class GracePeriodBase extends ImmutableObject {
   * period is not AUTO_RENEW.
   */
  public VKey<BillingEvent.OneTime> getOneTimeBillingEvent() {
+    restoreOfyKeys();
    return billingEventOneTime;
  }

@@ -109,6 +124,63 @@ public class GracePeriodBase extends ImmutableObject {
   * period is AUTO_RENEW.
   */
  public VKey<BillingEvent.Recurring> getRecurringBillingEvent() {
+    restoreOfyKeys();
    return billingEventRecurring;
  }
+
+  /**
+   * Restores history ids for composite VKeys after a load from datastore.
+   *
+   * <p>For use by DomainContent.load() ONLY.
+   */
+  protected void restoreHistoryIds() {
+    billingEventOneTimeHistoryId = DomainBase.getHistoryId(billingEventOneTime);
+    billingEventRecurringHistoryId = DomainBase.getHistoryId(billingEventRecurring);
+  }
+
+  /**
+   * Override {@link ImmutableObject#getSignificantFields()} to exclude "id", which breaks equality
+   * testing in the unit tests.
+   */
+  @Override
+  protected Map<Field, Object> getSignificantFields() {
+    restoreOfyKeys();
+    // Can't use streams or ImmutableMap because we can have null values.
+    Map<Field, Object> result = new LinkedHashMap();
+    for (Map.Entry<Field, Object> entry : ModelUtils.getFieldValues(this).entrySet()) {
+      if (!entry.getKey().getName().equals("id")) {
+        result.put(entry.getKey(), entry.getValue());
+      }
+    }
+    return result;
+  }
+
+  /**
+   * Restores Ofy keys in the billing events.
+   *
+   * <p>This must be called by all methods that access the one time or recurring billing event keys.
+   * When the billing event keys are loaded from SQL, they are loaded as asymmetric keys because the
+   * database columns that we load them from do not contain all of the information necessary to
+   * reconsitute the Ofy side of the key. In other cases, we restore the Ofy key during the
+   * hibernate {@link javax.persistence.PostLoad} method from the other fields of the object, but we
+   * have been unable to make this work with hibernate's internal persistence model in this case
+   * because the {@link GracePeriod}'s hash code is evaluated prior to these calls, and would be
+   * invalidated by changing the fields.
+   */
+  private final synchronized void restoreOfyKeys() {
+    if (billingEventOneTime != null && !billingEventOneTime.maybeGetOfyKey().isPresent()) {
+      billingEventOneTime =
+          DomainBase.restoreOfyFrom(
+              Key.create(DomainBase.class, domainRepoId),
+              billingEventOneTime,
+              billingEventOneTimeHistoryId);
+    }
+    if (billingEventRecurring != null && !billingEventRecurring.maybeGetOfyKey().isPresent()) {
+      billingEventRecurring =
+          DomainBase.restoreOfyFrom(
+              Key.create(DomainBase.class, domainRepoId),
+              billingEventRecurring,
+              billingEventRecurringHistoryId);
+    }
+  }
 }
@@ -14,96 +14,235 @@

 package google.registry.model.domain.secdns;

+import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
+
 import com.googlecode.objectify.annotation.Embed;
 import google.registry.model.ImmutableObject;
+import google.registry.model.domain.secdns.DelegationSignerData.DomainDsDataId;
+import java.io.Serializable;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.Index;
+import javax.persistence.Table;
 import javax.xml.bind.DatatypeConverter;
-import javax.xml.bind.annotation.XmlElement;
 import javax.xml.bind.annotation.XmlType;
-import javax.xml.bind.annotation.adapters.HexBinaryAdapter;
-import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;

 /**
 * Holds the data necessary to construct a single Delegation Signer (DS) record for a domain.
 *
 * @see <a href="http://tools.ietf.org/html/rfc5910">RFC 5910</a>
 * @see <a href="http://tools.ietf.org/html/rfc4034">RFC 4034</a>
+ *     <p>TODO(shicong): Rename this class to DomainDsData.
 */
@Embed
@XmlType(name = "dsData")
-@javax.persistence.Entity
-public class DelegationSignerData extends ImmutableObject {
+@Entity
+@IdClass(DomainDsDataId.class)
+@Table(indexes = @Index(columnList = "domainRepoId"))
+public class DelegationSignerData extends DomainDsDataBase {

  private DelegationSignerData() {}

-  /** The identifier for this particular key in the domain. */
-  @javax.persistence.Id int keyTag;
-
-  /**
-   * The algorithm used by this key.
-   *
-   * @see <a href="http://tools.ietf.org/html/rfc4034#appendix-A.1">RFC 4034 Appendix A.1</a>
-   */
-  @XmlElement(name = "alg")
-  int algorithm;
-
-  /**
-   * The algorithm used to generate the digest.
-   *
-   * @see <a href="http://tools.ietf.org/html/rfc4034#appendix-A.2">RFC 4034 Appendix A.2</a>
-   */
-  int digestType;
-
-  /**
-   * The hexBinary digest of the public key.
-   *
-   * @see <a href="http://tools.ietf.org/html/rfc4034#section-5.1.4">RFC 4034 Section 5.1.4</a>
-   */
-  @XmlJavaTypeAdapter(HexBinaryAdapter.class)
-  byte[] digest;
+  @Override
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getDomainRepoId() {
+    return super.getDomainRepoId();
+  }

+  @Override
+  @Id
+  @Access(AccessType.PROPERTY)
  public int getKeyTag() {
-    return keyTag;
+    return super.getKeyTag();
  }

+  @Override
+  @Id
+  @Access(AccessType.PROPERTY)
  public int getAlgorithm() {
-    return algorithm;
+    return super.getAlgorithm();
  }

+  @Override
+  @Id
+  @Access(AccessType.PROPERTY)
  public int getDigestType() {
-    return digestType;
+    return super.getDigestType();
  }

+  @Override
+  @Id
+  @Access(AccessType.PROPERTY)
  public byte[] getDigest() {
-    return digest;
+    return super.getDigest();
  }

-  public String getDigestAsString() {
-    return digest == null ? "" : DatatypeConverter.printHexBinary(digest);
+  public DelegationSignerData cloneWithDomainRepoId(String domainRepoId) {
+    DelegationSignerData clone = clone(this);
+    clone.domainRepoId = checkArgumentNotNull(domainRepoId);
+    return clone;
+  }
+
+  public DelegationSignerData cloneWithoutDomainRepoId() {
+    DelegationSignerData clone = clone(this);
+    clone.domainRepoId = null;
+    return clone;
  }

  public static DelegationSignerData create(
-      int keyTag, int algorithm, int digestType, byte[] digest) {
+      int keyTag, int algorithm, int digestType, byte[] digest, String domainRepoId) {
    DelegationSignerData instance = new DelegationSignerData();
    instance.keyTag = keyTag;
    instance.algorithm = algorithm;
    instance.digestType = digestType;
    instance.digest = digest;
+    instance.domainRepoId = domainRepoId;
    return instance;
  }

+  public static DelegationSignerData create(
+      int keyTag, int algorithm, int digestType, byte[] digest) {
+    return create(keyTag, algorithm, digestType, digest, null);
+  }
+
  public static DelegationSignerData create(
      int keyTag, int algorithm, int digestType, String digestAsHex) {
    return create(keyTag, algorithm, digestType, DatatypeConverter.parseHexBinary(digestAsHex));
  }

-  /**
-   * Returns the presentation format of this DS record.
-   *
-   * @see <a href="https://tools.ietf.org/html/rfc4034#section-5.3">RFC 4034 Section 5.3</a>
-   */
-  public String toRrData() {
-    return String.format(
-        "%d %d %d %s",
-        this.keyTag, this.algorithm, this.digestType, DatatypeConverter.printHexBinary(digest));
+  /** Class to represent the composite primary key of {@link DelegationSignerData} entity. */
+  static class DomainDsDataId extends ImmutableObject implements Serializable {
+
+    String domainRepoId;
+
+    int keyTag;
+
+    int algorithm;
+
+    int digestType;
+
+    byte[] digest;
+
+    /** Hibernate requires this default constructor. */
+    private DomainDsDataId() {}
+
+    /** Constructs a {link DomainDsDataId} instance. */
+    DomainDsDataId(String domainRepoId, int keyTag, int algorithm, int digestType, byte[] digest) {
+      this.domainRepoId = domainRepoId;
+      this.keyTag = keyTag;
+      this.algorithm = algorithm;
+      this.digestType = digestType;
+      this.digest = digest;
+    }
+
+    /**
+     * Returns the domain repository ID.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getDomainRepoId() {
+      return domainRepoId;
+    }
+
+    /**
+     * Returns the key tag.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private int getKeyTag() {
+      return keyTag;
+    }
+
+    /**
+     * Returns the algorithm.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private int getAlgorithm() {
+      return algorithm;
+    }
+
+    /**
+     * Returns the digest type.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private int getDigestType() {
+      return digestType;
+    }
+
+    /**
+     * Returns the digest.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private byte[] getDigest() {
+      return digest;
+    }
+
+    /**
+     * Sets the domain repository ID.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private void setDomainRepoId(String domainRepoId) {
+      this.domainRepoId = domainRepoId;
+    }
+
+    /**
+     * Sets the key tag.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private void setKeyTag(int keyTag) {
+      this.keyTag = keyTag;
+    }
+
+    /**
+     * Sets the algorithm.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private void setAlgorithm(int algorithm) {
+      this.algorithm = algorithm;
+    }
+
+    /**
+     * Sets the digest type.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private void setDigestType(int digestType) {
+      this.digestType = digestType;
+    }
+
+    /**
+     * Sets the digest.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private void setDigest(byte[] digest) {
+      this.digest = digest;
+    }
+
+    public static DomainDsDataId create(
+        String domainRepoId, int keyTag, int algorithm, int digestType, byte[] digest) {
+      return new DomainDsDataId(
+          domainRepoId, keyTag, algorithm, digestType, checkArgumentNotNull(digest));
+    }
  }
 }
@@ -0,0 +1,150 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.domain.secdns;
+
+import com.googlecode.objectify.annotation.Embed;
+import com.googlecode.objectify.annotation.Ignore;
+import google.registry.model.ImmutableObject;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.MappedSuperclass;
+import javax.persistence.Transient;
+import javax.xml.bind.DatatypeConverter;
+import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlTransient;
+import javax.xml.bind.annotation.adapters.HexBinaryAdapter;
+import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
+
+/** Base class for {@link DelegationSignerData} and {@link DomainDsDataHistory}. */
+@Embed
+@MappedSuperclass
+@Access(AccessType.FIELD)
+public abstract class DomainDsDataBase extends ImmutableObject {
+
+  @Ignore @XmlTransient @Transient String domainRepoId;
+
+  /** The identifier for this particular key in the domain. */
+  @Transient int keyTag;
+
+  /**
+   * The algorithm used by this key.
+   *
+   * @see <a href="http://tools.ietf.org/html/rfc4034#appendix-A.1">RFC 4034 Appendix A.1</a>
+   */
+  @Transient
+  @XmlElement(name = "alg")
+  int algorithm;
+
+  /**
+   * The algorithm used to generate the digest.
+   *
+   * @see <a href="http://tools.ietf.org/html/rfc4034#appendix-A.2">RFC 4034 Appendix A.2</a>
+   */
+  @Transient int digestType;
+
+  /**
+   * The hexBinary digest of the public key.
+   *
+   * @see <a href="http://tools.ietf.org/html/rfc4034#section-5.1.4">RFC 4034 Section 5.1.4</a>
+   */
+  @Transient
+  @XmlJavaTypeAdapter(HexBinaryAdapter.class)
+  byte[] digest;
+
+  public String getDomainRepoId() {
+    return domainRepoId;
+  }
+
+  public int getKeyTag() {
+    return keyTag;
+  }
+
+  public int getAlgorithm() {
+    return algorithm;
+  }
+
+  public int getDigestType() {
+    return digestType;
+  }
+
+  public byte[] getDigest() {
+    return digest;
+  }
+
+  /**
+   * Sets the domain repository ID.
+   *
+   * <p>This method is private because it is only used by Hibernate.
+   */
+  @SuppressWarnings("unused")
+  private void setDomainRepoId(String domainRepoId) {
+    this.domainRepoId = domainRepoId;
+  }
+
+  /**
+   * Sets the key tag.
+   *
+   * <p>This method is private because it is only used by Hibernate.
+   */
+  @SuppressWarnings("unused")
+  private void setKeyTag(int keyTag) {
+    this.keyTag = keyTag;
+  }
+
+  /**
+   * Sets the algorithm.
+   *
+   * <p>This method is private because it is only used by Hibernate.
+   */
+  @SuppressWarnings("unused")
+  private void setAlgorithm(int algorithm) {
+    this.algorithm = algorithm;
+  }
+
+  /**
+   * Sets the digest type.
+   *
+   * <p>This method is private because it is only used by Hibernate.
+   */
+  @SuppressWarnings("unused")
+  private void setDigestType(int digestType) {
+    this.digestType = digestType;
+  }
+
+  /**
+   * Sets the digest.
+   *
+   * <p>This method is private because it is only used by Hibernate.
+   */
+  @SuppressWarnings("unused")
+  private void setDigest(byte[] digest) {
+    this.digest = digest;
+  }
+
+  public String getDigestAsString() {
+    return digest == null ? "" : DatatypeConverter.printHexBinary(digest);
+  }
+
+  /**
+   * Returns the presentation format of this DS record.
+   *
+   * @see <a href="https://tools.ietf.org/html/rfc4034#section-5.3">RFC 4034 Section 5.3</a>
+   */
+  public String toRrData() {
+    return String.format(
+        "%d %d %d %s",
+        this.keyTag, this.algorithm, this.digestType, DatatypeConverter.printHexBinary(digest));
+  }
+}
@@ -0,0 +1,92 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.domain.secdns;
+
+import com.google.common.collect.ImmutableList;
+import google.registry.model.domain.DomainHistory;
+import google.registry.model.ofy.ObjectifyService;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+
+/** Entity class to represent a historic {@link DelegationSignerData}. */
+@Entity
+public class DomainDsDataHistory extends DomainDsDataBase implements SqlEntity {
+
+  @Id Long dsDataHistoryRevisionId;
+
+  /** ID of the {@link DomainHistory} entity that this entity is associated with. */
+  @Column(nullable = false)
+  Long domainHistoryRevisionId;
+
+  private DomainDsDataHistory() {}
+
+  /**
+   * Creates a {@link DomainDsDataHistory} instance from given {@link #domainHistoryRevisionId} and
+   * {@link DelegationSignerData} instance.
+   */
+  public static DomainDsDataHistory createFrom(
+      long domainHistoryRevisionId, DelegationSignerData dsData) {
+    DomainDsDataHistory instance = new DomainDsDataHistory();
+    instance.domainHistoryRevisionId = domainHistoryRevisionId;
+    instance.domainRepoId = dsData.domainRepoId;
+    instance.keyTag = dsData.getKeyTag();
+    instance.algorithm = dsData.getAlgorithm();
+    instance.digestType = dsData.getDigestType();
+    instance.digest = dsData.getDigest();
+    instance.dsDataHistoryRevisionId = ObjectifyService.allocateId();
+    return instance;
+  }
+
+  @Override
+  @Access(AccessType.PROPERTY)
+  public String getDomainRepoId() {
+    return super.getDomainRepoId();
+  }
+
+  @Override
+  @Access(AccessType.PROPERTY)
+  public int getKeyTag() {
+    return super.getKeyTag();
+  }
+
+  @Override
+  @Access(AccessType.PROPERTY)
+  public int getAlgorithm() {
+    return super.getAlgorithm();
+  }
+
+  @Override
+  @Access(AccessType.PROPERTY)
+  public int getDigestType() {
+    return super.getDigestType();
+  }
+
+  @Override
+  @Access(AccessType.PROPERTY)
+  @Column(nullable = false)
+  public byte[] getDigest() {
+    return super.getDigest();
+  }
+
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(); // not persisted in Datastore
+  }
+}
@@ -20,8 +20,10 @@ import static com.google.common.base.Strings.nullToEmpty;

 import com.google.common.collect.ImmutableSet;
 import google.registry.model.EppResource;
+import google.registry.model.contact.ContactBase;
 import google.registry.model.contact.ContactResource;
 import google.registry.model.domain.DomainBase;
+import google.registry.model.domain.DomainContent;
 import google.registry.model.host.HostBase;
 import google.registry.model.host.HostResource;
 import google.registry.model.translators.EnumToAttributeAdapter.EppEnum;
@@ -128,9 +130,15 @@ public enum StatusValue implements EppEnum {

  /** Enum to help clearly list which resource types a status value is allowed to be present on. */
  private enum AllowedOn {
-    ALL(ContactResource.class, DomainBase.class, HostBase.class, HostResource.class),
+    ALL(
+        ContactBase.class,
+        ContactResource.class,
+        DomainContent.class,
+        DomainBase.class,
+        HostBase.class,
+        HostResource.class),
    NONE,
-    DOMAINS(DomainBase.class);
+    DOMAINS(DomainContent.class, DomainBase.class);

    private final ImmutableSet<Class<? extends EppResource>> classes;

@@ -14,20 +14,24 @@

 package google.registry.model.host;

+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EntitySubclass;
-import google.registry.model.EppResource;
+import google.registry.model.ImmutableObject;
+import google.registry.model.host.HostHistory.HostHistoryId;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import java.io.Serializable;
 import java.util.Optional;
 import javax.annotation.Nullable;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
 import javax.persistence.Column;
 import javax.persistence.Entity;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.Id;
+import javax.persistence.IdClass;
 import javax.persistence.PostLoad;

 /**
@@ -48,16 +52,27 @@ import javax.persistence.PostLoad;
    })
@EntitySubclass
@Access(AccessType.FIELD)
-public class HostHistory extends HistoryEntry {
+@IdClass(HostHistoryId.class)
+public class HostHistory extends HistoryEntry implements SqlEntity {

  // Store HostBase instead of HostResource so we don't pick up its @Id
  @Nullable HostBase hostBase;

-  @Column(nullable = false)
-  VKey<HostResource> hostRepoId;
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getHostRepoId() {
+    // We need to handle null case here because Hibernate sometimes accesses this method before
+    // parent gets initialized
+    return parent == null ? null : parent.getName();
+  }
+
+  /** This method is private because it is only used by Hibernate. */
+  @SuppressWarnings("unused")
+  private void setHostRepoId(String hostRepoId) {
+    parent = Key.create(HostResource.class, hostRepoId);
+  }

  @Id
-  @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "TempHistorySequenceGenerator")
  @Column(name = "historyRevisionId")
  @Access(AccessType.PROPERTY)
  @Override
@@ -76,8 +91,14 @@ public class HostHistory extends HistoryEntry {
  }

  /** The key to the {@link google.registry.model.host.HostResource} this is based off of. */
-  public VKey<HostResource> getHostRepoId() {
-    return hostRepoId;
+  public VKey<HostResource> getParentVKey() {
+    return VKey.create(HostResource.class, getHostRepoId());
+  }
+
+  /** Creates a {@link VKey} instance for this entity. */
+  public VKey<HostHistory> createVKey() {
+    return VKey.create(
+        HostHistory.class, new HostHistoryId(getHostRepoId(), getId()), Key.create(this));
  }

  @PostLoad
@@ -87,10 +108,70 @@ public class HostHistory extends HistoryEntry {
    if (hostBase != null && hostBase.getHostName() == null) {
      hostBase = null;
    }
-    // Fill in the full, symmetric, parent repo ID key
-    Key<HostResource> parentKey = Key.create(HostResource.class, (String) hostRepoId.getSqlKey());
-    parent = parentKey;
-    hostRepoId = VKey.create(HostResource.class, hostRepoId.getSqlKey(), parentKey);
+  }
+
+  // In Datastore, save as a HistoryEntry object regardless of this object's type
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(asHistoryEntry());
+  }
+
+  /** Class to represent the composite primary key of {@link HostHistory} entity. */
+  static class HostHistoryId extends ImmutableObject implements Serializable {
+
+    private String hostRepoId;
+
+    private Long id;
+
+    /** Hibernate requires this default constructor. */
+    private HostHistoryId() {}
+
+    HostHistoryId(String hostRepoId, long id) {
+      this.hostRepoId = hostRepoId;
+      this.id = id;
+    }
+
+    /**
+     * Returns the host repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getHostRepoId() {
+      return hostRepoId;
+    }
+
+    /**
+     * Returns the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private long getId() {
+      return id;
+    }
+
+    /**
+     * Sets the host repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setHostRepoId(String hostRepoId) {
+      this.hostRepoId = hostRepoId;
+    }
+
+    /**
+     * Sets the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setId(long id) {
+      this.id = id;
+    }
  }

  @Override
@@ -111,18 +192,8 @@ public class HostHistory extends HistoryEntry {
      return this;
    }

-    public Builder setHostRepoId(VKey<HostResource> hostRepoId) {
-      getInstance().hostRepoId = hostRepoId;
-      hostRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
-      return this;
-    }
-
-    // We can remove this once all HistoryEntries are converted to History objects
-    @Override
-    public Builder setParent(Key<? extends EppResource> parent) {
-      super.setParent(parent);
-      getInstance().hostRepoId =
-          VKey.create(HostResource.class, parent.getName(), (Key<HostResource>) parent);
+    public Builder setHostRepoId(String hostRepoId) {
+      getInstance().parent = Key.create(HostResource.class, hostRepoId);
      return this;
    }
  }
@@ -20,7 +20,6 @@ import static google.registry.config.RegistryConfig.getEppResourceMaxCachedEntri
 import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.TypeUtils.instantiate;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;

 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.cache.CacheBuilder;
@@ -244,7 +243,7 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
  private static LoadingCache<Key<ForeignKeyIndex<?>>, Optional<ForeignKeyIndex<?>>>
      createForeignKeyIndexesCache(Duration expiry) {
    return CacheBuilder.newBuilder()
-        .expireAfterWrite(expiry.getMillis(), MILLISECONDS)
+        .expireAfterWrite(java.time.Duration.ofMillis(expiry.getMillis()))
        .maximumSize(getEppResourceMaxCachedEntries())
        .build(CACHE_LOADER);
  }
@@ -23,7 +23,9 @@ import com.google.common.base.Functions;
 import com.google.common.collect.ImmutableCollection;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Streams;
 import com.googlecode.objectify.Key;
+import com.googlecode.objectify.Result;
 import google.registry.model.contact.ContactHistory;
 import google.registry.model.host.HostHistory;
 import google.registry.model.reporting.HistoryEntry;
@@ -102,12 +104,22 @@ public class DatastoreTransactionManager implements TransactionManager {

  @Override
  public void insert(Object entity) {
-    saveEntity(entity);
+    put(entity);
  }

  @Override
  public void insertAll(ImmutableCollection<?> entities) {
-    getOfy().save().entities(entities);
+    putAll(entities);
+  }
+
+  @Override
+  public void insertWithoutBackup(Object entity) {
+    putWithoutBackup(entity);
+  }
+
+  @Override
+  public void insertAllWithoutBackup(ImmutableCollection<?> entities) {
+    putAllWithoutBackup(entities);
  }

  @Override
@@ -117,17 +129,37 @@ public class DatastoreTransactionManager implements TransactionManager {

  @Override
  public void putAll(ImmutableCollection<?> entities) {
-    getOfy().save().entities(entities);
+    syncIfTransactionless(getOfy().save().entities(entities));
+  }
+
+  @Override
+  public void putWithoutBackup(Object entity) {
+    syncIfTransactionless(getOfy().saveWithoutBackup().entities(entity));
+  }
+
+  @Override
+  public void putAllWithoutBackup(ImmutableCollection<?> entities) {
+    syncIfTransactionless(getOfy().saveWithoutBackup().entities(entities));
  }

  @Override
  public void update(Object entity) {
-    saveEntity(entity);
+    put(entity);
  }

  @Override
  public void updateAll(ImmutableCollection<?> entities) {
-    getOfy().save().entities(entities);
+    putAll(entities);
+  }
+
+  @Override
+  public void updateWithoutBackup(Object entity) {
+    putWithoutBackup(entity);
+  }
+
+  @Override
+  public void updateAllWithoutBackup(ImmutableCollection<?> entities) {
+    putAllWithoutBackup(entities);
  }

  @Override
@@ -158,6 +190,11 @@ public class DatastoreTransactionManager implements TransactionManager {
    return result;
  }

+  @Override
+  public <T> T load(T entity) {
+    return ofy().load().entity(entity).now();
+  }
+
  @Override
  public <T> ImmutableMap<VKey<? extends T>, T> load(Iterable<? extends VKey<? extends T>> keys) {
    // Keep track of the Key -> VKey mapping so we can translate them back.
@@ -175,13 +212,17 @@ public class DatastoreTransactionManager implements TransactionManager {

  @Override
  public <T> ImmutableList<T> loadAll(Class<T> clazz) {
-    // We can do a ofy().load().type(clazz), but this doesn't work in a transaction.
-    throw new UnsupportedOperationException("Not available in the Datastore transaction manager");
+    return ImmutableList.copyOf(getOfy().load().type(clazz));
+  }
+
+  @Override
+  public <T> ImmutableList<T> loadAll(Iterable<T> entities) {
+    return ImmutableList.copyOf(getOfy().load().entities(entities).values());
  }

  @Override
  public void delete(VKey<?> key) {
-    getOfy().delete().key(key.getOfyKey()).now();
+    syncIfTransactionless(getOfy().delete().key(key.getOfyKey()));
  }

  @Override
@@ -192,7 +233,35 @@ public class DatastoreTransactionManager implements TransactionManager {
        StreamSupport.stream(vKeys.spliterator(), false)
            .map(VKey::getOfyKey)
            .collect(toImmutableList());
-    getOfy().delete().keys(list).now();
+    syncIfTransactionless(getOfy().delete().keys(list));
+  }
+
+  @Override
+  public void delete(Object entity) {
+    syncIfTransactionless(getOfy().delete().entity(entity));
+  }
+
+  @Override
+  public void deleteWithoutBackup(VKey<?> key) {
+    syncIfTransactionless(getOfy().deleteWithoutBackup().key(key.getOfyKey()));
+  }
+
+  @Override
+  public void deleteWithoutBackup(Iterable<? extends VKey<?>> keys) {
+    syncIfTransactionless(
+        getOfy()
+            .deleteWithoutBackup()
+            .keys(Streams.stream(keys).map(VKey::getOfyKey).collect(toImmutableList())));
+  }
+
+  @Override
+  public void deleteWithoutBackup(Object entity) {
+    syncIfTransactionless(getOfy().deleteWithoutBackup().entity(entity));
+  }
+
+  @Override
+  public void clearSessionCache() {
+    getOfy().clearSessionCache();
  }

  /**
@@ -209,7 +278,7 @@ public class DatastoreTransactionManager implements TransactionManager {
    if (entity instanceof HistoryEntry) {
      entity = ((HistoryEntry) entity).asHistoryEntry();
    }
-    getOfy().save().entity(entity);
+    syncIfTransactionless(getOfy().save().entity(entity));
  }

  @SuppressWarnings("unchecked")
@@ -227,4 +296,19 @@ public class DatastoreTransactionManager implements TransactionManager {
  private <T> T loadNullable(VKey<T> key) {
    return toChildHistoryEntryIfPossible(getOfy().load().key(key.getOfyKey()).now());
  }
+
+  /**
+   * Executes the given {@link Result} instance synchronously if not in a transaction.
+   *
+   * <p>The {@link Result} instance contains a task that will be executed by Objectify
+   * asynchronously. If it is in a transaction, we don't need to execute the task immediately
+   * because it is guaranteed to be done by the end of the transaction. However, if it is not in a
+   * transaction, we need to execute it in case the following code expects that happens before
+   * themselves.
+   */
+  private void syncIfTransactionless(Result<?> result) {
+    if (!inTransaction()) {
+      result.now();
+    }
+  }
 }
@@ -27,14 +27,18 @@ import com.googlecode.objectify.annotation.EntitySubclass;
 import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.Index;
+import com.googlecode.objectify.annotation.OnLoad;
 import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.Buildable;
 import google.registry.model.EppResource;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ExternalMessagingName;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.model.contact.ContactResource;
+import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.DomainRenewData;
 import google.registry.model.eppoutput.EppResponse.ResponseData;
+import google.registry.model.host.HostResource;
 import google.registry.model.poll.PendingActionNotificationResponse.ContactPendingActionNotificationResponse;
 import google.registry.model.poll.PendingActionNotificationResponse.DomainPendingActionNotificationResponse;
 import google.registry.model.poll.PendingActionNotificationResponse.HostPendingActionNotificationResponse;
@@ -54,10 +58,9 @@ import javax.persistence.Column;
 import javax.persistence.DiscriminatorColumn;
 import javax.persistence.DiscriminatorValue;
 import javax.persistence.Embedded;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.Inheritance;
 import javax.persistence.InheritanceType;
+import javax.persistence.PostLoad;
 import javax.persistence.Transient;
 import org.joda.time.DateTime;

@@ -98,7 +101,6 @@ public abstract class PollMessage extends ImmutableObject
  /** Entity id. */
  @Id
  @javax.persistence.Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
  @Column(name = "poll_message_id")
  Long id;

@@ -124,11 +126,11 @@ public abstract class PollMessage extends ImmutableObject

  @Ignore String hostRepoId;

-  @Ignore Long domainRevisionId;
+  @Ignore Long domainHistoryRevisionId;

-  @Ignore Long contactRevisionId;
+  @Ignore Long contactHistoryRevisionId;

-  @Ignore Long hostRevisionId;
+  @Ignore Long hostHistoryRevisionId;

  public Key<HistoryEntry> getParentKey() {
    return parent;
@@ -152,6 +154,34 @@ public abstract class PollMessage extends ImmutableObject

  public abstract ImmutableList<ResponseData> getResponseData();

+  @PostLoad
+  void postLoad() {
+    if (domainRepoId != null) {
+      parent =
+          Key.create(
+              Key.create(DomainBase.class, domainRepoId),
+              HistoryEntry.class,
+              domainHistoryRevisionId);
+    } else if (contactRepoId != null) {
+      parent =
+          Key.create(
+              Key.create(ContactResource.class, contactRepoId),
+              HistoryEntry.class,
+              contactHistoryRevisionId);
+    } else if (hostHistoryRevisionId != null) {
+      parent =
+          Key.create(
+              Key.create(HostResource.class, hostRepoId),
+              HistoryEntry.class,
+              hostHistoryRevisionId);
+    }
+  }
+
+  @OnLoad
+  void onLoad() {
+    setSqlForeignKeys(this);
+  }
+
  @Override
  public abstract VKey<? extends PollMessage> createVKey();

@@ -217,10 +247,30 @@ public abstract class PollMessage extends ImmutableObject
      checkArgumentNotNull(instance.clientId, "clientId must be specified");
      checkArgumentNotNull(instance.eventTime, "eventTime must be specified");
      checkArgumentNotNull(instance.parent, "parent must be specified");
+      checkArgumentNotNull(instance.parent.getParent(), "parent.getParent() must be specified");
+      setSqlForeignKeys(instance);
      return super.build();
    }
  }

+  private static void setSqlForeignKeys(PollMessage pollMessage) {
+    String grandparentKind = pollMessage.parent.getParent().getKind();
+    String repoId = pollMessage.parent.getParent().getName();
+    long historyRevisionId = pollMessage.parent.getId();
+    if (Key.getKind(DomainBase.class).equals(grandparentKind)) {
+      pollMessage.domainRepoId = repoId;
+      pollMessage.domainHistoryRevisionId = historyRevisionId;
+    } else if (Key.getKind(ContactResource.class).equals(grandparentKind)) {
+      pollMessage.contactRepoId = repoId;
+      pollMessage.contactHistoryRevisionId = historyRevisionId;
+    } else if (Key.getKind(HostResource.class).equals(grandparentKind)) {
+      pollMessage.hostRepoId = repoId;
+      pollMessage.hostHistoryRevisionId = historyRevisionId;
+    } else {
+      throw new IllegalArgumentException("Unknown grandparent kind: " + grandparentKind);
+    }
+  }
+
  /**
   * A one-time poll message.
   *
@@ -15,17 +15,30 @@
 package google.registry.model.rde;

 import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Verify.verify;
-import static com.google.common.base.Verify.verifyNotNull;
-import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.model.rde.RdeNamingUtils.makePartialName;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

 import com.google.common.base.VerifyException;
+import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Id;
+import com.googlecode.objectify.annotation.Ignore;
+import google.registry.model.BackupGroupRoot;
 import google.registry.model.ImmutableObject;
+import google.registry.model.rde.RdeRevision.RdeRevisionId;
+import google.registry.persistence.VKey;
+import google.registry.persistence.converter.LocalDateConverter;
+import google.registry.schema.replay.NonReplicatedEntity;
+import java.io.Serializable;
+import java.util.Optional;
+import javax.persistence.Column;
+import javax.persistence.Convert;
+import javax.persistence.EnumType;
+import javax.persistence.Enumerated;
+import javax.persistence.IdClass;
+import javax.persistence.Transient;
 import org.joda.time.DateTime;
+import org.joda.time.LocalDate;

 /**
 * Datastore entity for tracking RDE revisions.
@@ -35,19 +48,41 @@ import org.joda.time.DateTime;
 * flag is included in the generated XML.
 */
@Entity
-public final class RdeRevision extends ImmutableObject {
+@javax.persistence.Entity
+@IdClass(RdeRevisionId.class)
+public final class RdeRevision extends BackupGroupRoot implements NonReplicatedEntity {

  /** String triplet of tld, date, and mode, e.g. {@code soy_2015-09-01_full}. */
-  @Id
-  String id;
+  @Id @Transient String id;
+
+  @javax.persistence.Id @Ignore String tld;
+
+  @javax.persistence.Id @Ignore LocalDate date;
+
+  @javax.persistence.Id @Ignore RdeMode mode;

  /**
   * Number of last revision successfully staged to GCS.
   *
   * <p>This values begins at zero upon object creation and thenceforth incremented transactionally.
   */
+  @Column(nullable = false)
  int revision;

+  /** Hibernate requires an empty constructor. */
+  private RdeRevision() {}
+
+  public static RdeRevision create(
+      String id, String tld, LocalDate date, RdeMode mode, int revision) {
+    RdeRevision instance = new RdeRevision();
+    instance.id = id;
+    instance.tld = tld;
+    instance.date = date;
+    instance.mode = mode;
+    instance.revision = revision;
+    return instance;
+  }
+
  public int getRevision() {
    return revision;
  }
@@ -58,9 +93,12 @@ public final class RdeRevision extends ImmutableObject {
   * @return {@code 0} for first deposit generation and {@code >0} for resends
   */
  public static int getNextRevision(String tld, DateTime date, RdeMode mode) {
-    RdeRevision object =
-        ofy().load().type(RdeRevision.class).id(makePartialName(tld, date, mode)).now();
-    return object == null ? 0 : object.revision + 1;
+    String id = makePartialName(tld, date, mode);
+    RdeRevisionId sqlKey = RdeRevisionId.create(tld, date.toLocalDate(), mode);
+    Key<RdeRevision> ofyKey = Key.create(RdeRevision.class, id);
+    Optional<RdeRevision> revisionOptional =
+        tm().maybeLoad(VKey.create(RdeRevision.class, sqlKey, ofyKey));
+    return revisionOptional.map(rdeRevision -> rdeRevision.revision + 1).orElse(0);
  }

  /**
@@ -76,17 +114,56 @@ public final class RdeRevision extends ImmutableObject {
    checkArgument(revision >= 0, "Negative revision: %s", revision);
    String triplet = makePartialName(tld, date, mode);
    tm().assertInTransaction();
-    RdeRevision object = ofy().load().type(RdeRevision.class).id(triplet).now();
+    RdeRevisionId sqlKey = RdeRevisionId.create(tld, date.toLocalDate(), mode);
+    Key<RdeRevision> ofyKey = Key.create(RdeRevision.class, triplet);
+    Optional<RdeRevision> revisionOptional =
+        tm().maybeLoad(VKey.create(RdeRevision.class, sqlKey, ofyKey));
    if (revision == 0) {
-      verify(object == null, "RdeRevision object already created: %s", object);
+      revisionOptional.ifPresent(
+          rdeRevision -> {
+            throw new IllegalArgumentException(
+                String.format(
+                    "RdeRevision object already created and revision 0 specified: %s",
+                    rdeRevision));
+          });
    } else {
-      verifyNotNull(object, "RDE revision object missing for %s?! revision=%s", triplet, revision);
-      verify(object.revision == revision - 1,
-          "RDE revision object should be at %s but was: %s", revision - 1, object);
+      checkArgument(
+          revisionOptional.isPresent(),
+          "Couldn't find existing RDE revision %s when trying to save new revision %s",
+          triplet,
+          revision);
+      checkArgument(
+          revisionOptional.get().revision == revision - 1,
+          "RDE revision object should be at revision %s but was: %s",
+          revision - 1,
+          revisionOptional.get());
+    }
+    RdeRevision object = RdeRevision.create(triplet, tld, date.toLocalDate(), mode, revision);
+    tm().put(object);
+  }
+
+  /** Class to represent the composite primary key of {@link RdeRevision} entity. */
+  static class RdeRevisionId extends ImmutableObject implements Serializable {
+
+    String tld;
+
+    // Auto-conversion doesn't work for ID classes, we must specify @Column and @Convert
+    @Column(columnDefinition = "date")
+    @Convert(converter = LocalDateConverter.class)
+    LocalDate date;
+
+    @Enumerated(EnumType.STRING)
+    RdeMode mode;
+
+    /** Hibernate requires this default constructor. */
+    private RdeRevisionId() {}
+
+    static RdeRevisionId create(String tld, LocalDate date, RdeMode mode) {
+      RdeRevisionId instance = new RdeRevisionId();
+      instance.tld = tld;
+      instance.date = date;
+      instance.mode = mode;
+      return instance;
    }
-    object = new RdeRevision();
-    object.id = triplet;
-    object.revision = revision;
-    ofy().save().entity(object);
  }
 }
@@ -32,6 +32,7 @@ import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
 import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.model.registry.Registries.assertTldsExist;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
+import static google.registry.persistence.transaction.TransactionManagerUtil.transactIfJpaTm;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableSortedCopy;
 import static google.registry.util.PasswordUtils.SALT_SUPPLIER;
@@ -704,10 +705,18 @@ public class Registrar extends ImmutableObject
    return new Builder(clone(this));
  }

+  /** Creates a {@link VKey} for this instance. */
  public VKey<Registrar> createVKey() {
    return VKey.create(Registrar.class, clientIdentifier, Key.create(this));
  }

+  /** Creates a {@link VKey} for the given {@code registrarId}. */
+  public static VKey<Registrar> createVKey(String registrarId) {
+    checkArgumentNotNull(registrarId, "registrarId must be specified");
+    return VKey.create(
+        Registrar.class, registrarId, Key.create(getCrossTldKey(), Registrar.class, registrarId));
+  }
+
  /** A builder for constructing {@link Registrar}, since it is immutable. */
  public static class Builder extends Buildable.Builder<Registrar> {
    public Builder() {}
@@ -991,8 +1000,7 @@ public class Registrar extends ImmutableObject
  /** Loads and returns a registrar entity by its client id directly from Datastore. */
  public static Optional<Registrar> loadByClientId(String clientId) {
    checkArgument(!Strings.isNullOrEmpty(clientId), "clientId must be specified");
-    return Optional.ofNullable(
-        ofy().load().type(Registrar.class).parent(getCrossTldKey()).id(clientId).now());
+    return transactIfJpaTm(() -> tm().maybeLoad(createVKey(clientId)));
  }

  /**
@@ -59,15 +59,19 @@ public final class Registries {
            tm().doTransactionless(
                    () -> {
                      ImmutableSet<String> tlds =
-                          ofy()
-                              .load()
-                              .type(Registry.class)
-                              .ancestor(getCrossTldKey())
-                              .keys()
-                              .list()
-                              .stream()
-                              .map(Key::getName)
-                              .collect(toImmutableSet());
+                          tm().isOfy()
+                              ? ofy()
+                                  .load()
+                                  .type(Registry.class)
+                                  .ancestor(getCrossTldKey())
+                                  .keys()
+                                  .list()
+                                  .stream()
+                                  .map(Key::getName)
+                                  .collect(toImmutableSet())
+                              : tm().loadAll(Registry.class).stream()
+                                  .map(Registry::getTldStr)
+                                  .collect(toImmutableSet());
                      return Registry.getAll(tlds).stream()
                          .map(e -> Maps.immutableEntry(e.getTldStr(), e.getTldType()))
                          .collect(entriesToImmutableMap());
@@ -22,13 +22,11 @@ import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static com.google.common.collect.Maps.toMap;
 import static google.registry.config.RegistryConfig.getSingletonCacheRefreshDuration;
 import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
-import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
 import static google.registry.util.DateTimeUtils.END_OF_TIME;
 import static google.registry.util.DateTimeUtils.START_OF_TIME;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;
 import static org.joda.money.CurrencyUnit.USD;

 import com.google.common.annotations.VisibleForTesting;
@@ -62,6 +60,8 @@ import google.registry.model.domain.fee.BaseFee.FeeType;
 import google.registry.model.domain.fee.Fee;
 import google.registry.model.registry.label.PremiumList;
 import google.registry.model.registry.label.ReservedList;
+import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import google.registry.util.Idn;
 import java.util.Map;
 import java.util.Optional;
@@ -86,7 +86,7 @@ import org.joda.time.Duration;
@ReportedOn
@Entity
@javax.persistence.Entity(name = "Tld")
-public class Registry extends ImmutableObject implements Buildable {
+public class Registry extends ImmutableObject implements Buildable, DatastoreAndSqlEntity {

  @Parent @Transient Key<EntityGroupRoot> parent = getCrossTldKey();

@@ -259,35 +259,36 @@ public class Registry extends ImmutableObject implements Buildable {
  /** A cache that loads the {@link Registry} for a given tld. */
  private static final LoadingCache<String, Optional<Registry>> CACHE =
      CacheBuilder.newBuilder()
-          .expireAfterWrite(getSingletonCacheRefreshDuration().getMillis(), MILLISECONDS)
+          .expireAfterWrite(
+              java.time.Duration.ofMillis(getSingletonCacheRefreshDuration().getMillis()))
          .build(
              new CacheLoader<String, Optional<Registry>>() {
                @Override
                public Optional<Registry> load(final String tld) {
                  // Enter a transaction-less context briefly; we don't want to enroll every TLD in
                  // a transaction that might be wrapping this call.
-                  return Optional.ofNullable(
-                      tm().doTransactionless(
-                              () ->
-                                  ofy()
-                                      .load()
-                                      .key(Key.create(getCrossTldKey(), Registry.class, tld))
-                                      .now()));
+                  return tm().doTransactionless(() -> tm().maybeLoad(createVKey(tld)));
                }

                @Override
                public Map<String, Optional<Registry>> loadAll(Iterable<? extends String> tlds) {
-                  ImmutableMap<String, Key<Registry>> keysMap =
-                      toMap(
-                          ImmutableSet.copyOf(tlds),
-                          tld -> Key.create(getCrossTldKey(), Registry.class, tld));
-                  Map<Key<Registry>, Registry> entities =
-                      tm().doTransactionless(() -> ofy().load().keys(keysMap.values()));
+                  ImmutableMap<String, VKey<Registry>> keysMap =
+                      toMap(ImmutableSet.copyOf(tlds), Registry::createVKey);
+                  Map<VKey<? extends Registry>, Registry> entities =
+                      tm().doTransactionless(() -> tm().load(keysMap.values()));
                  return Maps.transformEntries(
                      keysMap, (k, v) -> Optional.ofNullable(entities.getOrDefault(v, null)));
                }
              });

+  public static VKey<Registry> createVKey(String tld) {
+    return VKey.create(Registry.class, tld, Key.create(getCrossTldKey(), Registry.class, tld));
+  }
+
+  public static VKey<Registry> createVKey(Key<Registry> key) {
+    return createVKey(key.getName());
+  }
+
  /**
   * The name of the pricing engine that this TLD uses.
   *
@@ -385,7 +386,7 @@ public class Registry extends ImmutableObject implements Buildable {
  CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);

  /** The set of reserved lists that are applicable to this registry. */
-  @Column(name = "reserved_list_names", nullable = false)
+  @Column(name = "reserved_list_names")
  Set<Key<ReservedList>> reservedLists;

  /** Retrieves an ImmutableSet of all ReservedLists associated with this tld. */
@@ -25,7 +25,6 @@ import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
 import static google.registry.model.ofy.ObjectifyService.allocateId;
 import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;

 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Splitter;
@@ -46,8 +45,8 @@ import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ReportedOn;
 import google.registry.model.registry.Registry;
-import google.registry.schema.replay.DatastoreAndSqlEntity;
 import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.NonReplicatedEntity;
 import google.registry.schema.replay.SqlEntity;
 import google.registry.schema.tld.PremiumListDao;
 import google.registry.util.NonFinalForTesting;
@@ -56,6 +55,7 @@ import java.io.IOException;
 import java.math.BigDecimal;
 import java.util.List;
 import java.util.Map;
+import java.util.NoSuchElementException;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
@@ -89,7 +89,7 @@ import org.joda.time.Duration;
@javax.persistence.Entity
@Table(indexes = {@Index(columnList = "name", name = "premiumlist_name_idx")})
 public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.PremiumListEntry>
-    implements DatastoreAndSqlEntity {
+    implements NonReplicatedEntity {

  /** Stores the revision key for the set of currently used premium list entry entities. */
  @Transient Key<PremiumListRevision> revisionKey;
@@ -197,7 +197,7 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
  @VisibleForTesting
  static LoadingCache<String, PremiumList> createCachePremiumLists(Duration cachePersistDuration) {
    return CacheBuilder.newBuilder()
-        .expireAfterWrite(cachePersistDuration.getMillis(), MILLISECONDS)
+        .expireAfterWrite(java.time.Duration.ofMillis(cachePersistDuration.getMillis()))
        .build(
            new CacheLoader<String, PremiumList>() {
              @Override
@@ -208,7 +208,9 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
  }

  private static PremiumList loadPremiumList(String name) {
-    return ofy().load().type(PremiumList.class).parent(getCrossTldKey()).id(name).now();
+    return tm().isOfy()
+        ? ofy().load().type(PremiumList.class).parent(getCrossTldKey()).id(name).now()
+        : PremiumListDao.getLatestRevision(name).orElseThrow(NoSuchElementException::new);
  }

  /**
@@ -221,7 +223,8 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
  static final LoadingCache<Key<PremiumListRevision>, PremiumListRevision>
      cachePremiumListRevisions =
          CacheBuilder.newBuilder()
-              .expireAfterWrite(getSingletonCachePersistDuration().getMillis(), MILLISECONDS)
+              .expireAfterWrite(
+                  java.time.Duration.ofMillis(getSingletonCachePersistDuration().getMillis()))
              .build(
                  new CacheLoader<Key<PremiumListRevision>, PremiumListRevision>() {
                    @Override
@@ -260,14 +263,14 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
  static LoadingCache<Key<PremiumListEntry>, Optional<PremiumListEntry>>
      createCachePremiumListEntries(Duration cachePersistDuration) {
    return CacheBuilder.newBuilder()
-        .expireAfterWrite(cachePersistDuration.getMillis(), MILLISECONDS)
+        .expireAfterWrite(java.time.Duration.ofMillis(cachePersistDuration.getMillis()))
        .maximumSize(getStaticPremiumListMaxCachedEntries())
        .build(
            new CacheLoader<Key<PremiumListEntry>, Optional<PremiumListEntry>>() {
              @Override
              public Optional<PremiumListEntry> load(final Key<PremiumListEntry> entryKey) {
-                return tm()
-                    .doTransactionless(() -> Optional.ofNullable(ofy().load().key(entryKey).now()));
+                return tm().doTransactionless(
+                        () -> Optional.ofNullable(ofy().load().key(entryKey).now()));
              }
            });
  }
@@ -348,7 +351,7 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr

    @Override
    public ImmutableList<SqlEntity> toSqlEntities() {
-      return null;
+      return ImmutableList.of();
    }

    /** A builder for constructing {@link PremiumListEntry} objects, since they are immutable. */
@@ -19,8 +19,8 @@ import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.config.RegistryConfig.getDomainLabelListCacheDuration;
 import static google.registry.model.registry.label.ReservationType.FULLY_BLOCKED;
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.util.CollectionUtils.nullToEmpty;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;
 import static org.joda.time.DateTimeZone.UTC;

 import com.google.common.base.Splitter;
@@ -38,7 +38,7 @@ import com.googlecode.objectify.mapper.Mapper;
 import google.registry.model.Buildable;
 import google.registry.model.registry.Registry;
 import google.registry.model.registry.label.DomainLabelMetrics.MetricsReservedListMatch;
-import google.registry.schema.replay.DatastoreAndSqlEntity;
+import google.registry.schema.replay.NonReplicatedEntity;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
@@ -67,7 +67,7 @@ import org.joda.time.DateTime;
@Table(indexes = {@Index(columnList = "name", name = "reservedlist_name_idx")})
 public final class ReservedList
    extends BaseDomainLabelList<ReservationType, ReservedList.ReservedListEntry>
-    implements DatastoreAndSqlEntity {
+    implements NonReplicatedEntity {

  @Mapify(ReservedListEntry.LabelMapper.class)
  @ElementCollection
@@ -241,12 +241,15 @@ public final class ReservedList

  private static LoadingCache<String, ReservedList> cache =
      CacheBuilder.newBuilder()
-          .expireAfterWrite(getDomainLabelListCacheDuration().getMillis(), MILLISECONDS)
+          .expireAfterWrite(
+              java.time.Duration.ofMillis(getDomainLabelListCacheDuration().getMillis()))
          .build(
              new CacheLoader<String, ReservedList>() {
                @Override
                public ReservedList load(String listName) {
-                  return ReservedListDualWriteDao.getLatestRevision(listName).orElse(null);
+                  return tm().isOfy()
+                      ? ReservedListDualWriteDao.getLatestRevision(listName).orElse(null)
+                      : ReservedListSqlDao.getLatestRevision(listName).orElse(null);
                }
              });

@@ -22,6 +22,7 @@ import com.googlecode.objectify.annotation.Embed;
 import com.googlecode.objectify.annotation.Ignore;
 import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import javax.persistence.Column;
 import javax.persistence.Entity;
 import javax.persistence.EnumType;
@@ -43,7 +44,8 @@ import org.joda.time.DateTime;
 */
@Embed
@Entity
-public class DomainTransactionRecord extends ImmutableObject implements Buildable {
+public class DomainTransactionRecord extends ImmutableObject
+    implements Buildable, DatastoreAndSqlEntity {

  @Id
  @Ignore
@@ -62,7 +62,7 @@ import org.joda.time.DateTime;
@MappedSuperclass
@WithStringVKey // TODO(b/162229294): This should be resolved during the course of that bug
@Access(AccessType.FIELD)
-public class HistoryEntry extends ImmutableObject implements Buildable, DatastoreEntity, SqlEntity {
+public class HistoryEntry extends ImmutableObject implements Buildable, DatastoreEntity {

  /** Represents the type of history entry. */
  public enum Type {
@@ -193,7 +193,7 @@ public class HistoryEntry extends ImmutableObject implements Buildable, Datastor
   * transaction counts (such as contact or host mutations).
   */
  @Transient // domain-specific
-  Set<DomainTransactionRecord> domainTransactionRecords;
+  protected Set<DomainTransactionRecord> domainTransactionRecords;

  public long getId() {
    // For some reason, Hibernate throws NPE during some initialization phase if we don't deal with
@@ -273,7 +273,8 @@ public class HistoryEntry extends ImmutableObject implements Buildable, Datastor
  /** This method exists solely to satisfy Hibernate. Use the {@link Builder} instead. */
  @SuppressWarnings("UnusedMethod")
  private void setDomainTransactionRecords(Set<DomainTransactionRecord> domainTransactionRecords) {
-    this.domainTransactionRecords = ImmutableSet.copyOf(domainTransactionRecords);
+    this.domainTransactionRecords =
+        domainTransactionRecords == null ? null : ImmutableSet.copyOf(domainTransactionRecords);
  }

  public static VKey<HistoryEntry> createVKey(Key<HistoryEntry> key) {
@@ -308,19 +309,10 @@ public class HistoryEntry extends ImmutableObject implements Buildable, Datastor
          new DomainHistory.Builder().copyFrom(this).setDomainRepoId(parent.getName()).build();
    } else if (parentKind.equals(getKind(HostResource.class))) {
      resultEntity =
-          new HostHistory.Builder()
-              .copyFrom(this)
-              .setHostRepoId(
-                  VKey.create(HostResource.class, parent.getName(), (Key<HostResource>) parent))
-              .build();
+          new HostHistory.Builder().copyFrom(this).setHostRepoId(parent.getName()).build();
    } else if (parentKind.equals(getKind(ContactResource.class))) {
      resultEntity =
-          new ContactHistory.Builder()
-              .copyFrom(this)
-              .setContactRepoId(
-                  VKey.create(
-                      ContactResource.class, parent.getName(), (Key<ContactResource>) parent))
-              .build();
+          new ContactHistory.Builder().copyFrom(this).setContactRepoId(parent.getName()).build();
    } else {
      throw new IllegalStateException(
          String.format("Unknown kind of HistoryEntry parent %s", parentKind));
@@ -331,13 +323,7 @@ public class HistoryEntry extends ImmutableObject implements Buildable, Datastor
  // In SQL, save the child type
  @Override
  public ImmutableList<SqlEntity> toSqlEntities() {
-    return ImmutableList.of(toChildHistoryEntity());
-  }
-
-  // In Datastore, save as a HistoryEntry object regardless of this object's type
-  @Override
-  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
-    return ImmutableList.of(asHistoryEntry());
+    return ImmutableList.of((SqlEntity) toChildHistoryEntity());
  }

  /** A builder for {@link HistoryEntry} since it is immutable */
@@ -75,7 +75,7 @@ public class Spec11ThreatMatch extends ImmutableObject implements Buildable, Sql
  String registrarId;

  /** Date on which the check was run, on which the domain was flagged as abusive. */
-  @Column(nullable = false)
+  @Column(nullable = false, columnDefinition = "date")
  LocalDate checkDate;

  /** The domain's top-level domain. */
@@ -16,23 +16,25 @@ package google.registry.model.reporting;

 import com.google.common.collect.ImmutableList;
 import google.registry.persistence.transaction.JpaTransactionManager;
+import google.registry.util.DateTimeUtils;
+import javax.persistence.TemporalType;
 import org.joda.time.LocalDate;

 /**
 * Data access object for {@link google.registry.model.reporting.Spec11ThreatMatch}.
 *
 * <p>A JpaTransactionManager is passed into each static method because they are called from a BEAM
- * pipeline and we don't know where it's coming from.</p>
+ * pipeline and we don't know where it's coming from.
 */
 public class Spec11ThreatMatchDao {
-  
+
  /** Delete all entries with the specified date from the database. */
  public static void deleteEntriesByDate(JpaTransactionManager jpaTm, LocalDate date) {
    jpaTm.assertInTransaction();
    jpaTm
        .getEntityManager()
        .createQuery("DELETE FROM Spec11ThreatMatch WHERE check_date = :date")
-        .setParameter("date", date.toString())
+        .setParameter("date", DateTimeUtils.toSqlDate(date), TemporalType.DATE)
        .executeUpdate();
  }

@@ -16,6 +16,7 @@ package google.registry.model.server;

 import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;

+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Id;
@@ -23,11 +24,13 @@ import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ReportedOn;
 import google.registry.model.common.EntityGroupRoot;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;

 /** Pointer to the latest {@link KmsSecretRevision}. */
@Entity
@ReportedOn
-public class KmsSecret extends ImmutableObject {
+public class KmsSecret extends ImmutableObject implements DatastoreEntity {

  /** The unique name of this {@link KmsSecret}. */
  @Id String name;
@@ -45,6 +48,11 @@ public class KmsSecret extends ImmutableObject {
    return latestRevision;
  }

+  @Override
+  public ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of(); // not persisted in SQL
+  }
+
  public static KmsSecret create(String name, KmsSecretRevision latestRevision) {
    KmsSecret instance = new KmsSecret();
    instance.name = name;
@@ -20,11 +20,19 @@ import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Id;
+import com.googlecode.objectify.annotation.Ignore;
+import com.googlecode.objectify.annotation.OnLoad;
 import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.Buildable;
 import google.registry.model.CreateAutoTimestamp;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.schema.replay.NonReplicatedEntity;
+import javax.persistence.Column;
+import javax.persistence.Index;
+import javax.persistence.PostLoad;
+import javax.persistence.Table;
+import javax.persistence.Transient;

 /**
 * An encrypted value.
@@ -35,13 +43,22 @@ import google.registry.model.annotations.ReportedOn;
 *
 * <p>The value can be encrypted and decrypted using Cloud KMS.
 *
+ * <p>Note that the primary key of this entity is {@link #revisionKey}, which is auto-generated by
+ * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
+ * succeeds, we will end up with having two exact same revisions that differ only by revisionKey.
+ * This is fine though, because we only use the revision with the highest revisionKey.
+ *
+ * <p>TODO: remove Datastore-specific fields post-Registry-3.0-migration and rename to KmsSecret.
+ *
 * @see <a href="https://cloud.google.com/kms/docs/">Google Cloud Key Management Service
 *     Documentation</a>
 * @see google.registry.keyring.kms.KmsKeyring
 */
@Entity
@ReportedOn
-public class KmsSecretRevision extends ImmutableObject {
+@javax.persistence.Entity(name = "KmsSecret")
+@Table(indexes = {@Index(columnList = "secretName")})
+public class KmsSecretRevision extends ImmutableObject implements NonReplicatedEntity {

  /**
   * The maximum allowable secret size. Although Datastore allows entities up to 1 MB in size,
@@ -49,18 +66,31 @@ public class KmsSecretRevision extends ImmutableObject {
   */
  private static final int MAX_SECRET_SIZE_BYTES = 64 * 1024 * 1024;

-  /** The revision of this secret. */
-  @Id long revisionKey;
+  /**
+   * The revision of this secret.
+   *
+   * <p>TODO: change name of the variable to revisionId once we're off Datastore
+   */
+  @Id
+  @javax.persistence.Id
+  @Column(name = "revisionId")
+  long revisionKey;

  /** The parent {@link KmsSecret} which contains metadata about this {@link KmsSecretRevision}. */
-  @Parent Key<KmsSecret> parent;
+  @Parent @Transient Key<KmsSecret> parent;
+  @Column(nullable = false)
+  @Ignore
+  String secretName;

  /**
   * The name of the {@code cryptoKeyVersion} associated with this {@link KmsSecretRevision}.
   *
+   * <p>TODO: change name of the variable to cryptoKeyVersionName once we're off Datastore
+   *
   * @see <a
   *     href="https://cloud.google.com/kms/docs/reference/rest/v1/projects.locations.keyRings.cryptoKeys.cryptoKeyVersions">projects.locations.keyRings.cryptoKeys.cryptoKeyVersions</a>
   */
+  @Column(nullable = false, name = "cryptoKeyVersionName")
  String kmsCryptoKeyVersionName;

  /**
@@ -70,9 +100,11 @@ public class KmsSecretRevision extends ImmutableObject {
   * @see <a
   *     href="https://cloud.google.com/kms/docs/reference/rest/v1/projects.locations.keyRings.cryptoKeys/encrypt">projects.locations.keyRings.cryptoKeys.encrypt</a>
   */
+  @Column(nullable = false)
  String encryptedValue;

  /** An automatically managed creation timestamp. */
+  @Column(nullable = false)
  CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);

  public String getKmsCryptoKeyVersionName() {
@@ -83,6 +115,18 @@ public class KmsSecretRevision extends ImmutableObject {
    return encryptedValue;
  }

+  // When loading from SQL, fill out the Datastore-specific field
+  @PostLoad
+  void postLoad() {
+    parent = Key.create(getCrossTldKey(), KmsSecret.class, secretName);
+  }
+
+  // When loading from Datastore, fill out the SQL-specific field
+  @OnLoad
+  void onLoad() {
+    secretName = parent.getName();
+  }
+
  /** A builder for constructing {@link KmsSecretRevision} entities, since they are immutable. */
  public static class Builder extends Buildable.Builder<KmsSecretRevision> {

@@ -108,6 +152,7 @@ public class KmsSecretRevision extends ImmutableObject {
     */
    public Builder setParent(String secretName) {
      getInstance().parent = Key.create(getCrossTldKey(), KmsSecret.class, secretName);
+      getInstance().secretName = secretName;
      return this;
    }
  }
@@ -0,0 +1,54 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.server;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Strings.isNullOrEmpty;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
+
+import java.util.Optional;
+
+/**
+ * A {@link KmsSecretRevision} DAO for Cloud SQL.
+ *
+ * <p>TODO: Rename this class to KmsSecretDao after migrating to Cloud SQL.
+ */
+public class KmsSecretRevisionSqlDao {
+
+  private KmsSecretRevisionSqlDao() {}
+
+  /** Saves the given KMS secret revision. */
+  public static void save(KmsSecretRevision kmsSecretRevision) {
+    checkArgumentNotNull(kmsSecretRevision, "kmsSecretRevision cannot be null");
+    jpaTm().assertInTransaction();
+    jpaTm().put(kmsSecretRevision);
+  }
+
+  /** Returns the latest revision for the secret name given, or absent if nonexistent. */
+  public static Optional<KmsSecretRevision> getLatestRevision(String secretName) {
+    checkArgument(!isNullOrEmpty(secretName), "secretName cannot be null or empty");
+    jpaTm().assertInTransaction();
+    return jpaTm()
+        .getEntityManager()
+        .createQuery(
+            "FROM KmsSecret ks WHERE ks.revisionKey IN (SELECT MAX(revisionKey) FROM "
+                + "KmsSecret subKs WHERE subKs.secretName = :secretName)",
+            KmsSecretRevision.class)
+        .setParameter("secretName", secretName)
+        .getResultStream()
+        .findFirst();
+  }
+}
@@ -14,29 +14,43 @@

 package google.registry.model.server;

-import static google.registry.model.ofy.ObjectifyService.ofy;
+import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.persistence.transaction.TransactionManagerFactory.ofyTm;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
 import com.google.common.cache.LoadingCache;
+import com.google.common.collect.ImmutableList;
 import com.google.common.primitives.Longs;
+import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
+import com.googlecode.objectify.annotation.Ignore;
+import com.googlecode.objectify.annotation.OnLoad;
 import com.googlecode.objectify.annotation.Unindex;
 import google.registry.model.annotations.NotBackedUp;
 import google.registry.model.annotations.NotBackedUp.Reason;
 import google.registry.model.common.CrossTldSingleton;
+import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
 import java.nio.ByteBuffer;
 import java.util.UUID;
 import java.util.concurrent.ExecutionException;
+import javax.persistence.Column;
+import javax.persistence.Id;
+import javax.persistence.PostLoad;
+import javax.persistence.Transient;

 /** A secret number used for generating tokens (such as XSRF tokens). */
@Entity
+@javax.persistence.Entity
@Unindex
@NotBackedUp(reason = Reason.AUTO_GENERATED)
 // TODO(b/27427316): Replace this with an entry in KMSKeyring
-public class ServerSecret extends CrossTldSingleton {
+public class ServerSecret extends CrossTldSingleton implements DatastoreEntity, SqlEntity {

  /**
   * Cache of the singleton ServerSecret instance that creates it if not present.
@@ -45,28 +59,34 @@ public class ServerSecret extends CrossTldSingleton {
   * Supplier that can be reset for testing purposes.
   */
  private static final LoadingCache<Class<ServerSecret>, ServerSecret> CACHE =
-      CacheBuilder.newBuilder().build(
-          new CacheLoader<Class<ServerSecret>, ServerSecret>() {
-            @Override
-            public ServerSecret load(Class<ServerSecret> unused) {
-              // Fast path - non-transactional load to hit memcache.
-              ServerSecret secret = ofy().load().entity(new ServerSecret()).now();
-              if (secret != null) {
-                return secret;
-              }
-              // Slow path - transactionally create a new ServerSecret (once per app setup).
-              return tm().transact(() -> {
-                // Check again for an existing secret within the transaction to avoid races.
-                ServerSecret secret1 = ofy().load().entity(new ServerSecret()).now();
-                if (secret1 == null) {
-                  UUID uuid = UUID.randomUUID();
-                  secret1 = create(uuid.getMostSignificantBits(), uuid.getLeastSignificantBits());
-                  ofy().saveWithoutBackup().entity(secret1).now();
+      CacheBuilder.newBuilder()
+          .build(
+              new CacheLoader<Class<ServerSecret>, ServerSecret>() {
+                @Override
+                public ServerSecret load(Class<ServerSecret> unused) {
+                  return retrieveAndSaveSecret();
                }
-                return secret1;
              });
-            }
-          });
+
+  private static ServerSecret retrieveAndSaveSecret() {
+    VKey<ServerSecret> key =
+        VKey.create(
+            ServerSecret.class,
+            SINGLETON_ID,
+            Key.create(getCrossTldKey(), ServerSecret.class, SINGLETON_ID));
+    return tm().transact(
+            () -> {
+              // transactionally create a new ServerSecret (once per app setup) if necessary.
+              // return the ofy() result during Datastore-primary phase
+              ServerSecret secret =
+                  ofyTm().maybeLoad(key).orElseGet(() -> create(UUID.randomUUID()));
+              // During a dual-write period, write it to both Datastore and SQL
+              // even if we didn't have to retrieve it from the DB
+              ofyTm().transact(() -> ofyTm().putWithoutBackup(secret));
+              jpaTm().transact(() -> jpaTm().putWithoutBackup(secret));
+              return secret;
+            });
+  }

  /** Returns the global ServerSecret instance, creating it if one isn't already in Datastore. */
  public static ServerSecret get() {
@@ -77,23 +97,38 @@ public class ServerSecret extends CrossTldSingleton {
    }
  }

-  /** Most significant 8 bytes of the UUID value. */
-  long mostSignificant;
+  /** Most significant 8 bytes of the UUID value (stored separately for legacy purposes). */
+  @Transient long mostSignificant;

-  /** Least significant 8 bytes of the UUID value. */
-  long leastSignificant;
+  /** Least significant 8 bytes of the UUID value (stored separately for legacy purposes). */
+  @Transient long leastSignificant;

-  @VisibleForTesting
-  static ServerSecret create(long mostSignificant, long leastSignificant) {
-    ServerSecret secret = new ServerSecret();
-    secret.mostSignificant = mostSignificant;
-    secret.leastSignificant = leastSignificant;
-    return secret;
+  /** The UUID value itself. */
+  @Id
+  @Column(columnDefinition = "uuid")
+  @Ignore
+  UUID secret;
+
+  /** Convert the Datastore representation to SQL. */
+  @OnLoad
+  void onLoad() {
+    secret = new UUID(mostSignificant, leastSignificant);
  }

-  /** Returns the value of this ServerSecret as a UUID. */
-  public UUID asUuid() {
-    return new UUID(mostSignificant, leastSignificant);
+  /** Convert the SQL representation to Datastore. */
+  @PostLoad
+  void postLoad() {
+    mostSignificant = secret.getMostSignificantBits();
+    leastSignificant = secret.getLeastSignificantBits();
+  }
+
+  @VisibleForTesting
+  static ServerSecret create(UUID uuid) {
+    ServerSecret secret = new ServerSecret();
+    secret.mostSignificant = uuid.getMostSignificantBits();
+    secret.leastSignificant = uuid.getLeastSignificantBits();
+    secret.secret = uuid;
+    return secret;
  }

  /** Returns the value of this ServerSecret as a byte array. */
@@ -104,6 +139,16 @@ public class ServerSecret extends CrossTldSingleton {
        .array();
  }

+  @Override
+  public ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of(); // dually-written
+  }
+
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(); // dually-written
+  }
+
  @VisibleForTesting
  static void resetCache() {
    CACHE.invalidateAll();
@@ -30,6 +30,9 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Supplier;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Iterables;
+import com.google.common.collect.MapDifference;
+import com.google.common.collect.Maps;
+import com.google.common.flogger.FluentLogger;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EmbedMap;
 import com.googlecode.objectify.annotation.Entity;
@@ -41,8 +44,18 @@ import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.NotBackedUp;
 import google.registry.model.annotations.NotBackedUp.Reason;
 import google.registry.model.common.EntityGroupRoot;
+import google.registry.schema.replay.NonReplicatedEntity;
 import google.registry.util.CollectionUtils;
 import java.util.Map;
+import java.util.Optional;
+import javax.persistence.CollectionTable;
+import javax.persistence.Column;
+import javax.persistence.ElementCollection;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.JoinColumn;
+import javax.persistence.MapKeyColumn;
+import javax.persistence.Transient;
 import org.joda.time.DateTime;

 /**
@@ -56,35 +69,49 @@ import org.joda.time.DateTime;
 * order to avoid exceeding the one megabyte max entity size limit, we'll also be sharding that
 * entity into multiple entities, each entity containing {@value #SHARD_SIZE} rows.
 *
+ * <p>TODO: We can remove the sharding once we have converted entirely to Cloud SQL storage during
+ * the Registry 3.0 migration. Then, the entire table will be stored conceptually as one entity (in
+ * fact in SignedMarkRevocationList and SignedMarkRevocationEntry tables).
+ *
 * @see google.registry.tmch.SmdrlCsvParser
- * @see <a href="http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.2">
- *     TMCH functional specifications - SMD Revocation List</a>
+ * @see <a href="http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.2">TMCH
+ *     functional specifications - SMD Revocation List</a>
 */
@Entity
+@javax.persistence.Entity
@NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
-public class SignedMarkRevocationList extends ImmutableObject {
+public class SignedMarkRevocationList extends ImmutableObject implements NonReplicatedEntity {

-  @VisibleForTesting
-  static final int SHARD_SIZE = 10000;
+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
+  @VisibleForTesting static final int SHARD_SIZE = 10000;

  /** Common ancestor for queries. */
-  @Parent
-  Key<EntityGroupRoot> parent = getCrossTldKey();
+  @Parent @Transient Key<EntityGroupRoot> parent = getCrossTldKey();

  /** ID for the sharded entity. */
-  @Id
-  long id;
+  @Id @Transient long id;
+
+  @Ignore
+  @javax.persistence.Id
+  @GeneratedValue(strategy = GenerationType.IDENTITY)
+  Long revisionId;

  /** Time when this list was last updated, as specified in the first line of the CSV file. */
  DateTime creationTime;

  /** A map from SMD IDs to revocation time. */
  @EmbedMap
+  @ElementCollection
+  @CollectionTable(
+      name = "SignedMarkRevocationEntry",
+      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
+  @MapKeyColumn(name = "smdId")
+  @Column(name = "revocationTime", nullable = false)
  Map</*@MatchesPattern("[0-9]+-[0-9]+")*/ String, DateTime> revokes;

  /** Indicates that this is a shard rather than a "full" list. */
-  @Ignore
-  boolean isShard;
+  @Ignore @Transient boolean isShard;

  /**
   * A cached supplier that fetches the SMDRL shards from Datastore and recombines them into a
@@ -92,32 +119,16 @@ public class SignedMarkRevocationList extends ImmutableObject {
   */
  private static final Supplier<SignedMarkRevocationList> CACHE =
      memoizeWithShortExpiration(
-          () ->
-              tm()
-                  .transactNewReadOnly(
-                      () -> {
-                        Iterable<SignedMarkRevocationList> shards =
-                            ofy()
-                                .load()
-                                .type(SignedMarkRevocationList.class)
-                                .ancestor(getCrossTldKey());
-                        DateTime creationTime =
-                            isEmpty(shards)
-                                ? START_OF_TIME
-                                : checkNotNull(
-                                    Iterables.get(shards, 0).creationTime, "creationTime");
-                        ImmutableMap.Builder<String, DateTime> revokes =
-                            new ImmutableMap.Builder<>();
-                        for (SignedMarkRevocationList shard : shards) {
-                          revokes.putAll(shard.revokes);
-                          checkState(
-                              creationTime.equals(shard.creationTime),
-                              "Inconsistent creation times: %s vs. %s",
-                              creationTime,
-                              shard.creationTime);
-                        }
-                        return create(creationTime, revokes.build());
-                      }));
+          () -> {
+            SignedMarkRevocationList datastoreList = loadFromDatastore();
+            // Also load the list from Cloud SQL, compare the two lists, and log if different.
+            try {
+              loadAndCompareCloudSqlList(datastoreList);
+            } catch (Throwable t) {
+              logger.atSevere().withCause(t).log("Error comparing signed mark revocation lists.");
+            }
+            return datastoreList;
+          });

  /** Return a single logical instance that combines all Datastore shards. */
  public static SignedMarkRevocationList get() {
@@ -149,10 +160,39 @@ public class SignedMarkRevocationList extends ImmutableObject {
    return revokes.size();
  }

-  /** Save this list to Datastore in sharded form. Returns {@code this}. */
+  /** Save this list to Datastore in sharded form and to Cloud SQL. Returns {@code this}. */
  public SignedMarkRevocationList save() {
-    tm()
-        .transact(
+    saveToDatastore();
+    SignedMarkRevocationListDao.trySave(this);
+    return this;
+  }
+
+  /** Loads the shards from Datastore and combines them into one list. */
+  private static SignedMarkRevocationList loadFromDatastore() {
+    return tm().transactNewReadOnly(
+            () -> {
+              Iterable<SignedMarkRevocationList> shards =
+                  ofy().load().type(SignedMarkRevocationList.class).ancestor(getCrossTldKey());
+              DateTime creationTime =
+                  isEmpty(shards)
+                      ? START_OF_TIME
+                      : checkNotNull(Iterables.get(shards, 0).creationTime, "creationTime");
+              ImmutableMap.Builder<String, DateTime> revokes = new ImmutableMap.Builder<>();
+              for (SignedMarkRevocationList shard : shards) {
+                revokes.putAll(shard.revokes);
+                checkState(
+                    creationTime.equals(shard.creationTime),
+                    "Inconsistent creation times: %s vs. %s",
+                    creationTime,
+                    shard.creationTime);
+              }
+              return create(creationTime, revokes.build());
+            });
+  }
+
+  /** Save this list to Datastore in sharded form. */
+  private SignedMarkRevocationList saveToDatastore() {
+    tm().transact(
            () -> {
              ofy()
                  .deleteWithoutBackup()
@@ -165,8 +205,7 @@ public class SignedMarkRevocationList extends ImmutableObject {
              ofy()
                  .saveWithoutBackup()
                  .entities(
-                      CollectionUtils.partitionMap(revokes, SHARD_SIZE)
-                          .stream()
+                      CollectionUtils.partitionMap(revokes, SHARD_SIZE).stream()
                          .map(
                              shardRevokes -> {
                                SignedMarkRevocationList shard = create(creationTime, shardRevokes);
@@ -180,6 +219,38 @@ public class SignedMarkRevocationList extends ImmutableObject {
    return this;
  }

+  private static void loadAndCompareCloudSqlList(SignedMarkRevocationList datastoreList) {
+    // Lifted with some modifications from ClaimsListShard
+    Optional<SignedMarkRevocationList> maybeCloudSqlList =
+        SignedMarkRevocationListDao.getLatestRevision();
+    if (maybeCloudSqlList.isPresent()) {
+      SignedMarkRevocationList cloudSqlList = maybeCloudSqlList.get();
+      MapDifference<String, DateTime> diff =
+          Maps.difference(datastoreList.revokes, cloudSqlList.revokes);
+      if (!diff.areEqual()) {
+        if (diff.entriesDiffering().size() > 10) {
+          logger.atWarning().log(
+              String.format(
+                  "Unequal SM revocation lists detected, Cloud SQL list with revision id %d has %d"
+                      + " different records than the current Datastore list.",
+                  cloudSqlList.revisionId, diff.entriesDiffering().size()));
+        } else {
+          StringBuilder diffMessage = new StringBuilder("Unequal SM revocation lists detected:\n");
+          diff.entriesDiffering()
+              .forEach(
+                  (label, valueDiff) ->
+                      diffMessage.append(
+                          String.format(
+                              "SMD %s has key %s in Datastore and key %s in Cloud SQL.\n",
+                              label, valueDiff.leftValue(), valueDiff.rightValue())));
+          logger.atWarning().log(diffMessage.toString());
+        }
+      }
+    } else {
+      logger.atWarning().log("Signed mark revocation list in Cloud SQL is empty.");
+    }
+  }
+
  /** As a safety mechanism, fail if someone tries to save this class directly. */
  @OnSave
  void disallowUnshardedSaves() {
@@ -0,0 +1,76 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.model.smd;
+
+import static google.registry.model.CacheUtils.memoizeWithShortExpiration;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+
+import com.google.common.base.Supplier;
+import com.google.common.flogger.FluentLogger;
+import java.util.Optional;
+import javax.persistence.EntityManager;
+
+public class SignedMarkRevocationListDao {
+
+  private static final FluentLogger logger = FluentLogger.forEnclosingClass();
+
+  private static final Supplier<Optional<SignedMarkRevocationList>> CACHE =
+      memoizeWithShortExpiration(SignedMarkRevocationListDao::getLatestRevision);
+
+  /** Returns the most recent revision of the {@link SignedMarkRevocationList}, from cache. */
+  public static Optional<SignedMarkRevocationList> getLatestRevisionCached() {
+    return CACHE.get();
+  }
+
+  public static Optional<SignedMarkRevocationList> getLatestRevision() {
+    return jpaTm()
+        .transact(
+            () -> {
+              EntityManager em = jpaTm().getEntityManager();
+              Long revisionId =
+                  em.createQuery("SELECT MAX(revisionId) FROM SignedMarkRevocationList", Long.class)
+                      .getSingleResult();
+              return em.createQuery(
+                      "FROM SignedMarkRevocationList smrl LEFT JOIN FETCH smrl.revokes "
+                          + "WHERE smrl.revisionId = :revisionId",
+                      SignedMarkRevocationList.class)
+                  .setParameter("revisionId", revisionId)
+                  .getResultStream()
+                  .findFirst();
+            });
+  }
+
+  /**
+   * Try to save the given {@link SignedMarkRevocationList} into Cloud SQL. If the save fails, the
+   * error will be logged but no exception will be thrown.
+   *
+   * <p>This method is used during the dual-write phase of database migration as Datastore is still
+   * the authoritative database.
+   */
+  static void trySave(SignedMarkRevocationList signedMarkRevocationList) {
+    try {
+      SignedMarkRevocationListDao.save(signedMarkRevocationList);
+      logger.atInfo().log(
+          "Inserted %,d signed mark revocations into Cloud SQL",
+          signedMarkRevocationList.revokes.size());
+    } catch (Throwable e) {
+      logger.atSevere().withCause(e).log("Error inserting signed mark revocations into Cloud SQL");
+    }
+  }
+
+  private static void save(SignedMarkRevocationList signedMarkRevocationList) {
+    jpaTm().transact(() -> jpaTm().getEntityManager().persist(signedMarkRevocationList));
+  }
+}
@@ -46,6 +46,7 @@ import google.registry.model.annotations.NotBackedUp;
 import google.registry.model.annotations.NotBackedUp.Reason;
 import google.registry.model.annotations.VirtualEntity;
 import google.registry.model.common.CrossTldSingleton;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import google.registry.schema.replay.DatastoreEntity;
 import google.registry.schema.replay.SqlEntity;
 import google.registry.util.CollectionUtils;
@@ -56,7 +57,6 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
-import java.util.concurrent.Callable;
 import javax.annotation.Nullable;
 import javax.persistence.CollectionTable;
 import javax.persistence.Column;
@@ -97,7 +97,7 @@ import org.joda.time.DateTime;
@NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
@javax.persistence.Entity(name = "ClaimsList")
@Table
-public class ClaimsListShard extends ImmutableObject implements DatastoreEntity {
+public class ClaimsListShard extends ImmutableObject implements DatastoreAndSqlEntity {

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

@@ -145,8 +145,7 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity

  private static final Retrier LOADER_RETRIER = new Retrier(new SystemSleeper(), 2);

-  private static final Callable<ClaimsListShard> LOADER_CALLABLE =
-      () -> {
+  private static ClaimsListShard loadClaimsListShard() {
        // Find the most recent revision.
        Key<ClaimsListRevision> revisionKey = getCurrentRevision();

@@ -245,7 +244,9 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
   */
  private static final Supplier<ClaimsListShard> CACHE =
      memoizeWithShortExpiration(
-          () -> LOADER_RETRIER.callWithRetry(LOADER_CALLABLE, IllegalStateException.class));
+          () ->
+              LOADER_RETRIER.callWithRetry(
+                  ClaimsListShard::loadClaimsListShard, IllegalStateException.class));

  /** Returns the revision id of this claims list, or throws exception if it is null. */
  public Long getRevisionId() {
@@ -354,11 +355,6 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
    }
  }

-  @Override
-  public ImmutableList<SqlEntity> toSqlEntities() {
-    return ImmutableList.of(); // ClaimsLists are dually written
-  }
-
  /** Virtual parent entity for claims list shards of a specific revision. */
  @Entity
  @VirtualEntity
@@ -15,31 +15,50 @@
 package google.registry.model.tmch;

 import static com.google.common.base.Preconditions.checkNotNull;
-import static google.registry.model.ofy.ObjectifyService.ofy;
+import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.persistence.transaction.TransactionManagerFactory.ofyTm;
 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

+import com.google.common.collect.ImmutableList;
+import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
 import google.registry.model.annotations.NotBackedUp;
 import google.registry.model.annotations.NotBackedUp.Reason;
 import google.registry.model.common.CrossTldSingleton;
-import javax.annotation.Nullable;
+import google.registry.model.tmch.TmchCrl.TmchCrlId;
+import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import java.io.Serializable;
+import java.util.Optional;
 import javax.annotation.concurrent.Immutable;
+import javax.persistence.Column;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
 import org.joda.time.DateTime;

 /** Datastore singleton for ICANN's TMCH CA certificate revocation list (CRL). */
@Entity
+@javax.persistence.Entity
@Immutable
@NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
-public final class TmchCrl extends CrossTldSingleton {
+@IdClass(TmchCrlId.class)
+public final class TmchCrl extends CrossTldSingleton implements DatastoreEntity, SqlEntity {

-  String crl;
-  DateTime updated;
-  String url;
+  @Id String crl;
+
+  @Id DateTime updated;
+
+  @Id String url;

  /** Returns the singleton instance of this entity, without memoization. */
-  @Nullable
-  public static TmchCrl get() {
-    return ofy().load().entity(new TmchCrl()).now();
+  public static Optional<TmchCrl> get() {
+    VKey<TmchCrl> key =
+        VKey.create(
+            TmchCrl.class, SINGLETON_ID, Key.create(getCrossTldKey(), TmchCrl.class, SINGLETON_ID));
+    // return the ofy() result during Datastore-primary phase
+    return ofyTm().transact(() -> ofyTm().maybeLoad(key));
  }

  /**
@@ -47,16 +66,27 @@ public final class TmchCrl extends CrossTldSingleton {
   *
   * <p>Please do not call this function unless your CRL is properly formatted, signed by the root,
   * and actually newer than the one currently in Datastore.
+   *
+   * <p>During the dual-write period, we write to both Datastore and SQL
   */
  public static void set(final String crl, final String url) {
-    tm()
-        .transactNew(
+    tm().transact(
            () -> {
              TmchCrl tmchCrl = new TmchCrl();
              tmchCrl.updated = tm().getTransactionTime();
              tmchCrl.crl = checkNotNull(crl, "crl");
              tmchCrl.url = checkNotNull(url, "url");
-              ofy().saveWithoutBackup().entity(tmchCrl);
+              ofyTm().transactNew(() -> ofyTm().putWithoutBackup(tmchCrl));
+              jpaTm()
+                  .transactNew(
+                      () -> {
+                        // Delete the old one and insert the new one
+                        jpaTm()
+                            .getEntityManager()
+                            .createQuery("DELETE FROM TmchCrl")
+                            .executeUpdate();
+                        jpaTm().putWithoutBackup(tmchCrl);
+                      });
            });
  }

@@ -74,4 +104,36 @@ public final class TmchCrl extends CrossTldSingleton {
  public final DateTime getUpdated() {
    return updated;
  }
+
+  @Override
+  public ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of(); // dually-written
+  }
+
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(); // dually-written
+  }
+
+  static class TmchCrlId implements Serializable {
+
+    @Column(name = "certificateRevocations")
+    String crl;
+
+    @Column(name = "updateTimestamp")
+    DateTime updated;
+
+    String url;
+
+    /** Hibernate requires this default constructor. */
+    private TmchCrlId() {}
+
+    static TmchCrlId create(String crl, DateTime updated, String url) {
+      TmchCrlId result = new TmchCrlId();
+      result.crl = crl;
+      result.updated = updated;
+      result.url = url;
+      return result;
+    }
+  }
 }
@@ -14,12 +14,16 @@

 package google.registry.model.transfer;

+import com.google.common.annotations.VisibleForTesting;
+import com.googlecode.objectify.Key;
+import com.googlecode.objectify.annotation.AlsoLoad;
 import com.googlecode.objectify.annotation.Embed;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.IgnoreSave;
 import com.googlecode.objectify.annotation.Unindex;
 import com.googlecode.objectify.condition.IfNull;
 import google.registry.model.billing.BillingEvent;
+import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.Period;
 import google.registry.model.domain.Period.Unit;
 import google.registry.model.poll.PollMessage;
@@ -86,6 +90,10 @@ public class DomainTransferData extends TransferData<DomainTransferData.Builder>
  @Column(name = "transfer_billing_event_id")
  VKey<BillingEvent.OneTime> serverApproveBillingEvent;

+  @Ignore
+  @Column(name = "transfer_billing_event_history_id")
+  Long serverApproveBillingEventHistoryId;
+
  /**
   * The autorenew billing event that should be associated with this resource after the transfer.
   *
@@ -96,6 +104,10 @@ public class DomainTransferData extends TransferData<DomainTransferData.Builder>
  @Column(name = "transfer_billing_recurrence_id")
  VKey<BillingEvent.Recurring> serverApproveAutorenewEvent;

+  @Ignore
+  @Column(name = "transfer_billing_recurrence_history_id")
+  Long serverApproveAutorenewEventHistoryId;
+
  /**
   * The autorenew poll message that should be associated with this resource after the transfer.
   *
@@ -106,11 +118,50 @@ public class DomainTransferData extends TransferData<DomainTransferData.Builder>
  @Column(name = "transfer_autorenew_poll_message_id")
  VKey<PollMessage.Autorenew> serverApproveAutorenewPollMessage;

+  @Ignore
+  @Column(name = "transfer_autorenew_poll_message_history_id")
+  Long serverApproveAutorenewPollMessageHistoryId;
+
  @Override
  public Builder copyConstantFieldsToBuilder() {
    return super.copyConstantFieldsToBuilder().setTransferPeriod(this.transferPeriod);
  }

+  /**
+   * Restores the set of ofy keys after loading from SQL using the specified {@code rootKey}.
+   *
+   * <p>This is for use by DomainBase/DomainHistory PostLoad methods ONLY.
+   */
+  public void restoreOfyKeys(Key<DomainBase> rootKey) {
+    serverApproveBillingEvent =
+        DomainBase.restoreOfyFrom(
+            rootKey, serverApproveBillingEvent, serverApproveBillingEventHistoryId);
+    serverApproveAutorenewEvent =
+        DomainBase.restoreOfyFrom(
+            rootKey, serverApproveAutorenewEvent, serverApproveAutorenewEventHistoryId);
+    serverApproveAutorenewPollMessage =
+        DomainBase.restoreOfyFrom(
+            rootKey, serverApproveAutorenewPollMessage, serverApproveAutorenewPollMessageHistoryId);
+  }
+
+  @SuppressWarnings("unused") // For Hibernate.
+  private void loadServerApproveBillingEventHistoryId(
+      @AlsoLoad("serverApproveBillingEvent") VKey<BillingEvent.OneTime> val) {
+    serverApproveBillingEventHistoryId = DomainBase.getHistoryId(val);
+  }
+
+  @SuppressWarnings("unused") // For Hibernate.
+  private void loadServerApproveAutorenewEventHistoryId(
+      @AlsoLoad("serverApproveAutorenewEvent") VKey<BillingEvent.Recurring> val) {
+    serverApproveAutorenewEventHistoryId = DomainBase.getHistoryId(val);
+  }
+
+  @SuppressWarnings("unused") // For Hibernate.
+  private void loadServerApproveAutorenewPollMessageHistoryId(
+      @AlsoLoad("serverApproveAutorenewPollMessage") VKey<PollMessage.Autorenew> val) {
+    serverApproveAutorenewPollMessageHistoryId = DomainBase.getHistoryId(val);
+  }
+
  public Period getTransferPeriod() {
    return transferPeriod;
  }
@@ -125,16 +176,34 @@ public class DomainTransferData extends TransferData<DomainTransferData.Builder>
    return serverApproveBillingEvent;
  }

+  @VisibleForTesting
+  @Nullable
+  public Long getServerApproveBillingEventHistoryId() {
+    return serverApproveBillingEventHistoryId;
+  }
+
  @Nullable
  public VKey<BillingEvent.Recurring> getServerApproveAutorenewEvent() {
    return serverApproveAutorenewEvent;
  }

+  @VisibleForTesting
+  @Nullable
+  public Long getServerApproveAutorenewEventHistoryId() {
+    return serverApproveAutorenewEventHistoryId;
+  }
+
  @Nullable
  public VKey<PollMessage.Autorenew> getServerApproveAutorenewPollMessage() {
    return serverApproveAutorenewPollMessage;
  }

+  @VisibleForTesting
+  @Nullable
+  public Long getServerApproveAutorenewPollMessageHistoryId() {
+    return serverApproveAutorenewPollMessageHistoryId;
+  }
+
  @Override
  public boolean isEmpty() {
    return EMPTY.equals(this);
@@ -168,18 +237,24 @@ public class DomainTransferData extends TransferData<DomainTransferData.Builder>
    public Builder setServerApproveBillingEvent(
        VKey<BillingEvent.OneTime> serverApproveBillingEvent) {
      getInstance().serverApproveBillingEvent = serverApproveBillingEvent;
+      getInstance().serverApproveBillingEventHistoryId =
+          DomainBase.getHistoryId(serverApproveBillingEvent);
      return this;
    }

    public Builder setServerApproveAutorenewEvent(
        VKey<BillingEvent.Recurring> serverApproveAutorenewEvent) {
      getInstance().serverApproveAutorenewEvent = serverApproveAutorenewEvent;
+      getInstance().serverApproveAutorenewEventHistoryId =
+          DomainBase.getHistoryId(serverApproveAutorenewEvent);
      return this;
    }

    public Builder setServerApproveAutorenewPollMessage(
        VKey<PollMessage.Autorenew> serverApproveAutorenewPollMessage) {
      getInstance().serverApproveAutorenewPollMessage = serverApproveAutorenewPollMessage;
+      getInstance().serverApproveAutorenewPollMessageHistoryId =
+          DomainBase.getHistoryId(serverApproveAutorenewPollMessage);
      return this;
    }
  }
@@ -22,7 +22,6 @@ import google.registry.request.RequestHandler;
 import google.registry.util.SystemClock;
 import java.io.IOException;
 import java.security.Security;
-import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import javax.servlet.http.HttpServlet;
 import javax.servlet.http.HttpServletRequest;
@@ -51,13 +50,16 @@ public class ServletBase extends HttpServlet {
    // etc), we log the error but keep the main thread running. Also the shutdown hook will only be
    // registered if metric reporter starts up correctly.
    try {
-      metricReporter.get().startAsync().awaitRunning(10, TimeUnit.SECONDS);
+      metricReporter.get().startAsync().awaitRunning(java.time.Duration.ofSeconds(10));
      logger.atInfo().log("Started up MetricReporter");
      LifecycleManager.getInstance()
          .setShutdownHook(
              () -> {
                try {
-                  metricReporter.get().stopAsync().awaitTerminated(10, TimeUnit.SECONDS);
+                  metricReporter
+                      .get()
+                      .stopAsync()
+                      .awaitTerminated(java.time.Duration.ofSeconds(10));
                  logger.atInfo().log("Shut down MetricReporter");
                } catch (TimeoutException e) {
                  logger.atSevere().withCause(e).log("Failed to stop MetricReporter.");
@@ -88,7 +88,7 @@ public class VKey<T> extends ImmutableObject implements Serializable {
   */
  public static <T> VKey<T> create(Class<T> kind, long id) {
    checkArgument(
-        kind.isAssignableFrom(BackupGroupRoot.class),
+        BackupGroupRoot.class.isAssignableFrom(kind),
        "The kind %s is not a BackupGroupRoot and thus needs its entire entity group chain"
            + " specified in a parent",
        kind.getCanonicalName());
@@ -106,13 +106,90 @@ public class VKey<T> extends ImmutableObject implements Serializable {
   */
  public static <T> VKey<T> create(Class<T> kind, String name) {
    checkArgument(
-        kind.isAssignableFrom(BackupGroupRoot.class),
+        BackupGroupRoot.class.isAssignableFrom(kind),
        "The kind %s is not a BackupGroupRoot and thus needs its entire entity group chain"
            + " specified in a parent",
        kind.getCanonicalName());
    return new VKey<T>(kind, Key.create(kind, name), name);
  }

+  /**
+   * Returns a clone with an ofy key restored from {@code ancestors}.
+   *
+   * <p>The arguments should generally consist of pairs of Class and value, where the Class is the
+   * kind of the ancestor key and the value is either a String or a Long.
+   *
+   * <p>For example, to restore the objectify key for
+   * DomainBase("COM-1234")/HistoryEntry(123)/PollEvent(567), one might use:
+   *
+   * <pre>{@code
+   * pollEvent.restoreOfy(DomainBase.class, "COM-1234", HistoryEntry.class, 567)
+   * }</pre>
+   *
+   * <p>The final key id or name is obtained from the SQL key. It is assumed that this value must be
+   * either a long integer or a {@code String} and that this proper identifier for the objectify
+   * key.
+   *
+   * <p>As a special case, an objectify Key may be used as the first ancestor instead of a Class,
+   * value pair.
+   */
+  public VKey<T> restoreOfy(Object... ancestors) {
+    Class lastClass = null;
+    Key<?> lastKey = null;
+    for (Object ancestor : ancestors) {
+      if (ancestor instanceof Class) {
+        if (lastClass != null) {
+          throw new IllegalArgumentException(ancestor + " used as a key value.");
+        }
+        lastClass = (Class) ancestor;
+        continue;
+      } else if (ancestor instanceof Key) {
+        if (lastKey != null) {
+          throw new IllegalArgumentException(
+              "Objectify keys may only be used for the first argument");
+        }
+        lastKey = (Key) ancestor;
+        continue;
+      }
+
+      // The argument should be a value.
+      if (lastClass == null) {
+        throw new IllegalArgumentException("Argument " + ancestor + " should be a class.");
+      }
+      if (ancestor instanceof Long) {
+        lastKey = Key.create(lastKey, lastClass, (Long) ancestor);
+      } else if (ancestor instanceof String) {
+        lastKey = Key.create(lastKey, lastClass, (String) ancestor);
+      } else {
+        throw new IllegalArgumentException("Key value " + ancestor + " must be a string or long.");
+      }
+      lastClass = null;
+    }
+
+    // Make sure we didn't end up with a dangling class with no value.
+    if (lastClass != null) {
+      throw new IllegalArgumentException("Missing value for last key of type " + lastClass);
+    }
+
+    Object sqlKey = getSqlKey();
+    Key<T> ofyKey =
+        sqlKey instanceof Long
+            ? Key.create(lastKey, getKind(), (Long) sqlKey)
+            : Key.create(lastKey, getKind(), (String) sqlKey);
+
+    return VKey.create((Class<T>) getKind(), sqlKey, ofyKey);
+  }
+
+  /**
+   * Returns a clone of {@code key} with an ofy key restored from {@code ancestors}.
+   *
+   * <p>This is the static form of the method restoreOfy() above. If {@code key} is null, it returns
+   * null.
+   */
+  public static <T> VKey<T> restoreOfyFrom(@Nullable VKey<T> key, Object... ancestors) {
+    return key == null ? null : key.restoreOfy(ancestors);
+  }
+
  /** Returns the type of the entity. */
  public Class<? extends T> getKind() {
    return this.kind;
@@ -14,17 +14,23 @@

 package google.registry.persistence.converter;

+import google.registry.util.DateTimeUtils;
+import java.sql.Date;
+import javax.persistence.AttributeConverter;
 import javax.persistence.Converter;
 import org.joda.time.LocalDate;
-import org.joda.time.format.ISODateTimeFormat;

-/** JPA converter for {@link LocalDate}. */
+/** JPA converter for {@link LocalDate}, to/from {@link Date}. */
@Converter(autoApply = true)
-public class LocalDateConverter extends ToStringConverterBase<LocalDate> {
+public class LocalDateConverter implements AttributeConverter<LocalDate, Date> {

-  /** Converts the string (a date in ISO-8601 format) into a LocalDate. */
  @Override
-  public LocalDate convertToEntityAttribute(String columnValue) {
-    return (columnValue == null) ? null : LocalDate.parse(columnValue, ISODateTimeFormat.date());
+  public Date convertToDatabaseColumn(LocalDate attribute) {
+    return attribute == null ? null : DateTimeUtils.toSqlDate(attribute);
+  }
+
+  @Override
+  public LocalDate convertToEntityAttribute(Date dbData) {
+    return dbData == null ? null : DateTimeUtils.toLocalDate(dbData);
  }
 }
@@ -15,6 +15,7 @@
 package google.registry.persistence.transaction;

 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.collect.ImmutableMap.toImmutableMap;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
@@ -25,6 +26,7 @@ import com.google.common.collect.ImmutableCollection;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Streams;
 import com.google.common.flogger.FluentLogger;
 import google.registry.config.RegistryConfig;
 import google.registry.persistence.JpaRetries;
@@ -238,6 +240,16 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    entities.forEach(this::insert);
  }

+  @Override
+  public void insertWithoutBackup(Object entity) {
+    insert(entity);
+  }
+
+  @Override
+  public void insertAllWithoutBackup(ImmutableCollection<?> entities) {
+    insertAll(entities);
+  }
+
  @Override
  public void put(Object entity) {
    checkArgumentNotNull(entity, "entity must be specified");
@@ -253,6 +265,16 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    entities.forEach(this::put);
  }

+  @Override
+  public void putWithoutBackup(Object entity) {
+    put(entity);
+  }
+
+  @Override
+  public void putAllWithoutBackup(ImmutableCollection<?> entities) {
+    putAll(entities);
+  }
+
  @Override
  public void update(Object entity) {
    checkArgumentNotNull(entity, "entity must be specified");
@@ -269,6 +291,16 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    entities.forEach(this::update);
  }

+  @Override
+  public void updateWithoutBackup(Object entity) {
+    update(entity);
+  }
+
+  @Override
+  public void updateAllWithoutBackup(ImmutableCollection<?> entities) {
+    updateAll(entities);
+  }
+
  @Override
  public <T> boolean exists(VKey<T> key) {
    checkArgumentNotNull(key, "key must be specified");
@@ -315,6 +347,14 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    return result;
  }

+  @Override
+  public <T> T load(T entity) {
+    checkArgumentNotNull(entity, "entity must be specified");
+    assertInTransaction();
+    return (T)
+        load(VKey.createSql(entity.getClass(), emf.getPersistenceUnitUtil().getIdentifier(entity)));
+  }
+
  @Override
  public <T> ImmutableMap<VKey<? extends T>, T> load(Iterable<? extends VKey<? extends T>> keys) {
    checkArgumentNotNull(keys, "keys must be specified");
@@ -342,6 +382,11 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
            .getResultList());
  }

+  @Override
+  public <T> ImmutableList<T> loadAll(Iterable<T> entities) {
+    return Streams.stream(entities).map(this::load).collect(toImmutableList());
+  }
+
  private int internalDelete(VKey<?> key) {
    checkArgumentNotNull(key, "key must be specified");
    assertInTransaction();
@@ -366,6 +411,37 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    vKeys.forEach(this::internalDelete);
  }

+  @Override
+  public void delete(Object entity) {
+    checkArgumentNotNull(entity, "entity must be specified");
+    assertInTransaction();
+    Object managedEntity = entity;
+    if (!getEntityManager().contains(entity)) {
+      managedEntity = getEntityManager().merge(entity);
+    }
+    getEntityManager().remove(managedEntity);
+  }
+
+  @Override
+  public void deleteWithoutBackup(VKey<?> key) {
+    delete(key);
+  }
+
+  @Override
+  public void deleteWithoutBackup(Iterable<? extends VKey<?>> keys) {
+    delete(keys);
+  }
+
+  @Override
+  public void deleteWithoutBackup(Object entity) {
+    delete(entity);
+  }
+
+  @Override
+  public void clearSessionCache() {
+    // This is an intended no-op method as there is no session cache in Postgresql.
+  }
+
  @Override
  public <T> void assertDelete(VKey<T> key) {
    if (internalDelete(key) != 1) {
@@ -17,6 +17,7 @@ package google.registry.persistence.transaction;
 import com.google.common.collect.ImmutableCollection;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
+import google.registry.model.ofy.DatastoreTransactionManager;
 import google.registry.persistence.VKey;
 import java.util.NoSuchElementException;
 import java.util.Optional;
@@ -91,18 +92,90 @@ public interface TransactionManager {
  /** Persists all new entities in the database, throws exception if any entity already exists. */
  void insertAll(ImmutableCollection<?> entities);

+  /**
+   * Persists a new entity in the database without writing any backup if the underlying database is
+   * Datastore.
+   *
+   * <p>This method is for the sake of keeping a single code path when replacing ofy() with tm() in
+   * the application code. When the method is invoked with Datastore, it won't write the commit log
+   * backup; when invoked with Cloud SQL, it behaves the same as the method which doesn't have
+   * "WithoutBackup" in its method name because it is not necessary to have the backup mechanism in
+   * SQL.
+   */
+  void insertWithoutBackup(Object entity);
+
+  /**
+   * Persists all new entities in the database without writing any backup if the underlying database
+   * is Datastore.
+   *
+   * <p>This method is for the sake of keeping a single code path when replacing ofy() with tm() in
+   * the application code. When the method is invoked with Datastore, it won't write the commit log
+   * backup; when invoked with Cloud SQL, it behaves the same as the method which doesn't have
+   * "WithoutBackup" in its method name because it is not necessary to have the backup mechanism in
+   * SQL.
+   */
+  void insertAllWithoutBackup(ImmutableCollection<?> entities);
+
  /** Persists a new entity or update the existing entity in the database. */
  void put(Object entity);

  /** Persists all new entities or update the existing entities in the database. */
  void putAll(ImmutableCollection<?> entities);

+  /**
+   * Persists a new entity or update the existing entity in the database without writing any backup
+   * if the underlying database is Datastore.
+   *
+   * <p>This method is for the sake of keeping a single code path when replacing ofy() with tm() in
+   * the application code. When the method is invoked with Datastore, it won't write the commit log
+   * backup; when invoked with Cloud SQL, it behaves the same as the method which doesn't have
+   * "WithoutBackup" in its method name because it is not necessary to have the backup mechanism in
+   * SQL.
+   */
+  void putWithoutBackup(Object entity);
+
+  /**
+   * Persists all new entities or update the existing entities in the database without writing any
+   * backup if the underlying database is Datastore.
+   *
+   * <p>This method is for the sake of keeping a single code path when replacing ofy() with tm() in
+   * the application code. When the method is invoked with Datastore, it won't write the commit log
+   * backup; when invoked with Cloud SQL, it behaves the same as the method which doesn't have
+   * "WithoutBackup" in its method name because it is not necessary to have the backup mechanism in
+   * SQL.
+   */
+  void putAllWithoutBackup(ImmutableCollection<?> entities);
+
  /** Updates an entity in the database, throws exception if the entity does not exist. */
  void update(Object entity);

  /** Updates all entities in the database, throws exception if any entity does not exist. */
  void updateAll(ImmutableCollection<?> entities);

+  /**
+   * Updates an entity in the database without writing any backup if the underlying database is
+   * Datastore.
+   *
+   * <p>This method is for the sake of keeping a single code path when replacing ofy() with tm() in
+   * the application code. When the method is invoked with Datastore, it won't write the commit log
+   * backup; when invoked with Cloud SQL, it behaves the same as the method which doesn't have
+   * "WithoutBackup" in its method name because it is not necessary to have the backup mechanism in
+   * SQL.
+   */
+  void updateWithoutBackup(Object entity);
+
+  /**
+   * Updates all entities in the database without writing any backup if the underlying database is
+   * Datastore.
+   *
+   * <p>This method is for the sake of keeping a single code path when replacing ofy() with tm() in
+   * the application code. When the method is invoked with Datastore, it won't write the commit log
+   * backup; when invoked with Cloud SQL, it behaves the same as the method which doesn't have
+   * "WithoutBackup" in its method name because it is not necessary to have the backup mechanism in
+   * SQL.
+   */
+  void updateAllWithoutBackup(ImmutableCollection<?> entities);
+
  /** Returns whether the given entity with same ID exists. */
  boolean exists(Object entity);

@@ -115,6 +188,11 @@ public interface TransactionManager {
  /** Loads the entity by its id, throws NoSuchElementException if it doesn't exist. */
  <T> T load(VKey<T> key);

+  /**
+   * Loads the given entity from the database, throws NoSuchElementException if it doesn't exist.
+   */
+  <T> T load(T entity);
+
  /**
   * Loads the set of entities by their key id.
   *
@@ -125,9 +203,43 @@ public interface TransactionManager {
  /** Loads all entities of the given type, returns empty if there is no such entity. */
  <T> ImmutableList<T> loadAll(Class<T> clazz);

+  /**
+   * Loads all given entities from the database, throws NoSuchElementException if it doesn't exist.
+   */
+  <T> ImmutableList<T> loadAll(Iterable<T> entities);
+
  /** Deletes the entity by its id. */
  void delete(VKey<?> key);

  /** Deletes the set of entities by their key id. */
  void delete(Iterable<? extends VKey<?>> keys);
+
+  /** Deletes the given entity from the database. */
+  void delete(Object entity);
+
+  /**
+   * Deletes the entity by its id without writing any backup if the underlying database is
+   * Datastore.
+   */
+  void deleteWithoutBackup(VKey<?> key);
+
+  /**
+   * Deletes the set of entities by their key id without writing any backup if the underlying
+   * database is Datastore.
+   */
+  void deleteWithoutBackup(Iterable<? extends VKey<?>> keys);
+
+  /**
+   * Deletes the given entity from the database without writing any backup if the underlying
+   * database is Datastore.
+   */
+  void deleteWithoutBackup(Object entity);
+
+  /** Clears the session cache if the underlying database is Datastore, otherwise it is a no-op. */
+  void clearSessionCache();
+
+  /** Returns true if the transaction manager is DatastoreTransactionManager. */
+  default boolean isOfy() {
+    return this instanceof DatastoreTransactionManager;
+  }
 }
@@ -0,0 +1,76 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.transaction;
+
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
+
+import google.registry.model.ofy.DatastoreTransactionManager;
+import java.util.function.Supplier;
+
+/** Utility class that provides supplementary methods for {@link TransactionManager}. */
+public class TransactionManagerUtil {
+
+  /**
+   * Returns the result of the given {@link Supplier}.
+   *
+   * <p>If {@link TransactionManagerFactory#tm()} returns a {@link JpaTransactionManager} instance,
+   * the {@link Supplier} is executed in a transaction.
+   */
+  public static <T> T transactIfJpaTm(Supplier<T> supplier) {
+    if (tm() instanceof JpaTransactionManager) {
+      return tm().transact(supplier);
+    } else {
+      return supplier.get();
+    }
+  }
+
+  /**
+   * Executes the given {@link Runnable}.
+   *
+   * <p>If {@link TransactionManagerFactory#tm()} returns a {@link JpaTransactionManager} instance,
+   * the {@link Runnable} is executed in a transaction.
+   */
+  public static void transactIfJpaTm(Runnable runnable) {
+    transactIfJpaTm(
+        () -> {
+          runnable.run();
+          return null;
+        });
+  }
+
+  /**
+   * Executes the given {@link Runnable} if {@link TransactionManagerFactory#tm()} returns a {@link
+   * DatastoreTransactionManager} instance, otherwise does nothing.
+   */
+  public static void ofyTmOrDoNothing(Runnable ofyRunnable) {
+    if (tm() instanceof DatastoreTransactionManager) {
+      ofyRunnable.run();
+    }
+  }
+
+  /**
+   * Returns the result from the given {@link Supplier} if {@link TransactionManagerFactory#tm()}
+   * returns a {@link DatastoreTransactionManager} instance, otherwise returns null.
+   */
+  public static <T> T ofyTmOrDoNothing(Supplier<T> ofySupplier) {
+    if (tm() instanceof DatastoreTransactionManager) {
+      return ofySupplier.get();
+    } else {
+      return null;
+    }
+  }
+
+  private TransactionManagerUtil() {}
+}
@@ -0,0 +1,35 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.schema.replay;
+
+import com.google.common.collect.ImmutableList;
+
+/**
+ * Represents an entity that should not participate in asynchronous replication.
+ *
+ * <p>We expect that this is a result of the entity being dually-written.
+ */
+public interface NonReplicatedEntity extends DatastoreEntity, SqlEntity {
+
+  @Override
+  default ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of();
+  }
+
+  @Override
+  default ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of();
+  }
+}
@@ -0,0 +1,58 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.schema.replay;
+
+import static google.registry.model.common.CrossTldSingleton.SINGLETON_ID;
+import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.util.DateTimeUtils.START_OF_TIME;
+
+import com.google.common.collect.ImmutableList;
+import javax.persistence.Entity;
+import javax.persistence.Id;
+import org.joda.time.DateTime;
+
+@Entity
+public class SqlReplayCheckpoint implements SqlEntity {
+
+  // Hibernate doesn't allow us to have a converted DateTime as our primary key so we need this
+  @Id private long revisionId = SINGLETON_ID;
+
+  private DateTime lastReplayTime;
+
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(); // not necessary to persist in Datastore
+  }
+
+  public static DateTime get() {
+    jpaTm().assertInTransaction();
+    return jpaTm()
+        .getEntityManager()
+        .createQuery("FROM SqlReplayCheckpoint", SqlReplayCheckpoint.class)
+        .setMaxResults(1)
+        .getResultStream()
+        .findFirst()
+        .map(checkpoint -> checkpoint.lastReplayTime)
+        .orElse(START_OF_TIME);
+  }
+
+  public static void set(DateTime lastReplayTime) {
+    jpaTm().assertInTransaction();
+    SqlReplayCheckpoint checkpoint = new SqlReplayCheckpoint();
+    checkpoint.lastReplayTime = lastReplayTime;
+    // this will overwrite the existing object due to the constant revisionId
+    jpaTm().put(checkpoint);
+  }
+}
@@ -18,7 +18,6 @@ import static google.registry.config.RegistryConfig.getDomainLabelListCacheDurat
 import static google.registry.config.RegistryConfig.getSingletonCachePersistDuration;
 import static google.registry.config.RegistryConfig.getStaticPremiumListMaxCachedEntries;
 import static google.registry.schema.tld.PremiumListDao.getPriceForLabel;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;

 import com.google.auto.value.AutoValue;
 import com.google.common.annotations.VisibleForTesting;
@@ -48,7 +47,7 @@ class PremiumListCache {
  static LoadingCache<String, Optional<PremiumList>> createCachePremiumLists(
      Duration cachePersistDuration) {
    return CacheBuilder.newBuilder()
-        .expireAfterWrite(cachePersistDuration.getMillis(), MILLISECONDS)
+        .expireAfterWrite(java.time.Duration.ofMillis(cachePersistDuration.getMillis()))
        .build(
            new CacheLoader<String, Optional<PremiumList>>() {
              @Override
@@ -81,7 +80,7 @@ class PremiumListCache {
  static LoadingCache<RevisionIdAndLabel, Optional<BigDecimal>> createCachePremiumEntries(
      Duration cachePersistDuration) {
    return CacheBuilder.newBuilder()
-        .expireAfterWrite(cachePersistDuration.getMillis(), MILLISECONDS)
+        .expireAfterWrite(java.time.Duration.ofMillis(cachePersistDuration.getMillis()))
        .maximumSize(getStaticPremiumListMaxCachedEntries())
        .build(
            new CacheLoader<RevisionIdAndLabel, Optional<BigDecimal>>() {
@@ -18,7 +18,6 @@ import static google.registry.config.RegistryConfig.ConfigModule.TmchCaMode.PILO
 import static google.registry.config.RegistryConfig.ConfigModule.TmchCaMode.PRODUCTION;
 import static google.registry.config.RegistryConfig.getSingletonCacheRefreshDuration;
 import static google.registry.util.ResourceUtils.readResourceUtf8;
-import static java.util.concurrent.TimeUnit.MILLISECONDS;

 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
@@ -33,6 +32,7 @@ import java.security.GeneralSecurityException;
 import java.security.cert.CertificateParsingException;
 import java.security.cert.X509CRL;
 import java.security.cert.X509Certificate;
+import java.util.Optional;
 import javax.annotation.concurrent.Immutable;
 import javax.annotation.concurrent.ThreadSafe;
 import javax.inject.Inject;
@@ -77,19 +77,21 @@ public final class TmchCertificateAuthority {
   */
  private static final LoadingCache<TmchCaMode, X509CRL> CRL_CACHE =
      CacheBuilder.newBuilder()
-          .expireAfterWrite(getSingletonCacheRefreshDuration().getMillis(), MILLISECONDS)
+          .expireAfterWrite(
+              java.time.Duration.ofMillis(getSingletonCacheRefreshDuration().getMillis()))
          .build(
              new CacheLoader<TmchCaMode, X509CRL>() {
                @Override
                public X509CRL load(final TmchCaMode tmchCaMode) throws GeneralSecurityException {
-                  TmchCrl storedCrl = TmchCrl.get();
-                  String crlContents;
-                  if (storedCrl == null) {
-                    String file = (tmchCaMode == PILOT) ? CRL_PILOT_FILE : CRL_FILE;
-                    crlContents = readResourceUtf8(TmchCertificateAuthority.class, file);
-                  } else {
-                    crlContents = storedCrl.getCrl();
-                  }
+                  Optional<TmchCrl> storedCrl = TmchCrl.get();
+                  String crlContents =
+                      storedCrl
+                          .map(TmchCrl::getCrl)
+                          .orElseGet(
+                              () -> {
+                                String file = (tmchCaMode == PILOT) ? CRL_PILOT_FILE : CRL_FILE;
+                                return readResourceUtf8(TmchCertificateAuthority.class, file);
+                              });
                  X509CRL crl = X509Utils.loadCrl(crlContents);
                  crl.verify(ROOT_CERTS.get(tmchCaMode).getPublicKey());
                  return crl;
@@ -18,7 +18,6 @@ import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Sets;
 import com.google.common.collect.Sets.SetView;
 import java.io.File;
-import java.util.function.Predicate;

 /**
 * Compares two Datastore backups in V3 format on local file system. This is for use in tests and
@@ -30,8 +29,10 @@ import java.util.function.Predicate;
 */
 class CompareDbBackups {
  private static final String DS_V3_BACKUP_FILE_PREFIX = "output-";
-  private static final Predicate<File> DATA_FILE_MATCHER =
-      file -> file.isFile() && file.getName().startsWith(DS_V3_BACKUP_FILE_PREFIX);
+
+  private static boolean isDatastoreV3File(File file) {
+    return file.isFile() && file.getName().startsWith(DS_V3_BACKUP_FILE_PREFIX);
+  }

  public static void main(String[] args) {
    if (args.length != 2) {
@@ -40,9 +41,11 @@ class CompareDbBackups {
    }

    ImmutableSet<EntityWrapper> entities1 =
-        RecordAccumulator.readDirectory(new File(args[0]), DATA_FILE_MATCHER).getEntityWrapperSet();
+        RecordAccumulator.readDirectory(new File(args[0]), CompareDbBackups::isDatastoreV3File)
+            .getEntityWrapperSet();
    ImmutableSet<EntityWrapper> entities2 =
-        RecordAccumulator.readDirectory(new File(args[1]), DATA_FILE_MATCHER).getEntityWrapperSet();
+        RecordAccumulator.readDirectory(new File(args[1]), CompareDbBackups::isDatastoreV3File)
+            .getEntityWrapperSet();

    // Calculate the entities added and removed.
    SetView<EntityWrapper> added = Sets.difference(entities2, entities1);
@@ -30,6 +30,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Sets;
 import com.google.common.flogger.FluentLogger;
+import google.registry.flows.certs.CertificateChecker;
 import google.registry.model.registrar.Registrar;
 import google.registry.model.registrar.RegistrarAddress;
 import google.registry.model.registry.Registry;
@@ -49,6 +50,7 @@ import java.util.Map;
 import java.util.Optional;
 import java.util.Set;
 import javax.annotation.Nullable;
+import javax.inject.Inject;
 import org.joda.money.CurrencyUnit;
 import org.joda.time.DateTime;

@@ -57,9 +59,9 @@ abstract class CreateOrUpdateRegistrarCommand extends MutatingCommand {

  static final FluentLogger logger = FluentLogger.forEnclosingClass();

-  @Parameter(
-      description = "Client identifier of the registrar account",
-      required = true)
+  @Inject CertificateChecker certificateChecker;
+
+  @Parameter(description = "Client identifier of the registrar account", required = true)
  List<String> mainParameters;

  @Parameter(
@@ -356,11 +358,21 @@ abstract class CreateOrUpdateRegistrarCommand extends MutatingCommand {
      }
      if (clientCertificateFilename != null) {
        String asciiCert = new String(Files.readAllBytes(clientCertificateFilename), US_ASCII);
+        // An empty certificate file is allowed in order to provide a functionality for removing an
+        // existing certificate without providing a replacement. An uploaded empty certificate file
+        // will prevent the registrar from being able to establish EPP connections.
+        if (!asciiCert.equals("")) {
+          certificateChecker.validateCertificate(asciiCert);
+        }
        builder.setClientCertificate(asciiCert, now);
      }
+
      if (failoverClientCertificateFilename != null) {
        String asciiCert =
            new String(Files.readAllBytes(failoverClientCertificateFilename), US_ASCII);
+        if (!asciiCert.equals("")) {
+          certificateChecker.validateCertificate(asciiCert);
+        }
        builder.setFailoverClientCertificate(asciiCert, now);
      }
      if (!isNullOrEmpty(clientCertificateHash)) {
@@ -19,42 +19,22 @@ import static google.registry.model.ofy.ObjectifyService.ofy;
 import static java.nio.charset.StandardCharsets.UTF_8;

 import com.beust.jcommander.Parameter;
-import com.beust.jcommander.Parameters;
 import com.google.appengine.api.datastore.KeyFactory;
 import com.google.common.base.Splitter;
 import com.google.common.io.CharStreams;
 import com.google.common.io.Files;
 import com.googlecode.objectify.Key;
 import google.registry.model.ImmutableObject;
-import google.registry.model.billing.BillingEvent;
 import google.registry.model.domain.DomainBase;
 import google.registry.util.NonFinalForTesting;
+import google.registry.util.TypeUtils.TypeInstantiator;
 import java.io.File;
 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.util.List;

-/**
- * Command to resave entities with a unique id.
- *
- * <p>This command is used to address the duplicate id issue we found for certain {@link
- * BillingEvent.OneTime} entities. The command reassigns an application wide unique id to the
- * problematic entity and resaves it, it also resaves the entity having reference to the problematic
- * entity with the updated id.
- *
- * <p>To use this command, you will need to provide the path to a file containing a list of strings
- * representing the literal of Objectify key for the problematic entities. An example key literal
- * is:
- *
- * <pre>
- * "DomainBase", "111111-TEST", "HistoryEntry", 2222222, "OneTime", 3333333
- * </pre>
- *
- * <p>Note that the double quotes are part of the key literal. The key literal can be retrieved from
- * the column <code>__key__.path</code> in BigQuery.
- */
-@Parameters(separators = " =", commandDescription = "Resave entities with a unique id.")
-public class ResaveEntitiesWithUniqueIdCommand extends MutatingCommand {
+/** Base Command to dedupe entities with duplicate IDs. */
+abstract class DedupeEntityIdsCommand<T> extends MutatingCommand {

  @Parameter(
      names = "--key_paths_file",
@@ -66,7 +46,9 @@ public class ResaveEntitiesWithUniqueIdCommand extends MutatingCommand {

  @NonFinalForTesting private static InputStream stdin = System.in;

-  private String keyChangeMessage;
+  private StringBuilder changeMessage = new StringBuilder();
+
+  abstract void dedupe(T entity);

  @Override
  protected void init() throws Exception {
@@ -85,8 +67,9 @@ public class ResaveEntitiesWithUniqueIdCommand extends MutatingCommand {
                keyPathsFile == null ? "STDIN" : "File " + keyPathsFile.getAbsolutePath()));
        continue;
      }
-      if (entity instanceof BillingEvent.OneTime) {
-        resaveBillingEvent((BillingEvent.OneTime) entity);
+      Class<T> clazz = new TypeInstantiator<T>(getClass()) {}.getExactType();
+      if (clazz.isInstance(entity)) {
+        dedupe((T) entity);
      } else {
        throw new IllegalArgumentException("Unsupported entity key: " + untypedKey);
      }
@@ -96,37 +79,19 @@ public class ResaveEntitiesWithUniqueIdCommand extends MutatingCommand {

  @Override
  protected void postBatchExecute() {
-    System.out.println(keyChangeMessage);
+    System.out.println(changeMessage);
  }

-  private void deleteOldAndSaveNewEntity(ImmutableObject oldEntity, ImmutableObject newEntity) {
+  void stageEntityKeyChange(ImmutableObject oldEntity, ImmutableObject newEntity) {
    stageEntityChange(oldEntity, null);
    stageEntityChange(null, newEntity);
+    appendChangeMessage(
+        String.format(
+            "Changed entity key from: %s to: %s", Key.create(oldEntity), Key.create(newEntity)));
  }

-  private void resaveBillingEvent(BillingEvent.OneTime billingEvent) {
-    Key<BillingEvent> key = Key.create(billingEvent);
-    Key<DomainBase> domainKey = getGrandParentAsDomain(key);
-    DomainBase domain = ofy().load().key(domainKey).now();
-
-    // The BillingEvent.OneTime entity to be resaved should be the billing event created a few
-    // years ago, so they should not be referenced from TransferData and GracePeriod in the domain.
-    assertNotInDomainTransferData(domain, key);
-    domain
-        .getGracePeriods()
-        .forEach(
-            gracePeriod ->
-                checkState(
-                    !gracePeriod.getOneTimeBillingEvent().getOfyKey().equals(key),
-                    "Entity %s is referenced by a grace period in domain %s",
-                    key,
-                    domainKey));
-
-    // By setting id to 0L, Buildable.build() will assign an application wide unique id to it.
-    BillingEvent.OneTime uniqIdBillingEvent = billingEvent.asBuilder().setId(0L).build();
-    deleteOldAndSaveNewEntity(billingEvent, uniqIdBillingEvent);
-    keyChangeMessage =
-        String.format("Old Entity Key: %s New Entity Key: %s", key, Key.create(uniqIdBillingEvent));
+  void appendChangeMessage(String message) {
+    changeMessage.append(message);
  }

  private static boolean isKind(Key<?> key, Class<?> clazz) {
@@ -165,7 +130,7 @@ public class ResaveEntitiesWithUniqueIdCommand extends MutatingCommand {
    return literal.substring(1, literal.length() - 1);
  }

-  private static Key<DomainBase> getGrandParentAsDomain(Key<?> key) {
+  static Key<DomainBase> getGrandParentAsDomain(Key<?> key) {
    Key<?> grandParent;
    try {
      grandParent = key.getParent().getParent();
@@ -178,19 +143,4 @@ public class ResaveEntitiesWithUniqueIdCommand extends MutatingCommand {
    }
    return (Key<DomainBase>) grandParent;
  }
-
-  private static void assertNotInDomainTransferData(DomainBase domainBase, Key<?> key) {
-    if (!domainBase.getTransferData().isEmpty()) {
-      domainBase
-          .getTransferData()
-          .getServerApproveEntities()
-          .forEach(
-              entityKey ->
-                  checkState(
-                      !entityKey.getOfyKey().equals(key),
-                      "Entity %s is referenced by the transfer data in domain %s",
-                      key,
-                      domainBase.createVKey().getOfyKey()));
-    }
-  }
 }
@@ -0,0 +1,88 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.tools;
+
+import static com.google.common.base.Preconditions.checkState;
+import static google.registry.model.ofy.ObjectifyService.ofy;
+
+import com.beust.jcommander.Parameters;
+import com.googlecode.objectify.Key;
+import google.registry.model.billing.BillingEvent;
+import google.registry.model.billing.BillingEvent.OneTime;
+import google.registry.model.domain.DomainBase;
+
+/**
+ * Command to dedupe {@link BillingEvent.OneTime} entities having duplicate IDs.
+ *
+ * <p>This command is used to address the duplicate id issue we found for certain {@link
+ * BillingEvent.OneTime} entities. The command reassigns an application wide unique id to the
+ * problematic entity and resaves it, it also resaves the entity having reference to the problematic
+ * entity with the updated id.
+ *
+ * <p>To use this command, you will need to provide the path to a file containing a list of strings
+ * representing the literal of Objectify key for the problematic entities. An example key literal
+ * is:
+ *
+ * <pre>
+ * "DomainBase", "111111-TEST", "HistoryEntry", 2222222, "OneTime", 3333333
+ * </pre>
+ *
+ * <p>Note that the double quotes are part of the key literal. The key literal can be retrieved from
+ * the column <code>__key__.path</code> in BigQuery.
+ */
+@Parameters(
+    separators = " =",
+    commandDescription = "Dedupe BillingEvent.OneTime entities with duplicate IDs.")
+public class DedupeOneTimeBillingEventIdsCommand extends DedupeEntityIdsCommand<OneTime> {
+
+  @Override
+  void dedupe(OneTime entity) {
+    Key<BillingEvent> key = Key.create(entity);
+    Key<DomainBase> domainKey = getGrandParentAsDomain(key);
+    DomainBase domain = ofy().load().key(domainKey).now();
+
+    // The BillingEvent.OneTime entity to be resaved should be the billing event created a few
+    // years ago, so they should not be referenced from TransferData and GracePeriod in the domain.
+    assertNotInDomainTransferData(domain, key);
+    domain
+        .getGracePeriods()
+        .forEach(
+            gracePeriod ->
+                checkState(
+                    !gracePeriod.getOneTimeBillingEvent().getOfyKey().equals(key),
+                    "Entity %s is referenced by a grace period in domain %s",
+                    key,
+                    domainKey));
+
+    // By setting id to 0L, Buildable.build() will assign an application wide unique id to it.
+    BillingEvent.OneTime uniqIdBillingEvent = entity.asBuilder().setId(0L).build();
+    stageEntityKeyChange(entity, uniqIdBillingEvent);
+  }
+
+  private static void assertNotInDomainTransferData(DomainBase domainBase, Key<?> key) {
+    if (!domainBase.getTransferData().isEmpty()) {
+      domainBase
+          .getTransferData()
+          .getServerApproveEntities()
+          .forEach(
+              entityKey ->
+                  checkState(
+                      !entityKey.getOfyKey().equals(key),
+                      "Entity %s is referenced by the transfer data in domain %s",
+                      key,
+                      domainBase.createVKey().getOfyKey()));
+    }
+  }
+}
@@ -0,0 +1,192 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.tools;
+
+import static com.google.common.collect.ImmutableSet.toImmutableSet;
+import static google.registry.model.ofy.ObjectifyService.ofy;
+
+import com.beust.jcommander.Parameters;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Sets;
+import com.googlecode.objectify.Key;
+import google.registry.model.billing.BillingEvent;
+import google.registry.model.billing.BillingEvent.OneTime;
+import google.registry.model.billing.BillingEvent.Recurring;
+import google.registry.model.domain.DomainBase;
+import google.registry.model.domain.GracePeriod;
+import google.registry.model.transfer.DomainTransferData;
+import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
+import google.registry.persistence.VKey;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A command that re-saves the problematic {@link BillingEvent.Recurring} entities with unique IDs.
+ *
+ * <p>This command is used to address the duplicate id issue we found for certain {@link
+ * BillingEvent.Recurring} entities. The command reassigns an application wide unique id to the
+ * problematic entity and resaves it, it also resaves the entity having reference to the problematic
+ * entity with the updated id.
+ *
+ * <p>To use this command, you will need to provide the path to a file containing a list of strings
+ * representing the literal of Objectify key for the problematic entities. An example key literal
+ * is:
+ *
+ * <pre>
+ * "DomainBase", "111111-TEST", "HistoryEntry", 2222222, "Recurring", 3333333
+ * </pre>
+ *
+ * <p>Note that the double quotes are part of the key literal. The key literal can be retrieved from
+ * the column <code>__key__.path</code> in BigQuery.
+ */
+@Parameters(
+    separators = " =",
+    commandDescription = "Dedupe BillingEvent.Recurring entities with duplicate IDs.")
+public class DedupeRecurringBillingEventIdsCommand
+    extends DedupeEntityIdsCommand<BillingEvent.Recurring> {
+
+  @Override
+  void dedupe(Recurring recurring) {
+    // Loads the associated DomainBase and BillingEvent.OneTime entities that
+    // may have reference to this BillingEvent.Recurring entity.
+    Key<DomainBase> domainKey = getGrandParentAsDomain(Key.create(recurring));
+    DomainBase domain = ofy().load().key(domainKey).now();
+    List<BillingEvent.OneTime> oneTimes =
+        ofy().load().type(BillingEvent.OneTime.class).ancestor(domainKey).list();
+
+    VKey<Recurring> oldRecurringVKey = recurring.createVKey();
+    // By setting id to 0L, Buildable.build() will assign an application wide unique id to it.
+    Recurring uniqIdRecurring = recurring.asBuilder().setId(0L).build();
+    VKey<Recurring> newRecurringVKey = uniqIdRecurring.createVKey();
+
+    // After having the unique id for the BillingEvent.Recurring entity, we also need to
+    // update the references in other entities to point to the new BillingEvent.Recurring
+    // entity.
+    updateReferenceInOneTimeBillingEvent(oneTimes, oldRecurringVKey, newRecurringVKey);
+    updateReferenceInDomain(domain, oldRecurringVKey, newRecurringVKey);
+
+    stageEntityKeyChange(recurring, uniqIdRecurring);
+  }
+
+  /**
+   * Resaves {@link BillingEvent.OneTime} entities with updated {@link
+   * BillingEvent.OneTime#cancellationMatchingBillingEvent}.
+   *
+   * <p>{@link BillingEvent.OneTime#cancellationMatchingBillingEvent} is a {@link VKey} to a {@link
+   * BillingEvent.Recurring} entity. So, if the {@link BillingEvent.Recurring} entity gets a new key
+   * by changing its id, we need to update {@link
+   * BillingEvent.OneTime#cancellationMatchingBillingEvent} as well.
+   */
+  private void updateReferenceInOneTimeBillingEvent(
+      List<OneTime> oneTimes, VKey<Recurring> oldRecurringVKey, VKey<Recurring> newRecurringVKey) {
+    oneTimes.forEach(
+        oneTime -> {
+          if (oneTime.getCancellationMatchingBillingEvent() != null
+              && oneTime.getCancellationMatchingBillingEvent().equals(oldRecurringVKey)) {
+            BillingEvent.OneTime updatedOneTime =
+                oneTime.asBuilder().setCancellationMatchingBillingEvent(newRecurringVKey).build();
+            stageEntityChange(oneTime, updatedOneTime);
+            appendChangeMessage(
+                String.format(
+                    "Changed cancellationMatchingBillingEvent in entity %s from %s to %s\n",
+                    oneTime.createVKey().getOfyKey(),
+                    oneTime.getCancellationMatchingBillingEvent().getOfyKey(),
+                    updatedOneTime.getCancellationMatchingBillingEvent().getOfyKey()));
+          }
+        });
+  }
+
+  /**
+   * Resaves {@link DomainBase} entity with updated references to {@link BillingEvent.Recurring}
+   * entity.
+   *
+   * <p>The following 4 fields in the domain entity can be or have a reference to this
+   * BillingEvent.Recurring entity, so we need to check them and replace them with the new entity
+   * when necessary:
+   *
+   * <ol>
+   *   <li>domain.autorenewBillingEvent, see {@link DomainBase#autorenewBillingEvent}
+   *   <li>domain.transferData.serverApproveAutorenewEvent, see {@link
+   *       DomainTransferData#serverApproveAutorenewEvent}
+   *   <li>domain.transferData.serverApproveEntities, see {@link
+   *       DomainTransferData#serverApproveEntities}
+   *   <li>domain.gracePeriods.billingEventRecurring, see {@link GracePeriod#billingEventRecurring}
+   * </ol>
+   */
+  private void updateReferenceInDomain(
+      DomainBase domain, VKey<Recurring> oldRecurringVKey, VKey<Recurring> newRecurringVKey) {
+    DomainBase.Builder domainBuilder = domain.asBuilder();
+    StringBuilder domainChange =
+        new StringBuilder(
+            String.format(
+                "Resaved domain %s with following changes:\n", domain.createVKey().getOfyKey()));
+
+    if (domain.getAutorenewBillingEvent() != null
+        && domain.getAutorenewBillingEvent().equals(oldRecurringVKey)) {
+      domainBuilder.setAutorenewBillingEvent(newRecurringVKey);
+      domainChange.append(
+          String.format(
+              "  Changed autorenewBillingEvent from %s to %s.\n",
+              oldRecurringVKey, newRecurringVKey));
+    }
+
+    if (domain.getTransferData().getServerApproveAutorenewEvent() != null
+        && domain.getTransferData().getServerApproveAutorenewEvent().equals(oldRecurringVKey)) {
+      Set<VKey<? extends TransferServerApproveEntity>> serverApproveEntities =
+          Sets.union(
+              Sets.difference(
+                  domain.getTransferData().getServerApproveEntities(),
+                  ImmutableSet.of(oldRecurringVKey)),
+              ImmutableSet.of(newRecurringVKey));
+      domainBuilder.setTransferData(
+          domain
+              .getTransferData()
+              .asBuilder()
+              .setServerApproveEntities(ImmutableSet.copyOf(serverApproveEntities))
+              .setServerApproveAutorenewEvent(newRecurringVKey)
+              .build());
+      domainChange.append(
+          String.format(
+              "  Changed transferData.serverApproveAutoRenewEvent from %s to %s.\n",
+              oldRecurringVKey, newRecurringVKey));
+      domainChange.append(
+          String.format(
+              "  Changed transferData.serverApproveEntities to remove %s and add %s.\n",
+              oldRecurringVKey, newRecurringVKey));
+    }
+
+    ImmutableSet<GracePeriod> updatedGracePeriod =
+        domain.getGracePeriods().stream()
+            .map(
+                gracePeriod ->
+                    gracePeriod.getRecurringBillingEvent().equals(oldRecurringVKey)
+                        ? gracePeriod.cloneWithRecurringBillingEvent(newRecurringVKey)
+                        : gracePeriod)
+            .collect(toImmutableSet());
+    if (!updatedGracePeriod.equals(domain.getGracePeriods())) {
+      domainBuilder.setGracePeriods(updatedGracePeriod);
+      domainChange.append(
+          String.format(
+              "  Changed gracePeriods to remove %s and add %s.\n",
+              oldRecurringVKey, newRecurringVKey));
+    }
+
+    DomainBase updatedDomain = domainBuilder.build();
+    if (!updatedDomain.equals(domain)) {
+      stageEntityChange(domain, updatedDomain);
+      appendChangeMessage(domainChange.toString());
+    }
+  }
+}
@@ -48,6 +48,8 @@ public final class RegistryTool {
          .put("create_reserved_list", CreateReservedListCommand.class)
          .put("create_tld", CreateTldCommand.class)
          .put("curl", CurlCommand.class)
+          .put("dedupe_one_time_billing_event_ids", DedupeOneTimeBillingEventIdsCommand.class)
+          .put("dedupe_recurring_billing_event_ids", DedupeRecurringBillingEventIdsCommand.class)
          .put("delete_allocation_tokens", DeleteAllocationTokensCommand.class)
          .put("delete_domain", DeleteDomainCommand.class)
          .put("delete_host", DeleteHostCommand.class)
@@ -99,7 +101,6 @@ public final class RegistryTool {
          .put("remove_ip_address", RemoveIpAddressCommand.class)
          .put("renew_domain", RenewDomainCommand.class)
          .put("resave_entities", ResaveEntitiesCommand.class)
-          .put("resave_entities_with_unique_id", ResaveEntitiesWithUniqueIdCommand.class)
          .put("resave_environment_entities", ResaveEnvironmentEntitiesCommand.class)
          .put("resave_epp_resource", ResaveEppResourceCommand.class)
          .put("send_escrow_report_to_icann", SendEscrowReportToIcannCommand.class)
@@ -83,42 +83,83 @@ import javax.inject.Singleton;
    })
 interface RegistryToolComponent {
  void inject(AckPollMessagesCommand command);
+
  void inject(CheckDomainClaimsCommand command);
+
  void inject(CheckDomainCommand command);
+
  void inject(CountDomainsCommand command);
+
  void inject(CreateAnchorTenantCommand command);
+
  void inject(CreateCdnsTld command);
+
  void inject(CreateContactCommand command);
+
  void inject(CreateDomainCommand command);
+
+  void inject(CreateRegistrarCommand command);
+
  void inject(CreateTldCommand command);
+
  void inject(DeployInvoicingPipelineCommand command);
+
  void inject(DeploySpec11PipelineCommand command);
+
  void inject(EncryptEscrowDepositCommand command);
+
  void inject(GenerateAllocationTokensCommand command);
+
  void inject(GenerateDnsReportCommand command);
+
  void inject(GenerateEscrowDepositCommand command);
+
  void inject(GetKeyringSecretCommand command);
+
  void inject(GetOperationStatusCommand command);
+
  void inject(GhostrydeCommand command);
+
  void inject(ImportDatastoreCommand command);
+
  void inject(ListCursorsCommand command);
+
  void inject(ListDatastoreOperationsCommand command);
+
  void inject(LoadSnapshotCommand command);
+
  void inject(LockDomainCommand command);
+
  void inject(LoginCommand command);
+
  void inject(LogoutCommand command);
+
  void inject(PendingEscrowCommand command);
+
  void inject(RenewDomainCommand command);
+
  void inject(SendEscrowReportToIcannCommand command);
+
  void inject(SetNumInstancesCommand command);
+
  void inject(SetupOteCommand command);
+
  void inject(UnlockDomainCommand command);
+
  void inject(UnrenewDomainCommand command);
+
  void inject(UpdateCursorsCommand command);
+
  void inject(UpdateDomainCommand command);
+
  void inject(UpdateKmsKeyringCommand command);
+
+  void inject(UpdateRegistrarCommand command);
+
  void inject(UpdateTldCommand command);
+
  void inject(ValidateEscrowDepositCommand command);
+
  void inject(WhoisQueryCommand command);

  AppEngineConnection appEngineConnection();
@@ -38,6 +38,7 @@ import com.google.common.collect.Sets;
 import com.google.common.collect.Streams;
 import com.google.common.flogger.FluentLogger;
 import google.registry.config.RegistryEnvironment;
+import google.registry.flows.certs.CertificateChecker;
 import google.registry.model.registrar.Registrar;
 import google.registry.model.registrar.RegistrarContact;
 import google.registry.model.registrar.RegistrarContact.Type;
@@ -64,7 +65,6 @@ import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
-import java.util.function.Predicate;
 import javax.inject.Inject;
 import org.joda.time.DateTime;

@@ -93,11 +93,13 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
  @Inject SendEmailUtils sendEmailUtils;
  @Inject AuthenticatedRegistrarAccessor registrarAccessor;
  @Inject AuthResult authResult;
+  @Inject CertificateChecker certificateChecker;

  @Inject RegistrarSettingsAction() {}

-  private static final Predicate<RegistrarContact> HAS_PHONE =
-      contact -> contact.getPhoneNumber() != null;
+  private static boolean hasPhone(RegistrarContact contact) {
+    return contact.getPhoneNumber() != null;
+  }

  @Override
  public void run() {
@@ -306,19 +308,40 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
        RegistrarFormFields.IP_ADDRESS_ALLOW_LIST_FIELD
            .extractUntyped(args)
            .orElse(ImmutableList.of()));
-    RegistrarFormFields.CLIENT_CERTIFICATE_FIELD
-        .extractUntyped(args)
-        .ifPresent(
-            certificate -> builder.setClientCertificate(certificate, tm().getTransactionTime()));
-    RegistrarFormFields.FAILOVER_CLIENT_CERTIFICATE_FIELD
-        .extractUntyped(args)
-        .ifPresent(
-            certificate ->
-                builder.setFailoverClientCertificate(certificate, tm().getTransactionTime()));
+
+    Optional<String> certificateString =
+        RegistrarFormFields.CLIENT_CERTIFICATE_FIELD.extractUntyped(args);
+    if (certificateString.isPresent()) {
+      if (validateCertificate(initialRegistrar.getClientCertificate(), certificateString.get())) {
+        builder.setClientCertificate(certificateString.get(), tm().getTransactionTime());
+      }
+    }
+
+    Optional<String> failoverCertificateString =
+        RegistrarFormFields.FAILOVER_CLIENT_CERTIFICATE_FIELD.extractUntyped(args);
+    if (failoverCertificateString.isPresent()) {
+      if (validateCertificate(
+          initialRegistrar.getFailoverClientCertificate(), failoverCertificateString.get())) {
+        builder.setFailoverClientCertificate(
+            failoverCertificateString.get(), tm().getTransactionTime());
+      }
+    }

    return checkNotChangedUnlessAllowed(builder, initialRegistrar, Role.OWNER);
  }

+  /**
+   * Returns true if the registrar should accept the new certificate. Returns false if the
+   * certificate is already the one stored for the registrar.
+   */
+  private boolean validateCertificate(String existingCertificate, String certificateString) {
+    if ((existingCertificate == null) || !existingCertificate.equals(certificateString)) {
+        certificateChecker.validateCertificate(certificateString);
+      return true;
+    }
+    return false;
+  }
+
  /**
   * Updates a registrar with the ADMIN-controlled args from the http request.
   *
@@ -512,8 +535,8 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
      Multimap<Type, RegistrarContact> newContactsByType,
      Type... types) {
    for (Type type : types) {
-      if (oldContactsByType.get(type).stream().anyMatch(HAS_PHONE)
-          && newContactsByType.get(type).stream().noneMatch(HAS_PHONE)) {
+      if (oldContactsByType.get(type).stream().anyMatch(RegistrarSettingsAction::hasPhone)
+          && newContactsByType.get(type).stream().noneMatch(RegistrarSettingsAction::hasPhone)) {
        throw new ContactRequirementException(
            String.format(
                "Please provide a phone number for at least one %s contact",
@@ -11,11 +11,6 @@
    </attributes>
  </embeddable>

-  <sequence-generator name="HistorySequenceGenerator" sequence-name="history_id_sequence"/>
-
-  <!-- TODO(shicong): Drop this sequence and change all history tables to use the above one. -->
-  <sequence-generator name="TempHistorySequenceGenerator" sequence-name="temp_history_id_sequence"/>
-
  <persistence-unit-metadata>
    <persistence-unit-defaults>
      <entity-listeners>
@@ -47,12 +47,14 @@
    <class>google.registry.model.domain.DomainHistory</class>
    <class>google.registry.model.domain.GracePeriod</class>
    <class>google.registry.model.domain.secdns.DelegationSignerData</class>
+    <class>google.registry.model.domain.secdns.DomainDsDataHistory</class>
    <class>google.registry.model.domain.token.AllocationToken</class>
    <class>google.registry.model.host.HostHistory</class>
    <class>google.registry.model.host.HostResource</class>
    <class>google.registry.model.poll.PollMessage</class>
    <class>google.registry.model.poll.PollMessage$OneTime</class>
    <class>google.registry.model.poll.PollMessage$Autorenew</class>
+    <class>google.registry.model.rde.RdeRevision</class>
    <class>google.registry.model.registrar.Registrar</class>
    <class>google.registry.model.registrar.RegistrarContact</class>
    <class>google.registry.model.registry.label.PremiumList</class>
@@ -60,10 +62,15 @@
    <class>google.registry.model.registry.Registry</class>
    <class>google.registry.model.reporting.DomainTransactionRecord</class>
    <class>google.registry.model.reporting.Spec11ThreatMatch</class>
+    <class>google.registry.model.server.KmsSecretRevision</class>
+    <class>google.registry.model.server.ServerSecret</class>
+    <class>google.registry.model.smd.SignedMarkRevocationList</class>
    <class>google.registry.model.tmch.ClaimsListShard</class>
+    <class>google.registry.model.tmch.TmchCrl</class>
    <class>google.registry.persistence.transaction.TransactionEntity</class>
    <class>google.registry.schema.cursor.Cursor</class>
    <class>google.registry.schema.domain.RegistryLock</class>
+    <class>google.registry.schema.replay.SqlReplayCheckpoint</class>
    <class>google.registry.schema.server.Lock</class>
    <class>google.registry.schema.tld.PremiumEntry</class>

@@ -94,7 +101,10 @@
    <class>google.registry.persistence.converter.ZonedDateTimeConverter</class>

    <!-- Generated converters for VKey -->
-    <class>google.registry.model.billing.VKeyConverter_BillingEvent</class>
+    <class>google.registry.model.billing.VKeyConverter_Cancellation</class>
+    <class>google.registry.model.billing.VKeyConverter_Modification</class>
+    <class>google.registry.model.billing.VKeyConverter_OneTime</class>
+    <class>google.registry.model.billing.VKeyConverter_Recurring</class>
    <class>google.registry.model.contact.VKeyConverter_ContactResource</class>
    <class>google.registry.model.domain.VKeyConverter_DomainBase</class>
    <class>google.registry.model.domain.token.VKeyConverter_AllocationToken</class>
@@ -28,6 +28,7 @@ public class DevTool {
  public static final ImmutableMap<String, Class<? extends Command>> COMMAND_MAP =
      ImmutableMap.of(
          "dump_golden_schema", DumpGoldenSchemaCommand.class,
+          "generate_sql_er_diagram", GenerateSqlErDiagramCommand.class,
          "generate_sql_schema", GenerateSqlSchemaCommand.class);

  public static void main(String[] args) throws Exception {
@@ -0,0 +1,227 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.tools;
+
+import static com.google.common.base.Preconditions.checkState;
+import static google.registry.tools.GenerateSqlErDiagramCommand.DiagramType.ALL;
+import static google.registry.tools.GenerateSqlErDiagramCommand.DiagramType.BRIEF;
+import static google.registry.tools.GenerateSqlErDiagramCommand.DiagramType.FULL;
+
+import com.beust.jcommander.Parameter;
+import com.beust.jcommander.Parameters;
+import com.beust.jcommander.converters.PathConverter;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Iterables;
+import com.google.common.io.Resources;
+import google.registry.persistence.NomulusPostgreSql;
+import google.registry.util.ResourceUtils;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Properties;
+import org.jsoup.Jsoup;
+import org.jsoup.nodes.Document;
+import org.testcontainers.containers.PostgreSQLContainer;
+import schemacrawler.schemacrawler.LoadOptionsBuilder;
+import schemacrawler.schemacrawler.SchemaCrawlerOptions;
+import schemacrawler.schemacrawler.SchemaCrawlerOptionsBuilder;
+import schemacrawler.schemacrawler.SchemaInfoLevelBuilder;
+import schemacrawler.tools.executable.SchemaCrawlerExecutable;
+import schemacrawler.tools.integration.diagram.DiagramOutputFormat;
+import schemacrawler.tools.options.OutputOptions;
+import schemacrawler.tools.options.OutputOptionsBuilder;
+
+/** Command to generate ER diagrams for SQL schema. */
+@Parameters(separators = " =", commandDescription = "Generate ER diagrams for SQL schmea.")
+public class GenerateSqlErDiagramCommand implements Command {
+
+  private static final String DB_NAME = "postgres";
+  private static final String DB_USER = "username";
+  private static final String DB_PASSWORD = "password";
+  private static final String FULL_DIAGRAM_COMMAND = "schema";
+  private static final String BRIEF_DIAGRAM_COMMAND = "brief";
+  private static final String FULL_DIAGRAM_FILE_NAME = "full_er_diagram.html";
+  private static final String BRIEF_DIAGRAM_FILE_NAME = "brief_er_diagram.html";
+  private static final String NOMULUS_GOLDEN_SCHEMA = "sql/schema/nomulus.golden.sql";
+  private static final String FLYWAY_FILE = "sql/flyway.txt";
+  private static final String SVG_PAN_ZOOM_LIB = "google/registry/tools/svg-pan-zoom.min.js";
+
+  // The HTML element ID for the last flyway file name
+  static final String FLYWAY_FILE_ELEMENT_ID = "lastFlywayFile";
+
+  @Parameter(
+      names = {"-o", "--out_dir"},
+      description = "Name of the output directory to store ER diagrams.",
+      converter = PathConverter.class,
+      required = true)
+  private Path outDir;
+
+  @Parameter(
+      names = "--diagram_type",
+      description =
+          "Type of the generated ER diagram, can be FULL, BRIEF and ALL (defaults to ALL).")
+  private DiagramType diagramType = ALL;
+
+  /** The type of ER diagram. */
+  public enum DiagramType {
+    /** An HTML file that has an embedded ER diagram showing the full SQL schema. */
+    FULL,
+
+    /**
+     * An HTML file that has an embedded ER diagram showing only significant columns, such as
+     * primary and foreign key columns, and columns that are part of unique indexes.
+     */
+    BRIEF,
+
+    /** Generates all types of ER diagrams. */
+    ALL
+  }
+
+  @Override
+  public void run() throws Exception {
+    if (!outDir.toFile().exists()) {
+      checkState(outDir.toFile().mkdirs(), "Failed to create directory %s", outDir);
+    }
+
+    PostgreSQLContainer postgresContainer =
+        new PostgreSQLContainer(NomulusPostgreSql.getDockerTag())
+            .withDatabaseName(DB_NAME)
+            .withUsername(DB_USER)
+            .withPassword(DB_PASSWORD);
+    postgresContainer.start();
+
+    try (Connection conn = getConnection(postgresContainer)) {
+      initDb(conn);
+      if (diagramType == ALL || diagramType == FULL) {
+        improveDiagramHtml(generateErDiagram(conn, FULL_DIAGRAM_COMMAND, FULL_DIAGRAM_FILE_NAME));
+      }
+      if (diagramType == ALL || diagramType == BRIEF) {
+        improveDiagramHtml(generateErDiagram(conn, BRIEF_DIAGRAM_COMMAND, BRIEF_DIAGRAM_FILE_NAME));
+      }
+    } finally {
+      postgresContainer.stop();
+    }
+  }
+
+  private void improveDiagramHtml(Path diagram) {
+    try {
+      Document doc = Jsoup.parse(diagram.toFile(), StandardCharsets.UTF_8.name());
+
+      // Add the last name of the flyway file to the HTML so we can have a test to verify that if
+      // the generated diagram is up to date.
+      doc.select("body > table > tbody")
+          .first()
+          .append(
+              String.format(
+                  "<tr>"
+                      + "<td class=\"property_name\">last flyway file</td>"
+                      + "<td id=\""
+                      + FLYWAY_FILE_ELEMENT_ID
+                      + "\" class=\"property_value\">"
+                      + getLastFlywayFileName()
+                      + "</td>"
+                      + "</tr>"));
+
+      // Add pan and zoom support for the embedded SVG in the HTML.
+      StringBuilder svgPanZoomLib =
+          new StringBuilder("<script>")
+              .append(ResourceUtils.readResourceUtf8(Resources.getResource(SVG_PAN_ZOOM_LIB)))
+              .append("</script>");
+      doc.select("head").first().append(svgPanZoomLib.toString());
+      doc.select("svg")
+          .first()
+          .attributes()
+          .add("id", "erDiagram")
+          .add("style", "overflow: hidden; width: 100%; height: 800px");
+      doc.select("body")
+          .first()
+          .append(
+              "<script>"
+                  + "svgPanZoom('#erDiagram', {"
+                  + "  zoomEnabled: true,"
+                  + "  controlIconsEnabled: true,"
+                  + "  fit: true,"
+                  + "  center: true,"
+                  + "  minZoom: 0.1"
+                  + "});"
+                  + "</script>");
+
+      Files.write(
+          diagram, doc.outerHtml().getBytes(StandardCharsets.UTF_8), StandardOpenOption.WRITE);
+    } catch (IOException e) {
+      throw new UncheckedIOException(e);
+    }
+  }
+
+  private Path generateErDiagram(Connection connection, String command, String fileName) {
+    Path outputFile = outDir.resolve(fileName);
+
+    LoadOptionsBuilder loadOptionsBuilder =
+        LoadOptionsBuilder.builder().withSchemaInfoLevel(SchemaInfoLevelBuilder.standard());
+    SchemaCrawlerOptions options =
+        SchemaCrawlerOptionsBuilder.newSchemaCrawlerOptions()
+            .withLoadOptions(loadOptionsBuilder.toOptions());
+    OutputOptions outputOptions =
+        OutputOptionsBuilder.newOutputOptions(DiagramOutputFormat.htmlx, outputFile);
+
+    SchemaCrawlerExecutable executable = new SchemaCrawlerExecutable(command);
+    executable.setSchemaCrawlerOptions(options);
+    executable.setOutputOptions(outputOptions);
+    executable.setConnection(connection);
+    try {
+      executable.execute();
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+
+    return outputFile;
+  }
+
+  private static Connection getConnection(PostgreSQLContainer container) {
+    Properties info = new Properties();
+    info.put("user", container.getUsername());
+    info.put("password", container.getPassword());
+    try {
+      return container.getJdbcDriverInstance().connect(container.getJdbcUrl(), info);
+    } catch (SQLException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static void initDb(Connection connection) {
+    try (Statement statement = connection.createStatement()) {
+      statement.execute(
+          ResourceUtils.readResourceUtf8(Resources.getResource(NOMULUS_GOLDEN_SCHEMA)));
+    } catch (SQLException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @VisibleForTesting
+  static String getLastFlywayFileName() {
+    try {
+      return Iterables.getLast(
+          Resources.readLines(Resources.getResource(FLYWAY_FILE), StandardCharsets.UTF_8));
+    } catch (IOException e) {
+      throw new UncheckedIOException(e);
+    }
+  }
+}
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Michael MullerandGitHub	78bc967357	Replace ofyOrJpaTm with tm().isOfy() conditionals (#870 ) * Replace ofyOrJpaTm with tm().isOfy() conditionals Replace existing ofyOrJpaTm() calls with conditionals (either "if" statements or ternary expressions) gated on tm().isOfy().	2020-11-11 07:40:43 -05:00
gbrodmanandGitHub	88f4f36678	Maintain only one row in the TmchCrl table at all times (#869 ) * Maintain only one row in the TmchCrl table at all times	2020-11-10 17:09:26 -05:00
gbrodmanandGitHub	f1eb1a8fe7	Add SQL replay checkpoint object to SQL (#868 ) * Add SQL replay checkpoint object to Datastore This will be part of the asynchronous commit-log replay to SQL. Whenever we successfully export commits up to a particular time, we should persist that time so we don't replay the same commits again (it is not idempotent) * Move SqlReplayCheckpoint from DS to SQL * Responses to CR	2020-11-10 17:09:18 -05:00
Michael MullerandGitHub	82cc7b59fe	Small SQL persistence fixes to model classes (#863 ) * Small SQL persistence fixes to model classes - Add a createVKey() method to Registry (Registry vkeys are composite) - Add/fix toSqlEntities() methods in premium and reserved list classes. * Remove fixes addressed by #866	2020-11-10 11:12:09 -05:00
Michael MullerandGitHub	633377a594	Generate ER diagrams in :nom:generate_golden_file (#867 ) Generate entity relationship diagrams during the same step in which we generate the golden schema.	2020-11-10 10:03:28 -05:00
sarahcaseybotandGitHub	c729a30d38	Add ECDSA key validation to Certificate Checker (#855 ) * Add ecdsa key validation * Add some comments * fix merge conflicts * change variable names * Separate tests * separate curve tests	2020-11-09 15:28:48 -05:00
gbrodmanandGitHub	acce1a7d3b	Add DuallyWrittenEntity convenience interface (#866 ) * Add DuallyWrittenEntity convenience interface For classes that are dually written we don't need to worry about replay conversion, so this just allows us to remove unnecessary unimportant methods * Rename DWE to NonReplicatedEntity and enable test	2020-11-09 15:13:40 -05:00
gbrodmanandGitHub	0c6363c04f	Persist two singleton entities in SQL tables (#860 ) * Persist two singleton entities in SQL tables A table might not be the best place to store singleton entities, but by doing this we ensure we can easily inspect them later and use the same sort of persistence logic for these that we do elsewhere. ServerSecret is stored upon retrieval so that we make sure that the same secret is used in both Datastore and SQL (we wouldn't want to change it). * Responses to CR * Don't have a separate ID for the singleton entities * Rename secret UUID * Rename and regenerate	2020-11-09 13:47:42 -05:00
Shicong HuangandGitHub	cb764b5d30	Convert Registry realted tests to working with SQL (#862 )	2020-11-09 12:10:55 -05:00
Shicong HuangandGitHub	420f3bf380	Add missing foreign key constraints (#845 ) * Add missing foreign key constraints * Fix failed unit tests	2020-11-09 10:55:29 -05:00
Michael MullerandGitHub	7097b0f5e6	Allow InjectExtension to apply in "before each" (#864 ) * Allow InjectExtension to apply in "before each" InjectExtension is non-standard in that, while it cleans up the changes it has applied in its "after each" method, those changes must be applied by the test driver or by another extension. This breaks extension ordering, which is something we're going to need to make heavy use of for the ReplayExtension. Allow changes to be specified (but not activated) after construction and apply all unactivated changes during "before each."	2020-11-09 07:52:55 -05:00
Shicong HuangandGitHub	30d57d9476	Upgrade Postgres realted packages to latest version (#861 )	2020-11-06 10:14:44 -05:00
sarahcaseybotandGitHub	700d612ff9	Remove November 1 check (#859 )	2020-11-04 13:10:48 -05:00
Shicong HuangandGitHub	e3d400958c	Add a command to fix duplicate id issue for BillingEvent.Recurring (#798 ) * Refactor ResaveEntitiesWithUniqueIdCommand to extract common methods * Add a command to dedupe BillingEvent.Recurring entities	2020-11-03 16:11:56 -05:00
gbrodmanandGitHub	aa84d5d138	Fix entity annotations for a few classes (#856 ) * Fix entity annotations for a few classes - Abstract classes shouldn't implement DatastoreEntity/SqlEntity - We aren't persisting Modification in SQL - Because DelegationSignerData is embedded, we don't need to worry about converting and persisting it - DomainDsDataHistory isn't persisted in Datastore	2020-11-03 10:28:34 -05:00
gbrodmanandGitHub	d685f7e2df	Add a SQL schema and DAO for KmsSecretRevision (#840 ) * Add a SQL schema and DAO for KmsSecretRevision The dual-object nature of KmsSecret and KmsSecretRevision will not be necessary once we have moved to SQL. In that world, the only object will be the one now called KmsSecretRevision. KmsSecretRevision already stores its parent so all we need to do is convert that key to the String secretName (or from the secretName to the key, if loading from SQL) and select the max revision ID for a given secret name. In a future PR, we will add a dual-writing DAO to these objects and perform the dual writes, similar to how ReservedList functions. * Regenerate diagram * Rename revisionId and cryptoKeyVersionName * Fix SQL files and diagram	2020-10-30 18:45:43 -04:00
gbrodmanandGitHub	40eef2a06c	Add SQL schema and DAO for SignedMarkRevocationList (#850 ) * Add SQL schema and DAO for SignedMarkRevocationList This gets saved every day so we're not concerned about history, meaning we can dual-write and/or dual-read without concern. The structure here is somewhat similar to the ClaimsListDao and related classes. * Update the DB files	2020-10-30 17:52:09 -04:00
sarahcaseybotandGitHub	8bd5eb4eca	Move CertificateChecker to core/ (#852 ) * Move CertificateChecker to core/ * rename certificates/ to certs/	2020-10-30 15:57:12 -04:00
Weimin YuandGitHub	83918e92b5	Sync the live folder after Nomulus rollback (#854 ) * Sync the live folder after Nomulus rollback To update the nomulus tool on corp desktop, the artifacts from the rollback target release should be copied to the 'live' folder. * Fix a test	2020-10-29 16:21:56 -04:00
Shicong HuangandGitHub	5bba65835a	Change primary key of DelegationSignerData and add its history table (#841 ) * Change primary key of DelegationSignerData and add its history table * Change primary key and resolve comments * Rebase on HEAD	2020-10-29 16:19:15 -04:00
Shicong HuangandGitHub	1e51f51979	Use TransactionManager APIs in DatastoreHelper (#849 ) * Make DatastoreHelper support Postgresql * Rebase on HEAD * Resolve comments * Use put* inside insert* and update* * Resolve comments	2020-10-29 11:41:04 -04:00
Weimin YuandGitHub	db2e896d42	An automated rollback tool for Nomulus (#847 ) * An automated rollback tool for Nomulus A tool that directs traffic between deployed versions. It handles the conversion between Nomulus tags and AppEngine versions, executes schema compatibility tests, ensures that steps are executed in the correct order, and updates deployment records appropriately.	2020-10-29 10:37:20 -04:00
Michael MullerandGitHub	478064f32b	Add a test for ImmutableObject significant fields (#853 ) This should have been introduced in #846. Better late than never.	2020-10-28 11:11:42 -04:00
Shicong HuangandGitHub	0db535b838	Add ER diagram links to db/README.md (#851 )	2020-10-28 10:13:39 -04:00
Shicong HuangandGitHub	3705f37fab	Add a build task to upload ER diagrams to GCS (#844 ) * Add a build task to upload ER diagrams to GCS * Merge ER diagram task into cloudbuild-javadoc	2020-10-27 10:41:12 -04:00
Michael MullerandGitHub	86bdd154bc	Restore ofy keys in GracePeriod objects (#846 ) * Restore ofy keys in GracePeriod objects Restore the ofy keys when loading GracePeriod object from SQL. There's no clear way to do this using the normal approach (fix-up during a PostLoad method) because fixups to these violate immutability after hibernate has already obtained their hash values. Instead, we force reconstitution of the ofy keys in all public methods that access them (including equals() and hashCode()) so that they can be generated before an invalid hash is generated. As part of this change, convert the GracePeriod id from an autogenerated sequence to a UUID allocated from ObjectifyService and enhance ImmutableObject to allow it to exclude certain fields from hash/equals and print. The ImmutableObject enhancements are necessary because we compare grace periods against locally created test objects in a number of unit tests and there's no way this can work with GracePeriods loaded from SQL currently, as they will have an identifier field generated from the database and the test objects will have an identifier field of null (or a new unique value, after this change). Removing autogeneration from GracePeriod ids ended up being likely not strictly necessary for this change (it was a consequence of an earlier iteration). However, it does alleviate the problem of mutation of an immutable object after creation and is more in line with how we've decided to allocate other identifiers. * Changed needed after rebase.	2020-10-26 13:38:14 -04:00
sarahcaseybotandGitHub	576c05ff5f	Add certificate checks to RegistrarSettingsAction (#843 ) * Add certificate checks to RegistrarSettingsAction * Add some comments * Add more functionality to CertificateChecker and update call sites * Small code cleanups * Small format fix	2020-10-23 15:46:57 -04:00
gbrodmanandGitHub	f52e887db5	Create SQL schema for RdeRevision (#835 ) * Create SQL schema for RdeRevision * Split RdeRevision IDs into three separate DB fields as unified pkey * Rename variable * Merge remote-tracking branch 'origin/master' into rdeRevision * Rename variable in one other location * Implement no-op toDatastore/Sql for RdeRevision * Responses to CR * Merge remote-tracking branch 'origin/master' into rdeRevision * Use a date for the date column * Fix exception messages in tests * Regen diagram to fix the test * Use assignment in static factory methods * Merge remote-tracking branch 'origin/master' into rdeRevision	2020-10-23 13:14:07 -04:00
Weimin YuandGitHub	6ed286e3bc	Upgrade error-prone to 3.3.4 (#848 ) * Upgrade error-prone to 3.3.4 This would fix the failure with openjdk 11.0.9 in 3.3.3. Fixed new antipatterns raised by the new version: - Replaced unnecessary lambdas with methods. - Switched wait/sleep calls to equivalent methods using java.time types - Types inheriting Object.toString() should not be assigned to string parameter in logging statements.	2020-10-23 11:17:57 -04:00
sarahcaseybotandGitHub	93d922af6f	Add certificate checks for create and update registrar commands (#837 ) * Add certificatechecks for create and update registrar commands * Add CertificateCheckerModule * Remove commented out code * Still tring to get dependency injection to work * Get this actually working * Add tests for multiple violations * Small formatting fixes * Rename configs and fix collectors * Add checks for failover client certificate * Fix formatting	2020-10-22 11:43:22 -04:00
gbrodmanandGitHub	0b73e9032c	Use a SQL date object for LocalDates (#842 ) * Use a SQL date object for LocalDates * Clean up comment	2020-10-20 15:44:23 -04:00
Shicong HuangandGitHub	4d5d9700b8	Add a command to generate ER diagram for SQL schema (#839 ) * Add a command to generate ER diagram for SQL schema * Add graphviz as runtime dependency * Update ER diagrams for #838	2020-10-15 17:31:43 -04:00
Michael MullerandGitHub	3534a146e4	Restore ofy keys in DomainTransferData (#838 ) * Restore ofy keys in DomainTransferData Restore composite VKeys correctly in DomainTransferData (they were previously missing their ofy keys). * Use "AlsoLoad" to populate history ids	2020-10-15 07:54:47 -04:00
gbrodmanandGitHub	4ec7f23e84	Use the parent to store the history repo ID and fill in the base object (#830 ) * Use the parent to store the history repo ID and fill in the base object Storing the repo ID in the parent and in the base object has two primary benefits. First, it unifies the parent information in the HistoryEntry's `parent` object. This simplifies the builders and the data flow. Second, when possible (which should be always, post-migration) we fill out the DomainContent's repo ID (similarly for the other EPP resources) which means that when reconstituting the ofy keys we don't need to pass the repo ID in from a separate object. This way, all the data are encapsulated where they should be. The primary downside here is that it further reduces the "immutability" of the history objects (since we're using the Hibernate setter for the parent repo ID) but we weren't immutable anyway. * Respond to CR - compare the entire vkeys in tests - always return the parent for repo ID * Simplify creation of parent VKeys * Fix flipped isAssignableFrom check in VKey * Merge remote-tracking branch 'origin/master' into historyRepoId	2020-10-09 16:01:51 -04:00
Shicong Huang GitHub gbrodman	7a68b1b6f0	Revert package-lock.json to version from #676 (#834 ) Co-authored-by: gbrodman <gbrodman@google.com>	2020-10-09 15:57:23 -04:00
Shicong HuangandGitHub	14e593d9e1	Add SchemaCrawler as dependency (#833 )	2020-10-09 15:02:11 -04:00
Weimin YuandGitHub	2d5de96fbd	Minor python changes (#832 ) * Minor python changes Use dataclasses instead of attrs. The former is part of the standard lib while the latter may need to be installed separately. Also added python3 to the list of prerequisites.	2020-10-09 14:50:21 -04:00
Weimin YuandGitHub	13d30b0bfb	Maintain a release-to-Version map in deployment (#831 ) * Maintain a release-to-Version map in deployment Keep track of the mapping between Nomulus release tags and AppEngine version ids with a mapping file. This is necessary because AppEngine does not support custom versioning. With this mapping, rollbacks could be automated. Automation of rollbacks is important since there are test-supporting metadata to be updated, but are easily forgotten. During the last stage of deployment, current per-service version ids are fetched using gcloud and are appended to a file on GCS. Each line is of the format "{RELEASE_TAG},{APPENGINE_SERVICE},{APPENGINE_VERSION}. This change has been tested in crash. The rollback script is still a work in progress.	2020-10-09 13:32:52 -04:00
Shicong HuangandGitHub	b05f6b4ba3	Add SQL schema for DelegationSignerData (#713 ) * Add SQL schema for DelegationSignerData * Remove join table * Rebased on HEAD * Rebase on head	2020-10-09 10:22:31 -04:00
Shicong HuangandGitHub	17a1387184	Disable auto-generation on id for HostHistory and ContactHistory (#827 )	2020-10-08 12:30:54 -04:00
Ben McIlwainandGitHub	2e230664fd	Convert CertificateViolation into an enum (#829 ) * Convert CertificateViolation into an enum This ends up being nicer to deal with from callsites than class instances, while still permitting full configurability of all parameters. There are various other changes/fixes as well.	2020-10-07 22:19:36 -04:00
Michael MullerandGitHub	299b093f78	Correctly restore composite VKeys in DomainContent (#825 ) * Restore composite vkeys in DomainContent PollMessage/BillingEvent vkeys in DomainContent must have their ofy keys restored from other fields in DomainContent (namely the repo id and their specific history event ids). Add PostLoad methods to DomainContent and DomainHistory to do the restoration. * Fixes for review. * Deal with foreign-key cycles	2020-10-07 12:42:01 -04:00
Ben McIlwainandGitHub	61e7fa89f7	Fix incorrect repackaged App Engine import (#828 ) * Fix incorrect repackaged App Engine import	2020-10-07 11:52:17 -04:00
sarahcaseybotandGitHub	6ab69d4226	Add a CertificateChecker class (#793 ) * CertificateChecker with checks for expiration and key length * Add validity length check * Get rid of hard-coded constants and DSA checks * add files that for some reason weren't included in last commit * Rename violations and other fixes * Add displayMessage to CertificateViolation enum * Switch violations from an enum to a class * small changes * Get rid of ECDSA checks * add checks for old validity length * Change error message for validity length	2020-10-06 15:47:42 -04:00
gbrodmanandGitHub	0f09a4a0ab	Add more Datastore/Sql Entity annotations (#826 ) * Add more Datastore/Sql Entity annotations * Move comments up a line	2020-10-05 13:07:53 -04:00
Shicong HuangandGitHub	95f6ccc657	Fix vkey reconstruction for PollMessage (#823 ) * Fix vkey reconstruction for PollMessage * Add foreign key * Rebase on HEAD	2020-10-05 10:35:40 -04:00
Michael MullerandGitHub	77fabe4dc4	Move "WithLongVKey" to BillingEvent subclasses (#821 ) When loading the VKeys for the BillingEvents hierarchy, it is necessary to restore the original concrete class for the type, otherwise we end up with a different (and incompatible) VKey. As part of this, convert the cancellation matching billing event to VKey<Recurring>, which seems like the only thing it actually can be.	2020-10-02 15:20:23 -04:00
Lai JiangandGitHub	71fa12f773	Fix invoicing SQL (#824 )	2020-10-01 14:29:49 -04:00
Shicong HuangandGitHub	fd40a6a2b9	Use composite primary key for HostHistory and ContactHistory (#809 ) * Use composite primary key for HostHistory and ContactHistory * Update flyway file version * Make getters private * Add javadoc * Rebase on HEAD	2020-10-01 11:01:57 -04:00
Michael MullerandGitHub	71f86c9970	Add VKey.restoreOfy() method for fixing ofy keys (#820 ) Add a restoreOfy() instance method and a restoreOfyFrom() static method to assist in restoring the objectify key for classes that have composite keys that do not restore automatically.	2020-09-30 11:15:58 -04:00