Use the parent to store the history repo ID and fill in the base object (#830 )

* Use the parent to store the history repo ID and fill in the base object Storing the repo ID in the parent and in the base object has two primary benefits. First, it unifies the parent information in the HistoryEntry's `parent` object. This simplifies the builders and the data flow. Second, when possible (which should be always, post-migration) we fill out the DomainContent's repo ID (similarly for the other EPP resources) which means that when reconstituting the ofy keys we don't need to pass the repo ID in from a separate object. This way, all the data are encapsulated where they should be. The primary downside here is that it further reduces the "immutability" of the history objects (since we're using the Hibernate setter for the parent repo ID) but we weren't immutable anyway. * Respond to CR - compare the entire vkeys in tests - always return the parent for repo ID * Simplify creation of parent VKeys * Fix flipped isAssignableFrom check in VKey * Merge remote-tracking branch 'origin/master' into historyRepoId
Revert package-lock.json to version from #676 (#834 )
2026-06-09 16:33:02 +00:00 · 2020-10-09 16:01:51 -04:00 · 2020-10-09 15:57:23 -04:00 · 2020-10-09 15:02:11 -04:00 · 2020-10-09 14:50:21 -04:00 · 2020-10-09 13:32:52 -04:00
266 changed files with 5574 additions and 1569 deletions
@@ -191,7 +191,7 @@ allprojects {
 }

 task runPresubmits(type: Exec) {
-  executable '/usr/bin/python'
+  executable '/usr/bin/python3'
  args('config/presubmits.py')
 }

@@ -260,7 +260,7 @@ subprojects {
      // in the 'configurations' block, the following code must run after
      // project evaluation, when all configurations have been created.
      configurations.each {
-        if (it.name != 'dependencyLicenseReport') {
+        if (it.name != 'dependencyLicenseReport' && it.name != 'integration') {
          it.resolutionStrategy.activateDependencyLocking()
        }
      }
@@ -16,7 +16,7 @@
 """

 import argparse
-import attr
+import dataclasses
 import io
 import os
 import shutil
@@ -25,7 +25,7 @@ import sys
 from typing import List, Union


-@attr.s(auto_attribs=True)
+@dataclasses.dataclass
 class Property:
    name : str = ''
    desc : str = ''
@@ -39,7 +39,7 @@ class Property:
                raise ValidationError('value of {self.name} must be "true" or '
                                      '"false".')

-@attr.s(auto_attribs=True)
+@dataclasses.dataclass
 class GradleFlag:
    flags : Union[str, List[str]]
    desc : str
@@ -75,6 +75,7 @@ Pseudo-commands:
 """

 # Define all of our special gradle properties here.
+# TODO(b/169318491): use consistent naming style for properties and variables.
 PROPERTIES = [
    Property('mavenUrl',
             'URL to use for the main maven repository (defaults to maven '
@@ -124,6 +125,9 @@ PROPERTIES = [
             'server/schema integration tests. Please refer to <a '
             'href="./integration/README.md">integration project</a> for more '
             'information.'),
+    Property('baseSchemaTag',
+             'The nomulus version tag of the schema for use in the schema'
+             'deployment integration test (:db:schemaIncrementalDeployTest)'),
    Property('schema_version',
             'The nomulus version tag of the schema for use in a database'
             'integration test.'),
@@ -18,6 +18,7 @@ Error Prone) so we must write them manually.
 """

 import os
+from typing import List, Tuple
 import sys
 import re

@@ -178,6 +179,90 @@ PRESUBMITS = {
        "JavaScript files should not include console logging."
 }

+# Note that this regex only works for one kind of Flyway file.  If we want to
+# start using "R" and "U" files we'll need to update this script.
+FLYWAY_FILE_RX = re.compile(r'V(\d+)__.*')
+
+
+def get_seqnum(filename: str, location: str) -> int:
+  """Extracts the sequence number from a filename."""
+  m = FLYWAY_FILE_RX.match(filename)
+  if m is None:
+    raise ValueError('Illegal Flyway filename: %s in %s' % (filename, location))
+  return int(m.group(1))
+
+
+def files_by_seqnum(files: List[str], location: str) -> List[Tuple[int, str]]:
+  """Returns the list of seqnum, filename sorted by sequence number."""
+  return [(get_seqnum(filename, location), filename) for filename in files]
+
+
+def has_valid_order(indexed_files: List[Tuple[int, str]], location: str) -> bool:
+  """Verify that sequence numbers are in order without gaps or duplicates.
+
+  Args:
+    files: List of seqnum, filename for a list of Flyway files.
+    location: Where the list of files came from (for error reporting).
+
+  Returns:
+    True if the file list is valid.
+  """
+  last_index = 0
+  valid = True
+  for seqnum, filename in indexed_files:
+    if seqnum == last_index:
+      print('duplicate Flyway file sequence number found in %s: %s' %
+            (location, filename))
+      valid = False
+    elif seqnum < last_index:
+      print('File %s in %s is out of order.' % (filename, location))
+      valid = False
+    elif seqnum != last_index + 1:
+      print('Missing Flyway sequence number %d in %s.  Next file is %s' %
+            (last_index + 1, location, filename))
+      valid = False
+    last_index = seqnum
+  return valid
+
+
+def verify_flyway_index():
+  """Verifies that the Flyway index file is in sync with the directory."""
+  success = True
+
+  # Sort the files in the Flyway directory by their sequence number.
+  files = sorted(
+      files_by_seqnum(os.listdir('db/src/main/resources/sql/flyway'),
+                      'Flyway directory'))
+
+  # Make sure that there are no gaps and no duplicate sequence numbers in the
+  # files themselves.
+  if not has_valid_order(files, 'Flyway directory'):
+    success = False
+
+  # Remove the sequence numbers and compare against the index file contents.
+  files = [filename[1] for filename in sorted(files)]
+  with open('db/src/main/resources/sql/flyway.txt') as index:
+    indexed_files = index.read().splitlines()
+  if files != indexed_files:
+    unindexed = set(files) - set(indexed_files)
+    if unindexed:
+      print('The following Flyway files are not in flyway.txt: %s' % unindexed)
+
+    nonexistent = set(indexed_files) - set(files)
+    if nonexistent:
+      print('The following files are in flyway.txt but not in the Flyway '
+            'directory: %s' % nonexistent)
+
+    # Do an ordering check on the index file (ignore the result, we're failing
+    # anyway).
+    has_valid_order(files_by_seqnum(indexed_files, 'flyway.txt'), 'flyway.txt')
+    success = False
+
+  if not success:
+    print('Please fix any conflicts and run "./nom_build :db:generateFlywayIndex"')
+
+  return not success
+

 def get_files():
  for root, dirnames, filenames in os.walk("."):
@@ -197,5 +282,10 @@ if __name__ == "__main__":
      failed = True
      print("%s had errors: \n  %s" % (file, "\n  ".join(error_messages)))

+  # And now for something completely different: check to see if the Flyway
+  # index is up-to-date.  It's quicker to do it here than in the unit tests:
+  # when we put it here it fails fast before all of the tests are run.
+  failed |= verify_flyway_index()
+
  if failed:
    sys.exit(1)
@@ -63,8 +63,6 @@ def dockerIncompatibleTestPatterns = [
    // methods, so we exclude the whole test class.
    "google/registry/tools/params/PathParameterTest.*",
    "google/registry/persistence/PersistenceModuleTest.*",
-    // This test is failing in docker when using Java 11. The cause is unclear.
-    "google/registry/tools/DomainLockUtilsTest.*",
 ]

 // Tests that conflict with members of both the main test suite and the
@@ -247,6 +245,7 @@ dependencies {
  testCompile deps['org.apache.ftpserver:ftpserver-core']
  compile deps['org.apache.httpcomponents:httpclient']
  compile deps['org.apache.httpcomponents:httpcore']
+  runtime deps['org.apache.logging.log4j:log4j-core']
  testCompile deps['org.apache.sshd:sshd-core']
  testCompile deps['org.apache.sshd:sshd-scp']
  testCompile deps['org.apache.sshd:sshd-sftp']
@@ -271,6 +270,8 @@ dependencies {
  compile deps['org.testcontainers:postgresql']
  testCompile deps['org.testcontainers:selenium']
  testCompile deps['org.testcontainers:testcontainers']
+  compile deps['us.fatehi:schemacrawler']
+  compile deps['us.fatehi:schemacrawler-diagram']
  compile deps['xerces:xmlParserAPIs']
  compile deps['xpp3:xpp3']
  // This dependency must come after javax.mail:mail as it would otherwise
@@ -202,6 +202,7 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +250,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -200,6 +200,7 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -246,5 +247,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -201,8 +201,10 @@ org.apache.commons:commons-compress:1.20
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -251,5 +253,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -200,8 +200,10 @@ org.apache.commons:commons-compress:1.20
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -202,6 +202,7 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +250,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -201,6 +201,7 @@ org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
 org.apache.logging.log4j:log4j-api:2.6.2
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -248,5 +249,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -200,8 +200,10 @@ org.apache.commons:commons-compress:1.20
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -200,8 +200,10 @@ org.apache.commons:commons-compress:1.20
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -200,8 +200,10 @@ org.apache.commons:commons-compress:1.20
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -200,8 +200,10 @@ org.apache.commons:commons-compress:1.20
 org.apache.commons:commons-lang3:3.5
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.bouncycastle:bcpg-jdk15on:1.61
+org.bouncycastle:bcpkix-jdk15on:1.61
 org.bouncycastle:bcprov-jdk15on:1.61
 org.checkerframework:checker-compat-qual:2.5.5
 org.checkerframework:checker-qual:2.11.1
@@ -249,5 +251,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,9 +6,9 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
-com.fasterxml.jackson.core:jackson-annotations:2.10.2
-com.fasterxml.jackson.core:jackson-core:2.10.2
-com.fasterxml.jackson.core:jackson-databind:2.10.2
+com.fasterxml.jackson.core:jackson-annotations:2.11.2
+com.fasterxml.jackson.core:jackson-core:2.11.2
+com.fasterxml.jackson.core:jackson-databind:2.11.2
 com.fasterxml:classmate:1.5.1
 com.github.jnr:jffi:1.2.23
 com.github.jnr:jnr-a64asm:1.0.0
@@ -298,5 +298,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,9 +6,9 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
-com.fasterxml.jackson.core:jackson-annotations:2.10.2
-com.fasterxml.jackson.core:jackson-core:2.10.2
-com.fasterxml.jackson.core:jackson-databind:2.10.2
+com.fasterxml.jackson.core:jackson-annotations:2.11.2
+com.fasterxml.jackson.core:jackson-core:2.11.2
+com.fasterxml.jackson.core:jackson-databind:2.11.2
 com.fasterxml:classmate:1.5.1
 com.github.jnr:jffi:1.2.23
 com.github.jnr:jnr-a64asm:1.0.0
@@ -297,5 +297,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,9 +6,9 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
-com.fasterxml.jackson.core:jackson-annotations:2.10.2
-com.fasterxml.jackson.core:jackson-core:2.10.2
-com.fasterxml.jackson.core:jackson-databind:2.10.2
+com.fasterxml.jackson.core:jackson-annotations:2.11.2
+com.fasterxml.jackson.core:jackson-core:2.11.2
+com.fasterxml.jackson.core:jackson-databind:2.11.2
 com.fasterxml:classmate:1.5.1
 com.github.jnr:jffi:1.2.23
 com.github.jnr:jnr-a64asm:1.0.0
@@ -216,7 +216,8 @@ org.apache.ftpserver:ftplet-api:1.0.6
 org.apache.ftpserver:ftpserver-core:1.0.6
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.apache.mina:mina-core:2.0.4
 org.apache.sshd:sshd-core:2.0.0
 org.apache.sshd:sshd-scp:2.0.0
@@ -301,5 +302,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -6,9 +6,9 @@ aopalliance:aopalliance:1.0
 args4j:args4j:2.33
 cglib:cglib-nodep:2.2
 com.beust:jcommander:1.60
-com.fasterxml.jackson.core:jackson-annotations:2.10.2
-com.fasterxml.jackson.core:jackson-core:2.10.2
-com.fasterxml.jackson.core:jackson-databind:2.10.2
+com.fasterxml.jackson.core:jackson-annotations:2.11.2
+com.fasterxml.jackson.core:jackson-core:2.11.2
+com.fasterxml.jackson.core:jackson-databind:2.11.2
 com.fasterxml:classmate:1.5.1
 com.github.jnr:jffi:1.2.23
 com.github.jnr:jnr-a64asm:1.0.0
@@ -216,7 +216,8 @@ org.apache.ftpserver:ftplet-api:1.0.6
 org.apache.ftpserver:ftpserver-core:1.0.6
 org.apache.httpcomponents:httpclient:4.5.11
 org.apache.httpcomponents:httpcore:4.4.13
-org.apache.logging.log4j:log4j-api:2.6.2
+org.apache.logging.log4j:log4j-api:2.13.3
+org.apache.logging.log4j:log4j-core:2.13.3
 org.apache.mina:mina-core:2.0.4
 org.apache.sshd:sshd-core:2.0.0
 org.apache.sshd:sshd-scp:2.0.0
@@ -302,5 +303,10 @@ org.tukaani:xz:1.8
 org.w3c.css:sac:1.3
 org.xerial.snappy:snappy-java:1.1.4
 org.yaml:snakeyaml:1.17
+us.fatehi:schemacrawler-api:16.10.1
+us.fatehi:schemacrawler-diagram:16.10.1
+us.fatehi:schemacrawler-tools:16.10.1
+us.fatehi:schemacrawler-utility:16.10.1
+us.fatehi:schemacrawler:16.10.1
 xerces:xmlParserAPIs:2.6.2
 xpp3:xpp3:1.1.4c
@@ -169,16 +169,20 @@ public final class AsyncTaskEnqueuer {
        lock.getRelockDuration().isPresent(),
        "Lock with ID %s not configured for relock",
        lock.getRevisionId());
+    enqueueDomainRelock(lock.getRelockDuration().get(), lock.getRevisionId(), 0);
+  }
+
+  /** Enqueues a task to asynchronously re-lock a registry-locked domain after it was unlocked. */
+  void enqueueDomainRelock(Duration countdown, long lockRevisionId, int previousAttempts) {
    String backendHostname = appEngineServiceUtils.getServiceHostname("backend");
    addTaskToQueueWithRetry(
        asyncActionsPushQueue,
        TaskOptions.Builder.withUrl(RelockDomainAction.PATH)
            .method(Method.POST)
            .header("Host", backendHostname)
-            .param(
-                RelockDomainAction.OLD_UNLOCK_REVISION_ID_PARAM,
-                String.valueOf(lock.getRevisionId()))
-            .countdownMillis(lock.getRelockDuration().get().getMillis()));
+            .param(RelockDomainAction.OLD_UNLOCK_REVISION_ID_PARAM, String.valueOf(lockRevisionId))
+            .param(RelockDomainAction.PREVIOUS_ATTEMPTS_PARAM, String.valueOf(previousAttempts))
+            .countdownMillis(countdown.getMillis()));
  }

  /**
@@ -21,6 +21,7 @@ import static google.registry.batch.AsyncTaskEnqueuer.PARAM_RESOURCE_KEY;
 import static google.registry.batch.AsyncTaskEnqueuer.QUEUE_ASYNC_ACTIONS;
 import static google.registry.batch.AsyncTaskEnqueuer.QUEUE_ASYNC_DELETE;
 import static google.registry.batch.AsyncTaskEnqueuer.QUEUE_ASYNC_HOST_RENAME;
+import static google.registry.request.RequestParameters.extractIntParameter;
 import static google.registry.request.RequestParameters.extractLongParameter;
 import static google.registry.request.RequestParameters.extractOptionalBooleanParameter;
 import static google.registry.request.RequestParameters.extractOptionalIntParameter;
@@ -94,9 +95,15 @@ public class BatchModule {
  }

  @Provides
-  @Parameter("oldUnlockRevisionId")
+  @Parameter(RelockDomainAction.OLD_UNLOCK_REVISION_ID_PARAM)
  static long provideOldUnlockRevisionId(HttpServletRequest req) {
-    return extractLongParameter(req, "oldUnlockRevisionId");
+    return extractLongParameter(req, RelockDomainAction.OLD_UNLOCK_REVISION_ID_PARAM);
+  }
+
+  @Provides
+  @Parameter(RelockDomainAction.PREVIOUS_ATTEMPTS_PARAM)
+  static int providePreviousAttempts(HttpServletRequest req) {
+    return extractIntParameter(req, RelockDomainAction.PREVIOUS_ATTEMPTS_PARAM);
  }

  @Provides
@@ -464,7 +464,7 @@ public class DeleteContactsAndHostsAction implements Runnable {
        HostResource host = (HostResource) existingResource;
        if (host.isSubordinate()) {
          dnsQueue.addHostRefreshTask(host.getHostName());
-          tm().saveNewOrUpdate(
+          tm().put(
                  tm().load(host.getSuperordinateDomain())
                      .asBuilder()
                      .removeSubordinateHost(host.getHostName())
@@ -15,19 +15,23 @@
 package google.registry.batch;

 import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.collect.ImmutableSet.toImmutableSet;
 import static google.registry.model.ofy.ObjectifyService.ofy;
 import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
 import static google.registry.request.Action.Method.POST;
 import static google.registry.tools.LockOrUnlockDomainCommand.REGISTRY_LOCK_STATUSES;
-import static javax.servlet.http.HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
 import static javax.servlet.http.HttpServletResponse.SC_NO_CONTENT;
 import static javax.servlet.http.HttpServletResponse.SC_OK;

 import com.google.common.collect.ImmutableSet;
 import com.google.common.flogger.FluentLogger;
 import com.google.common.net.MediaType;
+import google.registry.config.RegistryConfig.Config;
 import google.registry.model.domain.DomainBase;
 import google.registry.model.eppcommon.StatusValue;
+import google.registry.model.registrar.Registrar;
+import google.registry.model.registrar.RegistrarContact;
 import google.registry.model.registry.RegistryLockDao;
 import google.registry.request.Action;
 import google.registry.request.Parameter;
@@ -36,11 +40,15 @@ import google.registry.request.auth.Auth;
 import google.registry.schema.domain.RegistryLock;
 import google.registry.tools.DomainLockUtils;
 import google.registry.util.DateTimeUtils;
+import google.registry.util.EmailMessage;
+import google.registry.util.SendEmailService;
+import java.util.Optional;
 import javax.inject.Inject;
+import javax.mail.internet.AddressException;
+import javax.mail.internet.InternetAddress;
+import org.joda.time.Duration;

-/**
- * Task that relocks a previously-Registry-Locked domain after some predetermined period of time.
- */
+/** Task that re-locks a previously-Registry-Locked domain after a predetermined period of time. */
@Action(
    service = Action.Service.BACKEND,
    path = RelockDomainAction.PATH,
@@ -51,30 +59,78 @@ public class RelockDomainAction implements Runnable {

  public static final String PATH = "/_dr/task/relockDomain";
  public static final String OLD_UNLOCK_REVISION_ID_PARAM = "oldUnlockRevisionId";
+  public static final String PREVIOUS_ATTEMPTS_PARAM = "previousAttempts";
+
+  static final int ATTEMPTS_BEFORE_SLOWDOWN = 36; // every ten minutes for six hours then every hour
+  static final int FAILURES_BEFORE_EMAIL = 2; // email after three failures, one half hour
+  private static final Duration TEN_MINUTES = Duration.standardMinutes(10);
+  private static final Duration ONE_HOUR = Duration.standardHours(1);
+
+  private static final String RELOCK_SUCCESS_EMAIL_TEMPLATE =
+      "The domain %s was successfully re-locked.\n\nPlease contact support at %s if you have any "
+          + "questions.";
+  private static final String RELOCK_NON_RETRYABLE_FAILURE_EMAIL_TEMPLATE =
+      "There was an error when automatically re-locking %s. Error message: %s\n\nPlease contact "
+          + "support at %s if you have any questions.";
+  private static final String RELOCK_TRANSIENT_FAILURE_EMAIL_TEMPLATE =
+      "There was an unexpected error when automatically re-locking %s. We will continue retrying "
+          + "the lock for five hours. Please contact support at %s if you have any questions";
+  private static final String RELOCK_UNKNOWN_ID_FAILURE_EMAIL_TEMPLATE =
+      "The old lock with revision ID %d is not present or is not accessible";

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

  private final long oldUnlockRevisionId;
+  private final int previousAttempts;
+  private final InternetAddress alertRecipientAddress;
+  private final InternetAddress gSuiteOutgoingEmailAddress;
+  private final String supportEmail;
+  private final SendEmailService sendEmailService;
  private final DomainLockUtils domainLockUtils;
  private final Response response;
+  private final AsyncTaskEnqueuer asyncTaskEnqueuer;

  @Inject
  public RelockDomainAction(
      @Parameter(OLD_UNLOCK_REVISION_ID_PARAM) long oldUnlockRevisionId,
+      @Parameter(PREVIOUS_ATTEMPTS_PARAM) int previousAttempts,
+      @Config("alertRecipientEmailAddress") InternetAddress alertRecipientAddress,
+      @Config("gSuiteOutgoingEmailAddress") InternetAddress gSuiteOutgoingEmailAddress,
+      @Config("supportEmail") String supportEmail,
+      SendEmailService sendEmailService,
      DomainLockUtils domainLockUtils,
-      Response response) {
+      Response response,
+      AsyncTaskEnqueuer asyncTaskEnqueuer) {
    this.oldUnlockRevisionId = oldUnlockRevisionId;
+    this.previousAttempts = previousAttempts;
+    this.alertRecipientAddress = alertRecipientAddress;
+    this.gSuiteOutgoingEmailAddress = gSuiteOutgoingEmailAddress;
+    this.supportEmail = supportEmail;
+    this.sendEmailService = sendEmailService;
    this.domainLockUtils = domainLockUtils;
    this.response = response;
+    this.asyncTaskEnqueuer = asyncTaskEnqueuer;
  }

  @Override
  public void run() {
-    jpaTm().transact(this::relockDomain);
+    /* We wish to manually control our retry behavior, in order to limit the number of retries
+     * and/or notify registrars / support only after a certain number of retries, or only
+     * with a certain type of failure. AppEngine will automatically retry on any non-2xx status
+     * code, so return SC_NO_CONTENT (204) by default to avoid this auto-retry.
+     *
+     * See https://cloud.google.com/appengine/docs/standard/java/taskqueue/push/retrying-tasks
+     * for more details on retry behavior. */
+    response.setStatus(SC_NO_CONTENT);
+    response.setContentType(MediaType.PLAIN_TEXT_UTF_8);
+
+    // nb: DomainLockUtils relies on the JPA transaction being the outermost transaction
+    jpaTm().transact(() -> tm().transact(this::relockDomain));
  }

  private void relockDomain() {
-    RegistryLock oldLock;
+    RegistryLock oldLock = null;
+    DomainBase domain;
    try {
      oldLock =
          RegistryLockDao.getByRevisionId(oldUnlockRevisionId)
@@ -82,87 +138,187 @@ public class RelockDomainAction implements Runnable {
                  () ->
                      new IllegalArgumentException(
                          String.format("Unknown revision ID %d", oldUnlockRevisionId)));
-      DomainBase domain =
+      domain =
          ofy()
              .load()
              .type(DomainBase.class)
              .id(oldLock.getRepoId())
              .now()
              .cloneProjectedAtTime(jpaTm().getTransactionTime());
-
-      if (domain.getStatusValues().containsAll(REGISTRY_LOCK_STATUSES)
-          || oldLock.getRelock() != null) {
-        // The domain was manually locked, so we shouldn't worry about relocking
-        String message =
-            String.format(
-                "Domain %s is already manually relocked, skipping automated relock.",
-                domain.getDomainName());
-        logger.atInfo().log(message);
-        // SC_NO_CONTENT (204) skips retry -- see the comment below
-        response.setStatus(SC_NO_CONTENT);
-        response.setContentType(MediaType.PLAIN_TEXT_UTF_8);
-        response.setPayload(message);
-        return;
-      }
-      verifyDomainAndLockState(oldLock, domain);
    } catch (Throwable t) {
-      /* If there's a bad verification code or the domain is in a bad state, we won't want to retry.
-       * AppEngine will retry on non-2xx error codes, so we return SC_NO_CONTENT (204) to avoid it.
-       *
-       * See https://cloud.google.com/appengine/docs/standard/java/taskqueue/push/retrying-tasks
-       * for more details on retry behavior. */
-      logger.atWarning().withCause(t).log(
-          "Exception when attempting to relock domain with old revision ID %d.",
-          oldUnlockRevisionId);
-      response.setStatus(SC_NO_CONTENT);
-      response.setContentType(MediaType.PLAIN_TEXT_UTF_8);
-      response.setPayload(String.format("Relock failed: %s", t.getMessage()));
+      handleTransientFailure(Optional.ofNullable(oldLock), t);
      return;
    }
-    applyRelock(oldLock);
+
+    if (domain.getStatusValues().containsAll(REGISTRY_LOCK_STATUSES)
+        || oldLock.getRelock() != null) {
+      // The domain was manually locked, so we shouldn't worry about re-locking
+      String message =
+          String.format(
+              "Domain %s is already manually re-locked, skipping automated re-lock.",
+              domain.getDomainName());
+      logger.atInfo().log(message);
+      response.setPayload(message);
+      return;
+    }
+    try {
+      verifyDomainAndLockState(oldLock, domain);
+    } catch (Throwable t) {
+      // If the domain was, for example, transferred, then notify the old registrar and don't retry.
+      handleNonRetryableFailure(oldLock, t);
+      return;
+    }
+    try {
+      applyRelock(oldLock);
+    } catch (Throwable t) {
+      handleTransientFailure(Optional.of(oldLock), t);
+    }
  }

  private void applyRelock(RegistryLock oldLock) {
-    try {
-      domainLockUtils.administrativelyApplyLock(
-          oldLock.getDomainName(),
-          oldLock.getRegistrarId(),
-          oldLock.getRegistrarPocId(),
-          oldLock.isSuperuser());
-      logger.atInfo().log("Relocked domain %s.", oldLock.getDomainName());
-      response.setStatus(SC_OK);
-    } catch (Throwable t) {
-      // Any errors that occur here are unexpected, so we should retry. Return a non-2xx
-      // error code to get AppEngine to retry
-      logger.atSevere().withCause(t).log(
-          "Exception when attempting to relock domain %s.", oldLock.getDomainName());
-      response.setStatus(SC_INTERNAL_SERVER_ERROR);
-      response.setContentType(MediaType.PLAIN_TEXT_UTF_8);
-      response.setPayload(String.format("Relock failed: %s", t.getMessage()));
+    domainLockUtils.administrativelyApplyLock(
+        oldLock.getDomainName(),
+        oldLock.getRegistrarId(),
+        oldLock.getRegistrarPocId(),
+        oldLock.isSuperuser());
+    logger.atInfo().log("Re-locked domain %s.", oldLock.getDomainName());
+    response.setStatus(SC_OK);
+    // Only send a success email if we previously sent a failure email
+    if (previousAttempts > FAILURES_BEFORE_EMAIL) {
+      sendSuccessEmail(oldLock);
    }
  }

  private void verifyDomainAndLockState(RegistryLock oldLock, DomainBase domain) {
    // Domain shouldn't be deleted or have a pending transfer/delete
    String domainName = domain.getDomainName();
-    checkArgument(
-        !DateTimeUtils.isAtOrAfter(jpaTm().getTransactionTime(), domain.getDeletionTime()),
-        "Domain %s has been deleted",
-        domainName);
    ImmutableSet<StatusValue> statusValues = domain.getStatusValues();
    checkArgument(
        !statusValues.contains(StatusValue.PENDING_DELETE),
-        "Domain %s has a pending delete",
+        "Domain %s has a pending delete.",
+        domainName);
+    checkArgument(
+        !DateTimeUtils.isAtOrAfter(jpaTm().getTransactionTime(), domain.getDeletionTime()),
+        "Domain %s has been deleted.",
        domainName);
    checkArgument(
        !statusValues.contains(StatusValue.PENDING_TRANSFER),
-        "Domain %s has a pending transfer",
+        "Domain %s has a pending transfer.",
        domainName);
    checkArgument(
        domain.getCurrentSponsorClientId().equals(oldLock.getRegistrarId()),
-        "Domain %s has been transferred from registrar %s to registrar %s since the unlock",
+        "Domain %s has been transferred from registrar %s to registrar %s since the unlock.",
        domainName,
        oldLock.getRegistrarId(),
        domain.getCurrentSponsorClientId());
  }
+
+  private void handleNonRetryableFailure(RegistryLock oldLock, Throwable t) {
+    logger.atWarning().withCause(t).log(
+        "Exception thrown when attempting to re-lock domain with old revision ID %d.",
+        oldUnlockRevisionId);
+    response.setPayload(String.format("Re-lock failed: %s", t.getMessage()));
+
+    String body =
+        String.format(
+            RELOCK_NON_RETRYABLE_FAILURE_EMAIL_TEMPLATE,
+            oldLock.getDomainName(),
+            t.getMessage(),
+            supportEmail);
+    sendEmailService.sendEmail(
+        EmailMessage.newBuilder()
+            .setFrom(gSuiteOutgoingEmailAddress)
+            .setBody(body)
+            .setSubject(String.format("Error re-locking domain %s", oldLock.getDomainName()))
+            .setRecipients(getEmailRecipients(oldLock.getRegistrarId()))
+            .build());
+  }
+
+  private void handleTransientFailure(Optional<RegistryLock> oldLock, Throwable t) {
+    String message = String.format("Re-lock failed: %s", t.getMessage());
+    logger.atSevere().withCause(t).log(message);
+    response.setPayload(message);
+
+    if (previousAttempts == FAILURES_BEFORE_EMAIL) {
+      if (oldLock.isPresent()) {
+        sendGenericTransientFailureEmail(oldLock.get());
+      } else {
+        // if the old lock isn't present, something has gone horribly wrong
+        sendUnknownRevisionIdAlertEmail();
+      }
+    }
+    Duration timeBeforeRetry = previousAttempts < ATTEMPTS_BEFORE_SLOWDOWN ? TEN_MINUTES : ONE_HOUR;
+    asyncTaskEnqueuer.enqueueDomainRelock(
+        timeBeforeRetry, oldUnlockRevisionId, previousAttempts + 1);
+  }
+
+  private void sendSuccessEmail(RegistryLock oldLock) {
+    String body =
+        String.format(RELOCK_SUCCESS_EMAIL_TEMPLATE, oldLock.getDomainName(), supportEmail);
+
+    sendEmailService.sendEmail(
+        EmailMessage.newBuilder()
+            .setFrom(gSuiteOutgoingEmailAddress)
+            .setBody(body)
+            .setSubject(String.format("Successful re-lock of domain %s", oldLock.getDomainName()))
+            .setRecipients(getEmailRecipients(oldLock.getRegistrarId()))
+            .build());
+  }
+
+  private void sendGenericTransientFailureEmail(RegistryLock oldLock) {
+    String body =
+        String.format(
+            RELOCK_TRANSIENT_FAILURE_EMAIL_TEMPLATE, oldLock.getDomainName(), supportEmail);
+    // For an unexpected failure, notify both the lock-enabled contacts and our alerting email
+    ImmutableSet<InternetAddress> allRecipients =
+        new ImmutableSet.Builder<InternetAddress>()
+            .addAll(getEmailRecipients(oldLock.getRegistrarId()))
+            .add(alertRecipientAddress)
+            .build();
+    sendEmailService.sendEmail(
+        EmailMessage.newBuilder()
+            .setFrom(gSuiteOutgoingEmailAddress)
+            .setBody(body)
+            .setSubject(String.format("Error re-locking domain %s", oldLock.getDomainName()))
+            .setRecipients(allRecipients)
+            .build());
+  }
+
+  private void sendUnknownRevisionIdAlertEmail() {
+    sendEmailService.sendEmail(
+        EmailMessage.newBuilder()
+            .setFrom(gSuiteOutgoingEmailAddress)
+            .setBody(String.format(RELOCK_UNKNOWN_ID_FAILURE_EMAIL_TEMPLATE, oldUnlockRevisionId))
+            .setSubject("Error re-locking domain")
+            .setRecipients(ImmutableSet.of(alertRecipientAddress))
+            .build());
+  }
+
+  private ImmutableSet<InternetAddress> getEmailRecipients(String registrarId) {
+    Registrar registrar =
+        Registrar.loadByClientIdCached(registrarId)
+            .orElseThrow(
+                () ->
+                    new IllegalStateException(String.format("Unknown registrar %s", registrarId)));
+
+    ImmutableSet<String> registryLockEmailAddresses =
+        registrar.getContacts().stream()
+            .filter(RegistrarContact::isRegistryLockAllowed)
+            .map(RegistrarContact::getRegistryLockEmailAddress)
+            .filter(Optional::isPresent)
+            .map(Optional::get)
+            .collect(toImmutableSet());
+
+    ImmutableSet.Builder<InternetAddress> builder = new ImmutableSet.Builder<>();
+    // can't use streams due to the 'throws' in the InternetAddress constructor
+    for (String registryLockEmailAddress : registryLockEmailAddresses) {
+      try {
+        builder.add(new InternetAddress(registryLockEmailAddress));
+      } catch (AddressException e) {
+        // This shouldn't stop any other emails going out, so swallow it
+        logger.atWarning().log("Invalid email address %s", registryLockEmailAddress);
+      }
+    }
+    return builder.build();
+  }
 }
@@ -437,7 +437,7 @@ public final class Transforms {
                .map(Optional::get)
                .map(ofy::toPojo)
                .collect(ImmutableList.toImmutableList());
-        retry(() -> jpaTm().transact(() -> jpaTm().saveNewOrUpdateAll(ofyEntities)));
+        retry(() -> jpaTm().transact(() -> jpaTm().putAll(ofyEntities)));
      }
    }

@@ -55,9 +55,7 @@ FROM (
    FROM
      `%PROJECT_ID%.%DATASTORE_EXPORT_DATA_SET%.%REGISTRY_TABLE%`
    WHERE
-    -- TODO(b/18092292): Add a filter for tldState (not PDT/PREDELEGATION)
-      tldType = 'REAL'
-    AND disableInvoicing is not TRUE) ) AS BillingEvent
+      invoicingEnabled IS TRUE) ) AS BillingEvent
  -- Gather billing ID from registrar table
  -- This is a 'JOIN' as opposed to 'LEFT JOIN' to filter out
  -- non-billable registrars
@@ -215,7 +215,7 @@ public class Spec11Pipeline implements Serializable {
                          .setRegistrarId(subdomain.registrarId())
                          .build();
                  JpaTransactionManager jpaTransactionManager = jpaSupplierFactory.get();
-                  jpaTransactionManager.transact(() -> jpaTransactionManager.saveNew(threatMatch));
+                  jpaTransactionManager.transact(() -> jpaTransactionManager.insert(threatMatch));
                }
              }
            }));
@@ -173,7 +173,7 @@ public final class DomainRestoreRequestFlow implements TransactionalFlow  {
            existingDomain, newExpirationTime, autorenewEvent, autorenewPollMessage, now, clientId);
    updateForeignKeyIndexDeletionTime(newDomain);
    entitiesToSave.add(newDomain, historyEntry, autorenewEvent, autorenewPollMessage);
-    tm().saveNewOrUpdateAll(entitiesToSave.build());
+    tm().putAll(entitiesToSave.build());
    tm().delete(existingDomain.getDeletePollMessage());
    dnsQueue.addDomainRefreshTask(existingDomain.getDomainName());
    return responseBuilder
@@ -67,6 +67,7 @@ import google.registry.model.domain.DomainCommand.Update.AddRemove;
 import google.registry.model.domain.DomainCommand.Update.Change;
 import google.registry.model.domain.fee.FeeUpdateCommandExtension;
 import google.registry.model.domain.metadata.MetadataExtension;
+import google.registry.model.domain.secdns.DelegationSignerData;
 import google.registry.model.domain.secdns.SecDnsUpdateExtension;
 import google.registry.model.domain.superuser.DomainUpdateSuperuserExtension;
 import google.registry.model.eppcommon.AuthInfo;
@@ -238,10 +239,16 @@ public final class DomainUpdateFlow implements TransactionalFlow {
    DomainBase.Builder domainBuilder =
        domain
            .asBuilder()
-            // Handle the secDNS extension.
+            // Handle the secDNS extension. As dsData in secDnsUpdate is read from EPP input and
+            // does not have domainRepoId set, we create a copy of the existing dsData without
+            // domainRepoId for comparison.
            .setDsData(
                secDnsUpdate.isPresent()
-                    ? updateDsData(domain.getDsData(), secDnsUpdate.get())
+                    ? updateDsData(
+                        domain.getDsData().stream()
+                            .map(DelegationSignerData::cloneWithoutDomainRepoId)
+                            .collect(toImmutableSet()),
+                        secDnsUpdate.get())
                    : domain.getDsData())
            .setLastEppUpdateTime(now)
            .setLastEppUpdateClientId(clientId)
@@ -285,7 +285,7 @@ public final class HostUpdateFlow implements TransactionalFlow {
        && newHost.isSubordinate()
        && Objects.equals(
            existingHost.getSuperordinateDomain(), newHost.getSuperordinateDomain())) {
-      tm().saveNewOrUpdate(
+      tm().put(
              tm().load(existingHost.getSuperordinateDomain())
                  .asBuilder()
                  .removeSubordinateHost(existingHost.getHostName())
@@ -294,14 +294,14 @@ public final class HostUpdateFlow implements TransactionalFlow {
      return;
    }
    if (existingHost.isSubordinate()) {
-      tm().saveNewOrUpdate(
+      tm().put(
              tm().load(existingHost.getSuperordinateDomain())
                  .asBuilder()
                  .removeSubordinateHost(existingHost.getHostName())
                  .build());
    }
    if (newHost.isSubordinate()) {
-      tm().saveNewOrUpdate(
+      tm().put(
              tm().load(newHost.getSuperordinateDomain())
                  .asBuilder()
                  .addSubordinateHost(newHost.getHostName())
@@ -64,7 +64,7 @@ public final class PollFlowUtils {
      // and re-save it for future autorenew poll messages to be delivered. Otherwise, this
      // autorenew poll message has no more events to deliver and should be deleted.
      if (nextEventTime.isBefore(autorenewPollMessage.getAutorenewEndTime())) {
-        tm().saveNewOrUpdate(autorenewPollMessage.asBuilder().setEventTime(nextEventTime).build());
+        tm().put(autorenewPollMessage.asBuilder().setEventTime(nextEventTime).build());
        includeAckedMessageInCount = isBeforeOrAt(nextEventTime, tm().getTransactionTime());
      } else {
        tm().delete(autorenewPollMessage.createVKey());
@@ -2,7 +2,7 @@
 # Script: Arab
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Armn
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Beng
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -1,24 +1,24 @@
 # Registry: Charleston Road Registry Inc.
-# Script: Chinese
+# Script: zh-Hans
 # Version: 1.0
 # Effective Date: 04-12-2012
 #
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
 # Notes: This table describes codepoints allowed for the Chinese script.

-Reference 0 Unicode 3.2
-Reference 1 A Complete Set of Simplified Chinese Characters
-Reference 2 Chinese Variants Collation Table
-Reference 3 Chinese Big Dictionary
-Reference 4 Chinese Relationship Table for Unihan Project
-Reference 5 GB2312
-Reference 6 General Table for Modern Chinese
-Reference 7 International Chinese Standard Big Dictionary
-Reference 8 Unihan Database
-Reference 9 BIG5
+# Reference 0 Unicode 3.2
+# Reference 1 A Complete Set of Simplified Chinese Characters
+# Reference 2 Chinese Variants Collation Table
+# Reference 3 Chinese Big Dictionary
+# Reference 4 Chinese Relationship Table for Unihan Project
+# Reference 5 GB2312
+# Reference 6 General Table for Modern Chinese
+# Reference 7 International Chinese Standard Big Dictionary
+# Reference 8 Unihan Database
+# Reference 9 BIG5

 U+002D(0);U+002D(0);
 U+0030(0);U+0030(0);
@@ -2,7 +2,7 @@
 # Script: Cyrl
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Deva
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Ethi
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -3,7 +3,7 @@
 #
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -3,7 +3,7 @@
 # Script: Grek
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -3,7 +3,7 @@
 # Script: Guru
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Hebr
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Knda
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Khmr
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 #
 # Telephone: +1 (650) 253-0000
@@ -2,7 +2,7 @@
 # Script: Kore
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Mlym
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -3,7 +3,7 @@
 # Version: 1.0
 # Effective Date: 04-12-2012
 #
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Orya
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Sinh
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Taml
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Telu
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Thai
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -2,7 +2,7 @@
 # Script: Tibt
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
@@ -1,23 +1,23 @@
 # Registry: Charleston Road Registry Inc.
-# Script: Traditional Chinese
+# Script: zh-Hant
 # Version: 1.0
 # Effective Date: 04-12-2012
-# Contact: tas-contact.google.com
+# Contact: iana-contact@google.com
 # Address: 1600 Amphitheatre Parkway Mountain View, CA 94043, USA
 # Telephone: +1 (650) 253-0000
 # Website: www.google.com
 # Notes: This table describes codepoints allowed for the Traditional Chinese script.

-Reference 0 Unicode 3.2
-Reference 1 A Complete Set of Simplified Chinese Characters
-Reference 2 Chinese Variants Collation Table
-Reference 3 Chinese Big Dictionary
-Reference 4 Chinese Relationship Table for Unihan Project
-Reference 5 GB2312
-Reference 6 General Table for Modern Chinese
-Reference 7 International Chinese Standard Big Dictionary
-Reference 8 Unihan Database
-Reference 9 BIG5
+# Reference 0 Unicode 3.2
+# Reference 1 A Complete Set of Simplified Chinese Characters
+# Reference 2 Chinese Variants Collation Table
+# Reference 3 Chinese Big Dictionary
+# Reference 4 Chinese Relationship Table for Unihan Project
+# Reference 5 GB2312
+# Reference 6 General Table for Modern Chinese
+# Reference 7 International Chinese Standard Big Dictionary
+# Reference 8 Unihan Database
+# Reference 9 BIG5

 U+002D(0);U+002D(0);
 U+0030(0);U+0030(0);
@@ -62,23 +62,32 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
  /**
   * Unique identifier in the registry for this resource.
   *
+   * <p>Not persisted so that we can store these in references to other objects. Subclasses that
+   * wish to use this as the primary key should create a getter method annotated with @Id
+   *
   * <p>This is in the (\w|_){1,80}-\w{1,8} format specified by RFC 5730 for roidType.
   *
   * @see <a href="https://tools.ietf.org/html/rfc5730">RFC 5730</a>
   */
-  @Id
-  // not persisted so that we can store these in references to other objects. Subclasses that wish
-  // to use this as the primary key should create a getter method annotated with @Id
-  @Transient
-  String repoId;
+  @Id @Transient String repoId;

-  /** The ID of the registrar that is currently sponsoring this resource. */
+  /**
+   * The ID of the registrar that is currently sponsoring this resource.
+   *
+   * <p>This can be null in the case of pre-Registry-3.0-migration history objects with null
+   * resource fields.
+   */
  @Index
-  @Column(name = "currentSponsorRegistrarId", nullable = false)
+  @Column(name = "currentSponsorRegistrarId")
  String currentSponsorClientId;

-  /** The ID of the registrar that created this resource. */
-  @Column(name = "creationRegistrarId", nullable = false)
+  /**
+   * The ID of the registrar that created this resource.
+   *
+   * <p>This can be null in the case of pre-Registry-3.0-migration history objects with null
+   * resource fields.
+   */
+  @Column(name = "creationRegistrarId")
  String creationClientId;

  /**
@@ -91,13 +100,17 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
  @Column(name = "lastEppUpdateRegistrarId")
  String lastEppUpdateClientId;

-  /** The time when this resource was created. */
-  // Map the method to XML, not the field, because if we map the field (with an adaptor class) it
-  // will never be omitted from the xml even if the timestamp inside creationTime is null and we
-  // return null from the adaptor. (Instead it gets written as an empty tag.)
-  @Column(nullable = false)
-  @Index
-  CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);
+  /**
+   * The time when this resource was created.
+   *
+   * <p>Map the method to XML, not the field, because if we map the field (with an adaptor class) it
+   * will never be omitted from the xml even if the timestamp inside creationTime is null and we
+   * return null from the adaptor (instead it gets written as an empty tag).
+   *
+   * <p>This can be null in the case of pre-Registry-3.0-migration history objects with null
+   * resource fields.
+   */
+  @Index CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);

  /**
   * The time when this resource was or will be deleted.
@@ -112,8 +125,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
   * out of the index at that time, as long as we query for resources whose deletion time is before
   * now.
   */
-  @Index
-  DateTime deletionTime;
+  @Index DateTime deletionTime;

  /**
   * The time that this resource was last updated.
@@ -144,7 +156,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
    return repoId;
  }

-  // Hibernate needs this to populate the repo ID, but no one else should ever use it
+  /** This method exists solely to satisfy Hibernate. Use {@link Builder} instead. */
  @SuppressWarnings("UnusedMethod")
  private void setRepoId(String repoId) {
    this.repoId = repoId;
@@ -252,7 +252,8 @@ public final class OteAccountBuilder {
  private void saveAllEntities() {
    tm().assertInTransaction();

-    ImmutableList<Registry> registries = ImmutableList.of(sunriseTld, gaTld, eapTld);
+    // use ImmutableObject instead of Registry so that the Key generation doesn't break
+    ImmutableList<ImmutableObject> registries = ImmutableList.of(sunriseTld, gaTld, eapTld);
    ImmutableList<RegistrarContact> contacts = contactsBuilder.build();

    if (!replaceExisting) {
@@ -33,12 +33,14 @@ import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.IgnoreSave;
 import com.googlecode.objectify.annotation.Index;
+import com.googlecode.objectify.annotation.OnLoad;
 import com.googlecode.objectify.annotation.Parent;
 import com.googlecode.objectify.condition.IfNull;
 import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ReportedOn;
 import google.registry.model.common.TimeOfYear;
+import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.GracePeriod;
 import google.registry.model.domain.rgp.GracePeriodStatus;
 import google.registry.model.domain.token.AllocationToken;
@@ -46,6 +48,7 @@ import google.registry.model.reporting.HistoryEntry;
 import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithLongVKey;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
@@ -56,16 +59,14 @@ import javax.persistence.Column;
 import javax.persistence.Embedded;
 import javax.persistence.EnumType;
 import javax.persistence.Enumerated;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.MappedSuperclass;
+import javax.persistence.PostLoad;
 import javax.persistence.Transient;
 import org.joda.money.Money;
 import org.joda.time.DateTime;

 /** A billable event in a domain's lifecycle. */
@MappedSuperclass
-@WithLongVKey
 public abstract class BillingEvent extends ImmutableObject
    implements Buildable, TransferServerApproveEntity {

@@ -107,10 +108,7 @@ public abstract class BillingEvent extends ImmutableObject
  }

  /** Entity id. */
-  @Id
-  @javax.persistence.Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
-  Long id;
+  @Id @javax.persistence.Id Long id;

  @Parent @DoNotHydrate @Transient Key<HistoryEntry> parent;

@@ -148,6 +146,21 @@ public abstract class BillingEvent extends ImmutableObject
  @Nullable
  Set<Flag> flags;

+  @PostLoad
+  void postLoad() {
+    parent =
+        Key.create(
+            Key.create(DomainBase.class, domainRepoId),
+            HistoryEntry.class,
+            domainHistoryRevisionId);
+  }
+
+  @OnLoad
+  void onLoad() {
+    domainHistoryRevisionId = parent.getId();
+    domainRepoId = parent.getParent().getName();
+  }
+
  public String getClientId() {
    return clientId;
  }
@@ -244,7 +257,6 @@ public abstract class BillingEvent extends ImmutableObject
    }

    public B setParent(Key<HistoryEntry> parentKey) {
-      // TODO(shicong): Figure out how to set domainHistoryRevisionId and domainRepoId
      getInstance().parent = parentKey;
      return thisCastToDerived();
    }
@@ -257,6 +269,11 @@ public abstract class BillingEvent extends ImmutableObject
      checkNotNull(instance.eventTime, "Event time must be set");
      checkNotNull(instance.targetId, "Target ID must be set");
      checkNotNull(instance.parent, "Parent must be set");
+      checkNotNull(instance.parent.getParent(), "parent.getParent() must be set");
+      checkNotNull(
+          instance.parent.getParent().getName(), "parent.getParent().getName() must be set");
+      instance.domainHistoryRevisionId = instance.parent.getId();
+      instance.domainRepoId = instance.parent.getParent().getName();
      return super.build();
    }
  }
@@ -274,7 +291,8 @@ public abstract class BillingEvent extends ImmutableObject
        @javax.persistence.Index(columnList = "allocation_token_id")
      })
  @AttributeOverride(name = "id", column = @Column(name = "billing_event_id"))
-  public static class OneTime extends BillingEvent {
+  @WithLongVKey
+  public static class OneTime extends BillingEvent implements DatastoreAndSqlEntity {

    /** The billable value. */
    @AttributeOverrides({
@@ -309,7 +327,7 @@ public abstract class BillingEvent extends ImmutableObject
     * Cancellation}s.
     */
    @Column(name = "cancellation_matching_billing_recurrence_id")
-    VKey<? extends BillingEvent> cancellationMatchingBillingEvent;
+    VKey<Recurring> cancellationMatchingBillingEvent;

    /**
     * The {@link AllocationToken} used in the creation of this event, or null if one was not used.
@@ -391,7 +409,7 @@ public abstract class BillingEvent extends ImmutableObject
      }

      public Builder setCancellationMatchingBillingEvent(
-          VKey<? extends BillingEvent> cancellationMatchingBillingEvent) {
+          VKey<Recurring> cancellationMatchingBillingEvent) {
        getInstance().cancellationMatchingBillingEvent = cancellationMatchingBillingEvent;
        return this;
      }
@@ -450,7 +468,8 @@ public abstract class BillingEvent extends ImmutableObject
        @javax.persistence.Index(columnList = "recurrence_time_of_year")
      })
  @AttributeOverride(name = "id", column = @Column(name = "billing_recurrence_id"))
-  public static class Recurring extends BillingEvent {
+  @WithLongVKey
+  public static class Recurring extends BillingEvent implements DatastoreAndSqlEntity {

    /**
     * The billing event recurs every year between {@link #eventTime} and this time on the
@@ -544,7 +563,8 @@ public abstract class BillingEvent extends ImmutableObject
        @javax.persistence.Index(columnList = "billingTime")
      })
  @AttributeOverride(name = "id", column = @Column(name = "billing_cancellation_id"))
-  public static class Cancellation extends BillingEvent {
+  @WithLongVKey
+  public static class Cancellation extends BillingEvent implements DatastoreAndSqlEntity {

    /** The billing time of the charge that is being cancelled. */
    @Index
@@ -664,7 +684,8 @@ public abstract class BillingEvent extends ImmutableObject
  /** An event representing a modification of an existing one-time billing event. */
  @ReportedOn
  @Entity
-  public static class Modification extends BillingEvent {
+  @WithLongVKey
+  public static class Modification extends BillingEvent implements DatastoreAndSqlEntity {

    /** The change in cost that should be applied to the original billing event. */
    Money cost;
@@ -14,13 +14,25 @@

 package google.registry.model.contact;

+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EntitySubclass;
-import google.registry.model.EppResource;
+import google.registry.model.ImmutableObject;
+import google.registry.model.contact.ContactHistory.ContactHistoryId;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import java.io.Serializable;
+import java.util.Optional;
+import javax.annotation.Nullable;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
 import javax.persistence.Column;
 import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.PostLoad;

 /**
 * A persisted history entry representing an EPP modification to a contact.
@@ -38,21 +50,125 @@ import javax.persistence.Entity;
      @javax.persistence.Index(columnList = "historyModificationTime")
    })
@EntitySubclass
-public class ContactHistory extends HistoryEntry {
+@Access(AccessType.FIELD)
+@IdClass(ContactHistoryId.class)
+public class ContactHistory extends HistoryEntry implements SqlEntity {
+
  // Store ContactBase instead of ContactResource so we don't pick up its @Id
-  ContactBase contactBase;
+  @Nullable ContactBase contactBase;

-  @Column(nullable = false)
-  VKey<ContactResource> contactRepoId;
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getContactRepoId() {
+    return parent.getName();
+  }

-  /** The state of the {@link ContactBase} object at this point in time. */
-  public ContactBase getContactBase() {
-    return contactBase;
+  /** This method is private because it is only used by Hibernate. */
+  @SuppressWarnings("unused")
+  private void setContactRepoId(String contactRepoId) {
+    parent = Key.create(ContactResource.class, contactRepoId);
+  }
+
+  @Id
+  @Column(name = "historyRevisionId")
+  @Access(AccessType.PROPERTY)
+  @Override
+  public long getId() {
+    return super.getId();
+  }
+
+  /**
+   * The values of all the fields on the {@link ContactBase} object after the action represented by
+   * this history object was executed.
+   *
+   * <p>Will be absent for objects created prior to the Registry 3.0 SQL migration.
+   */
+  public Optional<ContactBase> getContactBase() {
+    return Optional.ofNullable(contactBase);
  }

  /** The key to the {@link ContactResource} this is based off of. */
-  public VKey<ContactResource> getContactRepoId() {
-    return contactRepoId;
+  public VKey<ContactResource> getParentVKey() {
+    return VKey.create(ContactResource.class, getContactRepoId());
+  }
+
+  /** Creates a {@link VKey} instance for this entity. */
+  public VKey<ContactHistory> createVKey() {
+    return VKey.create(
+        ContactHistory.class, new ContactHistoryId(getContactRepoId(), getId()), Key.create(this));
+  }
+
+  @PostLoad
+  void postLoad() {
+    // Normally Hibernate would see that the contact fields are all null and would fill contactBase
+    // with a null object. Unfortunately, the updateTimestamp is never null in SQL.
+    if (contactBase != null && contactBase.getContactId() == null) {
+      contactBase = null;
+    }
+  }
+
+  // In Datastore, save as a HistoryEntry object regardless of this object's type
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(asHistoryEntry());
+  }
+
+  /** Class to represent the composite primary key of {@link ContactHistory} entity. */
+  static class ContactHistoryId extends ImmutableObject implements Serializable {
+
+    private String contactRepoId;
+
+    private Long id;
+
+    /** Hibernate requires this default constructor. */
+    private ContactHistoryId() {}
+
+    ContactHistoryId(String contactRepoId, long id) {
+      this.contactRepoId = contactRepoId;
+      this.id = id;
+    }
+
+    /**
+     * Returns the contact repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getContactRepoId() {
+      return contactRepoId;
+    }
+
+    /**
+     * Returns the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private long getId() {
+      return id;
+    }
+
+    /**
+     * Sets the contact repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setContactRepoId(String contactRepoId) {
+      this.contactRepoId = contactRepoId;
+    }
+
+    /**
+     * Sets the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setId(long id) {
+      this.id = id;
+    }
  }

  @Override
@@ -73,18 +189,8 @@ public class ContactHistory extends HistoryEntry {
      return this;
    }

-    public Builder setContactRepoId(VKey<ContactResource> contactRepoId) {
-      getInstance().contactRepoId = contactRepoId;
-      contactRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
-      return this;
-    }
-
-    // We can remove this once all HistoryEntries are converted to History objects
-    @Override
-    public Builder setParent(Key<? extends EppResource> parent) {
-      super.setParent(parent);
-      getInstance().contactRepoId =
-          VKey.create(ContactResource.class, parent.getName(), (Key<ContactResource>) parent);
+    public Builder setContactRepoId(String contactRepoId) {
+      getInstance().parent = Key.create(ContactResource.class, contactRepoId);
      return this;
    }
  }
@@ -51,7 +51,6 @@ public class ContactResource extends ContactBase

  @Override
  public VKey<ContactResource> createVKey() {
-    // TODO(mmuller): create symmetric keys if we can ever reload both sides.
    return VKey.create(ContactResource.class, getRepoId(), Key.create(this));
  }

@@ -14,12 +14,12 @@

 package google.registry.model.domain;

-
 import com.googlecode.objectify.Key;
 import google.registry.model.EppResource;
 import google.registry.model.EppResource.ForeignKeyedEppResource;
 import google.registry.model.annotations.ExternalMessagingName;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.model.domain.secdns.DelegationSignerData;
 import google.registry.model.host.HostResource;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithStringVKey;
@@ -82,17 +82,57 @@ public class DomainBase extends DomainContent
    return super.nsHosts;
  }

+  /**
+   * Returns the set of {@link GracePeriod} associated with the domain.
+   *
+   * <p>This is the getter method specific for Hibernate to access the field so it is set to
+   * private. The caller can use the public {@link #getGracePeriods()} to get the grace periods.
+   *
+   * <p>Note that we need to set `insertable = false, updatable = false` for @JoinColumn, otherwise
+   * Hibernate would try to set the foreign key to null(through an UPDATE TABLE sql) instead of
+   * deleting the whole entry from the table when the {@link GracePeriod} is removed from the set.
+   */
  @Access(AccessType.PROPERTY)
  @OneToMany(
      cascade = {CascadeType.ALL},
      fetch = FetchType.EAGER,
      orphanRemoval = true)
-  @JoinColumn(name = "domainRepoId", referencedColumnName = "repoId")
+  @JoinColumn(
+      name = "domainRepoId",
+      referencedColumnName = "repoId",
+      insertable = false,
+      updatable = false)
  @SuppressWarnings("UnusedMethod")
  private Set<GracePeriod> getInternalGracePeriods() {
    return gracePeriods;
  }

+  /**
+   * Returns the set of {@link DelegationSignerData} associated with the domain.
+   *
+   * <p>This is the getter method specific for Hibernate to access the field so it is set to
+   * private. The caller can use the public {@link #getDsData()} to get the DS data.
+   *
+   * <p>Note that we need to set `insertable = false, updatable = false` for @JoinColumn, otherwise
+   * Hibernate would try to set the foreign key to null(through an UPDATE TABLE sql) instead of
+   * deleting the whole entry from the table when the {@link DelegationSignerData} is removed from
+   * the set.
+   */
+  @Access(AccessType.PROPERTY)
+  @OneToMany(
+      cascade = {CascadeType.ALL},
+      fetch = FetchType.EAGER,
+      orphanRemoval = true)
+  @JoinColumn(
+      name = "domainRepoId",
+      referencedColumnName = "repoId",
+      insertable = false,
+      updatable = false)
+  @SuppressWarnings("UnusedMethod")
+  private Set<DelegationSignerData> getInternalDelegationSignerData() {
+    return dsData;
+  }
+
  @Override
  public VKey<DomainBase> createVKey() {
    return VKey.create(DomainBase.class, getRepoId(), Key.create(this));
@@ -35,11 +35,13 @@ import static google.registry.util.DomainNameUtils.canonicalizeDomainName;
 import static google.registry.util.DomainNameUtils.getTldFromDomainName;
 import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

+import com.google.common.base.Predicate;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedSet;
 import com.google.common.collect.Ordering;
 import com.google.common.collect.Sets;
 import com.google.common.collect.Streams;
+import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.IgnoreSave;
 import com.googlecode.objectify.annotation.Index;
@@ -49,6 +51,7 @@ import google.registry.flows.ResourceFlowUtils;
 import google.registry.model.EppResource;
 import google.registry.model.EppResource.ResourceWithTransferData;
 import google.registry.model.billing.BillingEvent;
+import google.registry.model.common.EntityGroupRoot;
 import google.registry.model.contact.ContactResource;
 import google.registry.model.domain.launch.LaunchNotice;
 import google.registry.model.domain.rgp.GracePeriodStatus;
@@ -57,6 +60,7 @@ import google.registry.model.eppcommon.StatusValue;
 import google.registry.model.host.HostResource;
 import google.registry.model.poll.PollMessage;
 import google.registry.model.registry.Registry;
+import google.registry.model.reporting.HistoryEntry;
 import google.registry.model.transfer.DomainTransferData;
 import google.registry.model.transfer.TransferStatus;
 import google.registry.persistence.VKey;
@@ -65,7 +69,6 @@ import java.util.HashSet;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
-import java.util.function.Predicate;
 import javax.annotation.Nullable;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
@@ -209,6 +212,15 @@ public class DomainContent extends EppResource
  @Column(name = "deletion_poll_message_id")
  VKey<PollMessage.OneTime> deletePollMessage;

+  /**
+   * History record for the delete poll message.
+   *
+   * <p>Here so we can restore the original ofy key from sql.
+   */
+  @Column(name = "deletion_poll_message_history_id")
+  @Ignore
+  Long deletePollMessageHistoryId;
+
  /**
   * The recurring billing event associated with this domain's autorenewals.
   *
@@ -220,6 +232,15 @@ public class DomainContent extends EppResource
  @Column(name = "billing_recurrence_id")
  VKey<BillingEvent.Recurring> autorenewBillingEvent;

+  /**
+   * History record for the autorenew billing event.
+   *
+   * <p>Here so we can restore the original ofy key from sql.
+   */
+  @Column(name = "billing_recurrence_history_id")
+  @Ignore
+  Long autorenewBillingEventHistoryId;
+
  /**
   * The recurring poll message associated with this domain's autorenewals.
   *
@@ -231,6 +252,13 @@ public class DomainContent extends EppResource
  @Column(name = "autorenew_poll_message_id")
  VKey<PollMessage.Autorenew> autorenewPollMessage;

+  /**
+   * History record for the autorenew poll message.
+   *
+   * <p>Here so we can restore the original ofy key from sql.
+   */
+  @Ignore Long autorenewPollMessageHistoryId;
+
  /** The unexpired grace periods for this domain (some of which may not be active yet). */
  @Transient Set<GracePeriod> gracePeriods;

@@ -280,16 +308,24 @@ public class DomainContent extends EppResource
    // object will have a null hashcode so that it can get a recalculated hashcode
    // when its hashCode() is invoked.
    // TODO(b/162739503): Remove this after fully migrating to Cloud SQL.
-    if (gracePeriods != null) {
-      gracePeriods =
-          gracePeriods.stream()
-              .map(gracePeriod -> gracePeriod.cloneWithDomainRepoId(getRepoId()))
-              .collect(toImmutableSet());
-    }
+    gracePeriods =
+        nullToEmptyImmutableCopy(gracePeriods).stream()
+            .map(gracePeriod -> gracePeriod.cloneWithDomainRepoId(getRepoId()))
+            .collect(toImmutableSet());
+
+    // Restore history record ids.
+    autorenewPollMessageHistoryId = getHistoryId(autorenewPollMessage);
+    autorenewBillingEventHistoryId = getHistoryId(autorenewBillingEvent);
+    deletePollMessageHistoryId = getHistoryId(deletePollMessage);
+    dsData =
+        nullToEmptyImmutableCopy(dsData).stream()
+            .map(dsData -> dsData.cloneWithDomainRepoId(getRepoId()))
+            .collect(toImmutableSet());
  }

  @PostLoad
-  void postLoad() {
+  @SuppressWarnings("UnusedMethod")
+  private final void postLoad() {
    // Reconstitute the contact list.
    ImmutableSet.Builder<DesignatedContact> contactsBuilder = new ImmutableSet.Builder<>();

@@ -308,6 +344,23 @@ public class DomainContent extends EppResource
    }

    allContacts = contactsBuilder.build();
+
+    // Reconstitute the composite ofy keys from the SQL data.
+    Key<DomainBase> myKey = Key.create(DomainBase.class, getRepoId());
+    deletePollMessage = restoreOfyFrom(myKey, deletePollMessage, deletePollMessageHistoryId);
+    autorenewBillingEvent =
+        restoreOfyFrom(myKey, autorenewBillingEvent, autorenewBillingEventHistoryId);
+    autorenewPollMessage =
+        restoreOfyFrom(myKey, autorenewPollMessage, autorenewPollMessageHistoryId);
+  }
+
+  private <T> VKey<T> restoreOfyFrom(Key<DomainBase> domainKey, VKey<T> key, Long historyId) {
+    if (historyId == null) {
+      // This is a legacy key (or a null key, in which case this works too)
+      return VKey.restoreOfyFrom(key, EntityGroupRoot.class, "per-tld");
+    } else {
+      return VKey.restoreOfyFrom(key, domainKey, HistoryEntry.class, historyId);
+    }
  }

  public ImmutableSet<String> getSubordinateHosts() {
@@ -389,6 +442,12 @@ public class DomainContent extends EppResource
    this.gracePeriods = gracePeriods;
  }

+  // Hibernate needs this in order to populate dsData but no one else should ever use it
+  @SuppressWarnings("UnusedMethod")
+  private void setInternalDelegationSignerData(Set<DelegationSignerData> dsData) {
+    this.dsData = dsData;
+  }
+
  public final String getCurrentSponsorClientId() {
    return getPersistedCurrentSponsorClientId();
  }
@@ -657,13 +716,24 @@ public class DomainContent extends EppResource
            + " use DomainBase instead");
  }

+  private static Long getHistoryId(VKey<?> key) {
+    if (key == null) {
+      return null;
+    }
+    Key<?> parent = key.getOfyKey().getParent();
+    if (parent == null || parent.getKind().equals("EntityGroupRoot")) {
+      return null;
+    }
+    return parent.getId();
+  }
+
  /** Predicate to determine if a given {@link DesignatedContact} is the registrant. */
  static final Predicate<DesignatedContact> IS_REGISTRANT =
      (DesignatedContact contact) -> DesignatedContact.Type.REGISTRANT.equals(contact.type);

  /** An override of {@link EppResource#asBuilder} with tighter typing. */
  @Override
-  public Builder asBuilder() {
+  public Builder<? extends DomainContent, ?> asBuilder() {
    return new Builder<>(clone(this));
  }

@@ -698,7 +768,19 @@ public class DomainContent extends EppResource
      }
      checkArgumentNotNull(instance.getRegistrant(), "Missing registrant");
      instance.tld = getTldFromDomainName(instance.fullyQualifiedDomainName);
-      return super.build();
+
+      T newDomain = super.build();
+      // Hibernate throws exception if gracePeriods or dsData is null because we enabled all
+      // cascadable operations and orphan removal.
+      newDomain.gracePeriods =
+          newDomain.gracePeriods == null ? ImmutableSet.of() : newDomain.gracePeriods;
+      newDomain.dsData =
+          newDomain.dsData == null
+              ? ImmutableSet.of()
+              : newDomain.dsData.stream()
+                  .map(ds -> ds.cloneWithDomainRepoId(instance.getRepoId()))
+                  .collect(toImmutableSet());
+      return newDomain;
    }

    public B setDomainName(String domainName) {
@@ -820,16 +902,19 @@ public class DomainContent extends EppResource

    public B setDeletePollMessage(VKey<PollMessage.OneTime> deletePollMessage) {
      getInstance().deletePollMessage = deletePollMessage;
+      getInstance().deletePollMessageHistoryId = getHistoryId(deletePollMessage);
      return thisCastToDerived();
    }

    public B setAutorenewBillingEvent(VKey<BillingEvent.Recurring> autorenewBillingEvent) {
      getInstance().autorenewBillingEvent = autorenewBillingEvent;
+      getInstance().autorenewBillingEventHistoryId = getHistoryId(autorenewBillingEvent);
      return thisCastToDerived();
    }

    public B setAutorenewPollMessage(VKey<PollMessage.Autorenew> autorenewPollMessage) {
      getInstance().autorenewPollMessage = autorenewPollMessage;
+      getInstance().autorenewPollMessageHistoryId = getHistoryId(autorenewPollMessage);
      return thisCastToDerived();
    }

@@ -14,20 +14,40 @@

 package google.registry.model.domain;

+import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
+
+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EntitySubclass;
-import google.registry.model.EppResource;
-import google.registry.model.contact.ContactResource;
+import com.googlecode.objectify.annotation.Ignore;
+import google.registry.model.ImmutableObject;
+import google.registry.model.domain.DomainHistory.DomainHistoryId;
 import google.registry.model.host.HostResource;
+import google.registry.model.reporting.DomainTransactionRecord;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import java.io.Serializable;
+import java.util.Optional;
 import java.util.Set;
+import javax.annotation.Nullable;
 import javax.persistence.Access;
 import javax.persistence.AccessType;
+import javax.persistence.AttributeOverride;
+import javax.persistence.AttributeOverrides;
+import javax.persistence.CascadeType;
 import javax.persistence.Column;
 import javax.persistence.ElementCollection;
 import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.Index;
+import javax.persistence.JoinColumn;
 import javax.persistence.JoinTable;
+import javax.persistence.OneToMany;
+import javax.persistence.PostLoad;
+import javax.persistence.Table;

 /**
 * A persisted history entry representing an EPP modification to a domain.
@@ -37,44 +57,197 @@ import javax.persistence.JoinTable;
 * the foreign-keyed fields in that class can refer to this object.
 */
@Entity
-@javax.persistence.Table(
+@Table(
    indexes = {
-      @javax.persistence.Index(columnList = "creationTime"),
-      @javax.persistence.Index(columnList = "historyRegistrarId"),
-      @javax.persistence.Index(columnList = "historyType"),
-      @javax.persistence.Index(columnList = "historyModificationTime")
+      @Index(columnList = "creationTime"),
+      @Index(columnList = "historyRegistrarId"),
+      @Index(columnList = "historyType"),
+      @Index(columnList = "historyModificationTime")
    })
@EntitySubclass
-public class DomainHistory extends HistoryEntry {
+@Access(AccessType.FIELD)
+@IdClass(DomainHistoryId.class)
+public class DomainHistory extends HistoryEntry implements SqlEntity {
+
  // Store DomainContent instead of DomainBase so we don't pick up its @Id
-  DomainContent domainContent;
+  @Nullable DomainContent domainContent;

-  @Column(nullable = false)
-  VKey<DomainBase> domainRepoId;
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getDomainRepoId() {
+    return parent.getName();
+  }

+  /** This method is private because it is only used by Hibernate. */
+  @SuppressWarnings("unused")
+  private void setDomainRepoId(String domainRepoId) {
+    parent = Key.create(DomainBase.class, domainRepoId);
+  }
+
+  // We could have reused domainContent.nsHosts here, but Hibernate throws a weird exception after
+  // we change to use a composite primary key.
+  // TODO(b/166776754): Investigate if we can reuse domainContent.nsHosts for storing host keys.
+  @Ignore
  @ElementCollection
  @JoinTable(name = "DomainHistoryHost")
-  @Access(AccessType.PROPERTY)
  @Column(name = "host_repo_id")
+  Set<VKey<HostResource>> nsHosts;
+
+  @Override
+  @Nullable
+  @Access(AccessType.PROPERTY)
+  @AttributeOverrides({
+      @AttributeOverride(
+          name = "unit",
+          column = @Column(name = "historyPeriodUnit")),
+      @AttributeOverride(
+          name = "value",
+          column = @Column(name = "historyPeriodValue"))
+  })
+  public Period getPeriod() {
+    return super.getPeriod();
+  }
+
+  /**
+   * For transfers, the id of the other registrar.
+   *
+   * <p>For requests and cancels, the other registrar is the losing party (because the registrar
+   * sending the EPP transfer command is the gaining party). For approves and rejects, the other
+   * registrar is the gaining party.
+   */
+  @Nullable
+  @Access(AccessType.PROPERTY)
+  @Column(name = "historyOtherRegistrarId")
+  public String getOtherRegistrarId() {
+    return super.getOtherClientId();
+  }
+
+  /**
+   * Logging field for transaction reporting.
+   *
+   * <p>This will be empty for any DomainHistory/HistoryEntry generated before this field was added,
+   * mid-2017, as well as any action that does not generate billable events (e.g. updates).
+   */
+  @Access(AccessType.PROPERTY)
+  @OneToMany(cascade = {CascadeType.ALL})
+  @JoinColumn(name = "historyRevisionId", referencedColumnName = "historyRevisionId")
+  @JoinColumn(name = "domainRepoId", referencedColumnName = "domainRepoId")
+  @Override
+  public Set<DomainTransactionRecord> getDomainTransactionRecords() {
+    return super.getDomainTransactionRecords();
+  }
+
+  @Id
+  @Column(name = "historyRevisionId")
+  @Access(AccessType.PROPERTY)
+  @Override
+  public long getId() {
+    return super.getId();
+  }
+
+  /** Returns keys to the {@link HostResource} that are the nameservers for the domain. */
  public Set<VKey<HostResource>> getNsHosts() {
-    return domainContent.nsHosts;
+    return nsHosts;
  }

-  /** The state of the {@link DomainContent} object at this point in time. */
-  public DomainContent getDomainContent() {
-    return domainContent;
+  /**
+   * The values of all the fields on the {@link DomainContent} object after the action represented
+   * by this history object was executed.
+   *
+   * <p>Will be absent for objects created prior to the Registry 3.0 SQL migration.
+   */
+  public Optional<DomainContent> getDomainContent() {
+    return Optional.ofNullable(domainContent);
  }

-  /** The key to the {@link ContactResource} this is based off of. */
-  public VKey<DomainBase> getDomainRepoId() {
-    return domainRepoId;
+  /** The key to the {@link DomainBase} this is based off of. */
+  public VKey<DomainBase> getParentVKey() {
+    return VKey.create(DomainBase.class, getDomainRepoId());
  }

-  // Hibernate needs this in order to populate nsHosts but no one else should ever use it
-  @SuppressWarnings("UnusedMethod")
-  private void setNsHosts(Set<VKey<HostResource>> nsHosts) {
+  /** Creates a {@link VKey} instance for this entity. */
+  public VKey<DomainHistory> createVKey() {
+    return VKey.create(
+        DomainHistory.class, new DomainHistoryId(getDomainRepoId(), getId()), Key.create(this));
+  }
+
+  @PostLoad
+  void postLoad() {
    if (domainContent != null) {
-      domainContent.nsHosts = nsHosts;
+      domainContent.nsHosts = nullToEmptyImmutableCopy(nsHosts);
+      // Normally Hibernate would see that the domain fields are all null and would fill
+      // domainContent with a null object. Unfortunately, the updateTimestamp is never null in SQL.
+      if (domainContent.getDomainName() == null) {
+        domainContent = null;
+      } else {
+        if (domainContent.getRepoId() == null) {
+          domainContent = domainContent.asBuilder().setRepoId(parent.getName()).build();
+        }
+      }
+    }
+  }
+
+  // In Datastore, save as a HistoryEntry object regardless of this object's type
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(asHistoryEntry());
+  }
+
+  /** Class to represent the composite primary key of {@link DomainHistory} entity. */
+  static class DomainHistoryId extends ImmutableObject implements Serializable {
+
+    private String domainRepoId;
+
+    private Long id;
+
+    /** Hibernate requires this default constructor. */
+    private DomainHistoryId() {}
+
+    DomainHistoryId(String domainRepoId, long id) {
+      this.domainRepoId = domainRepoId;
+      this.id = id;
+    }
+
+    /**
+     * Returns the domain repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getDomainRepoId() {
+      return domainRepoId;
+    }
+
+    /**
+     * Returns the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private long getId() {
+      return id;
+    }
+
+    /**
+     * Sets the domain repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setDomainRepoId(String domainRepoId) {
+      this.domainRepoId = domainRepoId;
+    }
+
+    /**
+     * Sets the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setId(long id) {
+      this.id = id;
    }
  }

@@ -93,21 +266,14 @@ public class DomainHistory extends HistoryEntry {

    public Builder setDomainContent(DomainContent domainContent) {
      getInstance().domainContent = domainContent;
+      if (domainContent != null) {
+        getInstance().nsHosts = nullToEmptyImmutableCopy(domainContent.nsHosts);
+      }
      return this;
    }

-    public Builder setDomainRepoId(VKey<DomainBase> domainRepoId) {
-      getInstance().domainRepoId = domainRepoId;
-      domainRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
-      return this;
-    }
-
-    // We can remove this once all HistoryEntries are converted to History objects
-    @Override
-    public Builder setParent(Key<? extends EppResource> parent) {
-      super.setParent(parent);
-      getInstance().domainRepoId =
-          VKey.create(DomainBase.class, parent.getName(), (Key<DomainBase>) parent);
+    public Builder setDomainRepoId(String domainRepoId) {
+      getInstance().parent = Key.create(DomainBase.class, domainRepoId);
      return this;
    }
  }
@@ -22,6 +22,7 @@ import google.registry.model.billing.BillingEvent;
 import google.registry.model.billing.BillingEvent.Recurring;
 import google.registry.model.domain.rgp.GracePeriodStatus;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import javax.annotation.Nullable;
 import javax.persistence.Entity;
 import javax.persistence.Index;
@@ -37,7 +38,7 @@ import org.joda.time.DateTime;
@Embed
@Entity
@Table(indexes = @Index(columnList = "domainRepoId"))
-public class GracePeriod extends GracePeriodBase {
+public class GracePeriod extends GracePeriodBase implements DatastoreAndSqlEntity {

  private static GracePeriod createInternal(
      GracePeriodStatus type,
@@ -31,9 +31,9 @@ public class Period extends ImmutableObject {
  @XmlAttribute
  Unit unit;

-  @XmlValue
-  Integer value;
+  @XmlValue Integer value;

+  @Enumerated(EnumType.STRING)
  public Unit getUnit() {
    return unit;
  }
@@ -42,6 +42,18 @@ public class Period extends ImmutableObject {
    return value;
  }

+  /** This method exists solely to satisfy Hibernate. Use {@link #create(int, Unit)} instead. */
+  @SuppressWarnings("UnusedMethod")
+  private void setUnit(Unit unit) {
+    this.unit = unit;
+  }
+
+  /** This method exists solely to satisfy Hibernate. Use {@link #create(int, Unit)} instead. */
+  @SuppressWarnings("UnusedMethod")
+  private void setValue(Integer value) {
+    this.value = value;
+  }
+
  /** The unit enum. */
  public enum Unit {
    @XmlEnumValue("y")
@@ -14,10 +14,22 @@

 package google.registry.model.domain.secdns;

+import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
+
 import com.googlecode.objectify.annotation.Embed;
+import com.googlecode.objectify.annotation.Ignore;
 import google.registry.model.ImmutableObject;
+import google.registry.model.domain.secdns.DelegationSignerData.DelegationSignerDataId;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
+import java.io.Serializable;
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.IdClass;
+import javax.persistence.Index;
+import javax.persistence.Table;
 import javax.xml.bind.DatatypeConverter;
 import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlTransient;
 import javax.xml.bind.annotation.XmlType;
 import javax.xml.bind.annotation.adapters.HexBinaryAdapter;
 import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
@@ -30,19 +42,26 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
 */
@Embed
@XmlType(name = "dsData")
-@javax.persistence.Entity
-public class DelegationSignerData extends ImmutableObject {
+@Entity
+@Table(indexes = @Index(columnList = "domainRepoId"))
+@IdClass(DelegationSignerDataId.class)
+public class DelegationSignerData extends ImmutableObject implements DatastoreAndSqlEntity {

  private DelegationSignerData() {}

+  @Ignore @XmlTransient @javax.persistence.Id String domainRepoId;
+
  /** The identifier for this particular key in the domain. */
-  @javax.persistence.Id int keyTag;
+  @javax.persistence.Id
+  @Column(nullable = false)
+  int keyTag;

  /**
   * The algorithm used by this key.
   *
   * @see <a href="http://tools.ietf.org/html/rfc4034#appendix-A.1">RFC 4034 Appendix A.1</a>
   */
+  @Column(nullable = false)
  @XmlElement(name = "alg")
  int algorithm;

@@ -51,6 +70,7 @@ public class DelegationSignerData extends ImmutableObject {
   *
   * @see <a href="http://tools.ietf.org/html/rfc4034#appendix-A.2">RFC 4034 Appendix A.2</a>
   */
+  @Column(nullable = false)
  int digestType;

  /**
@@ -58,6 +78,7 @@ public class DelegationSignerData extends ImmutableObject {
   *
   * @see <a href="http://tools.ietf.org/html/rfc4034#section-5.1.4">RFC 4034 Section 5.1.4</a>
   */
+  @Column(nullable = false)
  @XmlJavaTypeAdapter(HexBinaryAdapter.class)
  byte[] digest;

@@ -81,16 +102,34 @@ public class DelegationSignerData extends ImmutableObject {
    return digest == null ? "" : DatatypeConverter.printHexBinary(digest);
  }

+  public DelegationSignerData cloneWithDomainRepoId(String domainRepoId) {
+    DelegationSignerData clone = clone(this);
+    clone.domainRepoId = checkArgumentNotNull(domainRepoId);
+    return clone;
+  }
+
+  public DelegationSignerData cloneWithoutDomainRepoId() {
+    DelegationSignerData clone = clone(this);
+    clone.domainRepoId = null;
+    return clone;
+  }
+
  public static DelegationSignerData create(
-      int keyTag, int algorithm, int digestType, byte[] digest) {
+      int keyTag, int algorithm, int digestType, byte[] digest, String domainRepoId) {
    DelegationSignerData instance = new DelegationSignerData();
    instance.keyTag = keyTag;
    instance.algorithm = algorithm;
    instance.digestType = digestType;
    instance.digest = digest;
+    instance.domainRepoId = domainRepoId;
    return instance;
  }

+  public static DelegationSignerData create(
+      int keyTag, int algorithm, int digestType, byte[] digest) {
+    return create(keyTag, algorithm, digestType, digest, null);
+  }
+
  public static DelegationSignerData create(
      int keyTag, int algorithm, int digestType, String digestAsHex) {
    return create(keyTag, algorithm, digestType, DatatypeConverter.parseHexBinary(digestAsHex));
@@ -106,4 +145,20 @@ public class DelegationSignerData extends ImmutableObject {
        "%d %d %d %s",
        this.keyTag, this.algorithm, this.digestType, DatatypeConverter.printHexBinary(digest));
  }
+
+  static class DelegationSignerDataId extends ImmutableObject implements Serializable {
+    String domainRepoId;
+    int keyTag;
+
+    private DelegationSignerDataId() {}
+
+    private DelegationSignerDataId(String domainRepoId, int keyTag) {
+      this.domainRepoId = domainRepoId;
+      this.keyTag = keyTag;
+    }
+
+    public static DelegationSignerDataId create(String domainRepoId, int keyTag) {
+      return new DelegationSignerDataId(checkArgumentNotNull(domainRepoId), keyTag);
+    }
+  }
 }
@@ -20,8 +20,10 @@ import static com.google.common.base.Strings.nullToEmpty;

 import com.google.common.collect.ImmutableSet;
 import google.registry.model.EppResource;
+import google.registry.model.contact.ContactBase;
 import google.registry.model.contact.ContactResource;
 import google.registry.model.domain.DomainBase;
+import google.registry.model.domain.DomainContent;
 import google.registry.model.host.HostBase;
 import google.registry.model.host.HostResource;
 import google.registry.model.translators.EnumToAttributeAdapter.EppEnum;
@@ -128,9 +130,15 @@ public enum StatusValue implements EppEnum {

  /** Enum to help clearly list which resource types a status value is allowed to be present on. */
  private enum AllowedOn {
-    ALL(ContactResource.class, DomainBase.class, HostBase.class, HostResource.class),
+    ALL(
+        ContactBase.class,
+        ContactResource.class,
+        DomainContent.class,
+        DomainBase.class,
+        HostBase.class,
+        HostResource.class),
    NONE,
-    DOMAINS(DomainBase.class);
+    DOMAINS(DomainContent.class, DomainBase.class);

    private final ImmutableSet<Class<? extends EppResource>> classes;

@@ -14,13 +14,25 @@

 package google.registry.model.host;

+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.EntitySubclass;
-import google.registry.model.EppResource;
+import google.registry.model.ImmutableObject;
+import google.registry.model.host.HostHistory.HostHistoryId;
 import google.registry.model.reporting.HistoryEntry;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
+import java.io.Serializable;
+import java.util.Optional;
+import javax.annotation.Nullable;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
 import javax.persistence.Column;
 import javax.persistence.Entity;
+import javax.persistence.Id;
+import javax.persistence.IdClass;
+import javax.persistence.PostLoad;

 /**
 * A persisted history entry representing an EPP modification to a host.
@@ -39,22 +51,125 @@ import javax.persistence.Entity;
      @javax.persistence.Index(columnList = "historyModificationTime")
    })
@EntitySubclass
-public class HostHistory extends HistoryEntry {
+@Access(AccessType.FIELD)
+@IdClass(HostHistoryId.class)
+public class HostHistory extends HistoryEntry implements SqlEntity {

  // Store HostBase instead of HostResource so we don't pick up its @Id
-  HostBase hostBase;
+  @Nullable HostBase hostBase;

-  @Column(nullable = false)
-  VKey<HostResource> hostRepoId;
+  @Id
+  @Access(AccessType.PROPERTY)
+  public String getHostRepoId() {
+    return parent.getName();
+  }

-  /** The state of the {@link HostBase} object at this point in time. */
-  public HostBase getHostBase() {
-    return hostBase;
+  /** This method is private because it is only used by Hibernate. */
+  @SuppressWarnings("unused")
+  private void setHostRepoId(String hostRepoId) {
+    parent = Key.create(HostResource.class, hostRepoId);
+  }
+
+  @Id
+  @Column(name = "historyRevisionId")
+  @Access(AccessType.PROPERTY)
+  @Override
+  public long getId() {
+    return super.getId();
+  }
+
+  /**
+   * The values of all the fields on the {@link HostBase} object after the action represented by
+   * this history object was executed.
+   *
+   * <p>Will be absent for objects created prior to the Registry 3.0 SQL migration.
+   */
+  public Optional<HostBase> getHostBase() {
+    return Optional.ofNullable(hostBase);
  }

  /** The key to the {@link google.registry.model.host.HostResource} this is based off of. */
-  public VKey<HostResource> getHostRepoId() {
-    return hostRepoId;
+  public VKey<HostResource> getParentVKey() {
+    return VKey.create(HostResource.class, getHostRepoId());
+  }
+
+  /** Creates a {@link VKey} instance for this entity. */
+  public VKey<HostHistory> createVKey() {
+    return VKey.create(
+        HostHistory.class, new HostHistoryId(getHostRepoId(), getId()), Key.create(this));
+  }
+
+  @PostLoad
+  void postLoad() {
+    // Normally Hibernate would see that the host fields are all null and would fill hostBase
+    // with a null object. Unfortunately, the updateTimestamp is never null in SQL.
+    if (hostBase != null && hostBase.getHostName() == null) {
+      hostBase = null;
+    }
+  }
+
+  // In Datastore, save as a HistoryEntry object regardless of this object's type
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(asHistoryEntry());
+  }
+
+  /** Class to represent the composite primary key of {@link HostHistory} entity. */
+  static class HostHistoryId extends ImmutableObject implements Serializable {
+
+    private String hostRepoId;
+
+    private Long id;
+
+    /** Hibernate requires this default constructor. */
+    private HostHistoryId() {}
+
+    HostHistoryId(String hostRepoId, long id) {
+      this.hostRepoId = hostRepoId;
+      this.id = id;
+    }
+
+    /**
+     * Returns the host repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private String getHostRepoId() {
+      return hostRepoId;
+    }
+
+    /**
+     * Returns the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate.
+     */
+    @SuppressWarnings("unused")
+    private long getId() {
+      return id;
+    }
+
+    /**
+     * Sets the host repository id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setHostRepoId(String hostRepoId) {
+      this.hostRepoId = hostRepoId;
+    }
+
+    /**
+     * Sets the history revision id.
+     *
+     * <p>This method is private because it is only used by Hibernate and should not be used
+     * externally to keep immutability.
+     */
+    @SuppressWarnings("unused")
+    private void setId(long id) {
+      this.id = id;
+    }
  }

  @Override
@@ -75,18 +190,8 @@ public class HostHistory extends HistoryEntry {
      return this;
    }

-    public Builder setHostRepoId(VKey<HostResource> hostRepoId) {
-      getInstance().hostRepoId = hostRepoId;
-      hostRepoId.maybeGetOfyKey().ifPresent(parent -> getInstance().parent = parent);
-      return this;
-    }
-
-    // We can remove this once all HistoryEntries are converted to History objects
-    @Override
-    public Builder setParent(Key<? extends EppResource> parent) {
-      super.setParent(parent);
-      getInstance().hostRepoId =
-          VKey.create(HostResource.class, parent.getName(), (Key<HostResource>) parent);
+    public Builder setHostRepoId(String hostRepoId) {
+      getInstance().parent = Key.create(HostResource.class, hostRepoId);
      return this;
    }
  }
@@ -32,7 +32,7 @@ import javax.persistence.AccessType;
 */
@ReportedOn
@Entity
-@javax.persistence.Entity
+@javax.persistence.Entity(name = "Host")
@ExternalMessagingName("host")
@WithStringVKey
@Access(AccessType.FIELD) // otherwise it'll use the default if the repoId (property)
@@ -17,6 +17,7 @@ package google.registry.model.index;
 import static google.registry.util.TypeUtils.instantiate;

 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
 import com.googlecode.objectify.annotation.Id;
@@ -25,11 +26,13 @@ import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.BackupGroupRoot;
 import google.registry.model.EppResource;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;

 /** An index that allows for quick enumeration of all EppResource entities (e.g. via map reduce). */
@ReportedOn
@Entity
-public class EppResourceIndex extends BackupGroupRoot {
+public class EppResourceIndex extends BackupGroupRoot implements DatastoreEntity {

  @Id
  String id;
@@ -74,4 +77,9 @@ public class EppResourceIndex extends BackupGroupRoot {
  public static <T extends EppResource> EppResourceIndex create(Key<T> resourceKey) {
    return create(EppResourceIndexBucket.getBucketKey(resourceKey), resourceKey);
  }
+
+  @Override
+  public ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of(); // not relevant in SQL
+  }
 }
@@ -24,16 +24,23 @@ import com.googlecode.objectify.annotation.Id;
 import google.registry.model.EppResource;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.VirtualEntity;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;

 /** A virtual entity to represent buckets to which EppResourceIndex objects are randomly added. */
@Entity
@VirtualEntity
-public class EppResourceIndexBucket extends ImmutableObject {
+public class EppResourceIndexBucket extends ImmutableObject implements DatastoreEntity {

  @SuppressWarnings("unused")
  @Id
  private long bucketId;

+  @Override
+  public ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of(); // not relevant in SQL
+  }
+
  /**
   * Deterministic function that returns a bucket id based on the resource's roid.
   * NB: At the moment, nothing depends on this being deterministic, so we have the ability to
@@ -43,6 +43,8 @@ import google.registry.model.contact.ContactResource;
 import google.registry.model.domain.DomainBase;
 import google.registry.model.host.HostResource;
 import google.registry.persistence.VKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
 import google.registry.util.NonFinalForTesting;
 import java.util.Map;
 import java.util.Optional;
@@ -61,28 +63,44 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
  /** The {@link ForeignKeyIndex} type for {@link ContactResource} entities. */
  @ReportedOn
  @Entity
-  public static class ForeignKeyContactIndex extends ForeignKeyIndex<ContactResource> {}
+  public static class ForeignKeyContactIndex extends ForeignKeyIndex<ContactResource>
+      implements DatastoreEntity {
+    @Override
+    public ImmutableList<SqlEntity> toSqlEntities() {
+      return ImmutableList.of(); // not relevant in SQL
+    }
+  }

  /** The {@link ForeignKeyIndex} type for {@link DomainBase} entities. */
  @ReportedOn
  @Entity
-  public static class ForeignKeyDomainIndex extends ForeignKeyIndex<DomainBase> {}
+  public static class ForeignKeyDomainIndex extends ForeignKeyIndex<DomainBase>
+      implements DatastoreEntity {
+    @Override
+    public ImmutableList<SqlEntity> toSqlEntities() {
+      return ImmutableList.of(); // not relevant in SQL
+    }
+  }

  /** The {@link ForeignKeyIndex} type for {@link HostResource} entities. */
  @ReportedOn
  @Entity
-  public static class ForeignKeyHostIndex extends ForeignKeyIndex<HostResource> {}
+  public static class ForeignKeyHostIndex extends ForeignKeyIndex<HostResource>
+      implements DatastoreEntity {
+    @Override
+    public ImmutableList<SqlEntity> toSqlEntities() {
+      return ImmutableList.of(); // not relevant in SQL
+    }
+  }

-  static final ImmutableMap<
-          Class<? extends EppResource>, Class<? extends ForeignKeyIndex<?>>>
+  static final ImmutableMap<Class<? extends EppResource>, Class<? extends ForeignKeyIndex<?>>>
      RESOURCE_CLASS_TO_FKI_CLASS =
          ImmutableMap.of(
              ContactResource.class, ForeignKeyContactIndex.class,
              DomainBase.class, ForeignKeyDomainIndex.class,
              HostResource.class, ForeignKeyHostIndex.class);

-  @Id
-  String foreignKey;
+  @Id String foreignKey;

  /**
   * The deletion time of this {@link ForeignKeyIndex}.
@@ -90,8 +108,7 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
   * <p>This will generally be equal to the deletion time of {@link #topReference}. However, in the
   * case of a {@link HostResource} that was renamed, this field will hold the time of the rename.
   */
-  @Index
-  DateTime deletionTime;
+  @Index DateTime deletionTime;

  /**
   * The referenced resource.
@@ -101,22 +101,22 @@ public class DatastoreTransactionManager implements TransactionManager {
  }

  @Override
-  public void saveNew(Object entity) {
+  public void insert(Object entity) {
    saveEntity(entity);
  }

  @Override
-  public void saveAllNew(ImmutableCollection<?> entities) {
+  public void insertAll(ImmutableCollection<?> entities) {
    getOfy().save().entities(entities);
  }

  @Override
-  public void saveNewOrUpdate(Object entity) {
+  public void put(Object entity) {
    saveEntity(entity);
  }

  @Override
-  public void saveNewOrUpdateAll(ImmutableCollection<?> entities) {
+  public void putAll(ImmutableCollection<?> entities) {
    getOfy().save().entities(entities);
  }

@@ -131,12 +131,12 @@ public class DatastoreTransactionManager implements TransactionManager {
  }

  @Override
-  public boolean checkExists(Object entity) {
+  public boolean exists(Object entity) {
    return getOfy().load().key(Key.create(entity)).now() != null;
  }

  @Override
-  public <T> boolean checkExists(VKey<T> key) {
+  public <T> boolean exists(VKey<T> key) {
    return loadNullable(key) != null;
  }

@@ -27,14 +27,18 @@ import com.googlecode.objectify.annotation.EntitySubclass;
 import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.Index;
+import com.googlecode.objectify.annotation.OnLoad;
 import com.googlecode.objectify.annotation.Parent;
 import google.registry.model.Buildable;
 import google.registry.model.EppResource;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.ExternalMessagingName;
 import google.registry.model.annotations.ReportedOn;
+import google.registry.model.contact.ContactResource;
+import google.registry.model.domain.DomainBase;
 import google.registry.model.domain.DomainRenewData;
 import google.registry.model.eppoutput.EppResponse.ResponseData;
+import google.registry.model.host.HostResource;
 import google.registry.model.poll.PendingActionNotificationResponse.ContactPendingActionNotificationResponse;
 import google.registry.model.poll.PendingActionNotificationResponse.DomainPendingActionNotificationResponse;
 import google.registry.model.poll.PendingActionNotificationResponse.HostPendingActionNotificationResponse;
@@ -45,6 +49,7 @@ import google.registry.model.transfer.TransferResponse.ContactTransferResponse;
 import google.registry.model.transfer.TransferResponse.DomainTransferResponse;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithLongVKey;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import java.util.List;
 import java.util.Optional;
 import javax.persistence.AttributeOverride;
@@ -53,10 +58,9 @@ import javax.persistence.Column;
 import javax.persistence.DiscriminatorColumn;
 import javax.persistence.DiscriminatorValue;
 import javax.persistence.Embedded;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.Inheritance;
 import javax.persistence.InheritanceType;
+import javax.persistence.PostLoad;
 import javax.persistence.Transient;
 import org.joda.time.DateTime;

@@ -92,12 +96,11 @@ import org.joda.time.DateTime;
      @javax.persistence.Index(columnList = "eventTime")
    })
 public abstract class PollMessage extends ImmutableObject
-    implements Buildable, TransferServerApproveEntity {
+    implements Buildable, DatastoreAndSqlEntity, TransferServerApproveEntity {

  /** Entity id. */
  @Id
  @javax.persistence.Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
  @Column(name = "poll_message_id")
  Long id;

@@ -123,11 +126,11 @@ public abstract class PollMessage extends ImmutableObject

  @Ignore String hostRepoId;

-  @Ignore Long domainRevisionId;
+  @Ignore Long domainHistoryRevisionId;

-  @Ignore Long contactRevisionId;
+  @Ignore Long contactHistoryRevisionId;

-  @Ignore Long hostRevisionId;
+  @Ignore Long hostHistoryRevisionId;

  public Key<HistoryEntry> getParentKey() {
    return parent;
@@ -151,6 +154,34 @@ public abstract class PollMessage extends ImmutableObject

  public abstract ImmutableList<ResponseData> getResponseData();

+  @PostLoad
+  void postLoad() {
+    if (domainRepoId != null) {
+      parent =
+          Key.create(
+              Key.create(DomainBase.class, domainRepoId),
+              HistoryEntry.class,
+              domainHistoryRevisionId);
+    } else if (contactRepoId != null) {
+      parent =
+          Key.create(
+              Key.create(ContactResource.class, contactRepoId),
+              HistoryEntry.class,
+              contactHistoryRevisionId);
+    } else if (hostHistoryRevisionId != null) {
+      parent =
+          Key.create(
+              Key.create(HostResource.class, hostRepoId),
+              HistoryEntry.class,
+              hostHistoryRevisionId);
+    }
+  }
+
+  @OnLoad
+  void onLoad() {
+    setSqlForeignKeys(this);
+  }
+
  @Override
  public abstract VKey<? extends PollMessage> createVKey();

@@ -216,10 +247,30 @@ public abstract class PollMessage extends ImmutableObject
      checkArgumentNotNull(instance.clientId, "clientId must be specified");
      checkArgumentNotNull(instance.eventTime, "eventTime must be specified");
      checkArgumentNotNull(instance.parent, "parent must be specified");
+      checkArgumentNotNull(instance.parent.getParent(), "parent.getParent() must be specified");
+      setSqlForeignKeys(instance);
      return super.build();
    }
  }

+  private static void setSqlForeignKeys(PollMessage pollMessage) {
+    String grandparentKind = pollMessage.parent.getParent().getKind();
+    String repoId = pollMessage.parent.getParent().getName();
+    long historyRevisionId = pollMessage.parent.getId();
+    if (Key.getKind(DomainBase.class).equals(grandparentKind)) {
+      pollMessage.domainRepoId = repoId;
+      pollMessage.domainHistoryRevisionId = historyRevisionId;
+    } else if (Key.getKind(ContactResource.class).equals(grandparentKind)) {
+      pollMessage.contactRepoId = repoId;
+      pollMessage.contactHistoryRevisionId = historyRevisionId;
+    } else if (Key.getKind(HostResource.class).equals(grandparentKind)) {
+      pollMessage.hostRepoId = repoId;
+      pollMessage.hostHistoryRevisionId = historyRevisionId;
+    } else {
+      throw new IllegalArgumentException("Unknown grandparent kind: " + grandparentKind);
+    }
+  }
+
  /**
   * A one-time poll message.
   *
@@ -62,6 +62,7 @@ import google.registry.model.domain.fee.BaseFee.FeeType;
 import google.registry.model.domain.fee.Fee;
 import google.registry.model.registry.label.PremiumList;
 import google.registry.model.registry.label.ReservedList;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import google.registry.util.Idn;
 import java.util.Map;
 import java.util.Optional;
@@ -70,6 +71,13 @@ import java.util.concurrent.ExecutionException;
 import java.util.function.Predicate;
 import java.util.regex.Pattern;
 import javax.annotation.Nullable;
+import javax.persistence.AttributeOverride;
+import javax.persistence.AttributeOverrides;
+import javax.persistence.Column;
+import javax.persistence.EnumType;
+import javax.persistence.Enumerated;
+import javax.persistence.PostLoad;
+import javax.persistence.Transient;
 import org.joda.money.CurrencyUnit;
 import org.joda.money.Money;
 import org.joda.time.DateTime;
@@ -78,21 +86,31 @@ import org.joda.time.Duration;
 /** Persisted per-TLD configuration data. */
@ReportedOn
@Entity
-public class Registry extends ImmutableObject implements Buildable {
+@javax.persistence.Entity(name = "Tld")
+public class Registry extends ImmutableObject implements Buildable, DatastoreAndSqlEntity {

-  @Parent Key<EntityGroupRoot> parent = getCrossTldKey();
+  @Parent @Transient Key<EntityGroupRoot> parent = getCrossTldKey();

  /**
   * The canonical string representation of the TLD associated with this {@link Registry}, which is
   * the standard ASCII for regular TLDs and punycoded ASCII for IDN TLDs.
   */
-  @Id String tldStrId;
+  @Id
+  @javax.persistence.Id
+  @Column(name = "tld_name", nullable = false)
+  String tldStrId;

  /**
   * A duplicate of {@link #tldStrId}, to simplify BigQuery reporting since the id field becomes
   * {@code __key__.name} rather than being exported as a named field.
   */
-  String tldStr;
+  @Transient String tldStr;
+
+  /** Sets the Datastore specific field, tldStr, when the entity is loaded from Cloud SQL */
+  @PostLoad
+  void postLoad() {
+    tldStr = tldStrId;
+  }

  /** The suffix that identifies roids as belonging to this specific tld, e.g. -HOW for .how. */
  String roidSuffix;
@@ -116,6 +134,7 @@ public class Registry extends ImmutableObject implements Buildable {
  public static final Money DEFAULT_RENEW_BILLING_COST = Money.of(USD, 8);
  public static final Money DEFAULT_RESTORE_BILLING_COST = Money.of(USD, 100);
  public static final Money DEFAULT_SERVER_STATUS_CHANGE_BILLING_COST = Money.of(USD, 20);
+  public static final Money DEFAULT_REGISTRY_LOCK_OR_UNLOCK_BILLING_COST = Money.of(USD, 0);

  /** The type of TLD, which determines things like backups and escrow policy. */
  public enum TldType {
@@ -289,6 +308,7 @@ public class Registry extends ImmutableObject implements Buildable {
   * <p>All entries of this list must be valid keys for the map of {@code DnsWriter}s injected by
   * <code>@Inject Map<String, DnsWriter></code>
   */
+  @Column(nullable = false)
  Set<String> dnsWriters;

  /**
@@ -312,6 +332,7 @@ public class Registry extends ImmutableObject implements Buildable {
   * <p>Failure to do so can result in parallel writes to the {@link
   * google.registry.dns.writer.DnsWriter}, which may be dangerous depending on your implementation.
   */
+  @Column(nullable = false)
  int numDnsPublishLocks;

  /** Updates an unset numDnsPublishLocks (0) to the standard default of 1. */
@@ -327,6 +348,7 @@ public class Registry extends ImmutableObject implements Buildable {
   * <p>This will be equal to {@link #tldStr} for ASCII TLDs, but will be non-ASCII for IDN TLDs. We
   * store this in a field so that it will be retained upon import into BigQuery.
   */
+  @Column(nullable = false)
  String tldUnicode;

  /**
@@ -334,31 +356,37 @@ public class Registry extends ImmutableObject implements Buildable {
   *
   * <p>This is optional; if not configured, then information won't be exported for this TLD.
   */
-  String driveFolderId;
+  @Nullable String driveFolderId;

  /** The type of the TLD, whether it's real or for testing. */
+  @Column(nullable = false)
+  @Enumerated(EnumType.STRING)
  TldType tldType = TldType.REAL;

  /**
-   * Whether to disable invoicing for a {@link TldType#REAL} TLD.
+   * Whether to enable invoicing for this TLD.
   *
-   * <p>Note that invoicing is always disabled for {@link TldType#TEST} TLDs. Setting this field has
-   * no effect for {@link TldType#TEST} TLDs.
+   * <p>Note that this boolean is the sole determiner on whether invoices should be generated for a
+   * TLD. This applies to {@link TldType#TEST} TLDs as well.
   */
-  boolean disableInvoicing = false;
+  @Column(nullable = false)
+  boolean invoicingEnabled = false;

  /**
   * A property that transitions to different TldStates at different times. Stored as a list of
   * TldStateTransition embedded objects using the @Mapify annotation.
   */
+  @Column(nullable = false)
  @Mapify(TimedTransitionProperty.TimeMapper.class)
  TimedTransitionProperty<TldState, TldStateTransition> tldStateTransitions =
      TimedTransitionProperty.forMapify(DEFAULT_TLD_STATE, TldStateTransition.class);

  /** An automatically managed creation timestamp. */
+  @Column(nullable = false)
  CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);

  /** The set of reserved lists that are applicable to this registry. */
+  @Column(name = "reserved_list_names", nullable = false)
  Set<Key<ReservedList>> reservedLists;

  /** Retrieves an ImmutableSet of all ReservedLists associated with this tld. */
@@ -367,12 +395,15 @@ public class Registry extends ImmutableObject implements Buildable {
  }

  /** The static {@link PremiumList} for this TLD, if there is one. */
+  @Column(name = "premium_list_name", nullable = true)
  Key<PremiumList> premiumList;

  /** Should RDE upload a nightly escrow deposit for this TLD? */
+  @Column(nullable = false)
  boolean escrowEnabled = DEFAULT_ESCROW_ENABLED;

  /** Whether the pull queue that writes to authoritative DNS is paused for this TLD. */
+  @Column(nullable = false)
  boolean dnsPaused = DEFAULT_DNS_PAUSED;

  /**
@@ -381,41 +412,85 @@ public class Registry extends ImmutableObject implements Buildable {
   * <p>Domain deletes are free and effective immediately so long as they take place within this
   * amount of time following creation.
   */
+  @Column(nullable = false)
  Duration addGracePeriodLength = DEFAULT_ADD_GRACE_PERIOD;

  /** The length of the anchor tenant add grace period for this TLD. */
+  @Column(nullable = false)
  Duration anchorTenantAddGracePeriodLength = DEFAULT_ANCHOR_TENANT_ADD_GRACE_PERIOD;

  /** The length of the auto renew grace period for this TLD. */
+  @Column(nullable = false)
  Duration autoRenewGracePeriodLength = DEFAULT_AUTO_RENEW_GRACE_PERIOD;

  /** The length of the redemption grace period for this TLD. */
+  @Column(nullable = false)
  Duration redemptionGracePeriodLength = DEFAULT_REDEMPTION_GRACE_PERIOD;

  /** The length of the renew grace period for this TLD. */
+  @Column(nullable = false)
  Duration renewGracePeriodLength = DEFAULT_RENEW_GRACE_PERIOD;

  /** The length of the transfer grace period for this TLD. */
+  @Column(nullable = false)
  Duration transferGracePeriodLength = DEFAULT_TRANSFER_GRACE_PERIOD;

  /** The length of time before a transfer is automatically approved for this TLD. */
+  @Column(nullable = false)
  Duration automaticTransferLength = DEFAULT_AUTOMATIC_TRANSFER_LENGTH;

  /** The length of time a domain spends in the non-redeemable pending delete phase for this TLD. */
+  @Column(nullable = false)
  Duration pendingDeleteLength = DEFAULT_PENDING_DELETE_LENGTH;

  /** The currency unit for all costs associated with this TLD. */
+  @Column(nullable = false)
  CurrencyUnit currency = DEFAULT_CURRENCY;

  /** The per-year billing cost for registering a new domain name. */
+  @AttributeOverrides({
+    @AttributeOverride(
+        name = "money.amount",
+        column = @Column(name = "create_billing_cost_amount")),
+    @AttributeOverride(
+        name = "money.currency",
+        column = @Column(name = "create_billing_cost_currency"))
+  })
  Money createBillingCost = DEFAULT_CREATE_BILLING_COST;

  /** The one-time billing cost for restoring a domain name from the redemption grace period. */
+  @AttributeOverrides({
+    @AttributeOverride(
+        name = "money.amount",
+        column = @Column(name = "restore_billing_cost_amount")),
+    @AttributeOverride(
+        name = "money.currency",
+        column = @Column(name = "restore_billing_cost_currency"))
+  })
  Money restoreBillingCost = DEFAULT_RESTORE_BILLING_COST;

  /** The one-time billing cost for changing the server status (i.e. lock). */
+  @AttributeOverrides({
+    @AttributeOverride(
+        name = "money.amount",
+        column = @Column(name = "server_status_change_billing_cost_amount")),
+    @AttributeOverride(
+        name = "money.currency",
+        column = @Column(name = "server_status_change_billing_cost_currency"))
+  })
  Money serverStatusChangeBillingCost = DEFAULT_SERVER_STATUS_CHANGE_BILLING_COST;

+  /** The one-time billing cost for a registry lock/unlock action initiated by a registrar. */
+  @AttributeOverrides({
+      @AttributeOverride(
+          name = "money.amount",
+          column = @Column(name = "registry_lock_or_unlock_cost_amount")),
+      @AttributeOverride(
+          name = "money.currency",
+          column = @Column(name = "registry_lock_or_unlock_cost_currency"))
+  })
+  Money registryLockOrUnlockBillingCost = DEFAULT_REGISTRY_LOCK_OR_UNLOCK_BILLING_COST;
+
  /**
   * A property that transitions to different renew billing costs at different times. Stored as a
   * list of BillingCostTransition embedded objects using the @Mapify annotation.
@@ -424,11 +499,13 @@ public class Registry extends ImmutableObject implements Buildable {
   * name. This cost is also used to compute costs for transfers, since each transfer includes a
   * renewal to ensure transfers have a cost.
   */
+  @Column(nullable = false)
  @Mapify(TimedTransitionProperty.TimeMapper.class)
  TimedTransitionProperty<Money, BillingCostTransition> renewBillingCostTransitions =
      TimedTransitionProperty.forMapify(DEFAULT_RENEW_BILLING_COST, BillingCostTransition.class);

  /** A property that tracks the EAP fee schedule (if any) for the TLD. */
+  @Column(nullable = false)
  @Mapify(TimedTransitionProperty.TimeMapper.class)
  TimedTransitionProperty<Money, BillingCostTransition> eapFeeSchedule =
      TimedTransitionProperty.forMapify(DEFAULT_EAP_BILLING_COST, BillingCostTransition.class);
@@ -437,13 +514,14 @@ public class Registry extends ImmutableObject implements Buildable {
  String lordnUsername;

  /** The end of the claims period (at or after this time, claims no longer applies). */
+  @Column(nullable = false)
  DateTime claimsPeriodEnd = END_OF_TIME;

  /** An allow list of clients allowed to be used on domains on this TLD (ignored if empty). */
-  Set<String> allowedRegistrantContactIds;
+  @Nullable Set<String> allowedRegistrantContactIds;

  /** An allow list of hosts allowed to be used on domains on this TLD (ignored if empty). */
-  Set<String> allowedFullyQualifiedHostNames;
+  @Nullable Set<String> allowedFullyQualifiedHostNames;

  public String getTldStr() {
    return tldStr;
@@ -566,6 +644,11 @@ public class Registry extends ImmutableObject implements Buildable {
    return serverStatusChangeBillingCost;
  }

+  /** Returns the cost of a registry lock/unlock. */
+  public Money getRegistryLockOrUnlockBillingCost() {
+    return registryLockOrUnlockBillingCost;
+  }
+
  public ImmutableSortedMap<DateTime, TldState> getTldStateTransitions() {
    return tldStateTransitions.toValueMap();
  }
@@ -646,8 +729,8 @@ public class Registry extends ImmutableObject implements Buildable {
      return this;
    }

-    public Builder setDisableInvoicing(boolean disableInvoicing) {
-      getInstance().disableInvoicing = disableInvoicing;
+    public Builder setInvoicingEnabled(boolean invoicingEnabled) {
+      getInstance().invoicingEnabled = invoicingEnabled;
      return this;
    }

@@ -867,6 +950,12 @@ public class Registry extends ImmutableObject implements Buildable {
      return this;
    }

+    public Builder setRegistryLockOrUnlockBillingCost(Money amount) {
+      checkArgument(amount.isPositiveOrZero(), "Registry lock/unlock cost cannot be negative");
+      getInstance().registryLockOrUnlockBillingCost = amount;
+      return this;
+    }
+
    public Builder setLordnUsername(String username) {
      getInstance().lordnUsername = username;
      return this;
@@ -918,6 +1007,9 @@ public class Registry extends ImmutableObject implements Buildable {
      checkArgument(
          instance.getServerStatusChangeCost().getCurrencyUnit().equals(instance.currency),
          "Server status change cost must be in the registry's currency");
+      checkArgument(
+          instance.getRegistryLockOrUnlockBillingCost().getCurrencyUnit().equals(instance.currency),
+          "Registry lock/unlock cost must be in the registry's currency");
      Predicate<Money> currencyCheck =
          (Money money) -> money.getCurrencyUnit().equals(instance.currency);
      checkArgument(
@@ -114,7 +114,7 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
  /** Virtual parent entity for premium list entry entities associated with a single revision. */
  @ReportedOn
  @Entity
-  public static class PremiumListRevision extends ImmutableObject {
+  public static class PremiumListRevision extends ImmutableObject implements DatastoreEntity {

    @Parent Key<PremiumList> parent;

@@ -171,6 +171,11 @@ public final class PremiumList extends BaseDomainLabelList<Money, PremiumList.Pr
      }
      return revision;
    }
+
+    @Override
+    public ImmutableList<SqlEntity> toSqlEntities() {
+      return ImmutableList.of(); // not persisted in SQL
+    }
  }

  /**
@@ -43,7 +43,7 @@ public class ReservedListDualWriteDao {

  /** Persist a new reserved list to Cloud SQL. */
  public static void save(ReservedList reservedList) {
-    ofyTm().transact(() -> ofyTm().saveNewOrUpdate(reservedList));
+    ofyTm().transact(() -> ofyTm().put(reservedList));
    try {
      logger.atInfo().log("Saving reserved list %s to Cloud SQL", reservedList.getName());
      ReservedListSqlDao.save(reservedList);
@@ -31,7 +31,7 @@ public class ReservedListSqlDao {
  /** Persist a new reserved list to Cloud SQL. */
  public static void save(ReservedList reservedList) {
    checkArgumentNotNull(reservedList, "Must specify reservedList");
-    jpaTm().transact(() -> jpaTm().saveNew(reservedList));
+    jpaTm().transact(() -> jpaTm().insert(reservedList));
  }

  /**
@@ -19,8 +19,17 @@ import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;

 import com.google.common.collect.ImmutableSet;
 import com.googlecode.objectify.annotation.Embed;
+import com.googlecode.objectify.annotation.Ignore;
 import google.registry.model.Buildable;
 import google.registry.model.ImmutableObject;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
+import javax.persistence.Column;
+import javax.persistence.Entity;
+import javax.persistence.EnumType;
+import javax.persistence.Enumerated;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.Id;
 import org.joda.time.DateTime;

 /**
@@ -34,9 +43,17 @@ import org.joda.time.DateTime;
 * uses HistoryEntry.otherClientId because the losing party in a transfer is always the otherClient.
 */
@Embed
-public class DomainTransactionRecord extends ImmutableObject implements Buildable {
+@Entity
+public class DomainTransactionRecord extends ImmutableObject
+    implements Buildable, DatastoreAndSqlEntity {
+
+  @Id
+  @Ignore
+  @GeneratedValue(strategy = GenerationType.IDENTITY)
+  Long id;

  /** The TLD this record operates on. */
+  @Column(nullable = false)
  String tld;

  /**
@@ -50,9 +67,12 @@ public class DomainTransactionRecord extends ImmutableObject implements Buildabl
   *     href="https://www.icann.org/resources/unthemed-pages/registry-agmt-appc-10-2001-05-11-en">
   *     Grace period spec</a>
   */
+  @Column(nullable = false)
  DateTime reportingTime;

  /** The transaction report field we add reportAmount to for this registrar. */
+  @Column(nullable = false)
+  @Enumerated(value = EnumType.STRING)
  TransactionReportField reportField;

  /**
@@ -67,6 +87,7 @@ public class DomainTransactionRecord extends ImmutableObject implements Buildabl
   * original SUCCESSFUL transfer counters. Finally, if we explicitly allow a transfer, the report
   * amount is 0, as we've already counted the transfer in the original request.
   */
+  @Column(nullable = false)
  Integer reportAmount;

  /**
@@ -18,6 +18,7 @@ import static com.googlecode.objectify.Key.getKind;
 import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;

 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
 import com.googlecode.objectify.Key;
 import com.googlecode.objectify.annotation.Entity;
@@ -40,17 +41,18 @@ import google.registry.model.host.HostHistory;
 import google.registry.model.host.HostResource;
 import google.registry.persistence.VKey;
 import google.registry.persistence.WithStringVKey;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
 import java.util.Set;
 import javax.annotation.Nullable;
+import javax.persistence.Access;
+import javax.persistence.AccessType;
 import javax.persistence.AttributeOverride;
 import javax.persistence.AttributeOverrides;
 import javax.persistence.Column;
 import javax.persistence.EnumType;
 import javax.persistence.Enumerated;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
 import javax.persistence.MappedSuperclass;
-import javax.persistence.SequenceGenerator;
 import javax.persistence.Transient;
 import org.joda.time.DateTime;

@@ -59,7 +61,8 @@ import org.joda.time.DateTime;
@Entity
@MappedSuperclass
@WithStringVKey // TODO(b/162229294): This should be resolved during the course of that bug
-public class HistoryEntry extends ImmutableObject implements Buildable {
+@Access(AccessType.FIELD)
+public class HistoryEntry extends ImmutableObject implements Buildable, DatastoreEntity {

  /** Represents the type of history entry. */
  public enum Type {
@@ -110,17 +113,13 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
    SYNTHETIC
  }

-  /** The autogenerated id of this event. */
-  @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "HistorySequenceGenerator")
-  @SequenceGenerator(
-      name = "HistorySequenceGenerator",
-      sequenceName = "history_id_sequence",
-      allocationSize = 1)
-  @Id
-  @javax.persistence.Id
-  @Column(name = "historyRevisionId")
-  @VisibleForTesting
-  public Long id;
+  /**
+   * The autogenerated id of this event. Note that, this field is marked as {@link Transient} in the
+   * SQL schema, this is because the child class of {@link HistoryEntry}, e.g. {@link
+   * DomainHistory}, uses a composite primary key which the id is part of, and Hibernate requires
+   * that all the {@link javax.persistence.Id} fields must be put in the exact same class.
+   */
+  @Id @Transient @VisibleForTesting public Long id;

  /** The resource this event mutated. */
  @Parent @Transient protected Key<? extends EppResource> parent;
@@ -179,7 +178,7 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
  boolean bySuperuser;

  /** Reason for the change. */
-  @Column(nullable = false, name = "historyReason")
+  @Column(name = "historyReason")
  String reason;

  /** Whether this change was requested by a registrar. */
@@ -196,8 +195,17 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
  @Transient // domain-specific
  Set<DomainTransactionRecord> domainTransactionRecords;

-  public Long getId() {
-    return id;
+  public long getId() {
+    // For some reason, Hibernate throws NPE during some initialization phase if we don't deal with
+    // the null case. Setting the id to 0L when it is null should be fine because 0L for primitive
+    // type is considered as null for wrapper class in the Hibernate context.
+    return id == null ? 0L : id;
+  }
+
+  /** This method exists solely to satisfy Hibernate. Use the {@link Builder} instead. */
+  @SuppressWarnings("UnusedMethod")
+  private void setId(long id) {
+    this.id = id;
  }

  public Key<? extends EppResource> getParent() {
@@ -246,10 +254,28 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
    return requestedByRegistrar;
  }

-  public ImmutableSet<DomainTransactionRecord> getDomainTransactionRecords() {
+  public Set<DomainTransactionRecord> getDomainTransactionRecords() {
    return nullToEmptyImmutableCopy(domainTransactionRecords);
  }

+  /** This method exists solely to satisfy Hibernate. Use the {@link Builder} instead. */
+  @SuppressWarnings("UnusedMethod")
+  private void setPeriod(Period period) {
+    this.period = period;
+  }
+
+  /** This method exists solely to satisfy Hibernate. Use the {@link Builder} instead. */
+  @SuppressWarnings("UnusedMethod")
+  private void setOtherRegistrarId(String otherRegistrarId) {
+    this.otherClientId = otherRegistrarId;
+  }
+
+  /** This method exists solely to satisfy Hibernate. Use the {@link Builder} instead. */
+  @SuppressWarnings("UnusedMethod")
+  private void setDomainTransactionRecords(Set<DomainTransactionRecord> domainTransactionRecords) {
+    this.domainTransactionRecords = ImmutableSet.copyOf(domainTransactionRecords);
+  }
+
  public static VKey<HistoryEntry> createVKey(Key<HistoryEntry> key) {
    // TODO(b/159207551): This will likely need some revision.  As it stands, this method was
    // introduced purely to facilitate testing of VKey specialization in VKeyTranslatorFactory.
@@ -279,26 +305,13 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
    // can't use a switch statement since we're calling getKind()
    if (parentKind.equals(getKind(DomainBase.class))) {
      resultEntity =
-          new DomainHistory.Builder()
-              .copyFrom(this)
-              .setDomainRepoId(
-                  VKey.create(DomainBase.class, parent.getName(), (Key<DomainBase>) parent))
-              .build();
+          new DomainHistory.Builder().copyFrom(this).setDomainRepoId(parent.getName()).build();
    } else if (parentKind.equals(getKind(HostResource.class))) {
      resultEntity =
-          new HostHistory.Builder()
-              .copyFrom(this)
-              .setHostRepoId(
-                  VKey.create(HostResource.class, parent.getName(), (Key<HostResource>) parent))
-              .build();
+          new HostHistory.Builder().copyFrom(this).setHostRepoId(parent.getName()).build();
    } else if (parentKind.equals(getKind(ContactResource.class))) {
      resultEntity =
-          new ContactHistory.Builder()
-              .copyFrom(this)
-              .setContactRepoId(
-                  VKey.create(
-                      ContactResource.class, parent.getName(), (Key<ContactResource>) parent))
-              .build();
+          new ContactHistory.Builder().copyFrom(this).setContactRepoId(parent.getName()).build();
    } else {
      throw new IllegalStateException(
          String.format("Unknown kind of HistoryEntry parent %s", parentKind));
@@ -306,6 +319,12 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
    return resultEntity;
  }

+  // In SQL, save the child type
+  @Override
+  public ImmutableList<SqlEntity> toSqlEntities() {
+    return ImmutableList.of((SqlEntity) toChildHistoryEntity());
+  }
+
  /** A builder for {@link HistoryEntry} since it is immutable */
  public static class Builder<T extends HistoryEntry, B extends Builder<?, ?>>
      extends GenericBuilder<T, B> {
@@ -330,7 +349,10 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
      setBySuperuser(historyEntry.bySuperuser);
      setReason(historyEntry.reason);
      setRequestedByRegistrar(historyEntry.requestedByRegistrar);
-      setDomainTransactionRecords(nullToEmptyImmutableCopy(historyEntry.domainTransactionRecords));
+      setDomainTransactionRecords(
+          historyEntry.domainTransactionRecords == null
+              ? null
+              : ImmutableSet.copyOf(historyEntry.domainTransactionRecords));
      return thisCastToDerived();
    }

@@ -349,6 +371,7 @@ public class HistoryEntry extends ImmutableObject implements Buildable {
      return thisCastToDerived();
    }

+    // Until we move completely to SQL, override this in subclasses (e.g. HostHistory) to set VKeys
    public B setParent(Key<? extends EppResource> parent) {
      getInstance().parent = parent;
      return thisCastToDerived();
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-package google.registry.schema.tmch;
+package google.registry.model.tmch;

 import static google.registry.config.RegistryConfig.getDomainLabelListCacheDuration;
 import static google.registry.model.CacheUtils.tryMemoizeWithExpiration;
@@ -24,28 +24,28 @@ import google.registry.util.NonFinalForTesting;
 import java.util.Optional;
 import javax.persistence.EntityManager;

-/** Data access object for {@link ClaimsList}. */
+/** Data access object for {@link ClaimsListShard}. */
 public class ClaimsListDao {

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

  /** In-memory cache for claims list. */
  @NonFinalForTesting
-  private static Supplier<Optional<ClaimsList>> cacheClaimsList =
+  private static Supplier<Optional<ClaimsListShard>> cacheClaimsList =
      tryMemoizeWithExpiration(getDomainLabelListCacheDuration(), ClaimsListDao::getLatestRevision);

-  private static void save(ClaimsList claimsList) {
+  private static void save(ClaimsListShard claimsList) {
    jpaTm().transact(() -> jpaTm().getEntityManager().persist(claimsList));
  }

  /**
-   * Try to save the given {@link ClaimsList} into Cloud SQL. If the save fails, the error will be
-   * logged but no exception will be thrown.
+   * Try to save the given {@link ClaimsListShard} into Cloud SQL. If the save fails, the error will
+   * be logged but no exception will be thrown.
   *
   * <p>This method is used during the dual-write phase of database migration as Datastore is still
   * the authoritative database.
   */
-  public static void trySave(ClaimsList claimsList) {
+  static void trySave(ClaimsListShard claimsList) {
    try {
      ClaimsListDao.save(claimsList);
      logger.atInfo().log(
@@ -57,12 +57,12 @@ public class ClaimsListDao {
  }

  /**
-   * Returns the most recent revision of the {@link ClaimsList} in Cloud SQL, if it exists.
+   * Returns the most recent revision of the {@link ClaimsListShard} in Cloud SQL, if it exists.
   * TODO(shicong): Change this method to package level access after dual-read phase.
   * ClaimsListShard uses this method to retrieve claims list in Cloud SQL for the comparison, and
   * ClaimsListShard is not in this package.
   */
-  public static Optional<ClaimsList> getLatestRevision() {
+  public static Optional<ClaimsListShard> getLatestRevision() {
    return jpaTm()
        .transact(
            () -> {
@@ -73,15 +73,15 @@ public class ClaimsListDao {
              return em.createQuery(
                      "FROM ClaimsList cl LEFT JOIN FETCH cl.labelsToKeys WHERE cl.revisionId ="
                          + " :revisionId",
-                      ClaimsList.class)
+                      ClaimsListShard.class)
                  .setParameter("revisionId", revisionId)
                  .getResultStream()
                  .findFirst();
            });
  }

-  /** Returns the most recent revision of the {@link ClaimsList}, from cache. */
-  public static Optional<ClaimsList> getLatestRevisionCached() {
+  /** Returns the most recent revision of the {@link ClaimsListShard}, from cache. */
+  public static Optional<ClaimsListShard> getLatestRevisionCached() {
    return cacheClaimsList.get();
  }

@@ -40,15 +40,15 @@ import com.googlecode.objectify.annotation.Id;
 import com.googlecode.objectify.annotation.Ignore;
 import com.googlecode.objectify.annotation.OnSave;
 import com.googlecode.objectify.annotation.Parent;
+import google.registry.model.CreateAutoTimestamp;
 import google.registry.model.ImmutableObject;
 import google.registry.model.annotations.NotBackedUp;
 import google.registry.model.annotations.NotBackedUp.Reason;
 import google.registry.model.annotations.VirtualEntity;
 import google.registry.model.common.CrossTldSingleton;
+import google.registry.schema.replay.DatastoreAndSqlEntity;
 import google.registry.schema.replay.DatastoreEntity;
 import google.registry.schema.replay.SqlEntity;
-import google.registry.schema.tmch.ClaimsList;
-import google.registry.schema.tmch.ClaimsListDao;
 import google.registry.util.CollectionUtils;
 import google.registry.util.Concurrent;
 import google.registry.util.Retrier;
@@ -59,6 +59,15 @@ import java.util.Map;
 import java.util.Optional;
 import java.util.concurrent.Callable;
 import javax.annotation.Nullable;
+import javax.persistence.CollectionTable;
+import javax.persistence.Column;
+import javax.persistence.ElementCollection;
+import javax.persistence.GeneratedValue;
+import javax.persistence.GenerationType;
+import javax.persistence.JoinColumn;
+import javax.persistence.MapKeyColumn;
+import javax.persistence.Table;
+import javax.persistence.Transient;
 import org.joda.time.DateTime;

 /**
@@ -74,33 +83,66 @@ import org.joda.time.DateTime;
 * 10MB per transaction limit.
 *
 * <p>Therefore, it is never OK to save an instance of this class directly to Datastore. Instead you
- * must use the {@link #save} method to do it for you.
+ * must use the {@link #saveToDatastore} method to do it for you.
+ *
+ * <p>Note that the primary key of this entity is {@link #revisionId}, which is auto-generated by
+ * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
+ * succeeds, we will end up with having two exact same claims list with only different {@link
+ * #revisionId}. However, this is not an actual problem because we only use the claims list with
+ * highest {@link #revisionId}.
+ *
+ * <p>TODO(b/162007765): Rename the class to ClaimsList and remove Datastore related fields and
+ * methods.
 */
@Entity
@NotBackedUp(reason = Reason.EXTERNALLY_SOURCED)
-public class ClaimsListShard extends ImmutableObject implements DatastoreEntity {
+@javax.persistence.Entity(name = "ClaimsList")
+@Table
+public class ClaimsListShard extends ImmutableObject implements DatastoreAndSqlEntity {

  private static final FluentLogger logger = FluentLogger.forEnclosingClass();

  /** The number of claims list entries to store per shard. */
  private static final int SHARD_SIZE = 10000;

-  @Id
-  long id;
+  @Transient @Id long id;

-  @Parent
-  Key<ClaimsListRevision> parent;
+  @Transient @Parent Key<ClaimsListRevision> parent;

-  /** When the claims list was last updated. */
+  @Ignore
+  @javax.persistence.Id
+  @GeneratedValue(strategy = GenerationType.IDENTITY)
+  Long revisionId;
+
+  @Ignore
+  @Column(nullable = false)
+  CreateAutoTimestamp creationTimestamp = CreateAutoTimestamp.create(null);
+
+  /**
+   * When the claims list was last updated.
+   *
+   * <p>Note that the value of this field is parsed from the claims list file(See this <a
+   * href="https://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.1">RFC</>), it is
+   * the DNL List creation datetime from the rfc. Since this field has been used by Datastore, we
+   * cannot change its name until we finish the migration.
+   *
+   * <p>TODO(b/166784536): Rename this field to tmdbGenerationTime.
+   */
+  @Column(name = "tmdb_generation_time", nullable = false)
  DateTime creationTime;

  /** A map from labels to claims keys. */
  @EmbedMap
+  @ElementCollection
+  @CollectionTable(
+      name = "ClaimsEntry",
+      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
+  @MapKeyColumn(name = "domainLabel", nullable = false)
+  @Column(name = "claimKey", nullable = false)
  Map<String, String> labelsToKeys;

  /** Indicates that this is a shard rather than a "full" list. */
-  @Ignore
-  boolean isShard = false;
+  @Ignore @Transient boolean isShard = false;

  private static final Retrier LOADER_RETRIER = new Retrier(new SystemSleeper(), 2);

@@ -164,10 +206,10 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
        return datastoreList;
      };

-  private static final void loadAndCompareCloudSqlList(ClaimsListShard datastoreList) {
-    Optional<ClaimsList> maybeCloudSqlList = ClaimsListDao.getLatestRevision();
+  private static void loadAndCompareCloudSqlList(ClaimsListShard datastoreList) {
+    Optional<ClaimsListShard> maybeCloudSqlList = ClaimsListDao.getLatestRevision();
    if (maybeCloudSqlList.isPresent()) {
-      ClaimsList cloudSqlList = maybeCloudSqlList.get();
+      ClaimsListShard cloudSqlList = maybeCloudSqlList.get();
      MapDifference<String, String> diff =
          Maps.difference(datastoreList.labelsToKeys, cloudSqlList.getLabelsToKeys());
      if (!diff.areEqual()) {
@@ -206,15 +248,34 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
      memoizeWithShortExpiration(
          () -> LOADER_RETRIER.callWithRetry(LOADER_CALLABLE, IllegalStateException.class));

-  public DateTime getCreationTime() {
+  /** Returns the revision id of this claims list, or throws exception if it is null. */
+  public Long getRevisionId() {
+    checkState(
+        revisionId != null, "revisionId is null because it is not persisted in the database");
+    return revisionId;
+  }
+
+  /**
+   * Returns the time when the external TMDB service generated this revision of the claims list.
+   *
+   * @see <a href="https://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.1">DNL List
+   *     creation datetime</a>
+   */
+  public DateTime getTmdbGenerationTime() {
    return creationTime;
  }

+  /** Returns the creation time of this claims list. */
+  public DateTime getCreationTimestamp() {
+    return creationTimestamp.getTimestamp();
+  }
+
  /** Returns the claim key for a given domain if there is one, empty otherwise. */
  public Optional<String> getClaimKey(String label) {
    return Optional.ofNullable(labelsToKeys.get(label));
  }

+  /** Returns an {@link Map} mapping domain label to its lookup key. */
  public ImmutableMap<String, String> getLabelsToKeys() {
    return ImmutableMap.copyOf(labelsToKeys);
  }
@@ -229,11 +290,12 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
   * switching over to using them atomically, then deleting the old ones.
   */
  public void save() {
-    save(SHARD_SIZE);
+    saveToDatastore(SHARD_SIZE);
+    ClaimsListDao.trySave(this);
  }

  @VisibleForTesting
-  void save(int shardSize) {
+  void saveToDatastore(int shardSize) {
    // Figure out what the next versionId should be based on which ones already exist.
    final Key<ClaimsListRevision> oldRevision = getCurrentRevision();
    final Key<ClaimsListRevision> parentKey = ClaimsListRevision.createKey();
@@ -270,10 +332,11 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
            });
  }

-  public static ClaimsListShard create(DateTime creationTime, Map<String, String> labelsToKeys) {
+  public static ClaimsListShard create(
+      DateTime tmdbGenerationTime, Map<String, String> labelsToKeys) {
    ClaimsListShard instance = new ClaimsListShard();
    instance.id = allocateId();
-    instance.creationTime = checkNotNull(creationTime);
+    instance.creationTime = checkNotNull(tmdbGenerationTime);
    instance.labelsToKeys = checkNotNull(labelsToKeys);
    return instance;
  }
@@ -292,11 +355,6 @@ public class ClaimsListShard extends ImmutableObject implements DatastoreEntity
    }
  }

-  @Override
-  public ImmutableList<SqlEntity> toSqlEntities() {
-    return ImmutableList.of(); // ClaimsLists are dually written
-  }
-
  /** Virtual parent entity for claims list shards of a specific revision. */
  @Entity
  @VirtualEntity
@@ -88,7 +88,7 @@ public class VKey<T> extends ImmutableObject implements Serializable {
   */
  public static <T> VKey<T> create(Class<T> kind, long id) {
    checkArgument(
-        kind.isAssignableFrom(BackupGroupRoot.class),
+        BackupGroupRoot.class.isAssignableFrom(kind),
        "The kind %s is not a BackupGroupRoot and thus needs its entire entity group chain"
            + " specified in a parent",
        kind.getCanonicalName());
@@ -106,13 +106,90 @@ public class VKey<T> extends ImmutableObject implements Serializable {
   */
  public static <T> VKey<T> create(Class<T> kind, String name) {
    checkArgument(
-        kind.isAssignableFrom(BackupGroupRoot.class),
+        BackupGroupRoot.class.isAssignableFrom(kind),
        "The kind %s is not a BackupGroupRoot and thus needs its entire entity group chain"
            + " specified in a parent",
        kind.getCanonicalName());
    return new VKey<T>(kind, Key.create(kind, name), name);
  }

+  /**
+   * Returns a clone with an ofy key restored from {@code ancestors}.
+   *
+   * <p>The arguments should generally consist of pairs of Class and value, where the Class is the
+   * kind of the ancestor key and the value is either a String or a Long.
+   *
+   * <p>For example, to restore the objectify key for
+   * DomainBase("COM-1234")/HistoryEntry(123)/PollEvent(567), one might use:
+   *
+   * <pre>{@code
+   * pollEvent.restoreOfy(DomainBase.class, "COM-1234", HistoryEntry.class, 567)
+   * }</pre>
+   *
+   * <p>The final key id or name is obtained from the SQL key. It is assumed that this value must be
+   * either a long integer or a {@code String} and that this proper identifier for the objectify
+   * key.
+   *
+   * <p>As a special case, an objectify Key may be used as the first ancestor instead of a Class,
+   * value pair.
+   */
+  public VKey<T> restoreOfy(Object... ancestors) {
+    Class lastClass = null;
+    Key<?> lastKey = null;
+    for (Object ancestor : ancestors) {
+      if (ancestor instanceof Class) {
+        if (lastClass != null) {
+          throw new IllegalArgumentException(ancestor + " used as a key value.");
+        }
+        lastClass = (Class) ancestor;
+        continue;
+      } else if (ancestor instanceof Key) {
+        if (lastKey != null) {
+          throw new IllegalArgumentException(
+              "Objectify keys may only be used for the first argument");
+        }
+        lastKey = (Key) ancestor;
+        continue;
+      }
+
+      // The argument should be a value.
+      if (lastClass == null) {
+        throw new IllegalArgumentException("Argument " + ancestor + " should be a class.");
+      }
+      if (ancestor instanceof Long) {
+        lastKey = Key.create(lastKey, lastClass, (Long) ancestor);
+      } else if (ancestor instanceof String) {
+        lastKey = Key.create(lastKey, lastClass, (String) ancestor);
+      } else {
+        throw new IllegalArgumentException("Key value " + ancestor + " must be a string or long.");
+      }
+      lastClass = null;
+    }
+
+    // Make sure we didn't end up with a dangling class with no value.
+    if (lastClass != null) {
+      throw new IllegalArgumentException("Missing value for last key of type " + lastClass);
+    }
+
+    Object sqlKey = getSqlKey();
+    Key<T> ofyKey =
+        sqlKey instanceof Long
+            ? Key.create(lastKey, getKind(), (Long) sqlKey)
+            : Key.create(lastKey, getKind(), (String) sqlKey);
+
+    return VKey.create((Class<T>) getKind(), sqlKey, ofyKey);
+  }
+
+  /**
+   * Returns a clone of {@code key} with an ofy key restored from {@code ancestors}.
+   *
+   * <p>This is the static form of the method restoreOfy() above. If {@code key} is null, it returns
+   * null.
+   */
+  public static <T> VKey<T> restoreOfyFrom(@Nullable VKey<T> key, Object... ancestors) {
+    return key == null ? null : key.restoreOfy(ancestors);
+  }
+
  /** Returns the type of the entity. */
  public Class<? extends T> getKind() {
    return this.kind;
@@ -32,7 +32,11 @@ public class CreateAutoTimestampConverter
    implements AttributeConverter<CreateAutoTimestamp, Timestamp> {

  @Override
-  public Timestamp convertToDatabaseColumn(CreateAutoTimestamp entity) {
+  @Nullable
+  public Timestamp convertToDatabaseColumn(@Nullable CreateAutoTimestamp entity) {
+    if (entity == null) {
+      return null;
+    }
    DateTime dateTime = firstNonNull(entity.getTimestamp(), jpaTm().getTransactionTime());
    return Timestamp.from(DateTimeUtils.toZonedDateTime(dateTime).toInstant());
  }
@@ -40,11 +40,17 @@ public class DurationConverter implements AttributeConverter<Duration, PGInterva
    if (duration == null) {
      return new PGInterval();
    }
+    // When the period is created from duration by calling duration.toPeriod(), only precise fields
+    // in the period type will be used. Thus, only the hour, minute, second and millisecond fields
+    // on the period will be used. The year, month, week and day fields will not be populated:
+    //   1. If the duration is small, less than one day, then this method will just set
+    //      hours/minutes/seconds correctly.
+    //   2. If the duration is larger than one day then all the remaining duration will
+    //      be stored in the largest available field, hours in this case.
+    // So, when we convert the period to a PGInterval instance, we set the days field by extracting
+    // it from period's hours field.
+    Period period = duration.toPeriod();
    PGInterval interval = new PGInterval();
-    Period period = new Period(duration);
-    // For some reason when the period is created from the duration, it does not set days, but
-    // instead just a total number of hours. Years and months are not created because those can
-    // differ in length of milliseconds.
    interval.setDays(period.getHours() / 24);
    interval.setHours(period.getHours() % 24);
    interval.setMinutes(period.getMinutes());
@@ -0,0 +1,37 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.converter;
+
+import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
+
+import com.googlecode.objectify.Key;
+import google.registry.model.registry.label.PremiumList;
+import javax.persistence.AttributeConverter;
+import javax.persistence.Converter;
+
+/** JPA converter for a {@link Key} containing a {@link PremiumList} */
+@Converter(autoApply = true)
+public class PremiumListKeyConverter implements AttributeConverter<Key<PremiumList>, String> {
+
+  @Override
+  public String convertToDatabaseColumn(Key<PremiumList> attribute) {
+    return (attribute == null) ? null : attribute.getName();
+  }
+
+  @Override
+  public Key<PremiumList> convertToEntityAttribute(String dbData) {
+    return (dbData == null) ? null : Key.create(getCrossTldKey(), PremiumList.class, dbData);
+  }
+}
@@ -0,0 +1,36 @@
+// Copyright 2020 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.persistence.converter;
+
+import static google.registry.model.common.EntityGroupRoot.getCrossTldKey;
+
+import com.googlecode.objectify.Key;
+import google.registry.model.registry.label.ReservedList;
+import javax.persistence.Converter;
+
+/** JPA converter for a set of {@link Key} containing a {@link ReservedList} */
+@Converter(autoApply = true)
+public class ReservedListKeySetConverter extends StringSetConverterBase<Key<ReservedList>> {
+
+  @Override
+  String toString(Key<ReservedList> key) {
+    return key.getName();
+  }
+
+  @Override
+  Key<ReservedList> fromString(String value) {
+    return Key.create(getCrossTldKey(), ReservedList.class, value);
+  }
+}
@@ -224,7 +224,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  }

  @Override
-  public void saveNew(Object entity) {
+  public void insert(Object entity) {
    checkArgumentNotNull(entity, "entity must be specified");
    assertInTransaction();
    getEntityManager().persist(entity);
@@ -232,14 +232,14 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  }

  @Override
-  public void saveAllNew(ImmutableCollection<?> entities) {
+  public void insertAll(ImmutableCollection<?> entities) {
    checkArgumentNotNull(entities, "entities must be specified");
    assertInTransaction();
-    entities.forEach(this::saveNew);
+    entities.forEach(this::insert);
  }

  @Override
-  public void saveNewOrUpdate(Object entity) {
+  public void put(Object entity) {
    checkArgumentNotNull(entity, "entity must be specified");
    assertInTransaction();
    getEntityManager().merge(entity);
@@ -247,17 +247,17 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  }

  @Override
-  public void saveNewOrUpdateAll(ImmutableCollection<?> entities) {
+  public void putAll(ImmutableCollection<?> entities) {
    checkArgumentNotNull(entities, "entities must be specified");
    assertInTransaction();
-    entities.forEach(this::saveNewOrUpdate);
+    entities.forEach(this::put);
  }

  @Override
  public void update(Object entity) {
    checkArgumentNotNull(entity, "entity must be specified");
    assertInTransaction();
-    checkArgument(checkExists(entity), "Given entity does not exist");
+    checkArgument(exists(entity), "Given entity does not exist");
    getEntityManager().merge(entity);
    transactionInfo.get().addUpdate(entity);
  }
@@ -270,22 +270,22 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  }

  @Override
-  public <T> boolean checkExists(VKey<T> key) {
+  public <T> boolean exists(VKey<T> key) {
    checkArgumentNotNull(key, "key must be specified");
    EntityType<?> entityType = getEntityType(key.getKind());
    ImmutableSet<EntityId> entityIds = getEntityIdsFromSqlKey(entityType, key.getSqlKey());
-    return checkExists(entityType.getName(), entityIds);
+    return exists(entityType.getName(), entityIds);
  }

  @Override
-  public boolean checkExists(Object entity) {
+  public boolean exists(Object entity) {
    checkArgumentNotNull(entity, "entity must be specified");
    EntityType<?> entityType = getEntityType(entity.getClass());
    ImmutableSet<EntityId> entityIds = getEntityIdsFromEntity(entityType, entity);
-    return checkExists(entityType.getName(), entityIds);
+    return exists(entityType.getName(), entityIds);
  }

-  private boolean checkExists(String entityName, ImmutableSet<EntityId> entityIds) {
+  private boolean exists(String entityName, ImmutableSet<EntityId> entityIds) {
    assertInTransaction();
    TypedQuery<Integer> query =
        getEntityManager()
@@ -391,7 +391,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  private static ImmutableSet<EntityId> getEntityIdsFromEntity(
      EntityType<?> entityType, Object entity) {
    if (entityType.hasSingleIdAttribute()) {
-      String idName = entityType.getDeclaredId(entityType.getIdType().getJavaType()).getName();
+      String idName = entityType.getId(entityType.getIdType().getJavaType()).getName();
      Object idValue = getFieldValue(entity, idName);
      return ImmutableSet.of(new EntityId(idName, idValue));
    } else {
@@ -402,7 +402,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
  private static ImmutableSet<EntityId> getEntityIdsFromSqlKey(
      EntityType<?> entityType, Object sqlKey) {
    if (entityType.hasSingleIdAttribute()) {
-      String idName = entityType.getDeclaredId(entityType.getIdType().getJavaType()).getName();
+      String idName = entityType.getId(entityType.getIdType().getJavaType()).getName();
      return ImmutableSet.of(new EntityId(idName, sqlKey));
    } else {
      return getEntityIdsFromIdContainer(entityType, sqlKey);
@@ -429,7 +429,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {

  private static Object getFieldValue(Object object, String fieldName) {
    try {
-      Field field = object.getClass().getDeclaredField(fieldName);
+      Field field = getField(object.getClass(), fieldName);
      field.setAccessible(true);
      return field.get(object);
    } catch (NoSuchFieldException | IllegalAccessException e) {
@@ -437,6 +437,21 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
    }
  }

+  /** Gets the field definition from clazz or any superclass. */
+  private static Field getField(Class clazz, String fieldName) throws NoSuchFieldException {
+    try {
+      // Note that we have to use getDeclaredField() for this, getField() just finds public fields.
+      return clazz.getDeclaredField(fieldName);
+    } catch (NoSuchFieldException e) {
+      Class base = clazz.getSuperclass();
+      if (base != null) {
+        return getField(base, fieldName);
+      } else {
+        throw e;
+      }
+    }
+  }
+
  private static class TransactionInfo {
    EntityManager entityManager;
    boolean inTransaction = false;
@@ -208,7 +208,7 @@ public class Transaction extends ImmutableObject implements Buildable {

    @Override
    public void writeToDatastore() {
-      ofyTm().saveNewOrUpdate(entity);
+      ofyTm().put(entity);
    }

    @Override
@@ -14,6 +14,9 @@

 package google.registry.persistence.transaction;

+import com.google.common.collect.ImmutableList;
+import google.registry.schema.replay.DatastoreEntity;
+import google.registry.schema.replay.SqlEntity;
 import javax.persistence.Entity;
 import javax.persistence.GeneratedValue;
 import javax.persistence.GenerationType;
@@ -27,7 +30,7 @@ import javax.persistence.Table;
 */
@Entity
@Table(name = "Transaction")
-public class TransactionEntity {
+public class TransactionEntity implements SqlEntity {

  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
@@ -40,4 +43,9 @@ public class TransactionEntity {
  TransactionEntity(byte[] contents) {
    this.contents = contents;
  }
+
+  @Override
+  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
+    return ImmutableList.of(); // not stored in Datastore per se
+  }
 }
@@ -86,16 +86,16 @@ public interface TransactionManager {
  DateTime getTransactionTime();

  /** Persists a new entity in the database, throws exception if the entity already exists. */
-  void saveNew(Object entity);
+  void insert(Object entity);

  /** Persists all new entities in the database, throws exception if any entity already exists. */
-  void saveAllNew(ImmutableCollection<?> entities);
+  void insertAll(ImmutableCollection<?> entities);

  /** Persists a new entity or update the existing entity in the database. */
-  void saveNewOrUpdate(Object entity);
+  void put(Object entity);

  /** Persists all new entities or update the existing entities in the database. */
-  void saveNewOrUpdateAll(ImmutableCollection<?> entities);
+  void putAll(ImmutableCollection<?> entities);

  /** Updates an entity in the database, throws exception if the entity does not exist. */
  void update(Object entity);
@@ -104,10 +104,10 @@ public interface TransactionManager {
  void updateAll(ImmutableCollection<?> entities);

  /** Returns whether the given entity with same ID exists. */
-  boolean checkExists(Object entity);
+  boolean exists(Object entity);

  /** Returns whether the entity of given key exists. */
-  <T> boolean checkExists(VKey<T> key);
+  <T> boolean exists(VKey<T> key);

  /** Loads the entity by its id, returns empty if the entity doesn't exist. */
  <T> Optional<T> maybeLoad(VKey<T> key);
@@ -24,6 +24,8 @@ import com.google.api.client.http.GenericUrl;
 import com.google.api.client.http.HttpHeaders;
 import com.google.api.client.http.HttpRequest;
 import com.google.api.client.http.HttpResponse;
+import com.google.api.client.http.HttpResponseException;
+import com.google.api.client.http.HttpStatusCodes;
 import com.google.api.client.http.HttpTransport;
 import com.google.common.base.Ascii;
 import com.google.common.base.Splitter;
@@ -80,6 +82,7 @@ public class IcannHttpReporter {
    headers.setContentType(CSV_UTF_8.toString());
    request.setHeaders(headers);
    request.setFollowRedirects(false);
+    request.setThrowExceptionOnExecuteError(false);

    HttpResponse response = null;
    logger.atInfo().log(
@@ -87,6 +90,12 @@ public class IcannHttpReporter {
    boolean success = true;
    try {
      response = request.execute();
+      // Only responses with a 200 or 400 status have a body. For everything else, throw so that
+      // the caller catches it and prints the stack trace.
+      if (response.getStatusCode() != HttpStatusCodes.STATUS_CODE_OK
+          && response.getStatusCode() != HttpStatusCodes.STATUS_CODE_BAD_REQUEST) {
+        throw new HttpResponseException(response);
+      }
      byte[] content;
      try {
        content = ByteStreams.toByteArray(response.getContent());
@@ -94,16 +103,23 @@ public class IcannHttpReporter {
        response.getContent().close();
      }
      logger.atInfo().log(
-          "Received response code %d with content: %s\n\nResponse content in hex: %s",
+          "Received response code %d\n\n"
+              + "Response headers: %s\n\n"
+              + "Response content in UTF-8: %s\n\n"
+              + "Response content in HEX: %s",
          response.getStatusCode(),
+          response.getHeaders(),
          new String(content, UTF_8),
          BaseEncoding.base16().encode(content));
-      XjcIirdeaResult result = parseResult(content);
-      if (result.getCode().getValue() != 1000) {
+      // For reasons unclear at the moment, when we parse the response content using UTF-8 we get
+      // garbled texts. Since we know that an HTTP 200 response can only contain a result code of
+      // 1000 (i. e. success), there is no need to parse it.
+      if (response.getStatusCode() == HttpStatusCodes.STATUS_CODE_BAD_REQUEST) {
        success = false;
+        XjcIirdeaResult result = parseResult(content);
        logger.atWarning().log(
            "PUT rejected, status code %s:\n%s\n%s",
-            result.getCode(), result.getMsg(), result.getDescription());
+            result.getCode().getValue(), result.getMsg(), result.getDescription());
      }
    } finally {
      if (response != null) {
@@ -1,116 +0,0 @@
-// Copyright 2019 The Nomulus Authors. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package google.registry.schema.tmch;
-
-import static com.google.common.base.Preconditions.checkState;
-import static google.registry.util.DateTimeUtils.toJodaDateTime;
-import static google.registry.util.DateTimeUtils.toZonedDateTime;
-
-import com.google.common.collect.ImmutableList;
-import google.registry.model.CreateAutoTimestamp;
-import google.registry.model.ImmutableObject;
-import google.registry.schema.replay.DatastoreEntity;
-import google.registry.schema.replay.SqlEntity;
-import java.time.ZonedDateTime;
-import java.util.Map;
-import java.util.Optional;
-import javax.persistence.CollectionTable;
-import javax.persistence.Column;
-import javax.persistence.ElementCollection;
-import javax.persistence.Entity;
-import javax.persistence.GeneratedValue;
-import javax.persistence.GenerationType;
-import javax.persistence.Id;
-import javax.persistence.JoinColumn;
-import javax.persistence.MapKeyColumn;
-import javax.persistence.Table;
-import org.joda.time.DateTime;
-
-/**
- * A list of TMCH claims labels and their associated claims keys.
- *
- * <p>Note that the primary key of this entity is {@link #revisionId}, which is auto-generated by
- * the database. So, if a retry of insertion happens after the previous attempt unexpectedly
- * succeeds, we will end up with having two exact same claims list with only different {@link
- * #revisionId}. However, this is not an actual problem because we only use the claims list with
- * highest {@link #revisionId}.
- */
-@Entity
-@Table
-public class ClaimsList extends ImmutableObject implements SqlEntity {
-  @Id
-  @GeneratedValue(strategy = GenerationType.IDENTITY)
-  @Column
-  private Long revisionId;
-
-  @Column(nullable = false)
-  private CreateAutoTimestamp creationTimestamp = CreateAutoTimestamp.create(null);
-
-  @Column(nullable = false)
-  private ZonedDateTime tmdbGenerationTime;
-
-  @ElementCollection
-  @CollectionTable(
-      name = "ClaimsEntry",
-      joinColumns = @JoinColumn(name = "revisionId", referencedColumnName = "revisionId"))
-  @MapKeyColumn(name = "domainLabel", nullable = false)
-  @Column(name = "claimKey", nullable = false)
-  private Map<String, String> labelsToKeys;
-
-  private ClaimsList(ZonedDateTime tmdbGenerationTime, Map<String, String> labelsToKeys) {
-    this.tmdbGenerationTime = tmdbGenerationTime;
-    this.labelsToKeys = labelsToKeys;
-  }
-
-  // Hibernate requires this default constructor.
-  private ClaimsList() {}
-
-  /** Constructs a {@link ClaimsList} object. */
-  public static ClaimsList create(DateTime creationTimestamp, Map<String, String> labelsToKeys) {
-    return new ClaimsList(toZonedDateTime(creationTimestamp), labelsToKeys);
-  }
-
-  /** Returns the revision id of this claims list, or throws exception if it is null. */
-  public Long getRevisionId() {
-    checkState(
-        revisionId != null, "revisionId is null because it is not persisted in the database");
-    return revisionId;
-  }
-
-  /** Returns the TMDB generation time of this claims list. */
-  public DateTime getTmdbGenerationTime() {
-    return toJodaDateTime(tmdbGenerationTime);
-  }
-
-  /** Returns the creation time of this claims list. */
-  public DateTime getCreationTimestamp() {
-    return creationTimestamp.getTimestamp();
-  }
-
-  /** Returns an {@link Map} mapping domain label to its lookup key. */
-  public Map<String, String> getLabelsToKeys() {
-    return labelsToKeys;
-  }
-
-  /** Returns the claim key for a given domain if there is one, empty otherwise. */
-  public Optional<String> getClaimKey(String label) {
-    return Optional.ofNullable(labelsToKeys.get(label));
-  }
-
-  @Override
-  public ImmutableList<DatastoreEntity> toDatastoreEntities() {
-    return ImmutableList.of(); // ClaimsList is dual-written
-  }
-}
@@ -18,7 +18,7 @@ import static com.google.common.base.Preconditions.checkArgument;

 import com.google.common.base.Splitter;
 import com.google.common.collect.ImmutableMap;
-import google.registry.schema.tmch.ClaimsList;
+import google.registry.model.tmch.ClaimsListShard;
 import java.util.List;
 import org.joda.time.DateTime;

@@ -34,11 +34,11 @@ import org.joda.time.DateTime;
 public class ClaimsListParser {

  /**
-   * Converts the lines from the DNL CSV file into a {@link ClaimsList} object.
+   * Converts the lines from the DNL CSV file into a {@link ClaimsListShard} object.
   *
   * <p>Please note that this does <b>not</b> insert the object into Datastore.
   */
-  public static ClaimsList parse(List<String> lines) {
+  public static ClaimsListShard parse(List<String> lines) {
    ImmutableMap.Builder<String, String> builder = new ImmutableMap.Builder<>();

    // First line: <version>,<DNL List creation datetime>
@@ -74,6 +74,6 @@ public class ClaimsListParser {
      builder.put(label, lookupKey);
    }

-    return ClaimsList.create(creationTime, builder.build());
+    return ClaimsListShard.create(creationTime, builder.build());
  }
 }
@@ -21,8 +21,6 @@ import google.registry.keyring.api.KeyModule.Key;
 import google.registry.model.tmch.ClaimsListShard;
 import google.registry.request.Action;
 import google.registry.request.auth.Auth;
-import google.registry.schema.tmch.ClaimsList;
-import google.registry.schema.tmch.ClaimsListDao;
 import java.io.IOException;
 import java.security.SignatureException;
 import java.util.List;
@@ -56,14 +54,10 @@ public final class TmchDnlAction implements Runnable {
    } catch (SignatureException | IOException | PGPException e) {
      throw new RuntimeException(e);
    }
-    ClaimsList claims = ClaimsListParser.parse(lines);
-    ClaimsListShard claimsListShard =
-        ClaimsListShard.create(claims.getTmdbGenerationTime(), claims.getLabelsToKeys());
-    claimsListShard.save();
+    ClaimsListShard claims = ClaimsListParser.parse(lines);
+    claims.save();
    logger.atInfo().log(
        "Inserted %,d claims into Datastore, created at %s",
-        claimsListShard.size(), claimsListShard.getCreationTime());
-
-    ClaimsListDao.trySave(claims);
+        claims.size(), claims.getTmdbGenerationTime());
  }
 }
@@ -110,6 +110,12 @@ abstract class CreateOrUpdateTldCommand extends MutatingCommand {
      description = "One-time billing cost for a server status change")
  private Money serverStatusChangeCost;

+  @Nullable
+  @Parameter(
+      names = "--registry_lock_or_unlock_cost",
+      description = "One-time billing cost for a registry lock or unlock")
+  private Money registryLockOrUnlockCost;
+
  @Nullable
  @Parameter(
      names = "--tld_type",
@@ -118,10 +124,10 @@ abstract class CreateOrUpdateTldCommand extends MutatingCommand {

  @Nullable
  @Parameter(
-      names = "--disable_invoicing",
-      description = "Whether invoicing is disabled for a REAL tld.",
+      names = "--invoicing_enabled",
+      description = "Whether invoicing is enabled for this tld.",
      arity = 1)
-  private Boolean disableInvoicing;
+  private Boolean invoicingEnabled;

  @Nullable
  @Parameter(
@@ -326,8 +332,10 @@ abstract class CreateOrUpdateTldCommand extends MutatingCommand {
      Optional.ofNullable(roidSuffix).ifPresent(builder::setRoidSuffix);
      Optional.ofNullable(serverStatusChangeCost)
          .ifPresent(builder::setServerStatusChangeBillingCost);
+      Optional.ofNullable(registryLockOrUnlockCost)
+          .ifPresent(builder::setRegistryLockOrUnlockBillingCost);
      Optional.ofNullable(tldType).ifPresent(builder::setTldType);
-      Optional.ofNullable(disableInvoicing).ifPresent(builder::setDisableInvoicing);
+      Optional.ofNullable(invoicingEnabled).ifPresent(builder::setInvoicingEnabled);
      Optional.ofNullable(lordnUsername).ifPresent(u -> builder.setLordnUsername(u.orElse(null)));
      Optional.ofNullable(claimsPeriodEnd).ifPresent(builder::setClaimsPeriodEnd);
      Optional.ofNullable(numDnsPublishShards).ifPresent(builder::setNumDnsPublishLocks);
@@ -72,7 +72,7 @@ final class CreateRegistrarCommand extends CreateOrUpdateRegistrarCommand

  @Override
  void saveToCloudSql(Registrar registrar) {
-    jpaTm().saveNew(registrar);
+    jpaTm().insert(registrar);
  }

  @Nullable
@@ -380,7 +380,7 @@ public final class DomainLockUtils {
              .setReason(Reason.SERVER_STATUS)
              .setTargetId(domain.getForeignKey())
              .setClientId(domain.getCurrentSponsorClientId())
-              .setCost(Registry.get(domain.getTld()).getServerStatusChangeCost())
+              .setCost(Registry.get(domain.getTld()).getRegistryLockOrUnlockBillingCost())
              .setEventTime(now)
              .setBillingTime(now)
              .setParent(historyEntry)
@@ -22,8 +22,6 @@ import com.beust.jcommander.Parameters;
 import com.google.common.base.Joiner;
 import com.google.common.io.Files;
 import google.registry.model.tmch.ClaimsListShard;
-import google.registry.schema.tmch.ClaimsList;
-import google.registry.schema.tmch.ClaimsListDao;
 import google.registry.tmch.ClaimsListParser;
 import java.io.File;
 import java.io.IOException;
@@ -39,7 +37,7 @@ final class UploadClaimsListCommand extends ConfirmingCommand implements Command

  private String claimsListFilename;

-  private ClaimsList claimsList;
+  private ClaimsListShard claimsList;

  @Override
  protected void init() throws IOException {
@@ -58,8 +56,7 @@ final class UploadClaimsListCommand extends ConfirmingCommand implements Command

  @Override
  public String execute() {
-    ClaimsListShard.create(claimsList.getTmdbGenerationTime(), claimsList.getLabelsToKeys()).save();
-    ClaimsListDao.trySave(claimsList);
+    claimsList.save();
    return String.format("Successfully uploaded claims list %s", claimsListFilename);
  }
 }
@@ -56,7 +56,8 @@ registry.registrar.RegistryLock.prototype.runAfterRender = function(objArgs) {
  } else {
    goog.soy.renderElement(
        goog.dom.getRequiredElement('locks-content'),
-        registry.soy.registrar.registrylock.lockNotAllowedOnRegistrar);
+        registry.soy.registrar.registrylock.lockNotAllowedOnRegistrar,
+        {supportEmail: objArgs.supportEmail});
  }
 };

@@ -10,6 +10,7 @@
      <basic name="amount" access="FIELD"/>
    </attributes>
  </embeddable>
+
  <persistence-unit-metadata>
    <persistence-unit-defaults>
      <entity-listeners>
@@ -45,25 +45,27 @@
    <class>google.registry.model.contact.ContactResource</class>
    <class>google.registry.model.domain.DomainBase</class>
    <class>google.registry.model.domain.DomainHistory</class>
+    <class>google.registry.model.domain.GracePeriod</class>
+    <class>google.registry.model.domain.secdns.DelegationSignerData</class>
    <class>google.registry.model.domain.token.AllocationToken</class>
    <class>google.registry.model.host.HostHistory</class>
    <class>google.registry.model.host.HostResource</class>
-    <class>google.registry.model.registrar.Registrar</class>
-    <class>google.registry.model.registrar.RegistrarContact</class>
-    <class>google.registry.model.registry.label.PremiumList</class>
-    <class>google.registry.model.reporting.Spec11ThreatMatch</class>
-    <class>google.registry.persistence.transaction.TransactionEntity</class>
-    <class>google.registry.schema.domain.RegistryLock</class>
-    <class>google.registry.schema.tmch.ClaimsList</class>
-    <class>google.registry.schema.cursor.Cursor</class>
-    <class>google.registry.schema.server.Lock</class>
-    <class>google.registry.schema.tld.PremiumEntry</class>
-    <class>google.registry.model.domain.secdns.DelegationSignerData</class>
-    <class>google.registry.model.domain.GracePeriod</class>
    <class>google.registry.model.poll.PollMessage</class>
    <class>google.registry.model.poll.PollMessage$OneTime</class>
    <class>google.registry.model.poll.PollMessage$Autorenew</class>
+    <class>google.registry.model.registrar.Registrar</class>
+    <class>google.registry.model.registrar.RegistrarContact</class>
+    <class>google.registry.model.registry.label.PremiumList</class>
    <class>google.registry.model.registry.label.ReservedList</class>
+    <class>google.registry.model.registry.Registry</class>
+    <class>google.registry.model.reporting.DomainTransactionRecord</class>
+    <class>google.registry.model.reporting.Spec11ThreatMatch</class>
+    <class>google.registry.model.tmch.ClaimsListShard</class>
+    <class>google.registry.persistence.transaction.TransactionEntity</class>
+    <class>google.registry.schema.cursor.Cursor</class>
+    <class>google.registry.schema.domain.RegistryLock</class>
+    <class>google.registry.schema.server.Lock</class>
+    <class>google.registry.schema.tld.PremiumEntry</class>

    <!-- Customized type converters -->
    <class>google.registry.persistence.converter.AllocationTokenStatusTransitionConverter</class>
@@ -79,7 +81,9 @@
    <class>google.registry.persistence.converter.InetAddressSetConverter</class>
    <class>google.registry.persistence.converter.LocalDateConverter</class>
    <class>google.registry.persistence.converter.PostalInfoChoiceListConverter</class>
+    <class>google.registry.persistence.converter.PremiumListKeyConverter</class>
    <class>google.registry.persistence.converter.RegistrarPocSetConverter</class>
+    <class>google.registry.persistence.converter.ReservedListKeySetConverter</class>
    <class>google.registry.persistence.converter.Spec11ThreatMatchThreatTypeSetConverter</class>
    <class>google.registry.persistence.converter.StatusValueSetConverter</class>
    <class>google.registry.persistence.converter.StringListConverter</class>
@@ -90,7 +94,10 @@
    <class>google.registry.persistence.converter.ZonedDateTimeConverter</class>

    <!-- Generated converters for VKey -->
-    <class>google.registry.model.billing.VKeyConverter_BillingEvent</class>
+    <class>google.registry.model.billing.VKeyConverter_Cancellation</class>
+    <class>google.registry.model.billing.VKeyConverter_Modification</class>
+    <class>google.registry.model.billing.VKeyConverter_OneTime</class>
+    <class>google.registry.model.billing.VKeyConverter_Recurring</class>
    <class>google.registry.model.contact.VKeyConverter_ContactResource</class>
    <class>google.registry.model.domain.VKeyConverter_DomainBase</class>
    <class>google.registry.model.domain.token.VKeyConverter_AllocationToken</class>
@@ -163,5 +163,7 @@

 /** Content if the registrar is not allowed to use registry lock. */
 {template .lockNotAllowedOnRegistrar}
-  <h2>Registry Lock is coming soon; please stay tuned for updates.</h2>
+  {@param supportEmail: string}
+  <h2>Sorry, your registrar hasn't enrolled in registry lock yet. To do so, please
+    contact {$supportEmail}.</h2>
 {/template}
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
gbrodman	4ec7f23e84	Use the parent to store the history repo ID and fill in the base object (#830 ) * Use the parent to store the history repo ID and fill in the base object Storing the repo ID in the parent and in the base object has two primary benefits. First, it unifies the parent information in the HistoryEntry's `parent` object. This simplifies the builders and the data flow. Second, when possible (which should be always, post-migration) we fill out the DomainContent's repo ID (similarly for the other EPP resources) which means that when reconstituting the ofy keys we don't need to pass the repo ID in from a separate object. This way, all the data are encapsulated where they should be. The primary downside here is that it further reduces the "immutability" of the history objects (since we're using the Hibernate setter for the parent repo ID) but we weren't immutable anyway. * Respond to CR - compare the entire vkeys in tests - always return the parent for repo ID * Simplify creation of parent VKeys * Fix flipped isAssignableFrom check in VKey * Merge remote-tracking branch 'origin/master' into historyRepoId	2020-10-09 16:01:51 -04:00
Shicong Huang	7a68b1b6f0	Revert package-lock.json to version from #676 (#834 ) Co-authored-by: gbrodman <gbrodman@google.com>	2020-10-09 15:57:23 -04:00
Shicong Huang	14e593d9e1	Add SchemaCrawler as dependency (#833 )	2020-10-09 15:02:11 -04:00
Weimin Yu	2d5de96fbd	Minor python changes (#832 ) * Minor python changes Use dataclasses instead of attrs. The former is part of the standard lib while the latter may need to be installed separately. Also added python3 to the list of prerequisites.	2020-10-09 14:50:21 -04:00
Weimin Yu	13d30b0bfb	Maintain a release-to-Version map in deployment (#831 ) * Maintain a release-to-Version map in deployment Keep track of the mapping between Nomulus release tags and AppEngine version ids with a mapping file. This is necessary because AppEngine does not support custom versioning. With this mapping, rollbacks could be automated. Automation of rollbacks is important since there are test-supporting metadata to be updated, but are easily forgotten. During the last stage of deployment, current per-service version ids are fetched using gcloud and are appended to a file on GCS. Each line is of the format "{RELEASE_TAG},{APPENGINE_SERVICE},{APPENGINE_VERSION}. This change has been tested in crash. The rollback script is still a work in progress.	2020-10-09 13:32:52 -04:00
Shicong Huang	b05f6b4ba3	Add SQL schema for DelegationSignerData (#713 ) * Add SQL schema for DelegationSignerData * Remove join table * Rebased on HEAD * Rebase on head	2020-10-09 10:22:31 -04:00
Shicong Huang	17a1387184	Disable auto-generation on id for HostHistory and ContactHistory (#827 )	2020-10-08 12:30:54 -04:00
Ben McIlwain	2e230664fd	Convert CertificateViolation into an enum (#829 ) * Convert CertificateViolation into an enum This ends up being nicer to deal with from callsites than class instances, while still permitting full configurability of all parameters. There are various other changes/fixes as well.	2020-10-07 22:19:36 -04:00
Michael Muller	299b093f78	Correctly restore composite VKeys in DomainContent (#825 ) * Restore composite vkeys in DomainContent PollMessage/BillingEvent vkeys in DomainContent must have their ofy keys restored from other fields in DomainContent (namely the repo id and their specific history event ids). Add PostLoad methods to DomainContent and DomainHistory to do the restoration. * Fixes for review. * Deal with foreign-key cycles	2020-10-07 12:42:01 -04:00
Ben McIlwain	61e7fa89f7	Fix incorrect repackaged App Engine import (#828 ) * Fix incorrect repackaged App Engine import	2020-10-07 11:52:17 -04:00
sarahcaseybot	6ab69d4226	Add a CertificateChecker class (#793 ) * CertificateChecker with checks for expiration and key length * Add validity length check * Get rid of hard-coded constants and DSA checks * add files that for some reason weren't included in last commit * Rename violations and other fixes * Add displayMessage to CertificateViolation enum * Switch violations from an enum to a class * small changes * Get rid of ECDSA checks * add checks for old validity length * Change error message for validity length	2020-10-06 15:47:42 -04:00
gbrodman	0f09a4a0ab	Add more Datastore/Sql Entity annotations (#826 ) * Add more Datastore/Sql Entity annotations * Move comments up a line	2020-10-05 13:07:53 -04:00
Shicong Huang	95f6ccc657	Fix vkey reconstruction for PollMessage (#823 ) * Fix vkey reconstruction for PollMessage * Add foreign key * Rebase on HEAD	2020-10-05 10:35:40 -04:00
Michael Muller	77fabe4dc4	Move "WithLongVKey" to BillingEvent subclasses (#821 ) When loading the VKeys for the BillingEvents hierarchy, it is necessary to restore the original concrete class for the type, otherwise we end up with a different (and incompatible) VKey. As part of this, convert the cancellation matching billing event to VKey<Recurring>, which seems like the only thing it actually can be.	2020-10-02 15:20:23 -04:00
Lai Jiang	71fa12f773	Fix invoicing SQL (#824 )	2020-10-01 14:29:49 -04:00
Shicong Huang	fd40a6a2b9	Use composite primary key for HostHistory and ContactHistory (#809 ) * Use composite primary key for HostHistory and ContactHistory * Update flyway file version * Make getters private * Add javadoc * Rebase on HEAD	2020-10-01 11:01:57 -04:00
Michael Muller	71f86c9970	Add VKey.restoreOfy() method for fixing ofy keys (#820 ) Add a restoreOfy() instance method and a restoreOfyFrom() static method to assist in restoring the objectify key for classes that have composite keys that do not restore automatically.	2020-09-30 11:15:58 -04:00
Michael Muller	6f75dfd116	Create a flyway index file and verify correctness (#819 ) * Create a flyway index file and verify correctness Create an index file (flyway.txt) containing the names of all of the flyway files and verify that it is ordered and in sync with the actual contents of the flyway directory. Also provide a target (generateFlywayIndex) to automatically generate it. The purpose of flyway.txt is to cause a merge conflict in the event that two different developers add a flyway file with the same sequence number, an event which has occurred multiple times.	2020-09-29 11:26:05 -04:00
Lai Jiang	ad5a74fee9	Revert "Request 101m CPU in sandbox proxy (#813 )" (#818 ) This reverts commit `e30c0f9a11`. The proposed solution didn't work.	2020-09-25 11:55:46 -04:00
Lai Jiang	29b1ec4211	Add log4j-core as a runtime dependency (#817 ) Without it we kept getting the following warning: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...	2020-09-24 19:59:39 -04:00
Weimin Yu	553d5717cb	Enhance the test for forbidden Schema changes (#815 ) * Enhance the test for forbidden Schema changes Current test is git-based. It is difficult to maintain and does not catch out-of-order version numbers. It is also more aggressive than necessary, failing on changes to submitted scripts that have not been deployed yet. The new test starts a database, deploys the current schema to it, then deploys the set of Flyway scripts in this repository to the database.	2020-09-24 12:31:08 -04:00
Shicong Huang	1056fdbb64	Fix VKey reconstruction issue in BillingEvent (#805 ) * Fix VKey reconstruction issue in BillingEvent * Rebase on head	2020-09-23 19:04:58 -04:00
Lai Jiang	4aaf31be9f	Update IDN tables per ICANN's request (#812 ) See b/168508962 for the request.	2020-09-21 23:08:10 -04:00
Lai Jiang	e30c0f9a11	Request 101m CPU in sandbox proxy (#813 ) This is suggested as a mitigation to allow us to deploy to sandbox. The default value is 100m. See: https://b.corp.google.com/issues/167295064#comment36.	2020-09-21 21:22:41 -04:00
gbrodman	2a5d9c8ef5	Allow explicitly for null EPP resources in History objects (#790 ) * Allow explicitly for null EPP resources in History objects * Repo IDs should always be nonnull * Add a test to verify loading / comparison of legacy HistoryEntry objects * Format javadoc + annotations * More javadoc changes * V52 -> V56 * V56 -> V57 * saveNew -> insert in new tests	2020-09-21 15:50:15 -04:00
gbrodman	597f5746a4	Rename V54 -> V56 on host table rename (#811 )	2020-09-21 13:34:37 -04:00
Michael Muller	5bff53a711	Rename HostResource table to Host (#804 ) * Rename HostResource table to Host * Convert drop/creates to renames	2020-09-21 11:43:25 -04:00
Michael Muller	933394e8c3	Improve naming of TransactionManager methods (#802 ) * Improve naming of TransactionManager methods Per internal discussion, convert names of methods as follows: saveNew -> insert saveNewOrUpdate -> put checkExists -> exists Likewise, convert derived names to their corresponding forms, e.g. saveNewOrUpdateAll -> putAll.	2020-09-21 09:10:01 -04:00
gbrodman	3b841bbb5b	Add domain-specific history fields to DomainHistory objects (#794 ) * Add domain-specific history fields to DomainHistory objects * Add javadoc for Hibernate-only methods * V52 -> V54 * Use only a single DomainTransactionRecord table * Add nullables and fix up a comment * V54 -> V55 * Regenerate db schema * Regen SQL file	2020-09-18 15:55:17 -04:00
gbrodman	798879d031	Fix semantic merge conflict in Registry (#810 ) * Fix semantic merge conflict in Registry	2020-09-18 14:40:11 -04:00
gbrodman	1a63d50b82	Create a separate per-tld registry lock/unlock cost (#800 ) * Create a separate per-tld registry lock/unlock cost Currently we use the standard server status change cost for this, but this might not be ideal at some point in the future if we wish to allow manual forced updates outside of the standard registry lock system (we would charge for these manual forced updates, even if we don't charge for registry locks). * Remove period	2020-09-18 13:22:29 -04:00
Lai Jiang	054571a625	Update ICANN activity reporting logging (#808 ) 1. It appears that when we have a 200 response, the response content is garbled, but we don't care since we know the request is successful. When we have a 400 response, the response is indeed UTF-8 encoded. Print the stack trace of the HTTP exception doesn't help anymore. 2. The result code is a complex type which includes the xml element with all its attributes, which don't care. We only want to print the value.	2020-09-18 11:37:23 -04:00
sarahcaseybot	7468a9915b	Migrate Registry objects to a TLD table in Cloud SQL (#803 ) * Add TLD table * Change reservedLists to array * Change ReservedLists back to a set * Rename reservedListKeyConverter to ReservedListKeySetConverter * Add a postload method	2020-09-17 12:47:50 -04:00
Lai Jiang	157d9f75c1	Use the save API version for the HPA controller and the deployment (#807 ) This supposedly can fix our deployment problem. Tested on alpha. Also updated the deployment script to replace the service object as well.	2020-09-16 09:15:35 -04:00
Lai Jiang	b2e4f07bb9	Update IDN tables (#806 ) See b/168508962 for context. 1. Changed the contact to iana-contact@google.com 2. Changed the header from "Script" to "Language" for zh-Hans and zh-Hant. 3. Commented out the references in zh-Hans and Zh-Hant	2020-09-16 09:15:27 -04:00
Michael Muller	5488e1b323	Fix accessing superclass fields in checkExists() (#799 ) * Fix accessing superclass fields in checkExists() JpaTransactionManagerImpl doesn't respect @Id fields in mapped superclasses. Replace calls to getDeclaredId() and getDeclaredField() with superclass friendly counterparts.	2020-09-11 13:45:51 -04:00
Shicong Huang	5ab0f97351	Add and use temp_history_id_sequence to avoid release error (#795 )	2020-09-11 12:25:08 -04:00
sarahcaseybot	f7b65327da	Add type converter for Key<ReservedList> and Key<PremiumList> (#796 ) * Add converter for reservedlist and premiumlist keys * Remove public modifier from test classes	2020-09-10 17:36:22 -04:00
Michael Muller	36482ce94f	Fix the billing occurrence foreign key (#797 ) * Fix the billing occurrence foreign key Fix the Domain.billing_occurrence_id foreign key constraint to reference the correct table (BillingRecurrence, not BillingEvent).	2020-09-10 12:02:24 -04:00
Lai Jiang	125f509b46	Change disable invoicing flag to enable invoicing flag (#783 ) * Change disable invoicing flag to enable invoicing flag This flag will be the sole determinor on if invoicing is enabled, regardless of TLD types. Once this PR is deployed we will need to run the nomulus command to update this flag on all launched open TLDs. For context on why this change is made, see b/159626744. * Rename enableInvoicing to InvoicingEnabled	2020-09-09 19:37:41 -04:00
gbrodman	fb7ba80b86	Implement DatastoreEntity/SqlEntity for many more classes (#788 ) * Implement DatastoreEntity/SqlEntity for many more classes We still have many more classes to go, but this gets us closer to guaranteeing that we can convert from Datastore to SQL objects and back again. * Shift SqlEntity impl to HistoryEntry	2020-09-09 13:56:59 -04:00
Lai Jiang	a86fcf79f7	Make ICANN reporting not fail on success upload (#791 ) * Make ICANN reporting not fail on success upload According to the spec (https://tools.ietf.org/html/draft-lozano-icann-registry-interfaces-13#page-16), when an upload succeeds (HTTP response code 200), the result code contained in the response message is always 1000 (success). So there is no need to parse the response content and check the result code. Given that we are having a problem parsing the response content due to encoding, it is best that we don't check it so as to not get false negative alerts when the upload is successful. The current logic also has a bug: HttpRequest.execute() will by default throw when the response code is non-20X. Therefore for a 400 response, our parsing logic never runs on it. Coincidentally, this month when we uploaded the July activity report (due to stale cursors), we get 400 responses (due to existing reports on the ICANN servers). The stack trace printed for the thrown exceptions from the 400 responses contained correctly parsed response contents. This lead us to believe that the issue with encoding was transient last month. However when we tried again to upload this month's report, our parser failed again (because the response code was 200 this time, and our parser actually ran on the response contents). This seems to suggest that ICANN is sending back readable response contents, but our parser somehow failed to understand it, assuming that ICANN is using the same encoding for 200 (which we tried and failed to parse) and 400 response contents (which caused an exception and was printed corrected in the stack trace). This PR changed the transport behavior so that it doesn't throw automatically for non-20X responses. We will print the content for both 200 and 400 responses, but only try to parse 400 response content. We put the 400 response in an HttpResponseException and print stack trace from it, which should display the content correctly so that we can compare it with the result of our own parsing. * Add tests	2020-09-03 15:57:30 -04:00
Lai Jiang	dc8e095e55	Upgrade to Gradle 6.6.1 (#792 )	2020-09-03 15:56:52 -04:00
Shicong Huang	cdf2c7f7cb	Merge ClaimsList into ClaimsListShard (#694 ) * Merge ClaimsList into ClaimsListShard * Add a TODO to rename the class * Rebase on HEAD * Improve javadoc	2020-09-03 11:18:40 -04:00
Shicong Huang	ecafebdc3d	Use composite primary key for DomainHistory (#767 ) * Use composite primary key for DomainHistory * Move History table's SequenceGenerator to orm.xml * Rebase on HEAD and remove default value for key in History tables * Use primitive type for id. * Revert the cache change	2020-09-03 10:21:23 -04:00
Lai Jiang	c6c8d21281	Update jackson-core to the latest version (#789 ) Vomit identified a vulnerability in the current version.	2020-09-03 09:11:12 -04:00
Shicong Huang	5f6ea2cbf2	Fix cascade issue for GracePeriod (#775 ) * Fix cascade issue for GracePeriod * Rebase on HEAD * Make GracePeriod immutable * Add javadoc and use nullToEmptyImmutableCopy	2020-09-02 20:05:53 -04:00
Shicong Huang	393c388e0d	Consolidate conversion from Duration to Period in DurationConverter (#786 ) * Consolidate conversion from Duration to Period in DurationConverter * Resolve comment	2020-09-01 11:29:28 -04:00
gbrodman	5a08ce498e	Revert "Change the wording on the lock-not-enabled page (#504 )" (#787 ) This reverts commit `28d3af0ee9`. We are now ready to accept new Registry Lock requests so we can have the originally-designed wording back in place	2020-08-31 15:19:42 -04:00
Weimin Yu	5db8cbc994	Fix flaky web driver tests (#784 ) * Fix flaky web driver tests Identified two flaky tests in RegistrarConsoleScreenshotTest through local testing and fixed them by waiting for specific web elements instead of using fixed delays. Refactored the wait methods to support different test scenarios, and removed unnecessary delays. Extensively tested locally. Also ran multiple presubmits on Kokoro.	2020-08-31 15:09:54 -04:00
Weimin Yu	bbcafea98e	Cover more base in forbidden SQL change check (#785 ) * Cover more base in forbidden SQL change check Update the forbidden SQL change detection script to include file deletion and renaming as well as edits.	2020-08-31 15:08:37 -04:00
gbrodman	1bba68dd96	Add success/failure notifications for the RelockDomainAction (#733 ) * Add success/failure notifications for the RelockDomainAction If a relock fails for some reason, we should noisily notify both our alerting email and also the registry lock contacts for the registrar in question. The consequences of a silent failure could be large so it's something we want to avoid if at all possible. In addition, we only retry tasks up to two times (one in 5min, one in 10min). This model of retries / notifications, as well as the language contained in the emails, have been LGTMed by Bruno and Kirsten * Change the wording on the success email * Change the times in which we send emails For transient failures: - Retry every ten minutes for six hours - Send an email after a half hour (three failures) saying that we'll retry - Send a success email if we succeed any time after that For non-transient failures: Send an email with the error message and don't retry * Add a test for the max-failure-email * Responses to CR - retry indefinitely - send an email to just the alert address if we can't find the lock - refactor the task enqueuer a bit * non-transient -> non-retryable * Use a lenient stubber for the AESU * Add a DS transaction around the re-lock	2020-08-31 14:15:47 -04:00