1
0
mirror of https://github.com/google/nomulus synced 2026-07-19 06:22:33 +00:00

Compare commits

...

34 Commits

Author SHA1 Message Date
Lai Jiang 1c90a6648e Remove bulk query entities (#1834)
These alternative ORMs are introduced as a way to make querying large number of
domains and domain histories more efficient through bulk loading from several
to-be-joined tables separately, then in-memory re-assembly of the final entity,
bypassing the need to query multiple tables each time an entity is queried.

Their primary use case is loading these entities for comparison between
datastore and SQL during the migration, which has been completed. The
code remain unused as of now and their existence makes refactoring and
general maintenance more complicated than necessary due to the need to keep
them up to date.

Therefore we remove the related code.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1834)
<!-- Reviewable:end -->
2022-10-28 12:25:57 -04:00
Lai Jiang 3f68ad5ea3 Rename BackupGroupRoot (#1829)
Also removed the ability to disable update timestamp auto update as it
was only needed during the migration.

Lastly, rectified the use of raw Coder in RegistryJpaIO.
2022-10-28 12:22:53 -04:00
gbrodman 9c6c210e21 Check for entity nonexistence in SqlBatchWriter (#1824)
Passing in an already-existing instance is an antipattern because it can
lead to race conditions where something else modified the object in
between when the pipeline loaded it and when you're saving it. The Write
action should only be writing new entities.

We cannot check IDs for the objects (some IDs are not autogenerated so
they might exist already). We also cannot call `insert` on the objects
because the underlying JPA `persist` call adds the input object to the
persistence context, meaning that any modifications (e.g.
updateTimestamp) are reflected in the input object. Beam doesn't allow
modification of input objects.
2022-10-27 14:46:26 -04:00
sarahcaseybot ca60ca159f Add DEFAULT_PROMO token type (#1832)
* Add DEFAULT_PROMO token type

* Fix test error message check
2022-10-27 13:08:15 -04:00
Lai Jiang 82092b3516 Remove ofy-only functions in TransactionManager (#1826)
Also remove the use of auditedOfy in places other than the
GaeUserIdConverter.
2022-10-25 15:52:00 -04:00
sarahcaseybot 0746d28e0c Check token type of currentPackageToken (#1825)
* Check currentPackageToken TokenType

* Check TokenType of currentPackageToken

* Check that token already exists
2022-10-25 12:39:33 -04:00
Lai Jiang aaa311ec40 Remove the mechanism to compare objects across database (#1822)
The migration is done.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1822)
<!-- Reviewable:end -->
2022-10-20 13:19:48 -04:00
Lai Jiang addef17904 Does not self allocate IDs in Beam by default. (#1809)
* Does not self allocate IDs in Beam by default.

Per b/250948425, it is dangerous to implicitly allow all Beam pipelines
to create buildables by self allocating the IDs. This change makes it so
that one has to explicitly request self allocation in Beam.

A boolean is added to the pipeline option so that it can be passed to
the beam worker initializer that controls the behavior of the JVM on
each worker. Note that we did not add the option in the metadata.json file
because we did not want people to use the override at run time when launching
a pipeline, due to the risk. As shown in RdePipeline.java, we instead
explicitly hard-code the option in the pipeline. There is nothing that
stops one to supply that option when launching the pipeline, but it's
not advised.

Tested=deployed the pipeline alpha and ran it.
2022-10-19 20:44:06 -04:00
Weimin Yu 8fe3c08069 Properly create and use default credential (#1818)
* Properly create and use default credential

This PR consists of the following changes:

- Stopped adding scopes to the default credential when using it to access other
  non-workspace GCP APIs. Scopes are not needed here.

- Started applying scopes to the default credential when using to access
  Drive and Sheets APIs.
  - Upgraded Drive access from the deprecated credential lib to the
    up-to-date one
  - Switched Sheet access from the exported json credential to the
    scoped default credential.

This PR requires that the affected files be writable to the default
service account (project-name@appspot.gserviceaccount.com) of the
project.

- This is already the case for exported files (premium terms, reserved
  terms, and domain list).

- The registrar sync sheets in alpha, sandbox, and production have been
  updated with the new permissions.

All impacted operations have been tested in alpha.

* Properly create and use default credential

This PR consists of the following changes:

- Added a new method to generate scope-less default credential when using it to
  access other non-workspace GCP APIs. Scopes are not needed here.

  - Started to use the new credential in the SecreteManager.
  - Will migrate other usages to this new credential gradually.
  - Marked the old DefaultCredential as deprecated.

- Started applying scopes to the default credential when using to access Drive
  and Sheets APIs.

  - Upgraded Drive access from the deprecated credentials lib
  - Switched Sheet access from the exported json credential to the scoped
    default credential.

This PR requires that the affected files be writable to the default service
account (project-name@appspot.gserviceaccount.com) of the project.

- This is already the case for exported files (premium terms, reserved terms,
  and domain list).

- The registrar sync sheets in alpha, sandbox, and production have been
  updated with the new permissions.

All impacted operations have been tested in alpha.
2022-10-18 20:20:36 -04:00
sarahcaseybot 5dc796b1f7 Add monitoring for package max create limit (#1798)
* Add action for checking package domain create limit compliance

* Add create limit monitoring

* Change variable name

* Add more logging
2022-10-18 12:39:53 -04:00
Ben McIlwain 8bddf35d0d Revert "Upgrade App Engine Standard to Java 17 w/ bundled APIs (#1816)" (#1817)
This reverts commit 1ab077d267.

Apparently the new version of Spinnaker that is compatible with this doesn't
work for our release, so we need to roll this back for now. (Again!)
2022-10-13 10:05:47 -04:00
Pavlo Tkach 7b9c16ca3e Update conditions when domain update flow triggers dns publish task (#1811)
Addressing b/246375161
2022-10-12 10:25:33 -04:00
Ben McIlwain 1ab077d267 Upgrade App Engine Standard to Java 17 w/ bundled APIs (#1816) 2022-10-11 20:06:37 -04:00
gbrodman ca65fbcc79 Refactor createSynthetic to be a command instead of a pipeline (#1813) 2022-10-11 12:23:31 -04:00
sarahcaseybot 0cfa7f8081 Remove allocation token check for transfering package domains (#1814) 2022-10-11 11:37:52 -04:00
Lai Jiang 9e31047c3a Fix nomulus command (#1812)
go/r3pr/1805 introduced an injectable clock in a few commands, but we
forgot to add the corresponding injector in the component. This PR fixes
it.
2022-10-09 16:45:42 -04:00
Pavlo Tkach b7c2e8fba5 Limit environments allowed to send emails out (#1807) 2022-10-07 12:12:57 -04:00
Pavlo Tkach a299df3005 Add fallback for Spec11 ThreatMatch parser (#1806) 2022-10-06 13:54:43 +00:00
Pavlo Tkach a9b35c163d Revert "Do not enqueue DNS updates when flow doesn't affect nameservers (#1785)" (#1808)
This reverts commit 775f672f2a.
2022-10-05 14:13:52 -04:00
gbrodman 9da24d114c Use injected times in URSC and CommandTestCase (#1805)
We started getting failures because some of the tests used October. In
general we should freeze the clock for testing as much as possible.

Same thing with the Get*Commands
2022-10-04 15:36:41 -04:00
Lai Jiang 7dd5876315 Refactor VKeyConverter (#1794)
Remove the redundant composite key boolean and simply the annotation
structure a bit.
2022-10-03 15:49:18 -04:00
gbrodman d1a259f63a Modify the CreateSynthetic pipeline to run over all non-deleted domains (#1803) 2022-10-03 15:15:41 -04:00
sarahcaseybot 8c5d2e9d92 Don't allow package tokens to discount premium names (#1804) 2022-10-03 14:27:10 -04:00
gbrodman cca1306b09 Change some READ_COMMITTED levels to REPEATABLE_READ (#1802)
Basically, any time we're loading a bunch of linked objects that might
change, we want to have REPEATABLE_READ so that another transaction
doesn't come along and smush whatever we think we're loading.

The following instances of READ_COMMITTED haven't changed:
- RdePipeline (it only loads immutable objects like histories)
- Invoicing pipeline (only immutable objects like BillingEvents)
- Spec11 (doesn't use any linked info from Domain)

This also changes the PersistenceModule to use REPEATABLE_READ by
default on the replica JPA TM, for the standard reasoning.
2022-09-30 14:44:50 -04:00
Weimin Yu 47071b0fbb Restore log4j exclusion in gradle build (#1801) 2022-09-30 14:04:00 -04:00
Weimin Yu d83565d37e Add a new allowed license string (#1800)
There are sporadic errors when building on desktop using maven central.
2022-09-30 14:03:17 -04:00
Weimin Yu a557b3f376 Disable the cron job for ResaveAllEppResourcesPipelineAction (#1799)
See b/249863289 for more information.
2022-09-30 12:05:29 -04:00
sarahcaseybot f4a49864b5 Add a get_package_promotion Command (#1793)
* Add a get_package_promotion Command

* add changes to loadByTokenString

* Fix test
2022-09-29 15:02:16 -04:00
gbrodman acdecca181 Don't create unnecessary synthetic History objects (#1796) 2022-09-26 13:41:57 -04:00
gbrodman 5264ab3fc3 Create pipeline to save synthetic DomainHistory objects (#1795)
This runs over all domains that weren't deleted as of September 5. This
will fix most of b/248112997, which is itself caused by b/245940594 --
creating synthetic history objects means that the RDE pipeline will look
at those instead of the potentially-no-longer-valid data in the old
history objects.
2022-09-22 14:58:50 -04:00
sarahcaseybot a9d59e4d6e Fix id generation in PackagePromotion (#1788)
* Fix id generation in PackagePromotion

* Fix update command tests
2022-09-21 15:19:49 -04:00
sarahcaseybot 1d3738da27 Add mutating commands for PackagePromotion (#1769)
* Add mutating commands for PackagePromotion

* Add checkAllocationToken methods

* Remove abstract methods

* Add better comments

* Small fixes

* Remove unneccesary init method

* Only assert in transaction in helper method
2022-09-21 12:38:09 -04:00
Lai Jiang 82a3a49268 Rename various fields and classes after migration (#1784)
Also fixed a bug introduced in #1785 where identity checked were performed instead of equality. This resulted in two sets containing the same elements not being regarded as equal and subsequent DNS updated being unnecessarily enqueued.
2022-09-21 11:49:22 -04:00
Pavlo Tkach 5bbad483e4 Fail genenerate invoices job when billing events not finished expanding (#1791) 2022-09-21 09:20:05 -04:00
236 changed files with 3240 additions and 4641 deletions
@@ -207,6 +207,9 @@
{
"moduleLicense": "GNU Library General Public License v2.1 or later"
},
{
"moduleLicense": "GNU Lesser General Public License v3.0"
},
// This is just 3-clause BSD.
{
"moduleLicense": "Go License"
+4
View File
@@ -707,6 +707,10 @@ createToolTask(
createToolTask(
'jpaDemoPipeline', 'google.registry.beam.common.JpaDemoPipeline')
createToolTask(
'createSyntheticDomainHistories',
'google.registry.tools.javascrap.CreateSyntheticDomainHistoriesPipeline')
project.tasks.create('generateSqlSchema', JavaExec) {
classpath = sourceSets.nonprod.runtimeClasspath
main = 'google.registry.tools.DevTool'
@@ -0,0 +1,80 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.batch;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableList;
import com.google.common.flogger.FluentLogger;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.token.PackagePromotion;
import google.registry.request.Action;
import google.registry.request.Action.Service;
import google.registry.request.auth.Auth;
import java.util.List;
/**
* An action that checks all {@link PackagePromotion} objects for compliance with their max create
* limit.
*/
@Action(
service = Service.BACKEND,
path = CheckPackagesComplianceAction.PATH,
auth = Auth.AUTH_INTERNAL_OR_ADMIN)
public class CheckPackagesComplianceAction implements Runnable {
public static final String PATH = "/_dr/task/checkPackagesCompliance";
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
@Override
public void run() {
tm().transact(
() -> {
ImmutableList<PackagePromotion> packages = tm().loadAllOf(PackagePromotion.class);
ImmutableList.Builder<PackagePromotion> packagesOverCreateLimit =
new ImmutableList.Builder<>();
for (PackagePromotion packagePromo : packages) {
List<DomainHistory> creates =
jpaTm()
.query(
"FROM DomainHistory WHERE current_package_token = :token AND"
+ " modificationTime >= :lastBilling AND type = 'DOMAIN_CREATE'",
DomainHistory.class)
.setParameter("token", packagePromo.getToken().getSqlKey().toString())
.setParameter(
"lastBilling", packagePromo.getNextBillingDate().minusYears(1))
.getResultList();
if (creates.size() > packagePromo.getMaxCreates()) {
int overage = creates.size() - packagePromo.getMaxCreates();
logger.atInfo().log(
"Package with package token %s has exceeded their max domain creation limit"
+ " by %d name(s).",
packagePromo.getToken().getSqlKey(), overage);
packagesOverCreateLimit.add(packagePromo);
}
}
if (packagesOverCreateLimit.build().isEmpty()) {
logger.atInfo().log("Found no packages over their create limit.");
} else {
logger.atInfo().log(
"Found %d packages over their create limit.",
packagesOverCreateLimit.build().size());
// TODO(sarahbot@) Send email to registrar and registry informing of creation
// overage once email template is finalized.
}
});
}
}
@@ -91,7 +91,7 @@ public class DeleteProberDataAction implements Runnable {
// Note: creationTime must be compared to a Java object (CreateAutoTimestamp) but deletionTime can
// be compared directly to the SQL timestamp (it's a DateTime)
private static final String DOMAIN_QUERY_STRING =
"FROM Domain d WHERE d.tld IN :tlds AND d.fullyQualifiedDomainName NOT LIKE 'nic.%' AND"
"FROM Domain d WHERE d.tld IN :tlds AND d.domainName NOT LIKE 'nic.%' AND"
+ " (d.subordinateHosts IS EMPTY OR d.subordinateHosts IS NULL) AND d.creationTime <"
+ " :creationTimeCutoff AND ((d.creationTime <= :nowAutoTimestamp AND d.deletionTime >"
+ " current_timestamp()) OR d.deletionTime < :nowMinusSoftDeleteDelay) ORDER BY d.repoId";
@@ -220,7 +220,7 @@ public class DeleteProberDataAction implements Runnable {
private void hardDeleteDomainsAndHosts(
ImmutableList<String> domainRepoIds, ImmutableList<String> hostNames) {
jpaTm()
.query("DELETE FROM Host WHERE fullyQualifiedHostName IN :hostNames")
.query("DELETE FROM Host WHERE hostName IN :hostNames")
.setParameter("hostNames", hostNames)
.executeUpdate();
jpaTm()
@@ -14,6 +14,7 @@
package google.registry.beam.common;
import static com.google.common.base.Preconditions.checkArgument;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
@@ -21,18 +22,14 @@ import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Streams;
import google.registry.beam.common.RegistryQuery.CriteriaQuerySupplier;
import google.registry.model.UpdateAutoTimestamp;
import google.registry.model.UpdateAutoTimestamp.DisableAutoUpdateResource;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.persistence.transaction.TransactionManagerFactory;
import java.io.Serializable;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ThreadLocalRandom;
import javax.annotation.Nullable;
import javax.persistence.criteria.CriteriaQuery;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.transforms.Create;
@@ -135,6 +132,7 @@ public final class RegistryJpaIO {
abstract SerializableFunction<R, T> resultMapper();
@Nullable
abstract Coder<T> coder();
@Nullable
@@ -145,13 +143,16 @@ public final class RegistryJpaIO {
@Override
@SuppressWarnings("deprecation") // Reshuffle still recommended by GCP.
public PCollection<T> expand(PBegin input) {
return input
.apply("Starting " + name(), Create.of((Void) null))
.apply(
"Run query for " + name(),
ParDo.of(new QueryRunner<>(query(), resultMapper(), snapshotId())))
.setCoder(coder())
.apply("Reshuffle", Reshuffle.viaRandomKey());
PCollection<T> output =
input
.apply("Starting " + name(), Create.of((Void) null))
.apply(
"Run query for " + name(),
ParDo.of(new QueryRunner<>(query(), resultMapper(), snapshotId())));
if (coder() != null) {
output = output.setCoder(coder());
}
return output.apply("Reshuffle", Reshuffle.viaRandomKey());
}
public Read<R, T> withName(String name) {
@@ -179,9 +180,7 @@ public final class RegistryJpaIO {
}
static <R, T> Builder<R, T> builder() {
return new AutoValue_RegistryJpaIO_Read.Builder<R, T>()
.name(DEFAULT_NAME)
.coder(SerializableCoder.of(Serializable.class));
return new AutoValue_RegistryJpaIO_Read.Builder<R, T>().name(DEFAULT_NAME);
}
@AutoValue.Builder
@@ -193,7 +192,7 @@ public final class RegistryJpaIO {
abstract Builder<R, T> resultMapper(SerializableFunction<R, T> mapper);
abstract Builder<R, T> coder(Coder coder);
abstract Builder<R, T> coder(Coder<T> coder);
abstract Builder<R, T> snapshotId(@Nullable String sharedSnapshotId);
@@ -298,12 +297,6 @@ public final class RegistryJpaIO {
public abstract SerializableFunction<T, Object> jpaConverter();
/**
* Signal to the writer that the {@link UpdateAutoTimestamp} property should be allowed to
* manipulate its value before persistence. The default value is {@code true}.
*/
abstract boolean withUpdateAutoTimestamp();
public Write<T> withName(String name) {
return toBuilder().name(name).build();
}
@@ -324,10 +317,6 @@ public final class RegistryJpaIO {
return toBuilder().jpaConverter(jpaConverter).build();
}
public Write<T> disableUpdateAutoTimestamp() {
return toBuilder().withUpdateAutoTimestamp(false).build();
}
abstract Builder<T> toBuilder();
@Override
@@ -344,7 +333,7 @@ public final class RegistryJpaIO {
GroupIntoBatches.<Integer, T>ofSize(batchSize()).withShardedKey())
.apply(
"Write in batch for " + name(),
ParDo.of(new SqlBatchWriter<>(name(), jpaConverter(), withUpdateAutoTimestamp())));
ParDo.of(new SqlBatchWriter<>(name(), jpaConverter())));
}
static <T> Builder<T> builder() {
@@ -352,8 +341,7 @@ public final class RegistryJpaIO {
.name(DEFAULT_NAME)
.batchSize(DEFAULT_BATCH_SIZE)
.shards(DEFAULT_SHARDS)
.jpaConverter(x -> x)
.withUpdateAutoTimestamp(true);
.jpaConverter(x -> x);
}
@AutoValue.Builder
@@ -367,8 +355,6 @@ public final class RegistryJpaIO {
abstract Builder<T> jpaConverter(SerializableFunction<T, Object> jpaConverter);
abstract Builder<T> withUpdateAutoTimestamp(boolean withUpdateAutoTimestamp);
abstract Write<T> build();
}
}
@@ -377,24 +363,15 @@ public final class RegistryJpaIO {
private static class SqlBatchWriter<T> extends DoFn<KV<ShardedKey<Integer>, Iterable<T>>, Void> {
private final Counter counter;
private final SerializableFunction<T, Object> jpaConverter;
private final boolean withAutoTimestamp;
SqlBatchWriter(
String type, SerializableFunction<T, Object> jpaConverter, boolean withAutoTimestamp) {
SqlBatchWriter(String type, SerializableFunction<T, Object> jpaConverter) {
counter = Metrics.counter("SQL_WRITE", type);
this.jpaConverter = jpaConverter;
this.withAutoTimestamp = withAutoTimestamp;
}
@ProcessElement
public void processElement(@Element KV<ShardedKey<Integer>, Iterable<T>> kv) {
if (withAutoTimestamp) {
actuallyProcessElement(kv);
return;
}
try (DisableAutoUpdateResource disable = UpdateAutoTimestamp.disableAutoUpdate()) {
actuallyProcessElement(kv);
}
actuallyProcessElement(kv);
}
private void actuallyProcessElement(@Element KV<ShardedKey<Integer>, Iterable<T>> kv) {
@@ -405,7 +382,13 @@ public final class RegistryJpaIO {
.filter(Objects::nonNull)
.collect(ImmutableList.toImmutableList());
try {
jpaTm().transact(() -> jpaTm().putAll(entities));
jpaTm()
.transact(
() -> {
// Don't modify existing objects as it could lead to race conditions
entities.forEach(this::verifyObjectNonexistence);
jpaTm().putAll(entities);
});
counter.inc(entities.size());
} catch (RuntimeException e) {
processSingly(entities);
@@ -419,7 +402,13 @@ public final class RegistryJpaIO {
private void processSingly(ImmutableList<Object> entities) {
for (Object entity : entities) {
try {
jpaTm().transact(() -> jpaTm().put(entity));
jpaTm()
.transact(
() -> {
// Don't modify existing objects as it could lead to race conditions
verifyObjectNonexistence(entity);
jpaTm().put(entity);
});
counter.inc();
} catch (RuntimeException e) {
throw new RuntimeException(toEntityKeyString(entity), e);
@@ -445,5 +434,16 @@ public final class RegistryJpaIO {
return "Non-SqlEntity: " + entity;
}
}
/** SqlBatchWriter should not re-write existing entities due to potential race conditions. */
private void verifyObjectNonexistence(Object obj) {
// We cannot rely on calling "insert" on the objects because the underlying JPA persist call
// adds the input object to the persistence context, meaning that any modifications (e.g.
// updateTimestamp) are reflected in the input object. Beam doesn't allow modification of
// input objects, so this throws an exception.
// TODO(go/non-datastore-allocateid): also check that all the objects have IDs
checkArgument(
!jpaTm().exists(obj), "Entities created in SqlBatchWriter must not already exist");
}
}
}
@@ -20,9 +20,7 @@ import dagger.Lazy;
import google.registry.config.CredentialModule;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.model.domain.Domain;
import google.registry.persistence.PersistenceModule;
import google.registry.persistence.PersistenceModule.BeamBulkQueryJpaTm;
import google.registry.persistence.PersistenceModule.BeamJpaTm;
import google.registry.persistence.PersistenceModule.BeamReadOnlyReplicaJpaTm;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
@@ -52,14 +50,6 @@ public interface RegistryPipelineComponent {
@BeamJpaTm
Lazy<JpaTransactionManager> getJpaTransactionManager();
/**
* Returns a {@link JpaTransactionManager} optimized for bulk loading multi-level JPA entities
* ({@link Domain} and {@link google.registry.model.domain.DomainHistory}). Please refer to {@link
* google.registry.model.bulkquery.BulkQueryEntities} for more information.
*/
@BeamBulkQueryJpaTm
Lazy<JpaTransactionManager> getBulkQueryJpaTransactionManager();
/**
* A {@link JpaTransactionManager} that uses the Postgres read-only replica if configured (uses
* the standard DB otherwise).
@@ -16,6 +16,7 @@ package google.registry.beam.common;
import google.registry.beam.common.RegistryJpaIO.Write;
import google.registry.config.RegistryEnvironment;
import google.registry.model.annotations.DeleteAfterMigration;
import google.registry.persistence.PersistenceModule.JpaTransactionManagerType;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import java.util.Objects;
@@ -65,6 +66,17 @@ public interface RegistryPipelineOptions extends GcpOptions {
void setSqlWriteShards(int maxConcurrentSqlWriters);
@DeleteAfterMigration
@Description(
"Whether to use self allocated primary IDs when building entities. This should only be used"
+ " when the IDs are not significant and the resulting entities are not persisted back to"
+ " the database. Use with caution as self allocated IDs are not unique across workers,"
+ " and persisting entities with these IDs can be dangerous.")
@Default.Boolean(false)
boolean getUseSelfAllocatedId();
void setUseSelfAllocatedId(boolean useSelfAllocatedId);
static RegistryPipelineComponent toRegistryPipelineComponent(RegistryPipelineOptions options) {
return DaggerRegistryPipelineComponent.builder()
.isolationOverride(options.getIsolationOverride())
@@ -22,6 +22,8 @@ import dagger.Lazy;
import google.registry.config.RegistryEnvironment;
import google.registry.config.SystemPropertySetter;
import google.registry.model.AppEngineEnvironment;
import google.registry.model.IdService;
import google.registry.model.IdService.SelfAllocatedIdSupplier;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.persistence.transaction.TransactionManagerFactory;
import org.apache.beam.sdk.harness.JvmInitializer;
@@ -53,9 +55,6 @@ public class RegistryPipelineWorkerInitializer implements JvmInitializer {
toRegistryPipelineComponent(registryOptions);
Lazy<JpaTransactionManager> transactionManagerLazy;
switch (registryOptions.getJpaTransactionManagerType()) {
case BULK_QUERY:
transactionManagerLazy = registryPipelineComponent.getBulkQueryJpaTransactionManager();
break;
case READ_ONLY_REPLICA:
transactionManagerLazy =
registryPipelineComponent.getReadOnlyReplicaJpaTransactionManager();
@@ -65,12 +64,20 @@ public class RegistryPipelineWorkerInitializer implements JvmInitializer {
transactionManagerLazy = registryPipelineComponent.getJpaTransactionManager();
}
TransactionManagerFactory.setJpaTmOnBeamWorker(transactionManagerLazy::get);
// Masquerade all threads as App Engine threads so we can create Ofy keys in the pipeline. Also
// Masquerade all threads as App Engine threads, so we can create Ofy keys in the pipeline. Also
// loads all ofy entities.
new AppEngineEnvironment("s~" + registryPipelineComponent.getProjectId())
.setEnvironmentForAllThreads();
// Set the system property so that we can call IdService.allocateId() without access to
// datastore.
SystemPropertySetter.PRODUCTION_IMPL.setProperty(PROPERTY, "true");
// Use self-allocated IDs if requested. Note that this inevitably results in duplicate IDs from
// multiple workers, which can also collide with existing IDs in the database. So they cannot be
// dependent upon for comparison or anything significant. The resulting entities can never be
// persisted back into the database. This is a stop-gap measure that should only be used when
// you need to create Buildables in Beam, but do not have control over how the IDs are
// allocated, and you don't care about the generated IDs as long
// as you can build the entities.
if (registryOptions.getUseSelfAllocatedId()) {
IdService.setIdSupplier(SelfAllocatedIdSupplier.getInstance());
}
}
}
@@ -37,6 +37,7 @@ import java.util.Optional;
import java.util.regex.Pattern;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.io.TextIO;
@@ -93,8 +94,9 @@ public class InvoicingPipeline implements Serializable {
static PCollection<BillingEvent> readFromCloudSql(
InvoicingPipelineOptions options, Pipeline pipeline) {
Read<Object[], BillingEvent> read =
RegistryJpaIO.read(
makeCloudSqlQuery(options.getYearMonth()), false, row -> parseRow(row).orElse(null));
RegistryJpaIO.<Object[], BillingEvent>read(
makeCloudSqlQuery(options.getYearMonth()), false, row -> parseRow(row).orElse(null))
.withCoder(SerializableCoder.of(BillingEvent.class));
PCollection<BillingEvent> billingEventsWithNulls =
pipeline.apply("Read BillingEvents from Cloud SQL", read);
@@ -128,7 +128,7 @@ import org.joda.time.DateTime;
* <h2>{@link EppResource}</h2>
*
* All EPP resources are loaded from the corresponding {@link HistoryEntry}, which has the resource
* embedded. In general we find most recent history entry before watermark and filter out the ones
* embedded. In general, we find most recent history entry before watermark and filter out the ones
* that are soft-deleted by watermark. The history is emitted as pairs of (resource repo ID: history
* revision ID) from the SQL query.
*
@@ -164,7 +164,7 @@ import org.joda.time.DateTime;
*
* The (pending deposit: deposit fragment) pairs from different resources are combined and grouped
* by pending deposit. For each pending deposit, all the relevant deposit fragments are written into
* a encrypted file stored on GCS. The filename is uniquely determined by the Beam job ID so there
* an encrypted file stored on GCS. The filename is uniquely determined by the Beam job ID so there
* is no need to lock the GCS write operation to prevent stomping. The cursor for staging the
* pending deposit is then rolled forward, and the next action is enqueued. The latter two
* operations are performed in a transaction so the cursor is rolled back if enqueueing failed.
@@ -301,10 +301,11 @@ public class RdePipeline implements Serializable {
.apply(
"Read all production Registrars",
RegistryJpaIO.read(
"SELECT clientIdentifier FROM Registrar WHERE type NOT IN (:types)",
ImmutableMap.of("types", IGNORED_REGISTRAR_TYPES),
String.class,
id -> VKey.createSql(Registrar.class, id)))
"SELECT registrarId FROM Registrar WHERE type NOT IN (:types)",
ImmutableMap.of("types", IGNORED_REGISTRAR_TYPES),
String.class,
x -> x)
.withCoder(StringUtf8Coder.of()))
.apply(
"Marshall Registrar into DepositFragment",
FlatMapElements.into(
@@ -312,7 +313,8 @@ public class RdePipeline implements Serializable {
TypeDescriptor.of(PendingDeposit.class),
TypeDescriptor.of(DepositFragment.class)))
.via(
(VKey<Registrar> key) -> {
(String registrarRepoId) -> {
VKey<Registrar> key = VKey.createSql(Registrar.class, registrarRepoId);
includedRegistrarCounter.inc();
Registrar registrar = jpaTm().transact(() -> jpaTm().loadByKey(key));
DepositFragment fragment = marshaller.marshalRegistrar(registrar);
@@ -337,10 +339,9 @@ public class RdePipeline implements Serializable {
Pipeline pipeline, Class<T> historyClass) {
String repoIdFieldName = HistoryEntryDao.REPO_ID_FIELD_NAMES.get(historyClass);
String resourceFieldName = EPP_RESOURCE_FIELD_NAME.get(historyClass);
return pipeline
.apply(
String.format("Load most recent %s", historyClass.getSimpleName()),
RegistryJpaIO.read(
return pipeline.apply(
String.format("Load most recent %s", historyClass.getSimpleName()),
RegistryJpaIO.read(
("SELECT %repoIdField%, id FROM %entity% WHERE (%repoIdField%, modificationTime)"
+ " IN (SELECT %repoIdField%, MAX(modificationTime) FROM %entity% WHERE"
+ " modificationTime <= :watermark GROUP BY %repoIdField%) AND"
@@ -358,8 +359,8 @@ public class RdePipeline implements Serializable {
.replace("%resourceField%", resourceFieldName),
ImmutableMap.of("watermark", watermark),
Object[].class,
row -> KV.of((String) row[0], (long) row[1])))
.setCoder(KvCoder.of(StringUtf8Coder.of(), VarLongCoder.of()));
row -> KV.of((String) row[0], (long) row[1]))
.withCoder(KvCoder.of(StringUtf8Coder.of(), VarLongCoder.of())));
}
private <T extends HistoryEntry> EppResource loadResourceByHistoryEntryId(
@@ -698,8 +699,8 @@ public class RdePipeline implements Serializable {
}
/**
* Encodes the pending deposit set in an URL safe string that is sent to the pipeline worker by
* the pipeline launcher as a pipeline option.
* Encodes the pending deposit set in a URL safe string that is sent to the pipeline worker by the
* pipeline launcher as a pipeline option.
*/
public static String encodePendingDeposits(ImmutableSet<PendingDeposit> pendingDeposits)
throws IOException {
@@ -715,6 +716,12 @@ public class RdePipeline implements Serializable {
PipelineOptionsFactory.register(RdePipelineOptions.class);
RdePipelineOptions options =
PipelineOptionsFactory.fromArgs(args).withValidation().as(RdePipelineOptions.class);
// We need to self allocate the IDs because the pipeline creates EPP resources from history
// entries and projects them to watermark. These buildable entities would otherwise request an
// ID from datastore, which Beam does not have access to. The IDs are not included in the
// deposits or are these entities persisted back to the database, so it is OK to use a self
// allocated ID to get around the limitations of beam.
options.setUseSelfAllocatedId(true);
RegistryPipelineOptions.validateRegistryPipelineOptions(options);
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_COMMITTED);
DaggerRdePipeline_RdePipelineComponent.builder().options(options).build().rdePipeline().run();
@@ -36,6 +36,7 @@ import java.io.Serializable;
import java.util.concurrent.ThreadLocalRandom;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.GroupIntoBatches;
@@ -91,7 +92,6 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
}
void setupPipeline(Pipeline pipeline) {
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_COMMITTED);
if (options.getFast()) {
fastResaveContacts(pipeline);
fastResaveDomains(pipeline);
@@ -104,10 +104,11 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
private void fastResaveContacts(Pipeline pipeline) {
Read<String, String> repoIdRead =
RegistryJpaIO.read(
"SELECT repoId FROM Contact WHERE transferData.transferStatus = 'PENDING' AND"
+ " transferData.pendingTransferExpirationTime < current_timestamp()",
String.class,
r -> r);
"SELECT repoId FROM Contact WHERE transferData.transferStatus = 'PENDING' AND"
+ " transferData.pendingTransferExpirationTime < current_timestamp()",
String.class,
r -> r)
.withCoder(StringUtf8Coder.of());
projectAndResaveResources(pipeline, Contact.class, repoIdRead);
}
@@ -121,10 +122,11 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
private void fastResaveDomains(Pipeline pipeline) {
Read<String, String> repoIdRead =
RegistryJpaIO.read(
DOMAINS_TO_PROJECT_QUERY,
ImmutableMap.of("END_OF_TIME", DateTimeUtils.END_OF_TIME),
String.class,
r -> r);
DOMAINS_TO_PROJECT_QUERY,
ImmutableMap.of("END_OF_TIME", DateTimeUtils.END_OF_TIME),
String.class,
r -> r)
.withCoder(StringUtf8Coder.of());
projectAndResaveResources(pipeline, Domain.class, repoIdRead);
}
@@ -132,8 +134,9 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
private <T extends EppResource> void forceResaveAllResources(Pipeline pipeline, Class<T> clazz) {
Read<String, String> repoIdRead =
RegistryJpaIO.read(
// Note: cannot use SQL parameters for the table name
String.format("SELECT repoId FROM %s", clazz.getSimpleName()), String.class, r -> r);
// Note: cannot use SQL parameters for the table name
String.format("SELECT repoId FROM %s", clazz.getSimpleName()), String.class, r -> r)
.withCoder(StringUtf8Coder.of());
projectAndResaveResources(pipeline, clazz, repoIdRead);
}
@@ -194,6 +197,7 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
PipelineOptionsFactory.fromArgs(args)
.withValidation()
.as(ResaveAllEppResourcesPipelineOptions.class);
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_REPEATABLE_READ);
new ResaveAllEppResourcesPipeline(options).run();
}
}
@@ -75,8 +75,8 @@ public class SafeBrowsingTransforms {
private final String apiKey;
/**
* Maps a domain name's {@code fullyQualifiedDomainName} to its corresponding {@link
* DomainNameInfo} to facilitate batching SafeBrowsing API requests.
* Maps a domain name's {@code domainName} to its corresponding {@link DomainNameInfo} to
* facilitate batching SafeBrowsing API requests.
*/
private final Map<String, DomainNameInfo> domainNameInfoBuffer =
new LinkedHashMap<>(BATCH_SIZE);
@@ -186,8 +186,8 @@ public class SafeBrowsingTransforms {
private JSONObject createRequestBody() throws JSONException {
// Accumulate all domain names to evaluate.
JSONArray threatArray = new JSONArray();
for (String fullyQualifiedDomainName : domainNameInfoBuffer.keySet()) {
threatArray.put(new JSONObject().put("url", fullyQualifiedDomainName));
for (String domainName : domainNameInfoBuffer.keySet()) {
threatArray.put(new JSONObject().put("url", domainName));
}
// Construct the JSON request body
return new JSONObject()
@@ -37,6 +37,8 @@ import java.io.Serializable;
import javax.inject.Singleton;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
@@ -112,11 +114,12 @@ public class Spec11Pipeline implements Serializable {
static PCollection<DomainNameInfo> readFromCloudSql(Pipeline pipeline) {
Read<Object[], KV<String, String>> read =
RegistryJpaIO.read(
"select d.repoId, r.emailAddress from Domain d join Registrar r on"
+ " d.currentSponsorClientId = r.clientIdentifier where r.type = 'REAL' and"
+ " d.deletionTime > now()",
false,
Spec11Pipeline::parseRow);
"select d.repoId, r.emailAddress from Domain d join Registrar r on"
+ " d.currentSponsorClientId = r.registrarId where r.type = 'REAL' and"
+ " d.deletionTime > now()",
false,
Spec11Pipeline::parseRow)
.withCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()));
return pipeline
.apply("Read active domains from Cloud SQL", read)
@@ -214,8 +217,7 @@ public class Spec11Pipeline implements Serializable {
return output.toString();
} catch (JSONException e) {
throw new RuntimeException(
String.format(
"Encountered an error constructing the JSON for %s", kv.toString()),
String.format("Encountered an error constructing the JSON for %s", kv),
e);
}
}))
@@ -25,28 +25,34 @@ import org.json.JSONObject;
public abstract class ThreatMatch implements Serializable {
private static final String THREAT_TYPE_FIELD = "threatType";
private static final String DOMAIN_NAME_FIELD = "fullyQualifiedDomainName";
private static final String DOMAIN_NAME_FIELD = "domainName";
private static final String OUTDATED_NAME_FIELD = "fullyQualifiedDomainName";
/** Returns what kind of threat it is (malware, phishing etc.) */
public abstract String threatType();
/** Returns the fully qualified domain name [SLD].[TLD] of the matched threat. */
public abstract String fullyQualifiedDomainName();
public abstract String domainName();
@VisibleForTesting
static ThreatMatch create(String threatType, String fullyQualifiedDomainName) {
return new AutoValue_ThreatMatch(threatType, fullyQualifiedDomainName);
static ThreatMatch create(String threatType, String domainName) {
return new AutoValue_ThreatMatch(threatType, domainName);
}
/** Returns a {@link JSONObject} representing a subset of this object's data. */
JSONObject toJSON() throws JSONException {
return new JSONObject()
.put(THREAT_TYPE_FIELD, threatType())
.put(DOMAIN_NAME_FIELD, fullyQualifiedDomainName());
.put(DOMAIN_NAME_FIELD, domainName());
}
/** Parses a {@link JSONObject} and returns an equivalent {@link ThreatMatch}. */
public static ThreatMatch fromJSON(JSONObject threatMatch) throws JSONException {
// TODO: delete OUTDATED_NAME_FIELD once we no longer process reports saved with
// fullyQualifiedDomainName in them, likely 2023
return new AutoValue_ThreatMatch(
threatMatch.getString(THREAT_TYPE_FIELD), threatMatch.getString(DOMAIN_NAME_FIELD));
threatMatch.getString(THREAT_TYPE_FIELD),
threatMatch.has(OUTDATED_NAME_FIELD)
? threatMatch.getString(OUTDATED_NAME_FIELD)
: threatMatch.getString(DOMAIN_NAME_FIELD));
}
}
@@ -16,7 +16,6 @@ package google.registry.config;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.common.collect.ImmutableList;
import dagger.Module;
@@ -37,6 +36,36 @@ import javax.inject.Singleton;
@Module
public abstract class CredentialModule {
/**
* Provides a {@link GoogleCredentialsBundle} backed by the application default credential from
* the Google Cloud Runtime. This credential may be used to access GCP APIs that are NOT part of
* the Google Workspace.
*
* <p>The credential returned by the Cloud Runtime depends on the runtime environment:
*
* <ul>
* <li>On App Engine, returns a scope-less {@code ComputeEngineCredentials} for
* PROJECT_ID@appspot.gserviceaccount.com
* <li>On Compute Engine, returns a scope-less {@code ComputeEngineCredentials} for
* PROJECT_NUMBER-compute@developer.gserviceaccount.com
* <li>On end user host, this returns the credential downloaded by gcloud. Please refer to <a
* href="https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login">Cloud
* SDK documentation</a> for details.
* </ul>
*/
@ApplicationDefaultCredential
@Provides
@Singleton
public static GoogleCredentialsBundle provideApplicationDefaultCredential() {
GoogleCredentials credential;
try {
credential = GoogleCredentials.getApplicationDefault();
} catch (IOException e) {
throw new RuntimeException(e);
}
return GoogleCredentialsBundle.create(credential);
}
/**
* Provides the default {@link GoogleCredentialsBundle} from the Google Cloud runtime.
*
@@ -70,26 +99,19 @@ public abstract class CredentialModule {
}
/**
* Provides the default {@link GoogleCredential} from the Google Cloud runtime for G Suite
* Drive API.
* TODO(b/138195359): Deprecate this credential once we figure out how to use
* {@link GoogleCredentials} for G Suite Drive API.
* Provides a {@link GoogleCredentialsBundle} for accessing Google Workspace APIs, such as Drive
* and Sheets.
*/
@GSuiteDriveCredential
@GoogleWorkspaceCredential
@Provides
@Singleton
public static GoogleCredential provideGSuiteDriveCredential(
public static GoogleCredentialsBundle provideGSuiteDriveCredential(
@ApplicationDefaultCredential GoogleCredentialsBundle applicationDefaultCredential,
@Config("defaultCredentialOauthScopes") ImmutableList<String> requiredScopes) {
GoogleCredential credential;
try {
credential = GoogleCredential.getApplicationDefault();
} catch (IOException e) {
throw new RuntimeException(e);
}
if (credential.createScopedRequired()) {
credential = credential.createScoped(requiredScopes);
}
return credential;
GoogleCredentials credential = applicationDefaultCredential.getGoogleCredentials();
// Although credential is scope-less, its `createScopedRequired` method still returns false.
credential = credential.createScoped(requiredScopes);
return GoogleCredentialsBundle.create(credential);
}
/**
@@ -136,18 +158,24 @@ public abstract class CredentialModule {
.createScoped(requiredScopes));
}
/** Dagger qualifier for the scope-less Application Default Credential. */
@Qualifier
@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface ApplicationDefaultCredential {}
/** Dagger qualifier for the Application Default Credential. */
@Qualifier
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Deprecated // Switching to @ApplicationDefaultCredential
public @interface DefaultCredential {}
/** Dagger qualifier for the credential for G Suite Drive API. */
/** Dagger qualifier for the credential for Google Workspace APIs. */
@Qualifier
@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface GSuiteDriveCredential {}
public @interface GoogleWorkspaceCredential {}
/**
* Dagger qualifier for a credential from a service account's JSON key, to be used in non-request
@@ -37,7 +37,7 @@ import google.registry.dns.writer.BaseDnsWriter;
import google.registry.dns.writer.DnsWriter;
import google.registry.dns.writer.DnsWriterZone;
import google.registry.model.domain.Domain;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.host.Host;
import google.registry.model.tld.Registries;
import google.registry.util.Clock;
@@ -134,10 +134,10 @@ public class CloudDnsWriter extends BaseDnsWriter {
ImmutableSet.Builder<ResourceRecordSet> domainRecords = new ImmutableSet.Builder<>();
// Construct DS records (if any).
Set<DelegationSignerData> dsData = domain.get().getDsData();
Set<DomainDsData> dsData = domain.get().getDsData();
if (!dsData.isEmpty()) {
HashSet<String> dsRrData = new HashSet<>();
for (DelegationSignerData ds : dsData) {
for (DomainDsData ds : dsData) {
dsRrData.add(ds.toRrData());
}
@@ -28,7 +28,7 @@ import google.registry.config.RegistryConfig.Config;
import google.registry.dns.writer.BaseDnsWriter;
import google.registry.dns.writer.DnsWriterZone;
import google.registry.model.domain.Domain;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.host.Host;
import google.registry.model.tld.Registries;
import google.registry.util.Clock;
@@ -185,7 +185,7 @@ public class DnsUpdateWriter extends BaseDnsWriter {
private RRset makeDelegationSignerSet(Domain domain) {
RRset signerSet = new RRset();
for (DelegationSignerData signerData : domain.getDsData()) {
for (DomainDsData signerData : domain.getDsData()) {
DSRecord dsRecord =
new DSRecord(
toAbsoluteName(domain.getDomainName()),
@@ -102,6 +102,7 @@
<target>backend</target>
</cron>
<!-- TODO(b/249863289): disable until it is safe to run this pipeline
<cron>
<url><![CDATA[/_dr/task/resaveAllEppResourcesPipeline?fast=true]]></url>
<description>
@@ -110,6 +111,7 @@
<schedule>1st monday of month 09:00</schedule>
<target>backend</target>
</cron>
-->
<cron>
<url><![CDATA[/_dr/task/updateRegistrarRdapBaseUrls]]></url>
@@ -267,7 +269,7 @@
about 2 hours to complete, so we give 11 hours to be safe. Normally, we give 24+ hours (see
icannReportingStaging), but the invoicing team prefers receiving the e-mail on the first of
each month. -->
<schedule>1 of month 17:00</schedule>
<schedule>1 of month 19:00</schedule>
<target>backend</target>
</cron>
@@ -86,6 +86,7 @@
<target>backend</target>
</cron>
<!-- TODO(b/249863289): disable until it is safe to run this pipeline
<cron>
<url><![CDATA[/_dr/task/resaveAllEppResourcesPipeline?fast=true]]></url>
<description>
@@ -94,6 +95,7 @@
<schedule>1st monday of month 09:00</schedule>
<target>backend</target>
</cron>
-->
<cron>
<url><![CDATA[/_dr/cron/fanout?queue=retryable-cron-tasks&endpoint=/_dr/task/exportDomainLists&runInEmpty]]></url>
@@ -14,16 +14,16 @@
package google.registry.export;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.services.drive.Drive;
import dagger.Component;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule;
import google.registry.config.CredentialModule.GSuiteDriveCredential;
import google.registry.config.CredentialModule.GoogleWorkspaceCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.storage.drive.DriveConnection;
import google.registry.util.GoogleCredentialsBundle;
import javax.inject.Singleton;
/** Dagger module for Google {@link Drive} service connection objects. */
@@ -32,13 +32,13 @@ public final class DriveModule {
@Provides
static Drive provideDrive(
@GSuiteDriveCredential GoogleCredential googleCredential,
@GoogleWorkspaceCredential GoogleCredentialsBundle googleCredential,
@Config("projectId") String projectId) {
return new Drive.Builder(
googleCredential.getTransport(),
googleCredential.getHttpTransport(),
googleCredential.getJsonFactory(),
googleCredential)
googleCredential.getHttpRequestInitializer())
.setApplicationName(projectId)
.build();
}
@@ -91,10 +91,10 @@ public class ExportDomainListsAction implements Runnable {
// field that compares with the substituted value.
jpaTm()
.query(
"SELECT fullyQualifiedDomainName FROM Domain "
"SELECT domainName FROM Domain "
+ "WHERE tld = :tld "
+ "AND deletionTime > :now "
+ "ORDER by fullyQualifiedDomainName ASC",
+ "ORDER by domainName ASC",
String.class)
.setParameter("tld", tld)
.setParameter("now", clock.nowUtc())
@@ -163,7 +163,7 @@ public final class SyncGroupMembersAction implements Runnable {
registrarsToSave.add(result.getKey().asBuilder().setContactsRequireSyncing(false).build());
}
}
tm().transactNew(() -> tm().updateAll(registrarsToSave.build()));
tm().transact(() -> tm().updateAll(registrarsToSave.build()));
return errors;
}
@@ -17,7 +17,7 @@ package google.registry.export.sheet;
import com.google.api.services.sheets.v4.Sheets;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule.JsonCredential;
import google.registry.config.CredentialModule.GoogleWorkspaceCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.util.GoogleCredentialsBundle;
@@ -27,7 +27,7 @@ public final class SheetsServiceModule {
@Provides
static Sheets provideSheets(
@JsonCredential GoogleCredentialsBundle credentialsBundle,
@GoogleWorkspaceCredential GoogleCredentialsBundle credentialsBundle,
@Config("projectId") String projectId) {
return new Sheets.Builder(
credentialsBundle.getHttpTransport(),
@@ -114,7 +114,7 @@ class SyncRegistrarsSheet {
// and you'll need to remove deleted columns probably like a week after
// deployment.
//
builder.put("clientIdentifier", convert(registrar.getRegistrarId()));
builder.put("registrarId", convert(registrar.getRegistrarId()));
builder.put("registrarName", convert(registrar.getRegistrarName()));
builder.put("state", convert(registrar.getState()));
builder.put("ianaIdentifier", convert(registrar.getIanaIdentifier()));
@@ -248,7 +248,7 @@ public final class DomainCreateFlow implements TransactionalFlow {
validateRegistrationPeriod(years);
verifyResourceDoesNotExist(Domain.class, targetId, now, registrarId);
// Validate that this is actually a legal domain name on a TLD that the registrar has access to.
InternetDomainName domainName = validateDomainName(command.getFullyQualifiedDomainName());
InternetDomainName domainName = validateDomainName(command.getDomainName());
String domainLabel = domainName.parts().get(0);
Registry registry = Registry.get(domainName.parent().toString());
validateCreateCommandContactsAndNameservers(command, registry, domainName);
@@ -639,10 +639,7 @@ public final class DomainCreateFlow implements TransactionalFlow {
}
private static PollMessage.OneTime createNameCollisionOneTimePollMessage(
String fullyQualifiedDomainName,
HistoryEntry historyEntry,
String registrarId,
DateTime now) {
String domainName, HistoryEntry historyEntry, String registrarId, DateTime now) {
return new PollMessage.OneTime.Builder()
.setRegistrarId(registrarId)
.setEventTime(now)
@@ -650,7 +647,7 @@ public final class DomainCreateFlow implements TransactionalFlow {
.setResponseData(
ImmutableList.of(
DomainPendingActionNotificationResponse.create(
fullyQualifiedDomainName, true, historyEntry.getTrid(), now)))
domainName, true, historyEntry.getTrid(), now)))
.setHistoryEntry(historyEntry)
.build();
}
@@ -105,7 +105,7 @@ import google.registry.model.domain.launch.LaunchNotice;
import google.registry.model.domain.launch.LaunchNotice.InvalidChecksumException;
import google.registry.model.domain.launch.LaunchPhase;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.domain.secdns.SecDnsCreateExtension;
import google.registry.model.domain.secdns.SecDnsInfoExtension;
import google.registry.model.domain.secdns.SecDnsUpdateExtension;
@@ -316,14 +316,14 @@ public class DomainFlowUtils {
}
/** Check that the DS data that will be set on a domain is valid. */
static void validateDsData(Set<DelegationSignerData> dsData) throws EppException {
static void validateDsData(Set<DomainDsData> dsData) throws EppException {
if (dsData != null) {
if (dsData.size() > MAX_DS_RECORDS_PER_DOMAIN) {
throw new TooManyDsRecordsException(
String.format(
"A maximum of %s DS records are allowed per domain.", MAX_DS_RECORDS_PER_DOMAIN));
}
ImmutableList<DelegationSignerData> invalidAlgorithms =
ImmutableList<DomainDsData> invalidAlgorithms =
dsData.stream()
.filter(ds -> !validateAlgorithm(ds.getAlgorithm()))
.collect(toImmutableList());
@@ -333,7 +333,7 @@ public class DomainFlowUtils {
"Domain contains DS record(s) with an invalid algorithm wire value: %s",
invalidAlgorithms));
}
ImmutableList<DelegationSignerData> invalidDigestTypes =
ImmutableList<DomainDsData> invalidDigestTypes =
dsData.stream()
.filter(ds -> !DigestType.fromWireValue(ds.getDigestType()).isPresent())
.collect(toImmutableList());
@@ -343,7 +343,7 @@ public class DomainFlowUtils {
"Domain contains DS record(s) with an invalid digest type: %s",
invalidDigestTypes));
}
ImmutableList<DelegationSignerData> digestsWithInvalidDigestLength =
ImmutableList<DomainDsData> digestsWithInvalidDigestLength =
dsData.stream()
.filter(
ds ->
@@ -920,16 +920,15 @@ public class DomainFlowUtils {
* and we are going to ignore it; clients who don't care about secDNS can just ignore it.
*/
static void addSecDnsExtensionIfPresent(
ImmutableList.Builder<ResponseExtension> extensions,
ImmutableSet<DelegationSignerData> dsData) {
ImmutableList.Builder<ResponseExtension> extensions, ImmutableSet<DomainDsData> dsData) {
if (!dsData.isEmpty()) {
extensions.add(SecDnsInfoExtension.create(dsData));
}
}
/** Update {@link DelegationSignerData} based on an update extension command. */
static ImmutableSet<DelegationSignerData> updateDsData(
ImmutableSet<DelegationSignerData> oldDsData, SecDnsUpdateExtension secDnsUpdate)
/** Update {@link DomainDsData} based on an update extension command. */
static ImmutableSet<DomainDsData> updateDsData(
ImmutableSet<DomainDsData> oldDsData, SecDnsUpdateExtension secDnsUpdate)
throws EppException {
// We don't support 'urgent' because we do everything as fast as we can anyways.
if (Boolean.TRUE.equals(secDnsUpdate.getUrgent())) { // We allow both false and null.
@@ -948,8 +947,8 @@ public class DomainFlowUtils {
if (remove != null && Boolean.FALSE.equals(remove.getAll())) {
throw new SecDnsAllUsageException(); // Explicit all=false is meaningless.
}
Set<DelegationSignerData> toAdd = (add == null) ? ImmutableSet.of() : add.getDsData();
Set<DelegationSignerData> toRemove =
Set<DomainDsData> toAdd = (add == null) ? ImmutableSet.of() : add.getDsData();
Set<DomainDsData> toRemove =
(remove == null)
? ImmutableSet.of()
: (remove.getAll() == null) ? remove.getDsData() : oldDsData;
@@ -1001,9 +1000,9 @@ public class DomainFlowUtils {
validateRegistrantAllowedOnTld(tld, command.getRegistrantContactId());
validateNoDuplicateContacts(command.getContacts());
validateRequiredContactsPresent(command.getRegistrant(), command.getContacts());
ImmutableSet<String> fullyQualifiedHostNames = command.getNameserverFullyQualifiedHostNames();
validateNameserversCountForTld(tld, domainName, fullyQualifiedHostNames.size());
validateNameserversAllowedOnTld(tld, fullyQualifiedHostNames);
ImmutableSet<String> hostNames = command.getNameserverHostNames();
validateNameserversCountForTld(tld, domainName, hostNames.size());
validateNameserversAllowedOnTld(tld, hostNames);
}
/** Validate the secDNS extension, if present. */
@@ -1542,11 +1541,11 @@ public class DomainFlowUtils {
/** Nameservers are not allow-listed for this TLD. */
public static class NameserversNotAllowedForTldException
extends StatusProhibitsOperationException {
public NameserversNotAllowedForTldException(Set<String> fullyQualifiedHostNames) {
public NameserversNotAllowedForTldException(Set<String> hostNames) {
super(
String.format(
"Nameservers '%s' are not allow-listed for this TLD",
Joiner.on(',').join(fullyQualifiedHostNames)));
Joiner.on(',').join(hostNames)));
}
}
@@ -109,7 +109,7 @@ public final class DomainInfoFlow implements Flow {
// This is a policy decision that is left up to us by the rfcs.
DomainInfoData.Builder infoBuilder =
DomainInfoData.newBuilder()
.setFullyQualifiedDomainName(domain.getDomainName())
.setDomainName(domain.getDomainName())
.setRepoId(domain.getRepoId())
.setCurrentSponsorClientId(domain.getCurrentSponsorRegistrarId())
.setRegistrant(
@@ -32,8 +32,6 @@ import static google.registry.flows.domain.DomainTransferUtils.createLosingTrans
import static google.registry.flows.domain.DomainTransferUtils.createPendingTransferData;
import static google.registry.flows.domain.DomainTransferUtils.createTransferResponse;
import static google.registry.flows.domain.DomainTransferUtils.createTransferServerApproveEntities;
import static google.registry.flows.domain.token.AllocationTokenFlowUtils.maybeApplyPackageRemovalToken;
import static google.registry.flows.domain.token.AllocationTokenFlowUtils.verifyTokenAllowedOnDomain;
import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
import static google.registry.model.reporting.HistoryEntry.Type.DOMAIN_TRANSFER_REQUEST;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
@@ -65,7 +63,6 @@ import google.registry.model.domain.fee.FeeTransferCommandExtension;
import google.registry.model.domain.fee.FeeTransformResponseExtension;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.domain.superuser.DomainTransferRequestSuperuserExtension;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.domain.token.AllocationTokenExtension;
import google.registry.model.eppcommon.AuthInfo;
import google.registry.model.eppcommon.StatusValue;
@@ -135,8 +132,6 @@ import org.joda.time.DateTime;
* google.registry.flows.domain.token.AllocationTokenFlowUtils.AllocationTokenNotValidForTldException}
* @error {@link
* google.registry.flows.domain.token.AllocationTokenFlowUtils.AlreadyRedeemedAllocationTokenException}
* @error {@link
* google.registry.flows.domain.token.AllocationTokenFlowUtils.MissingRemovePackageTokenOnPackageDomainException}
*/
@ReportingSpec(ActivityReportField.DOMAIN_TRANSFER_REQUEST)
public final class DomainTransferRequestFlow implements TransactionalFlow {
@@ -174,23 +169,19 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
extensionManager.validate();
DateTime now = tm().getTransactionTime();
Domain existingDomain = loadAndVerifyExistence(Domain.class, targetId, now);
Optional<AllocationToken> allocationToken =
allocationTokenFlowUtils.verifyAllocationTokenIfPresent(
existingDomain,
Registry.get(existingDomain.getTld()),
gainingClientId,
now,
eppInput.getSingleExtension(AllocationTokenExtension.class));
allocationTokenFlowUtils.verifyAllocationTokenIfPresent(
existingDomain,
Registry.get(existingDomain.getTld()),
gainingClientId,
now,
eppInput.getSingleExtension(AllocationTokenExtension.class));
Optional<DomainTransferRequestSuperuserExtension> superuserExtension =
eppInput.getSingleExtension(DomainTransferRequestSuperuserExtension.class);
Period period =
superuserExtension.isPresent()
? superuserExtension.get().getRenewalPeriod()
: ((Transfer) resourceCommand).getPeriod();
verifyTransferAllowed(existingDomain, period, now, superuserExtension, allocationToken);
// If client passed an applicable static token this updates the domain
existingDomain = maybeApplyPackageRemovalToken(existingDomain, allocationToken);
verifyTransferAllowed(existingDomain, period, now, superuserExtension);
String tld = existingDomain.getTld();
Registry registry = Registry.get(tld);
@@ -303,11 +294,9 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
Domain existingDomain,
Period period,
DateTime now,
Optional<DomainTransferRequestSuperuserExtension> superuserExtension,
Optional<AllocationToken> allocationToken)
Optional<DomainTransferRequestSuperuserExtension> superuserExtension)
throws EppException {
verifyNoDisallowedStatuses(existingDomain, DISALLOWED_STATUSES);
verifyTokenAllowedOnDomain(existingDomain, allocationToken);
if (!isSuperuser) {
verifyAuthInfoPresentForResourceTransfer(authInfo);
verifyAuthInfo(authInfo.get(), existingDomain);
@@ -218,7 +218,7 @@ public final class DomainTransferUtils {
TransferData transferData,
@Nullable DateTime extendedRegistrationExpirationTime) {
return new DomainTransferResponse.Builder()
.setFullyQualifiedDomainName(targetId)
.setDomainName(targetId)
.setGainingRegistrarId(transferData.getGainingRegistrarId())
.setLosingRegistrarId(transferData.getLosingRegistrarId())
.setPendingTransferExpirationTime(transferData.getPendingTransferExpirationTime())
@@ -74,7 +74,7 @@ import google.registry.model.domain.DomainCommand.Update.Change;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.fee.FeeUpdateCommandExtension;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.domain.secdns.SecDnsUpdateExtension;
import google.registry.model.domain.superuser.DomainUpdateSuperuserExtension;
import google.registry.model.eppcommon.AuthInfo;
@@ -87,6 +87,7 @@ import google.registry.model.poll.PendingActionNotificationResponse.DomainPendin
import google.registry.model.poll.PollMessage;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.model.tld.Registry;
import java.util.Objects;
import java.util.Optional;
import javax.inject.Inject;
import org.joda.time.DateTime;
@@ -181,8 +182,7 @@ public final class DomainUpdateFlow implements TransactionalFlow {
DomainHistory domainHistory =
historyBuilder.setType(DOMAIN_UPDATE).setDomain(newDomain).build();
validateNewState(newDomain);
if (newDomain.getDsData() != existingDomain.getDsData()
|| newDomain.getNsHosts() != existingDomain.getNsHosts()) {
if (requiresDnsUpdate(existingDomain, newDomain)) {
dnsQueue.addDomainRefreshTask(targetId);
}
ImmutableSet.Builder<ImmutableObject> entitiesToSave = new ImmutableSet.Builder<>();
@@ -206,6 +206,16 @@ public final class DomainUpdateFlow implements TransactionalFlow {
return responseBuilder.build();
}
/** Determines if any of the changes to new domain should trigger DNS update. */
private boolean requiresDnsUpdate(Domain existingDomain, Domain newDomain) {
if (existingDomain.shouldPublishToDns() != newDomain.shouldPublishToDns()
|| !Objects.equals(newDomain.getDsData(), existingDomain.getDsData())
|| !Objects.equals(newDomain.getNsHosts(), existingDomain.getNsHosts())) {
return true;
}
return false;
}
/** Fail if the object doesn't exist or was deleted. */
private void verifyUpdateAllowed(Update command, Domain existingDomain, DateTime now)
throws EppException {
@@ -232,8 +242,7 @@ public final class DomainUpdateFlow implements TransactionalFlow {
validateContactsHaveTypes(add.getContacts());
validateContactsHaveTypes(remove.getContacts());
validateRegistrantAllowedOnTld(tld, command.getInnerChange().getRegistrantContactId());
validateNameserversAllowedOnTld(
tld, add.getNameserverFullyQualifiedHostNames());
validateNameserversAllowedOnTld(tld, add.getNameserverHostNames());
}
private Domain performUpdate(Update command, Domain domain, DateTime now) throws EppException {
@@ -263,7 +272,7 @@ public final class DomainUpdateFlow implements TransactionalFlow {
secDnsUpdate.isPresent()
? updateDsData(
domain.getDsData().stream()
.map(DelegationSignerData::cloneWithoutDomainRepoId)
.map(DomainDsData::cloneWithoutDomainRepoId)
.collect(toImmutableSet()),
secDnsUpdate.get())
: domain.getDsData())
@@ -175,11 +175,7 @@ public class AllocationTokenFlowUtils {
return Optional.empty();
}
AllocationToken tokenEntity = loadToken(extension.get().getAllocationToken());
validateToken(
InternetDomainName.from(command.getFullyQualifiedDomainName()),
tokenEntity,
registrarId,
now);
validateToken(InternetDomainName.from(command.getDomainName()), tokenEntity, registrarId, now);
return Optional.of(
tokenCustomLogic.validateToken(command, tokenEntity, registry, registrarId, now));
}
@@ -134,7 +134,7 @@ public final class HostCreateFlow implements TransactionalFlow {
superordinateDomain
.get()
.asBuilder()
.addSubordinateHost(command.getFullyQualifiedHostName())
.addSubordinateHost(command.getHostName())
.build());
// Only update DNS if this is a subordinate host. External hosts have no glue to write, so
// they are only written as NS records from the referencing domain.
@@ -93,7 +93,7 @@ public final class HostInfoFlow implements Flow {
return responseBuilder
.setResData(
hostInfoDataBuilder
.setFullyQualifiedHostName(host.getHostName())
.setHostName(host.getHostName())
.setRepoId(host.getRepoId())
.setStatusValues(statusValues.build())
.setInetAddresses(host.getInetAddresses())
@@ -129,7 +129,7 @@ public final class HostUpdateFlow implements TransactionalFlow {
extensionManager.validate();
Update command = (Update) resourceCommand;
Change change = command.getInnerChange();
String suppliedNewHostName = change.getFullyQualifiedHostName();
String suppliedNewHostName = change.getHostName();
DateTime now = tm().getTransactionTime();
validateHostName(targetId);
Host existingHost = loadAndVerifyExistence(Host.class, targetId, now);
@@ -260,7 +260,7 @@ public final class HostUpdateFlow implements TransactionalFlow {
dnsQueue.addHostRefreshTask(existingHost.getHostName());
}
// In case of a rename, there are many updates we need to queue up.
if (((Update) resourceCommand).getInnerChange().getFullyQualifiedHostName() != null) {
if (((Update) resourceCommand).getInnerChange().getHostName() != null) {
// If the renamed host is also subordinate, then we must enqueue an update to write the new
// glue.
if (newHost.isSubordinate()) {
@@ -108,7 +108,7 @@ public final class PollAckFlow implements TransactionalFlow {
// acked, then we return a special status code indicating that. Note that the query will
// include the message being acked.
int messageCount = tm().doTransactionless(() -> getPollMessageCount(registrarId, now));
int messageCount = tm().transact(() -> getPollMessageCount(registrarId, now));
if (messageCount <= 0) {
return responseBuilder.setResultFromCode(SUCCESS_WITH_NO_MESSAGES).build();
}
@@ -56,7 +56,7 @@ import org.joda.time.DateTime;
/** An EPP entity object (i.e. a domain, contact, or host). */
@MappedSuperclass
@Access(AccessType.FIELD) // otherwise it'll use the default if the repoId (property)
public abstract class EppResource extends BackupGroupRoot implements Buildable {
public abstract class EppResource extends UpdateAutoTimestampEntity implements Buildable {
private static final long serialVersionUID = -252782773382339534L;
@@ -140,10 +140,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
DateTime lastEppUpdateTime;
/** Status values associated with this resource. */
@Ignore
@Column(name = "statuses")
// TODO(b/177567432): rename to "statuses" once we're off datastore.
Set<StatusValue> status;
@Ignore Set<StatusValue> statuses;
public String getRepoId() {
return repoId;
@@ -189,7 +186,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
}
public final ImmutableSet<StatusValue> getStatusValues() {
return nullToEmptyImmutableCopy(status);
return nullToEmptyImmutableCopy(statuses);
}
public DateTime getDeletionTime() {
@@ -307,7 +304,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
statusValue,
resourceClass.getSimpleName());
}
getInstance().status = statusValues;
getInstance().statuses = statusValues;
return thisCastToDerived();
}
@@ -342,7 +339,7 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
/**
* Set the update timestamp.
*
* <p>This is provided at EppResource since BackupGroupRoot doesn't have a Builder.
* <p>This is provided at EppResource since UpdateAutoTimestampEntity doesn't have a Builder.
*/
public B setUpdateTimestamp(UpdateAutoTimestamp updateTimestamp) {
getInstance().setUpdateTimestamp(updateTimestamp);
@@ -368,13 +365,13 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
@Override
public EppResource load(VKey<? extends EppResource> key) {
return replicaTm().doTransactionless(() -> replicaTm().loadByKey(key));
return replicaTm().transact(() -> replicaTm().loadByKey(key));
}
@Override
public Map<VKey<? extends EppResource>, EppResource> loadAll(
Iterable<? extends VKey<? extends EppResource>> keys) {
return replicaTm().doTransactionless(() -> replicaTm().loadByKeys(keys));
return replicaTm().transact(() -> replicaTm().loadByKeys(keys));
}
};
@@ -16,7 +16,6 @@ package google.registry.model;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.DateTimeUtils.START_OF_TIME;
@@ -67,7 +66,7 @@ public final class EppResourceUtils {
+ "AND deletionTime > :now";
// We have to use the native SQL query here because DomainHost table doesn't have its entity
// class so we cannot reference its property like domainHost.hostRepoId in a JPQL query.
// class, so we cannot reference its property like domainHost.hostRepoId in a JPQL query.
private static final String HOST_LINKED_DOMAIN_QUERY =
"SELECT d.repo_id FROM \"Domain\" d "
+ "JOIN \"DomainHost\" dh ON dh.domain_repo_id = d.repo_id "
@@ -260,7 +259,7 @@ public final class EppResourceUtils {
/**
* Rewinds an {@link EppResource} object to a given point in time.
*
* <p>This method costs nothing if {@code resource} is already current. Otherwise it needs to
* <p>This method costs nothing if {@code resource} is already current. Otherwise, it needs to
* perform a single fetch operation.
*
* <p><b>Warning:</b> A resource can only be rolled backwards in time, not forwards; therefore
@@ -292,7 +291,7 @@ public final class EppResourceUtils {
/**
* Rewinds an {@link EppResource} object to a given point in time.
*
* <p>This method costs nothing if {@code resource} is already current. Otherwise it returns an
* <p>This method costs nothing if {@code resource} is already current. Otherwise, it returns an
* async operation that performs a single fetch operation.
*
* @return an asynchronous operation returning resource at {@code timestamp} or {@code null} if
@@ -346,50 +345,37 @@ public final class EppResourceUtils {
"key must be either VKey<Contact> or VKey<Host>, but it is %s",
key);
boolean isContactKey = key.getKind().equals(Contact.class);
if (tm().isOfy()) {
com.googlecode.objectify.cmd.Query<Domain> query =
auditedOfy()
.load()
.type(Domain.class)
.filter(isContactKey ? "allContacts.contact" : "nsHosts", key.getOfyKey())
.filter("deletionTime >", now);
if (limit != null) {
query.limit(limit);
}
return query.keys().list().stream().map(Domain::createVKey).collect(toImmutableSet());
} else {
return tm().transact(
() -> {
Query query;
if (isContactKey) {
query =
jpaTm()
.query(CONTACT_LINKED_DOMAIN_QUERY, String.class)
.setParameter("fkRepoId", key)
.setParameter("now", now);
} else {
query =
jpaTm()
.getEntityManager()
.createNativeQuery(HOST_LINKED_DOMAIN_QUERY)
.setParameter("fkRepoId", key.getSqlKey())
.setParameter("now", now.toDate());
}
if (limit != null) {
query.setMaxResults(limit);
}
@SuppressWarnings("unchecked")
ImmutableSet<VKey<Domain>> domainKeySet =
(ImmutableSet<VKey<Domain>>)
query
.getResultStream()
.map(
repoId ->
Domain.createVKey(Key.create(Domain.class, (String) repoId)))
.collect(toImmutableSet());
return domainKeySet;
});
}
return tm().transact(
() -> {
Query query;
if (isContactKey) {
query =
jpaTm()
.query(CONTACT_LINKED_DOMAIN_QUERY, String.class)
.setParameter("fkRepoId", key)
.setParameter("now", now);
} else {
query =
jpaTm()
.getEntityManager()
.createNativeQuery(HOST_LINKED_DOMAIN_QUERY)
.setParameter("fkRepoId", key.getSqlKey())
.setParameter("now", now.toDate());
}
if (limit != null) {
query.setMaxResults(limit);
}
@SuppressWarnings("unchecked")
ImmutableSet<VKey<Domain>> domainKeySet =
(ImmutableSet<VKey<Domain>>)
query
.getResultStream()
.map(
repoId ->
Domain.createVKey(Key.create(Domain.class, (String) repoId)))
.collect(toImmutableSet());
return domainKeySet;
});
}
/**
@@ -58,8 +58,8 @@ public final class ForeignKeyUtils {
RESOURCE_TYPE_TO_FK_PROPERTY =
ImmutableMap.of(
Contact.class, "contactId",
Domain.class, "fullyQualifiedDomainName",
Host.class, "fullyQualifiedHostName");
Domain.class, "domainName",
Host.class, "hostName");
/**
* Loads a {@link VKey} to an {@link EppResource} from the database by foreign key.
@@ -17,58 +17,114 @@ package google.registry.model;
import static com.google.common.base.Preconditions.checkState;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.flogger.FluentLogger;
import google.registry.beam.common.RegistryPipelineWorkerInitializer;
import google.registry.config.RegistryEnvironment;
import google.registry.model.annotations.DeleteAfterMigration;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Supplier;
/**
* Allocates a globally unique {@link Long} number to use as an Ofy {@code @Id}.
* Allocates a {@link long} to use as a {@code @Id}, (part) of the primary SQL key for an entity.
*
* <p>In non-test, non-beam environments the Id is generated by Datastore, otherwise it's from an
* atomic long number that's incremented every time this method is called.
* <p>Normally, the ID is globally unique and allocated by Datastore. It is possible to override
* this behavior by providing an ID supplier, such as in unit tests, where a self-allocated ID based
* on a monotonically increasing atomic {@link long} is used. Such an ID supplier can also be used
* in other scenarios, such as in a Beam pipeline to get around the limitation of Beam's inability
* to use GAE SDK to access Datastore. The override should be used with great care lest it results
* in irreversible data corruption.
*
* @see #setIdSupplier(Supplier)
*/
@DeleteAfterMigration
public final class IdService {
/**
* A placeholder String passed into DatastoreService.allocateIds that ensures that all ids are
* initialized from the same id pool.
*/
private static final String APP_WIDE_ALLOCATION_KIND = "common";
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private IdService() {}
private static Supplier<Long> idSupplier =
RegistryEnvironment.UNITTEST.equals(RegistryEnvironment.get())
? SelfAllocatedIdSupplier.getInstance()
: DatastoreIdSupplier.getInstance();
/**
* Counts of used ids for use in unit tests or Beam.
* Provides a {@link Supplier} of ID that overrides the default.
*
* <p>Note that one should only use self-allocate Ids in Beam for entities whose Ids are not
* important and are not persisted back to the database, i. e. nowhere the uniqueness of the ID is
* required.
* <p>Currently, the only use case for an override is in the Beam pipeline, where access to
* Datastore is not possible through the App Engine API. As such, the setter explicitly checks if
* the runtime is Beam.
*
* <p>Because the provided supplier is not guaranteed to be globally unique and compatible with
* existing IDs in the database, one should proceed with great care. It is safe to use an
* arbitrary supplier when the resulting IDs are not significant and not persisted back to the
* database, i.e. the IDs are only required by the {@link Buildable} contract but are not used in
* any meaningful way. One example is the RDE pipeline where we project EPP resource entities from
* history entries to watermark time, which are then marshalled into XML elements in the RDE
* deposits, where the IDs are omitted.
*/
private static final AtomicLong nextSelfAllocatedId = new AtomicLong(1); // ids cannot be zero
private static final boolean isSelfAllocated() {
return RegistryEnvironment.UNITTEST.equals(RegistryEnvironment.get())
|| "true".equals(System.getProperty(RegistryPipelineWorkerInitializer.PROPERTY, "false"));
public static void setIdSupplier(Supplier<Long> idSupplier) {
checkState(
"true".equals(System.getProperty(RegistryPipelineWorkerInitializer.PROPERTY, "false")),
"Can only set ID supplier in a Beam pipeline");
logger.atWarning().log("Using ID supplier override!");
IdService.idSupplier = idSupplier;
}
/** Allocates an id. */
// TODO(b/201547855): Find a way to allocate a unique ID without datastore.
public static long allocateId() {
return isSelfAllocated()
? nextSelfAllocatedId.getAndIncrement()
: DatastoreServiceFactory.getDatastoreService()
.allocateIds(APP_WIDE_ALLOCATION_KIND, 1)
.iterator()
.next()
.getId();
return idSupplier.get();
}
/** Resets the global self-allocated id counter (i.e. sets the next id to 1). */
@VisibleForTesting
public static void resetSelfAllocatedId() {
checkState(
isSelfAllocated(), "Can only call resetSelfAllocatedId() in unit tests or Beam pipelines");
nextSelfAllocatedId.set(1); // ids cannot be zero
// TODO(b/201547855): Find a way to allocate a unique ID without datastore.
private static class DatastoreIdSupplier implements Supplier<Long> {
private static final DatastoreIdSupplier INSTANCE = new DatastoreIdSupplier();
/**
* A placeholder String passed into {@code DatastoreService.allocateIds} that ensures that all
* IDs are initialized from the same ID pool.
*/
private static final String APP_WIDE_ALLOCATION_KIND = "common";
public static DatastoreIdSupplier getInstance() {
return INSTANCE;
}
@Override
public Long get() {
return DatastoreServiceFactory.getDatastoreService()
.allocateIds(APP_WIDE_ALLOCATION_KIND, 1)
.iterator()
.next()
.getId();
}
}
/**
* An ID supplier that allocates an ID from a monotonically increasing atomic {@link long}.
*
* <p>The generated IDs are only unique within the same JVM. It is not suitable for production use
* unless in cases the IDs are not significant.
*/
public static class SelfAllocatedIdSupplier implements Supplier<Long> {
private static final SelfAllocatedIdSupplier INSTANCE = new SelfAllocatedIdSupplier();
/** Counts of used ids for self allocating IDs. */
private static final AtomicLong nextSelfAllocatedId = new AtomicLong(1); // ids cannot be zero
public static SelfAllocatedIdSupplier getInstance() {
return INSTANCE;
}
@Override
public Long get() {
return nextSelfAllocatedId.getAndIncrement();
}
public void reset() {
nextSelfAllocatedId.set(1);
}
}
}
@@ -14,17 +14,13 @@
package google.registry.model;
import static com.google.common.collect.Iterables.transform;
import static com.google.common.collect.Maps.transformValues;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static java.lang.annotation.ElementType.FIELD;
import static java.lang.annotation.RetentionPolicy.RUNTIME;
import static java.util.stream.Collectors.toCollection;
import static java.util.stream.Collectors.toList;
import com.google.common.base.Joiner;
import com.google.common.collect.Maps;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Ignore;
import google.registry.persistence.VKey;
@@ -56,15 +52,6 @@ public abstract class ImmutableObject implements Cloneable {
@Target(FIELD)
public @interface DoNotHydrate {}
/**
* Indicates that the field should be ignored when comparing an object in the datastore to the
* corresponding object in Cloud SQL.
*/
@Documented
@Retention(RUNTIME)
@Target(FIELD)
public @interface DoNotCompare {}
/**
* Indicates that the field stores a null value to indicate an empty set. This is also used in
* object comparison.
@@ -105,7 +92,7 @@ public abstract class ImmutableObject implements Cloneable {
*/
protected Map<Field, Object> getSignificantFields() {
// Can't use streams or ImmutableMap because we can have null values.
Map<Field, Object> result = new LinkedHashMap();
Map<Field, Object> result = new LinkedHashMap<>();
for (Map.Entry<Field, Object> entry : ModelUtils.getFieldValues(this).entrySet()) {
if (!entry.getKey().isAnnotationPresent(Insignificant.class)) {
result.put(entry.getKey(), entry.getValue());
@@ -190,15 +177,15 @@ public abstract class ImmutableObject implements Cloneable {
/** Helper function to recursively hydrate an ImmutableObject. */
private static Object hydrate(Object value) {
if (value instanceof Key) {
if (tm().isOfy()) {
return hydrate(auditedOfy().load().key((Key<?>) value).now());
}
return value;
} else if (value instanceof Map) {
}
if (value instanceof Map) {
return transformValues((Map<?, ?>) value, ImmutableObject::hydrate);
} else if (value instanceof Collection) {
return transform((Collection<?>) value, ImmutableObject::hydrate);
} else if (value instanceof ImmutableObject) {
}
if (value instanceof Collection) {
return ((Collection<?>) value).stream().map(ImmutableObject::hydrate);
}
if (value instanceof ImmutableObject) {
return ((ImmutableObject) value).toHydratedString();
}
return value;
@@ -220,7 +207,7 @@ public abstract class ImmutableObject implements Cloneable {
}
return result;
} else if (o instanceof Map) {
return Maps.transformValues((Map<?, ?>) o, ImmutableObject::toMapRecursive);
return transformValues((Map<?, ?>) o, ImmutableObject::toMapRecursive);
} else if (o instanceof Set) {
return ((Set<?>) o)
.stream()
@@ -68,7 +68,7 @@ public class OteStats {
((DomainCommand.Create)
((ResourceCommandWrapper) eppInput.getCommandWrapper().getCommand())
.getResourceCommand())
.getFullyQualifiedDomainName()
.getDomainName()
.startsWith(ACE_PREFIX);
private static final Predicate<EppInput> IS_SUBORDINATE =
@@ -64,7 +64,7 @@ public final class ResourceTransferUtils {
DomainTransferData domainTransferData = (DomainTransferData) transferData;
builder =
new DomainTransferResponse.Builder()
.setFullyQualifiedDomainName(eppResource.getForeignKey())
.setDomainName(eppResource.getForeignKey())
.setExtendedRegistrationExpirationTime(
ADD_EXDATE_STATUSES.contains(domainTransferData.getTransferStatus())
? domainTransferData.getTransferredRegistrationExpirationTime()
@@ -29,12 +29,7 @@ import org.joda.time.DateTime;
@Embeddable
public class UpdateAutoTimestamp extends ImmutableObject implements UnsafeSerializable {
// When set to true, database converters/translators should do the auto update. When set to
// false, auto update should be suspended (this exists to allow us to preserve the original value
// during a replay).
private static final ThreadLocal<Boolean> autoUpdateEnabled = ThreadLocal.withInitial(() -> true);
@Column(nullable = false)
@Column(name = "updateTimestamp")
DateTime lastUpdateTime;
// Unfortunately, we cannot use the @UpdateTimestamp annotation on "lastUpdateTime" in this class
@@ -43,9 +38,7 @@ public class UpdateAutoTimestamp extends ImmutableObject implements UnsafeSerial
@PrePersist
@PreUpdate
void setTimestamp() {
if (autoUpdateEnabled() || lastUpdateTime == null) {
lastUpdateTime = jpaTm().getTransactionTime();
}
lastUpdateTime = jpaTm().getTransactionTime();
}
/** Returns the timestamp, or {@code START_OF_TIME} if it's null. */
@@ -58,30 +51,4 @@ public class UpdateAutoTimestamp extends ImmutableObject implements UnsafeSerial
instance.lastUpdateTime = timestamp;
return instance;
}
// TODO(b/175610935): Remove the auto-update disabling code below after migration.
/** Class to allow us to safely disable auto-update in a try-with-resources block. */
public static class DisableAutoUpdateResource implements AutoCloseable {
DisableAutoUpdateResource() {
autoUpdateEnabled.set(false);
}
@Override
public void close() {
autoUpdateEnabled.set(true);
}
}
/**
* Resturns a resource that disables auto-updates on all {@link UpdateAutoTimestamp}s in the
* current thread, suitable for use with in a try-with-resources block.
*/
public static DisableAutoUpdateResource disableAutoUpdate() {
return new DisableAutoUpdateResource();
}
public static boolean autoUpdateEnabled() {
return autoUpdateEnabled.get();
}
}
@@ -18,21 +18,16 @@ import com.googlecode.objectify.annotation.Ignore;
import google.registry.util.PreconditionsUtils;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.AttributeOverride;
import javax.persistence.Column;
import javax.persistence.MappedSuperclass;
import javax.xml.bind.annotation.XmlTransient;
/**
* Base class for entities that are the root of a Registry 2.0 entity group that gets enrolled in
* commit logs for backup purposes.
*
* <p>The commit log system needs to preserve the ordering of closely timed mutations to entities in
* a single entity group. We require an {@link UpdateAutoTimestamp} field on the root of a group so
* that we can enforce strictly increasing timestamps.
* Base class for entities that contains an {@link UpdateAutoTimestamp} which is updated every time
* the entity is persisted.
*/
@MappedSuperclass
public abstract class BackupGroupRoot extends ImmutableObject implements UnsafeSerializable {
public abstract class UpdateAutoTimestampEntity extends ImmutableObject
implements UnsafeSerializable {
/**
* An automatically managed timestamp of when this object was last written to Datastore.
@@ -44,7 +39,6 @@ public abstract class BackupGroupRoot extends ImmutableObject implements UnsafeS
// Prevents subclasses from unexpectedly accessing as property (e.g., Host), which would
// require an unnecessary non-private setter method.
@Access(AccessType.FIELD)
@AttributeOverride(name = "lastUpdateTime", column = @Column(name = "updateTimestamp"))
@Ignore
UpdateAutoTimestamp updateTimestamp = UpdateAutoTimestamp.create(null);
@@ -59,7 +53,7 @@ public abstract class BackupGroupRoot extends ImmutableObject implements UnsafeS
* <p>This method is for the few cases when {@code updateTimestamp} is copied between different
* types of entities. Use {@link #clone} for same-type copying.
*/
protected void copyUpdateTimestamp(BackupGroupRoot other) {
protected void copyUpdateTimestamp(UpdateAutoTimestampEntity other) {
this.updateTimestamp = PreconditionsUtils.checkArgumentNotNull(other, "other").updateTimestamp;
}
@@ -38,7 +38,7 @@ import google.registry.model.domain.rgp.GracePeriodStatus;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
import google.registry.persistence.VKey;
import google.registry.persistence.WithLongVKey;
import google.registry.persistence.WithVKey;
import google.registry.persistence.converter.JodaMoneyType;
import java.util.Optional;
import java.util.Set;
@@ -295,7 +295,7 @@ public abstract class BillingEvent extends ImmutableObject
@Index(columnList = "cancellation_matching_billing_recurrence_id")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_event_id"))
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class OneTime extends BillingEvent {
/** The billable value. */
@@ -473,7 +473,7 @@ public abstract class BillingEvent extends ImmutableObject
@Index(columnList = "recurrence_time_of_year")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_recurrence_id"))
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class Recurring extends BillingEvent {
/**
@@ -606,7 +606,7 @@ public abstract class BillingEvent extends ImmutableObject
@Index(columnList = "billing_recurrence_id")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_cancellation_id"))
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class Cancellation extends BillingEvent {
/** The billing time of the charge that is being cancelled. */
@@ -1,114 +0,0 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.model.bulkquery;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import google.registry.model.domain.Domain;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.GracePeriod;
import google.registry.model.domain.GracePeriod.GracePeriodHistory;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsDataHistory;
import google.registry.model.host.Host;
import google.registry.model.reporting.DomainTransactionRecord;
import google.registry.persistence.VKey;
import google.registry.persistence.transaction.JpaTransactionManager;
/**
* Utilities for managing an alternative JPA entity model optimized for bulk loading multi-level
* entities such as {@link Domain} and {@link DomainHistory}.
*
* <p>In a bulk query for a multi-level JPA entity type, the JPA framework only generates a bulk
* query (SELECT * FROM table) for the base table. Then, for each row in the base table, additional
* queries are issued to load associated rows in child tables. This can be very slow when an entity
* type has multiple child tables.
*
* <p>We have defined an alternative entity model for {@link Domain} and {@link DomainHistory},
* where the base table as well as the child tables are mapped to single-level entity types. The
* idea is to load each of these types using a bulk query, and assemble them into the target type in
* memory in a pipeline. The main use case is Datastore-Cloud SQL validation during the Registry
* database migration, where we will need the full database snapshots frequently.
*/
public class BulkQueryEntities {
/**
* The JPA entity classes in persistence.xml to replace when creating the {@link
* JpaTransactionManager} for bulk query.
*/
public static final ImmutableMap<String, String> JPA_ENTITIES_REPLACEMENTS =
ImmutableMap.of(
Domain.class.getCanonicalName(),
DomainLite.class.getCanonicalName(),
DomainHistory.class.getCanonicalName(),
DomainHistoryLite.class.getCanonicalName());
/* The JPA entity classes that are not included in persistence.xml and need to be added to
* the {@link JpaTransactionManager} for bulk query.*/
public static final ImmutableList<String> JPA_ENTITIES_NEW =
ImmutableList.of(
DomainHost.class.getCanonicalName(), DomainHistoryHost.class.getCanonicalName());
public static Domain assembleDomain(
DomainLite domainLite,
ImmutableSet<GracePeriod> gracePeriods,
ImmutableSet<DelegationSignerData> delegationSignerData,
ImmutableSet<VKey<Host>> nsHosts) {
Domain.Builder builder = new Domain.Builder();
builder.copyFrom(domainLite);
builder.setGracePeriods(gracePeriods);
builder.setDsData(delegationSignerData);
builder.setNameservers(nsHosts);
// Restore the original update timestamp (this gets cleared when we set nameservers or DS data).
builder.setUpdateTimestamp(domainLite.getUpdateTimestamp());
return builder.build();
}
public static DomainHistory assembleDomainHistory(
DomainHistoryLite domainHistoryLite,
ImmutableSet<DomainDsDataHistory> dsDataHistories,
ImmutableSet<VKey<Host>> domainHistoryHosts,
ImmutableSet<GracePeriodHistory> gracePeriodHistories,
ImmutableSet<DomainTransactionRecord> transactionRecords) {
DomainHistory.Builder builder = new DomainHistory.Builder();
builder.copyFrom(domainHistoryLite);
DomainBase rawDomainBase = domainHistoryLite.domainBase;
if (rawDomainBase != null) {
DomainBase newDomainBase =
domainHistoryLite
.domainBase
.asBuilder()
.setNameservers(domainHistoryHosts)
.setGracePeriods(
gracePeriodHistories.stream()
.map(GracePeriod::createFromHistory)
.collect(toImmutableSet()))
.setDsData(
dsDataHistories.stream()
.map(DelegationSignerData::create)
.collect(toImmutableSet()))
// Restore the original update timestamp (this gets cleared when we set nameservers or
// DS data).
.setUpdateTimestamp(domainHistoryLite.domainBase.getUpdateTimestamp())
.build();
builder.setDomain(newDomainBase);
}
return builder.buildAndAssemble(
dsDataHistories, domainHistoryHosts, gracePeriodHistories, transactionRecords);
}
}
@@ -1,69 +0,0 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.model.bulkquery;
import com.google.common.base.Objects;
import google.registry.model.domain.DomainHistory.DomainHistoryId;
import google.registry.model.host.Host;
import google.registry.persistence.VKey;
import java.io.Serializable;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.IdClass;
/**
* A name server host referenced by a {@link google.registry.model.domain.DomainHistory} record.
* Please refer to {@link BulkQueryEntities} for usage.
*/
@Entity
@Access(AccessType.FIELD)
@IdClass(DomainHistoryHost.class)
public class DomainHistoryHost implements Serializable {
@Id private Long domainHistoryHistoryRevisionId;
@Id private String domainHistoryDomainRepoId;
@Id private String hostRepoId;
private DomainHistoryHost() {}
public DomainHistoryId getDomainHistoryId() {
return new DomainHistoryId(domainHistoryDomainRepoId, domainHistoryHistoryRevisionId);
}
public VKey<Host> getHostVKey() {
return VKey.create(Host.class, hostRepoId);
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof DomainHistoryHost)) {
return false;
}
DomainHistoryHost that = (DomainHistoryHost) o;
return Objects.equal(domainHistoryHistoryRevisionId, that.domainHistoryHistoryRevisionId)
&& Objects.equal(domainHistoryDomainRepoId, that.domainHistoryDomainRepoId)
&& Objects.equal(hostRepoId, that.hostRepoId);
}
@Override
public int hashCode() {
return Objects.hashCode(domainHistoryHistoryRevisionId, domainHistoryDomainRepoId, hostRepoId);
}
}
@@ -1,125 +0,0 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.model.bulkquery;
import com.googlecode.objectify.Key;
import google.registry.model.domain.Domain;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.DomainHistory.DomainHistoryId;
import google.registry.model.domain.Period;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.VKey;
import javax.annotation.Nullable;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.AttributeOverride;
import javax.persistence.AttributeOverrides;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.IdClass;
import javax.persistence.PostLoad;
/**
* A 'light' version of {@link DomainHistory} with only base table ("DomainHistory") attributes,
* which allows fast bulk loading. They are used in in-memory assembly of {@code DomainHistory}
* instances along with bulk-loaded child entities ({@code GracePeriodHistory} etc). The in-memory
* assembly achieves much higher performance than loading {@code DomainHistory} directly.
*
* <p>Please refer to {@link BulkQueryEntities} for more information.
*
* <p>This class is adapted from {@link DomainHistory} by removing the {@code dsDataHistories},
* {@code gracePeriodHistories}, and {@code nsHosts} fields and associated methods.
*/
@Entity(name = "DomainHistory")
@Access(AccessType.FIELD)
@IdClass(DomainHistoryId.class)
public class DomainHistoryLite extends HistoryEntry {
// Store DomainBase instead of Domain so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@Nullable DomainBase domainBase;
@Id
@Access(AccessType.PROPERTY)
public String getDomainRepoId() {
// We need to handle null case here because Hibernate sometimes accesses this method before
// parent gets initialized
return parent == null ? null : parent.getName();
}
/** This method is private because it is only used by Hibernate. */
@SuppressWarnings("unused")
private void setDomainRepoId(String domainRepoId) {
parent = Key.create(Domain.class, domainRepoId);
}
@Override
@Nullable
@Access(AccessType.PROPERTY)
@AttributeOverrides({
@AttributeOverride(name = "unit", column = @Column(name = "historyPeriodUnit")),
@AttributeOverride(name = "value", column = @Column(name = "historyPeriodValue"))
})
public Period getPeriod() {
return super.getPeriod();
}
/**
* For transfers, the id of the other registrar.
*
* <p>For requests and cancels, the other registrar is the losing party (because the registrar
* sending the EPP transfer command is the gaining party). For approves and rejects, the other
* registrar is the gaining party.
*/
@Nullable
@Access(AccessType.PROPERTY)
@Column(name = "historyOtherRegistrarId")
@Override
public String getOtherRegistrarId() {
return super.getOtherRegistrarId();
}
@Id
@Column(name = "historyRevisionId")
@Access(AccessType.PROPERTY)
@Override
public long getId() {
return super.getId();
}
/** The key to the {@link Domain} this is based off of. */
public VKey<Domain> getParentVKey() {
return VKey.create(Domain.class, getDomainRepoId());
}
public DomainHistoryId getDomainHistoryId() {
return new DomainHistoryId(getDomainRepoId(), getId());
}
@PostLoad
void postLoad() {
if (domainBase == null) {
return;
}
// See inline comments in DomainHistory.postLoad for reasons for the following lines.
if (domainBase.getDomainName() == null) {
domainBase = null;
} else if (domainBase.getRepoId() == null) {
domainBase.setRepoId(parent.getName());
}
}
}
@@ -1,64 +0,0 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.model.bulkquery;
import com.google.common.base.Objects;
import google.registry.model.host.Host;
import google.registry.persistence.VKey;
import java.io.Serializable;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.IdClass;
/** A name server host of a domain. Please refer to {@link BulkQueryEntities} for usage. */
@Entity
@Access(AccessType.FIELD)
@IdClass(DomainHost.class)
public class DomainHost implements Serializable {
@Id private String domainRepoId;
@Id private String hostRepoId;
DomainHost() {}
public String getDomainRepoId() {
return domainRepoId;
}
public VKey<Host> getHostVKey() {
return VKey.create(Host.class, hostRepoId);
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof DomainHost)) {
return false;
}
DomainHost that = (DomainHost) o;
return Objects.equal(domainRepoId, that.domainRepoId)
&& Objects.equal(hostRepoId, that.hostRepoId);
}
@Override
public int hashCode() {
return Objects.hashCode(domainRepoId, hostRepoId);
}
}
@@ -1,48 +0,0 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.model.bulkquery;
import google.registry.model.domain.Domain;
import google.registry.model.domain.DomainBase;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Entity;
/**
* A 'light' version of {@link Domain} with only base table ("Domain") attributes, which allows fast
* bulk loading. They are used in in-memory assembly of {@code Domain} instances along with
* bulk-loaded child entities ({@code GracePeriod} etc). The in-memory assembly achieves much higher
* performance than loading {@code Domain} directly.
*
* <p>Please refer to {@link BulkQueryEntities} for more information.
*/
@Entity(name = "Domain")
@WithStringVKey
@Access(AccessType.FIELD)
public class DomainLite extends DomainBase {
@Override
@javax.persistence.Id
@Access(AccessType.PROPERTY)
public String getRepoId() {
return super.getRepoId();
}
public static VKey<DomainLite> createVKey(String repoId) {
return VKey.createSql(DomainLite.class, repoId);
}
}
@@ -20,11 +20,12 @@ import static google.registry.util.DateTimeUtils.START_OF_TIME;
import google.registry.model.ImmutableObject;
import google.registry.model.UnsafeSerializable;
import google.registry.model.UpdateAutoTimestamp;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.model.common.Cursor.CursorId;
import google.registry.model.tld.Registry;
import google.registry.persistence.VKey;
import java.util.Optional;
import javax.persistence.AttributeOverride;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.EnumType;
@@ -41,7 +42,10 @@ import org.joda.time.DateTime;
*/
@Entity
@IdClass(CursorId.class)
public class Cursor extends ImmutableObject implements UnsafeSerializable {
@AttributeOverride(
name = "updateTimestamp.lastUpdateTime",
column = @Column(nullable = false, name = "lastUpdateTime"))
public class Cursor extends UpdateAutoTimestampEntity {
private static final long serialVersionUID = 5777891565780594961L;
@@ -122,10 +126,6 @@ public class Cursor extends ImmutableObject implements UnsafeSerializable {
@Column(nullable = false)
DateTime cursorTime = START_OF_TIME;
/** An automatically managed timestamp of when this object was last written to Datastore. */
@Column(nullable = false)
UpdateAutoTimestamp lastUpdateTime = UpdateAutoTimestamp.create(null);
@Override
public VKey<Cursor> createVKey() {
return createVKey(type, scope);
@@ -145,9 +145,10 @@ public class Cursor extends ImmutableObject implements UnsafeSerializable {
}
public DateTime getLastUpdateTime() {
return lastUpdateTime.getTimestamp();
return getUpdateTimestamp().getTimestamp();
}
public String getScope() {
return scope;
}
@@ -22,8 +22,8 @@ import static google.registry.util.PasswordUtils.SALT_SUPPLIER;
import static google.registry.util.PasswordUtils.hashPassword;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import google.registry.model.BackupGroupRoot;
import google.registry.model.Buildable;
import google.registry.model.UpdateAutoTimestampEntity;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
@@ -39,7 +39,7 @@ import javax.persistence.Table;
@Index(columnList = "gaiaId", name = "user_gaia_id_idx"),
@Index(columnList = "emailAddress", name = "user_email_address_idx")
})
public class User extends BackupGroupRoot implements Buildable {
public class User extends UpdateAutoTimestampEntity implements Buildable {
/** Autogenerated unique ID of this user. */
@Id
@@ -18,7 +18,7 @@ import google.registry.model.EppResource.ForeignKeyedEppResource;
import google.registry.model.annotations.ExternalMessagingName;
import google.registry.model.annotations.ReportedOn;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Entity;
@@ -45,7 +45,7 @@ import org.joda.time.DateTime;
@Index(columnList = "searchName")
})
@ExternalMessagingName("contact")
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@Access(AccessType.FIELD)
public class Contact extends ContactBase implements ForeignKeyedEppResource {
@@ -60,7 +60,7 @@ public class ContactHistory extends HistoryEntry implements UnsafeSerializable {
// Store ContactBase instead of Contact so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@DoNotCompare @Nullable ContactBase contactBase;
@Nullable ContactBase contactBase;
@Id
@Access(AccessType.PROPERTY)
@@ -19,10 +19,10 @@ import google.registry.model.EppResource;
import google.registry.model.EppResource.ForeignKeyedEppResource;
import google.registry.model.annotations.ExternalMessagingName;
import google.registry.model.annotations.ReportedOn;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.host.Host;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import java.util.Set;
import javax.persistence.Access;
import javax.persistence.AccessType;
@@ -66,7 +66,7 @@ import org.joda.time.DateTime;
@Index(columnList = "transfer_billing_event_id"),
@Index(columnList = "transfer_billing_recurrence_id")
})
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@ExternalMessagingName("domain")
@Access(AccessType.FIELD)
public class Domain extends DomainBase implements ForeignKeyedEppResource {
@@ -116,15 +116,14 @@ public class Domain extends DomainBase implements ForeignKeyedEppResource {
}
/**
* Returns the set of {@link DelegationSignerData} associated with the domain.
* Returns the set of {@link DomainDsData} associated with the domain.
*
* <p>This is the getter method specific for Hibernate to access the field, so it is set to
* private. The caller can use the public {@link #getDsData()} to get the DS data.
*
* <p>Note that we need to set `insertable = false, updatable = false` for @JoinColumn, otherwise
* Hibernate would try to set the foreign key to null(through an UPDATE TABLE sql) instead of
* deleting the whole entry from the table when the {@link DelegationSignerData} is removed from
* the set.
* deleting the whole entry from the table when the {@link DomainDsData} is removed from the set.
*/
@Access(AccessType.PROPERTY)
@OneToMany(cascade = CascadeType.ALL, fetch = FetchType.EAGER, orphanRemoval = true)
@@ -134,7 +133,7 @@ public class Domain extends DomainBase implements ForeignKeyedEppResource {
insertable = false,
updatable = false)
@SuppressWarnings("UnusedMethod")
private Set<DelegationSignerData> getInternalDelegationSignerData() {
private Set<DomainDsData> getInternalDelegationSignerData() {
return dsData;
}
@@ -53,8 +53,9 @@ import google.registry.model.billing.BillingEvent;
import google.registry.model.contact.Contact;
import google.registry.model.domain.launch.LaunchNotice;
import google.registry.model.domain.rgp.GracePeriodStatus;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.domain.token.AllocationToken.TokenType;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.host.Host;
import google.registry.model.poll.PollMessage;
@@ -112,17 +113,14 @@ public class DomainBase extends EppResource
* Fully qualified domain name (puny-coded), which serves as the foreign key for this domain.
*
* <p>This is only unique in the sense that for any given lifetime specified as the time range
* from (creationTime, deletionTime) there can only be one domain in Datastore with this name.
* from (creationTime, deletionTime) there can only be one domain in the database with this name.
* However, there can be many domains with the same name and non-overlapping lifetimes.
*
* @invariant fullyQualifiedDomainName == fullyQualifiedDomainName.toLowerCase(Locale.ENGLISH)
* @invariant domainName == domainName.toLowerCase(Locale.ENGLISH)
*/
// TODO(b/177567432): Rename this to domainName when we are off Datastore
@Column(name = "domainName")
@Index
String fullyQualifiedDomainName;
@Index String domainName;
/** The top level domain this is under, dernormalized from {@link #fullyQualifiedDomainName}. */
/** The top level domain this is under, dernormalized from {@link #domainName}. */
@Index String tld;
/** References to hosts that are the nameservers for the domain. */
@@ -145,7 +143,7 @@ public class DomainBase extends EppResource
DomainAuthInfo authInfo;
/** Data used to construct DS records for this domain. */
@Ignore @Transient Set<DelegationSignerData> dsData;
@Ignore @Transient Set<DomainDsData> dsData;
/**
* The claims notice supplied when this domain was created, if there was one.
@@ -340,14 +338,14 @@ public class DomainBase extends EppResource
@Override
public String getForeignKey() {
return fullyQualifiedDomainName;
return domainName;
}
public String getDomainName() {
return fullyQualifiedDomainName;
return domainName;
}
public ImmutableSet<DelegationSignerData> getDsData() {
public ImmutableSet<DomainDsData> getDsData() {
return nullToEmptyImmutableCopy(dsData);
}
@@ -392,9 +390,9 @@ public class DomainBase extends EppResource
// Hibernate needs this in order to populate dsData but no one else should ever use it
@SuppressWarnings("UnusedMethod")
private void setInternalDelegationSignerData(Set<DelegationSignerData> dsData) {
private void setInternalDelegationSignerData(Set<DomainDsData> dsData) {
if (this.dsData instanceof PersistentSet) {
Set<DelegationSignerData> nonNullDsData = nullToEmpty(dsData);
Set<DomainDsData> nonNullDsData = nullToEmpty(dsData);
this.dsData.retainAll(nonNullDsData);
this.dsData.addAll(nonNullDsData);
} else {
@@ -728,9 +726,9 @@ public class DomainBase extends EppResource
// If there is no autorenew end time, set it to END_OF_TIME.
instance.autorenewEndTime = firstNonNull(getInstance().autorenewEndTime, END_OF_TIME);
checkArgumentNotNull(emptyToNull(instance.fullyQualifiedDomainName), "Missing domainName");
checkArgumentNotNull(emptyToNull(instance.domainName), "Missing domainName");
checkArgumentNotNull(instance.getRegistrant(), "Missing registrant");
instance.tld = getTldFromDomainName(instance.fullyQualifiedDomainName);
instance.tld = getTldFromDomainName(instance.domainName);
T newDomain = super.build();
// Hibernate throws exception if gracePeriods or dsData is null because we enabled all
@@ -751,11 +749,11 @@ public class DomainBase extends EppResource
domainName.equals(canonicalizeHostname(domainName)),
"Domain name %s not in puny-coded, lower-case form",
domainName);
getInstance().fullyQualifiedDomainName = domainName;
getInstance().domainName = domainName;
return thisCastToDerived();
}
public B setDsData(ImmutableSet<DelegationSignerData> dsData) {
public B setDsData(ImmutableSet<DomainDsData> dsData) {
getInstance().dsData = dsData;
getInstance().resetUpdateTimestamp();
return thisCastToDerived();
@@ -922,6 +920,21 @@ public class DomainBase extends EppResource
}
public B setCurrentPackageToken(@Nullable VKey<AllocationToken> currentPackageToken) {
if (currentPackageToken == null) {
getInstance().currentPackageToken = currentPackageToken;
return thisCastToDerived();
}
AllocationToken token =
tm().transact(() -> tm().loadByKeyIfPresent(currentPackageToken))
.orElseThrow(
() ->
new IllegalArgumentException(
String.format(
"The package token %s does not exist",
currentPackageToken.getSqlKey())));
checkArgument(
token.getTokenType().equals(TokenType.PACKAGE),
"The currentPackageToken must have a PACKAGE TokenType");
getInstance().currentPackageToken = currentPackageToken;
return thisCastToDerived();
}
@@ -105,9 +105,9 @@ public class DomainCommand {
@XmlRootElement
@XmlType(
propOrder = {
"fullyQualifiedDomainName",
"domainName",
"period",
"nameserverFullyQualifiedHostNames",
"nameserverHostNames",
"registrantContactId",
"foreignKeyedDesignatedContacts",
"authInfo"
@@ -117,12 +117,12 @@ public class DomainCommand {
/** Fully qualified domain name, which serves as a unique identifier for this domain. */
@XmlElement(name = "name")
String fullyQualifiedDomainName;
String domainName;
/** Fully qualified host names of the hosts that are the nameservers for the domain. */
@XmlElementWrapper(name = "ns")
@XmlElement(name = "hostObj")
Set<String> nameserverFullyQualifiedHostNames;
Set<String> nameserverHostNames;
/** Resolved keys to hosts that are the nameservers for the domain. */
@XmlTransient Set<VKey<Host>> nameservers;
@@ -144,15 +144,15 @@ public class DomainCommand {
@Override
public String getTargetId() {
return fullyQualifiedDomainName;
return domainName;
}
public String getFullyQualifiedDomainName() {
return fullyQualifiedDomainName;
public String getDomainName() {
return domainName;
}
public ImmutableSet<String> getNameserverFullyQualifiedHostNames() {
return nullToEmptyImmutableCopy(nameserverFullyQualifiedHostNames);
public ImmutableSet<String> getNameserverHostNames() {
return nullToEmptyImmutableCopy(nameserverHostNames);
}
public ImmutableSet<VKey<Host>> getNameservers() {
@@ -172,7 +172,7 @@ public class DomainCommand {
@Override
public Create cloneAndLinkReferences(DateTime now) throws InvalidReferencesException {
Create clone = clone(this);
clone.nameservers = linkHosts(clone.nameserverFullyQualifiedHostNames, now);
clone.nameservers = linkHosts(clone.nameserverHostNames, now);
if (registrantContactId == null) {
clone.contacts = linkContacts(clone.foreignKeyedDesignatedContacts, now);
} else {
@@ -205,7 +205,7 @@ public class DomainCommand {
/** The name of the domain to look up, and an attribute specifying the host lookup type. */
@XmlElement(name = "name")
NameWithHosts fullyQualifiedDomainName;
NameWithHosts domainName;
DomainAuthInfo authInfo;
@@ -244,12 +244,12 @@ public class DomainCommand {
/** Get the enum that specifies the requested hosts (applies only to info flows). */
public HostsRequest getHostsRequest() {
// Null "hosts" is implicitly ALL.
return MoreObjects.firstNonNull(fullyQualifiedDomainName.hosts, HostsRequest.ALL);
return MoreObjects.firstNonNull(domainName.hosts, HostsRequest.ALL);
}
@Override
public String getTargetId() {
return fullyQualifiedDomainName.name;
return domainName.name;
}
@Override
@@ -337,15 +337,12 @@ public class DomainCommand {
}
/** The inner change type on a domain update command. */
@XmlType(propOrder = {
"nameserverFullyQualifiedHostNames",
"foreignKeyedDesignatedContacts",
"statusValues"})
@XmlType(propOrder = {"nameserverHostNames", "foreignKeyedDesignatedContacts", "statusValues"})
public static class AddRemove extends ResourceUpdate.AddRemove {
/** Fully qualified host names of the hosts that are the nameservers for the domain. */
@XmlElementWrapper(name = "ns")
@XmlElement(name = "hostObj")
Set<String> nameserverFullyQualifiedHostNames;
Set<String> nameserverHostNames;
/** Resolved keys to hosts that are the nameservers for the domain. */
@XmlTransient Set<VKey<Host>> nameservers;
@@ -358,8 +355,8 @@ public class DomainCommand {
@XmlTransient
Set<DesignatedContact> contacts;
public ImmutableSet<String> getNameserverFullyQualifiedHostNames() {
return nullSafeImmutableCopy(nameserverFullyQualifiedHostNames);
public ImmutableSet<String> getNameserverHostNames() {
return nullSafeImmutableCopy(nameserverHostNames);
}
public ImmutableSet<VKey<Host>> getNameservers() {
@@ -373,7 +370,7 @@ public class DomainCommand {
/** Creates a copy of this {@link AddRemove} with hard links to hosts and contacts. */
private AddRemove cloneAndLinkReferences(DateTime now) throws InvalidReferencesException {
AddRemove clone = clone(this);
clone.nameservers = linkHosts(clone.nameserverFullyQualifiedHostNames, now);
clone.nameservers = linkHosts(clone.nameserverHostNames, now);
clone.contacts = linkContacts(clone.foreignKeyedDesignatedContacts, now);
return clone;
}
@@ -412,13 +409,12 @@ public class DomainCommand {
}
}
private static Set<VKey<Host>> linkHosts(Set<String> fullyQualifiedHostNames, DateTime now)
private static Set<VKey<Host>> linkHosts(Set<String> hostNames, DateTime now)
throws InvalidReferencesException {
if (fullyQualifiedHostNames == null) {
if (hostNames == null) {
return null;
}
return ImmutableSet.copyOf(
loadByForeignKeysCached(fullyQualifiedHostNames, Host.class, now).values());
return ImmutableSet.copyOf(loadByForeignKeysCached(hostNames, Host.class, now).values());
}
private static Set<DesignatedContact> linkContacts(
@@ -25,7 +25,7 @@ import google.registry.model.EppResource;
import google.registry.model.ImmutableObject;
import google.registry.model.domain.DomainHistory.DomainHistoryId;
import google.registry.model.domain.GracePeriod.GracePeriodHistory;
import google.registry.model.domain.secdns.DelegationSignerData;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.domain.secdns.DomainDsDataHistory;
import google.registry.model.host.Host;
import google.registry.model.reporting.DomainTransactionRecord;
@@ -83,7 +83,7 @@ public class DomainHistory extends HistoryEntry {
// Store DomainBase instead of Domain so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@DoNotCompare @Nullable DomainBase domainBase;
@Nullable DomainBase domainBase;
@Id
@Access(AccessType.PROPERTY)
@@ -102,7 +102,6 @@ public class DomainHistory extends HistoryEntry {
// We could have reused domainBase.nsHosts here, but Hibernate throws a weird exception after
// we change to use a composite primary key.
// TODO(b/166776754): Investigate if we can reuse domainBase.nsHosts for storing host keys.
@DoNotCompare
@ElementCollection
@JoinTable(
name = "DomainHistoryHost",
@@ -116,7 +115,6 @@ public class DomainHistory extends HistoryEntry {
@Column(name = "host_repo_id")
Set<VKey<Host>> nsHosts;
@DoNotCompare
@OneToMany(
cascade = {CascadeType.ALL},
fetch = FetchType.EAGER,
@@ -137,7 +135,6 @@ public class DomainHistory extends HistoryEntry {
@Ignore
Set<DomainDsDataHistory> dsDataHistories = new HashSet<>();
@DoNotCompare
@OneToMany(
cascade = {CascadeType.ALL},
fetch = FetchType.EAGER,
@@ -280,7 +277,7 @@ public class DomainHistory extends HistoryEntry {
.map(GracePeriod::createFromHistory)
.collect(toImmutableSet());
domainBase.dsData =
dsDataHistories.stream().map(DelegationSignerData::create).collect(toImmutableSet());
dsDataHistories.stream().map(DomainDsData::create).collect(toImmutableSet());
// Normally Hibernate would see that the domain fields are all null and would fill
// domainBase with a null object. Unfortunately, the updateTimestamp is never null in SQL.
if (domainBase.getDomainName() == null) {
@@ -406,19 +403,5 @@ public class DomainHistory extends HistoryEntry {
fillAuxiliaryFieldsFromDomain(instance);
return instance;
}
public DomainHistory buildAndAssemble(
ImmutableSet<DomainDsDataHistory> dsDataHistories,
ImmutableSet<VKey<Host>> domainHistoryHosts,
ImmutableSet<GracePeriodHistory> gracePeriodHistories,
ImmutableSet<DomainTransactionRecord> transactionRecords) {
DomainHistory instance = super.build();
instance.dsDataHistories = dsDataHistories;
instance.nsHosts = domainHistoryHosts;
instance.gracePeriodHistories = gracePeriodHistories;
instance.domainTransactionRecords = transactionRecords;
instance.hashCode = null;
return instance;
}
}
}
@@ -30,28 +30,30 @@ import org.joda.time.DateTime;
/** The {@link ResponseData} returned for an EPP info flow on a domain. */
@XmlRootElement(name = "infData")
@XmlType(propOrder = {
"fullyQualifiedDomainName",
"repoId",
"statusValues",
"registrant",
"contacts",
"nameservers",
"subordinateHosts",
"currentSponsorClientId",
"creationClientId",
"creationTime",
"lastEppUpdateClientId",
"lastEppUpdateTime",
"registrationExpirationTime",
"lastTransferTime",
"authInfo"})
@XmlType(
propOrder = {
"domainName",
"repoId",
"statusValues",
"registrant",
"contacts",
"nameservers",
"subordinateHosts",
"currentSponsorClientId",
"creationClientId",
"creationTime",
"lastEppUpdateClientId",
"lastEppUpdateTime",
"registrationExpirationTime",
"lastTransferTime",
"authInfo"
})
@AutoValue
@CopyAnnotations
public abstract class DomainInfoData implements ResponseData {
@XmlElement(name = "name")
abstract String getFullyQualifiedDomainName();
abstract String getDomainName();
@XmlElement(name = "roid")
abstract String getRepoId();
@@ -110,7 +112,8 @@ public abstract class DomainInfoData implements ResponseData {
/** Builder for {@link DomainInfoData}. */
@AutoValue.Builder
public abstract static class Builder {
public abstract Builder setFullyQualifiedDomainName(String fullyQualifiedDomainName);
public abstract Builder setDomainName(String domainName);
public abstract Builder setRepoId(String repoId);
public abstract Builder setStatusValues(@Nullable ImmutableSet<StatusValue> statusValues);
public abstract Builder setRegistrant(String registrant);
@@ -20,10 +20,11 @@ import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import google.registry.model.Buildable;
import google.registry.model.CreateAutoTimestamp;
import google.registry.model.ImmutableObject;
import google.registry.model.UpdateAutoTimestamp;
import google.registry.model.UpdateAutoTimestampEntity;
import java.util.Optional;
import javax.annotation.Nullable;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.AttributeOverride;
import javax.persistence.AttributeOverrides;
import javax.persistence.Column;
@@ -48,7 +49,7 @@ import org.joda.time.Duration;
* the completion time will remain null and the lock will have no effect. The same applies for
* unlock actions.
*
* <p>Note that there will be at most one row per domain with a null copmleted time -- this means
* <p>Note that there will be at most one row per domain with a null completed time -- this means
* that there is at most one pending action per domain. This is enforced at the logic level.
*
* <p>Note as well that in the case of a retry of a write after an unexpected success, the unique
@@ -56,12 +57,12 @@ import org.joda.time.Duration;
*/
@Entity
@Table(
/**
/*
* Unique constraint to get around Hibernate's failure to handle auto-increment field in
* composite primary key.
*
* <p>Note: indexes use the camelCase version of the field names because the {@link
* google.registry.persistence.NomulusNamingStrategy} does not translate the field name into the
* Note: indexes use the camelCase version of the field names because
* google.registry.persistence.NomulusNamingStrategy does not translate the field name into the
* snake_case column name until the write itself.
*/
indexes = {
@@ -72,7 +73,11 @@ import org.joda.time.Duration;
@Index(name = "idx_registry_lock_verification_code", columnList = "verificationCode"),
@Index(name = "idx_registry_lock_registrar_id", columnList = "registrarId")
})
public final class RegistryLock extends ImmutableObject implements Buildable {
@Access(AccessType.FIELD)
@AttributeOverride(
name = "updateTimestamp.lastUpdateTime",
column = @Column(nullable = false, name = "lastUpdateTime"))
public final class RegistryLock extends UpdateAutoTimestampEntity implements Buildable {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@@ -103,7 +108,7 @@ public final class RegistryLock extends ImmutableObject implements Buildable {
name = "creationTime",
column = @Column(name = "lockRequestTime", nullable = false))
})
private CreateAutoTimestamp lockRequestTime = CreateAutoTimestamp.create(null);
private final CreateAutoTimestamp lockRequestTime = CreateAutoTimestamp.create(null);
/** When the unlock is first requested. */
private DateTime unlockRequestTime;
@@ -140,9 +145,6 @@ public final class RegistryLock extends ImmutableObject implements Buildable {
/** The duration after which we will re-lock this domain after it is unlocked. */
private Duration relockDuration;
/** Time that this entity was last updated. */
private UpdateAutoTimestamp lastUpdateTime = UpdateAutoTimestamp.create(null);
public String getRepoId() {
return repoId;
}
@@ -189,7 +191,7 @@ public final class RegistryLock extends ImmutableObject implements Buildable {
}
public DateTime getLastUpdateTime() {
return lastUpdateTime.getTimestamp();
return getUpdateTimestamp().getTimestamp();
}
public Long getRevisionId() {
@@ -199,7 +201,7 @@ public final class RegistryLock extends ImmutableObject implements Buildable {
/**
* The lock that undoes this lock, if this lock has been unlocked and the domain locked again.
*
* <p>Note: this is lazily loaded, so it may not be initialized if referenced outside of the
* <p>Note: this is lazily loaded, so it may not be initialized if referenced outside the
* transaction in which this lock is loaded.
*/
public RegistryLock getRelock() {
@@ -17,7 +17,7 @@ package google.registry.model.domain.secdns;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import google.registry.model.ImmutableObject;
import google.registry.model.domain.secdns.DelegationSignerData.DomainDsDataId;
import google.registry.model.domain.secdns.DomainDsData.DomainDsDataId;
import java.io.Serializable;
import javax.persistence.Access;
import javax.persistence.AccessType;
@@ -34,15 +34,14 @@ import javax.xml.bind.annotation.XmlType;
*
* @see <a href="http://tools.ietf.org/html/rfc5910">RFC 5910</a>
* @see <a href="http://tools.ietf.org/html/rfc4034">RFC 4034</a>
* <p>TODO(b/177567432): Rename this class to DomainDsData.
*/
@XmlType(name = "dsData")
@Entity
@IdClass(DomainDsDataId.class)
@Table(indexes = @Index(columnList = "domainRepoId"))
public class DelegationSignerData extends DomainDsDataBase {
@Table(name = "DelegationSignerData", indexes = @Index(columnList = "domainRepoId"))
public class DomainDsData extends DomainDsDataBase {
private DelegationSignerData() {}
private DomainDsData() {}
@Override
@Id
@@ -79,21 +78,21 @@ public class DelegationSignerData extends DomainDsDataBase {
return super.getDigest();
}
public DelegationSignerData cloneWithDomainRepoId(String domainRepoId) {
DelegationSignerData clone = clone(this);
public DomainDsData cloneWithDomainRepoId(String domainRepoId) {
DomainDsData clone = clone(this);
clone.domainRepoId = checkArgumentNotNull(domainRepoId);
return clone;
}
public DelegationSignerData cloneWithoutDomainRepoId() {
DelegationSignerData clone = clone(this);
public DomainDsData cloneWithoutDomainRepoId() {
DomainDsData clone = clone(this);
clone.domainRepoId = null;
return clone;
}
public static DelegationSignerData create(
public static DomainDsData create(
int keyTag, int algorithm, int digestType, byte[] digest, String domainRepoId) {
DelegationSignerData instance = new DelegationSignerData();
DomainDsData instance = new DomainDsData();
instance.keyTag = keyTag;
instance.algorithm = algorithm;
instance.digestType = digestType;
@@ -102,17 +101,15 @@ public class DelegationSignerData extends DomainDsDataBase {
return instance;
}
public static DelegationSignerData create(
int keyTag, int algorithm, int digestType, byte[] digest) {
public static DomainDsData create(int keyTag, int algorithm, int digestType, byte[] digest) {
return create(keyTag, algorithm, digestType, digest, null);
}
public static DelegationSignerData create(
int keyTag, int algorithm, int digestType, String digestAsHex) {
public static DomainDsData create(int keyTag, int algorithm, int digestType, String digestAsHex) {
return create(keyTag, algorithm, digestType, DatatypeConverter.parseHexBinary(digestAsHex));
}
public static DelegationSignerData create(DomainDsDataHistory history) {
public static DomainDsData create(DomainDsDataHistory history) {
return create(
history.keyTag,
history.algorithm,
@@ -121,7 +118,7 @@ public class DelegationSignerData extends DomainDsDataBase {
history.domainRepoId);
}
/** Class to represent the composite primary key of {@link DelegationSignerData} entity. */
/** Class to represent the composite primary key of {@link DomainDsData} entity. */
static class DomainDsDataId extends ImmutableObject implements Serializable {
String domainRepoId;
@@ -26,7 +26,7 @@ import javax.xml.bind.annotation.XmlTransient;
import javax.xml.bind.annotation.adapters.HexBinaryAdapter;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
/** Base class for {@link DelegationSignerData} and {@link DomainDsDataHistory}. */
/** Base class for {@link DomainDsData} and {@link DomainDsDataHistory}. */
@MappedSuperclass
@Access(AccessType.FIELD)
public abstract class DomainDsDataBase extends ImmutableObject implements UnsafeSerializable {
@@ -25,7 +25,7 @@ import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
/** Entity class to represent a historic {@link DelegationSignerData}. */
/** Entity class to represent a historic {@link DomainDsData}. */
@Entity
public class DomainDsDataHistory extends DomainDsDataBase implements UnsafeSerializable {
@@ -39,10 +39,9 @@ public class DomainDsDataHistory extends DomainDsDataBase implements UnsafeSeria
/**
* Creates a {@link DomainDsDataHistory} instance from given {@link #domainHistoryRevisionId} and
* {@link DelegationSignerData} instance.
* {@link DomainDsData} instance.
*/
public static DomainDsDataHistory createFrom(
long domainHistoryRevisionId, DelegationSignerData dsData) {
public static DomainDsDataHistory createFrom(long domainHistoryRevisionId, DomainDsData dsData) {
DomainDsDataHistory instance = new DomainDsDataHistory();
instance.domainHistoryRevisionId = domainHistoryRevisionId;
instance.domainRepoId = dsData.domainRepoId;
@@ -36,13 +36,13 @@ public class SecDnsCreateExtension extends ImmutableObject implements CommandExt
Long maxSigLife;
/** Signatures for this domain. */
Set<DelegationSignerData> dsData;
Set<DomainDsData> dsData;
public Long getMaxSigLife() {
return maxSigLife;
}
public ImmutableSet<DelegationSignerData> getDsData() {
public ImmutableSet<DomainDsData> getDsData() {
return nullSafeImmutableCopy(dsData);
}
}
@@ -24,9 +24,9 @@ import javax.xml.bind.annotation.XmlRootElement;
public class SecDnsInfoExtension extends ImmutableObject implements ResponseExtension {
/** Signatures for this domain. */
ImmutableSet<DelegationSignerData> dsData;
ImmutableSet<DomainDsData> dsData;
public static SecDnsInfoExtension create(ImmutableSet<DelegationSignerData> dsData) {
public static SecDnsInfoExtension create(ImmutableSet<DomainDsData> dsData) {
SecDnsInfoExtension instance = new SecDnsInfoExtension();
instance.dsData = dsData;
return instance;
@@ -70,9 +70,9 @@ public class SecDnsUpdateExtension extends ImmutableObject implements CommandExt
@XmlTransient
abstract static class AddRemoveBase extends ImmutableObject {
/** Delegations to add or remove. */
Set<DelegationSignerData> dsData;
Set<DomainDsData> dsData;
public ImmutableSet<DelegationSignerData> getDsData() {
public ImmutableSet<DomainDsData> getDsData() {
return nullToEmptyImmutableCopy(dsData);
}
}
@@ -33,15 +33,15 @@ import com.google.common.collect.ImmutableSortedMap;
import com.google.common.collect.Range;
import google.registry.flows.EppException;
import google.registry.flows.domain.DomainFlowUtils;
import google.registry.model.BackupGroupRoot;
import google.registry.model.Buildable;
import google.registry.model.CreateAutoTimestamp;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.model.billing.BillingEvent.RenewalPriceBehavior;
import google.registry.model.common.TimedTransitionProperty;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.DomainHistoryVKey;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import java.util.Optional;
import java.util.Set;
import javax.annotation.Nullable;
@@ -58,7 +58,7 @@ import org.joda.time.DateTime;
/** An entity representing an allocation token. */
@Entity
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@Table(
indexes = {
@Index(columnList = "token", name = "allocation_token_token_idx", unique = true),
@@ -66,7 +66,7 @@ import org.joda.time.DateTime;
@Index(columnList = "tokenType"),
@Index(columnList = "redemption_domain_repo_id")
})
public class AllocationToken extends BackupGroupRoot implements Buildable {
public class AllocationToken extends UpdateAutoTimestampEntity implements Buildable {
private static final long serialVersionUID = -3954475393220876903L;
private static final String REMOVE_PACKAGE = "__REMOVEPACKAGE__";
@@ -109,13 +109,15 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
ANCHOR_TENANT
}
/**
* Single-use tokens are invalid after use. Infinite-use tokens, predictably, are not. Package
* tokens are used in package promotions.
*/
/** Type of the token that indicates how and where it should be used. */
public enum TokenType {
/** Token saved on a TLD to use if no other token is passed from the client */
DEFAULT_PROMO,
/** Token used for package pricing */
PACKAGE,
/** Invalid after use */
SINGLE_USE,
/** Do not expire after use */
UNLIMITED_USE,
}
@@ -300,6 +302,9 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
!getInstance().tokenType.equals(TokenType.PACKAGE)
|| getInstance().renewalPriceBehavior.equals(RenewalPriceBehavior.SPECIFIED),
"Package tokens must have renewalPriceBehavior set to SPECIFIED");
checkArgument(
!getInstance().tokenType.equals(TokenType.PACKAGE) || !getInstance().discountPremiums,
"Package tokens cannot discount premium names");
checkArgument(
getInstance().domainName == null || TokenType.SINGLE_USE.equals(getInstance().tokenType),
"Domain name can only be specified for SINGLE_USE tokens");
@@ -15,6 +15,7 @@
package google.registry.model.domain.token;
import static com.google.common.base.Preconditions.checkArgument;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
@@ -28,6 +29,8 @@ import java.util.Optional;
import javax.annotation.Nullable;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import org.hibernate.annotations.Columns;
import org.hibernate.annotations.Type;
@@ -40,7 +43,9 @@ import org.joda.time.DateTime;
public class PackagePromotion extends ImmutableObject implements Buildable {
/** An autogenerated identifier for the package promotion. */
@Id long packagePromotionId;
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
long packagePromotionId;
/** The allocation token string for the package. */
@Column(nullable = false)
@@ -94,6 +99,21 @@ public class PackagePromotion extends ImmutableObject implements Buildable {
return Optional.ofNullable(lastNotificationSent);
}
/** Loads and returns a PackagePromotion entity by its token string directly from Cloud SQL. */
public static Optional<PackagePromotion> loadByTokenString(String tokenString) {
jpaTm().assertInTransaction();
return jpaTm()
.query("FROM PackagePromotion WHERE token = :token", PackagePromotion.class)
.setParameter("token", VKey.createSql(AllocationToken.class, tokenString))
.getResultStream()
.findFirst();
}
@Override
public VKey<PackagePromotion> createVKey() {
return VKey.createSql(PackagePromotion.class, packagePromotionId);
}
@Override
public Builder asBuilder() {
return new Builder(clone(this));
@@ -143,7 +163,7 @@ public class PackagePromotion extends ImmutableObject implements Buildable {
return this;
}
public Builder setNextBillingDate(@Nullable DateTime nextBillingDate) {
public Builder setNextBillingDate(DateTime nextBillingDate) {
checkArgumentNotNull(nextBillingDate, "Next billing date must not be null");
getInstance().nextBillingDate = nextBillingDate;
return this;
@@ -19,7 +19,7 @@ import google.registry.model.EppResource.ForeignKeyedEppResource;
import google.registry.model.annotations.ExternalMessagingName;
import google.registry.model.annotations.ReportedOn;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import javax.persistence.Access;
import javax.persistence.AccessType;
@@ -51,7 +51,7 @@ import javax.persistence.AccessType;
@javax.persistence.Index(columnList = "currentSponsorRegistrarId")
})
@ExternalMessagingName("host")
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@Access(AccessType.FIELD) // otherwise it'll use the default if the repoId (property)
public class Host extends HostBase implements ForeignKeyedEppResource {
@@ -35,7 +35,6 @@ import java.util.Set;
import javax.annotation.Nullable;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Column;
import javax.persistence.Embeddable;
import javax.persistence.MappedSuperclass;
import org.joda.time.DateTime;
@@ -64,10 +63,7 @@ public class HostBase extends EppResource {
* from (creationTime, deletionTime) there can only be one host in Datastore with this name.
* However, there can be many hosts with the same name and non-overlapping lifetimes.
*/
// TODO(b/177567432): Rename this to hostName when we are off Datastore
@Index
@Column(name = "hostName")
String fullyQualifiedHostName;
@Index String hostName;
/** IP Addresses for this host. Can be null if this is an external host. */
@Index Set<InetAddress> inetAddresses;
@@ -95,7 +91,7 @@ public class HostBase extends EppResource {
DateTime lastSuperordinateChange;
public String getHostName() {
return fullyQualifiedHostName;
return hostName;
}
public VKey<Domain> getSuperordinateDomain() {
@@ -120,7 +116,7 @@ public class HostBase extends EppResource {
@Override
public String getForeignKey() {
return fullyQualifiedHostName;
return hostName;
}
@Override
@@ -197,7 +193,7 @@ public class HostBase extends EppResource {
hostName.equals(canonicalizeHostname(hostName)),
"Host name %s not in puny-coded, lower-case form",
hostName);
getInstance().fullyQualifiedHostName = hostName;
getInstance().hostName = hostName;
return thisCastToDerived();
}
@@ -36,7 +36,7 @@ public class HostCommand {
@XmlTransient
abstract static class HostCreateOrChange extends AbstractSingleResourceCommand
implements ResourceCreateOrChange<Host.Builder> {
public String getFullyQualifiedHostName() {
public String getHostName() {
return getTargetId();
}
}
@@ -61,7 +61,7 @@ public class HostHistory extends HistoryEntry implements UnsafeSerializable {
// Store HostBase instead of Host so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@DoNotCompare @Nullable HostBase hostBase;
@Nullable HostBase hostBase;
@Id
@Access(AccessType.PROPERTY)
@@ -28,23 +28,25 @@ import org.joda.time.DateTime;
/** The {@link ResponseData} returned for an EPP info flow on a host. */
@XmlRootElement(name = "infData")
@XmlType(propOrder = {
"fullyQualifiedHostName",
"repoId",
"statusValues",
"inetAddresses",
"currentSponsorClientId",
"creationClientId",
"creationTime",
"lastEppUpdateClientId",
"lastEppUpdateTime",
"lastTransferTime" })
@XmlType(
propOrder = {
"hostName",
"repoId",
"statusValues",
"inetAddresses",
"currentSponsorClientId",
"creationClientId",
"creationTime",
"lastEppUpdateClientId",
"lastEppUpdateTime",
"lastTransferTime"
})
@AutoValue
@CopyAnnotations
public abstract class HostInfoData implements ResponseData {
@XmlElement(name = "name")
abstract String getFullyQualifiedHostName();
abstract String getHostName();
@XmlElement(name = "roid")
abstract String getRepoId();
@@ -79,7 +81,8 @@ public abstract class HostInfoData implements ResponseData {
/** Builder for {@link HostInfoData}. */
@AutoValue.Builder
public abstract static class Builder {
public abstract Builder setFullyQualifiedHostName(String fullyQualifiedHostName);
public abstract Builder setHostName(String hostName);
public abstract Builder setRepoId(String repoId);
public abstract Builder setStatusValues(ImmutableSet<StatusValue> statusValues);
public abstract Builder setInetAddresses(ImmutableSet<InetAddress> inetAddresses);
@@ -56,11 +56,6 @@ public class ObjectifyService {
/** A singleton instance of our Ofy wrapper. */
private static final Ofy OFY = new Ofy(null);
/** Returns the singleton {@link Ofy} instance. */
public static Ofy ofy() {
return OFY;
}
/**
* Returns the singleton {@link Ofy} instance, signifying that the caller has been audited for the
* Registry 3.0 conversion.
@@ -16,7 +16,7 @@ package google.registry.model.ofy;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.Objectify;
import google.registry.model.BackupGroupRoot;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.model.annotations.DeleteAfterMigration;
import java.util.Arrays;
import java.util.Map;
@@ -34,7 +34,7 @@ class TimestampInversionException extends RuntimeException {
}
TimestampInversionException(
DateTime transactionTime, Map<Key<BackupGroupRoot>, DateTime> problematicRoots) {
DateTime transactionTime, Map<Key<UpdateAutoTimestampEntity>, DateTime> problematicRoots) {
this(transactionTime, "entities rooted under:\n" + problematicRoots);
}
@@ -91,10 +91,10 @@ public class PendingActionNotificationResponse extends ImmutableObject
}
public static DomainPendingActionNotificationResponse create(
String fullyQualifiedDomainName, boolean actionResult, Trid trid, DateTime processedDate) {
String domainName, boolean actionResult, Trid trid, DateTime processedDate) {
return init(
new DomainPendingActionNotificationResponse(),
fullyQualifiedDomainName,
domainName,
actionResult,
trid,
processedDate);
@@ -140,13 +140,9 @@ public class PendingActionNotificationResponse extends ImmutableObject
}
public static HostPendingActionNotificationResponse create(
String fullyQualifiedHostName, boolean actionResult, Trid trid, DateTime processedDate) {
String hostName, boolean actionResult, Trid trid, DateTime processedDate) {
return init(
new HostPendingActionNotificationResponse(),
fullyQualifiedHostName,
actionResult,
trid,
processedDate);
new HostPendingActionNotificationResponse(), hostName, actionResult, trid, processedDate);
}
}
}
@@ -46,7 +46,7 @@ import google.registry.model.transfer.TransferResponse;
import google.registry.model.transfer.TransferResponse.ContactTransferResponse;
import google.registry.model.transfer.TransferResponse.DomainTransferResponse;
import google.registry.persistence.VKey;
import google.registry.persistence.WithLongVKey;
import google.registry.persistence.WithVKey;
import google.registry.util.NullIgnoringCollectionBuilder;
import java.util.Optional;
import javax.persistence.AttributeOverride;
@@ -342,7 +342,7 @@ public abstract class PollMessage extends ImmutableObject
*/
@Entity
@DiscriminatorValue("ONE_TIME")
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class OneTime extends PollMessage {
@Embedded
@@ -386,7 +386,7 @@ public abstract class PollMessage extends ImmutableObject
TransferResponse transferResponse;
@Column(name = "transfer_response_domain_name")
String fullyQualifiedDomainName;
String domainName;
@Column(name = "transfer_response_domain_expiration_time")
DateTime extendedRegistrationExpirationTime;
@@ -427,7 +427,7 @@ public abstract class PollMessage extends ImmutableObject
pendingActionNotificationResponse.getActionResult(),
pendingActionNotificationResponse.getTrid(),
pendingActionNotificationResponse.processedDate);
} else if (fullyQualifiedDomainName != null) {
} else if (domainName != null) {
pendingActionNotificationResponse =
DomainPendingActionNotificationResponse.create(
pendingActionNotificationResponse.nameOrId.value,
@@ -457,10 +457,10 @@ public abstract class PollMessage extends ImmutableObject
.setPendingTransferExpirationTime(
transferResponse.getPendingTransferExpirationTime())
.build();
} else if (fullyQualifiedDomainName != null) {
} else if (domainName != null) {
transferResponse =
new DomainTransferResponse.Builder()
.setFullyQualifiedDomainName(fullyQualifiedDomainName)
.setDomainName(domainName)
.setGainingRegistrarId(transferResponse.getGainingRegistrarId())
.setLosingRegistrarId(transferResponse.getLosingRegistrarId())
.setTransferStatus(transferResponse.getTransferStatus())
@@ -486,7 +486,7 @@ public abstract class PollMessage extends ImmutableObject
OneTime instance = getInstance();
// Note: In its current form, the code will basically just ignore everything but the first
// PendingActionNotificationResponse and TransferResponse in responseData, and will override
// any identifier fields (e.g. contactId, fullyQualifiedDomainName) obtained from the
// any identifier fields (e.g. contactId, domainName) obtained from the
// PendingActionNotificationResponse if a TransferResponse is found with different values
// for those fields. It is not clear what the constraints should be on this data or
// whether we should enforce them here, though historically we have not, so the current
@@ -507,8 +507,7 @@ public abstract class PollMessage extends ImmutableObject
instance.contactId = instance.pendingActionNotificationResponse.nameOrId.value;
} else if (instance.pendingActionNotificationResponse
instanceof DomainPendingActionNotificationResponse) {
instance.fullyQualifiedDomainName =
instance.pendingActionNotificationResponse.nameOrId.value;
instance.domainName = instance.pendingActionNotificationResponse.nameOrId.value;
} else if (instance.pendingActionNotificationResponse
instanceof HostPendingActionNotificationResponse) {
instance.hostId = instance.pendingActionNotificationResponse.nameOrId.value;
@@ -527,7 +526,7 @@ public abstract class PollMessage extends ImmutableObject
instance.contactId = ((ContactTransferResponse) instance.transferResponse).getContactId();
} else if (instance.transferResponse instanceof DomainTransferResponse) {
DomainTransferResponse response = (DomainTransferResponse) instance.transferResponse;
instance.fullyQualifiedDomainName = response.getFullyQualifiedDomainName();
instance.domainName = response.getDomainName();
instance.extendedRegistrationExpirationTime =
response.getExtendedRegistrationExpirationTime();
}
@@ -545,7 +544,7 @@ public abstract class PollMessage extends ImmutableObject
*/
@Entity
@DiscriminatorValue("AUTORENEW")
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class Autorenew extends PollMessage {
/** The target id of the autorenew event. */
@@ -28,10 +28,10 @@ public interface PremiumPricingEngine {
/**
* Returns the prices for the given fully qualified domain name at the given time.
*
* <p>Note that the fullyQualifiedDomainName must only contain a single part left of the TLD, i.e.
* subdomains are not allowed, but multi-part TLDs are.
* <p>Note that the domainName must only contain a single part left of the TLD, i.e. subdomains
* are not allowed, but multi-part TLDs are.
*/
DomainPrices getDomainPrices(String fullyQualifiedDomainName, DateTime priceTime);
DomainPrices getDomainPrices(String domainName, DateTime priceTime);
/**
* A class containing information on premium prices for a specific domain name.
@@ -34,9 +34,9 @@ public final class StaticPremiumListPricingEngine implements PremiumPricingEngin
@Inject StaticPremiumListPricingEngine() {}
@Override
public DomainPrices getDomainPrices(String fullyQualifiedDomainName, DateTime priceTime) {
String tld = getTldFromDomainName(fullyQualifiedDomainName);
String label = InternetDomainName.from(fullyQualifiedDomainName).parts().get(0);
public DomainPrices getDomainPrices(String domainName, DateTime priceTime) {
String tld = getTldFromDomainName(domainName);
String label = InternetDomainName.from(domainName).parts().get(0);
Registry registry = Registry.get(checkNotNull(tld, "tld"));
Optional<Money> premiumPrice =
registry.getPremiumListName().flatMap(pl -> PremiumListDao.getPremiumPrice(pl, label));
@@ -19,8 +19,8 @@ import static google.registry.model.rde.RdeNamingUtils.makePartialName;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.base.VerifyException;
import google.registry.model.BackupGroupRoot;
import google.registry.model.ImmutableObject;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.model.rde.RdeRevision.RdeRevisionId;
import google.registry.persistence.VKey;
import google.registry.persistence.converter.LocalDateConverter;
@@ -45,7 +45,7 @@ import org.joda.time.LocalDate;
*/
@Entity
@IdClass(RdeRevisionId.class)
public final class RdeRevision extends BackupGroupRoot {
public final class RdeRevision extends UpdateAutoTimestampEntity {
@Id String tld;
@@ -54,11 +54,10 @@ import com.google.common.collect.Streams;
import com.google.re2j.Pattern;
import google.registry.model.Buildable;
import google.registry.model.CreateAutoTimestamp;
import google.registry.model.ImmutableObject;
import google.registry.model.JsonMapBuilder;
import google.registry.model.Jsonifiable;
import google.registry.model.UnsafeSerializable;
import google.registry.model.UpdateAutoTimestamp;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.model.tld.Registry;
import google.registry.model.tld.Registry.TldType;
import google.registry.persistence.VKey;
@@ -95,8 +94,10 @@ import org.joda.time.DateTime;
@Index(columnList = "registrarName", name = "registrar_name_idx"),
@Index(columnList = "ianaIdentifier", name = "registrar_iana_identifier_idx"),
})
public class Registrar extends ImmutableObject
implements Buildable, Jsonifiable, UnsafeSerializable {
@AttributeOverride(
name = "updateTimestamp.lastUpdateTime",
column = @Column(nullable = false, name = "lastUpdateTime"))
public class Registrar extends UpdateAutoTimestampEntity implements Buildable, Jsonifiable {
/** Represents the type of a registrar entity. */
public enum Type {
@@ -207,11 +208,10 @@ public class Registrar extends ImmutableObject
* Unique registrar client id. Must conform to "clIDType" as defined in RFC5730.
*
* @see <a href="http://tools.ietf.org/html/rfc5730#section-4.2">Shared Structure Schema</a>
* <p>TODO(b/177567432): Rename this field to registrarId.
*/
@Id
@Column(name = "registrarId", nullable = false)
String clientIdentifier;
@Column(nullable = false)
String registrarId;
/**
* Registrar name. This is a distinct from the client identifier since there are no restrictions
@@ -270,9 +270,7 @@ public class Registrar extends ImmutableObject
String failoverClientCertificateHash;
/** An allow list of netmasks (in CIDR notation) which the client is allowed to connect from. */
// TODO(b/177567432): Rename to ipAddressAllowList once Cloud SQL migration is complete.
@Column(name = "ip_address_allow_list")
List<CidrAddressBlock> ipAddressWhitelist;
List<CidrAddressBlock> ipAddressAllowList;
/** A hashed password for EPP access. The hash is a base64 encoded SHA256 string. */
String passwordHash;
@@ -381,9 +379,6 @@ public class Registrar extends ImmutableObject
/** The time when this registrar was created. */
CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);
/** An automatically managed last-saved timestamp. */
UpdateAutoTimestamp lastUpdateTime = UpdateAutoTimestamp.create(null);
/** The time that the certificate was last updated. */
DateTime lastCertificateUpdateTime;
@@ -408,7 +403,7 @@ public class Registrar extends ImmutableObject
boolean registryLockAllowed = false;
public String getRegistrarId() {
return clientIdentifier;
return registrarId;
}
public DateTime getCreationTime() {
@@ -431,7 +426,7 @@ public class Registrar extends ImmutableObject
}
public DateTime getLastUpdateTime() {
return lastUpdateTime.getTimestamp();
return getUpdateTimestamp().getTimestamp();
}
public DateTime getLastCertificateUpdateTime() {
@@ -498,7 +493,7 @@ public class Registrar extends ImmutableObject
}
public ImmutableList<CidrAddressBlock> getIpAddressAllowList() {
return nullToEmptyImmutableCopy(ipAddressWhitelist);
return nullToEmptyImmutableCopy(ipAddressAllowList);
}
public RegistrarAddress getLocalizedAddress() {
@@ -587,7 +582,7 @@ public class Registrar extends ImmutableObject
() ->
jpaTm()
.query("FROM RegistrarPoc WHERE registrarId = :registrarId", RegistrarPoc.class)
.setParameter("registrarId", clientIdentifier)
.setParameter("registrarId", registrarId)
.getResultStream()
.collect(toImmutableList()));
}
@@ -595,10 +590,10 @@ public class Registrar extends ImmutableObject
@Override
public Map<String, Object> toJsonMap() {
return new JsonMapBuilder()
.put("clientIdentifier", clientIdentifier)
.put("registrarId", registrarId)
.put("ianaIdentifier", ianaIdentifier)
.putString("creationTime", creationTime.getTimestamp())
.putString("lastUpdateTime", lastUpdateTime.getTimestamp())
.putString("lastUpdateTime", getUpdateTimestamp().getTimestamp())
.putString("lastCertificateUpdateTime", lastCertificateUpdateTime)
.putString("lastExpiringCertNotificationSentDate", lastExpiringCertNotificationSentDate)
.putString(
@@ -655,7 +650,7 @@ public class Registrar extends ImmutableObject
/** Creates a {@link VKey} for this instance. */
@Override
public VKey<Registrar> createVKey() {
return createVKey(clientIdentifier);
return createVKey(registrarId);
}
/** Creates a {@link VKey} for the given {@code registrarId}. */
@@ -678,7 +673,7 @@ public class Registrar extends ImmutableObject
checkArgument(
Range.closed(3, 16).contains(registrarId.length()),
"Registrar ID must be 3-16 characters long.");
getInstance().clientIdentifier = registrarId;
getInstance().registrarId = registrarId;
return this;
}
@@ -796,7 +791,7 @@ public class Registrar extends ImmutableObject
}
public Builder setIpAddressAllowList(Iterable<CidrAddressBlock> ipAddressAllowList) {
getInstance().ipAddressWhitelist = ImmutableList.copyOf(ipAddressAllowList);
getInstance().ipAddressAllowList = ImmutableList.copyOf(ipAddressAllowList);
return this;
}
@@ -889,7 +884,7 @@ public class Registrar extends ImmutableObject
*/
@VisibleForTesting
public Builder setLastUpdateTime(DateTime timestamp) {
getInstance().lastUpdateTime = UpdateAutoTimestamp.create(timestamp);
getInstance().setUpdateTimestamp(UpdateAutoTimestamp.create(timestamp));
return this;
}
@@ -210,9 +210,6 @@ public class Lock extends ImmutableObject implements Serializable {
RequestStatusChecker requestStatusChecker,
boolean checkThreadRunning) {
String scope = tld != null ? tld : GLOBAL;
// It's important to use transactNew rather than transact, because a Lock can be used to control
// access to resources like GCS that can't be transactionally rolled back. Therefore, the lock
// must be definitively acquired before it is used, even when called inside another transaction.
Supplier<AcquireResult> lockAcquirer =
() -> {
DateTime now = jpaTm().getTransactionTime();
@@ -22,22 +22,21 @@ import static com.google.common.base.Strings.emptyToNull;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.Maps.filterValues;
import static google.registry.model.CacheUtils.memoizeWithShortExpiration;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.CollectionUtils.entriesToImmutableMap;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import com.google.common.base.Joiner;
import com.google.common.base.Supplier;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.common.collect.Streams;
import com.google.common.net.InternetDomainName;
import com.googlecode.objectify.Key;
import google.registry.model.tld.Registry.TldType;
import google.registry.util.DomainNameUtils;
import java.util.Optional;
import java.util.function.Supplier;
import java.util.stream.Stream;
import javax.persistence.EntityManager;
@@ -58,32 +57,17 @@ public final class Registries {
private static Supplier<ImmutableMap<String, TldType>> createFreshCache() {
return memoizeWithShortExpiration(
() ->
tm().doTransactionless(
tm().transact(
() -> {
if (tm().isOfy()) {
ImmutableSet<String> tlds =
auditedOfy()
.load()
.type(Registry.class)
.keys()
.list()
.stream()
.map(Key::getName)
.collect(toImmutableSet());
return Registry.get(tlds).stream()
.map(e -> Maps.immutableEntry(e.getTldStr(), e.getTldType()))
.collect(entriesToImmutableMap());
} else {
EntityManager entityManager = jpaTm().getEntityManager();
Stream<?> resultStream =
entityManager
.createQuery("SELECT tldStr, tldType FROM Tld")
.getResultStream();
return resultStream
.map(e -> ((Object[]) e))
.map(e -> Maps.immutableEntry((String) e[0], ((TldType) e[1])))
.collect(entriesToImmutableMap());
}
EntityManager entityManager = jpaTm().getEntityManager();
Stream<?> resultStream =
entityManager
.createQuery("SELECT tldStr, tldType FROM Tld")
.getResultStream();
return resultStream
.map(e -> ((Object[]) e))
.map(e -> Maps.immutableEntry((String) e[0], ((TldType) e[1])))
.collect(entriesToImmutableMap());
}));
}
@@ -143,8 +127,7 @@ public final class Registries {
*
* <p><b>Note:</b> This routine will only work on names under TLDs for which this registry is
* authoritative. To extract TLDs from domains (not hosts) that other registries control, use
* {@link google.registry.util.DomainNameUtils#getTldFromDomainName(String)
* DomainNameUtils#getTldFromDomainName}.
* {@link DomainNameUtils#getTldFromDomainName(String) DomainNameUtils#getTldFromDomainName}.
*
* @param domainName domain name or host name (but not TLD) under an authoritative TLD
* @return TLD or absent if {@code domainName} has no labels under an authoritative TLD
@@ -72,11 +72,9 @@ public class ClaimsList extends ImmutableObject {
* <p>Note that the value of this field is parsed from the claims list file(See this <a
* href="https://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.1">RFC</>), it is
* the DNL List creation datetime from the rfc.
*
* <p>TODO(b/177567432): Rename this field to tmdbGenerationTime.
*/
@Column(name = "tmdb_generation_time", nullable = false)
DateTime creationTime;
@Column(nullable = false)
DateTime tmdbGenerationTime;
/**
* A map from labels to claims keys.
@@ -143,7 +141,7 @@ public class ClaimsList extends ImmutableObject {
* creation datetime</a>
*/
public DateTime getTmdbGenerationTime() {
return creationTime;
return tmdbGenerationTime;
}
/** Returns the creation time of this claims list. */
@@ -225,7 +223,7 @@ public class ClaimsList extends ImmutableObject {
public static ClaimsList create(
DateTime tmdbGenerationTime, ImmutableMap<String, String> labelsToKeys) {
ClaimsList instance = new ClaimsList();
instance.creationTime = checkNotNull(tmdbGenerationTime);
instance.tmdbGenerationTime = checkNotNull(tmdbGenerationTime);
instance.labelsToKeys = checkNotNull(labelsToKeys);
return instance;
}
@@ -33,22 +33,24 @@ public class TransferResponse extends BaseTransferObject implements ResponseData
/** An adapter to output the XML in response to a transfer command on a domain. */
@XmlRootElement(name = "trnData", namespace = "urn:ietf:params:xml:ns:domain-1.0")
@XmlType(propOrder = {
"fullyQualifiedDomainName",
"transferStatus",
"gainingClientId",
"transferRequestTime",
"losingClientId",
"pendingTransferExpirationTime",
"extendedRegistrationExpirationTime"},
namespace = "urn:ietf:params:xml:ns:domain-1.0")
@XmlType(
propOrder = {
"domainName",
"transferStatus",
"gainingClientId",
"transferRequestTime",
"losingClientId",
"pendingTransferExpirationTime",
"extendedRegistrationExpirationTime"
},
namespace = "urn:ietf:params:xml:ns:domain-1.0")
public static class DomainTransferResponse extends TransferResponse {
@XmlElement(name = "name")
String fullyQualifiedDomainName;
String domainName;
public String getFullyQualifiedDomainName() {
return fullyQualifiedDomainName;
public String getDomainName() {
return domainName;
}
/**
@@ -65,8 +67,8 @@ public class TransferResponse extends BaseTransferObject implements ResponseData
/** Builder for {@link DomainTransferResponse}. */
public static class Builder
extends BaseTransferObject.Builder<DomainTransferResponse, Builder> {
public Builder setFullyQualifiedDomainName(String fullyQualifiedDomainName) {
getInstance().fullyQualifiedDomainName = fullyQualifiedDomainName;
public Builder setDomainName(String domainName) {
getInstance().domainName = domainName;
return this;
}
@@ -1,65 +0,0 @@
// Copyright 2021 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Streams;
import google.registry.model.bulkquery.BulkQueryEntities;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.persistence.transaction.JpaTransactionManagerImpl;
import google.registry.util.Clock;
import java.util.List;
import javax.persistence.EntityManagerFactory;
import org.hibernate.jpa.boot.internal.ParsedPersistenceXmlDescriptor;
import org.hibernate.jpa.boot.spi.Bootstrap;
/**
* Defines factory method for instantiating the bulk-query optimized {@link JpaTransactionManager}.
*/
public final class BulkQueryJpaFactory {
private BulkQueryJpaFactory() {}
static EntityManagerFactory createBulkQueryEntityManagerFactory(
ImmutableMap<String, String> cloudSqlConfigs) {
ParsedPersistenceXmlDescriptor descriptor =
PersistenceXmlUtility.getParsedPersistenceXmlDescriptor();
List<String> updatedManagedClasses =
Streams.concat(
descriptor.getManagedClassNames().stream(),
BulkQueryEntities.JPA_ENTITIES_NEW.stream())
.map(
name -> {
if (BulkQueryEntities.JPA_ENTITIES_REPLACEMENTS.containsKey(name)) {
return BulkQueryEntities.JPA_ENTITIES_REPLACEMENTS.get(name);
}
return name;
})
.collect(ImmutableList.toImmutableList());
descriptor.getManagedClassNames().clear();
descriptor.getManagedClassNames().addAll(updatedManagedClasses);
return Bootstrap.getEntityManagerFactoryBuilder(descriptor, cloudSqlConfigs).build();
}
public static JpaTransactionManager createBulkQueryJpaTransactionManager(
ImmutableMap<String, String> cloudSqlConfigs, Clock clock) {
return new JpaTransactionManagerImpl(
createBulkQueryEntityManagerFactory(cloudSqlConfigs), clock);
}
}
@@ -14,12 +14,10 @@
package google.registry.persistence;
import com.google.common.base.Predicates;
import com.google.common.collect.ImmutableSet;
import java.sql.SQLException;
import java.util.function.Predicate;
import javax.persistence.OptimisticLockException;
import org.hibernate.exception.JDBCConnectionException;
/** Helpers for identifying retriable database operations. */
public final class JpaRetries {
@@ -35,18 +33,11 @@ public final class JpaRetries {
);
private static final Predicate<Throwable> RETRIABLE_TXN_PREDICATE =
Predicates.or(
OptimisticLockException.class::isInstance,
e ->
e instanceof SQLException
&& RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
private static final Predicate<Throwable> RETRIABLE_QUERY_PREDICATE =
Predicates.or(
JDBCConnectionException.class::isInstance,
e ->
e instanceof SQLException
&& RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
((Predicate<Throwable>) OptimisticLockException.class::isInstance)
.or(
e ->
e instanceof SQLException
&& RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
public static boolean isFailedTxnRetriable(Throwable throwable) {
Throwable t = throwable;
@@ -58,16 +49,4 @@ public final class JpaRetries {
}
return false;
}
public static boolean isFailedQueryRetriable(Throwable throwable) {
// TODO(weiminyu): check for more error codes.
Throwable t = throwable;
while (t != null) {
if (RETRIABLE_QUERY_PREDICATE.test(t)) {
return true;
}
t = t.getCause();
}
return false;
}
}
@@ -241,15 +241,6 @@ public abstract class PersistenceModule {
return new JpaTransactionManagerImpl(create(beamCloudSqlConfigs), clock);
}
@Provides
@Singleton
@BeamBulkQueryJpaTm
static JpaTransactionManager provideBeamBulkQueryJpaTm(
@BeamPipelineCloudSqlConfigs ImmutableMap<String, String> beamCloudSqlConfigs, Clock clock) {
return new JpaTransactionManagerImpl(
BulkQueryJpaFactory.createBulkQueryEntityManagerFactory(beamCloudSqlConfigs), clock);
}
@Provides
@Singleton
@NomulusToolJpaTm
@@ -278,7 +269,7 @@ public abstract class PersistenceModule {
replicaInstanceConnectionName.ifPresent(
name -> overrides.put(HIKARI_DS_CLOUD_SQL_INSTANCE, name));
overrides.put(
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_READ_COMMITTED.name());
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_REPEATABLE_READ.name());
return new JpaTransactionManagerImpl(create(overrides), clock);
}
@@ -294,7 +285,7 @@ public abstract class PersistenceModule {
replicaInstanceConnectionName.ifPresent(
name -> overrides.put(HIKARI_DS_CLOUD_SQL_INSTANCE, name));
overrides.put(
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_READ_COMMITTED.name());
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_REPEATABLE_READ.name());
return new JpaTransactionManagerImpl(create(overrides), clock);
}
@@ -371,11 +362,6 @@ public abstract class PersistenceModule {
public enum JpaTransactionManagerType {
/** The regular {@link JpaTransactionManager} for general use. */
REGULAR,
/**
* The {@link JpaTransactionManager} optimized for bulk loading multi-level JPA entities. Please
* see {@link google.registry.model.bulkquery.BulkQueryEntities} for more information.
*/
BULK_QUERY,
/**
* The {@link JpaTransactionManager} that uses the read-only Postgres replica if configured, or
* the standard DB if not.
@@ -398,14 +384,6 @@ public abstract class PersistenceModule {
@Documented
public @interface BeamJpaTm {}
/**
* Dagger qualifier for {@link JpaTransactionManager} that uses an alternative entity model for
* faster bulk queries.
*/
@Qualifier
@Documented
public @interface BeamBulkQueryJpaTm {}
/**
* Dagger qualifier for {@link JpaTransactionManager} used inside BEAM pipelines that uses the
* read-only Postgres replica if one is configured (otherwise it uses the standard DB).
@@ -16,14 +16,13 @@ package google.registry.persistence;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkState;
import static google.registry.model.ImmutableObject.Insignificant;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableMap;
import com.googlecode.objectify.Key;
import google.registry.model.BackupGroupRoot;
import google.registry.model.ImmutableObject;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.model.common.ClassPathManager;
import google.registry.model.translators.VKeyTranslatorFactory;
import google.registry.util.SerializeUtils;
@@ -104,9 +103,9 @@ public class VKey<T> extends ImmutableObject implements Serializable {
*/
public static <T> VKey<T> create(Class<T> kind, long id) {
checkArgument(
BackupGroupRoot.class.isAssignableFrom(kind),
"The kind %s is not a BackupGroupRoot and thus needs its entire entity group chain"
+ " specified in a parent",
UpdateAutoTimestampEntity.class.isAssignableFrom(kind),
"The kind %s is not a UpdateAutoTimestampEntity and thus needs its entire entity group"
+ " chain specified in a parent",
kind.getCanonicalName());
return new VKey<T>(kind, Key.create(kind, id), id);
}
@@ -122,9 +121,9 @@ public class VKey<T> extends ImmutableObject implements Serializable {
*/
public static <T> VKey<T> create(Class<T> kind, String name) {
checkArgument(
BackupGroupRoot.class.isAssignableFrom(kind),
"The kind %s is not a BackupGroupRoot and thus needs its entire entity group chain"
+ " specified in a parent",
UpdateAutoTimestampEntity.class.isAssignableFrom(kind),
"The kind %s is not a UpdateAutoTimestampEntity and thus needs its entire entity group"
+ " chain specified in a parent",
kind.getCanonicalName());
return new VKey<T>(kind, Key.create(kind, name), name);
}
@@ -1,42 +0,0 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;
import javax.persistence.AttributeConverter;
import javax.persistence.Entity;
/**
* Annotation for {@link Entity} which id is string type and needs an {@link AttributeConverter} for
* its VKey.
*/
@Target({ElementType.TYPE})
public @interface WithStringVKey {
/**
* Sets the suffix of the class name for the {@link AttributeConverter} generated by
* StringVKeyProcessor. If not set, the suffix will be the type name of the VKey. Note that the
* class name will be "VKeyConverter_" concatenated with the suffix.
*/
String classNameSuffix() default "";
/**
* Set to true if this is a composite vkey.
*
* <p>For composite VKeys, we don't attempt to define an objectify key when loading from SQL: the
* enclosing class has to take care of that.
*/
boolean compositeKey() default false;
}
@@ -1,4 +1,4 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@@ -14,29 +14,22 @@
package google.registry.persistence;
import java.io.Serializable;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;
import javax.persistence.AttributeConverter;
import javax.persistence.Entity;
/**
* Annotation for {@link Entity} which id is long type and needs an {@link AttributeConverter} for
* its VKey.
* Annotation for {@link Entity} that can be saved as a foreign key in the form of a {@link VKey} in
* another table.
*
* <p>A {@link AttributeConverter} named {@code VKeyConverter_[EntityClassSimpleName]} will be
* automatically generated by {@code google.registry.processors.VKeyProcessor}, this class must be
* manually added to {@code persistence.xml} in order for it to be picked up by Hibernate.
*/
@Target({ElementType.TYPE})
public @interface WithLongVKey {
/**
* Sets the suffix of the class name for the {@link AttributeConverter} generated by
* LongVKeyProcessor. If not set, the suffix will be the type name of the VKey. Note that the
* class name will be "VKeyConverter_" concatenated with the suffix.
*/
String classNameSuffix() default "";
/**
* Set to true if this is a composite vkey.
*
* <p>For composite VKeys, we don't attempt to define an objectify key when loading from SQL: the
* enclosing class has to take care of that.
*/
boolean compositeKey() default false;
public @interface WithVKey {
/** The type of the SQL primary ID of the entity that is saved in the {@link VKey} */
Class<? extends Serializable> value();
}
@@ -14,20 +14,34 @@
package google.registry.persistence.converter;
import com.googlecode.objectify.Key;
import google.registry.persistence.VKey;
import java.io.Serializable;
import javax.annotation.Nullable;
import javax.persistence.AttributeConverter;
/** Converts VKey to a string or long column. */
/**
* Converts {@link VKey} to/from a type that can be directly stored in the database.
*
* <p>Typically the converted type is {@link String} or {@link Long}.
*/
public abstract class VKeyConverter<T, C extends Serializable>
implements AttributeConverter<VKey<? extends T>, C> {
@Override
@Nullable
@SuppressWarnings("unchecked")
public C convertToDatabaseColumn(@Nullable VKey<? extends T> attribute) {
return attribute == null ? null : (C) attribute.getSqlKey();
if (attribute == null) {
return null;
}
try {
return getKeyClass().cast(attribute.getSqlKey());
} catch (ClassCastException e) {
throw new RuntimeException(
String.format(
"Cannot cast SQL key %s of type %s to type %s",
attribute.getSqlKey(), attribute.getSqlKey().getClass(), getKeyClass()),
e);
}
}
@Override
@@ -36,27 +50,12 @@ public abstract class VKeyConverter<T, C extends Serializable>
if (dbData == null) {
return null;
}
Class<T> clazz = getAttributeClass();
Key<T> ofyKey;
if (!hasCompositeOfyKey()) {
// If this isn't a composite key, we can create the Ofy key from the SQL key.
ofyKey =
dbData instanceof String
? Key.create(clazz, (String) dbData)
: Key.create(clazz, (Long) dbData);
return VKey.create(clazz, dbData, ofyKey);
} else {
// We don't know how to create the Ofy key and probably don't have everything necessary to do
// it anyway, so just create an asymmetric key - the containing object will have to convert it
// into a symmetric key.
return VKey.createSql(clazz, dbData);
}
return VKey.createSql(getEntityClass(), dbData);
}
protected boolean hasCompositeOfyKey() {
return false;
}
/** Returns the class of the entity that the VKey represents. */
protected abstract Class<T> getEntityClass();
/** Returns the class of the attribute. */
protected abstract Class<T> getAttributeClass();
/** Returns the class of the key that the VKey holds. */
protected abstract Class<C> getKeyClass();
}
@@ -61,7 +61,7 @@ public interface JpaTransactionManager extends TransactionManager {
*/
<T> TypedQuery<T> query(String sqlString, Class<T> resultClass);
/** Creates a JPA SQU query for the given criteria query. */
/** Creates a JPA SQL query for the given criteria query. */
<T> TypedQuery<T> criteriaQuery(CriteriaQuery<T> criteriaQuery);
/**
@@ -208,43 +208,6 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
});
}
// TODO(b/177674699): Remove all transactNew methods as they are same as transact after the
// database migration.
@Override
public <T> T transactNew(Supplier<T> work) {
return transact(work);
}
@Override
public void transactNew(Runnable work) {
transact(work);
}
// For now, read-only transactions and "transactNew" methods only create (or use existing)
// standard transactions. Attempting to use a read-only transaction can break larger transactions
// (if we were already in one) so we don't set read-only mode.
//
// TODO(gbrodman): If necessary, implement transactNew and readOnly transactions using Postgres
// savepoints, see https://www.postgresql.org/docs/8.1/sql-savepoint.html
@Override
public <T> T transactNewReadOnly(Supplier<T> work) {
return retrier.callWithRetry(() -> transact(work), JpaRetries::isFailedQueryRetriable);
}
@Override
public void transactNewReadOnly(Runnable work) {
transactNewReadOnly(
() -> {
work.run();
return null;
});
}
@Override
public <T> T doTransactionless(Supplier<T> work) {
return retrier.callWithRetry(() -> transact(work), JpaRetries::isFailedQueryRetriable);
}
@Override
public DateTime getTransactionTime() {
assertInTransaction();
@@ -493,16 +456,6 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
return new JpaQueryComposerImpl<>(entity);
}
@Override
public void clearSessionCache() {
// This is an intended no-op method as there is no session cache in Postgresql.
}
@Override
public boolean isOfy() {
return false;
}
@Override
public <T> void assertDelete(VKey<T> key) {
if (internalDelete(key) != 1) {
@@ -516,8 +469,8 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
}
private static class EntityId {
private String name;
private Object value;
private final String name;
private final Object value;
private EntityId(String name, Object value) {
this.name = name;
@@ -52,42 +52,6 @@ public interface TransactionManager {
/** Executes the work in a transaction. */
void transact(Runnable work);
/**
* Pauses the current transaction (if any), executes the work in a new transaction and returns the
* result.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
<T> T transactNew(Supplier<T> work);
/**
* Pauses the current transaction (if any) and executes the work in a new transaction.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
void transactNew(Runnable work);
/**
* Executes the work in a read-only transaction and returns the result.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
<R> R transactNewReadOnly(Supplier<R> work);
/**
* Executes the work in a read-only transaction.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
void transactNewReadOnly(Runnable work);
/** Executes the work in a transactionless context. */
<R> R doTransactionless(Supplier<R> work);
/** Returns the time associated with the start of this particular transaction attempt. */
DateTime getTransactionTime();
@@ -210,10 +174,4 @@ public interface TransactionManager {
/** Returns a QueryComposer which can be used to perform queries against the current database. */
<T> QueryComposer<T> createQueryComposer(Class<T> entity);
/** Clears the session cache if the underlying database is Datastore, otherwise it is a no-op. */
void clearSessionCache();
/** Returns true if the transaction manager is DatastoreTransactionManager, false otherwise. */
boolean isOfy();
}
@@ -109,7 +109,7 @@ public final class TransactionManagerFactory {
* however, this will be a reference to the read-only replica database if one is configured.
*/
public static TransactionManager replicaTm() {
return tm().isOfy() ? tm() : replicaJpaTm();
return replicaJpaTm();
}
/** Sets the return of {@link #jpaTm()} to the given instance of {@link JpaTransactionManager}. */
@@ -118,7 +118,7 @@ public final class TransactionManagerFactory {
checkState(
RegistryEnvironment.get().equals(RegistryEnvironment.UNITTEST)
|| RegistryToolEnvironment.get() != null,
"setJpamTm() should only be called by tools and tests.");
"setJpaTm() should only be called by tools and tests.");
jpaTm = Suppliers.memoize(jpaTmSupplier::get);
}
@@ -19,7 +19,7 @@ import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretManagerServiceSettings;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule.DefaultCredential;
import google.registry.config.CredentialModule.ApplicationDefaultCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.util.GoogleCredentialsBundle;
import google.registry.util.Retrier;
@@ -33,7 +33,7 @@ public abstract class SecretManagerModule {
@Provides
@Singleton
static SecretManagerServiceSettings provideSecretManagerSetting(
@DefaultCredential GoogleCredentialsBundle credentialsBundle) {
@ApplicationDefaultCredential GoogleCredentialsBundle credentialsBundle) {
try {
return SecretManagerServiceSettings.newBuilder()
.setCredentialsProvider(() -> credentialsBundle.getGoogleCredentials())
@@ -16,9 +16,7 @@ package google.registry.rdap;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.EppResourceUtils.loadByForeignKeyCached;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.request.Action.Method.GET;
import static google.registry.request.Action.Method.HEAD;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
@@ -30,11 +28,9 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.ImmutableSortedSet;
import com.google.common.collect.Iterables;
import com.google.common.collect.Streams;
import com.google.common.flogger.FluentLogger;
import com.google.common.net.InetAddresses;
import com.google.common.primitives.Booleans;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.ForeignKeyUtils;
import google.registry.model.domain.Domain;
import google.registry.model.host.Host;
@@ -58,7 +54,6 @@ import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import javax.inject.Inject;
import javax.persistence.criteria.CriteriaBuilder;
import org.hibernate.Hibernate;
@@ -199,95 +194,65 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
final RdapSearchPattern partialStringQuery) {
// We can't query for undeleted domains as part of the query itself; that would require an
// inequality query on deletion time, and we are already using inequality queries on
// fullyQualifiedDomainName. So we instead pick an arbitrary limit of
// RESULT_SET_SIZE_SCALING_FACTOR times the result set size limit, fetch up to that many, and
// weed out all deleted domains. If there still isn't a full result set's worth of domains, we
// give up and return just the ones we found. Don't use queryItems, because it checks that the
// initial string is at least a certain length, which we don't need in this case. Query the
// domains directly, rather than the foreign keys, because then we have an index on TLD if we
// need it.
// domainName. So we instead pick an arbitrary limit of RESULT_SET_SIZE_SCALING_FACTOR times the
// result set size limit, fetch up to that many, and weed out all deleted domains. If there
// still isn't a full result set's worth of domains, we give up and return just the ones we
// found. Don't use queryItems, because it checks that the initial string is at least a certain
// length, which we don't need in this case. Query the domains directly, rather than the foreign
// keys, because then we have an index on TLD if we need it.
int querySizeLimit = RESULT_SET_SIZE_SCALING_FACTOR * rdapResultSetMaxSize;
RdapResultSet<Domain> resultSet;
if (tm().isOfy()) {
Query<Domain> query =
auditedOfy()
.load()
.type(Domain.class)
.filter("fullyQualifiedDomainName <", partialStringQuery.getNextInitialString())
.filter("fullyQualifiedDomainName >=", partialStringQuery.getInitialString());
if (cursorString.isPresent()) {
query = query.filter("fullyQualifiedDomainName >", cursorString.get());
}
if (partialStringQuery.getSuffix() != null) {
query = query.filter("tld", partialStringQuery.getSuffix());
}
query = query.limit(querySizeLimit);
// Always check for visibility, because we couldn't look at the deletionTime in the query.
resultSet = getMatchingResources(query, true, querySizeLimit);
} else {
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaBuilder criteriaBuilder =
replicaJpaTm().getEntityManager().getCriteriaBuilder();
CriteriaQueryBuilder<Domain> queryBuilder =
CriteriaQueryBuilder.create(replicaJpaTm(), Domain.class)
.where(
"fullyQualifiedDomainName",
criteriaBuilder::like,
String.format("%s%%", partialStringQuery.getInitialString()))
.orderByAsc("fullyQualifiedDomainName");
if (cursorString.isPresent()) {
queryBuilder =
queryBuilder.where(
"fullyQualifiedDomainName",
criteriaBuilder::greaterThan,
cursorString.get());
}
if (partialStringQuery.getSuffix() != null) {
queryBuilder =
queryBuilder.where(
"tld", criteriaBuilder::equal, partialStringQuery.getSuffix());
}
return getMatchingResourcesSql(queryBuilder, true, querySizeLimit);
});
}
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaBuilder criteriaBuilder =
replicaJpaTm().getEntityManager().getCriteriaBuilder();
CriteriaQueryBuilder<Domain> queryBuilder =
CriteriaQueryBuilder.create(replicaJpaTm(), Domain.class)
.where(
"domainName",
criteriaBuilder::like,
String.format("%s%%", partialStringQuery.getInitialString()))
.orderByAsc("domainName");
if (cursorString.isPresent()) {
queryBuilder =
queryBuilder.where(
"domainName", criteriaBuilder::greaterThan, cursorString.get());
}
if (partialStringQuery.getSuffix() != null) {
queryBuilder =
queryBuilder.where(
"tld", criteriaBuilder::equal, partialStringQuery.getSuffix());
}
return getMatchingResources(queryBuilder, true, querySizeLimit);
});
return makeSearchResults(resultSet);
}
/** Searches for domains by domain name with a TLD suffix. */
private DomainSearchResponse searchByDomainNameByTld(String tld) {
// Even though we are not searching on fullyQualifiedDomainName, we want the results to come
// back ordered by name, so we are still in the same boat as
// searchByDomainNameWithInitialString, unable to perform an inequality query on deletion time.
// Don't use queryItems, because it doesn't handle pending deletes.
// Even though we are not searching on domainName, we want the results to come back ordered by
// name, so we are still in the same boat as searchByDomainNameWithInitialString, unable to
// perform an inequality query on deletion time. Don't use queryItems, because it doesn't handle
// pending deletes.
int querySizeLimit = RESULT_SET_SIZE_SCALING_FACTOR * rdapResultSetMaxSize;
RdapResultSet<Domain> resultSet;
if (tm().isOfy()) {
Query<Domain> query = auditedOfy().load().type(Domain.class).filter("tld", tld);
if (cursorString.isPresent()) {
query = query.filter("fullyQualifiedDomainName >", cursorString.get());
}
query = query.order("fullyQualifiedDomainName").limit(querySizeLimit);
resultSet = getMatchingResources(query, true, querySizeLimit);
} else {
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Domain> builder =
queryItemsSql(
Domain.class,
"tld",
tld,
Optional.of("fullyQualifiedDomainName"),
cursorString,
DeletedItemHandling.INCLUDE)
.orderByAsc("fullyQualifiedDomainName");
return getMatchingResourcesSql(builder, true, querySizeLimit);
});
}
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Domain> builder =
queryItems(
Domain.class,
"tld",
tld,
Optional.of("domainName"),
cursorString,
DeletedItemHandling.INCLUDE)
.orderByAsc("domainName");
return getMatchingResources(builder, true, querySizeLimit);
});
return makeSearchResults(resultSet);
}
@@ -340,46 +305,29 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
// incomplete result set if a search asks for something like "ns*", but we need to enforce a
// limit in order to avoid arbitrarily long-running queries.
Optional<String> desiredRegistrar = getDesiredRegistrar();
if (tm().isOfy()) {
Query<Host> query =
queryItems(
Host.class,
"fullyQualifiedHostName",
partialStringQuery,
Optional.empty(),
DeletedItemHandling.EXCLUDE,
maxNameserversInFirstStage);
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
return StreamSupport.stream(query.keys().spliterator(), false)
.map(VKey::from)
.collect(toImmutableSet());
} else {
return replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Host> builder =
queryItemsSql(
Host.class,
"fullyQualifiedHostName",
partialStringQuery,
Optional.empty(),
DeletedItemHandling.EXCLUDE);
if (desiredRegistrar.isPresent()) {
builder =
builder.where(
"currentSponsorClientId",
replicaJpaTm().getEntityManager().getCriteriaBuilder()::equal,
desiredRegistrar.get());
}
return getMatchingResourcesSql(builder, true, maxNameserversInFirstStage)
.resources()
.stream()
.map(Host::createVKey)
.collect(toImmutableSet());
});
}
return replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Host> builder =
queryItems(
Host.class,
"hostName",
partialStringQuery,
Optional.empty(),
DeletedItemHandling.EXCLUDE);
if (desiredRegistrar.isPresent()) {
builder =
builder.where(
"currentSponsorClientId",
replicaJpaTm().getEntityManager().getCriteriaBuilder()::equal,
desiredRegistrar.get());
}
return getMatchingResources(builder, true, maxNameserversInFirstStage)
.resources()
.stream()
.map(Host::createVKey)
.collect(toImmutableSet());
});
}
/** Assembles a list of {@link Host} keys by name when the pattern has no wildcard. */
@@ -474,24 +422,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
private DomainSearchResponse searchByNameserverIp(final InetAddress inetAddress) {
Optional<String> desiredRegistrar = getDesiredRegistrar();
ImmutableSet<VKey<Host>> hostKeys;
if (tm().isOfy()) {
Query<Host> query =
queryItems(
Host.class,
"inetAddresses",
inetAddress.getHostAddress(),
Optional.empty(),
Optional.empty(),
DeletedItemHandling.EXCLUDE,
maxNameserversInFirstStage);
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
hostKeys =
StreamSupport.stream(query.keys().spliterator(), false)
.map(VKey::from)
.collect(toImmutableSet());
} else {
// Hibernate does not allow us to query @Converted array fields directly, either
// in the CriteriaQuery or the raw text format. However, Postgres does -- so we
// use native queries to find hosts where any of the inetAddresses match.
@@ -523,7 +453,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
.map(repoId -> VKey.create(Host.class, repoId))
.collect(toImmutableSet());
});
}
return searchByNameserverRefs(hostKeys);
}
@@ -546,27 +475,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
int numHostKeysSearched = 0;
for (List<VKey<Host>> chunk : Iterables.partition(hostKeys, 30)) {
numHostKeysSearched += chunk.size();
if (tm().isOfy()) {
Query<Domain> query =
auditedOfy()
.load()
.type(Domain.class)
.filter(
"nsHosts in", chunk.stream().map(VKey::getOfyKey).collect(toImmutableSet()));
if (!shouldIncludeDeleted()) {
query = query.filter("deletionTime >", getRequestTime());
// If we are not performing an inequality query, we can filter on the cursor in the query.
// Otherwise, we will need to filter the results afterward.
} else if (cursorString.isPresent()) {
query = query.filter("fullyQualifiedDomainName >", cursorString.get());
}
Stream<Domain> stream = Streams.stream(query).filter(this::isAuthorized);
if (cursorString.isPresent()) {
stream =
stream.filter(domain -> (domain.getDomainName().compareTo(cursorString.get()) > 0));
}
stream.forEach(domainSetBuilder::add);
} else {
replicaJpaTm()
.transact(
() -> {
@@ -574,7 +482,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
CriteriaQueryBuilder<Domain> queryBuilder =
CriteriaQueryBuilder.create(replicaJpaTm(), Domain.class)
.whereFieldContains("nsHosts", hostKey)
.orderByAsc("fullyQualifiedDomainName");
.orderByAsc("domainName");
CriteriaBuilder criteriaBuilder =
replicaJpaTm().getEntityManager().getCriteriaBuilder();
if (!shouldIncludeDeleted()) {
@@ -585,9 +493,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
if (cursorString.isPresent()) {
queryBuilder =
queryBuilder.where(
"fullyQualifiedDomainName",
criteriaBuilder::greaterThan,
cursorString.get());
"domainName", criteriaBuilder::greaterThan, cursorString.get());
}
replicaJpaTm()
.criteriaQuery(queryBuilder.build())
@@ -600,7 +506,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
});
}
});
}
}
List<Domain> domains = domainSetBuilder.build().asList();
metricInformationBuilder.setNumHostsRetrieved(numHostKeysSearched);
@@ -16,7 +16,6 @@ package google.registry.rdap;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.rdap.RdapUtils.getRegistrarByIanaIdentifier;
import static google.registry.request.Action.Method.GET;
import static google.registry.request.Action.Method.HEAD;
@@ -27,7 +26,6 @@ import com.google.common.collect.Iterables;
import com.google.common.collect.Streams;
import com.google.common.primitives.Booleans;
import com.google.common.primitives.Longs;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.contact.Contact;
import google.registry.model.registrar.Registrar;
import google.registry.persistence.VKey;
@@ -261,39 +259,24 @@ public class RdapEntitySearchAction extends RdapSearchActionBase {
|| (cursorType == CursorType.REGISTRAR)) {
resultSet = RdapResultSet.create(ImmutableList.of());
} else {
if (tm().isOfy()) {
Query<Contact> query =
queryItems(
Contact.class,
"searchName",
partialStringQuery,
cursorQueryString, // if we get here and there's a cursor, it must be a contact
DeletedItemHandling.EXCLUDE,
rdapResultSetMaxSize + 1);
if (!rdapAuthorization.role().equals(Role.ADMINISTRATOR)) {
query = query.filter("currentSponsorClientId in", rdapAuthorization.registrarIds());
}
resultSet = getMatchingResources(query, false, rdapResultSetMaxSize + 1);
} else {
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Contact> builder =
queryItemsSql(
Contact.class,
"searchName",
partialStringQuery,
cursorQueryString,
DeletedItemHandling.EXCLUDE);
if (!rdapAuthorization.role().equals(Role.ADMINISTRATOR)) {
builder =
builder.whereFieldIsIn(
"currentSponsorClientId", rdapAuthorization.registrarIds());
}
return getMatchingResourcesSql(builder, false, rdapResultSetMaxSize + 1);
});
}
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Contact> builder =
queryItems(
Contact.class,
"searchName",
partialStringQuery,
cursorQueryString,
DeletedItemHandling.EXCLUDE);
if (!rdapAuthorization.role().equals(Role.ADMINISTRATOR)) {
builder =
builder.whereFieldIsIn(
"currentSponsorClientId", rdapAuthorization.registrarIds());
}
return getMatchingResources(builder, false, rdapResultSetMaxSize + 1);
});
}
}
return makeSearchResults(resultSet, registrars, QueryType.FULL_NAME);
@@ -386,31 +369,18 @@ public class RdapEntitySearchAction extends RdapSearchActionBase {
if (subtype == Subtype.REGISTRARS) {
contactResultSet = RdapResultSet.create(ImmutableList.of());
} else {
if (tm().isOfy()) {
contactResultSet =
getMatchingResources(
queryItemsByKey(
Contact.class,
partialStringQuery,
cursorQueryString,
getDeletedItemHandling(),
querySizeLimit),
shouldIncludeDeleted(),
querySizeLimit);
} else {
contactResultSet =
replicaJpaTm()
.transact(
() ->
getMatchingResourcesSql(
queryItemsByKeySql(
Contact.class,
partialStringQuery,
cursorQueryString,
getDeletedItemHandling()),
shouldIncludeDeleted(),
querySizeLimit));
}
contactResultSet =
replicaJpaTm()
.transact(
() ->
getMatchingResources(
queryItemsByKey(
Contact.class,
partialStringQuery,
cursorQueryString,
getDeletedItemHandling()),
shouldIncludeDeleted(),
querySizeLimit));
}
return makeSearchResults(contactResultSet, registrars, QueryType.HANDLE);
}
@@ -21,7 +21,6 @@ import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.ImmutableSetMultimap.toImmutableSetMultimap;
import static google.registry.model.EppResourceUtils.isLinked;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.rdap.RdapIcannStandardInformation.CONTACT_REDACTED_VALUE;
import static google.registry.util.CollectionUtils.union;
@@ -91,8 +90,9 @@ import org.joda.time.DateTime;
*
* <p>The JSON format specifies that entities should be supplied with links indicating how to fetch
* them via RDAP, which requires the URL to the RDAP server. The linkBase parameter, passed to many
* of the methods, is used as the first part of the link URL. For instance, if linkBase is
* "http://rdap.org/dir/", the link URLs will look like "http://rdap.org/dir/domain/XXXX", etc.
* of the methods, is used as the first part of the link URL. For instance, if linkBase is <a
* href="http://rdap.org/dir/"></a>, the link URLs will look like <a
* href="http://rdap.org/dir/domain/XXXX"></a>, etc.
*
* @see <a href="https://tools.ietf.org/html/rfc9083">RFC 9083: JSON Responses for the Registration
* Data Access Protocol (RDAP)</a>
@@ -211,15 +211,15 @@ public class RdapJsonFormatter {
.put(HistoryEntry.Type.CONTACT_DELETE, EventAction.DELETION)
.put(HistoryEntry.Type.CONTACT_TRANSFER_APPROVE, EventAction.TRANSFER)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_AUTORENEW, EventAction.REREGISTRATION)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_DELETE, EventAction.DELETION)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_RENEW, EventAction.REREGISTRATION)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_RESTORE, EventAction.REINSTANTIATION)
/** Section 2.3.2.3, optional. */
/* Section 2.3.2.3, optional. */
.put(HistoryEntry.Type.DOMAIN_TRANSFER_APPROVE, EventAction.TRANSFER)
.put(HistoryEntry.Type.HOST_CREATE, EventAction.REGISTRATION)
.put(HistoryEntry.Type.HOST_DELETE, EventAction.DELETION)
@@ -533,7 +533,7 @@ public class RdapJsonFormatter {
// state/province, postal code, country
//
// Note that in theory we have to show the Organization and state/province and country for the
// REGISTRANT. For now we won't do that until we make sure it's really OK for GDPR
// REGISTRANT. For now, we won't do that until we make sure it's really OK for GDPR
//
if (!isAuthorized) {
// RDAP Response Profile 2.7.4.3: if we redact values from the contact, we MUST include a
@@ -749,9 +749,9 @@ public class RdapJsonFormatter {
if (outputDataType != OutputDataType.SUMMARY) {
ImmutableList<RdapContactEntity> registrarContacts =
registrar.getContacts().stream()
.map(registrarContact -> makeRdapJsonForRegistrarContact(registrarContact))
.filter(optional -> optional.isPresent())
.map(optional -> optional.get())
.map(RdapJsonFormatter::makeRdapJsonForRegistrarContact)
.filter(Optional::isPresent)
.map(Optional::get)
.filter(
contact ->
outputDataType == OutputDataType.FULL
@@ -886,10 +886,6 @@ public class RdapJsonFormatter {
// 2.3.2.3 An event of *eventAction* type *transfer*, with the last date and time that the
// domain was transferred. The event of *eventAction* type *transfer* MUST be omitted if the
// domain name has not been transferred since it was created.
Iterable<? extends HistoryEntry> historyEntries;
if (tm().isOfy()) {
historyEntries = HistoryEntryDao.loadHistoryObjectsForResource(resource.createVKey());
} else {
VKey<? extends EppResource> resourceVkey = resource.createVKey();
Class<? extends HistoryEntry> historyClass =
HistoryEntryDao.getHistoryClassFromParent(resourceVkey.getKind());
@@ -903,10 +899,14 @@ public class RdapJsonFormatter {
.replace("%entityName%", entityName)
.replace("%repoIdField%", repoIdFieldName)
.replace("%repoIdValue%", resourceVkey.getSqlKey().toString());
historyEntries =
replicaJpaTm()
.transact(() -> replicaJpaTm().getEntityManager().createQuery(jpql).getResultList());
}
Iterable<HistoryEntry> historyEntries =
replicaJpaTm()
.transact(
() ->
replicaJpaTm()
.getEntityManager()
.createQuery(jpql, HistoryEntry.class)
.getResultList());
for (HistoryEntry historyEntry : historyEntries) {
EventAction rdapEventAction =
HISTORY_ENTRY_TYPE_TO_RDAP_EVENT_ACTION_MAP.get(historyEntry.getType());
@@ -1131,7 +1131,7 @@ public class RdapJsonFormatter {
* all these objects are projected to the same "now".
*
* <p>This "now" will also be considered the time of the "last update of RDAP database" event that
* RDAP sepc requires.
* RDAP spec requires.
*
* <p>We would have set this during the constructor, but the clock is injected after construction.
* So instead we set the time during the first call to this function.

Some files were not shown because too many files have changed in this diff Show More