Compare commits

..
Author SHA1 Message Date
Lai JiangandGitHub 82092b3516 Remove ofy-only functions in TransactionManager (#1826)
Also remove the use of auditedOfy in places other than the
GaeUserIdConverter.
2022-10-25 15:52:00 -04:00
sarahcaseybotandGitHub 0746d28e0c Check token type of currentPackageToken (#1825)
* Check currentPackageToken TokenType

* Check TokenType of currentPackageToken

* Check that token already exists
2022-10-25 12:39:33 -04:00
Lai JiangandGitHub aaa311ec40 Remove the mechanism to compare objects across database (#1822)
The migration is done.

<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1822)
<!-- Reviewable:end -->
2022-10-20 13:19:48 -04:00
Lai JiangandGitHub addef17904 Does not self allocate IDs in Beam by default. (#1809)
* Does not self allocate IDs in Beam by default.

Per b/250948425, it is dangerous to implicitly allow all Beam pipelines
to create buildables by self allocating the IDs. This change makes it so
that one has to explicitly request self allocation in Beam.

A boolean is added to the pipeline option so that it can be passed to
the beam worker initializer that controls the behavior of the JVM on
each worker. Note that we did not add the option in the metadata.json file
because we did not want people to use the override at run time when launching
a pipeline, due to the risk. As shown in RdePipeline.java, we instead
explicitly hard-code the option in the pipeline. There is nothing that
stops one to supply that option when launching the pipeline, but it's
not advised.

Tested=deployed the pipeline alpha and ran it.
2022-10-19 20:44:06 -04:00
Weimin YuandGitHub 8fe3c08069 Properly create and use default credential (#1818)
* Properly create and use default credential

This PR consists of the following changes:

- Stopped adding scopes to the default credential when using it to access other
  non-workspace GCP APIs. Scopes are not needed here.

- Started applying scopes to the default credential when using to access
  Drive and Sheets APIs.
  - Upgraded Drive access from the deprecated credential lib to the
    up-to-date one
  - Switched Sheet access from the exported json credential to the
    scoped default credential.

This PR requires that the affected files be writable to the default
service account (project-name@appspot.gserviceaccount.com) of the
project.

- This is already the case for exported files (premium terms, reserved
  terms, and domain list).

- The registrar sync sheets in alpha, sandbox, and production have been
  updated with the new permissions.

All impacted operations have been tested in alpha.

* Properly create and use default credential

This PR consists of the following changes:

- Added a new method to generate scope-less default credential when using it to
  access other non-workspace GCP APIs. Scopes are not needed here.

  - Started to use the new credential in the SecreteManager.
  - Will migrate other usages to this new credential gradually.
  - Marked the old DefaultCredential as deprecated.

- Started applying scopes to the default credential when using to access Drive
  and Sheets APIs.

  - Upgraded Drive access from the deprecated credentials lib
  - Switched Sheet access from the exported json credential to the scoped
    default credential.

This PR requires that the affected files be writable to the default service
account (project-name@appspot.gserviceaccount.com) of the project.

- This is already the case for exported files (premium terms, reserved terms,
  and domain list).

- The registrar sync sheets in alpha, sandbox, and production have been
  updated with the new permissions.

All impacted operations have been tested in alpha.
2022-10-18 20:20:36 -04:00
sarahcaseybotandGitHub 5dc796b1f7 Add monitoring for package max create limit (#1798)
* Add action for checking package domain create limit compliance

* Add create limit monitoring

* Change variable name

* Add more logging
2022-10-18 12:39:53 -04:00
Ben McIlwainandGitHub 8bddf35d0d Revert "Upgrade App Engine Standard to Java 17 w/ bundled APIs (#1816)" (#1817)
This reverts commit 1ab077d267.

Apparently the new version of Spinnaker that is compatible with this doesn't
work for our release, so we need to roll this back for now. (Again!)
2022-10-13 10:05:47 -04:00
Pavlo TkachandGitHub 7b9c16ca3e Update conditions when domain update flow triggers dns publish task (#1811)
Addressing b/246375161
2022-10-12 10:25:33 -04:00
Ben McIlwainandGitHub 1ab077d267 Upgrade App Engine Standard to Java 17 w/ bundled APIs (#1816) 2022-10-11 20:06:37 -04:00
gbrodmanandGitHub ca65fbcc79 Refactor createSynthetic to be a command instead of a pipeline (#1813) 2022-10-11 12:23:31 -04:00
sarahcaseybotandGitHub 0cfa7f8081 Remove allocation token check for transfering package domains (#1814) 2022-10-11 11:37:52 -04:00
Lai JiangandGitHub 9e31047c3a Fix nomulus command (#1812)
go/r3pr/1805 introduced an injectable clock in a few commands, but we
forgot to add the corresponding injector in the component. This PR fixes
it.
2022-10-09 16:45:42 -04:00
Pavlo TkachandGitHub b7c2e8fba5 Limit environments allowed to send emails out (#1807) 2022-10-07 12:12:57 -04:00
Pavlo TkachandGitHub a299df3005 Add fallback for Spec11 ThreatMatch parser (#1806) 2022-10-06 13:54:43 +00:00
Pavlo TkachandGitHub a9b35c163d Revert "Do not enqueue DNS updates when flow doesn't affect nameservers (#1785)" (#1808)
This reverts commit 775f672f2a.
2022-10-05 14:13:52 -04:00
gbrodmanandGitHub 9da24d114c Use injected times in URSC and CommandTestCase (#1805)
We started getting failures because some of the tests used October. In
general we should freeze the clock for testing as much as possible.

Same thing with the Get*Commands
2022-10-04 15:36:41 -04:00
Lai JiangandGitHub 7dd5876315 Refactor VKeyConverter (#1794)
Remove the redundant composite key boolean and simply the annotation
structure a bit.
2022-10-03 15:49:18 -04:00
gbrodmanandGitHub d1a259f63a Modify the CreateSynthetic pipeline to run over all non-deleted domains (#1803) 2022-10-03 15:15:41 -04:00
sarahcaseybotandGitHub 8c5d2e9d92 Don't allow package tokens to discount premium names (#1804) 2022-10-03 14:27:10 -04:00
gbrodmanandGitHub cca1306b09 Change some READ_COMMITTED levels to REPEATABLE_READ (#1802)
Basically, any time we're loading a bunch of linked objects that might
change, we want to have REPEATABLE_READ so that another transaction
doesn't come along and smush whatever we think we're loading.

The following instances of READ_COMMITTED haven't changed:
- RdePipeline (it only loads immutable objects like histories)
- Invoicing pipeline (only immutable objects like BillingEvents)
- Spec11 (doesn't use any linked info from Domain)

This also changes the PersistenceModule to use REPEATABLE_READ by
default on the replica JPA TM, for the standard reasoning.
2022-09-30 14:44:50 -04:00
Weimin YuandGitHub 47071b0fbb Restore log4j exclusion in gradle build (#1801) 2022-09-30 14:04:00 -04:00
Weimin YuandGitHub d83565d37e Add a new allowed license string (#1800)
There are sporadic errors when building on desktop using maven central.
2022-09-30 14:03:17 -04:00
Weimin YuandGitHub a557b3f376 Disable the cron job for ResaveAllEppResourcesPipelineAction (#1799)
See b/249863289 for more information.
2022-09-30 12:05:29 -04:00
sarahcaseybotandGitHub f4a49864b5 Add a get_package_promotion Command (#1793)
* Add a get_package_promotion Command

* add changes to loadByTokenString

* Fix test
2022-09-29 15:02:16 -04:00
110 changed files with 1809 additions and 2776 deletions
@@ -207,6 +207,9 @@
{
"moduleLicense": "GNU Library General Public License v2.1 or later"
},
{
"moduleLicense": "GNU Lesser General Public License v3.0"
},
// This is just 3-clause BSD.
{
"moduleLicense": "Go License"
@@ -0,0 +1,80 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.batch;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableList;
import com.google.common.flogger.FluentLogger;
import google.registry.model.domain.DomainHistory;
import google.registry.model.domain.token.PackagePromotion;
import google.registry.request.Action;
import google.registry.request.Action.Service;
import google.registry.request.auth.Auth;
import java.util.List;
/**
* An action that checks all {@link PackagePromotion} objects for compliance with their max create
* limit.
*/
@Action(
service = Service.BACKEND,
path = CheckPackagesComplianceAction.PATH,
auth = Auth.AUTH_INTERNAL_OR_ADMIN)
public class CheckPackagesComplianceAction implements Runnable {
public static final String PATH = "/_dr/task/checkPackagesCompliance";
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
@Override
public void run() {
tm().transact(
() -> {
ImmutableList<PackagePromotion> packages = tm().loadAllOf(PackagePromotion.class);
ImmutableList.Builder<PackagePromotion> packagesOverCreateLimit =
new ImmutableList.Builder<>();
for (PackagePromotion packagePromo : packages) {
List<DomainHistory> creates =
jpaTm()
.query(
"FROM DomainHistory WHERE current_package_token = :token AND"
+ " modificationTime >= :lastBilling AND type = 'DOMAIN_CREATE'",
DomainHistory.class)
.setParameter("token", packagePromo.getToken().getSqlKey().toString())
.setParameter(
"lastBilling", packagePromo.getNextBillingDate().minusYears(1))
.getResultList();
if (creates.size() > packagePromo.getMaxCreates()) {
int overage = creates.size() - packagePromo.getMaxCreates();
logger.atInfo().log(
"Package with package token %s has exceeded their max domain creation limit"
+ " by %d name(s).",
packagePromo.getToken().getSqlKey(), overage);
packagesOverCreateLimit.add(packagePromo);
}
}
if (packagesOverCreateLimit.build().isEmpty()) {
logger.atInfo().log("Found no packages over their create limit.");
} else {
logger.atInfo().log(
"Found %d packages over their create limit.",
packagesOverCreateLimit.build().size());
// TODO(sarahbot@) Send email to registrar and registry informing of creation
// overage once email template is finalized.
}
});
}
}
@@ -16,6 +16,7 @@ package google.registry.beam.common;
import google.registry.beam.common.RegistryJpaIO.Write;
import google.registry.config.RegistryEnvironment;
import google.registry.model.annotations.DeleteAfterMigration;
import google.registry.persistence.PersistenceModule.JpaTransactionManagerType;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import java.util.Objects;
@@ -65,6 +66,17 @@ public interface RegistryPipelineOptions extends GcpOptions {
void setSqlWriteShards(int maxConcurrentSqlWriters);
@DeleteAfterMigration
@Description(
"Whether to use self allocated primary IDs when building entities. This should only be used"
+ " when the IDs are not significant and the resulting entities are not persisted back to"
+ " the database. Use with caution as self allocated IDs are not unique across workers,"
+ " and persisting entities with these IDs can be dangerous.")
@Default.Boolean(false)
boolean getUseSelfAllocatedId();
void setUseSelfAllocatedId(boolean useSelfAllocatedId);
static RegistryPipelineComponent toRegistryPipelineComponent(RegistryPipelineOptions options) {
return DaggerRegistryPipelineComponent.builder()
.isolationOverride(options.getIsolationOverride())
@@ -22,6 +22,8 @@ import dagger.Lazy;
import google.registry.config.RegistryEnvironment;
import google.registry.config.SystemPropertySetter;
import google.registry.model.AppEngineEnvironment;
import google.registry.model.IdService;
import google.registry.model.IdService.SelfAllocatedIdSupplier;
import google.registry.persistence.transaction.JpaTransactionManager;
import google.registry.persistence.transaction.TransactionManagerFactory;
import org.apache.beam.sdk.harness.JvmInitializer;
@@ -65,12 +67,20 @@ public class RegistryPipelineWorkerInitializer implements JvmInitializer {
transactionManagerLazy = registryPipelineComponent.getJpaTransactionManager();
}
TransactionManagerFactory.setJpaTmOnBeamWorker(transactionManagerLazy::get);
// Masquerade all threads as App Engine threads so we can create Ofy keys in the pipeline. Also
// Masquerade all threads as App Engine threads, so we can create Ofy keys in the pipeline. Also
// loads all ofy entities.
new AppEngineEnvironment("s~" + registryPipelineComponent.getProjectId())
.setEnvironmentForAllThreads();
// Set the system property so that we can call IdService.allocateId() without access to
// datastore.
SystemPropertySetter.PRODUCTION_IMPL.setProperty(PROPERTY, "true");
// Use self-allocated IDs if requested. Note that this inevitably results in duplicate IDs from
// multiple workers, which can also collide with existing IDs in the database. So they cannot be
// dependent upon for comparison or anything significant. The resulting entities can never be
// persisted back into the database. This is a stop-gap measure that should only be used when
// you need to create Buildables in Beam, but do not have control over how the IDs are
// allocated, and you don't care about the generated IDs as long
// as you can build the entities.
if (registryOptions.getUseSelfAllocatedId()) {
IdService.setIdSupplier(SelfAllocatedIdSupplier.getInstance());
}
}
}
@@ -128,7 +128,7 @@ import org.joda.time.DateTime;
* <h2>{@link EppResource}</h2>
*
* All EPP resources are loaded from the corresponding {@link HistoryEntry}, which has the resource
* embedded. In general we find most recent history entry before watermark and filter out the ones
* embedded. In general, we find most recent history entry before watermark and filter out the ones
* that are soft-deleted by watermark. The history is emitted as pairs of (resource repo ID: history
* revision ID) from the SQL query.
*
@@ -164,7 +164,7 @@ import org.joda.time.DateTime;
*
* The (pending deposit: deposit fragment) pairs from different resources are combined and grouped
* by pending deposit. For each pending deposit, all the relevant deposit fragments are written into
* a encrypted file stored on GCS. The filename is uniquely determined by the Beam job ID so there
* an encrypted file stored on GCS. The filename is uniquely determined by the Beam job ID so there
* is no need to lock the GCS write operation to prevent stomping. The cursor for staging the
* pending deposit is then rolled forward, and the next action is enqueued. The latter two
* operations are performed in a transaction so the cursor is rolled back if enqueueing failed.
@@ -698,8 +698,8 @@ public class RdePipeline implements Serializable {
}
/**
* Encodes the pending deposit set in an URL safe string that is sent to the pipeline worker by
* the pipeline launcher as a pipeline option.
* Encodes the pending deposit set in a URL safe string that is sent to the pipeline worker by the
* pipeline launcher as a pipeline option.
*/
public static String encodePendingDeposits(ImmutableSet<PendingDeposit> pendingDeposits)
throws IOException {
@@ -715,6 +715,12 @@ public class RdePipeline implements Serializable {
PipelineOptionsFactory.register(RdePipelineOptions.class);
RdePipelineOptions options =
PipelineOptionsFactory.fromArgs(args).withValidation().as(RdePipelineOptions.class);
// We need to self allocate the IDs because the pipeline creates EPP resources from history
// entries and projects them to watermark. These buildable entities would otherwise request an
// ID from datastore, which Beam does not have access to. The IDs are not included in the
// deposits or are these entities persisted back to the database, so it is OK to use a self
// allocated ID to get around the limitations of beam.
options.setUseSelfAllocatedId(true);
RegistryPipelineOptions.validateRegistryPipelineOptions(options);
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_COMMITTED);
DaggerRdePipeline_RdePipelineComponent.builder().options(options).build().rdePipeline().run();
@@ -91,7 +91,6 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
}
void setupPipeline(Pipeline pipeline) {
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_COMMITTED);
if (options.getFast()) {
fastResaveContacts(pipeline);
fastResaveDomains(pipeline);
@@ -194,6 +193,7 @@ public class ResaveAllEppResourcesPipeline implements Serializable {
PipelineOptionsFactory.fromArgs(args)
.withValidation()
.as(ResaveAllEppResourcesPipelineOptions.class);
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_REPEATABLE_READ);
new ResaveAllEppResourcesPipeline(options).run();
}
}
@@ -26,6 +26,7 @@ public abstract class ThreatMatch implements Serializable {
private static final String THREAT_TYPE_FIELD = "threatType";
private static final String DOMAIN_NAME_FIELD = "domainName";
private static final String OUTDATED_NAME_FIELD = "fullyQualifiedDomainName";
/** Returns what kind of threat it is (malware, phishing etc.) */
public abstract String threatType();
@@ -46,7 +47,12 @@ public abstract class ThreatMatch implements Serializable {
/** Parses a {@link JSONObject} and returns an equivalent {@link ThreatMatch}. */
public static ThreatMatch fromJSON(JSONObject threatMatch) throws JSONException {
// TODO: delete OUTDATED_NAME_FIELD once we no longer process reports saved with
// fullyQualifiedDomainName in them, likely 2023
return new AutoValue_ThreatMatch(
threatMatch.getString(THREAT_TYPE_FIELD), threatMatch.getString(DOMAIN_NAME_FIELD));
threatMatch.getString(THREAT_TYPE_FIELD),
threatMatch.has(OUTDATED_NAME_FIELD)
? threatMatch.getString(OUTDATED_NAME_FIELD)
: threatMatch.getString(DOMAIN_NAME_FIELD));
}
}
@@ -16,7 +16,6 @@ package google.registry.config;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.common.collect.ImmutableList;
import dagger.Module;
@@ -37,6 +36,36 @@ import javax.inject.Singleton;
@Module
public abstract class CredentialModule {
/**
* Provides a {@link GoogleCredentialsBundle} backed by the application default credential from
* the Google Cloud Runtime. This credential may be used to access GCP APIs that are NOT part of
* the Google Workspace.
*
* <p>The credential returned by the Cloud Runtime depends on the runtime environment:
*
* <ul>
* <li>On App Engine, returns a scope-less {@code ComputeEngineCredentials} for
* PROJECT_ID@appspot.gserviceaccount.com
* <li>On Compute Engine, returns a scope-less {@code ComputeEngineCredentials} for
* PROJECT_NUMBER-compute@developer.gserviceaccount.com
* <li>On end user host, this returns the credential downloaded by gcloud. Please refer to <a
* href="https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login">Cloud
* SDK documentation</a> for details.
* </ul>
*/
@ApplicationDefaultCredential
@Provides
@Singleton
public static GoogleCredentialsBundle provideApplicationDefaultCredential() {
GoogleCredentials credential;
try {
credential = GoogleCredentials.getApplicationDefault();
} catch (IOException e) {
throw new RuntimeException(e);
}
return GoogleCredentialsBundle.create(credential);
}
/**
* Provides the default {@link GoogleCredentialsBundle} from the Google Cloud runtime.
*
@@ -70,26 +99,19 @@ public abstract class CredentialModule {
}
/**
* Provides the default {@link GoogleCredential} from the Google Cloud runtime for G Suite
* Drive API.
* TODO(b/138195359): Deprecate this credential once we figure out how to use
* {@link GoogleCredentials} for G Suite Drive API.
* Provides a {@link GoogleCredentialsBundle} for accessing Google Workspace APIs, such as Drive
* and Sheets.
*/
@GSuiteDriveCredential
@GoogleWorkspaceCredential
@Provides
@Singleton
public static GoogleCredential provideGSuiteDriveCredential(
public static GoogleCredentialsBundle provideGSuiteDriveCredential(
@ApplicationDefaultCredential GoogleCredentialsBundle applicationDefaultCredential,
@Config("defaultCredentialOauthScopes") ImmutableList<String> requiredScopes) {
GoogleCredential credential;
try {
credential = GoogleCredential.getApplicationDefault();
} catch (IOException e) {
throw new RuntimeException(e);
}
if (credential.createScopedRequired()) {
credential = credential.createScoped(requiredScopes);
}
return credential;
GoogleCredentials credential = applicationDefaultCredential.getGoogleCredentials();
// Although credential is scope-less, its `createScopedRequired` method still returns false.
credential = credential.createScoped(requiredScopes);
return GoogleCredentialsBundle.create(credential);
}
/**
@@ -136,18 +158,24 @@ public abstract class CredentialModule {
.createScoped(requiredScopes));
}
/** Dagger qualifier for the scope-less Application Default Credential. */
@Qualifier
@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface ApplicationDefaultCredential {}
/** Dagger qualifier for the Application Default Credential. */
@Qualifier
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Deprecated // Switching to @ApplicationDefaultCredential
public @interface DefaultCredential {}
/** Dagger qualifier for the credential for G Suite Drive API. */
/** Dagger qualifier for the credential for Google Workspace APIs. */
@Qualifier
@Documented
@Retention(RetentionPolicy.RUNTIME)
public @interface GSuiteDriveCredential {}
public @interface GoogleWorkspaceCredential {}
/**
* Dagger qualifier for a credential from a service account's JSON key, to be used in non-request
@@ -102,6 +102,7 @@
<target>backend</target>
</cron>
<!-- TODO(b/249863289): disable until it is safe to run this pipeline
<cron>
<url><![CDATA[/_dr/task/resaveAllEppResourcesPipeline?fast=true]]></url>
<description>
@@ -110,6 +111,7 @@
<schedule>1st monday of month 09:00</schedule>
<target>backend</target>
</cron>
-->
<cron>
<url><![CDATA[/_dr/task/updateRegistrarRdapBaseUrls]]></url>
@@ -86,6 +86,7 @@
<target>backend</target>
</cron>
<!-- TODO(b/249863289): disable until it is safe to run this pipeline
<cron>
<url><![CDATA[/_dr/task/resaveAllEppResourcesPipeline?fast=true]]></url>
<description>
@@ -94,6 +95,7 @@
<schedule>1st monday of month 09:00</schedule>
<target>backend</target>
</cron>
-->
<cron>
<url><![CDATA[/_dr/cron/fanout?queue=retryable-cron-tasks&endpoint=/_dr/task/exportDomainLists&runInEmpty]]></url>
@@ -14,16 +14,16 @@
package google.registry.export;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.services.drive.Drive;
import dagger.Component;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule;
import google.registry.config.CredentialModule.GSuiteDriveCredential;
import google.registry.config.CredentialModule.GoogleWorkspaceCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.storage.drive.DriveConnection;
import google.registry.util.GoogleCredentialsBundle;
import javax.inject.Singleton;
/** Dagger module for Google {@link Drive} service connection objects. */
@@ -32,13 +32,13 @@ public final class DriveModule {
@Provides
static Drive provideDrive(
@GSuiteDriveCredential GoogleCredential googleCredential,
@GoogleWorkspaceCredential GoogleCredentialsBundle googleCredential,
@Config("projectId") String projectId) {
return new Drive.Builder(
googleCredential.getTransport(),
googleCredential.getHttpTransport(),
googleCredential.getJsonFactory(),
googleCredential)
googleCredential.getHttpRequestInitializer())
.setApplicationName(projectId)
.build();
}
@@ -163,7 +163,7 @@ public final class SyncGroupMembersAction implements Runnable {
registrarsToSave.add(result.getKey().asBuilder().setContactsRequireSyncing(false).build());
}
}
tm().transactNew(() -> tm().updateAll(registrarsToSave.build()));
tm().transact(() -> tm().updateAll(registrarsToSave.build()));
return errors;
}
@@ -17,7 +17,7 @@ package google.registry.export.sheet;
import com.google.api.services.sheets.v4.Sheets;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule.JsonCredential;
import google.registry.config.CredentialModule.GoogleWorkspaceCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.util.GoogleCredentialsBundle;
@@ -27,7 +27,7 @@ public final class SheetsServiceModule {
@Provides
static Sheets provideSheets(
@JsonCredential GoogleCredentialsBundle credentialsBundle,
@GoogleWorkspaceCredential GoogleCredentialsBundle credentialsBundle,
@Config("projectId") String projectId) {
return new Sheets.Builder(
credentialsBundle.getHttpTransport(),
@@ -32,8 +32,6 @@ import static google.registry.flows.domain.DomainTransferUtils.createLosingTrans
import static google.registry.flows.domain.DomainTransferUtils.createPendingTransferData;
import static google.registry.flows.domain.DomainTransferUtils.createTransferResponse;
import static google.registry.flows.domain.DomainTransferUtils.createTransferServerApproveEntities;
import static google.registry.flows.domain.token.AllocationTokenFlowUtils.maybeApplyPackageRemovalToken;
import static google.registry.flows.domain.token.AllocationTokenFlowUtils.verifyTokenAllowedOnDomain;
import static google.registry.model.eppoutput.Result.Code.SUCCESS_WITH_ACTION_PENDING;
import static google.registry.model.reporting.HistoryEntry.Type.DOMAIN_TRANSFER_REQUEST;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
@@ -65,7 +63,6 @@ import google.registry.model.domain.fee.FeeTransferCommandExtension;
import google.registry.model.domain.fee.FeeTransformResponseExtension;
import google.registry.model.domain.metadata.MetadataExtension;
import google.registry.model.domain.superuser.DomainTransferRequestSuperuserExtension;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.domain.token.AllocationTokenExtension;
import google.registry.model.eppcommon.AuthInfo;
import google.registry.model.eppcommon.StatusValue;
@@ -135,8 +132,6 @@ import org.joda.time.DateTime;
* google.registry.flows.domain.token.AllocationTokenFlowUtils.AllocationTokenNotValidForTldException}
* @error {@link
* google.registry.flows.domain.token.AllocationTokenFlowUtils.AlreadyRedeemedAllocationTokenException}
* @error {@link
* google.registry.flows.domain.token.AllocationTokenFlowUtils.MissingRemovePackageTokenOnPackageDomainException}
*/
@ReportingSpec(ActivityReportField.DOMAIN_TRANSFER_REQUEST)
public final class DomainTransferRequestFlow implements TransactionalFlow {
@@ -174,23 +169,19 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
extensionManager.validate();
DateTime now = tm().getTransactionTime();
Domain existingDomain = loadAndVerifyExistence(Domain.class, targetId, now);
Optional<AllocationToken> allocationToken =
allocationTokenFlowUtils.verifyAllocationTokenIfPresent(
existingDomain,
Registry.get(existingDomain.getTld()),
gainingClientId,
now,
eppInput.getSingleExtension(AllocationTokenExtension.class));
allocationTokenFlowUtils.verifyAllocationTokenIfPresent(
existingDomain,
Registry.get(existingDomain.getTld()),
gainingClientId,
now,
eppInput.getSingleExtension(AllocationTokenExtension.class));
Optional<DomainTransferRequestSuperuserExtension> superuserExtension =
eppInput.getSingleExtension(DomainTransferRequestSuperuserExtension.class);
Period period =
superuserExtension.isPresent()
? superuserExtension.get().getRenewalPeriod()
: ((Transfer) resourceCommand).getPeriod();
verifyTransferAllowed(existingDomain, period, now, superuserExtension, allocationToken);
// If client passed an applicable static token this updates the domain
existingDomain = maybeApplyPackageRemovalToken(existingDomain, allocationToken);
verifyTransferAllowed(existingDomain, period, now, superuserExtension);
String tld = existingDomain.getTld();
Registry registry = Registry.get(tld);
@@ -303,11 +294,9 @@ public final class DomainTransferRequestFlow implements TransactionalFlow {
Domain existingDomain,
Period period,
DateTime now,
Optional<DomainTransferRequestSuperuserExtension> superuserExtension,
Optional<AllocationToken> allocationToken)
Optional<DomainTransferRequestSuperuserExtension> superuserExtension)
throws EppException {
verifyNoDisallowedStatuses(existingDomain, DISALLOWED_STATUSES);
verifyTokenAllowedOnDomain(existingDomain, allocationToken);
if (!isSuperuser) {
verifyAuthInfoPresentForResourceTransfer(authInfo);
verifyAuthInfo(authInfo.get(), existingDomain);
@@ -87,6 +87,7 @@ import google.registry.model.poll.PendingActionNotificationResponse.DomainPendin
import google.registry.model.poll.PollMessage;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.model.tld.Registry;
import java.util.Objects;
import java.util.Optional;
import javax.inject.Inject;
import org.joda.time.DateTime;
@@ -181,7 +182,9 @@ public final class DomainUpdateFlow implements TransactionalFlow {
DomainHistory domainHistory =
historyBuilder.setType(DOMAIN_UPDATE).setDomain(newDomain).build();
validateNewState(newDomain);
dnsQueue.addDomainRefreshTask(targetId);
if (requiresDnsUpdate(existingDomain, newDomain)) {
dnsQueue.addDomainRefreshTask(targetId);
}
ImmutableSet.Builder<ImmutableObject> entitiesToSave = new ImmutableSet.Builder<>();
entitiesToSave.add(newDomain, domainHistory);
Optional<BillingEvent.OneTime> statusUpdateBillingEvent =
@@ -203,6 +206,16 @@ public final class DomainUpdateFlow implements TransactionalFlow {
return responseBuilder.build();
}
/** Determines if any of the changes to new domain should trigger DNS update. */
private boolean requiresDnsUpdate(Domain existingDomain, Domain newDomain) {
if (existingDomain.shouldPublishToDns() != newDomain.shouldPublishToDns()
|| !Objects.equals(newDomain.getDsData(), existingDomain.getDsData())
|| !Objects.equals(newDomain.getNsHosts(), existingDomain.getNsHosts())) {
return true;
}
return false;
}
/** Fail if the object doesn't exist or was deleted. */
private void verifyUpdateAllowed(Update command, Domain existingDomain, DateTime now)
throws EppException {
@@ -267,12 +280,18 @@ public final class DomainUpdateFlow implements TransactionalFlow {
.setLastEppUpdateRegistrarId(registrarId)
.addStatusValues(add.getStatusValues())
.removeStatusValues(remove.getStatusValues())
.addNameservers(add.getNameservers().stream().collect(toImmutableSet()))
.removeNameservers(remove.getNameservers().stream().collect(toImmutableSet()))
.removeContacts(remove.getContacts())
.addContacts(add.getContacts())
.setRegistrant(firstNonNull(change.getRegistrant(), domain.getRegistrant()))
.setAuthInfo(firstNonNull(change.getAuthInfo(), domain.getAuthInfo()));
if (!add.getNameservers().isEmpty()) {
domainBuilder.addNameservers(add.getNameservers().stream().collect(toImmutableSet()));
}
if (!remove.getNameservers().isEmpty()) {
domainBuilder.removeNameservers(remove.getNameservers().stream().collect(toImmutableSet()));
}
Optional<DomainUpdateSuperuserExtension> superuserExt =
eppInput.getSingleExtension(DomainUpdateSuperuserExtension.class);
if (superuserExt.isPresent()) {
@@ -108,7 +108,7 @@ public final class PollAckFlow implements TransactionalFlow {
// acked, then we return a special status code indicating that. Note that the query will
// include the message being acked.
int messageCount = tm().doTransactionless(() -> getPollMessageCount(registrarId, now));
int messageCount = tm().transact(() -> getPollMessageCount(registrarId, now));
if (messageCount <= 0) {
return responseBuilder.setResultFromCode(SUCCESS_WITH_NO_MESSAGES).build();
}
@@ -365,13 +365,13 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
@Override
public EppResource load(VKey<? extends EppResource> key) {
return replicaTm().doTransactionless(() -> replicaTm().loadByKey(key));
return replicaTm().transact(() -> replicaTm().loadByKey(key));
}
@Override
public Map<VKey<? extends EppResource>, EppResource> loadAll(
Iterable<? extends VKey<? extends EppResource>> keys) {
return replicaTm().doTransactionless(() -> replicaTm().loadByKeys(keys));
return replicaTm().transact(() -> replicaTm().loadByKeys(keys));
}
};
@@ -16,7 +16,6 @@ package google.registry.model;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.DateTimeUtils.START_OF_TIME;
@@ -67,7 +66,7 @@ public final class EppResourceUtils {
+ "AND deletionTime > :now";
// We have to use the native SQL query here because DomainHost table doesn't have its entity
// class so we cannot reference its property like domainHost.hostRepoId in a JPQL query.
// class, so we cannot reference its property like domainHost.hostRepoId in a JPQL query.
private static final String HOST_LINKED_DOMAIN_QUERY =
"SELECT d.repo_id FROM \"Domain\" d "
+ "JOIN \"DomainHost\" dh ON dh.domain_repo_id = d.repo_id "
@@ -260,7 +259,7 @@ public final class EppResourceUtils {
/**
* Rewinds an {@link EppResource} object to a given point in time.
*
* <p>This method costs nothing if {@code resource} is already current. Otherwise it needs to
* <p>This method costs nothing if {@code resource} is already current. Otherwise, it needs to
* perform a single fetch operation.
*
* <p><b>Warning:</b> A resource can only be rolled backwards in time, not forwards; therefore
@@ -292,7 +291,7 @@ public final class EppResourceUtils {
/**
* Rewinds an {@link EppResource} object to a given point in time.
*
* <p>This method costs nothing if {@code resource} is already current. Otherwise it returns an
* <p>This method costs nothing if {@code resource} is already current. Otherwise, it returns an
* async operation that performs a single fetch operation.
*
* @return an asynchronous operation returning resource at {@code timestamp} or {@code null} if
@@ -346,50 +345,37 @@ public final class EppResourceUtils {
"key must be either VKey<Contact> or VKey<Host>, but it is %s",
key);
boolean isContactKey = key.getKind().equals(Contact.class);
if (tm().isOfy()) {
com.googlecode.objectify.cmd.Query<Domain> query =
auditedOfy()
.load()
.type(Domain.class)
.filter(isContactKey ? "allContacts.contact" : "nsHosts", key.getOfyKey())
.filter("deletionTime >", now);
if (limit != null) {
query.limit(limit);
}
return query.keys().list().stream().map(Domain::createVKey).collect(toImmutableSet());
} else {
return tm().transact(
() -> {
Query query;
if (isContactKey) {
query =
jpaTm()
.query(CONTACT_LINKED_DOMAIN_QUERY, String.class)
.setParameter("fkRepoId", key)
.setParameter("now", now);
} else {
query =
jpaTm()
.getEntityManager()
.createNativeQuery(HOST_LINKED_DOMAIN_QUERY)
.setParameter("fkRepoId", key.getSqlKey())
.setParameter("now", now.toDate());
}
if (limit != null) {
query.setMaxResults(limit);
}
@SuppressWarnings("unchecked")
ImmutableSet<VKey<Domain>> domainKeySet =
(ImmutableSet<VKey<Domain>>)
query
.getResultStream()
.map(
repoId ->
Domain.createVKey(Key.create(Domain.class, (String) repoId)))
.collect(toImmutableSet());
return domainKeySet;
});
}
return tm().transact(
() -> {
Query query;
if (isContactKey) {
query =
jpaTm()
.query(CONTACT_LINKED_DOMAIN_QUERY, String.class)
.setParameter("fkRepoId", key)
.setParameter("now", now);
} else {
query =
jpaTm()
.getEntityManager()
.createNativeQuery(HOST_LINKED_DOMAIN_QUERY)
.setParameter("fkRepoId", key.getSqlKey())
.setParameter("now", now.toDate());
}
if (limit != null) {
query.setMaxResults(limit);
}
@SuppressWarnings("unchecked")
ImmutableSet<VKey<Domain>> domainKeySet =
(ImmutableSet<VKey<Domain>>)
query
.getResultStream()
.map(
repoId ->
Domain.createVKey(Key.create(Domain.class, (String) repoId)))
.collect(toImmutableSet());
return domainKeySet;
});
}
/**
@@ -17,58 +17,114 @@ package google.registry.model;
import static com.google.common.base.Preconditions.checkState;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.flogger.FluentLogger;
import google.registry.beam.common.RegistryPipelineWorkerInitializer;
import google.registry.config.RegistryEnvironment;
import google.registry.model.annotations.DeleteAfterMigration;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Supplier;
/**
* Allocates a globally unique {@link Long} number to use as an Ofy {@code @Id}.
* Allocates a {@link long} to use as a {@code @Id}, (part) of the primary SQL key for an entity.
*
* <p>In non-test, non-beam environments the Id is generated by Datastore, otherwise it's from an
* atomic long number that's incremented every time this method is called.
* <p>Normally, the ID is globally unique and allocated by Datastore. It is possible to override
* this behavior by providing an ID supplier, such as in unit tests, where a self-allocated ID based
* on a monotonically increasing atomic {@link long} is used. Such an ID supplier can also be used
* in other scenarios, such as in a Beam pipeline to get around the limitation of Beam's inability
* to use GAE SDK to access Datastore. The override should be used with great care lest it results
* in irreversible data corruption.
*
* @see #setIdSupplier(Supplier)
*/
@DeleteAfterMigration
public final class IdService {
/**
* A placeholder String passed into DatastoreService.allocateIds that ensures that all ids are
* initialized from the same id pool.
*/
private static final String APP_WIDE_ALLOCATION_KIND = "common";
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private IdService() {}
private static Supplier<Long> idSupplier =
RegistryEnvironment.UNITTEST.equals(RegistryEnvironment.get())
? SelfAllocatedIdSupplier.getInstance()
: DatastoreIdSupplier.getInstance();
/**
* Counts of used ids for use in unit tests or Beam.
* Provides a {@link Supplier} of ID that overrides the default.
*
* <p>Note that one should only use self-allocate Ids in Beam for entities whose Ids are not
* important and are not persisted back to the database, i. e. nowhere the uniqueness of the ID is
* required.
* <p>Currently, the only use case for an override is in the Beam pipeline, where access to
* Datastore is not possible through the App Engine API. As such, the setter explicitly checks if
* the runtime is Beam.
*
* <p>Because the provided supplier is not guaranteed to be globally unique and compatible with
* existing IDs in the database, one should proceed with great care. It is safe to use an
* arbitrary supplier when the resulting IDs are not significant and not persisted back to the
* database, i.e. the IDs are only required by the {@link Buildable} contract but are not used in
* any meaningful way. One example is the RDE pipeline where we project EPP resource entities from
* history entries to watermark time, which are then marshalled into XML elements in the RDE
* deposits, where the IDs are omitted.
*/
private static final AtomicLong nextSelfAllocatedId = new AtomicLong(1); // ids cannot be zero
private static final boolean isSelfAllocated() {
return RegistryEnvironment.UNITTEST.equals(RegistryEnvironment.get())
|| "true".equals(System.getProperty(RegistryPipelineWorkerInitializer.PROPERTY, "false"));
public static void setIdSupplier(Supplier<Long> idSupplier) {
checkState(
"true".equals(System.getProperty(RegistryPipelineWorkerInitializer.PROPERTY, "false")),
"Can only set ID supplier in a Beam pipeline");
logger.atWarning().log("Using ID supplier override!");
IdService.idSupplier = idSupplier;
}
/** Allocates an id. */
// TODO(b/201547855): Find a way to allocate a unique ID without datastore.
public static long allocateId() {
return isSelfAllocated()
? nextSelfAllocatedId.getAndIncrement()
: DatastoreServiceFactory.getDatastoreService()
.allocateIds(APP_WIDE_ALLOCATION_KIND, 1)
.iterator()
.next()
.getId();
return idSupplier.get();
}
/** Resets the global self-allocated id counter (i.e. sets the next id to 1). */
@VisibleForTesting
public static void resetSelfAllocatedId() {
checkState(
isSelfAllocated(), "Can only call resetSelfAllocatedId() in unit tests or Beam pipelines");
nextSelfAllocatedId.set(1); // ids cannot be zero
// TODO(b/201547855): Find a way to allocate a unique ID without datastore.
private static class DatastoreIdSupplier implements Supplier<Long> {
private static final DatastoreIdSupplier INSTANCE = new DatastoreIdSupplier();
/**
* A placeholder String passed into {@code DatastoreService.allocateIds} that ensures that all
* IDs are initialized from the same ID pool.
*/
private static final String APP_WIDE_ALLOCATION_KIND = "common";
public static DatastoreIdSupplier getInstance() {
return INSTANCE;
}
@Override
public Long get() {
return DatastoreServiceFactory.getDatastoreService()
.allocateIds(APP_WIDE_ALLOCATION_KIND, 1)
.iterator()
.next()
.getId();
}
}
/**
* An ID supplier that allocates an ID from a monotonically increasing atomic {@link long}.
*
* <p>The generated IDs are only unique within the same JVM. It is not suitable for production use
* unless in cases the IDs are not significant.
*/
public static class SelfAllocatedIdSupplier implements Supplier<Long> {
private static final SelfAllocatedIdSupplier INSTANCE = new SelfAllocatedIdSupplier();
/** Counts of used ids for self allocating IDs. */
private static final AtomicLong nextSelfAllocatedId = new AtomicLong(1); // ids cannot be zero
public static SelfAllocatedIdSupplier getInstance() {
return INSTANCE;
}
@Override
public Long get() {
return nextSelfAllocatedId.getAndIncrement();
}
public void reset() {
nextSelfAllocatedId.set(1);
}
}
}
@@ -14,17 +14,13 @@
package google.registry.model;
import static com.google.common.collect.Iterables.transform;
import static com.google.common.collect.Maps.transformValues;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static java.lang.annotation.ElementType.FIELD;
import static java.lang.annotation.RetentionPolicy.RUNTIME;
import static java.util.stream.Collectors.toCollection;
import static java.util.stream.Collectors.toList;
import com.google.common.base.Joiner;
import com.google.common.collect.Maps;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Ignore;
import google.registry.persistence.VKey;
@@ -56,15 +52,6 @@ public abstract class ImmutableObject implements Cloneable {
@Target(FIELD)
public @interface DoNotHydrate {}
/**
* Indicates that the field should be ignored when comparing an object in the datastore to the
* corresponding object in Cloud SQL.
*/
@Documented
@Retention(RUNTIME)
@Target(FIELD)
public @interface DoNotCompare {}
/**
* Indicates that the field stores a null value to indicate an empty set. This is also used in
* object comparison.
@@ -105,7 +92,7 @@ public abstract class ImmutableObject implements Cloneable {
*/
protected Map<Field, Object> getSignificantFields() {
// Can't use streams or ImmutableMap because we can have null values.
Map<Field, Object> result = new LinkedHashMap();
Map<Field, Object> result = new LinkedHashMap<>();
for (Map.Entry<Field, Object> entry : ModelUtils.getFieldValues(this).entrySet()) {
if (!entry.getKey().isAnnotationPresent(Insignificant.class)) {
result.put(entry.getKey(), entry.getValue());
@@ -190,15 +177,15 @@ public abstract class ImmutableObject implements Cloneable {
/** Helper function to recursively hydrate an ImmutableObject. */
private static Object hydrate(Object value) {
if (value instanceof Key) {
if (tm().isOfy()) {
return hydrate(auditedOfy().load().key((Key<?>) value).now());
}
return value;
} else if (value instanceof Map) {
}
if (value instanceof Map) {
return transformValues((Map<?, ?>) value, ImmutableObject::hydrate);
} else if (value instanceof Collection) {
return transform((Collection<?>) value, ImmutableObject::hydrate);
} else if (value instanceof ImmutableObject) {
}
if (value instanceof Collection) {
return ((Collection<?>) value).stream().map(ImmutableObject::hydrate);
}
if (value instanceof ImmutableObject) {
return ((ImmutableObject) value).toHydratedString();
}
return value;
@@ -220,7 +207,7 @@ public abstract class ImmutableObject implements Cloneable {
}
return result;
} else if (o instanceof Map) {
return Maps.transformValues((Map<?, ?>) o, ImmutableObject::toMapRecursive);
return transformValues((Map<?, ?>) o, ImmutableObject::toMapRecursive);
} else if (o instanceof Set) {
return ((Set<?>) o)
.stream()
@@ -38,7 +38,7 @@ import google.registry.model.domain.rgp.GracePeriodStatus;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
import google.registry.persistence.VKey;
import google.registry.persistence.WithLongVKey;
import google.registry.persistence.WithVKey;
import google.registry.persistence.converter.JodaMoneyType;
import java.util.Optional;
import java.util.Set;
@@ -295,7 +295,7 @@ public abstract class BillingEvent extends ImmutableObject
@Index(columnList = "cancellation_matching_billing_recurrence_id")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_event_id"))
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class OneTime extends BillingEvent {
/** The billable value. */
@@ -473,7 +473,7 @@ public abstract class BillingEvent extends ImmutableObject
@Index(columnList = "recurrence_time_of_year")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_recurrence_id"))
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class Recurring extends BillingEvent {
/**
@@ -606,7 +606,7 @@ public abstract class BillingEvent extends ImmutableObject
@Index(columnList = "billing_recurrence_id")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_cancellation_id"))
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class Cancellation extends BillingEvent {
/** The billing time of the charge that is being cancelled. */
@@ -17,7 +17,7 @@ package google.registry.model.bulkquery;
import google.registry.model.domain.Domain;
import google.registry.model.domain.DomainBase;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Entity;
@@ -31,7 +31,7 @@ import javax.persistence.Entity;
* <p>Please refer to {@link BulkQueryEntities} for more information.
*/
@Entity(name = "Domain")
@WithStringVKey
@WithVKey(String.class)
@Access(AccessType.FIELD)
public class DomainLite extends DomainBase {
@@ -18,7 +18,7 @@ import google.registry.model.EppResource.ForeignKeyedEppResource;
import google.registry.model.annotations.ExternalMessagingName;
import google.registry.model.annotations.ReportedOn;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import javax.persistence.Access;
import javax.persistence.AccessType;
import javax.persistence.Entity;
@@ -45,7 +45,7 @@ import org.joda.time.DateTime;
@Index(columnList = "searchName")
})
@ExternalMessagingName("contact")
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@Access(AccessType.FIELD)
public class Contact extends ContactBase implements ForeignKeyedEppResource {
@@ -60,7 +60,7 @@ public class ContactHistory extends HistoryEntry implements UnsafeSerializable {
// Store ContactBase instead of Contact so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@DoNotCompare @Nullable ContactBase contactBase;
@Nullable ContactBase contactBase;
@Id
@Access(AccessType.PROPERTY)
@@ -22,7 +22,7 @@ import google.registry.model.annotations.ReportedOn;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.host.Host;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import java.util.Set;
import javax.persistence.Access;
import javax.persistence.AccessType;
@@ -66,7 +66,7 @@ import org.joda.time.DateTime;
@Index(columnList = "transfer_billing_event_id"),
@Index(columnList = "transfer_billing_recurrence_id")
})
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@ExternalMessagingName("domain")
@Access(AccessType.FIELD)
public class Domain extends DomainBase implements ForeignKeyedEppResource {
@@ -55,6 +55,7 @@ import google.registry.model.domain.launch.LaunchNotice;
import google.registry.model.domain.rgp.GracePeriodStatus;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.domain.token.AllocationToken.TokenType;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.host.Host;
import google.registry.model.poll.PollMessage;
@@ -919,6 +920,21 @@ public class DomainBase extends EppResource
}
public B setCurrentPackageToken(@Nullable VKey<AllocationToken> currentPackageToken) {
if (currentPackageToken == null) {
getInstance().currentPackageToken = currentPackageToken;
return thisCastToDerived();
}
AllocationToken token =
tm().transact(() -> tm().loadByKeyIfPresent(currentPackageToken))
.orElseThrow(
() ->
new IllegalArgumentException(
String.format(
"The package token %s does not exist",
currentPackageToken.getSqlKey())));
checkArgument(
token.getTokenType().equals(TokenType.PACKAGE),
"The currentPackageToken must have a PACKAGE TokenType");
getInstance().currentPackageToken = currentPackageToken;
return thisCastToDerived();
}
@@ -83,7 +83,7 @@ public class DomainHistory extends HistoryEntry {
// Store DomainBase instead of Domain so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@DoNotCompare @Nullable DomainBase domainBase;
@Nullable DomainBase domainBase;
@Id
@Access(AccessType.PROPERTY)
@@ -102,7 +102,6 @@ public class DomainHistory extends HistoryEntry {
// We could have reused domainBase.nsHosts here, but Hibernate throws a weird exception after
// we change to use a composite primary key.
// TODO(b/166776754): Investigate if we can reuse domainBase.nsHosts for storing host keys.
@DoNotCompare
@ElementCollection
@JoinTable(
name = "DomainHistoryHost",
@@ -116,7 +115,6 @@ public class DomainHistory extends HistoryEntry {
@Column(name = "host_repo_id")
Set<VKey<Host>> nsHosts;
@DoNotCompare
@OneToMany(
cascade = {CascadeType.ALL},
fetch = FetchType.EAGER,
@@ -137,7 +135,6 @@ public class DomainHistory extends HistoryEntry {
@Ignore
Set<DomainDsDataHistory> dsDataHistories = new HashSet<>();
@DoNotCompare
@OneToMany(
cascade = {CascadeType.ALL},
fetch = FetchType.EAGER,
@@ -41,7 +41,7 @@ import google.registry.model.common.TimedTransitionProperty;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.DomainHistoryVKey;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import java.util.Optional;
import java.util.Set;
import javax.annotation.Nullable;
@@ -58,7 +58,7 @@ import org.joda.time.DateTime;
/** An entity representing an allocation token. */
@Entity
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@Table(
indexes = {
@Index(columnList = "token", name = "allocation_token_token_idx", unique = true),
@@ -300,6 +300,9 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
!getInstance().tokenType.equals(TokenType.PACKAGE)
|| getInstance().renewalPriceBehavior.equals(RenewalPriceBehavior.SPECIFIED),
"Package tokens must have renewalPriceBehavior set to SPECIFIED");
checkArgument(
!getInstance().tokenType.equals(TokenType.PACKAGE) || !getInstance().discountPremiums,
"Package tokens cannot discount premium names");
checkArgument(
getInstance().domainName == null || TokenType.SINGLE_USE.equals(getInstance().tokenType),
"Domain name can only be specified for SINGLE_USE tokens");
@@ -19,7 +19,7 @@ import google.registry.model.EppResource.ForeignKeyedEppResource;
import google.registry.model.annotations.ExternalMessagingName;
import google.registry.model.annotations.ReportedOn;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.persistence.WithVKey;
import javax.persistence.Access;
import javax.persistence.AccessType;
@@ -51,7 +51,7 @@ import javax.persistence.AccessType;
@javax.persistence.Index(columnList = "currentSponsorRegistrarId")
})
@ExternalMessagingName("host")
@WithStringVKey(compositeKey = true)
@WithVKey(String.class)
@Access(AccessType.FIELD) // otherwise it'll use the default if the repoId (property)
public class Host extends HostBase implements ForeignKeyedEppResource {
@@ -61,7 +61,7 @@ public class HostHistory extends HistoryEntry implements UnsafeSerializable {
// Store HostBase instead of Host so we don't pick up its @Id
// Nullable for the sake of pre-Registry-3.0 history objects
@DoNotCompare @Nullable HostBase hostBase;
@Nullable HostBase hostBase;
@Id
@Access(AccessType.PROPERTY)
@@ -56,11 +56,6 @@ public class ObjectifyService {
/** A singleton instance of our Ofy wrapper. */
private static final Ofy OFY = new Ofy(null);
/** Returns the singleton {@link Ofy} instance. */
public static Ofy ofy() {
return OFY;
}
/**
* Returns the singleton {@link Ofy} instance, signifying that the caller has been audited for the
* Registry 3.0 conversion.
@@ -46,7 +46,7 @@ import google.registry.model.transfer.TransferResponse;
import google.registry.model.transfer.TransferResponse.ContactTransferResponse;
import google.registry.model.transfer.TransferResponse.DomainTransferResponse;
import google.registry.persistence.VKey;
import google.registry.persistence.WithLongVKey;
import google.registry.persistence.WithVKey;
import google.registry.util.NullIgnoringCollectionBuilder;
import java.util.Optional;
import javax.persistence.AttributeOverride;
@@ -342,7 +342,7 @@ public abstract class PollMessage extends ImmutableObject
*/
@Entity
@DiscriminatorValue("ONE_TIME")
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class OneTime extends PollMessage {
@Embedded
@@ -544,7 +544,7 @@ public abstract class PollMessage extends ImmutableObject
*/
@Entity
@DiscriminatorValue("AUTORENEW")
@WithLongVKey(compositeKey = true)
@WithVKey(Long.class)
public static class Autorenew extends PollMessage {
/** The target id of the autorenew event. */
@@ -210,9 +210,6 @@ public class Lock extends ImmutableObject implements Serializable {
RequestStatusChecker requestStatusChecker,
boolean checkThreadRunning) {
String scope = tld != null ? tld : GLOBAL;
// It's important to use transactNew rather than transact, because a Lock can be used to control
// access to resources like GCS that can't be transactionally rolled back. Therefore, the lock
// must be definitively acquired before it is used, even when called inside another transaction.
Supplier<AcquireResult> lockAcquirer =
() -> {
DateTime now = jpaTm().getTransactionTime();
@@ -22,22 +22,21 @@ import static com.google.common.base.Strings.emptyToNull;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.Maps.filterValues;
import static google.registry.model.CacheUtils.memoizeWithShortExpiration;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.CollectionUtils.entriesToImmutableMap;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import com.google.common.base.Joiner;
import com.google.common.base.Supplier;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.common.collect.Streams;
import com.google.common.net.InternetDomainName;
import com.googlecode.objectify.Key;
import google.registry.model.tld.Registry.TldType;
import google.registry.util.DomainNameUtils;
import java.util.Optional;
import java.util.function.Supplier;
import java.util.stream.Stream;
import javax.persistence.EntityManager;
@@ -58,32 +57,17 @@ public final class Registries {
private static Supplier<ImmutableMap<String, TldType>> createFreshCache() {
return memoizeWithShortExpiration(
() ->
tm().doTransactionless(
tm().transact(
() -> {
if (tm().isOfy()) {
ImmutableSet<String> tlds =
auditedOfy()
.load()
.type(Registry.class)
.keys()
.list()
.stream()
.map(Key::getName)
.collect(toImmutableSet());
return Registry.get(tlds).stream()
.map(e -> Maps.immutableEntry(e.getTldStr(), e.getTldType()))
.collect(entriesToImmutableMap());
} else {
EntityManager entityManager = jpaTm().getEntityManager();
Stream<?> resultStream =
entityManager
.createQuery("SELECT tldStr, tldType FROM Tld")
.getResultStream();
return resultStream
.map(e -> ((Object[]) e))
.map(e -> Maps.immutableEntry((String) e[0], ((TldType) e[1])))
.collect(entriesToImmutableMap());
}
EntityManager entityManager = jpaTm().getEntityManager();
Stream<?> resultStream =
entityManager
.createQuery("SELECT tldStr, tldType FROM Tld")
.getResultStream();
return resultStream
.map(e -> ((Object[]) e))
.map(e -> Maps.immutableEntry((String) e[0], ((TldType) e[1])))
.collect(entriesToImmutableMap());
}));
}
@@ -143,8 +127,7 @@ public final class Registries {
*
* <p><b>Note:</b> This routine will only work on names under TLDs for which this registry is
* authoritative. To extract TLDs from domains (not hosts) that other registries control, use
* {@link google.registry.util.DomainNameUtils#getTldFromDomainName(String)
* DomainNameUtils#getTldFromDomainName}.
* {@link DomainNameUtils#getTldFromDomainName(String) DomainNameUtils#getTldFromDomainName}.
*
* @param domainName domain name or host name (but not TLD) under an authoritative TLD
* @return TLD or absent if {@code domainName} has no labels under an authoritative TLD
@@ -14,12 +14,10 @@
package google.registry.persistence;
import com.google.common.base.Predicates;
import com.google.common.collect.ImmutableSet;
import java.sql.SQLException;
import java.util.function.Predicate;
import javax.persistence.OptimisticLockException;
import org.hibernate.exception.JDBCConnectionException;
/** Helpers for identifying retriable database operations. */
public final class JpaRetries {
@@ -35,18 +33,11 @@ public final class JpaRetries {
);
private static final Predicate<Throwable> RETRIABLE_TXN_PREDICATE =
Predicates.or(
OptimisticLockException.class::isInstance,
e ->
e instanceof SQLException
&& RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
private static final Predicate<Throwable> RETRIABLE_QUERY_PREDICATE =
Predicates.or(
JDBCConnectionException.class::isInstance,
e ->
e instanceof SQLException
&& RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
((Predicate<Throwable>) OptimisticLockException.class::isInstance)
.or(
e ->
e instanceof SQLException
&& RETRIABLE_TXN_SQL_STATE.contains(((SQLException) e).getSQLState()));
public static boolean isFailedTxnRetriable(Throwable throwable) {
Throwable t = throwable;
@@ -58,16 +49,4 @@ public final class JpaRetries {
}
return false;
}
public static boolean isFailedQueryRetriable(Throwable throwable) {
// TODO(weiminyu): check for more error codes.
Throwable t = throwable;
while (t != null) {
if (RETRIABLE_QUERY_PREDICATE.test(t)) {
return true;
}
t = t.getCause();
}
return false;
}
}
@@ -278,7 +278,7 @@ public abstract class PersistenceModule {
replicaInstanceConnectionName.ifPresent(
name -> overrides.put(HIKARI_DS_CLOUD_SQL_INSTANCE, name));
overrides.put(
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_READ_COMMITTED.name());
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_REPEATABLE_READ.name());
return new JpaTransactionManagerImpl(create(overrides), clock);
}
@@ -294,7 +294,7 @@ public abstract class PersistenceModule {
replicaInstanceConnectionName.ifPresent(
name -> overrides.put(HIKARI_DS_CLOUD_SQL_INSTANCE, name));
overrides.put(
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_READ_COMMITTED.name());
Environment.ISOLATION, TransactionIsolationLevel.TRANSACTION_REPEATABLE_READ.name());
return new JpaTransactionManagerImpl(create(overrides), clock);
}
@@ -1,42 +0,0 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;
import javax.persistence.AttributeConverter;
import javax.persistence.Entity;
/**
* Annotation for {@link Entity} which id is string type and needs an {@link AttributeConverter} for
* its VKey.
*/
@Target({ElementType.TYPE})
public @interface WithStringVKey {
/**
* Sets the suffix of the class name for the {@link AttributeConverter} generated by
* StringVKeyProcessor. If not set, the suffix will be the type name of the VKey. Note that the
* class name will be "VKeyConverter_" concatenated with the suffix.
*/
String classNameSuffix() default "";
/**
* Set to true if this is a composite vkey.
*
* <p>For composite VKeys, we don't attempt to define an objectify key when loading from SQL: the
* enclosing class has to take care of that.
*/
boolean compositeKey() default false;
}
@@ -1,4 +1,4 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
@@ -14,29 +14,22 @@
package google.registry.persistence;
import java.io.Serializable;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;
import javax.persistence.AttributeConverter;
import javax.persistence.Entity;
/**
* Annotation for {@link Entity} which id is long type and needs an {@link AttributeConverter} for
* its VKey.
* Annotation for {@link Entity} that can be saved as a foreign key in the form of a {@link VKey} in
* another table.
*
* <p>A {@link AttributeConverter} named {@code VKeyConverter_[EntityClassSimpleName]} will be
* automatically generated by {@code google.registry.processors.VKeyProcessor}, this class must be
* manually added to {@code persistence.xml} in order for it to be picked up by Hibernate.
*/
@Target({ElementType.TYPE})
public @interface WithLongVKey {
/**
* Sets the suffix of the class name for the {@link AttributeConverter} generated by
* LongVKeyProcessor. If not set, the suffix will be the type name of the VKey. Note that the
* class name will be "VKeyConverter_" concatenated with the suffix.
*/
String classNameSuffix() default "";
/**
* Set to true if this is a composite vkey.
*
* <p>For composite VKeys, we don't attempt to define an objectify key when loading from SQL: the
* enclosing class has to take care of that.
*/
boolean compositeKey() default false;
public @interface WithVKey {
/** The type of the SQL primary ID of the entity that is saved in the {@link VKey} */
Class<? extends Serializable> value();
}
@@ -14,20 +14,34 @@
package google.registry.persistence.converter;
import com.googlecode.objectify.Key;
import google.registry.persistence.VKey;
import java.io.Serializable;
import javax.annotation.Nullable;
import javax.persistence.AttributeConverter;
/** Converts VKey to a string or long column. */
/**
* Converts {@link VKey} to/from a type that can be directly stored in the database.
*
* <p>Typically the converted type is {@link String} or {@link Long}.
*/
public abstract class VKeyConverter<T, C extends Serializable>
implements AttributeConverter<VKey<? extends T>, C> {
@Override
@Nullable
@SuppressWarnings("unchecked")
public C convertToDatabaseColumn(@Nullable VKey<? extends T> attribute) {
return attribute == null ? null : (C) attribute.getSqlKey();
if (attribute == null) {
return null;
}
try {
return getKeyClass().cast(attribute.getSqlKey());
} catch (ClassCastException e) {
throw new RuntimeException(
String.format(
"Cannot cast SQL key %s of type %s to type %s",
attribute.getSqlKey(), attribute.getSqlKey().getClass(), getKeyClass()),
e);
}
}
@Override
@@ -36,27 +50,12 @@ public abstract class VKeyConverter<T, C extends Serializable>
if (dbData == null) {
return null;
}
Class<T> clazz = getAttributeClass();
Key<T> ofyKey;
if (!hasCompositeOfyKey()) {
// If this isn't a composite key, we can create the Ofy key from the SQL key.
ofyKey =
dbData instanceof String
? Key.create(clazz, (String) dbData)
: Key.create(clazz, (Long) dbData);
return VKey.create(clazz, dbData, ofyKey);
} else {
// We don't know how to create the Ofy key and probably don't have everything necessary to do
// it anyway, so just create an asymmetric key - the containing object will have to convert it
// into a symmetric key.
return VKey.createSql(clazz, dbData);
}
return VKey.createSql(getEntityClass(), dbData);
}
protected boolean hasCompositeOfyKey() {
return false;
}
/** Returns the class of the entity that the VKey represents. */
protected abstract Class<T> getEntityClass();
/** Returns the class of the attribute. */
protected abstract Class<T> getAttributeClass();
/** Returns the class of the key that the VKey holds. */
protected abstract Class<C> getKeyClass();
}
@@ -208,43 +208,6 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
});
}
// TODO(b/177674699): Remove all transactNew methods as they are same as transact after the
// database migration.
@Override
public <T> T transactNew(Supplier<T> work) {
return transact(work);
}
@Override
public void transactNew(Runnable work) {
transact(work);
}
// For now, read-only transactions and "transactNew" methods only create (or use existing)
// standard transactions. Attempting to use a read-only transaction can break larger transactions
// (if we were already in one) so we don't set read-only mode.
//
// TODO(gbrodman): If necessary, implement transactNew and readOnly transactions using Postgres
// savepoints, see https://www.postgresql.org/docs/8.1/sql-savepoint.html
@Override
public <T> T transactNewReadOnly(Supplier<T> work) {
return retrier.callWithRetry(() -> transact(work), JpaRetries::isFailedQueryRetriable);
}
@Override
public void transactNewReadOnly(Runnable work) {
transactNewReadOnly(
() -> {
work.run();
return null;
});
}
@Override
public <T> T doTransactionless(Supplier<T> work) {
return retrier.callWithRetry(() -> transact(work), JpaRetries::isFailedQueryRetriable);
}
@Override
public DateTime getTransactionTime() {
assertInTransaction();
@@ -493,16 +456,6 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
return new JpaQueryComposerImpl<>(entity);
}
@Override
public void clearSessionCache() {
// This is an intended no-op method as there is no session cache in Postgresql.
}
@Override
public boolean isOfy() {
return false;
}
@Override
public <T> void assertDelete(VKey<T> key) {
if (internalDelete(key) != 1) {
@@ -516,8 +469,8 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
}
private static class EntityId {
private String name;
private Object value;
private final String name;
private final Object value;
private EntityId(String name, Object value) {
this.name = name;
@@ -52,42 +52,6 @@ public interface TransactionManager {
/** Executes the work in a transaction. */
void transact(Runnable work);
/**
* Pauses the current transaction (if any), executes the work in a new transaction and returns the
* result.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
<T> T transactNew(Supplier<T> work);
/**
* Pauses the current transaction (if any) and executes the work in a new transaction.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
void transactNew(Runnable work);
/**
* Executes the work in a read-only transaction and returns the result.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
<R> R transactNewReadOnly(Supplier<R> work);
/**
* Executes the work in a read-only transaction.
*
* <p>Note that this function is kept for backward compatibility. We will review the use case
* later when adding the cloud sql implementation.
*/
void transactNewReadOnly(Runnable work);
/** Executes the work in a transactionless context. */
<R> R doTransactionless(Supplier<R> work);
/** Returns the time associated with the start of this particular transaction attempt. */
DateTime getTransactionTime();
@@ -210,10 +174,4 @@ public interface TransactionManager {
/** Returns a QueryComposer which can be used to perform queries against the current database. */
<T> QueryComposer<T> createQueryComposer(Class<T> entity);
/** Clears the session cache if the underlying database is Datastore, otherwise it is a no-op. */
void clearSessionCache();
/** Returns true if the transaction manager is DatastoreTransactionManager, false otherwise. */
boolean isOfy();
}
@@ -109,7 +109,7 @@ public final class TransactionManagerFactory {
* however, this will be a reference to the read-only replica database if one is configured.
*/
public static TransactionManager replicaTm() {
return tm().isOfy() ? tm() : replicaJpaTm();
return replicaJpaTm();
}
/** Sets the return of {@link #jpaTm()} to the given instance of {@link JpaTransactionManager}. */
@@ -118,7 +118,7 @@ public final class TransactionManagerFactory {
checkState(
RegistryEnvironment.get().equals(RegistryEnvironment.UNITTEST)
|| RegistryToolEnvironment.get() != null,
"setJpamTm() should only be called by tools and tests.");
"setJpaTm() should only be called by tools and tests.");
jpaTm = Suppliers.memoize(jpaTmSupplier::get);
}
@@ -19,7 +19,7 @@ import com.google.cloud.secretmanager.v1.SecretManagerServiceClient;
import com.google.cloud.secretmanager.v1.SecretManagerServiceSettings;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule.DefaultCredential;
import google.registry.config.CredentialModule.ApplicationDefaultCredential;
import google.registry.config.RegistryConfig.Config;
import google.registry.util.GoogleCredentialsBundle;
import google.registry.util.Retrier;
@@ -33,7 +33,7 @@ public abstract class SecretManagerModule {
@Provides
@Singleton
static SecretManagerServiceSettings provideSecretManagerSetting(
@DefaultCredential GoogleCredentialsBundle credentialsBundle) {
@ApplicationDefaultCredential GoogleCredentialsBundle credentialsBundle) {
try {
return SecretManagerServiceSettings.newBuilder()
.setCredentialsProvider(() -> credentialsBundle.getGoogleCredentials())
@@ -16,9 +16,7 @@ package google.registry.rdap;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.EppResourceUtils.loadByForeignKeyCached;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.request.Action.Method.GET;
import static google.registry.request.Action.Method.HEAD;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
@@ -30,11 +28,9 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.ImmutableSortedSet;
import com.google.common.collect.Iterables;
import com.google.common.collect.Streams;
import com.google.common.flogger.FluentLogger;
import com.google.common.net.InetAddresses;
import com.google.common.primitives.Booleans;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.ForeignKeyUtils;
import google.registry.model.domain.Domain;
import google.registry.model.host.Host;
@@ -58,7 +54,6 @@ import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import javax.inject.Inject;
import javax.persistence.criteria.CriteriaBuilder;
import org.hibernate.Hibernate;
@@ -207,49 +202,31 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
// keys, because then we have an index on TLD if we need it.
int querySizeLimit = RESULT_SET_SIZE_SCALING_FACTOR * rdapResultSetMaxSize;
RdapResultSet<Domain> resultSet;
if (tm().isOfy()) {
Query<Domain> query =
auditedOfy()
.load()
.type(Domain.class)
.filter("domainName <", partialStringQuery.getNextInitialString())
.filter("domainName >=", partialStringQuery.getInitialString());
if (cursorString.isPresent()) {
query = query.filter("domainName >", cursorString.get());
}
if (partialStringQuery.getSuffix() != null) {
query = query.filter("tld", partialStringQuery.getSuffix());
}
query = query.limit(querySizeLimit);
// Always check for visibility, because we couldn't look at the deletionTime in the query.
resultSet = getMatchingResources(query, true, querySizeLimit);
} else {
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaBuilder criteriaBuilder =
replicaJpaTm().getEntityManager().getCriteriaBuilder();
CriteriaQueryBuilder<Domain> queryBuilder =
CriteriaQueryBuilder.create(replicaJpaTm(), Domain.class)
.where(
"domainName",
criteriaBuilder::like,
String.format("%s%%", partialStringQuery.getInitialString()))
.orderByAsc("domainName");
if (cursorString.isPresent()) {
queryBuilder =
queryBuilder.where(
"domainName", criteriaBuilder::greaterThan, cursorString.get());
}
if (partialStringQuery.getSuffix() != null) {
queryBuilder =
queryBuilder.where(
"tld", criteriaBuilder::equal, partialStringQuery.getSuffix());
}
return getMatchingResourcesSql(queryBuilder, true, querySizeLimit);
});
}
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaBuilder criteriaBuilder =
replicaJpaTm().getEntityManager().getCriteriaBuilder();
CriteriaQueryBuilder<Domain> queryBuilder =
CriteriaQueryBuilder.create(replicaJpaTm(), Domain.class)
.where(
"domainName",
criteriaBuilder::like,
String.format("%s%%", partialStringQuery.getInitialString()))
.orderByAsc("domainName");
if (cursorString.isPresent()) {
queryBuilder =
queryBuilder.where(
"domainName", criteriaBuilder::greaterThan, cursorString.get());
}
if (partialStringQuery.getSuffix() != null) {
queryBuilder =
queryBuilder.where(
"tld", criteriaBuilder::equal, partialStringQuery.getSuffix());
}
return getMatchingResources(queryBuilder, true, querySizeLimit);
});
return makeSearchResults(resultSet);
}
@@ -261,30 +238,21 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
// pending deletes.
int querySizeLimit = RESULT_SET_SIZE_SCALING_FACTOR * rdapResultSetMaxSize;
RdapResultSet<Domain> resultSet;
if (tm().isOfy()) {
Query<Domain> query = auditedOfy().load().type(Domain.class).filter("tld", tld);
if (cursorString.isPresent()) {
query = query.filter("domainName >", cursorString.get());
}
query = query.order("domainName").limit(querySizeLimit);
resultSet = getMatchingResources(query, true, querySizeLimit);
} else {
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Domain> builder =
queryItemsSql(
Domain.class,
"tld",
tld,
Optional.of("domainName"),
cursorString,
DeletedItemHandling.INCLUDE)
.orderByAsc("domainName");
return getMatchingResourcesSql(builder, true, querySizeLimit);
});
}
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Domain> builder =
queryItems(
Domain.class,
"tld",
tld,
Optional.of("domainName"),
cursorString,
DeletedItemHandling.INCLUDE)
.orderByAsc("domainName");
return getMatchingResources(builder, true, querySizeLimit);
});
return makeSearchResults(resultSet);
}
@@ -337,46 +305,29 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
// incomplete result set if a search asks for something like "ns*", but we need to enforce a
// limit in order to avoid arbitrarily long-running queries.
Optional<String> desiredRegistrar = getDesiredRegistrar();
if (tm().isOfy()) {
Query<Host> query =
queryItems(
Host.class,
"hostName",
partialStringQuery,
Optional.empty(),
DeletedItemHandling.EXCLUDE,
maxNameserversInFirstStage);
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
return StreamSupport.stream(query.keys().spliterator(), false)
.map(VKey::from)
.collect(toImmutableSet());
} else {
return replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Host> builder =
queryItemsSql(
Host.class,
"hostName",
partialStringQuery,
Optional.empty(),
DeletedItemHandling.EXCLUDE);
if (desiredRegistrar.isPresent()) {
builder =
builder.where(
"currentSponsorClientId",
replicaJpaTm().getEntityManager().getCriteriaBuilder()::equal,
desiredRegistrar.get());
}
return getMatchingResourcesSql(builder, true, maxNameserversInFirstStage)
.resources()
.stream()
.map(Host::createVKey)
.collect(toImmutableSet());
});
}
return replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Host> builder =
queryItems(
Host.class,
"hostName",
partialStringQuery,
Optional.empty(),
DeletedItemHandling.EXCLUDE);
if (desiredRegistrar.isPresent()) {
builder =
builder.where(
"currentSponsorClientId",
replicaJpaTm().getEntityManager().getCriteriaBuilder()::equal,
desiredRegistrar.get());
}
return getMatchingResources(builder, true, maxNameserversInFirstStage)
.resources()
.stream()
.map(Host::createVKey)
.collect(toImmutableSet());
});
}
/** Assembles a list of {@link Host} keys by name when the pattern has no wildcard. */
@@ -471,24 +422,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
private DomainSearchResponse searchByNameserverIp(final InetAddress inetAddress) {
Optional<String> desiredRegistrar = getDesiredRegistrar();
ImmutableSet<VKey<Host>> hostKeys;
if (tm().isOfy()) {
Query<Host> query =
queryItems(
Host.class,
"inetAddresses",
inetAddress.getHostAddress(),
Optional.empty(),
Optional.empty(),
DeletedItemHandling.EXCLUDE,
maxNameserversInFirstStage);
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
hostKeys =
StreamSupport.stream(query.keys().spliterator(), false)
.map(VKey::from)
.collect(toImmutableSet());
} else {
// Hibernate does not allow us to query @Converted array fields directly, either
// in the CriteriaQuery or the raw text format. However, Postgres does -- so we
// use native queries to find hosts where any of the inetAddresses match.
@@ -520,7 +453,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
.map(repoId -> VKey.create(Host.class, repoId))
.collect(toImmutableSet());
});
}
return searchByNameserverRefs(hostKeys);
}
@@ -543,27 +475,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
int numHostKeysSearched = 0;
for (List<VKey<Host>> chunk : Iterables.partition(hostKeys, 30)) {
numHostKeysSearched += chunk.size();
if (tm().isOfy()) {
Query<Domain> query =
auditedOfy()
.load()
.type(Domain.class)
.filter(
"nsHosts in", chunk.stream().map(VKey::getOfyKey).collect(toImmutableSet()));
if (!shouldIncludeDeleted()) {
query = query.filter("deletionTime >", getRequestTime());
// If we are not performing an inequality query, we can filter on the cursor in the query.
// Otherwise, we will need to filter the results afterward.
} else if (cursorString.isPresent()) {
query = query.filter("domainName >", cursorString.get());
}
Stream<Domain> stream = Streams.stream(query).filter(this::isAuthorized);
if (cursorString.isPresent()) {
stream =
stream.filter(domain -> (domain.getDomainName().compareTo(cursorString.get()) > 0));
}
stream.forEach(domainSetBuilder::add);
} else {
replicaJpaTm()
.transact(
() -> {
@@ -595,7 +506,6 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
});
}
});
}
}
List<Domain> domains = domainSetBuilder.build().asList();
metricInformationBuilder.setNumHostsRetrieved(numHostKeysSearched);
@@ -16,7 +16,6 @@ package google.registry.rdap;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.rdap.RdapUtils.getRegistrarByIanaIdentifier;
import static google.registry.request.Action.Method.GET;
import static google.registry.request.Action.Method.HEAD;
@@ -27,7 +26,6 @@ import com.google.common.collect.Iterables;
import com.google.common.collect.Streams;
import com.google.common.primitives.Booleans;
import com.google.common.primitives.Longs;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.contact.Contact;
import google.registry.model.registrar.Registrar;
import google.registry.persistence.VKey;
@@ -261,39 +259,24 @@ public class RdapEntitySearchAction extends RdapSearchActionBase {
|| (cursorType == CursorType.REGISTRAR)) {
resultSet = RdapResultSet.create(ImmutableList.of());
} else {
if (tm().isOfy()) {
Query<Contact> query =
queryItems(
Contact.class,
"searchName",
partialStringQuery,
cursorQueryString, // if we get here and there's a cursor, it must be a contact
DeletedItemHandling.EXCLUDE,
rdapResultSetMaxSize + 1);
if (!rdapAuthorization.role().equals(Role.ADMINISTRATOR)) {
query = query.filter("currentSponsorClientId in", rdapAuthorization.registrarIds());
}
resultSet = getMatchingResources(query, false, rdapResultSetMaxSize + 1);
} else {
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Contact> builder =
queryItemsSql(
Contact.class,
"searchName",
partialStringQuery,
cursorQueryString,
DeletedItemHandling.EXCLUDE);
if (!rdapAuthorization.role().equals(Role.ADMINISTRATOR)) {
builder =
builder.whereFieldIsIn(
"currentSponsorClientId", rdapAuthorization.registrarIds());
}
return getMatchingResourcesSql(builder, false, rdapResultSetMaxSize + 1);
});
}
resultSet =
replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Contact> builder =
queryItems(
Contact.class,
"searchName",
partialStringQuery,
cursorQueryString,
DeletedItemHandling.EXCLUDE);
if (!rdapAuthorization.role().equals(Role.ADMINISTRATOR)) {
builder =
builder.whereFieldIsIn(
"currentSponsorClientId", rdapAuthorization.registrarIds());
}
return getMatchingResources(builder, false, rdapResultSetMaxSize + 1);
});
}
}
return makeSearchResults(resultSet, registrars, QueryType.FULL_NAME);
@@ -386,31 +369,18 @@ public class RdapEntitySearchAction extends RdapSearchActionBase {
if (subtype == Subtype.REGISTRARS) {
contactResultSet = RdapResultSet.create(ImmutableList.of());
} else {
if (tm().isOfy()) {
contactResultSet =
getMatchingResources(
queryItemsByKey(
Contact.class,
partialStringQuery,
cursorQueryString,
getDeletedItemHandling(),
querySizeLimit),
shouldIncludeDeleted(),
querySizeLimit);
} else {
contactResultSet =
replicaJpaTm()
.transact(
() ->
getMatchingResourcesSql(
queryItemsByKeySql(
Contact.class,
partialStringQuery,
cursorQueryString,
getDeletedItemHandling()),
shouldIncludeDeleted(),
querySizeLimit));
}
contactResultSet =
replicaJpaTm()
.transact(
() ->
getMatchingResources(
queryItemsByKey(
Contact.class,
partialStringQuery,
cursorQueryString,
getDeletedItemHandling()),
shouldIncludeDeleted(),
querySizeLimit));
}
return makeSearchResults(contactResultSet, registrars, QueryType.HANDLE);
}
@@ -21,7 +21,6 @@ import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.ImmutableSetMultimap.toImmutableSetMultimap;
import static google.registry.model.EppResourceUtils.isLinked;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.rdap.RdapIcannStandardInformation.CONTACT_REDACTED_VALUE;
import static google.registry.util.CollectionUtils.union;
@@ -91,8 +90,9 @@ import org.joda.time.DateTime;
*
* <p>The JSON format specifies that entities should be supplied with links indicating how to fetch
* them via RDAP, which requires the URL to the RDAP server. The linkBase parameter, passed to many
* of the methods, is used as the first part of the link URL. For instance, if linkBase is
* "http://rdap.org/dir/", the link URLs will look like "http://rdap.org/dir/domain/XXXX", etc.
* of the methods, is used as the first part of the link URL. For instance, if linkBase is <a
* href="http://rdap.org/dir/"></a>, the link URLs will look like <a
* href="http://rdap.org/dir/domain/XXXX"></a>, etc.
*
* @see <a href="https://tools.ietf.org/html/rfc9083">RFC 9083: JSON Responses for the Registration
* Data Access Protocol (RDAP)</a>
@@ -211,15 +211,15 @@ public class RdapJsonFormatter {
.put(HistoryEntry.Type.CONTACT_DELETE, EventAction.DELETION)
.put(HistoryEntry.Type.CONTACT_TRANSFER_APPROVE, EventAction.TRANSFER)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_AUTORENEW, EventAction.REREGISTRATION)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_DELETE, EventAction.DELETION)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_RENEW, EventAction.REREGISTRATION)
/** Not in the Response Profile. */
/* Not in the Response Profile. */
.put(HistoryEntry.Type.DOMAIN_RESTORE, EventAction.REINSTANTIATION)
/** Section 2.3.2.3, optional. */
/* Section 2.3.2.3, optional. */
.put(HistoryEntry.Type.DOMAIN_TRANSFER_APPROVE, EventAction.TRANSFER)
.put(HistoryEntry.Type.HOST_CREATE, EventAction.REGISTRATION)
.put(HistoryEntry.Type.HOST_DELETE, EventAction.DELETION)
@@ -533,7 +533,7 @@ public class RdapJsonFormatter {
// state/province, postal code, country
//
// Note that in theory we have to show the Organization and state/province and country for the
// REGISTRANT. For now we won't do that until we make sure it's really OK for GDPR
// REGISTRANT. For now, we won't do that until we make sure it's really OK for GDPR
//
if (!isAuthorized) {
// RDAP Response Profile 2.7.4.3: if we redact values from the contact, we MUST include a
@@ -749,9 +749,9 @@ public class RdapJsonFormatter {
if (outputDataType != OutputDataType.SUMMARY) {
ImmutableList<RdapContactEntity> registrarContacts =
registrar.getContacts().stream()
.map(registrarContact -> makeRdapJsonForRegistrarContact(registrarContact))
.filter(optional -> optional.isPresent())
.map(optional -> optional.get())
.map(RdapJsonFormatter::makeRdapJsonForRegistrarContact)
.filter(Optional::isPresent)
.map(Optional::get)
.filter(
contact ->
outputDataType == OutputDataType.FULL
@@ -886,10 +886,6 @@ public class RdapJsonFormatter {
// 2.3.2.3 An event of *eventAction* type *transfer*, with the last date and time that the
// domain was transferred. The event of *eventAction* type *transfer* MUST be omitted if the
// domain name has not been transferred since it was created.
Iterable<? extends HistoryEntry> historyEntries;
if (tm().isOfy()) {
historyEntries = HistoryEntryDao.loadHistoryObjectsForResource(resource.createVKey());
} else {
VKey<? extends EppResource> resourceVkey = resource.createVKey();
Class<? extends HistoryEntry> historyClass =
HistoryEntryDao.getHistoryClassFromParent(resourceVkey.getKind());
@@ -903,10 +899,14 @@ public class RdapJsonFormatter {
.replace("%entityName%", entityName)
.replace("%repoIdField%", repoIdFieldName)
.replace("%repoIdValue%", resourceVkey.getSqlKey().toString());
historyEntries =
replicaJpaTm()
.transact(() -> replicaJpaTm().getEntityManager().createQuery(jpql).getResultList());
}
Iterable<HistoryEntry> historyEntries =
replicaJpaTm()
.transact(
() ->
replicaJpaTm()
.getEntityManager()
.createQuery(jpql, HistoryEntry.class)
.getResultList());
for (HistoryEntry historyEntry : historyEntries) {
EventAction rdapEventAction =
HISTORY_ENTRY_TYPE_TO_RDAP_EVENT_ACTION_MAP.get(historyEntry.getType());
@@ -1131,7 +1131,7 @@ public class RdapJsonFormatter {
* all these objects are projected to the same "now".
*
* <p>This "now" will also be considered the time of the "last update of RDAP database" event that
* RDAP sepc requires.
* RDAP spec requires.
*
* <p>We would have set this during the constructor, but the clock is injected after construction.
* So instead we set the time during the first call to this function.
@@ -16,7 +16,6 @@ package google.registry.rdap;
import static google.registry.model.EppResourceUtils.loadByForeignKeyCached;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.request.Action.Method.GET;
import static google.registry.request.Action.Method.HEAD;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
@@ -26,7 +25,6 @@ import com.google.common.collect.ImmutableSortedSet;
import com.google.common.collect.Iterables;
import com.google.common.net.InetAddresses;
import com.google.common.primitives.Booleans;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.domain.Domain;
import google.registry.model.host.Host;
import google.registry.persistence.transaction.CriteriaQueryBuilder;
@@ -90,7 +88,7 @@ public class RdapNameserverSearchAction extends RdapSearchActionBase {
}
NameserverSearchResponse results;
if (nameParam.isPresent()) {
// RDAP Technical Implementation Guilde 2.2.3 - we MAY support nameserver search queries based
// RDAP Technical Implementation Guide 2.2.3 - we MAY support nameserver search queries based
// on a "nameserver search pattern" as defined in RFC 9082
//
// syntax: /rdap/nameservers?name=exam*.com
@@ -219,33 +217,20 @@ public class RdapNameserverSearchAction extends RdapSearchActionBase {
private NameserverSearchResponse searchByNameUsingPrefix(RdapSearchPattern partialStringQuery) {
// Add 1 so we can detect truncation.
int querySizeLimit = getStandardQuerySizeLimit();
if (tm().isOfy()) {
Query<Host> query =
queryItems(
Host.class,
"hostName",
partialStringQuery,
cursorString,
getDeletedItemHandling(),
querySizeLimit);
return makeSearchResults(
getMatchingResources(query, shouldIncludeDeleted(), querySizeLimit), CursorType.NAME);
} else {
return replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Host> queryBuilder =
queryItemsSql(
Host.class,
"hostName",
partialStringQuery,
cursorString,
getDeletedItemHandling());
return makeSearchResults(
getMatchingResourcesSql(queryBuilder, shouldIncludeDeleted(), querySizeLimit),
CursorType.NAME);
});
}
return replicaJpaTm()
.transact(
() -> {
CriteriaQueryBuilder<Host> queryBuilder =
queryItems(
Host.class,
"hostName",
partialStringQuery,
cursorString,
getDeletedItemHandling());
return makeSearchResults(
getMatchingResources(queryBuilder, shouldIncludeDeleted(), querySizeLimit),
CursorType.NAME);
});
}
/** Searches for nameservers by IP address, returning a JSON array of nameserver info maps. */
@@ -253,18 +238,6 @@ public class RdapNameserverSearchAction extends RdapSearchActionBase {
// Add 1 so we can detect truncation.
int querySizeLimit = getStandardQuerySizeLimit();
RdapResultSet<Host> rdapResultSet;
if (tm().isOfy()) {
Query<Host> query =
queryItems(
Host.class,
"inetAddresses",
inetAddress.getHostAddress(),
Optional.empty(),
cursorString,
getDeletedItemHandling(),
querySizeLimit);
rdapResultSet = getMatchingResources(query, shouldIncludeDeleted(), querySizeLimit);
} else {
// Hibernate does not allow us to query @Converted array fields directly, either in the
// CriteriaQuery or the raw text format. However, Postgres does -- so we use native queries to
// find hosts where any of the inetAddresses match.
@@ -301,7 +274,6 @@ public class RdapNameserverSearchAction extends RdapSearchActionBase {
List<Host> resultList = query.getResultList();
return filterResourcesByVisibility(resultList, querySizeLimit);
});
}
return makeSearchResults(rdapResultSet, CursorType.ADDRESS);
}
@@ -15,14 +15,11 @@
package google.registry.rdap;
import static com.google.common.base.Charsets.UTF_8;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.replicaJpaTm;
import static google.registry.util.DateTimeUtils.END_OF_TIME;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableListMultimap;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.EppResource;
import google.registry.model.registrar.Registrar;
import google.registry.persistence.transaction.CriteriaQueryBuilder;
@@ -142,10 +139,10 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
}
/**
* Runs the given query, and checks for permissioning if necessary.
* In Cloud SQL, builds and runs the given query, and checks for permissioning if necessary.
*
* @param query an already-defined query to be run; a filter on currentSponsorClientId will be
* added if appropriate
* @param builder a query builder that represents the various SELECT FROM, WHERE, ORDER BY, etc.
* clauses that make up this SQL query
* @param checkForVisibility true if the results should be checked to make sure they are visible;
* normally this should be equal to the shouldIncludeDeleted setting, but in cases where the
* query could not check deletion status (due to Datastore limitations such as the limit of
@@ -160,38 +157,6 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
* number we might have expected
*/
<T extends EppResource> RdapResultSet<T> getMatchingResources(
Query<T> query, boolean checkForVisibility, int querySizeLimit) {
Optional<String> desiredRegistrar = getDesiredRegistrar();
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
List<T> queryResult = query.list();
if (checkForVisibility) {
return filterResourcesByVisibility(queryResult, querySizeLimit);
} else {
return RdapResultSet.create(queryResult);
}
}
/**
* In Cloud SQL, builds and runs the given query, and checks for permissioning if necessary.
*
* @param builder a query builder that represents the various SELECT FROM, WHERE, ORDER BY and
* (etc) clauses that make up this SQL query
* @param checkForVisibility true if the results should be checked to make sure they are visible;
* normally this should be equal to the shouldIncludeDeleted setting, but in cases where the
* query could not check deletion status (due to Datastore limitations such as the limit of
* one field queried for inequality, for instance), it may need to be set to true even when
* not including deleted records
* @param querySizeLimit the maximum number of items the query is expected to return, usually
* because the limit has been set
* @return an {@link RdapResultSet} object containing the list of resources and an incompleteness
* warning flag, which is set to MIGHT_BE_INCOMPLETE iff any resources were excluded due to
* lack of visibility, and the resulting list of resources is less than the maximum allowable,
* and the number of items returned by the query is greater than or equal to the maximum
* number we might have expected
*/
<T extends EppResource> RdapResultSet<T> getMatchingResourcesSql(
CriteriaQueryBuilder<T> builder, boolean checkForVisibility, int querySizeLimit) {
replicaJpaTm().assertInTransaction();
Optional<String> desiredRegistrar = getDesiredRegistrar();
@@ -230,10 +195,10 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
}
// The incompleteness problem comes about because we don't know how many items to fetch. We want
// to return rdapResultSetMaxSize worth of items, but some might be excluded, so we fetch more
// just in case. But how many more? That's the potential problem, addressed with the three way
// just in case. But how many more? That's the potential problem, addressed with the three-way
// AND statement:
// 1. If we didn't exclude any items, then we can't have the incompleteness problem.
// 2. If have a full result set batch (rdapResultSetMaxSize items), we must by definition be
// 2. If we have a full result set batch (rdapResultSetMaxSize items), we must by definition be
// giving the user a complete result set.
// 3. If we started with fewer than querySizeLimit items, then there weren't any more items that
// we missed. Even if we return fewer than rdapResultSetMaxSize items, it isn't because we
@@ -321,56 +286,6 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
: (rdapResultSetMaxSize + 1);
}
/**
* Handles prefix searches in cases where, if we need to filter out deleted items, there are no
* pending deletes.
*
* <p>In such cases, it is sufficient to check whether {@code deletionTime} is equal to
* {@code END_OF_TIME}, because any other value means it has already been deleted. This allows us
* to use an equality query for the deletion time.
*
* @param clazz the type of resource to be queried
* @param filterField the database field of interest
* @param partialStringQuery the details of the search string; if there is no wildcard, an
* equality query is used; if there is a wildcard, a range query is used instead; the
* initial string should not be empty, and any search suffix will be ignored, so the caller
* must filter the results if a suffix is specified
* @param cursorString if a cursor is present, this parameter should specify the cursor string, to
* skip any results up to and including the string; empty() if there is no cursor
* @param deletedItemHandling whether to include or exclude deleted items
* @param resultSetMaxSize the maximum number of results to return
* @return the query object
*/
static <T extends EppResource> Query<T> queryItems(
Class<T> clazz,
String filterField,
RdapSearchPattern partialStringQuery,
Optional<String> cursorString,
DeletedItemHandling deletedItemHandling,
int resultSetMaxSize) {
if (partialStringQuery.getInitialString().length()
< RdapSearchPattern.MIN_INITIAL_STRING_LENGTH) {
throw new UnprocessableEntityException(
String.format(
"Initial search string must be at least %d characters",
RdapSearchPattern.MIN_INITIAL_STRING_LENGTH));
}
Query<T> query = auditedOfy().load().type(clazz);
if (!partialStringQuery.getHasWildcard()) {
query = query.filter(filterField, partialStringQuery.getInitialString());
} else {
// Ignore the suffix; the caller will need to filter on the suffix, if any.
query =
query
.filter(filterField + " >=", partialStringQuery.getInitialString())
.filter(filterField + " <", partialStringQuery.getNextInitialString());
}
if (cursorString.isPresent()) {
query = query.filter(filterField + " >", cursorString.get());
}
return setOtherQueryAttributes(query, deletedItemHandling, resultSetMaxSize);
}
/**
* In Cloud SQL, handles prefix searches in cases where, if we need to filter out deleted items,
* there are no pending deletes.
@@ -390,7 +305,7 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
* @param deletedItemHandling whether to include or exclude deleted items
* @return a {@link CriteriaQueryBuilder} object representing the query so far
*/
static <T extends EppResource> CriteriaQueryBuilder<T> queryItemsSql(
static <T extends EppResource> CriteriaQueryBuilder<T> queryItems(
Class<T> clazz,
String filterField,
RdapSearchPattern partialStringQuery,
@@ -420,49 +335,7 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
builder = builder.where(filterField, criteriaBuilder::greaterThan, cursorString.get());
}
builder = builder.orderByAsc(filterField);
return setDeletedItemHandlingSql(builder, deletedItemHandling);
}
/**
* Handles searches using a simple string rather than an {@link RdapSearchPattern}.
*
* <p>Since the filter is not an inequality, we can support also checking a cursor string against
* a different field (which involves an inequality on that field).
*
* @param clazz the type of resource to be queried
* @param filterField the database field of interest
* @param queryString the search string
* @param cursorField the field which should be compared to the cursor string, or empty() if the
* key should be compared to a key created from the cursor string
* @param cursorString if a cursor is present, this parameter should specify the cursor string, to
* skip any results up to and including the string; empty() if there is no cursor
* @param deletedItemHandling whether to include or exclude deleted items
* @param resultSetMaxSize the maximum number of results to return
* @return the query object
*/
static <T extends EppResource> Query<T> queryItems(
Class<T> clazz,
String filterField,
String queryString,
Optional<String> cursorField,
Optional<String> cursorString,
DeletedItemHandling deletedItemHandling,
int resultSetMaxSize) {
if (queryString.length() < RdapSearchPattern.MIN_INITIAL_STRING_LENGTH) {
throw new UnprocessableEntityException(
String.format(
"Initial search string must be at least %d characters",
RdapSearchPattern.MIN_INITIAL_STRING_LENGTH));
}
Query<T> query = auditedOfy().load().type(clazz).filter(filterField, queryString);
if (cursorString.isPresent()) {
if (cursorField.isPresent()) {
query = query.filter(cursorField.get() + " >", cursorString.get());
} else {
query = query.filterKey(">", Key.create(clazz, cursorString.get()));
}
}
return setOtherQueryAttributes(query, deletedItemHandling, resultSetMaxSize);
return setDeletedItemHandling(builder, deletedItemHandling);
}
/**
@@ -481,7 +354,7 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
* @param deletedItemHandling whether to include or exclude deleted items
* @return a {@link CriteriaQueryBuilder} object representing the query so far
*/
static <T extends EppResource> CriteriaQueryBuilder<T> queryItemsSql(
static <T extends EppResource> CriteriaQueryBuilder<T> queryItems(
Class<T> clazz,
String filterField,
String queryString,
@@ -506,50 +379,20 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
builder = builder.where("repoId", criteriaBuilder::greaterThan, cursorString.get());
}
}
return setDeletedItemHandlingSql(builder, deletedItemHandling);
}
/** Handles searches where the field to be searched is the key. */
static <T extends EppResource> Query<T> queryItemsByKey(
Class<T> clazz,
RdapSearchPattern partialStringQuery,
Optional<String> cursorString,
DeletedItemHandling deletedItemHandling,
int resultSetMaxSize) {
if (partialStringQuery.getInitialString().length()
< RdapSearchPattern.MIN_INITIAL_STRING_LENGTH) {
throw new UnprocessableEntityException(
String.format(
"Initial search string must be at least %d characters",
RdapSearchPattern.MIN_INITIAL_STRING_LENGTH));
}
Query<T> query = auditedOfy().load().type(clazz);
if (!partialStringQuery.getHasWildcard()) {
query = query.filterKey("=", Key.create(clazz, partialStringQuery.getInitialString()));
} else {
// Ignore the suffix; the caller will need to filter on the suffix, if any.
query =
query
.filterKey(">=", Key.create(clazz, partialStringQuery.getInitialString()))
.filterKey("<", Key.create(clazz, partialStringQuery.getNextInitialString()));
}
if (cursorString.isPresent()) {
query = query.filterKey(">", Key.create(clazz, cursorString.get()));
}
return setOtherQueryAttributes(query, deletedItemHandling, resultSetMaxSize);
return setDeletedItemHandling(builder, deletedItemHandling);
}
/** In Cloud SQL, handles searches where the field to be searched is the key. */
static <T extends EppResource> CriteriaQueryBuilder<T> queryItemsByKeySql(
static <T extends EppResource> CriteriaQueryBuilder<T> queryItemsByKey(
Class<T> clazz,
RdapSearchPattern partialStringQuery,
Optional<String> cursorString,
DeletedItemHandling deletedItemHandling) {
replicaJpaTm().assertInTransaction();
return queryItemsSql(clazz, "repoId", partialStringQuery, cursorString, deletedItemHandling);
return queryItems(clazz, "repoId", partialStringQuery, cursorString, deletedItemHandling);
}
static <T extends EppResource> CriteriaQueryBuilder<T> setDeletedItemHandlingSql(
static <T extends EppResource> CriteriaQueryBuilder<T> setDeletedItemHandling(
CriteriaQueryBuilder<T> builder, DeletedItemHandling deletedItemHandling) {
if (!Objects.equals(deletedItemHandling, DeletedItemHandling.INCLUDE)) {
builder =
@@ -560,12 +403,4 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
}
return builder;
}
static <T extends EppResource> Query<T> setOtherQueryAttributes(
Query<T> query, DeletedItemHandling deletedItemHandling, int resultSetMaxSize) {
if (deletedItemHandling != DeletedItemHandling.INCLUDE) {
query = query.filter("deletionTime", END_OF_TIME);
}
return query.limit(resultSetMaxSize);
}
}
@@ -103,7 +103,7 @@ public final class RdeUploadAction implements Runnable, EscrowTask {
//
// This prevents making an unnecessary time-expensive (and potentially failing) API call to the
// external KMS system when the RdeUploadAction ends up not being used (if the EscrowTaskRunner
// determins this EscrowTask was already completed today).
// determines this EscrowTask was already completed today).
@Inject Lazy<JSch> lazyJsch;
@Inject JSchSshSessionFactory jschSshSessionFactory;
@@ -128,9 +128,7 @@ public final class RdeUploadAction implements Runnable, EscrowTask {
runner.lockRunAndRollForward(this, Registry.get(tld), timeout, CursorType.RDE_UPLOAD, interval);
HashMultimap<String, String> params = HashMultimap.create();
params.put(RequestParameters.PARAM_TLD, tld);
if (prefix.isPresent()) {
params.put(RdeModule.PARAM_PREFIX, prefix.get());
}
prefix.ifPresent(s -> params.put(RdeModule.PARAM_PREFIX, s));
cloudTasksUtils.enqueue(
RDE_REPORT_QUEUE,
cloudTasksUtils.createPostTask(
@@ -142,7 +140,7 @@ public final class RdeUploadAction implements Runnable, EscrowTask {
// If a prefix is not provided, but we are in SQL mode, try to determine the prefix. This should
// only happen when the RDE upload cron job runs to catch up any un-retried (i. e. expected)
// RDE failures.
if (!prefix.isPresent() && !tm().isOfy()) {
if (!prefix.isPresent()) {
// The prefix is always in the format of: rde-2022-02-21t00-00-00z-2022-02-21t00-07-33z, where
// the first datetime is the watermark and the second one is the time when the RDE beam job
// launched. We search for the latest folder that starts with "rde-[watermark]".
@@ -246,7 +244,7 @@ public final class RdeUploadAction implements Runnable, EscrowTask {
* }</pre>
*/
@VisibleForTesting
protected void upload(
private void upload(
BlobId xmlFile, long xmlLength, DateTime watermark, String name, String nameWithoutPrefix)
throws Exception {
logger.atInfo().log("Uploading XML file '%s' to remote path '%s'.", xmlFile, uploadUrl);
@@ -14,8 +14,6 @@
package google.registry.reporting.icann;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.reporting.icann.IcannReportingModule.DATASTORE_EXPORT_DATA_SET;
import static google.registry.reporting.icann.IcannReportingModule.ICANN_REPORTING_DATA_SET;
import static google.registry.reporting.icann.QueryBuilderUtils.getQueryFromFile;
import static google.registry.reporting.icann.QueryBuilderUtils.getTableName;
@@ -72,19 +70,10 @@ public final class ActivityReportingQueryBuilder implements QueryBuilder {
ImmutableMap.Builder<String, String> queriesBuilder = ImmutableMap.builder();
String operationalRegistrarsQuery;
if (tm().isOfy()) {
operationalRegistrarsQuery =
SqlTemplate.create(getQueryFromFile("registrar_operating_status.sql"))
.put("PROJECT_ID", projectId)
.put("DATASTORE_EXPORT_DATA_SET", DATASTORE_EXPORT_DATA_SET)
.put("REGISTRAR_TABLE", "Registrar")
.build();
} else {
operationalRegistrarsQuery =
SqlTemplate.create(getQueryFromFile("cloud_sql_registrar_operating_status.sql"))
.put("PROJECT_ID", projectId)
.build();
}
queriesBuilder.put(
getTableName(REGISTRAR_OPERATING_STATUS, yearMonth), operationalRegistrarsQuery);
@@ -125,11 +114,7 @@ public final class ActivityReportingQueryBuilder implements QueryBuilder {
queriesBuilder.put(getTableName(WHOIS_COUNTS, yearMonth), whoisQuery);
SqlTemplate aggregateQuery =
SqlTemplate.create(
getQueryFromFile(
tm().isOfy()
? "activity_report_aggregation.sql"
: "cloud_sql_activity_report_aggregation.sql"))
SqlTemplate.create(getQueryFromFile("cloud_sql_activity_report_aggregation.sql"))
.put("PROJECT_ID", projectId)
.put(
"REGISTRAR_OPERATING_STATUS_TABLE",
@@ -139,13 +124,6 @@ public final class ActivityReportingQueryBuilder implements QueryBuilder {
.put("EPP_METRICS_TABLE", getTableName(EPP_METRICS, yearMonth))
.put("WHOIS_COUNTS_TABLE", getTableName(WHOIS_COUNTS, yearMonth));
if (tm().isOfy()) {
aggregateQuery =
aggregateQuery
.put("REGISTRY_TABLE", "Registry")
.put("DATASTORE_EXPORT_DATA_SET", DATASTORE_EXPORT_DATA_SET);
}
queriesBuilder.put(
getTableName(ACTIVITY_REPORT_AGGREGATION, yearMonth), aggregateQuery.build());
@@ -23,7 +23,6 @@ import com.google.common.util.concurrent.MoreExecutors;
import dagger.Module;
import dagger.Provides;
import google.registry.bigquery.BigqueryConnection;
import google.registry.persistence.transaction.TransactionManager;
import google.registry.request.HttpException.BadRequestException;
import google.registry.request.Parameter;
import java.util.Optional;
@@ -44,7 +43,6 @@ public final class IcannReportingModule {
static final String PARAM_SUBDIR = "subdir";
static final String PARAM_REPORT_TYPES = "reportTypes";
static final String ICANN_REPORTING_DATA_SET = "icannReportingDataSet";
static final String DATASTORE_EXPORT_DATA_SET = "latest_datastore_export";
static final String MANIFEST_FILE_NAME = "MANIFEST.txt";
/** Provides an optional subdirectory to store/upload reports to, extracted from the request. */
@@ -104,7 +102,7 @@ public final class IcannReportingModule {
@Provides
@Named(ICANN_REPORTING_DATA_SET)
static String provideIcannReportingDataSet(TransactionManager tm) {
return tm.isOfy() ? "icann_reporting" : "cloud_sql_icann_reporting";
static String provideIcannReportingDataSet() {
return "cloud_sql_icann_reporting";
}
}
@@ -1,27 +0,0 @@
#standardSQL
-- Copyright 2017 The Nomulus Authors. All Rights Reserved.
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
-- Query that counts the number of real registrars in system.
SELECT
-- Applies to all TLDs, hence the 'null' magic value.
STRING(NULL) AS tld,
'operational-registrars' AS metricName,
COUNT(registrarName) AS count
FROM
`%PROJECT_ID%.%DATASTORE_EXPORT_DATA_SET%.%REGISTRAR_TABLE%`
WHERE
(type = 'REAL' OR type = 'INTERNAL')
GROUP BY metricName
@@ -16,25 +16,19 @@ package google.registry.tools;
import static com.google.common.base.Strings.isNullOrEmpty;
import static google.registry.flows.poll.PollFlowUtils.createPollMessageQuery;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.model.poll.PollMessageExternalKeyConverter.makePollMessageExternalId;
import static google.registry.persistence.transaction.QueryComposer.Comparator.LIKE;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.google.common.base.Joiner;
import com.google.common.collect.Iterables;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.cmd.QueryKeys;
import google.registry.flows.poll.PollFlowUtils;
import google.registry.model.poll.PollMessage;
import google.registry.model.poll.PollMessage.Autorenew;
import google.registry.model.poll.PollMessage.OneTime;
import google.registry.persistence.transaction.QueryComposer;
import google.registry.util.Clock;
import java.util.List;
import javax.inject.Inject;
/**
@@ -80,40 +74,9 @@ final class AckPollMessagesCommand implements CommandWithRemoteApi {
@Inject Clock clock;
private static final int BATCH_SIZE = 20;
@Override
public void run() {
if (tm().isOfy()) {
ackPollMessagesDatastore();
} else {
ackPollMessagesSql();
}
}
/**
* Loads and acks the matching poll messages from Datastore.
*
* <p>We have to first load the poll message keys then batch-load the objects themselves due to
* the Datastore size limits.
*/
private void ackPollMessagesDatastore() {
QueryKeys<PollMessage> query =
auditedOfy()
.load()
.type(PollMessage.class)
.filter("clientId", clientId)
.filter("eventTime <=", clock.nowUtc())
.order("eventTime")
.keys();
for (List<Key<PollMessage>> keys : Iterables.partition(query, BATCH_SIZE)) {
tm().transact(
() ->
// Load poll messages and filter to just those of interest.
auditedOfy().load().keys(keys).values().stream()
.filter(pm -> isNullOrEmpty(message) || pm.getMsg().contains(message))
.forEach(this::actOnPollMessage));
}
}
/** Loads and acks all matching poll messages from SQL in one transaction. */
@@ -48,7 +48,7 @@ import org.json.simple.JSONValue;
* <p>By default - connects to the TOOLS service. To create a Connection to another service, call
* the {@link #withService} function.
*/
class AppEngineConnection {
public class AppEngineConnection {
/** Pattern to heuristically extract title tag contents in HTML responses. */
private static final Pattern HTML_TITLE_TAG_PATTERN = Pattern.compile("<title>(.*?)</title>");
@@ -35,6 +35,7 @@ import dagger.Binds;
import dagger.Lazy;
import dagger.Module;
import dagger.Provides;
import google.registry.config.CredentialModule.ApplicationDefaultCredential;
import google.registry.config.CredentialModule.DefaultCredential;
import google.registry.config.CredentialModule.LocalCredential;
import google.registry.config.CredentialModule.LocalCredentialJson;
@@ -228,6 +229,11 @@ public class AuthModule {
@DefaultCredential
abstract GoogleCredentialsBundle provideLocalCredentialAsDefaultCredential(
@LocalCredential GoogleCredentialsBundle credential);
@Binds
@ApplicationDefaultCredential
abstract GoogleCredentialsBundle provideLocalCredentialAsApplicationDefaultCredential(
@LocalCredential GoogleCredentialsBundle credential);
}
/** Raised when we need a user login. */
@@ -22,7 +22,7 @@ import org.joda.time.Duration;
/** Parameter delegate class to handle flag settings for a command's BigqueryConnection object. */
@Parameters(separators = " =")
final class BigqueryParameters {
public final class BigqueryParameters {
/**
* Default to 20 threads to stay within Bigquery's rate limit of 20 concurrent queries.
@@ -15,6 +15,6 @@
package google.registry.tools;
/** A command that can send HTTP requests to a backend module. */
interface CommandWithConnection extends Command {
public interface CommandWithConnection extends Command {
void setConnection(AppEngineConnection connection);
}
@@ -71,7 +71,7 @@ final class DeleteTldCommand extends ConfirmingCommand implements CommandWithRem
@Override
protected String execute() {
tm().transactNew(() -> tm().delete(registry));
tm().transact(() -> tm().delete(registry));
registry.invalidateInCache();
return String.format("Deleted TLD '%s'.\n", tld);
}
@@ -0,0 +1,47 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.util.PreconditionsUtils.checkArgumentPresent;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import google.registry.model.domain.token.PackagePromotion;
import java.util.List;
/** Command to show a {@link PackagePromotion} object. */
@Parameters(separators = " =", commandDescription = "Show package promotion object(s)")
public class GetPackagePromotionCommand extends GetEppResourceCommand {
@Parameter(description = "Package token(s)", required = true)
private List<String> mainParameters;
@Override
void runAndPrint() {
for (String token : mainParameters) {
jpaTm()
.transact(
() -> {
PackagePromotion packagePromotion =
checkArgumentPresent(
PackagePromotion.loadByTokenString(token),
"PackagePromotion with package token %s does not exist",
token);
System.out.println(packagePromotion);
});
}
}
}
@@ -263,7 +263,7 @@ final class RegistryCli implements AutoCloseable, CommandRunner {
ObjectifyService.initOfy();
// Make sure we start the command with a clean cache, so that any previous command won't
// interfere with this one.
ObjectifyService.ofy().clearSessionCache();
ObjectifyService.auditedOfy().clearSessionCache();
// Enable Cloud SQL for command that needs remote API as they will very likely use
// Cloud SQL after the database migration. Note that the DB password is stored in Datastore
@@ -17,6 +17,7 @@ package google.registry.tools;
import com.google.common.collect.ImmutableMap;
import google.registry.tools.javascrap.CompareEscrowDepositsCommand;
import google.registry.tools.javascrap.CreateCancellationsForOneTimesCommand;
import google.registry.tools.javascrap.CreateSyntheticDomainHistoriesCommand;
/** Container class to create and run remote commands against a Datastore instance. */
public final class RegistryTool {
@@ -47,6 +48,7 @@ public final class RegistryTool {
.put("create_registrar", CreateRegistrarCommand.class)
.put("create_registrar_groups", CreateRegistrarGroupsCommand.class)
.put("create_reserved_list", CreateReservedListCommand.class)
.put("create_synthetic_domain_histories", CreateSyntheticDomainHistoriesCommand.class)
.put("create_tld", CreateTldCommand.class)
.put("curl", CurlCommand.class)
.put("delete_allocation_tokens", DeleteAllocationTokensCommand.class)
@@ -71,6 +73,7 @@ public final class RegistryTool {
.put("get_history_entries", GetHistoryEntriesCommand.class)
.put("get_host", GetHostCommand.class)
.put("get_keyring_secret", GetKeyringSecretCommand.class)
.put("get_package_promotion", GetPackagePromotionCommand.class)
.put("get_premium_list", GetPremiumListCommand.class)
.put("get_registrar", GetRegistrarCommand.class)
.put("get_reserved_list", GetReservedListCommand.class)
@@ -43,6 +43,7 @@ import google.registry.request.Modules.UserServiceModule;
import google.registry.tools.AuthModule.LocalCredentialModule;
import google.registry.tools.javascrap.CompareEscrowDepositsCommand;
import google.registry.tools.javascrap.CreateCancellationsForOneTimesCommand;
import google.registry.tools.javascrap.CreateSyntheticDomainHistoriesCommand;
import google.registry.util.UtilsModule;
import google.registry.whois.NonCachingWhoisModule;
import javax.annotation.Nullable;
@@ -106,6 +107,8 @@ interface RegistryToolComponent {
void inject(CreateRegistrarCommand command);
void inject(CreateSyntheticDomainHistoriesCommand command);
void inject(CreateTldCommand command);
void inject(EncryptEscrowDepositCommand command);
@@ -118,6 +121,14 @@ interface RegistryToolComponent {
void inject(GenerateEscrowDepositCommand command);
void inject(GetContactCommand command);
void inject(GetDomainCommand command);
void inject(GetHostCommand command);
void inject(GetPackagePromotionCommand command);
void inject(GetKeyringSecretCommand command);
void inject(GetSqlCredentialCommand command);
@@ -144,6 +155,8 @@ interface RegistryToolComponent {
void inject(SetupOteCommand command);
void inject(UniformRapidSuspensionCommand command);
void inject(UnlockDomainCommand command);
void inject(UnrenewDomainCommand command);
@@ -28,8 +28,8 @@ import java.lang.reflect.Method;
* {@link RemoteApiOptions} with a JSON representing a user credential.
*/
public class RemoteApiOptionsUtil {
static RemoteApiOptions useGoogleCredentialStream(RemoteApiOptions options, InputStream stream)
throws Exception {
public static RemoteApiOptions useGoogleCredentialStream(
RemoteApiOptions options, InputStream stream) throws Exception {
Method method =
options.getClass().getDeclaredMethod("useGoogleCredentialStream", InputStream.class);
checkState(
@@ -0,0 +1,209 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools.javascrap;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static java.nio.charset.StandardCharsets.UTF_8;
import com.beust.jcommander.Parameters;
import com.google.appengine.tools.remoteapi.RemoteApiInstaller;
import com.google.appengine.tools.remoteapi.RemoteApiOptions;
import com.google.common.collect.ImmutableSet;
import com.google.common.flogger.FluentLogger;
import google.registry.config.CredentialModule;
import google.registry.config.RegistryConfig;
import google.registry.config.RegistryConfig.Config;
import google.registry.model.domain.Domain;
import google.registry.model.ofy.ObjectifyService;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.VKey;
import google.registry.tools.AppEngineConnection;
import google.registry.tools.CommandWithConnection;
import google.registry.tools.CommandWithRemoteApi;
import google.registry.tools.ConfirmingCommand;
import google.registry.tools.RemoteApiOptionsUtil;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import javax.inject.Inject;
import org.joda.time.DateTime;
/**
* Command that creates an additional synthetic history object for domains.
*
* <p>This is created to fix the issue identified in b/248112997. After b/245940594, there were some
* domains where the most recent history object did not represent the state of the domain as it
* exists in the world. Because RDE loads only from DomainHistory objects, this means that RDE was
* producing wrong data. This command mitigates that issue by creating synthetic history events for
* every domain that was not deleted as of the start of the bad {@link
* google.registry.beam.resave.ResaveAllEppResourcesPipeline} -- then, we can guarantee that this
* new history object represents the state of the domain as far as we know.
*
* <p>A previous run of this command (in pipeline form) attempted to do this and succeeded in most
* cases. Unfortunately, that pipeline had an issue where it used self-allocated IDs for some of the
* dependent objects (e.g. {@link google.registry.model.domain.secdns.DomainDsDataHistory}). As a
* result, we want to run this again as a command using Datastore-allocated IDs to re-create
* synthetic history objects for any domain whose last history object is one of the
* potentially-incorrect synthetic objects.
*
* <p>We further restrict the domains to domains whose latest history object is before October 4.
* This is an arbitrary date that is suitably far after the previous incorrect run of this synthetic
* history pipeline, with the purpose of making future runs of this command idempotent (in case the
* command fails, we can just run it again and again).
*/
@Parameters(
separators = " =",
commandDescription = "Create synthetic domain history objects to fix RDE.")
public class CreateSyntheticDomainHistoriesCommand extends ConfirmingCommand
implements CommandWithRemoteApi, CommandWithConnection {
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private static final String HISTORY_REASON =
"Create synthetic domain histories to fix RDE for b/248112997";
private static final DateTime BAD_PIPELINE_END_TIME = DateTime.parse("2022-09-10T12:00:00.000Z");
private static final DateTime NEW_SYNTHETIC_ROUND_START =
DateTime.parse("2022-10-04T00:00:00.000Z");
private static final ExecutorService executor = Executors.newFixedThreadPool(20);
private static final AtomicInteger numDomainsProcessed = new AtomicInteger();
private AppEngineConnection connection;
@Inject
@Config("registryAdminClientId")
String registryAdminRegistrarId;
@Inject @CredentialModule.LocalCredentialJson String localCredentialJson;
private final ThreadLocal<RemoteApiInstaller> installerThreadLocal =
ThreadLocal.withInitial(this::createInstaller);
private ImmutableSet<String> domainRepoIds;
@Override
protected String prompt() {
jpaTm()
.transact(
() -> {
domainRepoIds =
jpaTm()
.query(
"SELECT dh.domainRepoId FROM DomainHistory dh JOIN Tld t ON t.tldStr ="
+ " dh.domainBase.tld WHERE t.tldType = 'REAL' AND dh.type ="
+ " 'SYNTHETIC' AND dh.modificationTime > :badPipelineEndTime AND"
+ " dh.modificationTime < :newSyntheticRoundStart AND"
+ " (dh.domainRepoId, dh.modificationTime) IN (SELECT domainRepoId,"
+ " MAX(modificationTime) FROM DomainHistory GROUP BY domainRepoId)",
String.class)
.setParameter("badPipelineEndTime", BAD_PIPELINE_END_TIME)
.setParameter("newSyntheticRoundStart", NEW_SYNTHETIC_ROUND_START)
.getResultStream()
.collect(toImmutableSet());
});
return String.format(
"Attempt to create synthetic history entries for %d domains?", domainRepoIds.size());
}
@Override
protected String execute() throws Exception {
List<Future<?>> futures = new ArrayList<>();
for (String domainRepoId : domainRepoIds) {
futures.add(
executor.submit(
() -> {
// Make sure the remote API is installed for ID generation
installerThreadLocal.get();
jpaTm()
.transact(
() -> {
Domain domain =
jpaTm().loadByKey(VKey.createSql(Domain.class, domainRepoId));
jpaTm()
.put(
HistoryEntry.createBuilderForResource(domain)
.setRegistrarId(registryAdminRegistrarId)
.setBySuperuser(true)
.setRequestedByRegistrar(false)
.setModificationTime(jpaTm().getTransactionTime())
.setReason(HISTORY_REASON)
.setType(HistoryEntry.Type.SYNTHETIC)
.build());
});
int numProcessed = numDomainsProcessed.incrementAndGet();
if (numProcessed % 1000 == 0) {
System.out.printf("Saved histories for %d domains%n", numProcessed);
}
return null;
}));
}
for (Future<?> future : futures) {
try {
future.get();
} catch (Exception e) {
logger.atSevere().withCause(e).log("Error");
}
}
executor.shutdown();
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
return String.format("Saved entries for %d domains", numDomainsProcessed.get());
}
@Override
public void setConnection(AppEngineConnection connection) {
this.connection = connection;
}
/**
* Installs the remote API so that the worker threads can use Datastore for ID generation.
*
* <p>Lifted from the RegistryCli class
*/
private RemoteApiInstaller createInstaller() {
RemoteApiInstaller installer = new RemoteApiInstaller();
RemoteApiOptions options = new RemoteApiOptions();
options.server(connection.getServer().getHost(), getPort(connection.getServer()));
if (RegistryConfig.areServersLocal()) {
// Use dev credentials for localhost.
options.useDevelopmentServerCredential();
} else {
try {
RemoteApiOptionsUtil.useGoogleCredentialStream(
options, new ByteArrayInputStream(localCredentialJson.getBytes(UTF_8)));
} catch (Exception e) {
throw new RuntimeException(e);
}
}
try {
installer.install(options);
} catch (IOException e) {
throw new RuntimeException(e);
}
ObjectifyService.initOfy();
return installer;
}
private static int getPort(URL url) {
return url.getPort() == -1 ? url.getDefaultPort() : url.getPort();
}
}
@@ -1,137 +0,0 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools.javascrap;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import com.google.common.collect.ImmutableMap;
import dagger.Component;
import google.registry.beam.common.RegistryJpaIO;
import google.registry.beam.common.RegistryPipelineOptions;
import google.registry.config.RegistryConfig.Config;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.model.domain.Domain;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.VKey;
import java.io.Serializable;
import javax.inject.Singleton;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.joda.time.DateTime;
/**
* Pipeline that creates a synthetic history for every non-deleted {@link Domain} in SQL.
*
* <p>This is created to fix the issue identified in b/248112997. After b/245940594, there were some
* domains where the most recent history object did not represent the state of the domain as it
* exists in the world. Because RDE loads only from DomainHistory objects, this means that RDE was
* producing wrong data. This pipeline mitigates that issue by creating synthetic history events for
* every domain that was not deleted as of the start of the pipeline -- then, we can guarantee that
* this new history object represents the state of the domain as far as we know.
*
* <p>To run the pipeline (replace the environment as appropriate):
*
* <p><code>
* $ ./nom_build :core:createSyntheticDomainHistories --args="--region=us-central1
* --runner=DataflowRunner
* --registryEnvironment=CRASH
* --project={project-id}
* --workerMachineType=n2-standard-4"
* </code>
*/
public class CreateSyntheticDomainHistoriesPipeline implements Serializable {
private static final String HISTORY_REASON =
"Create synthetic domain histories to fix RDE for b/248112997";
private static final DateTime BAD_PIPELINE_START_TIME =
DateTime.parse("2022-09-05T09:00:00.000Z");
private static final DateTime BAD_PIPELINE_END_TIME = DateTime.parse("2022-09-10T12:00:00.000Z");
static void setup(Pipeline pipeline, String registryAdminRegistrarId) {
pipeline
.apply(
"Read all domain repo IDs",
RegistryJpaIO.read(
"SELECT d.repoId FROM Domain d WHERE deletionTime > :badPipelineStartTime AND NOT"
+ " EXISTS (SELECT 1 FROM DomainHistory dh WHERE dh.domainRepoId = d.repoId"
+ " AND dh.modificationTime > :badPipelineEndTime)",
ImmutableMap.of(
"badPipelineStartTime",
BAD_PIPELINE_START_TIME,
"badPipelineEndTime",
BAD_PIPELINE_END_TIME),
String.class,
repoId -> VKey.createSql(Domain.class, repoId)))
.apply(
"Save a synthetic DomainHistory for each domain",
ParDo.of(new DomainHistoryCreator(registryAdminRegistrarId)));
}
private static class DomainHistoryCreator extends DoFn<VKey<Domain>, Void> {
private final String registryAdminRegistrarId;
private DomainHistoryCreator(String registryAdminRegistrarId) {
this.registryAdminRegistrarId = registryAdminRegistrarId;
}
@ProcessElement
public void processElement(
@Element VKey<Domain> key, PipelineOptions options, OutputReceiver<Void> outputReceiver) {
jpaTm()
.transact(
() -> {
Domain domain = jpaTm().loadByKey(key);
jpaTm()
.put(
HistoryEntry.createBuilderForResource(domain)
.setRegistrarId(registryAdminRegistrarId)
.setBySuperuser(true)
.setRequestedByRegistrar(false)
.setModificationTime(jpaTm().getTransactionTime())
.setReason(HISTORY_REASON)
.setType(HistoryEntry.Type.SYNTHETIC)
.build());
outputReceiver.output(null);
});
}
}
public static void main(String[] args) {
RegistryPipelineOptions options =
PipelineOptionsFactory.fromArgs(args).withValidation().as(RegistryPipelineOptions.class);
RegistryPipelineOptions.validateRegistryPipelineOptions(options);
options.setIsolationOverride(TransactionIsolationLevel.TRANSACTION_READ_COMMITTED);
String registryAdminRegistrarId =
DaggerCreateSyntheticDomainHistoriesPipeline_ConfigComponent.create()
.getRegistryAdminRegistrarId();
Pipeline pipeline = Pipeline.create(options);
setup(pipeline, registryAdminRegistrarId);
pipeline.run();
}
@Singleton
@Component(modules = ConfigModule.class)
interface ConfigComponent {
@Config("registryAdminClientId")
String getRegistryAdminRegistrarId();
}
}
@@ -16,20 +16,15 @@ package google.registry.tools.server;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.model.tld.Registries.assertTldsExist;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.request.Action.Method.GET;
import static google.registry.request.Action.Method.POST;
import static google.registry.request.RequestParameters.PARAM_TLDS;
import static java.util.Comparator.comparing;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import google.registry.model.EppResource;
import google.registry.model.EppResourceUtils;
import google.registry.model.domain.Domain;
import google.registry.request.Action;
@@ -37,9 +32,7 @@ import google.registry.request.Parameter;
import google.registry.request.auth.Auth;
import google.registry.util.Clock;
import google.registry.util.NonFinalForTesting;
import java.util.List;
import javax.inject.Inject;
import org.joda.time.DateTime;
/** An action that lists domains, for use by the {@code nomulus list_domains} command. */
@Action(
@@ -76,38 +69,11 @@ public final class ListDomainsAction extends ListObjectsAction<Domain> {
public ImmutableSet<Domain> loadObjects() {
checkArgument(!tlds.isEmpty(), "Must specify TLDs to query");
assertTldsExist(tlds);
ImmutableList<Domain> domains = tm().isOfy() ? loadDomainsOfy() : loadDomainsSql();
ImmutableList<Domain> domains = loadDomains();
return ImmutableSet.copyOf(domains.reverse());
}
private ImmutableList<Domain> loadDomainsOfy() {
DateTime now = clock.nowUtc();
ImmutableList.Builder<Domain> domainsBuilder = new ImmutableList.Builder<>();
// Combine the batches together by sorting all domains together with newest first, applying the
// limit, and then reversing for display order.
for (List<String> tldsBatch : Lists.partition(tlds.asList(), maxNumSubqueries)) {
auditedOfy()
.load()
.type(Domain.class)
.filter("tld in", tldsBatch)
// Get the N most recently created domains (requires ordering in descending order).
.order("-creationTime")
.limit(limit)
.list()
.stream()
.map(EppResourceUtils.transformAtTime(now))
// Deleted entities must be filtered out post-query because queries don't allow
// ordering with two filters.
.filter(d -> d.getDeletionTime().isAfter(now))
.forEach(domainsBuilder::add);
}
return domainsBuilder.build().stream()
.sorted(comparing(EppResource::getCreationTime).reversed())
.limit(limit)
.collect(toImmutableList());
}
private ImmutableList<Domain> loadDomainsSql() {
private ImmutableList<Domain> loadDomains() {
return jpaTm()
.transact(
() ->
@@ -104,7 +104,7 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
* by default. Enqueuing is allowed only if the value of isInTestDriver is false. It's set to true
* in start() and set to false in stop() inside TestDriver.java, a class used in testing.
*/
private static ThreadLocal<Boolean> isInTestDriver = ThreadLocal.withInitial(() -> false);
private static final ThreadLocal<Boolean> isInTestDriver = ThreadLocal.withInitial(() -> false);
@Inject JsonActionRunner jsonActionRunner;
@Inject RegistrarConsoleMetrics registrarConsoleMetrics;
@@ -231,21 +231,19 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
private RegistrarResult update(final Map<String, ?> args, String registrarId) {
// Email the updates
sendExternalUpdatesIfNecessary(tm().transact(() -> saveUpdates(args, registrarId)));
// Reload the result outside of the transaction to get the most recent version
// Reload the result outside the transaction to get the most recent version
return RegistrarResult.create("Saved " + registrarId, loadRegistrarUnchecked(registrarId));
}
/** Saves the updates and returns info needed for the update email */
private EmailInfo saveUpdates(final Map<String, ?> args, String registrarId) {
// We load the registrar here rather than outside of the transaction - to make
// We load the registrar here rather than outside the transaction - to make
// sure we have the latest version. This one is loaded inside the transaction, so it's
// guaranteed to not change before we update it.
Registrar registrar = loadRegistrarUnchecked(registrarId);
// Detach the registrar to avoid Hibernate object-updates, since we wish to email
// out the diffs between the existing and updated registrar objects
if (!tm().isOfy()) {
jpaTm().getEntityManager().detach(registrar);
}
jpaTm().getEntityManager().detach(registrar);
// Verify that the registrar hasn't been changed.
// To do that - we find the latest update time (or null if the registrar has been
// deleted) and compare to the update time from the args. The update time in the args
@@ -262,7 +260,7 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
"Registrar has been changed by someone else. Please reload and retry.");
}
// Keep the current contacts so we can later check that no required contact was
// Keep the current contacts, so we can later check that no required contact was
// removed, email the changes to the contacts
ImmutableSet<RegistrarPoc> contacts = registrar.getContacts();
@@ -297,7 +295,7 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
ImmutableSet<Map<String, Object>> expandedContacts =
Streams.stream(contacts)
.map(RegistrarPoc::toDiffableFieldMap)
// Note: per the javadoc, toDiffableFieldMap includes sensitive data but we don't want
// Note: per the javadoc, toDiffableFieldMap includes sensitive data, but we don't want
// to display it here
.peek(
map -> {
@@ -416,7 +414,7 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
throw new ForbiddenException("Can't remove allowed TLDs using the console.");
}
if (!Sets.difference(updatedAllowedTlds, initialRegistrar.getAllowedTlds()).isEmpty()) {
// If a REAL registrar isn't in compliance with regards to having an abuse contact set,
// If a REAL registrar isn't in compliance with regard to having an abuse contact set,
// prevent addition of allowed TLDs until that's fixed.
if (Registrar.Type.REAL.equals(initialRegistrar.getType())
&& PRODUCTION.equals(RegistryEnvironment.get())) {
@@ -430,7 +428,8 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
}
/**
* Makes sure builder.build is different than originalRegistrar only if we have the correct role.
* Makes sure {@code builder.build}is different from {@code originalRegistrar} only if we have the
* correct role.
*
* <p>On success, returns {@code builder.build()}.
*/
@@ -610,7 +609,7 @@ public class RegistrarSettingsAction implements Runnable, JsonActionRunner.JsonA
* query as abuse contact (if any).
*
* <p>Frontend processing ensures that only one contact can be set as abuse contact in domain
* WHOIS record. Therefore it is possible to return inside the loop once one such contact is
* WHOIS record. Therefore, it is possible to return inside the loop once one such contact is
* found.
*/
private static Optional<RegistrarPoc> getDomainWhoisVisibleAbuseContact(
@@ -16,9 +16,7 @@ package google.registry.whois;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static javax.servlet.http.HttpServletResponse.SC_NOT_FOUND;
import com.google.common.annotations.VisibleForTesting;
@@ -52,35 +50,26 @@ final class NameserverLookupByIpCommand implements WhoisCommand {
@SuppressWarnings("unchecked")
public WhoisResponse executeQuery(DateTime now) throws WhoisException {
Iterable<Host> hostsFromDb;
if (tm().isOfy()) {
hostsFromDb =
auditedOfy()
.load()
.type(Host.class)
.filter("inetAddresses", ipAddress)
.filter("deletionTime >", now.toDate());
} else {
hostsFromDb =
jpaTm()
.transact(
() ->
// We cannot query @Convert-ed fields in HQL so we must use native Postgres.
jpaTm()
.getEntityManager()
/**
* Using array_operator <@ (contained-by) with gin index on inet_address.
* Without gin index, this is slightly slower than the alternative form of
* ':address = ANY(inet_address)'.
*/
.createNativeQuery(
"SELECT * From \"Host\" WHERE "
+ "ARRAY[ CAST(:address AS TEXT) ] <@ inet_addresses AND "
+ "deletion_time > CAST(:now AS timestamptz)",
Host.class)
.setParameter("address", InetAddresses.toAddrString(ipAddress))
.setParameter("now", now.toString())
.getResultList());
}
hostsFromDb =
jpaTm()
.transact(
() ->
// We cannot query @Convert-ed fields in HQL, so we must use native Postgres.
jpaTm()
.getEntityManager()
/*
* Using array_operator <@ (contained-by) with gin index on inet_address.
* Without gin index, this is slightly slower than the alternative form of
* ':address = ANY(inet_address)'.
*/
.createNativeQuery(
"SELECT * From \"Host\" WHERE "
+ "ARRAY[ CAST(:address AS TEXT) ] <@ inet_addresses AND "
+ "deletion_time > CAST(:now AS timestamptz)",
Host.class)
.setParameter("address", InetAddresses.toAddrString(ipAddress))
.setParameter("now", now.toString())
.getResultList());
ImmutableList<Host> hosts =
Streams.stream(hostsFromDb)
.filter(
@@ -0,0 +1,241 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.batch;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.persistActiveContact;
import static google.registry.testing.DatabaseHelper.persistEppResource;
import static google.registry.testing.DatabaseHelper.persistResource;
import static google.registry.testing.LogsSubject.assertAboutLogs;
import com.google.common.testing.TestLogHandler;
import google.registry.model.billing.BillingEvent.RenewalPriceBehavior;
import google.registry.model.contact.Contact;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.domain.token.AllocationToken.TokenType;
import google.registry.model.domain.token.PackagePromotion;
import google.registry.testing.AppEngineExtension;
import google.registry.testing.DatabaseHelper;
import google.registry.testing.FakeClock;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.joda.money.CurrencyUnit;
import org.joda.money.Money;
import org.joda.time.DateTime;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.testcontainers.shaded.com.google.common.collect.ImmutableSet;
/** Unit tests for {@link CheckPackagesComplianceAction}. */
public class CheckPackagesComplianceActionTest {
// This is the default creation time for test data.
private final FakeClock clock = new FakeClock(DateTime.parse("2012-03-25TZ"));
@RegisterExtension
public final AppEngineExtension appEngine =
AppEngineExtension.builder().withCloudSql().withClock(clock).build();
private CheckPackagesComplianceAction action;
private AllocationToken token;
private final TestLogHandler logHandler = new TestLogHandler();
private final Logger loggerToIntercept =
Logger.getLogger(CheckPackagesComplianceAction.class.getCanonicalName());
private Contact contact;
private PackagePromotion packagePromotion;
@BeforeEach
void beforeEach() {
loggerToIntercept.addHandler(logHandler);
createTld("tld");
action = new CheckPackagesComplianceAction();
token =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(TokenType.PACKAGE)
.setCreationTimeForTest(DateTime.parse("2010-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("foo"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountFraction(1)
.build());
packagePromotion =
new PackagePromotion.Builder()
.setToken(token)
.setMaxDomains(3)
.setMaxCreates(1)
.setPackagePrice(Money.of(CurrencyUnit.USD, 1000))
.setNextBillingDate(DateTime.parse("2012-11-12T05:00:00Z"))
.setLastNotificationSent(DateTime.parse("2010-11-12T05:00:00Z"))
.build();
jpaTm().transact(() -> jpaTm().put(packagePromotion));
contact = persistActiveContact("contact1234");
}
@AfterEach
void afterEach() {
loggerToIntercept.removeHandler(logHandler);
}
@Test
void testSuccess_noPackageOverCreateLimit() {
persistEppResource(
DatabaseHelper.newDomain("foo.tld", contact)
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
action.run();
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(Level.INFO, "Found no packages over their create limit.");
}
@Test
void testSuccess_onePackageOverCreateLimit() {
// Create limit is 1, creating 2 domains to go over the limit
persistEppResource(
DatabaseHelper.newDomain("foo.tld", contact)
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
persistEppResource(
DatabaseHelper.newDomain("buzz.tld", contact)
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
action.run();
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(Level.INFO, "Found 1 packages over their create limit.");
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(
Level.INFO,
"Package with package token abc123 has exceeded their max domain creation limit by 1"
+ " name(s).");
}
@Test
void testSuccess_multiplePackagesOverCreateLimit() {
// Create limit is 1, creating 2 domains to go over the limit
persistEppResource(
DatabaseHelper.newDomain("foo.tld", contact)
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
persistEppResource(
DatabaseHelper.newDomain("buzz.tld", contact)
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
AllocationToken token2 =
persistResource(
new AllocationToken.Builder()
.setToken("token")
.setTokenType(TokenType.PACKAGE)
.setCreationTimeForTest(DateTime.parse("2010-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("foo"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountFraction(1)
.build());
PackagePromotion packagePromotion2 =
new PackagePromotion.Builder()
.setToken(token2)
.setMaxDomains(8)
.setMaxCreates(1)
.setPackagePrice(Money.of(CurrencyUnit.USD, 1000))
.setNextBillingDate(DateTime.parse("2012-11-12T05:00:00Z"))
.build();
jpaTm().transact(() -> jpaTm().put(packagePromotion2));
persistEppResource(
DatabaseHelper.newDomain("foo2.tld", contact)
.asBuilder()
.setCurrentPackageToken(token2.createVKey())
.build());
persistEppResource(
DatabaseHelper.newDomain("buzz2.tld", contact)
.asBuilder()
.setCurrentPackageToken(token2.createVKey())
.build());
action.run();
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(Level.INFO, "Found 2 packages over their create limit.");
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(
Level.INFO,
"Package with package token abc123 has exceeded their max domain creation limit by 1"
+ " name(s).");
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(
Level.INFO,
"Package with package token token has exceeded their max domain creation limit by 1"
+ " name(s).");
}
@Test
void testSuccess_onlyChecksCurrentBillingYear() {
AllocationToken token2 =
persistResource(
new AllocationToken.Builder()
.setToken("token")
.setTokenType(TokenType.PACKAGE)
.setCreationTimeForTest(DateTime.parse("2010-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("foo"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountFraction(1)
.build());
PackagePromotion packagePromotion2 =
new PackagePromotion.Builder()
.setToken(token2)
.setMaxDomains(8)
.setMaxCreates(1)
.setPackagePrice(Money.of(CurrencyUnit.USD, 1000))
.setNextBillingDate(DateTime.parse("2015-11-12T05:00:00Z"))
.build();
jpaTm().transact(() -> jpaTm().put(packagePromotion2));
// Create limit is 1, creating 2 domains to go over the limit
persistEppResource(
DatabaseHelper.newDomain("foo.tld", contact)
.asBuilder()
.setCurrentPackageToken(token2.createVKey())
.build());
persistEppResource(
DatabaseHelper.newDomain("buzz.tld", contact)
.asBuilder()
.setCurrentPackageToken(token2.createVKey())
.build());
action.run();
assertAboutLogs()
.that(logHandler)
.hasLogAtLevelWithMessage(Level.INFO, "Found no packages over their create limit.");
}
}
@@ -74,12 +74,8 @@ public class SyncRegistrarsSheetTest {
void beforeEach() {
createTld("example");
// Remove Registrar entities created by AppEngineExtension (and RegistrarContact's, for jpa).
// We don't do this for ofy because ofy's loadAllOf() can't be called in a transaction but
// _must_ be called in a transaction in JPA.
if (!tm().isOfy()) {
tm().transact(() -> tm().loadAllOf(RegistrarPoc.class))
.forEach(DatabaseHelper::deleteResource);
}
Registrar.loadAll().forEach(DatabaseHelper::deleteResource);
}
@@ -17,7 +17,6 @@ package google.registry.flows;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.model.EppResourceUtils.loadAtPointInTime;
import static google.registry.model.ImmutableObjectSubject.assertAboutImmutableObjects;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.loadAllOf;
import static google.registry.testing.DatabaseHelper.loadByEntity;
@@ -94,7 +93,6 @@ class EppPointInTimeTest {
clock.setTo(timeAtCreate);
eppLoader = new EppLoader(this, "domain_create.xml", ImmutableMap.of("DOMAIN", "example.tld"));
runFlow();
tm().clearSessionCache();
Domain domainAfterCreate = Iterables.getOnlyElement(loadAllOf(Domain.class));
assertThat(domainAfterCreate.getDomainName()).isEqualTo("example.tld");
@@ -102,7 +100,6 @@ class EppPointInTimeTest {
DateTime timeAtFirstUpdate = clock.nowUtc();
eppLoader = new EppLoader(this, "domain_update_dsdata_add.xml");
runFlow();
tm().clearSessionCache();
Domain domainAfterFirstUpdate = loadByEntity(domainAfterCreate);
assertThat(domainAfterCreate).isNotEqualTo(domainAfterFirstUpdate);
@@ -111,14 +108,12 @@ class EppPointInTimeTest {
DateTime timeAtSecondUpdate = clock.nowUtc();
eppLoader = new EppLoader(this, "domain_update_dsdata_rem.xml");
runFlow();
tm().clearSessionCache();
Domain domainAfterSecondUpdate = loadByEntity(domainAfterCreate);
clock.advanceBy(standardDays(2));
DateTime timeAtDelete = clock.nowUtc(); // before 'add' grace period ends
eppLoader = new EppLoader(this, "domain_delete.xml", ImmutableMap.of("DOMAIN", "example.tld"));
runFlow();
tm().clearSessionCache();
assertThat(domainAfterFirstUpdate).isNotEqualTo(domainAfterSecondUpdate);
@@ -126,17 +121,14 @@ class EppPointInTimeTest {
Domain latest = loadByEntity(domainAfterCreate);
// Creation time has millisecond granularity due to isActive() check.
tm().clearSessionCache();
assertThat(loadAtPointInTime(latest, timeAtCreate.minusMillis(1))).isNull();
assertThat(loadAtPointInTime(latest, timeAtCreate)).isNotNull();
assertThat(loadAtPointInTime(latest, timeAtCreate.plusMillis(1))).isNotNull();
tm().clearSessionCache();
assertAboutImmutableObjects()
.that(loadAtPointInTime(latest, timeAtCreate.plusDays(1)))
.isEqualExceptFields(domainAfterCreate, "updateTimestamp");
tm().clearSessionCache();
// In SQL, we are not limited by the day granularity, so when we request the object
// at timeAtFirstUpdate we should receive the object at that first update, even though the
// second update occurred one millisecond later.
@@ -144,18 +136,15 @@ class EppPointInTimeTest {
.that(loadAtPointInTime(latest, timeAtFirstUpdate))
.isEqualExceptFields(domainAfterFirstUpdate, "updateTimestamp");
tm().clearSessionCache();
assertAboutImmutableObjects()
.that(loadAtPointInTime(latest, timeAtSecondUpdate))
.isEqualExceptFields(domainAfterSecondUpdate, "updateTimestamp");
tm().clearSessionCache();
assertAboutImmutableObjects()
.that(loadAtPointInTime(latest, timeAtSecondUpdate.plusDays(1)))
.isEqualExceptFields(domainAfterSecondUpdate, "updateTimestamp");
// Deletion time has millisecond granularity due to isActive() check.
tm().clearSessionCache();
assertThat(loadAtPointInTime(latest, timeAtDelete.minusMillis(1))).isNotNull();
assertThat(loadAtPointInTime(latest, timeAtDelete)).isNull();
assertThat(loadAtPointInTime(latest, timeAtDelete.plusMillis(1))).isNull();
@@ -16,7 +16,6 @@ package google.registry.flows;
import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.getOnlyHistoryEntryOfType;
import static google.registry.testing.DatabaseHelper.loadAllOf;
import static google.registry.testing.DatabaseHelper.stripBillingEventId;
@@ -226,7 +225,6 @@ public class EppTestCase {
"Running " + inputFilename + " => " + outputFilename,
"epp.response.resData.infData.roid",
"epp.response.trID.svTRID");
tm().clearSessionCache(); // Clear the cache like OfyFilter would.
return actualOutput;
}
@@ -277,8 +277,6 @@ public abstract class FlowTestCase<F extends Flow> {
Arrays.toString(marshal(output, ValidationMode.LENIENT))),
e);
}
// Clear the cache so that we don't see stale results in tests.
tm().clearSessionCache();
return output;
}
@@ -76,9 +76,6 @@ public abstract class ResourceFlowTestCase<F extends Flow, R extends EppResource
@Nullable
protected R reloadResourceByForeignKey(DateTime now) throws Exception {
// Force the session to be cleared so that when we read it back, we read from Datastore and not
// from the transaction's session cache.
tm().clearSessionCache();
return loadByForeignKey(getResourceClass(), getUniqueIdFromCommand(), now).orElse(null);
}
@@ -88,8 +85,6 @@ public abstract class ResourceFlowTestCase<F extends Flow, R extends EppResource
}
protected <T extends EppResource> T reloadResourceAndCloneAtTime(T resource, DateTime now) {
// Force the session to be cleared.
tm().clearSessionCache();
@SuppressWarnings("unchecked")
T refreshedResource =
(T) tm().transact(() -> tm().loadByEntity(resource)).cloneProjectedAtTime(now);
@@ -21,9 +21,6 @@ import static com.google.common.truth.Truth8.assertThat;
import static google.registry.batch.AsyncTaskEnqueuer.PARAM_REQUESTED_TIME;
import static google.registry.batch.AsyncTaskEnqueuer.PARAM_RESOURCE_KEY;
import static google.registry.batch.AsyncTaskEnqueuer.QUEUE_ASYNC_ACTIONS;
import static google.registry.model.billing.BillingEvent.RenewalPriceBehavior.DEFAULT;
import static google.registry.model.billing.BillingEvent.RenewalPriceBehavior.SPECIFIED;
import static google.registry.model.domain.token.AllocationToken.TokenType.PACKAGE;
import static google.registry.model.domain.token.AllocationToken.TokenType.SINGLE_USE;
import static google.registry.model.domain.token.AllocationToken.TokenType.UNLIMITED_USE;
import static google.registry.model.reporting.DomainTransactionRecord.TransactionReportField.TRANSFER_SUCCESSFUL;
@@ -64,7 +61,6 @@ import com.google.common.collect.Iterables;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import com.google.common.collect.Streams;
import com.google.common.truth.Truth8;
import com.googlecode.objectify.Key;
import google.registry.batch.ResaveEntityAction;
import google.registry.flows.EppException;
@@ -89,7 +85,6 @@ import google.registry.flows.domain.token.AllocationTokenFlowUtils.AllocationTok
import google.registry.flows.domain.token.AllocationTokenFlowUtils.AllocationTokenNotValidForTldException;
import google.registry.flows.domain.token.AllocationTokenFlowUtils.AlreadyRedeemedAllocationTokenException;
import google.registry.flows.domain.token.AllocationTokenFlowUtils.InvalidAllocationTokenException;
import google.registry.flows.domain.token.AllocationTokenFlowUtils.MissingRemovePackageTokenOnPackageDomainException;
import google.registry.flows.exceptions.AlreadyPendingTransferException;
import google.registry.flows.exceptions.InvalidTransferPeriodValueException;
import google.registry.flows.exceptions.MissingTransferRequestAuthInfoException;
@@ -1812,84 +1807,4 @@ class DomainTransferRequestFlowTest
assertThrows(AlreadyRedeemedAllocationTokenException.class, this::runFlow);
assertAboutEppExceptions().that(thrown).marshalsToXml();
}
@Test
void testFailsPackageDomainInvalidAllocationToken() throws Exception {
AllocationToken token =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(PACKAGE)
.setAllowedRegistrarIds(ImmutableSet.of("NewRegistrar"))
.setAllowedTlds(ImmutableSet.of("example", "tld"))
.setRenewalPriceBehavior(SPECIFIED)
.build());
setupDomain("example", "tld");
persistResource(
reloadResourceByForeignKey()
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
setEppInput("domain_transfer_request_allocation_token.xml", ImmutableMap.of("TOKEN", "abc123"));
assertThrows(MissingRemovePackageTokenOnPackageDomainException.class, this::runFlow);
}
@Test
void testFailsToTransferPackageDomainNoRemovePackageToken() throws Exception {
AllocationToken token =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(PACKAGE)
.setAllowedRegistrarIds(ImmutableSet.of("NewRegistrar"))
.setAllowedTlds(ImmutableSet.of("example", "tld"))
.setRenewalPriceBehavior(SPECIFIED)
.build());
setupDomain("example", "tld");
persistResource(
reloadResourceByForeignKey()
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
setEppInput("domain_transfer_request.xml");
assertThrows(MissingRemovePackageTokenOnPackageDomainException.class, this::runFlow);
}
@Test
void testSuccesfullyAppliesRemovePackageToken() throws Exception {
setupDomain("example", "tld");
AllocationToken token =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(PACKAGE)
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setAllowedTlds(ImmutableSet.of("tld"))
.setRenewalPriceBehavior(SPECIFIED)
.build());
domain = loadByEntity(domain);
persistResource(
loadByKey(domain.getAutorenewBillingEvent())
.asBuilder()
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setRenewalPrice(Money.of(USD, new BigDecimal("10.00")))
.build());
persistResource(
reloadResourceByForeignKey()
.asBuilder()
.setCurrentPackageToken(token.createVKey())
.build());
doSuccessfulTest(
"domain_transfer_request_allocation_token.xml",
"domain_transfer_request_response.xml",
ImmutableMap.of("TOKEN", "__REMOVEPACKAGE__"));
Domain domain = reloadResourceByForeignKey();
Truth8.assertThat(domain.getCurrentPackageToken()).isEmpty();
RenewalPriceBehavior priceBehavior =
loadByKey(domain.getAutorenewBillingEvent()).getRenewalPriceBehavior();
assertThat(priceBehavior).isEqualTo(DEFAULT);
}
}
@@ -47,6 +47,7 @@ import static google.registry.testing.DomainSubject.assertAboutDomains;
import static google.registry.testing.EppExceptionSubject.assertAboutEppExceptions;
import static google.registry.testing.HistoryEntrySubject.assertAboutHistoryEntries;
import static google.registry.testing.TaskQueueHelper.assertDnsTasksEnqueued;
import static google.registry.testing.TaskQueueHelper.assertNoDnsTasksEnqueued;
import static google.registry.util.DateTimeUtils.START_OF_TIME;
import static org.joda.money.CurrencyUnit.USD;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -496,13 +497,11 @@ class DomainUpdateFlowTest extends ResourceFlowTestCase<DomainUpdateFlow, Domain
expectedDsData.stream()
.map(ds -> ds.cloneWithDomainRepoId(resource.getRepoId()))
.collect(toImmutableSet()));
// TODO: REENABLE AFTER PROPER FIX FOR DNS PUBLISHING TASKS IS FOUND
// if (dnsTaskEnqueued) {
// assertDnsTasksEnqueued("example.tld");
// } else {
// assertNoDnsTasksEnqueued();
// }
if (dnsTaskEnqueued) {
assertDnsTasksEnqueued("example.tld");
} else {
assertNoDnsTasksEnqueued();
}
}
@Test
@@ -1747,4 +1746,51 @@ class DomainUpdateFlowTest extends ResourceFlowTestCase<DomainUpdateFlow, Domain
assertAboutDomains().that(reloadResourceByForeignKey()).hasNoAutorenewEndTime();
}
@Test
void testAddDnsPublishStatus_enqueueDnsTask() throws Exception {
setEppInput(
"domain_update_status_change.xml",
ImmutableMap.of("STATUS_ADD", "clientHold", "STATUS_REM", "clientTransferProhibited"));
persistReferencedEntities();
persistResource(
persistDomain()
.asBuilder()
.setDomainName("example.tld")
.setStatusValues(ImmutableSet.of(StatusValue.CLIENT_TRANSFER_PROHIBITED))
.build());
runFlowAsSuperuser();
assertDnsTasksEnqueued("example.tld");
}
@Test
void testRemoveEveryDnsPublishStatus_enqueueDnsTask() throws Exception {
setEppInput(
"domain_update_status_change.xml",
ImmutableMap.of("STATUS_REM", "serverHold", "STATUS_ADD", "clientTransferProhibited"));
persistReferencedEntities();
persistResource(
persistDomain()
.asBuilder()
.setDomainName("example.tld")
.setStatusValues(ImmutableSet.of(StatusValue.SERVER_HOLD))
.build());
runFlowAsSuperuser();
assertDnsTasksEnqueued("example.tld");
}
@Test
void testChangeSomeOrNoChangeDnsPublishStatus_doNotEnqueueDnsTask() throws Exception {
setEppInput(
"domain_update_status_change.xml",
ImmutableMap.of("STATUS_ADD", "clientUpdateProhibited", "STATUS_REM", "pendingDelete"));
persistReferencedEntities();
persistResource(
persistDomain()
.asBuilder()
.setDomainName("example.tld")
.setStatusValues(ImmutableSet.of(StatusValue.PENDING_DELETE, StatusValue.SERVER_HOLD))
.build());
runFlowAsSuperuser();
assertNoDnsTasksEnqueued();
}
}
@@ -58,7 +58,6 @@ public class CreateAutoTimestampTest {
tm().put(object);
return tm().getTransactionTime();
});
tm().clearSessionCache();
assertThat(reload().createTime.getTimestamp()).isEqualTo(transactionTime);
}
@@ -71,7 +70,6 @@ public class CreateAutoTimestampTest {
object.createTime = CreateAutoTimestamp.create(oldCreateTime);
tm().put(object);
});
tm().clearSessionCache();
assertThat(reload().createTime.getTimestamp()).isEqualTo(oldCreateTime);
}
}
@@ -18,26 +18,19 @@ import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.truth.Truth.assertAbout;
import static com.google.common.truth.Truth.assertThat;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Iterables;
import com.google.common.collect.Sets;
import com.google.common.truth.Correspondence;
import com.google.common.truth.Correspondence.BinaryPredicate;
import com.google.common.truth.FailureMetadata;
import com.google.common.truth.SimpleSubjectBuilder;
import com.google.common.truth.Subject;
import java.lang.annotation.Annotation;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collector;
import javax.annotation.Nullable;
/** Truth subject for asserting things about ImmutableObjects that are not built in. */
@@ -45,7 +38,7 @@ public final class ImmutableObjectSubject extends Subject {
@Nullable private final ImmutableObject actual;
protected ImmutableObjectSubject(
private ImmutableObjectSubject(
FailureMetadata failureMetadata, @Nullable ImmutableObject actual) {
super(failureMetadata, actual);
this.actual = actual;
@@ -73,261 +66,6 @@ public final class ImmutableObjectSubject extends Subject {
}
}
/**
* Checks that {@code expected} has the same contents as {@code actual} except for fields that are
* marked with {@link ImmutableObject.DoNotCompare}.
*
* <p>This is used to verify that entities stored in both cloud SQL and Datastore are identical.
*/
public void isEqualAcrossDatabases(@Nullable ImmutableObject expected) {
ComparisonResult result =
checkObjectAcrossDatabases(
actual, expected, actual != null ? actual.getClass().getName() : "null");
if (result.isFailure()) {
throw new AssertionError(result.getMessage());
}
}
// The following "check" methods implement a recursive check of immutable object equality across
// databases. All of them function in both assertive and predicate modes: if "path" is
// provided (not null) then they throw AssertionError's with detailed error messages. If
// it is null, they return true for equal objects and false for inequal ones.
//
// The reason for this dual-mode behavior is that all of these methods can either be used in the
// context of a test assertion (in which case we want a detailed error message describing exactly
// the location in a complex object where a difference was discovered) or in the context of a
// membership check in a set (in which case we don't care about the specific location of the first
// difference, we just want to be able to determine if the object "is equal to" another object as
// efficiently as possible -- see checkSetAcrossDatabase()).
@VisibleForTesting
static ComparisonResult checkObjectAcrossDatabases(
@Nullable Object actual, @Nullable Object expected, @Nullable String path) {
if (Objects.equals(actual, expected)) {
return ComparisonResult.createSuccess();
}
// They're different, do a more detailed comparison.
// Check for null first (we can assume both variables are not null at this point).
if (actual == null) {
return ComparisonResult.createFailure(path, "expected ", expected, "got null.");
} else if (expected == null) {
return ComparisonResult.createFailure(path, "expected null, got ", actual);
// For immutable objects, we have to recurse since the contained
// object could have DoNotCompare fields, too.
} else if (expected instanceof ImmutableObject) {
// We only verify that actual is an ImmutableObject so we get a good error message instead
// of a context-less ClassCastException.
if (!(actual instanceof ImmutableObject)) {
return ComparisonResult.createFailure(path, actual, " is not an immutable object.");
}
return checkImmutableAcrossDatabases(
(ImmutableObject) actual, (ImmutableObject) expected, path);
} else if (expected instanceof Map) {
if (!(actual instanceof Map)) {
return ComparisonResult.createFailure(path, actual, " is not a Map.");
}
// This would likely be more efficient if we could assume that keys can be compared across
// databases using .equals(), however we cannot guarantee key equality so the simplest and
// most correct way to accomplish this is by reusing the set comparison.
return checkSetAcrossDatabases(
((Map<?, ?>) actual).entrySet(), ((Map<?, ?>) expected).entrySet(), path, "Map");
} else if (expected instanceof Set) {
if (!(actual instanceof Set)) {
return ComparisonResult.createFailure(path, actual, " is not a Set.");
}
return checkSetAcrossDatabases((Set<?>) actual, (Set<?>) expected, path, "Set");
} else if (expected instanceof Collection) {
if (!(actual instanceof Collection)) {
return ComparisonResult.createFailure(path, actual, " is not a Collection.");
}
return checkListAcrossDatabases((Collection<?>) actual, (Collection<?>) expected, path);
// Give Map.Entry special treatment to facilitate the use of Set comparison for verification
// of Map.
} else if (expected instanceof Map.Entry) {
if (!(actual instanceof Map.Entry)) {
return ComparisonResult.createFailure(path, actual, " is not a Map.Entry.");
}
// Check both the key and value. We can always ignore the path here, this should only be
// called from within a set comparison.
ComparisonResult result;
if ((result =
checkObjectAcrossDatabases(
((Map.Entry<?, ?>) actual).getKey(), ((Map.Entry<?, ?>) expected).getKey(), null))
.isFailure()) {
return result;
}
if ((result =
checkObjectAcrossDatabases(
((Map.Entry<?, ?>) actual).getValue(),
((Map.Entry<?, ?>) expected).getValue(),
null))
.isFailure()) {
return result;
}
} else {
// Since we know that the objects are not equal and since any other types can not be expected
// to contain DoNotCompare elements, this condition is always a failure.
return ComparisonResult.createFailure(path, actual, " is not equal to ", expected);
}
return ComparisonResult.createSuccess();
}
private static ComparisonResult checkSetAcrossDatabases(
Set<?> actual, Set<?> expected, String path, String type) {
// Unfortunately, we can't just check to see whether one set "contains" all of the elements of
// the other, as the cross database checks don't require strict equality. Instead we have to do
// an N^2 comparison to search for an equivalent element.
// Objects in expected that aren't in actual. We use "identity sets" here and below because we
// want to keep track of the _objects themselves_ rather than rely upon any overridable notion
// of equality.
Set<Object> missing = path != null ? Sets.newIdentityHashSet() : null;
// Objects from actual that have matching elements in expected.
Set<Object> found = Sets.newIdentityHashSet();
// Build missing and found.
for (Object expectedElem : expected) {
boolean gotMatch = false;
for (Object actualElem : actual) {
if (!checkObjectAcrossDatabases(actualElem, expectedElem, null).isFailure()) {
gotMatch = true;
// Add the element to the set of expected elements that were "found" in actual. If the
// element matches multiple elements in "expected," we have a basic problem with this
// kind of set that we'll want to know about.
if (!found.add(actualElem)) {
return ComparisonResult.createFailure(
path, "element ", actualElem, " matches multiple elements in ", expected);
}
break;
}
}
if (!gotMatch) {
if (path == null) {
return ComparisonResult.createFailure();
}
missing.add(expectedElem);
}
}
if (path != null) {
// Provide a detailed message consisting of any missing or unexpected items.
// Build a set of all objects in actual that don't have counterparts in expected.
Set<Object> unexpected =
actual.stream()
.filter(actualElem -> !found.contains(actualElem))
.collect(
Collector.of(
Sets::newIdentityHashSet,
Set::add,
(result, values) -> {
result.addAll(values);
return result;
}));
if (!missing.isEmpty() || !unexpected.isEmpty()) {
String message = type + " does not contain the expected contents.";
if (!missing.isEmpty()) {
message += " It is missing: " + formatItems(missing.iterator());
}
if (!unexpected.isEmpty()) {
message += " It contains additional elements: " + formatItems(unexpected.iterator());
}
return ComparisonResult.createFailure(path, message);
}
// We just need to check if there were any objects in "actual" that were not in "expected"
// (where "found" is a proxy for "expected").
} else if (!found.containsAll(actual)) {
return ComparisonResult.createFailure();
}
return ComparisonResult.createSuccess();
}
private static ComparisonResult checkListAcrossDatabases(
Collection<?> actual, Collection<?> expected, @Nullable String path) {
Iterator<?> actualIter = actual.iterator();
Iterator<?> expectedIter = expected.iterator();
int index = 0;
while (actualIter.hasNext() && expectedIter.hasNext()) {
Object actualItem = actualIter.next();
Object expectedItem = expectedIter.next();
ComparisonResult result =
checkObjectAcrossDatabases(
actualItem, expectedItem, path != null ? path + "[" + index + "]" : null);
if (result.isFailure()) {
return result;
}
++index;
}
if (actualIter.hasNext()) {
return ComparisonResult.createFailure(
path, "has additional items: ", formatItems(actualIter));
}
if (expectedIter.hasNext()) {
return ComparisonResult.createFailure(path, "missing items: ", formatItems(expectedIter));
}
return ComparisonResult.createSuccess();
}
/** Recursive helper for isEqualAcrossDatabases. */
private static ComparisonResult checkImmutableAcrossDatabases(
ImmutableObject actual, ImmutableObject expected, String path) {
Map<Field, Object> actualFields = filterFields(actual, ImmutableObject.DoNotCompare.class);
Map<Field, Object> expectedFields = filterFields(expected, ImmutableObject.DoNotCompare.class);
for (Map.Entry<Field, Object> entry : expectedFields.entrySet()) {
if (!actualFields.containsKey(entry.getKey())) {
return ComparisonResult.createFailure(path, "is missing field ", entry.getKey().getName());
}
// Verify that the field values are the same.
Object expectedFieldValue = entry.getValue();
Object actualFieldValue = actualFields.get(entry.getKey());
ComparisonResult result =
checkObjectAcrossDatabases(
actualFieldValue,
expectedFieldValue,
path != null ? path + "." + entry.getKey().getName() : null);
if (result.isFailure()) {
return result;
}
}
// Check for fields in actual that are not in expected.
for (Map.Entry<Field, Object> entry : actualFields.entrySet()) {
if (!expectedFields.containsKey(entry.getKey())) {
return ComparisonResult.createFailure(
path, "has additional field ", entry.getKey().getName());
}
}
return ComparisonResult.createSuccess();
}
private static String formatItems(Iterator<?> iter) {
return Joiner.on(", ").join(iter);
}
/** Encapsulates success/failure result in recursive comparison with optional error string. */
static class ComparisonResult {
boolean succeeded;
@@ -412,7 +150,6 @@ public final class ImmutableObjectSubject extends Subject {
// don't use ImmutableMap or a stream->collect model since we can have nulls
Map<Field, Object> result = new LinkedHashMap<>();
for (Map.Entry<Field, Object> entry : originalFields.entrySet()) {
// TODO(b/203685960): filter by @DoNotCompare instead.
if (entry.getKey().isAnnotationPresent(ImmutableObject.Insignificant.class)) {
continue;
}
@@ -422,28 +159,4 @@ public final class ImmutableObjectSubject extends Subject {
}
return result;
}
/** Filter out fields with the given annotation. */
private static Map<Field, Object> filterFields(
ImmutableObject original, Class<? extends Annotation> annotation) {
Map<Field, Object> originalFields = ModelUtils.getFieldValues(original);
// don't use ImmutableMap or a stream->collect model since we can have nulls
Map<Field, Object> result = new LinkedHashMap<>();
for (Map.Entry<Field, Object> entry : originalFields.entrySet()) {
// TODO(b/203685960): filter by @DoNotCompare instead.
if (!entry.getKey().isAnnotationPresent(annotation)
&& !entry.getKey().isAnnotationPresent(ImmutableObject.Insignificant.class)) {
// Perform any necessary substitutions.
if (entry.getKey().isAnnotationPresent(ImmutableObject.EmptySetToNull.class)
&& entry.getValue() != null
&& ((Set<?>) entry.getValue()).isEmpty()) {
result.put(entry.getKey(), null);
} else {
result.put(entry.getKey(), entry.getValue());
}
}
}
return result;
}
}
@@ -14,380 +14,17 @@
package google.registry.model;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.model.ImmutableObjectSubject.ComparisonResult;
import static google.registry.model.ImmutableObjectSubject.assertAboutImmutableObjects;
import static google.registry.model.ImmutableObjectSubject.checkObjectAcrossDatabases;
import static org.junit.Assert.assertThrows;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import java.util.regex.Pattern;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
public class ImmutableObjectSubjectTest {
// Unique id to assign to the "ignored" field so that it always gets a unique value.
private static int uniqueId = 0;
@Test
void testCrossDatabase_nulls() {
assertAboutImmutableObjects().that(null).isEqualAcrossDatabases(null);
assertAboutImmutableObjects()
.that(makeTestAtom(null))
.isEqualAcrossDatabases(makeTestAtom(null));
assertThat(checkObjectAcrossDatabases(null, makeTestAtom("foo"), null).isFailure()).isTrue();
assertThat(checkObjectAcrossDatabases(null, makeTestAtom("foo"), null).isFailure()).isTrue();
}
@Test
void testCrossDatabase_equalObjects() {
TestImmutableObject actual = makeTestObj();
assertAboutImmutableObjects().that(actual).isEqualAcrossDatabases(actual);
assertAboutImmutableObjects().that(actual).isEqualAcrossDatabases(makeTestObj());
assertThat(checkObjectAcrossDatabases(makeTestObj(), makeTestObj(), null).isFailure())
.isFalse();
}
@Test
void testCrossDatabase_simpleFieldFailure() {
AssertionError e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(makeTestObj().withStringField("bar")));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$TestImmutableObject.stringField:");
assertThat(
checkObjectAcrossDatabases(makeTestObj(), makeTestObj().withStringField(null), null)
.isFailure())
.isTrue();
}
@Test
void testCrossDatabase_nestedImmutableFailure() {
// Repeat the null checks to verify that the attribute path is preserved.
AssertionError e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(makeTestObj().withNested(null)));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$TestImmutableObject.nested:"
+ " expected null, got TestImmutableObject");
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj().withNested(null))
.isEqualAcrossDatabases(makeTestObj()));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$TestImmutableObject.nested:"
+ " expected TestImmutableObject");
assertThat(e).hasMessageThat().contains("got null.");
// Test with a field difference.
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(
makeTestObj().withNested(makeTestObj().withNested(null))));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$"
+ "TestImmutableObject.nested.stringField:");
assertThat(
checkObjectAcrossDatabases(makeTestObj(), makeTestObj().withNested(null), null)
.isFailure())
.isTrue();
}
@Test
void testCrossDatabase_listFailure() {
AssertionError e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(makeTestObj().withList(null)));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$" + "TestImmutableObject.list:");
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(
makeTestObj().withList(ImmutableList.of(makeTestAtom("wack")))));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$"
+ "TestImmutableObject.list[0].stringField:");
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(
makeTestObj()
.withList(
ImmutableList.of(
makeTestAtom("baz"),
makeTestAtom("bot"),
makeTestAtom("boq")))));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$"
+ "TestImmutableObject.list: missing items");
// Make sure multiple additional items get formatted nicely.
assertThat(e).hasMessageThat().contains("}, TestImmutableObject");
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(makeTestObj().withList(ImmutableList.of())));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$"
+ "TestImmutableObject.list: has additional items");
assertThat(
checkObjectAcrossDatabases(
makeTestObj(),
makeTestObj()
.withList(ImmutableList.of(makeTestAtom("baz"), makeTestAtom("gauze"))),
null)
.isFailure())
.isTrue();
assertThat(
checkObjectAcrossDatabases(
makeTestObj(), makeTestObj().withList(ImmutableList.of()), null)
.isFailure())
.isTrue();
assertThat(
checkObjectAcrossDatabases(
makeTestObj(),
makeTestObj().withList(ImmutableList.of(makeTestAtom("gauze"))),
null)
.isFailure())
.isTrue();
}
@Test
void testCrossDatabase_setFailure() {
AssertionError e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(makeTestObj().withSet(null)));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$"
+ "TestImmutableObject.set: expected null, got ");
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(
makeTestObj().withSet(ImmutableSet.of(makeTestAtom("jim")))));
assertThat(e)
.hasMessageThat()
.containsMatch(
Pattern.compile(
"Set does not contain the expected contents. "
+ "It is missing: .*jim.* It contains additional elements: .*bob",
Pattern.DOTALL));
// Trickery here to verify that multiple items that both match existing items in the set trigger
// an error: we can add two of the same items because equality for purposes of the set includes
// the DoNotCompare field.
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(
makeTestObj()
.withSet(ImmutableSet.of(makeTestAtom("bob"), makeTestAtom("bob")))));
assertThat(e)
.hasMessageThat()
.containsMatch(
Pattern.compile(
"At google.registry.model.ImmutableObjectSubjectTest\\$TestImmutableObject.set: "
+ "element .*bob.* matches multiple elements in .*bob.*bob",
Pattern.DOTALL));
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(
makeTestObj()
.withSet(ImmutableSet.of(makeTestAtom("bob"), makeTestAtom("bob"))))
.isEqualAcrossDatabases(makeTestObj()));
assertThat(e)
.hasMessageThat()
.containsMatch(
Pattern.compile(
"At google.registry.model.ImmutableObjectSubjectTest\\$TestImmutableObject.set: "
+ "Set does not contain the expected contents. It contains additional "
+ "elements: .*bob",
Pattern.DOTALL));
assertThat(
checkObjectAcrossDatabases(
makeTestObj(),
makeTestObj()
.withSet(ImmutableSet.of(makeTestAtom("bob"), makeTestAtom("robert"))),
null)
.isFailure())
.isTrue();
assertThat(
checkObjectAcrossDatabases(
makeTestObj(), makeTestObj().withSet(ImmutableSet.of()), null)
.isFailure())
.isTrue();
assertThat(
checkObjectAcrossDatabases(
makeTestObj(),
makeTestObj()
.withSet(ImmutableSet.of(makeTestAtom("bob"), makeTestAtom("bob"))),
null)
.isFailure())
.isTrue();
// We don't test the case where actual's set contains multiple items matching the single item in
// the expected set: that path is the same as the "additional contents" path.
}
@Test
void testCrossDatabase_mapFailure() {
AssertionError e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(makeTestObj().withMap(null)));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$"
+ "TestImmutableObject.map: expected null, got ");
e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(makeTestObj())
.isEqualAcrossDatabases(
makeTestObj()
.withMap(ImmutableMap.of(makeTestAtom("difk"), makeTestAtom("difv")))));
assertThat(e)
.hasMessageThat()
.containsMatch(
Pattern.compile(
"Map does not contain the expected contents. "
+ "It is missing: .*difk.*difv.* It contains additional elements: .*key.*val",
Pattern.DOTALL));
assertThat(
checkObjectAcrossDatabases(
makeTestObj(),
makeTestObj()
.withMap(
ImmutableMap.of(
makeTestAtom("key"), makeTestAtom("val"),
makeTestAtom("otherk"), makeTestAtom("otherv"))),
null)
.isFailure())
.isTrue();
assertThat(
checkObjectAcrossDatabases(
makeTestObj(), makeTestObj().withMap(ImmutableMap.of()), null)
.isFailure())
.isTrue();
}
@Test
void testCrossDatabase_typeChecks() {
ComparisonResult result = checkObjectAcrossDatabases("blech", makeTestObj(), "xxx");
assertThat(result.getMessage()).isEqualTo("At xxx: blech is not an immutable object.");
assertThat(result.isFailure()).isTrue();
assertThat(checkObjectAcrossDatabases("blech", makeTestObj(), null).isFailure()).isTrue();
result = checkObjectAcrossDatabases("blech", ImmutableMap.of(), "xxx");
assertThat(result.getMessage()).isEqualTo("At xxx: blech is not a Map.");
assertThat(result.isFailure()).isTrue();
assertThat(checkObjectAcrossDatabases("blech", ImmutableMap.of(), null).isFailure()).isTrue();
result = checkObjectAcrossDatabases("blech", ImmutableList.of(), "xxx");
assertThat(result.getMessage()).isEqualTo("At xxx: blech is not a Collection.");
assertThat(result.isFailure()).isTrue();
assertThat(checkObjectAcrossDatabases("blech", ImmutableList.of(), null).isFailure()).isTrue();
for (ImmutableMap.Entry<String, String> entry : ImmutableMap.of("foo", "bar").entrySet()) {
result = checkObjectAcrossDatabases("blech", entry, "xxx");
assertThat(result.getMessage()).isEqualTo("At xxx: blech is not a Map.Entry.");
assertThat(result.isFailure()).isTrue();
assertThat(checkObjectAcrossDatabases("blech", entry, "xxx").isFailure()).isTrue();
}
}
@Test
void testCrossDatabase_checkAdditionalFields() {
AssertionError e =
assertThrows(
AssertionError.class,
() ->
assertAboutImmutableObjects()
.that(DerivedImmutableObject.create())
.isEqualAcrossDatabases(makeTestAtom(null)));
assertThat(e)
.hasMessageThat()
.contains(
"At google.registry.model.ImmutableObjectSubjectTest$DerivedImmutableObject: "
+ "has additional field extraField");
assertThat(
checkObjectAcrossDatabases(DerivedImmutableObject.create(), makeTestAtom(null), null)
.isFailure())
.isTrue();
}
@Test
void testHasCorrectHashValue() {
TestImmutableObject object = makeTestObj();
@@ -421,8 +58,6 @@ public class ImmutableObjectSubjectTest {
ImmutableSet<TestImmutableObject> set;
ImmutableMap<TestImmutableObject, TestImmutableObject> map;
@ImmutableObject.DoNotCompare int ignored;
static TestImmutableObject create(
String stringField,
TestImmutableObject nested,
@@ -435,7 +70,6 @@ public class ImmutableObjectSubjectTest {
instance.list = list;
instance.set = set;
instance.map = map;
instance.ignored = ++uniqueId;
return instance;
}
@@ -65,7 +65,6 @@ public class UpdateAutoTimestampTest {
tm().insert(object);
return tm().getTransactionTime();
});
tm().clearSessionCache();
assertThat(reload().updateTime.getTimestamp()).isEqualTo(transactionTime);
}
@@ -106,7 +105,6 @@ public class UpdateAutoTimestampTest {
tm().insert(object);
return tm().getTransactionTime();
});
tm().clearSessionCache();
assertThat(reload().updateTime.getTimestamp()).isEqualTo(transactionTime);
}
@@ -17,7 +17,7 @@ package google.registry.model.domain;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.model.ImmutableObjectSubject.assertAboutImmutableObjects;
import static google.registry.model.domain.token.AllocationToken.TokenStatus.NOT_STARTED;
import static google.registry.model.domain.token.AllocationToken.TokenType.UNLIMITED_USE;
import static google.registry.model.domain.token.AllocationToken.TokenType.PACKAGE;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.insertInDb;
@@ -39,6 +39,7 @@ import com.googlecode.objectify.Key;
import google.registry.model.billing.BillingEvent;
import google.registry.model.billing.BillingEvent.Flag;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.billing.BillingEvent.RenewalPriceBehavior;
import google.registry.model.contact.Contact;
import google.registry.model.domain.DesignatedContact.Type;
import google.registry.model.domain.launch.LaunchNotice;
@@ -144,13 +145,11 @@ public class DomainSqlTest {
allocationToken =
new AllocationToken.Builder()
.setToken("abc123Unlimited")
.setTokenType(UNLIMITED_USE)
.setTokenType(PACKAGE)
.setCreationTimeForTest(DateTime.parse("2010-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("dev", "app"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar, NewRegistrar"))
.setDiscountFraction(0.5)
.setDiscountPremiums(true)
.setDiscountYears(3)
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setTokenStatusTransitions(
ImmutableSortedMap.<DateTime, TokenStatus>naturalOrder()
.put(START_OF_TIME, NOT_STARTED)
@@ -168,8 +167,8 @@ public class DomainSqlTest {
@Test
void testDomainBasePersistenceWithCurrentPackageToken() {
domain = domain.asBuilder().setCurrentPackageToken(allocationToken.createVKey()).build();
persistResource(allocationToken);
domain = domain.asBuilder().setCurrentPackageToken(allocationToken.createVKey()).build();
persistDomain();
assertEqualDomainExcept(loadByKey(domain.createVKey()));
}
@@ -180,13 +179,6 @@ public class DomainSqlTest {
assertThrowForeignKeyViolation(() -> insertInDb(contact, contact2, domain));
}
@Test
void testCurrentPackageTokenForeignKeyConstraints() {
// Persist the domain without the associated allocation token object.
domain = domain.asBuilder().setCurrentPackageToken(allocationToken.createVKey()).build();
assertThrowForeignKeyViolation(() -> persistDomain());
}
@Test
void testContactForeignKeyConstraints() {
// Persist the domain without the associated contact objects.
@@ -19,6 +19,8 @@ import static com.google.common.collect.Iterables.getOnlyElement;
import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.model.EppResourceUtils.loadByForeignKey;
import static google.registry.model.domain.token.AllocationToken.TokenType.PACKAGE;
import static google.registry.model.domain.token.AllocationToken.TokenType.SINGLE_USE;
import static google.registry.testing.DatabaseHelper.cloneAndSetAutoTimestamps;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.insertInDb;
@@ -46,11 +48,13 @@ import google.registry.model.ImmutableObjectSubject;
import google.registry.model.billing.BillingEvent;
import google.registry.model.billing.BillingEvent.Flag;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.billing.BillingEvent.RenewalPriceBehavior;
import google.registry.model.contact.Contact;
import google.registry.model.domain.DesignatedContact.Type;
import google.registry.model.domain.launch.LaunchNotice;
import google.registry.model.domain.rgp.GracePeriodStatus;
import google.registry.model.domain.secdns.DomainDsData;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.eppcommon.AuthInfo.PasswordAuth;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.eppcommon.Trid;
@@ -978,4 +982,50 @@ public class DomainTest {
assertThat(domain.getBillingContact()).isNull();
assertThat(domain.getTechContact()).isNull();
}
@Test
void testFail_currentPackageTokenWrongPackageType() {
AllocationToken allocationToken =
persistResource(
new AllocationToken.Builder().setToken("abc123").setTokenType(SINGLE_USE).build());
IllegalArgumentException thrown =
assertThrows(
IllegalArgumentException.class,
() -> domain.asBuilder().setCurrentPackageToken(allocationToken.createVKey()).build());
assertThat(thrown)
.hasMessageThat()
.isEqualTo("The currentPackageToken must have a PACKAGE TokenType");
}
@Test
void testFailure_packageTokenDoesNotExist() {
AllocationToken allocationToken =
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(PACKAGE)
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.build();
IllegalArgumentException thrown =
assertThrows(
IllegalArgumentException.class,
() -> domain.asBuilder().setCurrentPackageToken(allocationToken.createVKey()).build());
assertThat(thrown).hasMessageThat().isEqualTo("The package token abc123 does not exist");
}
@Test
void testSuccess_removeCurrentPackageToken() {
AllocationToken allocationToken =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(PACKAGE)
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.build());
domain = domain.asBuilder().setCurrentPackageToken(allocationToken.createVKey()).build();
assertThat(domain.getCurrentPackageToken().get()).isEqualTo(allocationToken.createVKey());
domain = domain.asBuilder().setCurrentPackageToken(null).build();
assertThat(domain.getCurrentPackageToken()).isEmpty();
}
}
@@ -228,6 +228,18 @@ public class AllocationTokenTest extends EntityTestCase {
.isEqualTo("Package tokens must have renewalPriceBehavior set to SPECIFIED");
}
@Test
void testFail_packageTokenDiscountPremium() {
AllocationToken.Builder builder =
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(TokenType.PACKAGE)
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountPremiums(true);
IllegalArgumentException thrown = assertThrows(IllegalArgumentException.class, builder::build);
assertThat(thrown).hasMessageThat().isEqualTo("Package tokens cannot discount premium names");
}
@Test
void testBuild_DomainNameWithLessThanTwoParts() {
IllegalArgumentException thrown =
@@ -1,86 +0,0 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence.converter;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.testing.DatabaseHelper.insertInDb;
import google.registry.model.ImmutableObject;
import google.registry.persistence.VKey;
import google.registry.persistence.WithLongVKey;
import google.registry.testing.AppEngineExtension;
import javax.persistence.Entity;
import javax.persistence.Id;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Test SQL persistence of VKey. */
public class LongVKeyConverterTest {
@RegisterExtension
public final AppEngineExtension appEngineExtension =
new AppEngineExtension.Builder()
.withCloudSql()
.withoutCannedData()
.withJpaUnitTestEntities(
TestLongEntity.class,
VKeyConverter_LongType.class,
VKeyConverter_CompositeLongType.class)
.withOfyTestEntities(TestLongEntity.class, CompositeKeyTestLongEntity.class)
.build();
@Test
void testRoundTrip() {
TestLongEntity original =
new TestLongEntity(
VKey.createSql(TestLongEntity.class, 10L),
VKey.createSql(CompositeKeyTestLongEntity.class, 20L));
insertInDb(original);
TestLongEntity retrieved =
jpaTm().transact(() -> jpaTm().getEntityManager().find(TestLongEntity.class, "id"));
assertThat(retrieved.number.getSqlKey()).isEqualTo(10L);
assertThat(retrieved.number.getOfyKey().getId()).isEqualTo(10L);
assertThat(retrieved.composite.getSqlKey()).isEqualTo(20L);
assertThat(retrieved.composite.maybeGetOfyKey().isPresent()).isFalse();
}
@Entity(name = "TestLongEntity")
@com.googlecode.objectify.annotation.Entity
@WithLongVKey(classNameSuffix = "LongType")
static class TestLongEntity extends ImmutableObject {
@com.googlecode.objectify.annotation.Id @Id String id = "id";
VKey<TestLongEntity> number;
VKey<CompositeKeyTestLongEntity> composite;
TestLongEntity(VKey<TestLongEntity> number, VKey<CompositeKeyTestLongEntity> composite) {
this.number = number;
this.composite = composite;
}
/** Default constructor, needed for hibernate. */
public TestLongEntity() {}
}
@Entity(name = "CompositeKeyTestLongEntity")
@com.googlecode.objectify.annotation.Entity
@WithLongVKey(classNameSuffix = "CompositeLongType", compositeKey = true)
static class CompositeKeyTestLongEntity {
@com.googlecode.objectify.annotation.Id @Id String id = "id";
}
}
@@ -1,91 +0,0 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence.converter;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.testing.DatabaseHelper.insertInDb;
import google.registry.model.ImmutableObject;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.testing.AppEngineExtension;
import javax.persistence.Entity;
import javax.persistence.Id;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Test SQL persistence of VKey. */
public class StringVKeyConverterTest {
@RegisterExtension
public final AppEngineExtension appEngineExtension =
new AppEngineExtension.Builder()
.withCloudSql()
.withoutCannedData()
.withJpaUnitTestEntities(
TestStringEntity.class,
VKeyConverter_StringType.class,
VKeyConverter_CompositeStringType.class)
.withOfyTestEntities(TestStringEntity.class, CompositeKeyTestStringEntity.class)
.build();
@Test
void testRoundTrip() {
TestStringEntity original =
new TestStringEntity(
"TheRealSpartacus",
VKey.createSql(TestStringEntity.class, "ImSpartacus!"),
VKey.createSql(CompositeKeyTestStringEntity.class, "NoImSpartacus!"));
insertInDb(original);
TestStringEntity retrieved =
jpaTm()
.transact(
() -> jpaTm().getEntityManager().find(TestStringEntity.class, "TheRealSpartacus"));
assertThat(retrieved.other.getSqlKey()).isEqualTo("ImSpartacus!");
assertThat(retrieved.other.getOfyKey().getName()).isEqualTo("ImSpartacus!");
assertThat(retrieved.composite.getSqlKey()).isEqualTo("NoImSpartacus!");
assertThat(retrieved.composite.maybeGetOfyKey().isPresent()).isFalse();
}
@Entity(name = "TestStringEntity")
@com.googlecode.objectify.annotation.Entity
@WithStringVKey(classNameSuffix = "StringType")
static class TestStringEntity extends ImmutableObject {
@com.googlecode.objectify.annotation.Id @Id String id;
VKey<TestStringEntity> other;
VKey<CompositeKeyTestStringEntity> composite;
TestStringEntity(
String id, VKey<TestStringEntity> other, VKey<CompositeKeyTestStringEntity> composite) {
this.id = id;
this.other = other;
this.composite = composite;
}
/** Default constructor, needed for hibernate. */
public TestStringEntity() {}
}
@Entity(name = "CompositeKeyTestStringEntity")
@com.googlecode.objectify.annotation.Entity
@WithStringVKey(classNameSuffix = "CompositeStringType", compositeKey = true)
static class CompositeKeyTestStringEntity {
@com.googlecode.objectify.annotation.Id @Id String id = "id";
}
}
@@ -0,0 +1,103 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence.converter;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.testing.DatabaseHelper.insertInDb;
import google.registry.model.ImmutableObject;
import google.registry.persistence.VKey;
import google.registry.persistence.WithVKey;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaUnitTestExtension;
import javax.persistence.Entity;
import javax.persistence.Id;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Test SQL persistence of {@link VKey}. */
public class VKeyConverterTest {
@RegisterExtension
public final JpaUnitTestExtension jpa =
new JpaTestExtensions.Builder()
.withoutCannedData()
.withEntityClass(
TestEntity.class,
TestStringEntity.class,
TestLongEntity.class,
VKeyConverter_TestStringEntity.class,
VKeyConverter_TestLongEntity.class)
.buildUnitTestExtension();
@Test
void testRoundTrip() {
TestStringEntity stringEntity = new TestStringEntity("TheRealSpartacus");
VKey<TestStringEntity> stringKey = VKey.createSql(TestStringEntity.class, "TheRealSpartacus");
TestLongEntity longEntity = new TestLongEntity(300L);
VKey<TestLongEntity> longKey = VKey.createSql(TestLongEntity.class, 300L);
TestEntity original = new TestEntity(1984L, stringKey, longKey);
insertInDb(stringEntity, longEntity, original);
TestEntity retrieved =
jpaTm().transact(() -> jpaTm().getEntityManager().find(TestEntity.class, 1984L));
assertThat(retrieved.stringKey).isEqualTo(stringKey);
assertThat(retrieved.longKey).isEqualTo(longKey);
}
@Entity(name = "TestStringEntity")
@WithVKey(String.class)
protected static class TestStringEntity extends ImmutableObject {
@Id String id;
TestStringEntity(String id) {
this.id = id;
}
/** Default constructor, needed for hibernate. */
public TestStringEntity() {}
}
@Entity(name = "TestLongEntity")
@WithVKey(Long.class)
protected static class TestLongEntity extends ImmutableObject {
@Id Long id;
TestLongEntity(Long id) {
this.id = id;
}
/** Default constructor, needed for hibernate. */
public TestLongEntity() {}
}
@Entity(name = "TestEntity")
@WithVKey(String.class)
protected static class TestEntity extends ImmutableObject {
@Id Long id;
VKey<TestStringEntity> stringKey;
VKey<TestLongEntity> longKey;
TestEntity(Long id, VKey<TestStringEntity> stringKey, VKey<TestLongEntity> longKey) {
this.id = id;
this.stringKey = stringKey;
this.longKey = longKey;
}
/** Default constructor, needed for hibernate. */
public TestEntity() {}
}
}
@@ -38,7 +38,6 @@ import google.registry.testing.DatabaseHelper;
import google.registry.testing.FakeClock;
import java.io.Serializable;
import java.math.BigInteger;
import java.sql.SQLException;
import java.util.NoSuchElementException;
import java.util.function.Supplier;
import javax.persistence.Entity;
@@ -47,7 +46,6 @@ import javax.persistence.Id;
import javax.persistence.IdClass;
import javax.persistence.OptimisticLockException;
import javax.persistence.RollbackException;
import org.hibernate.exception.JDBCConnectionException;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -205,59 +203,6 @@ class JpaTransactionManagerImplTest {
verify(spyJpaTm, times(6)).delete(theEntityKey);
}
@Test
void transactNewReadOnly_retriesJdbcConnectionExceptions() {
JpaTransactionManager spyJpaTm = spy(jpaTm());
doThrow(JDBCConnectionException.class).when(spyJpaTm).loadByKey(any(VKey.class));
spyJpaTm.transact(() -> spyJpaTm.insert(theEntity));
assertThrows(
JDBCConnectionException.class,
() -> spyJpaTm.transactNewReadOnly(() -> spyJpaTm.loadByKey(theEntityKey)));
verify(spyJpaTm, times(3)).loadByKey(theEntityKey);
Supplier<Runnable> supplier =
() -> {
Runnable work = () -> spyJpaTm.loadByKey(theEntityKey);
work.run();
return null;
};
assertThrows(JDBCConnectionException.class, () -> spyJpaTm.transactNewReadOnly(supplier));
verify(spyJpaTm, times(6)).loadByKey(theEntityKey);
}
@Test
void transactNewReadOnly_retriesNestedJdbcConnectionExceptions() {
JpaTransactionManager spyJpaTm = spy(jpaTm());
doThrow(
new RuntimeException(
new JDBCConnectionException("connection exception", new SQLException())))
.when(spyJpaTm)
.loadByKey(any(VKey.class));
spyJpaTm.transact(() -> spyJpaTm.insert(theEntity));
assertThrows(
RuntimeException.class,
() -> spyJpaTm.transactNewReadOnly(() -> spyJpaTm.loadByKey(theEntityKey)));
verify(spyJpaTm, times(3)).loadByKey(theEntityKey);
Supplier<Runnable> supplier =
() -> {
Runnable work = () -> spyJpaTm.loadByKey(theEntityKey);
work.run();
return null;
};
assertThrows(RuntimeException.class, () -> spyJpaTm.transactNewReadOnly(supplier));
verify(spyJpaTm, times(6)).loadByKey(theEntityKey);
}
@Test
void doTransactionless_retriesJdbcConnectionExceptions() {
JpaTransactionManager spyJpaTm = spy(jpaTm());
doThrow(JDBCConnectionException.class).when(spyJpaTm).loadByKey(any(VKey.class));
spyJpaTm.transact(() -> spyJpaTm.insert(theEntity));
assertThrows(
RuntimeException.class,
() -> spyJpaTm.doTransactionless(() -> spyJpaTm.loadByKey(theEntityKey)));
verify(spyJpaTm, times(3)).loadByKey(theEntityKey);
}
@Test
void insert_throwsExceptionIfEntityExists() {
assertThat(existsInDb(theEntity)).isFalse();
@@ -787,6 +732,7 @@ class JpaTransactionManagerImplTest {
String name;
int age;
@SuppressWarnings("unused")
private CompoundId() {}
private CompoundId(String name, int age) {
@@ -834,6 +780,7 @@ class JpaTransactionManagerImplTest {
String nameField;
int ageField;
@SuppressWarnings("unused")
private NamedCompoundId() {}
private NamedCompoundId(String nameField, int ageField) {
@@ -72,7 +72,7 @@ public class QueryComposerTest {
tm().createQueryComposer(TestEntity.class)
.where("name", Comparator.GT, "bravo")
.first()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.get()))
.isEqualTo(charlie);
assertThat(
@@ -81,7 +81,7 @@ public class QueryComposerTest {
tm().createQueryComposer(TestEntity.class)
.where("name", Comparator.GTE, "charlie")
.first()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.get()))
.isEqualTo(charlie);
assertThat(
@@ -90,7 +90,7 @@ public class QueryComposerTest {
tm().createQueryComposer(TestEntity.class)
.where("name", Comparator.LT, "bravo")
.first()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.get()))
.isEqualTo(alpha);
assertThat(
@@ -99,7 +99,7 @@ public class QueryComposerTest {
tm().createQueryComposer(TestEntity.class)
.where("name", Comparator.LTE, "alpha")
.first()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.get()))
.isEqualTo(alpha);
}
@@ -120,7 +120,7 @@ public class QueryComposerTest {
assertThat(
tm().transact(
() ->
QueryComposerTest.assertDetachedIfJpa(
DatabaseHelper.assertDetachedFromEntityManager(
tm().createQueryComposer(TestEntity.class)
.where("name", Comparator.EQ, "alpha")
.getSingleResult())))
@@ -169,7 +169,7 @@ public class QueryComposerTest {
.createQueryComposer(TestEntity.class)
.where("name", Comparator.GT, "alpha")
.stream()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.collect(toImmutableList())))
.containsExactly(bravo, charlie);
assertThat(
@@ -179,7 +179,7 @@ public class QueryComposerTest {
.createQueryComposer(TestEntity.class)
.where("name", Comparator.GTE, "bravo")
.stream()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.collect(toImmutableList())))
.containsExactly(bravo, charlie);
assertThat(
@@ -189,7 +189,7 @@ public class QueryComposerTest {
.createQueryComposer(TestEntity.class)
.where("name", Comparator.LT, "charlie")
.stream()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.collect(toImmutableList())))
.containsExactly(alpha, bravo);
assertThat(
@@ -199,7 +199,7 @@ public class QueryComposerTest {
.createQueryComposer(TestEntity.class)
.where("name", Comparator.LTE, "bravo")
.stream()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.collect(toImmutableList())))
.containsExactly(alpha, bravo);
}
@@ -223,7 +223,7 @@ public class QueryComposerTest {
tm().createQueryComposer(TestEntity.class)
.where("val", Comparator.EQ, 2)
.first()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.get()))
.isEqualTo(bravo);
}
@@ -238,7 +238,7 @@ public class QueryComposerTest {
.where("val", Comparator.GT, 1)
.orderBy("val")
.stream()
.map(QueryComposerTest::assertDetachedIfJpa)
.map(DatabaseHelper::assertDetachedFromEntityManager)
.collect(toImmutableList())))
.containsExactly(bravo, alpha);
}
@@ -319,13 +319,6 @@ public class QueryComposerTest {
.isEmpty();
}
private static <T> T assertDetachedIfJpa(T entity) {
if (!tm().isOfy()) {
return DatabaseHelper.assertDetachedFromEntityManager(entity);
}
return entity;
}
@javax.persistence.Entity
@Entity(name = "QueryComposerTestEntity")
private static class TestEntity extends ImmutableObject {
@@ -128,31 +128,6 @@ public class ReplicaSimulatingJpaTransactionManager implements JpaTransactionMan
transact(work);
}
@Override
public <T> T transactNew(Supplier<T> work) {
return transact(work);
}
@Override
public void transactNew(Runnable work) {
transact(work);
}
@Override
public <T> T transactNewReadOnly(Supplier<T> work) {
return transact(work);
}
@Override
public void transactNewReadOnly(Runnable work) {
transact(work);
}
@Override
public <T> T doTransactionless(Supplier<T> work) {
return delegate.doTransactionless(work);
}
@Override
public DateTime getTransactionTime() {
return delegate.getTransactionTime();
@@ -285,16 +260,6 @@ public class ReplicaSimulatingJpaTransactionManager implements JpaTransactionMan
return delegate.createQueryComposer(entity);
}
@Override
public void clearSessionCache() {
delegate.clearSessionCache();
}
@Override
public boolean isOfy() {
return delegate.isOfy();
}
@Override
public <T> void assertDelete(VKey<T> key) {
delegate.assertDelete(key);
@@ -111,22 +111,6 @@ public class TransactionManagerTest {
assertEntityExists(theEntity);
}
@Test
void transactNew_succeeds() {
assertEntityNotExist(theEntity);
tm().transactNew(() -> tm().insert(theEntity));
assertEntityExists(theEntity);
}
@Test
void transactNewReadOnly_succeeds() {
assertEntityNotExist(theEntity);
tm().transact(() -> tm().insert(theEntity));
assertEntityExists(theEntity);
TestEntity persisted = tm().transactNewReadOnly(() -> tm().loadByKey(theEntity.key()));
assertThat(persisted).isEqualTo(theEntity);
}
@Test
void saveNew_succeeds() {
assertEntityNotExist(theEntity);
@@ -15,7 +15,6 @@
package google.registry.rde;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.loadByKey;
import static google.registry.testing.DatabaseHelper.persistResource;
@@ -79,7 +78,6 @@ public class EscrowTaskRunnerTest {
runner.lockRunAndRollForward(
task, registry, standardSeconds(30), CursorType.RDE_STAGING, standardDays(1));
verify(task).runWithLock(DateTime.parse("2006-06-06TZ"));
tm().clearSessionCache();
Cursor cursor = loadByKey(Cursor.createScopedVKey(CursorType.RDE_STAGING, registry));
assertThat(cursor.getCursorTime()).isEqualTo(DateTime.parse("2006-06-07TZ"));
}
@@ -15,7 +15,6 @@
package google.registry.reporting.icann;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.createTlds;
import static google.registry.testing.DatabaseHelper.loadByKey;
import static google.registry.testing.DatabaseHelper.persistResource;
@@ -179,7 +178,6 @@ class IcannReportingUploadActionTest {
when(mockReporter.send(PAYLOAD_SUCCESS, "tld-activity-200606.csv")).thenReturn(true);
IcannReportingUploadAction action = createAction();
action.run();
tm().clearSessionCache();
Cursor cursor =
loadByKey(Cursor.createScopedVKey(CursorType.ICANN_UPLOAD_ACTIVITY, Registry.get("tld")));
assertThat(cursor.getCursorTime()).isEqualTo(DateTime.parse("2006-08-01TZ"));
@@ -190,7 +188,6 @@ class IcannReportingUploadActionTest {
clock.setTo(DateTime.parse("2006-5-01T00:30:00Z"));
IcannReportingUploadAction action = createAction();
action.run();
tm().clearSessionCache();
verifyNoMoreInteractions(mockReporter);
verifyNoMoreInteractions(emailService);
}
@@ -238,7 +235,6 @@ class IcannReportingUploadActionTest {
void testFailure_cursorIsNotAdvancedForward() throws Exception {
runTest_nonRetryableException(
new IOException("Your IP address 25.147.130.158 is not allowed to connect"));
tm().clearSessionCache();
Cursor cursor =
loadByKey(Cursor.createScopedVKey(CursorType.ICANN_UPLOAD_ACTIVITY, Registry.get("tld")));
assertThat(cursor.getCursorTime()).isEqualTo(DateTime.parse("2006-07-01TZ"));
@@ -249,7 +245,6 @@ class IcannReportingUploadActionTest {
clock.setTo(DateTime.parse("2006-05-01T00:30:00Z"));
IcannReportingUploadAction action = createAction();
action.run();
tm().clearSessionCache();
Cursor cursor =
loadByKey(Cursor.createScopedVKey(CursorType.ICANN_UPLOAD_ACTIVITY, Registry.get("foo")));
assertThat(cursor.getCursorTime()).isEqualTo(DateTime.parse("2006-07-01TZ"));
@@ -100,6 +100,24 @@ public class Spec11RegistrarThreatMatchesParserTest {
assertThat(objectWithExtraFields).isEqualTo(objectWithoutExtraFields);
}
@Test
void testSuccess_worksWithOutdatedField() throws Exception {
ThreatMatch objectWithOutdatedField =
ThreatMatch.fromJSON(
new JSONObject(
ImmutableMap.of(
"threatType", "MALWARE",
"fullyQualifiedDomainName", "c.com")));
ThreatMatch objectWithoutOutdatedFields =
ThreatMatch.fromJSON(
new JSONObject(
ImmutableMap.of(
"threatType", "MALWARE",
"domainName", "c.com")));
assertThat(objectWithOutdatedField).isEqualTo(objectWithoutOutdatedFields);
}
/** The expected contents of the sample spec11 report file */
public static ImmutableSet<RegistrarThreatMatches> sampleThreatMatches() throws Exception {
return ImmutableSet.of(getMatchA(), getMatchB());
@@ -40,7 +40,7 @@ import com.google.common.collect.Sets;
import com.google.common.io.Files;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.ObjectifyFilter;
import google.registry.model.IdService;
import google.registry.model.IdService.SelfAllocatedIdSupplier;
import google.registry.model.ofy.ObjectifyService;
import google.registry.model.registrar.Registrar;
import google.registry.model.registrar.Registrar.State;
@@ -441,7 +441,7 @@ public final class AppEngineExtension implements BeforeEachCallback, AfterEachCa
ObjectifyService.initOfy();
// Reset id allocation in ObjectifyService so that ids are deterministic in tests.
IdService.resetSelfAllocatedId();
SelfAllocatedIdSupplier.getInstance().reset();
this.ofyTestEntities.forEach(AppEngineExtension::register);
}
@@ -32,7 +32,6 @@ import static google.registry.model.IdService.allocateId;
import static google.registry.model.ImmutableObjectSubject.assertAboutImmutableObjects;
import static google.registry.model.ImmutableObjectSubject.immutableObjectCorrespondence;
import static google.registry.model.ResourceTransferUtils.createTransferResponse;
import static google.registry.model.ofy.ObjectifyService.auditedOfy;
import static google.registry.model.tld.Registry.TldState.GENERAL_AVAILABILITY;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
@@ -319,7 +318,7 @@ public final class DatabaseHelper {
final Domain persistedDomain = persistResource(domain);
// Calls {@link LordnTaskUtils#enqueueDomainTask} wrapped in a transaction so that the
// transaction time is set correctly.
tm().transactNew(() -> LordnTaskUtils.enqueueDomainTask(persistedDomain));
tm().transact(() -> LordnTaskUtils.enqueueDomainTask(persistedDomain));
maybeAdvanceClock();
return persistedDomain;
}
@@ -399,7 +398,7 @@ public final class DatabaseHelper {
toImmutableMap(Map.Entry::getKey, entry -> entry.getValue().getValue())))
.build();
// Since we used to persist a PremiumList to Datastore here, it is necessary to allocate an ID
// here to prevent breaking some of the hard-coded flow tests. IDs in tests are allocated in a
// here to prevent breaking some hard-coded flow tests. IDs in tests are allocated in a
// strictly increasing sequence, if we don't pad out the ID here, we would have to renumber
// hundreds of unit tests.
allocateId();
@@ -990,11 +989,6 @@ public final class DatabaseHelper {
.isNotInstanceOf(Buildable.Builder.class);
tm().transact(() -> tm().put(resource));
maybeAdvanceClock();
// Force the session cache to be cleared so that when we read the resource back, we read from
// Datastore and not from the session cache. This is needed to trigger Objectify's load process
// (unmarshalling entity protos to POJOs, nulling out empty collections, calling @OnLoad
// methods, etc.) which is bypassed for entities loaded from the session cache.
tm().clearSessionCache();
return tm().transact(() -> tm().loadByEntity(resource));
}
@@ -1007,9 +1001,6 @@ public final class DatabaseHelper {
}
tm().transact(() -> resources.forEach(e -> tm().put(e)));
maybeAdvanceClock();
// Force the session to be cleared so that when we read it back, we read from Datastore
// and not from the transaction's session cache.
tm().clearSessionCache();
}
/**
@@ -1017,8 +1008,6 @@ public final class DatabaseHelper {
*
* <p>This was coded for testing RDE since its queries depend on the associated entries.
*
* <p><b>Warning:</b> If you call this multiple times in a single test, you need to inject Ofy's
* clock field and forward it by a millisecond between each subsequent call.
*
* @see #persistResource(ImmutableObject)
*/
@@ -1035,17 +1024,16 @@ public final class DatabaseHelper {
.build());
});
maybeAdvanceClock();
tm().clearSessionCache();
return tm().transact(() -> tm().loadByEntity(resource));
}
/** Returns all of the history entries that are parented off the given EppResource. */
/** Returns all the history entries that are parented off the given EppResource. */
public static List<HistoryEntry> getHistoryEntries(EppResource resource) {
return HistoryEntryDao.loadHistoryObjectsForResource(resource.createVKey());
}
/**
* Returns all of the history entries that are parented off the given EppResource, cast to the
* Returns all the history entries that are parented off the given EppResource, cast to the
* corresponding subclass.
*/
public static <T extends HistoryEntry> List<T> getHistoryEntries(
@@ -1054,7 +1042,7 @@ public final class DatabaseHelper {
}
/**
* Returns all of the history entries that are parented off the given EppResource with the given
* Returns all the history entries that are parented off the given EppResource with the given
* type.
*/
public static ImmutableList<HistoryEntry> getHistoryEntriesOfType(
@@ -1065,8 +1053,8 @@ public final class DatabaseHelper {
}
/**
* Returns all of the history entries that are parented off the given EppResource with the given
* type and cast to the corresponding subclass.
* Returns all the history entries that are parented off the given EppResource with the given type
* and cast to the corresponding subclass.
*/
public static <T extends HistoryEntry> ImmutableList<T> getHistoryEntriesOfType(
EppResource resource, final HistoryEntry.Type type, Class<T> subclazz) {
@@ -1161,16 +1149,10 @@ public final class DatabaseHelper {
public static <R> void insertSimpleResources(final Iterable<R> resources) {
tm().transact(() -> tm().putAll(ImmutableList.copyOf(resources)));
maybeAdvanceClock();
// Force the session to be cleared so that when we read it back, we read from Datastore
// and not from the transaction's session cache.
tm().clearSessionCache();
}
public static void deleteResource(final Object resource) {
tm().transact(() -> tm().delete(resource));
// Force the session to be cleared so that when we read it back, we read from Datastore and
// not from the transaction's session cache.
tm().clearSessionCache();
}
/** Force the create and update timestamps to get written into the resource. */
@@ -1201,14 +1183,11 @@ public final class DatabaseHelper {
* Loads all entities from all classes stored in the database.
*
* <p>This is not performant (it requires initializing and detaching all Hibernate entities so
* that they can be used outside of the transaction in which they're loaded) and it should only be
* that they can be used outside the transaction in which they're loaded) and it should only be
* used in situations where we need to verify that, for instance, a dry run flow hasn't affected
* the database at all.
*/
public static List<Object> loadAllEntities() {
if (tm().isOfy()) {
return auditedOfy().load().list();
} else {
return jpaTm()
.transact(
() -> {
@@ -1224,14 +1203,13 @@ public final class DatabaseHelper {
}
return result.build();
});
}
}
/**
* Loads (i.e. reloads) the specified entity from the DB.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*/
public static <T> T loadByEntity(T entity) {
return tm().transact(() -> tm().loadByEntity(entity));
@@ -1241,7 +1219,7 @@ public final class DatabaseHelper {
* Loads the specified entity by its key from the DB.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*/
public static <T> T loadByKey(VKey<T> key) {
return tm().transact(() -> tm().loadByKey(key));
@@ -1251,7 +1229,7 @@ public final class DatabaseHelper {
* Loads the specified entity by its key from the DB or empty if it doesn't exist.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*/
public static <T> Optional<T> loadByKeyIfPresent(VKey<T> key) {
return tm().transact(() -> tm().loadByKeyIfPresent(key));
@@ -1261,17 +1239,17 @@ public final class DatabaseHelper {
* Loads the specified entities by their keys from the DB.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*/
public static <T> ImmutableCollection<T> loadByKeys(Iterable<? extends VKey<? extends T>> keys) {
return tm().transact(() -> tm().loadByKeys(keys).values());
}
/**
* Loads all of the entities of the specified type from the DB.
* Loads all the entities of the specified type from the DB.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*/
public static <T> ImmutableList<T> loadAllOf(Class<T> clazz) {
return tm().transact(() -> tm().loadAllOf(clazz));
@@ -1281,7 +1259,7 @@ public final class DatabaseHelper {
* Loads the set of entities by their keys from the DB.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*
* <p>Nonexistent keys / entities are absent from the resulting map, but no {@link
* NoSuchElementException} will be thrown.
@@ -1295,7 +1273,7 @@ public final class DatabaseHelper {
* Loads all given entities from the database if possible.
*
* <p>If the transaction manager is Cloud SQL, then this creates an inner wrapping transaction for
* convenience, so you don't need to wrap it in a transaction at the callsite.
* convenience, so you don't need to wrap it in a transaction at the call site.
*
* <p>Nonexistent entities are absent from the resulting list, but no {@link
* NoSuchElementException} will be thrown.
@@ -102,6 +102,6 @@ public class LordnTaskUtilsTest {
void test_enqueueDomainTask_throwsNpeOnNullDomain() {
assertThrows(
NullPointerException.class,
() -> tm().transactNew(() -> LordnTaskUtils.enqueueDomainTask(null)));
() -> tm().transact(() -> LordnTaskUtils.enqueueDomainTask(null)));
}
}
@@ -100,9 +100,6 @@ public abstract class CommandTestCase<C extends Command> {
jcommander.parse(args);
command.run();
} finally {
// Clear the session cache so that subsequent reads for verification purposes hit Datastore.
// This primarily matters for AutoTimestamp fields, which otherwise won't have updated values.
tm().clearSessionCache();
// Reset back to UNITTEST environment.
RegistryToolEnvironment.UNITTEST.setup(systemPropertyExtension);
}
@@ -16,7 +16,6 @@ package google.registry.tools;
import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.CertificateSamples.SAMPLE_CERT;
import static google.registry.testing.CertificateSamples.SAMPLE_CERT3;
import static google.registry.testing.CertificateSamples.SAMPLE_CERT3_HASH;
@@ -89,9 +88,6 @@ class CreateRegistrarCommandTest extends CommandTestCase<CreateRegistrarCommand>
"clientz");
DateTime after = fakeClock.nowUtc();
// Clear the cache so that the CreateAutoTimestamp field gets reloaded.
tm().clearSessionCache();
Optional<Registrar> registrarOptional = Registrar.loadByRegistrarId("clientz");
assertThat(registrarOptional).isPresent();
Registrar registrar = registrarOptional.get();
@@ -0,0 +1,135 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.jpaTm;
import static google.registry.testing.DatabaseHelper.persistResource;
import static org.junit.jupiter.api.Assertions.assertThrows;
import com.beust.jcommander.ParameterException;
import com.google.common.collect.ImmutableSet;
import google.registry.model.billing.BillingEvent.RenewalPriceBehavior;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.domain.token.AllocationToken.TokenType;
import google.registry.model.domain.token.PackagePromotion;
import org.joda.money.CurrencyUnit;
import org.joda.money.Money;
import org.joda.time.DateTime;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
/** Unit tests for {@link GetPackagePromotionCommand}. */
public class GetPackagePromotionCommandTest extends CommandTestCase<GetPackagePromotionCommand> {
@BeforeEach
void beforeEach() {
command.clock = fakeClock;
}
@Test
void testSuccess() throws Exception {
AllocationToken token =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(TokenType.PACKAGE)
.setCreationTimeForTest(DateTime.parse("2010-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("foo"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountFraction(1)
.build());
PackagePromotion packagePromotion =
new PackagePromotion.Builder()
.setToken(token)
.setMaxDomains(100)
.setMaxCreates(500)
.setPackagePrice(Money.of(CurrencyUnit.USD, 1000))
.setNextBillingDate(DateTime.parse("2012-11-12T05:00:00Z"))
.setLastNotificationSent(DateTime.parse("2010-11-12T05:00:00Z"))
.build();
jpaTm().transact(() -> jpaTm().put(packagePromotion));
runCommand("abc123");
}
@Test
void testSuccessMultiplePackages() throws Exception {
AllocationToken token =
persistResource(
new AllocationToken.Builder()
.setToken("abc123")
.setTokenType(TokenType.PACKAGE)
.setCreationTimeForTest(DateTime.parse("2010-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("foo"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountFraction(1)
.build());
jpaTm()
.transact(
() ->
jpaTm()
.put(
new PackagePromotion.Builder()
.setToken(token)
.setMaxDomains(100)
.setMaxCreates(500)
.setPackagePrice(Money.of(CurrencyUnit.USD, 1000))
.setNextBillingDate(DateTime.parse("2012-11-12T05:00:00Z"))
.setLastNotificationSent(DateTime.parse("2010-11-12T05:00:00Z"))
.build()));
AllocationToken token2 =
persistResource(
new AllocationToken.Builder()
.setToken("123abc")
.setTokenType(TokenType.PACKAGE)
.setCreationTimeForTest(DateTime.parse("2012-11-12T05:00:00Z"))
.setAllowedTlds(ImmutableSet.of("foo"))
.setAllowedRegistrarIds(ImmutableSet.of("TheRegistrar"))
.setRenewalPriceBehavior(RenewalPriceBehavior.SPECIFIED)
.setDiscountFraction(1)
.build());
jpaTm()
.transact(
() ->
jpaTm()
.put(
new PackagePromotion.Builder()
.setToken(token2)
.setMaxDomains(1000)
.setMaxCreates(700)
.setPackagePrice(Money.of(CurrencyUnit.USD, 3000))
.setNextBillingDate(DateTime.parse("2014-11-12T05:00:00Z"))
.setLastNotificationSent(DateTime.parse("2013-11-12T05:00:00Z"))
.build()));
runCommand("abc123", "123abc");
}
@Test
void testFailure_packageDoesNotExist() {
IllegalArgumentException thrown =
assertThrows(IllegalArgumentException.class, () -> runCommand("fakeToken"));
assertThat(thrown)
.hasMessageThat()
.isEqualTo("PackagePromotion with package token fakeToken does not exist");
}
@Test
void testFailure_noToken() {
assertThrows(ParameterException.class, this::runCommand);
}
}
@@ -1,108 +0,0 @@
// Copyright 2022 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.tools.javascrap;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.model.ImmutableObjectSubject.assertAboutImmutableObjects;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.loadAllOf;
import static google.registry.testing.DatabaseHelper.newContact;
import static google.registry.testing.DatabaseHelper.persistActiveHost;
import static google.registry.testing.DatabaseHelper.persistDomainWithDependentResources;
import static google.registry.testing.DatabaseHelper.persistNewRegistrar;
import static google.registry.testing.DatabaseHelper.persistResource;
import static google.registry.testing.DatabaseHelper.persistSimpleResource;
import google.registry.beam.TestPipelineExtension;
import google.registry.model.domain.Domain;
import google.registry.model.domain.DomainHistory;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.HistoryEntryDao;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.testing.DatastoreEntityExtension;
import google.registry.testing.FakeClock;
import org.joda.time.DateTime;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Tests for {@link CreateSyntheticDomainHistoriesPipeline}. */
public class CreateSyntheticDomainHistoriesPipelineTest {
private final FakeClock fakeClock = new FakeClock(DateTime.parse("2022-09-01T00:00:00.000Z"));
@RegisterExtension
JpaTestExtensions.JpaIntegrationTestExtension jpaEextension =
new JpaTestExtensions.Builder().withClock(fakeClock).buildIntegrationTestExtension();
@RegisterExtension
DatastoreEntityExtension datastoreEntityExtension =
new DatastoreEntityExtension().allThreads(true);
@RegisterExtension TestPipelineExtension pipeline = TestPipelineExtension.create();
private Domain domain;
@BeforeEach
void beforeEach() {
persistNewRegistrar("TheRegistrar");
persistNewRegistrar("NewRegistrar");
createTld("tld");
domain =
persistDomainWithDependentResources(
"example",
"tld",
persistResource(newContact("contact1234")),
fakeClock.nowUtc(),
DateTime.parse("2022-09-01T00:00:00.000Z"),
DateTime.parse("2024-09-01T00:00:00.000Z"));
domain =
persistSimpleResource(
domain
.asBuilder()
.setNameservers(persistActiveHost("external.com").createVKey())
.build());
fakeClock.setTo(DateTime.parse("2022-09-20T00:00:00.000Z"));
// shouldn't create any history objects for this domain
persistDomainWithDependentResources(
"ignored-example",
"tld",
persistResource(newContact("contact12345")),
fakeClock.nowUtc(),
DateTime.parse("2022-09-20T00:00:00.000Z"),
DateTime.parse("2024-09-20T00:00:00.000Z"));
}
@Test
void testSuccess() {
assertThat(loadAllOf(DomainHistory.class)).hasSize(2);
CreateSyntheticDomainHistoriesPipeline.setup(pipeline, "NewRegistrar");
pipeline.run().waitUntilFinish();
DomainHistory syntheticHistory =
HistoryEntryDao.loadHistoryObjectsForResource(domain.createVKey(), DomainHistory.class)
.get(1);
assertThat(syntheticHistory.getType()).isEqualTo(HistoryEntry.Type.SYNTHETIC);
assertThat(syntheticHistory.getRegistrarId()).isEqualTo("NewRegistrar");
assertAboutImmutableObjects()
.that(syntheticHistory.getDomainBase().get())
.isEqualExceptFields(domain, "updateTimestamp");
// shouldn't create any entries on re-run
pipeline.run().waitUntilFinish();
assertThat(HistoryEntryDao.loadHistoryObjectsForResource(domain.createVKey())).hasSize(2);
// three total histories, two CREATE and one SYNTHETIC
assertThat(loadAllOf(DomainHistory.class)).hasSize(3);
}
}

Some files were not shown because too many files have changed in this diff Show More