Compare commits

..
Author SHA1 Message Date
gbrodmanandGitHub 0f415f78a6 Use the correct text VKey for HostResource's superordinateDomain (#608)
* Store the superordinateDomain reference as a VKey rather than Key

This is a reference to a Domain object, so we should store it as a VKey
in reference to the Domain table. This should not affect any business
logic, but rather will allow us to set up the SQL tables for
HostResource et al. properly.
2020-06-08 12:21:51 -04:00
gbrodmanandGitHub b324fb98d3 Only use asymmetric VKeys for EPP resources (#611)
Given that we currently have no way of reconstituting a symmetric key
from the asymmetric key (or at least, we don't have a 100% reliable way
of doing so) it's best to keep keys as asymmetrical, referring to the
correct database. That way, we don't get situations where we cannot
compare equality of two keys due to one being asymmetrical and one being
symmetrical.
2020-06-05 16:55:12 -04:00
Lai JiangandGitHub d27fe8ead5 Fix a typo (#610) 2020-06-05 15:53:17 -04:00
Lai JiangandGitHub da65a38782 Add a GCB job to build and publish javadoc (#609) 2020-06-05 13:00:15 -04:00
Legina ChenandGitHub 5a1f3d0376 Remove platformType and threatEntryMetaData fields from ThreatMatch (#607)
* Remove platformType and threatEntryMetaData fields from ThreatMatch

* Run google-java-format on both files

* Add test for removal of unnecessary fields

* Removed unnecessary fields from Spec11PipelineTest.testEndToEndPipeline_generatesExpectedFiles

* Added style check

* Fix typo
2020-06-05 09:00:07 -07:00
Shicong HuangandGitHub b1241b98b2 Generate sql schema for PollMessage (#582)
* Generate sql schema for PollMessage

* Rework columns and resolve comments

* Fix datastore schema
2020-06-04 18:24:59 -04:00
Shicong HuangandGitHub b42ded9451 Add test to verify the behavior of @DualDatabaseTest (#606) 2020-06-03 14:55:37 -04:00
Shicong HuangandGitHub 472503541b Add deleteAll method to TransactionManager (#604)
* Add deleteAll method to TransactionManager

* Rename deleteAll to delete

* Add bucket.getLastWrittenTime() before second mutation
2020-06-03 10:02:48 -04:00
Weimin YuandGitHub ed64dd3548 Load raw records from Datastore export (#605)
* Load raw records from Datastore export

Created a tool that can export from a test instance of Datastore.

Defined Beam pipeline transforms for loading raw records back from
the export.

This is the first part of the effort to create a consistent snapshot
of Datastore. The next step is to load entity records from CommitLog
files.
2020-06-02 18:55:03 -04:00
Michael MullerandGitHub 6a96b1a9cd Use TransactionManager for hosts and contacts (#603)
* Use TransactionManager for hosts and contacts

Replace Ofy calls with TransactionManager for most interactions involving
hosts and contacts.  In the course of this, also convert ForeignKeyIndex and
the EppResourceCache.

* Minor formatting fix
2020-06-02 13:17:16 -04:00
Michael MullerandGitHub c23d4f3ba5 Add createVKey() at the EppResource level (#600)
* Add createVKey() at the EppResource level

Also convert createKey() to createVKey() to normalize with what we've settled
on.
2020-05-29 08:36:57 -04:00
Weimin YuandGitHub 2b794347e6 Refactor LevelDbFileBuilder to accept DS Entity (#599)
* Refactor LevelDbFileBuilder to accept DS Entity

Builder now can directly work with Datastore Entity objects.
No need to wrap data in ComparableEntity.
2020-05-28 13:38:00 -04:00
Shicong HuangandGitHub 26fb5388a4 Generate sql schema for BillingEvent (#565)
* Generate sql schema for BillingEvent

* Change to use sequence

* Address comments

* Resolve warnings and remove duplicate cost related fields

* Increase the flayway file version to V25

* Remove extra space

* Split to 3 tables, merge VKey

* Rename talbes

* Rename repoId to domainRepoId

* Exclude VKey in schema.txt

* Rename target_id to domain_name

* Fix javadoc

* Resolve comments
2020-05-27 15:59:19 -04:00
Lai JiangandGitHub bd443633f6 Add a task to compile javadoc across all packages (#597)
Also fixes various issues that prevent javadoc compliation.
2020-05-27 10:33:46 -04:00
Weimin YuandGitHub d87f119b36 Add a test for SQL logging config (#598)
* Add a test for SQL logging config

Verifies that SQL statements are logged by Hibernate when
configured to do so.
2020-05-26 16:25:33 -04:00
Weimin YuandGitHub 54f1357d83 Fix show-sql which stopped working (#596)
* Fix show-sql which stopped working

Made show-sql property configurable in JpaUnitTestRules.

Added a few comments on foreign key constraint behavior.
2020-05-21 12:20:56 -04:00
Lai JiangandGitHub c73d154084 Do not enqueue update snapshot task if import fails (#578)
If the import from Datastore to BigQuery fails, there is no point
enqueuing a job to update the snapshot view.

Also when there's an error updating the snapshot view, log it at severe
level. The HTTP exception thrown is logged at info and triggers a retry
implicitly. I'm not sure if we want this behavior though. Do we want to
retry upon snapshot updating failures? Unless the failurs are transient,
retrying doesn't help. In our case the failure (End of time out of range
in Standard SQL) is not transient.
2020-05-21 11:40:45 -04:00
gbrodmanandGitHub 259d2e2cdc Run "npm audit fix" to fix a vulnerability (#592) 2020-05-20 15:12:27 -04:00
Shicong HuangandGitHub 0f174d9ce0 Add all existing entities to VKeyTranslatorFactory (#595)
EntityClasses.ALL_CLASSES has all of our registered entities so
we can use it to initialize VKeyTranslatorFactory.classRegistry
to avoid adding them one by one.

Also, this PR changed to use Key.getKind() to get the kind of
the entity to solve the problem that when the entity class
is an inner class, its kind should still be the class name
instead of OuterClass$InnerClass.
2020-05-20 14:24:45 -04:00
Weimin YuandGitHub ca2edb6a17 Close input channel in LevelDbLogReader (#594)
* Close input channel in LevelDbLogReader

Input channel should be closed when all data has been read.
2020-05-20 12:54:13 -04:00
Weimin YuandGitHub 3947ac6ef7 Read LevelDb incrementally (#593)
* Read LevelDb incrementally

Made LevelDbLogReader an iterator over a LevelDb data stream,
Reducing memory footprint which is important when used in a
Dataflow pipeline.
2020-05-20 10:26:34 -04:00
189 changed files with 3517 additions and 1259 deletions
+24
View File
@@ -197,6 +197,10 @@ task runPresubmits(type: Exec) {
args('config/presubmits.py')
}
def javadocSource = []
def javadocClasspath = []
def javadocDependentTasks = []
subprojects {
// Skip no-op project
if (project.name == 'services') return
@@ -317,6 +321,10 @@ subprojects {
}
}
}
javadocSource << project.sourceSets.main.allJava
javadocClasspath << project.sourceSets.main.compileClasspath
javadocDependentTasks << project.tasks.compileJava
}
// If "-P verboseTestOutput=true" is passed in, configure all subprojects to dump all of their
@@ -445,3 +453,19 @@ task javaIncrementalFormatApply {
}
tasks.build.dependsOn(tasks.javaIncrementalFormatCheck)
task javadoc(type: Javadoc) {
source javadocSource
classpath = files(javadocClasspath)
destinationDir = file("${buildDir}/docs/javadoc")
options.encoding = "UTF-8"
// In a lot of places we don't write @return so suppress warnings about that.
options.addBooleanOption('Xdoclint:all,-missing', true)
options.addBooleanOption("-allow-script-in-comments",true)
options.tags = ["type:a:Generic Type",
"error:a:Expected Error"]
}
tasks.build.dependsOn(tasks.javadoc)
javadocDependentTasks.each { tasks.javadoc.dependsOn(it) }
@@ -21,7 +21,6 @@ import org.joda.time.ReadableDuration;
* An object which accepts requests to put the current thread to sleep.
*
* @see SystemSleeper
* @see google.registry.testing.FakeSleeper
*/
@ThreadSafe
public interface Sleeper {
@@ -150,7 +150,7 @@ public final class AsyncTaskEnqueuer {
/** Enqueues a task to asynchronously refresh DNS for a renamed host. */
public void enqueueAsyncDnsRefresh(HostResource host, DateTime now) {
VKey<HostResource> hostKey = host.createKey();
VKey<HostResource> hostKey = host.createVKey();
logger.atInfo().log("Enqueuing async DNS refresh for renamed host %s.", hostKey);
addTaskToQueueWithRetry(
asyncDnsRefreshPullQueue,
@@ -347,7 +347,7 @@ public class DeleteContactsAndHostsAction implements Runnable {
String resourceClientId = resource.getPersistedCurrentSponsorClientId();
if (resource instanceof HostResource && ((HostResource) resource).isSubordinate()) {
resourceClientId =
ofy().load().key(((HostResource) resource).getSuperordinateDomain()).now()
tm().load(((HostResource) resource).getSuperordinateDomain())
.cloneProjectedAtTime(now)
.getCurrentSponsorClientId();
}
@@ -466,10 +466,11 @@ public class DeleteContactsAndHostsAction implements Runnable {
HostResource host = (HostResource) existingResource;
if (host.isSubordinate()) {
dnsQueue.addHostRefreshTask(host.getFullyQualifiedHostName());
ofy().save().entity(
ofy().load().key(host.getSuperordinateDomain()).now().asBuilder()
.removeSubordinateHost(host.getFullyQualifiedHostName())
.build());
tm().saveNewOrUpdate(
tm().load(host.getSuperordinateDomain())
.asBuilder()
.removeSubordinateHost(host.getFullyQualifiedHostName())
.build());
}
} else {
throw new IllegalStateException(
@@ -155,89 +155,100 @@ public class ExpandRecurringBillingEventsAction implements Runnable {
}
int numBillingEventsSaved = 0;
try {
numBillingEventsSaved = tm().transactNew(() -> {
ImmutableSet.Builder<OneTime> syntheticOneTimesBuilder =
new ImmutableSet.Builder<>();
final Registry tld = Registry.get(getTldFromDomainName(recurring.getTargetId()));
numBillingEventsSaved =
tm().transactNew(
() -> {
ImmutableSet.Builder<OneTime> syntheticOneTimesBuilder =
new ImmutableSet.Builder<>();
final Registry tld =
Registry.get(getTldFromDomainName(recurring.getTargetId()));
// Determine the complete set of times at which this recurring event should occur
// (up to and including the runtime of the mapreduce).
Iterable<DateTime> eventTimes =
recurring.getRecurrenceTimeOfYear().getInstancesInRange(Range.closed(
recurring.getEventTime(),
earliestOf(recurring.getRecurrenceEndTime(), executeTime)));
// Determine the complete set of times at which this recurring event should
// occur (up to and including the runtime of the mapreduce).
Iterable<DateTime> eventTimes =
recurring
.getRecurrenceTimeOfYear()
.getInstancesInRange(
Range.closed(
recurring.getEventTime(),
earliestOf(recurring.getRecurrenceEndTime(), executeTime)));
// Convert these event times to billing times
final ImmutableSet<DateTime> billingTimes =
getBillingTimesInScope(eventTimes, cursorTime, executeTime, tld);
// Convert these event times to billing times
final ImmutableSet<DateTime> billingTimes =
getBillingTimesInScope(eventTimes, cursorTime, executeTime, tld);
Key<? extends EppResource> domainKey = recurring.getParentKey().getParent();
Iterable<OneTime> oneTimesForDomain =
ofy().load().type(OneTime.class).ancestor(domainKey);
Key<? extends EppResource> domainKey = recurring.getParentKey().getParent();
Iterable<OneTime> oneTimesForDomain =
ofy().load().type(OneTime.class).ancestor(domainKey);
// Determine the billing times that already have OneTime events persisted.
ImmutableSet<DateTime> existingBillingTimes =
getExistingBillingTimes(oneTimesForDomain, recurring);
// Determine the billing times that already have OneTime events persisted.
ImmutableSet<DateTime> existingBillingTimes =
getExistingBillingTimes(oneTimesForDomain, recurring);
ImmutableSet.Builder<HistoryEntry> historyEntriesBuilder =
new ImmutableSet.Builder<>();
// Create synthetic OneTime events for all billing times that do not yet have an event
// persisted.
for (DateTime billingTime : difference(billingTimes, existingBillingTimes)) {
// Construct a new HistoryEntry that parents over the OneTime
HistoryEntry historyEntry = new HistoryEntry.Builder()
.setBySuperuser(false)
.setClientId(recurring.getClientId())
.setModificationTime(tm().getTransactionTime())
.setParent(domainKey)
.setPeriod(Period.create(1, YEARS))
.setReason("Domain autorenewal by ExpandRecurringBillingEventsAction")
.setRequestedByRegistrar(false)
.setType(DOMAIN_AUTORENEW)
// Don't write a domain transaction record if the recurrence was ended prior to the
// billing time (i.e. a domain was deleted during the autorenew grace period).
.setDomainTransactionRecords(
recurring.getRecurrenceEndTime().isBefore(billingTime)
? ImmutableSet.of()
: ImmutableSet.of(
DomainTransactionRecord.create(
tld.getTldStr(),
// We report this when the autorenew grace period ends
billingTime,
TransactionReportField.netRenewsFieldFromYears(1),
1)))
.build();
historyEntriesBuilder.add(historyEntry);
ImmutableSet.Builder<HistoryEntry> historyEntriesBuilder =
new ImmutableSet.Builder<>();
// Create synthetic OneTime events for all billing times that do not yet have
// an event persisted.
for (DateTime billingTime : difference(billingTimes, existingBillingTimes)) {
// Construct a new HistoryEntry that parents over the OneTime
HistoryEntry historyEntry =
new HistoryEntry.Builder()
.setBySuperuser(false)
.setClientId(recurring.getClientId())
.setModificationTime(tm().getTransactionTime())
.setParent(domainKey)
.setPeriod(Period.create(1, YEARS))
.setReason(
"Domain autorenewal by ExpandRecurringBillingEventsAction")
.setRequestedByRegistrar(false)
.setType(DOMAIN_AUTORENEW)
// Don't write a domain transaction record if the recurrence was
// ended prior to the billing time (i.e. a domain was deleted
// during the autorenew grace period).
.setDomainTransactionRecords(
recurring.getRecurrenceEndTime().isBefore(billingTime)
? ImmutableSet.of()
: ImmutableSet.of(
DomainTransactionRecord.create(
tld.getTldStr(),
// We report this when the autorenew grace period
// ends
billingTime,
TransactionReportField.netRenewsFieldFromYears(1),
1)))
.build();
historyEntriesBuilder.add(historyEntry);
DateTime eventTime = billingTime.minus(tld.getAutoRenewGracePeriodLength());
// Determine the cost for a one-year renewal.
Money renewCost = getDomainRenewCost(recurring.getTargetId(), eventTime, 1);
syntheticOneTimesBuilder.add(new OneTime.Builder()
.setBillingTime(billingTime)
.setClientId(recurring.getClientId())
.setCost(renewCost)
.setEventTime(eventTime)
.setFlags(union(recurring.getFlags(), Flag.SYNTHETIC))
.setParent(historyEntry)
.setPeriodYears(1)
.setReason(recurring.getReason())
.setSyntheticCreationTime(executeTime)
.setCancellationMatchingBillingEvent(Key.create(recurring))
.setTargetId(recurring.getTargetId())
.build());
}
Set<HistoryEntry> historyEntries = historyEntriesBuilder.build();
Set<OneTime> syntheticOneTimes = syntheticOneTimesBuilder.build();
if (!isDryRun) {
ImmutableSet<ImmutableObject> entitiesToSave =
new ImmutableSet.Builder<ImmutableObject>()
.addAll(historyEntries)
.addAll(syntheticOneTimes)
.build();
ofy().save().entities(entitiesToSave).now();
}
return syntheticOneTimes.size();
});
DateTime eventTime = billingTime.minus(tld.getAutoRenewGracePeriodLength());
// Determine the cost for a one-year renewal.
Money renewCost = getDomainRenewCost(recurring.getTargetId(), eventTime, 1);
syntheticOneTimesBuilder.add(
new OneTime.Builder()
.setBillingTime(billingTime)
.setClientId(recurring.getClientId())
.setCost(renewCost)
.setEventTime(eventTime)
.setFlags(union(recurring.getFlags(), Flag.SYNTHETIC))
.setParent(historyEntry)
.setPeriodYears(1)
.setReason(recurring.getReason())
.setSyntheticCreationTime(executeTime)
.setCancellationMatchingBillingEvent(recurring.createVKey())
.setTargetId(recurring.getTargetId())
.build());
}
Set<HistoryEntry> historyEntries = historyEntriesBuilder.build();
Set<OneTime> syntheticOneTimes = syntheticOneTimesBuilder.build();
if (!isDryRun) {
ImmutableSet<ImmutableObject> entitiesToSave =
new ImmutableSet.Builder<ImmutableObject>()
.addAll(historyEntries)
.addAll(syntheticOneTimes)
.build();
ofy().save().entities(entitiesToSave).now();
}
return syntheticOneTimes.size();
});
} catch (Throwable t) {
getContext().incrementCounter("error: " + t.getClass().getSimpleName());
getContext().incrementCounter(ERROR_COUNTER);
@@ -279,7 +290,8 @@ public class ExpandRecurringBillingEventsAction implements Runnable {
return Streams.stream(oneTimesForDomain)
.filter(
billingEvent ->
Key.create(recurringEvent)
recurringEvent
.createVKey()
.equals(billingEvent.getCancellationMatchingBillingEvent()))
.map(OneTime::getBillingTime)
.collect(toImmutableSet());
@@ -0,0 +1,86 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.initsql;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Strings.isNullOrEmpty;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* Helpers for determining the fully qualified paths to Nomulus backup files. A backup consists of a
* Datastore export and Nomulus CommitLogs that overlap with the export.
*/
public final class BackupPaths {
private BackupPaths() {}
private static final String WILDCARD_CHAR = "*";
private static final String EXPORT_PATTERN_TEMPLATE = "%s/all_namespaces/kind_%s/input-%s";
/**
* Regex pattern that captures the kind string in a file name. Datastore places no restrictions on
* what characters may be used in a kind string.
*/
private static final String FILENAME_TO_KIND_REGEX = ".+/all_namespaces/kind_(.+)/input-.+";
private static final Pattern FILENAME_TO_KIND_PATTERN = Pattern.compile(FILENAME_TO_KIND_REGEX);
/**
* Returns a regex pattern that matches all Datastore export files of a given {@code kind}.
*
* @param exportDir path to the top directory of a Datastore export
* @param kind the 'kind' of the Datastore entity
*/
public static String getExportFileNamePattern(String exportDir, String kind) {
checkArgument(!isNullOrEmpty(exportDir), "Null or empty exportDir.");
checkArgument(!isNullOrEmpty(kind), "Null or empty kind.");
return String.format(EXPORT_PATTERN_TEMPLATE, exportDir, kind, WILDCARD_CHAR);
}
/**
* Returns the fully qualified path of a Datastore export file with the given {@code kind} and
* {@code shard}.
*
* @param exportDir path to the top directory of a Datastore export
* @param kind the 'kind' of the Datastore entity
* @param shard an integer suffix of the file name
*/
public static String getExportFileNameByShard(String exportDir, String kind, int shard) {
checkArgument(!isNullOrEmpty(exportDir), "Null or empty exportDir.");
checkArgument(!isNullOrEmpty(kind), "Null or empty kind.");
checkArgument(shard >= 0, "Negative shard %s not allowed.", shard);
return String.format(EXPORT_PATTERN_TEMPLATE, exportDir, kind, Integer.toString(shard));
}
/**
* Returns the 'kind' of entity stored in a file based on the file name.
*
* <p>This method poses low risk and greatly simplifies the implementation of some transforms in
* {@link ExportLoadingTransforms}.
*
* @see ExportLoadingTransforms
*/
public static String getKindFromFileName(String fileName) {
checkArgument(!isNullOrEmpty(fileName), "Null or empty fileName.");
Matcher matcher = FILENAME_TO_KIND_PATTERN.matcher(fileName);
checkArgument(
matcher.matches(),
"Illegal file name %s, should match %s.",
fileName,
FILENAME_TO_KIND_REGEX);
return matcher.group(1);
}
}
@@ -0,0 +1,117 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.beam.initsql;
import static google.registry.beam.initsql.BackupPaths.getExportFileNamePattern;
import static google.registry.beam.initsql.BackupPaths.getKindFromFileName;
import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
import static org.apache.beam.sdk.values.TypeDescriptors.strings;
import com.google.common.collect.ImmutableList;
import google.registry.tools.LevelDbLogReader;
import java.io.IOException;
import java.util.Collection;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.Compression;
import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.io.FileIO.ReadableFile;
import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
/**
* {@link PTransform Pipeline transforms} for loading from Datastore export files. They are all part
* of a transformation that loads raw records from a Datastore export, and are broken apart for
* testing.
*
* <p>We drop the 'kind' information in {@link #getDatastoreExportFilePatterns} and recover it later
* using the file paths. Although we could have kept it by passing around {@link KV key-value
* pairs}, the code would be more complicated, especially in {@link #loadDataFromFiles()}.
*/
public class ExportLoadingTransforms {
/**
* Returns a {@link PTransform transform} that can generate a collection of patterns that match
* all Datastore export files of the given {@code kinds}.
*/
public static PTransform<PBegin, PCollection<String>> getDatastoreExportFilePatterns(
String exportDir, Collection<String> kinds) {
return Create.of(
kinds.stream()
.map(kind -> getExportFileNamePattern(exportDir, kind))
.collect(ImmutableList.toImmutableList()))
.withCoder(StringUtf8Coder.of());
}
/**
* Returns a {@link PTransform} from file name patterns to file {@link Metadata Metadata records}.
*/
public static PTransform<PCollection<String>, PCollection<Metadata>> getFilesByPatterns() {
return new PTransform<PCollection<String>, PCollection<Metadata>>() {
@Override
public PCollection<Metadata> expand(PCollection<String> input) {
return input.apply(FileIO.matchAll().withEmptyMatchTreatment(EmptyMatchTreatment.DISALLOW));
}
};
}
/**
* Returns a {@link PTransform} from file {@link Metadata} to {@link KV Key-Value pairs} with
* entity 'kind' as key and raw record as value.
*/
public static PTransform<PCollection<Metadata>, PCollection<KV<String, byte[]>>>
loadDataFromFiles() {
return new PTransform<PCollection<Metadata>, PCollection<KV<String, byte[]>>>() {
@Override
public PCollection<KV<String, byte[]>> expand(PCollection<Metadata> input) {
return input
.apply(FileIO.readMatches().withCompression(Compression.UNCOMPRESSED))
.apply(
MapElements.into(kvs(strings(), TypeDescriptor.of(ReadableFile.class)))
.via(file -> KV.of(getKindFromFileName(file.getMetadata().toString()), file)))
.apply("Load One LevelDb File", ParDo.of(new LoadOneFile()));
}
};
}
/**
* Reads a LevelDb file and converts each raw record into a {@link KV pair} of kind and bytes.
*
* <p>LevelDb files are not seekable because a large object may span multiple blocks. If a
* sequential read fails, the file needs to be retried from the beginning.
*/
private static class LoadOneFile extends DoFn<KV<String, ReadableFile>, KV<String, byte[]>> {
@ProcessElement
public void processElement(
@Element KV<String, ReadableFile> kv, OutputReceiver<KV<String, byte[]>> output) {
try {
LevelDbLogReader.from(kv.getValue().open())
.forEachRemaining(record -> output.output(KV.of(kv.getKey(), record)));
} catch (IOException e) {
// Let the pipeline retry the whole file.
throw new RuntimeException(e);
}
}
}
}
@@ -0,0 +1,3 @@
## Summary
This package contains a BEAM pipeline that populates a Cloud SQL database from a Datastore backup.
@@ -24,22 +24,10 @@ import org.json.JSONObject;
public abstract class ThreatMatch implements Serializable {
private static final String THREAT_TYPE_FIELD = "threatType";
private static final String PLATFORM_TYPE_FIELD = "platformType";
private static final String METADATA_FIELD = "threatEntryMetadata";
private static final String DOMAIN_NAME_FIELD = "fullyQualifiedDomainName";
/** Returns what kind of threat it is (malware, phishing etc.) */
public abstract String threatType();
/** Returns what platforms it affects (Windows, Linux etc.) */
abstract String platformType();
/**
* Returns a String representing a JSON Object containing arbitrary metadata associated with this
* threat, or "NONE" if there is no metadata to retrieve.
*
* <p>This ideally would be a {@link JSONObject} type, but can't be due to serialization
* requirements.
*/
abstract String metadata();
/** Returns the fully qualified domain name [SLD].[TLD] of the matched threat. */
public abstract String fullyQualifiedDomainName();
@@ -52,29 +40,19 @@ public abstract class ThreatMatch implements Serializable {
static ThreatMatch create(JSONObject threatMatchJSON, String fullyQualifiedDomainName)
throws JSONException {
return new AutoValue_ThreatMatch(
threatMatchJSON.getString(THREAT_TYPE_FIELD),
threatMatchJSON.getString(PLATFORM_TYPE_FIELD),
threatMatchJSON.has(METADATA_FIELD)
? threatMatchJSON.getJSONObject(METADATA_FIELD).toString()
: "NONE",
fullyQualifiedDomainName);
threatMatchJSON.getString(THREAT_TYPE_FIELD), fullyQualifiedDomainName);
}
/** Returns a {@link JSONObject} representing a subset of this object's data. */
JSONObject toJSON() throws JSONException {
return new JSONObject()
.put(THREAT_TYPE_FIELD, threatType())
.put(PLATFORM_TYPE_FIELD, platformType())
.put(METADATA_FIELD, metadata())
.put(DOMAIN_NAME_FIELD, fullyQualifiedDomainName());
}
/** Parses a {@link JSONObject} and returns an equivalent {@link ThreatMatch}. */
public static ThreatMatch fromJSON(JSONObject threatMatch) throws JSONException {
return new AutoValue_ThreatMatch(
threatMatch.getString(THREAT_TYPE_FIELD),
threatMatch.getString(PLATFORM_TYPE_FIELD),
threatMatch.getString(METADATA_FIELD),
threatMatch.getString(DOMAIN_NAME_FIELD));
threatMatch.getString(THREAT_TYPE_FIELD), threatMatch.getString(DOMAIN_NAME_FIELD));
}
}
@@ -442,10 +442,8 @@ public class BigqueryConnection implements AutoCloseable {
* Returns the result of calling queryToLocalTable, but synchronously to avoid spawning new
* background threads, which App Engine doesn't support.
*
* @see <a href="https://cloud.google.com/appengine/docs/standard/java/runtime#Threads">
* App Engine Runtime</a>
*
* <p>Returns the results of the query in an ImmutableTable on success.
* @see <a href="https://cloud.google.com/appengine/docs/standard/java/runtime#Threads">App Engine
* Runtime</a>
*/
public ImmutableTable<Integer, TableFieldSchema, Object> queryToLocalTableSync(String querySql) {
Job job = new Job()
@@ -576,8 +574,6 @@ public class BigqueryConnection implements AutoCloseable {
/**
* Launch a job, wait for it to complete, but <i>do not</i> check for errors.
*
* @throws BigqueryJobFailureException
*/
public Job runJob(Job job, @Nullable AbstractInputStreamContent data) {
return checkJob(waitForJob(launchJob(job, data)));
@@ -84,7 +84,7 @@ public class DnsMessageTransport {
* @param query a message to send
* @return the response received from the server
* @throws IOException if the Socket input/output streams throws one
* @throws IllegalArgumentException if the query is too large to be sent (> 65535 bytes)
* @throws IllegalArgumentException if the query is too large to be sent (&gt; 65535 bytes)
*/
public Message send(Message query) throws IOException {
try (Socket socket = factory.createSocket(InetAddress.getByName(updateHost), DNS_PORT)) {
@@ -73,8 +73,10 @@ public class BigqueryPollJobAction implements Runnable {
@Override
public void run() {
checkJobOutcome(); // Throws a NotModifiedException if the job hasn't completed.
if (payload == null || payload.length == 0) {
boolean jobOutcome =
checkJobOutcome(); // Throws a NotModifiedException if the job hasn't completed.
// If the job failed, do not enqueue the next step.
if (!jobOutcome || payload == null || payload.length == 0) {
return;
}
// If there is a payload, it's a chained task, so enqueue it.
@@ -42,8 +42,8 @@ import javax.xml.stream.events.XMLEvent;
/**
* Sanitizes sensitive data in incoming/outgoing EPP XML messages.
*
* <p>Current implementation masks user credentials (text following <pw> and <newPW> tags) as
* follows:
* <p>Current implementation masks user credentials (text following &lt;pw&gt; and &lt;newPW&gt;
* tags) as follows:
*
* <ul>
* <li>A control character (in ranges [0 - 1F] and [7F - 9F]) is replaced with 'C'.
@@ -94,10 +94,10 @@ public final class ResourceFlowUtils {
* trust the query and need to do the full mapreduce.
*/
Iterable<Key<DomainBase>> keys =
queryForLinkedDomains(fki.getResourceKey(), now)
queryForLinkedDomains(fki.getResourceKey().getOfyKey(), now)
.limit(FAILFAST_CHECK_COUNT)
.keys();
VKey<R> resourceVKey = VKey.createOfy(resourceClass, fki.getResourceKey());
VKey<R> resourceVKey = fki.getResourceKey();
Predicate<DomainBase> predicate =
domain -> getPotentialReferences.apply(domain).contains(resourceVKey);
return ofy().load().keys(keys).values().stream().anyMatch(predicate)
@@ -136,9 +136,9 @@ public final class ResourceFlowUtils {
public static <R extends EppResource> void verifyResourceDoesNotExist(
Class<R> clazz, String targetId, DateTime now, String clientId) throws EppException {
Key<R> key = loadAndGetKey(clazz, targetId, now);
VKey<R> key = loadAndGetKey(clazz, targetId, now);
if (key != null) {
R resource = ofy().load().key(key).now();
R resource = tm().load(key);
// These are similar exceptions, but we can track them internally as log-based metrics.
if (Objects.equals(clientId, resource.getPersistedCurrentSponsorClientId())) {
throw new ResourceAlreadyExistsForThisClientException(targetId);
@@ -75,7 +75,7 @@ public class DomainCheckFlowCustomLogic extends BaseFlowCustomLogic {
/**
* The time to perform the domain check as of. This defaults to the current time, but can be
* overridden in v>=0.12 of the fee extension.
* overridden in v&gt;=0.12 of the fee extension.
*/
public abstract DateTime asOfDate();
@@ -105,7 +105,7 @@ public class DomainCheckFlowCustomLogic extends BaseFlowCustomLogic {
/**
* The time to perform the domain check as of. This defaults to the current time, but can be
* overridden in v>=0.12 of the fee extension.
* overridden in v&gt;=0.12 of the fee extension.
*/
public abstract DateTime asOfDate();
@@ -77,7 +77,6 @@ import google.registry.model.billing.BillingEvent;
import google.registry.model.billing.BillingEvent.Flag;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.billing.BillingEvent.Recurring;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand;
import google.registry.model.domain.DomainCommand.Create;
@@ -353,14 +352,12 @@ public class DomainCreateFlow implements TransactionalFlow {
.setLaunchNotice(hasClaimsNotice ? launchCreate.get().getNotice() : null)
.setSmdId(signedMarkId)
.setDsData(secDnsCreate.isPresent() ? secDnsCreate.get().getDsData() : null)
.setRegistrant(VKey.createOfy(ContactResource.class, command.getRegistrant()))
.setRegistrant(command.getRegistrant())
.setAuthInfo(command.getAuthInfo())
.setFullyQualifiedDomainName(targetId)
.setNameservers(
(ImmutableSet<VKey<HostResource>>)
command.getNameservers().stream()
.map(key -> VKey.createOfy(HostResource.class, key))
.collect(toImmutableSet()))
command.getNameservers().stream().collect(toImmutableSet()))
.setStatusValues(statuses.build())
.setContacts(command.getContacts())
.addGracePeriod(GracePeriod.forBillingEvent(GracePeriodStatus.ADD, createBillingEvent))
@@ -534,7 +531,7 @@ public class DomainCreateFlow implements TransactionalFlow {
.setPeriodYears(years)
.setCost(feesAndCredits.getCreateCost())
.setEventTime(now)
.setAllocationToken(allocationToken.map(Key::create).orElse(null))
.setAllocationToken(allocationToken.map(AllocationToken::createVKey).orElse(null))
.setBillingTime(
now.plus(
isAnchorTenant
@@ -307,14 +307,11 @@ public class DomainFlowUtils {
/** Verify that no linked resources have disallowed statuses. */
static void verifyNotInPendingDelete(
Set<DesignatedContact> contacts,
Key<ContactResource> registrant,
Set<Key<HostResource>> nameservers)
VKey<ContactResource> registrant,
Set<VKey<HostResource>> nameservers)
throws EppException {
ImmutableList.Builder<Key<? extends EppResource>> keysToLoad = new ImmutableList.Builder<>();
contacts.stream()
.map(DesignatedContact::getContactKey)
.map(VKey::getOfyKey)
.forEach(keysToLoad::add);
ImmutableList.Builder<VKey<? extends EppResource>> keysToLoad = new ImmutableList.Builder<>();
contacts.stream().map(DesignatedContact::getContactKey).forEach(keysToLoad::add);
Optional.ofNullable(registrant).ifPresent(keysToLoad::add);
keysToLoad.addAll(nameservers);
verifyNotInPendingDelete(EppResource.loadCached(keysToLoad.build()).values());
@@ -381,7 +378,7 @@ public class DomainFlowUtils {
}
static void validateRequiredContactsPresent(
@Nullable Key<ContactResource> registrant, Set<DesignatedContact> contacts)
@Nullable VKey<ContactResource> registrant, Set<DesignatedContact> contacts)
throws RequiredParameterMissingException {
if (registrant == null) {
throw new MissingRegistrantException();
@@ -60,7 +60,6 @@ import google.registry.flows.domain.DomainFlowUtils.MissingRegistrantException;
import google.registry.model.ImmutableObject;
import google.registry.model.billing.BillingEvent;
import google.registry.model.billing.BillingEvent.Reason;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.DomainBase;
import google.registry.model.domain.DomainCommand.Update;
import google.registry.model.domain.DomainCommand.Update.AddRemove;
@@ -73,11 +72,9 @@ import google.registry.model.eppcommon.StatusValue;
import google.registry.model.eppinput.EppInput;
import google.registry.model.eppinput.ResourceCommand;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.host.HostResource;
import google.registry.model.registry.Registry;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.persistence.VKey;
import java.util.Optional;
import javax.inject.Inject;
import org.joda.time.DateTime;
@@ -247,22 +244,11 @@ public final class DomainUpdateFlow implements TransactionalFlow {
.setLastEppUpdateClientId(clientId)
.addStatusValues(add.getStatusValues())
.removeStatusValues(remove.getStatusValues())
.addNameservers(
add.getNameservers().stream()
.map(key -> VKey.createOfy(HostResource.class, key))
.collect(toImmutableSet()))
.removeNameservers(
remove.getNameservers().stream()
.map(key -> VKey.createOfy(HostResource.class, key))
.collect(toImmutableSet()))
.addNameservers(add.getNameservers().stream().collect(toImmutableSet()))
.removeNameservers(remove.getNameservers().stream().collect(toImmutableSet()))
.addContacts(add.getContacts())
.removeContacts(remove.getContacts())
.setRegistrant(
firstNonNull(
change.getRegistrant() != null
? VKey.createOfy(ContactResource.class, change.getRegistrant())
: null,
domain.getRegistrant()))
.setRegistrant(firstNonNull(change.getRegistrant(), domain.getRegistrant()))
.setAuthInfo(firstNonNull(change.getAuthInfo(), domain.getAuthInfo()));
return domainBuilder.build();
}
@@ -275,7 +261,7 @@ public final class DomainUpdateFlow implements TransactionalFlow {
private void validateNewState(DomainBase newDomain) throws EppException {
validateNoDuplicateContacts(newDomain.getContacts());
validateRequiredContactsPresent(newDomain.getRegistrant().getOfyKey(), newDomain.getContacts());
validateRequiredContactsPresent(newDomain.getRegistrant(), newDomain.getContacts());
validateDsData(newDomain.getDsData());
validateNameserversCountForTld(
newDomain.getTld(),
@@ -128,7 +128,7 @@ public final class HostCreateFlow implements TransactionalFlow {
.setFullyQualifiedHostName(targetId)
.setInetAddresses(command.getInetAddresses())
.setRepoId(createRepoId(ObjectifyService.allocateId(), roidSuffix))
.setSuperordinateDomain(superordinateDomain.map(Key::create).orElse(null))
.setSuperordinateDomain(superordinateDomain.map(DomainBase::createVKey).orElse(null))
.build();
historyBuilder
.setType(HistoryEntry.Type.HOST_CREATE)
@@ -96,8 +96,7 @@ public final class HostDeleteFlow implements TransactionalFlow {
// the client id, needs to be read off of it.
EppResource owningResource =
existingHost.isSubordinate()
? ofy().load().key(existingHost.getSuperordinateDomain()).now()
.cloneProjectedAtTime(now)
? tm().load(existingHost.getSuperordinateDomain()).cloneProjectedAtTime(now)
: existingHost;
verifyResourceOwnership(clientId, owningResource);
}
@@ -18,7 +18,7 @@ import static google.registry.flows.FlowUtils.validateClientIsLoggedIn;
import static google.registry.flows.ResourceFlowUtils.loadAndVerifyExistence;
import static google.registry.flows.host.HostFlowUtils.validateHostName;
import static google.registry.model.EppResourceUtils.isLinked;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
@@ -77,7 +77,7 @@ public final class HostInfoFlow implements Flow {
// there is no superordinate domain, the host's own values for these fields will be correct.
if (host.isSubordinate()) {
DomainBase superordinateDomain =
ofy().load().key(host.getSuperordinateDomain()).now().cloneProjectedAtTime(now);
tm().load(host.getSuperordinateDomain()).cloneProjectedAtTime(now);
hostInfoDataBuilder
.setCurrentSponsorClientId(superordinateDomain.getCurrentSponsorClientId())
.setLastTransferTime(host.computeLastTransferTime(superordinateDomain));
@@ -60,6 +60,7 @@ import google.registry.model.host.HostResource;
import google.registry.model.index.ForeignKeyIndex;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.reporting.IcannReportingTypes.ActivityReportField;
import google.registry.persistence.VKey;
import java.util.Objects;
import java.util.Optional;
import javax.inject.Inject;
@@ -136,9 +137,10 @@ public final class HostUpdateFlow implements TransactionalFlow {
boolean isHostRename = suppliedNewHostName != null;
String oldHostName = targetId;
String newHostName = firstNonNull(suppliedNewHostName, oldHostName);
DomainBase oldSuperordinateDomain = existingHost.isSubordinate()
? ofy().load().key(existingHost.getSuperordinateDomain()).now().cloneProjectedAtTime(now)
: null;
DomainBase oldSuperordinateDomain =
existingHost.isSubordinate()
? tm().load(existingHost.getSuperordinateDomain()).cloneProjectedAtTime(now)
: null;
// Note that lookupSuperordinateDomain calls cloneProjectedAtTime on the domain for us.
Optional<DomainBase> newSuperordinateDomain =
lookupSuperordinateDomain(validateHostName(newHostName), now);
@@ -153,8 +155,8 @@ public final class HostUpdateFlow implements TransactionalFlow {
AddRemove remove = command.getInnerRemove();
checkSameValuesNotAddedAndRemoved(add.getStatusValues(), remove.getStatusValues());
checkSameValuesNotAddedAndRemoved(add.getInetAddresses(), remove.getInetAddresses());
Key<DomainBase> newSuperordinateDomainKey =
newSuperordinateDomain.map(Key::create).orElse(null);
VKey<DomainBase> newSuperordinateDomainKey =
newSuperordinateDomain.map(DomainBase::createVKey).orElse(null);
// If the superordinateDomain field is changing, set the lastSuperordinateChange to now.
DateTime lastSuperordinateChange =
Objects.equals(newSuperordinateDomainKey, existingHost.getSuperordinateDomain())
@@ -280,28 +282,28 @@ public final class HostUpdateFlow implements TransactionalFlow {
if (existingHost.isSubordinate()
&& newHost.isSubordinate()
&& Objects.equals(
existingHost.getSuperordinateDomain(),
newHost.getSuperordinateDomain())) {
ofy().save().entity(
ofy().load().key(existingHost.getSuperordinateDomain()).now().asBuilder()
.removeSubordinateHost(existingHost.getFullyQualifiedHostName())
.addSubordinateHost(newHost.getFullyQualifiedHostName())
.build());
existingHost.getSuperordinateDomain(), newHost.getSuperordinateDomain())) {
tm().saveNewOrUpdate(
tm().load(existingHost.getSuperordinateDomain())
.asBuilder()
.removeSubordinateHost(existingHost.getFullyQualifiedHostName())
.addSubordinateHost(newHost.getFullyQualifiedHostName())
.build());
return;
}
if (existingHost.isSubordinate()) {
ofy().save().entity(
ofy().load().key(existingHost.getSuperordinateDomain()).now()
.asBuilder()
.removeSubordinateHost(existingHost.getFullyQualifiedHostName())
.build());
tm().saveNewOrUpdate(
tm().load(existingHost.getSuperordinateDomain())
.asBuilder()
.removeSubordinateHost(existingHost.getFullyQualifiedHostName())
.build());
}
if (newHost.isSubordinate()) {
ofy().save().entity(
ofy().load().key(newHost.getSuperordinateDomain()).now()
.asBuilder()
.addSubordinateHost(newHost.getFullyQualifiedHostName())
.build());
tm().saveNewOrUpdate(
tm().load(newHost.getSuperordinateDomain())
.asBuilder()
.addSubordinateHost(newHost.getFullyQualifiedHostName())
.build());
}
}
@@ -16,12 +16,11 @@ package google.registry.model;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkState;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static com.google.common.collect.ImmutableMap.toImmutableMap;
import static com.google.common.collect.Sets.difference;
import static com.google.common.collect.Sets.union;
import static google.registry.config.RegistryConfig.getEppResourceCachingDuration;
import static google.registry.config.RegistryConfig.getEppResourceMaxCachedEntries;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.CollectionUtils.nullToEmpty;
import static google.registry.util.CollectionUtils.nullToEmptyImmutableCopy;
@@ -32,11 +31,9 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.ImmutableSortedMap;
import com.google.common.collect.Streams;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Index;
@@ -44,12 +41,13 @@ import google.registry.config.RegistryConfig;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.ofy.CommitLogManifest;
import google.registry.model.transfer.TransferData;
import google.registry.persistence.VKey;
import google.registry.util.NonFinalForTesting;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.stream.StreamSupport;
import javax.persistence.Column;
import javax.persistence.MappedSuperclass;
import javax.persistence.Transient;
@@ -183,6 +181,9 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
/** Get the foreign key string for this resource. */
public abstract String getForeignKey();
/** Create the VKey for the specified EPP resource. */
public abstract VKey<? extends EppResource> createVKey();
/** Override of {@link Buildable#asBuilder} so that the extra methods are visible. */
@Override
public abstract Builder<?, ?> asBuilder();
@@ -329,18 +330,18 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
}
}
static final CacheLoader<Key<? extends EppResource>, EppResource> CACHE_LOADER =
new CacheLoader<Key<? extends EppResource>, EppResource>() {
static final CacheLoader<VKey<? extends EppResource>, EppResource> CACHE_LOADER =
new CacheLoader<VKey<? extends EppResource>, EppResource>() {
@Override
public EppResource load(Key<? extends EppResource> key) {
return tm().doTransactionless(() -> ofy().load().key(key).now());
public EppResource load(VKey<? extends EppResource> key) {
return tm().doTransactionless(() -> tm().load(key));
}
@Override
public Map<Key<? extends EppResource>, EppResource> loadAll(
Iterable<? extends Key<? extends EppResource>> keys) {
return tm().doTransactionless(() -> loadMultiple(keys));
public Map<VKey<? extends EppResource>, EppResource> loadAll(
Iterable<? extends VKey<? extends EppResource>> keys) {
return tm().doTransactionless(() -> loadAsMap(keys));
}
};
@@ -352,10 +353,10 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
* directly should of course never use the cache.
*/
@NonFinalForTesting
private static LoadingCache<Key<? extends EppResource>, EppResource> cacheEppResources =
private static LoadingCache<VKey<? extends EppResource>, EppResource> cacheEppResources =
createEppResourcesCache(getEppResourceCachingDuration());
private static LoadingCache<Key<? extends EppResource>, EppResource> createEppResourcesCache(
private static LoadingCache<VKey<? extends EppResource>, EppResource> createEppResourcesCache(
Duration expiry) {
return CacheBuilder.newBuilder()
.expireAfterWrite(expiry.getMillis(), MILLISECONDS)
@@ -369,29 +370,16 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
cacheEppResources = createEppResourcesCache(effectiveExpiry);
}
private static ImmutableMap<Key<? extends EppResource>, EppResource> loadMultiple(
Iterable<? extends Key<? extends EppResource>> keys) {
// This cast is safe because, in Objectify, Key<? extends EppResource> can also be
// treated as a Key<EppResource>.
@SuppressWarnings("unchecked")
ImmutableList<Key<EppResource>> typedKeys =
Streams.stream(keys).map(key -> (Key<EppResource>) key).collect(toImmutableList());
// Typing shenanigans are required to return the map with the correct required generic type.
return ofy().load().keys(typedKeys).entrySet().stream()
.collect(ImmutableMap.toImmutableMap(Entry::getKey, Entry::getValue));
}
/**
* Loads the given EppResources by their keys using the cache (if enabled).
*
* <p>Don't use this unless you really need it for performance reasons, and be sure that you are
* OK with the trade-offs in loss of transactional consistency.
*/
public static ImmutableMap<Key<? extends EppResource>, EppResource> loadCached(
Iterable<Key<? extends EppResource>> keys) {
public static ImmutableMap<VKey<? extends EppResource>, EppResource> loadCached(
Iterable<VKey<? extends EppResource>> keys) {
if (!RegistryConfig.isEppResourceCachingEnabled()) {
return loadMultiple(keys);
return loadAsMap(keys);
}
try {
return cacheEppResources.getAll(keys);
@@ -406,9 +394,9 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
* <p>Don't use this unless you really need it for performance reasons, and be sure that you are
* OK with the trade-offs in loss of transactional consistency.
*/
public static <T extends EppResource> T loadCached(Key<T> key) {
public static <T extends EppResource> T loadCached(VKey<T> key) {
if (!RegistryConfig.isEppResourceCachingEnabled()) {
return ofy().load().key(key).now();
return tm().load(key);
}
try {
// Safe to cast because loading a Key<T> returns an entity of type T.
@@ -419,4 +407,17 @@ public abstract class EppResource extends BackupGroupRoot implements Buildable {
throw new RuntimeException("Error loading cached EppResources", e.getCause());
}
}
private static ImmutableMap<VKey<? extends EppResource>, EppResource> loadAsMap(
Iterable<? extends VKey<? extends EppResource>> keys) {
return StreamSupport.stream(keys.spliterator(), false)
// It's possible for us to receive the same key more than once which causes
// the immutable map build to break with a duplicate key, so we have to ensure key
// uniqueness.
.distinct()
// We have to use "key -> key" here instead of the identity() function, because
// the latter breaks the fairly complicated generic type checking required by the
// caching interface.
.collect(toImmutableMap(key -> key, key -> tm().load(key)));
}
}
@@ -17,6 +17,7 @@ package google.registry.model;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.DateTimeUtils.isAtOrAfter;
import static google.registry.util.DateTimeUtils.isBeforeOrAt;
import static google.registry.util.DateTimeUtils.latestOf;
@@ -141,7 +142,7 @@ public final class EppResourceUtils {
T resource =
useCache
? EppResource.loadCached(fki.getResourceKey())
: ofy().load().key(fki.getResourceKey()).now();
: tm().maybeLoad(fki.getResourceKey()).orElse(null);
if (resource == null || isAtOrAfter(now, resource.getDeletionTime())) {
return Optional.empty();
}
@@ -34,6 +34,7 @@ import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Ignore;
import com.googlecode.objectify.annotation.Parent;
import google.registry.persistence.VKey;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
@@ -139,10 +140,13 @@ public class ModelUtils {
// If the field's type is the same as the field's class object, then it's a non-parameterized
// type, and thus we just add it directly. We also don't bother looking at the parameterized
// types of Key objects, since they are just references to other objects and don't actually
// embed themselves in the persisted object anyway.
// types of Key and VKey objects, since they are just references to other objects and don't
// actually embed themselves in the persisted object anyway.
Class<?> fieldClazz = field.getType();
Type fieldType = field.getGenericType();
if (VKey.class.equals(fieldClazz)) {
continue;
}
builder.add(fieldClazz);
if (fieldType.equals(fieldClazz) || Key.class.equals(clazz)) {
continue;
@@ -50,7 +50,8 @@ import java.util.stream.Collectors;
public class OteStats {
/**
* Returns the statistics about the OT&E actions that have been taken by a particular registrar.
* Returns the statistics about the OT&amp;E actions that have been taken by a particular
* registrar.
*/
public static OteStats getFromRegistrar(String registrarName) {
return new OteStats().recordRegistrarHistory(registrarName);
@@ -30,7 +30,7 @@ public class UpdateAutoTimestamp extends ImmutableObject {
DateTime timestamp;
/** Returns the timestamp, or {@link START_OF_TIME} if it's null. */
/** Returns the timestamp, or {@code START_OF_TIME} if it's null. */
public DateTime getTimestamp() {
return Optional.ofNullable(timestamp).orElse(START_OF_TIME);
}
@@ -30,6 +30,7 @@ import com.google.common.collect.Sets;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Ignore;
import com.googlecode.objectify.annotation.IgnoreSave;
import com.googlecode.objectify.annotation.Index;
import com.googlecode.objectify.annotation.Parent;
@@ -43,14 +44,28 @@ import google.registry.model.domain.rgp.GracePeriodStatus;
import google.registry.model.domain.token.AllocationToken;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
import google.registry.persistence.VKey;
import google.registry.persistence.WithLongVKey;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import javax.annotation.Nullable;
import javax.persistence.AttributeOverride;
import javax.persistence.AttributeOverrides;
import javax.persistence.Column;
import javax.persistence.Embedded;
import javax.persistence.EnumType;
import javax.persistence.Enumerated;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.MappedSuperclass;
import javax.persistence.Transient;
import org.joda.money.Money;
import org.joda.time.DateTime;
/** A billable event in a domain's lifecycle. */
@MappedSuperclass
@WithLongVKey
public abstract class BillingEvent extends ImmutableObject
implements Buildable, TransferServerApproveEntity {
@@ -93,24 +108,41 @@ public abstract class BillingEvent extends ImmutableObject
/** Entity id. */
@Id
long id;
@javax.persistence.Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
Long id;
@Parent
@DoNotHydrate
Key<HistoryEntry> parent;
@Parent @DoNotHydrate @Transient Key<HistoryEntry> parent;
/** The registrar to bill. */
@Index
@Column(nullable = false)
String clientId;
/** Revision id of the entry in DomainHistory table that ths bill belongs to. */
// TODO(shicong): Add foreign key constraint when DomainHistory table is generated
@Ignore
@Column(nullable = false)
Long domainHistoryRevisionId;
/** ID of the EPP resource that the bill is for. */
// TODO(shicong): Add foreign key constraint when we expand DatastoreHelp for Postgresql
@Ignore
@Column(nullable = false)
String domainRepoId;
/** When this event was created. For recurring events, this is also the recurrence start time. */
@Index
@Column(nullable = false)
DateTime eventTime;
/** The reason for the bill. */
@Enumerated(EnumType.STRING)
@Column(nullable = false)
Reason reason;
/** The fully qualified domain name of the domain that the bill is for. */
@Column(name = "domain_name", nullable = false)
String targetId;
@Nullable
@@ -120,6 +152,14 @@ public abstract class BillingEvent extends ImmutableObject
return clientId;
}
public long getDomainHistoryRevisionId() {
return domainHistoryRevisionId;
}
public String getDomainRepoId() {
return domainRepoId;
}
public DateTime getEventTime() {
return eventTime;
}
@@ -163,7 +203,7 @@ public abstract class BillingEvent extends ImmutableObject
return thisCastToDerived();
}
public B setId(Long id) {
public B setId(long id) {
getInstance().id = id;
return thisCastToDerived();
}
@@ -173,6 +213,16 @@ public abstract class BillingEvent extends ImmutableObject
return thisCastToDerived();
}
public B setDomainHistoryRevisionId(long domainHistoryRevisionId) {
getInstance().domainHistoryRevisionId = domainHistoryRevisionId;
return thisCastToDerived();
}
public B setDomainRepoId(String domainRepoId) {
getInstance().domainRepoId = domainRepoId;
return thisCastToDerived();
}
public B setEventTime(DateTime eventTime) {
getInstance().eventTime = eventTime;
return thisCastToDerived();
@@ -194,6 +244,7 @@ public abstract class BillingEvent extends ImmutableObject
}
public B setParent(Key<HistoryEntry> parentKey) {
// TODO(shicong): Figure out how to set domainHistoryRevisionId and domainRepoId
getInstance().parent = parentKey;
return thisCastToDerived();
}
@@ -213,9 +264,23 @@ public abstract class BillingEvent extends ImmutableObject
/** A one-time billable event. */
@ReportedOn
@Entity
@javax.persistence.Entity(name = "BillingEvent")
@javax.persistence.Table(
indexes = {
@javax.persistence.Index(columnList = "clientId"),
@javax.persistence.Index(columnList = "eventTime"),
@javax.persistence.Index(columnList = "billingTime"),
@javax.persistence.Index(columnList = "syntheticCreationTime"),
@javax.persistence.Index(columnList = "allocation_token_id")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_event_id"))
public static class OneTime extends BillingEvent {
/** The billable value. */
@AttributeOverrides({
@AttributeOverride(name = "money.amount", column = @Column(name = "cost_amount")),
@AttributeOverride(name = "money.currency", column = @Column(name = "cost_currency"))
})
Money cost;
/** When the cost should be billed. */
@@ -223,8 +288,8 @@ public abstract class BillingEvent extends ImmutableObject
DateTime billingTime;
/**
* The period in years of the action being billed for, if applicable, otherwise null.
* Used for financial reporting.
* The period in years of the action being billed for, if applicable, otherwise null. Used for
* financial reporting.
*/
@IgnoreSave(IfNull.class)
Integer periodYears = null;
@@ -240,15 +305,21 @@ public abstract class BillingEvent extends ImmutableObject
/**
* For {@link Flag#SYNTHETIC} events, a {@link Key} to the {@link BillingEvent} from which this
* OneTime was created. This is needed in order to properly match billing events against
* {@link Cancellation}s.
* OneTime was created. This is needed in order to properly match billing events against {@link
* Cancellation}s.
*/
Key<? extends BillingEvent> cancellationMatchingBillingEvent;
@Column(name = "cancellation_matching_billing_recurrence_id")
VKey<? extends BillingEvent> cancellationMatchingBillingEvent;
/**
* The {@link AllocationToken} used in the creation of this event, or null if one was not used.
*
* <p>TODO(shicong): Add foreign key constraint when AllocationToken schema is generated
*/
@Index @Nullable Key<AllocationToken> allocationToken;
@Column(name = "allocation_token_id")
@Index
@Nullable
VKey<AllocationToken> allocationToken;
public Money getCost() {
return cost;
@@ -266,14 +337,18 @@ public abstract class BillingEvent extends ImmutableObject
return syntheticCreationTime;
}
public Key<? extends BillingEvent> getCancellationMatchingBillingEvent() {
public VKey<? extends BillingEvent> getCancellationMatchingBillingEvent() {
return cancellationMatchingBillingEvent;
}
public Optional<Key<AllocationToken>> getAllocationToken() {
public Optional<VKey<AllocationToken>> getAllocationToken() {
return Optional.ofNullable(allocationToken);
}
public VKey<OneTime> createVKey() {
return VKey.createOfy(getClass(), Key.create(this));
}
@Override
public Builder asBuilder() {
return new Builder(clone(this));
@@ -311,12 +386,12 @@ public abstract class BillingEvent extends ImmutableObject
}
public Builder setCancellationMatchingBillingEvent(
Key<? extends BillingEvent> cancellationMatchingBillingEvent) {
VKey<? extends BillingEvent> cancellationMatchingBillingEvent) {
getInstance().cancellationMatchingBillingEvent = cancellationMatchingBillingEvent;
return this;
}
public Builder setAllocationToken(@Nullable Key<AllocationToken> allocationToken) {
public Builder setAllocationToken(@Nullable VKey<AllocationToken> allocationToken) {
getInstance().allocationToken = allocationToken;
return this;
}
@@ -361,6 +436,15 @@ public abstract class BillingEvent extends ImmutableObject
*/
@ReportedOn
@Entity
@javax.persistence.Entity(name = "BillingRecurrence")
@javax.persistence.Table(
indexes = {
@javax.persistence.Index(columnList = "clientId"),
@javax.persistence.Index(columnList = "eventTime"),
@javax.persistence.Index(columnList = "recurrenceEndTime"),
@javax.persistence.Index(columnList = "recurrence_time_of_year")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_recurrence_id"))
public static class Recurring extends BillingEvent {
/**
@@ -384,6 +468,10 @@ public abstract class BillingEvent extends ImmutableObject
* model, whereas the billing time is a fixed {@link org.joda.time.Duration} later.
*/
@Index
@Embedded
@AttributeOverrides({
@AttributeOverride(name = "timeString", column = @Column(name = "recurrence_time_of_year"))
})
TimeOfYear recurrenceTimeOfYear;
public DateTime getRecurrenceEndTime() {
@@ -394,6 +482,10 @@ public abstract class BillingEvent extends ImmutableObject
return recurrenceTimeOfYear;
}
public VKey<Recurring> createVKey() {
return VKey.createOfy(getClass(), Key.create(this));
}
@Override
public Builder asBuilder() {
return new Builder(clone(this));
@@ -434,6 +526,14 @@ public abstract class BillingEvent extends ImmutableObject
*/
@ReportedOn
@Entity
@javax.persistence.Entity(name = "BillingCancellation")
@javax.persistence.Table(
indexes = {
@javax.persistence.Index(columnList = "clientId"),
@javax.persistence.Index(columnList = "eventTime"),
@javax.persistence.Index(columnList = "billingTime")
})
@AttributeOverride(name = "id", column = @Column(name = "billing_cancellation_id"))
public static class Cancellation extends BillingEvent {
/** The billing time of the charge that is being cancelled. */
@@ -446,7 +546,8 @@ public abstract class BillingEvent extends ImmutableObject
* <p>Although the type is {@link Key} the name "ref" is preserved for historical reasons.
*/
@IgnoreSave(IfNull.class)
Key<BillingEvent.OneTime> refOneTime = null;
@Column(name = "billing_event_id")
VKey<BillingEvent.OneTime> refOneTime = null;
/**
* The recurring billing event to cancel, or null for non-autorenew cancellations.
@@ -454,13 +555,14 @@ public abstract class BillingEvent extends ImmutableObject
* <p>Although the type is {@link Key} the name "ref" is preserved for historical reasons.
*/
@IgnoreSave(IfNull.class)
Key<BillingEvent.Recurring> refRecurring = null;
@Column(name = "billing_recurrence_id")
VKey<BillingEvent.Recurring> refRecurring = null;
public DateTime getBillingTime() {
return billingTime;
}
public Key<? extends BillingEvent> getEventKey() {
public VKey<? extends BillingEvent> getEventKey() {
return firstNonNull(refOneTime, refRecurring);
}
@@ -492,13 +594,19 @@ public abstract class BillingEvent extends ImmutableObject
.setParent(historyEntry);
// Set the grace period's billing event using the appropriate Cancellation builder method.
if (gracePeriod.getOneTimeBillingEvent() != null) {
builder.setOneTimeEventKey(gracePeriod.getOneTimeBillingEvent());
builder.setOneTimeEventKey(
VKey.createOfy(BillingEvent.OneTime.class, gracePeriod.getOneTimeBillingEvent()));
} else if (gracePeriod.getRecurringBillingEvent() != null) {
builder.setRecurringEventKey(gracePeriod.getRecurringBillingEvent());
builder.setRecurringEventKey(
VKey.createOfy(BillingEvent.Recurring.class, gracePeriod.getRecurringBillingEvent()));
}
return builder.build();
}
public VKey<Cancellation> createVKey() {
return VKey.createOfy(getClass(), Key.create(this));
}
@Override
public Builder asBuilder() {
return new Builder(clone(this));
@@ -518,12 +626,12 @@ public abstract class BillingEvent extends ImmutableObject
return this;
}
public Builder setOneTimeEventKey(Key<BillingEvent.OneTime> eventKey) {
public Builder setOneTimeEventKey(VKey<BillingEvent.OneTime> eventKey) {
getInstance().refOneTime = eventKey;
return this;
}
public Builder setRecurringEventKey(Key<BillingEvent.Recurring> eventKey) {
public Builder setRecurringEventKey(VKey<BillingEvent.Recurring> eventKey) {
getInstance().refRecurring = eventKey;
return this;
}
@@ -540,9 +648,7 @@ public abstract class BillingEvent extends ImmutableObject
}
}
/**
* An event representing a modification of an existing one-time billing event.
*/
/** An event representing a modification of an existing one-time billing event. */
@ReportedOn
@Entity
public static class Modification extends BillingEvent {
@@ -29,20 +29,22 @@ import com.googlecode.objectify.annotation.Embed;
import com.googlecode.objectify.annotation.Index;
import google.registry.model.ImmutableObject;
import java.util.List;
import javax.persistence.Embeddable;
import org.joda.time.DateTime;
/**
* A time of year (month, day, millis of day) that can be stored in a sort-friendly format.
*
* <p>This is conceptually similar to {@code MonthDay} in Joda or more generally to Joda's
* {@code Partial}, but the parts we need are too simple to justify a full implementation of
* {@code Partial}.
* <p>This is conceptually similar to {@code MonthDay} in Joda or more generally to Joda's {@code
* Partial}, but the parts we need are too simple to justify a full implementation of {@code
* Partial}.
*
* <p>For simplicity, the native representation of this class's data is its stored format. This
* allows it to be embeddable with no translation needed and also delays parsing of the string on
* load until it's actually needed.
*/
@Embed
@Embeddable
public class TimeOfYear extends ImmutableObject {
/**
@@ -37,7 +37,7 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
/** A collection of {@link ContactResource} commands. */
public class ContactCommand {
/** The fields on "chgType" from {@link "http://tools.ietf.org/html/rfc5733"}. */
/** The fields on "chgType" from <a href="http://tools.ietf.org/html/rfc5733">RFC5733</a>. */
@XmlTransient
public static class ContactCreateOrChange extends ImmutableObject
implements ResourceCreateOrChange<ContactResource.Builder> {
@@ -111,8 +111,8 @@ public class ContactCommand {
}
/**
* A create command for a {@link ContactResource}, mapping "createType" from
* {@link "http://tools.ietf.org/html/rfc5733"}.
* A create command for a {@link ContactResource}, mapping "createType" from <a
* href="http://tools.ietf.org/html/rfc5733">RFC5733</a>}.
*/
@XmlType(propOrder = {"contactId", "postalInfo", "voice", "fax", "email", "authInfo", "disclose"})
@XmlRootElement
@@ -197,7 +197,9 @@ public class ContactResource extends EppResource
})
Disclose disclose;
@Override
public VKey<ContactResource> createVKey() {
// TODO(mmuller): create symmetric keys if we can ever reload both sides.
return VKey.createOfy(ContactResource.class, Key.create(this));
}
@@ -27,7 +27,7 @@ import javax.persistence.Embedded;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlType;
/** The "discloseType" from {@link "http://tools.ietf.org/html/rfc5733"}. */
/** The "discloseType" from <a href="http://tools.ietf.org/html/rfc5733">RFC5733</a>. */
@Embed
@Embeddable
@XmlType(propOrder = {"name", "org", "addr", "voice", "fax", "email"})
@@ -76,7 +76,7 @@ public class Disclose extends ImmutableObject {
return flag;
}
/** The "intLocType" from {@link "http://tools.ietf.org/html/rfc5733"}. */
/** The "intLocType" from <a href="http://tools.ietf.org/html/rfc5733">RFC5733</a>. */
@Embed
public static class PostalInfoChoice extends ImmutableObject {
@XmlAttribute
@@ -32,8 +32,8 @@ import javax.xml.bind.annotation.adapters.NormalizedStringAdapter;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
/**
* Implementation of both "postalInfoType" and "chgPostalInfoType" from {@link
* "http://tools.ietf.org/html/rfc5733"}.
* Implementation of both "postalInfoType" and "chgPostalInfoType" from <a href=
* "http://tools.ietf.org/html/rfc5733">RFC5733</a>.
*/
@Embed
@Embeddable
@@ -66,6 +66,7 @@ import google.registry.model.registry.Registry;
import google.registry.model.transfer.TransferData;
import google.registry.model.transfer.TransferStatus;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import google.registry.schema.replay.DatastoreAndSqlEntity;
import google.registry.util.CollectionUtils;
import java.util.HashSet;
@@ -106,6 +107,7 @@ import org.joda.time.Interval;
@javax.persistence.Index(columnList = "fullyQualifiedDomainName"),
@javax.persistence.Index(columnList = "tld")
})
@WithStringVKey
@ExternalMessagingName("domain")
public class DomainBase extends EppResource
implements DatastoreAndSqlEntity, ForeignKeyedEppResource, ResourceWithTransferData {
@@ -613,6 +615,12 @@ public class DomainBase extends EppResource
}
}
@Override
public VKey<DomainBase> createVKey() {
// TODO(mmuller): create symmetric keys if we can ever reload both sides.
return VKey.createOfy(DomainBase.class, Key.create(this));
}
/** Predicate to determine if a given {@link DesignatedContact} is the registrant. */
private static final Predicate<DesignatedContact> IS_REGISTRANT =
(DesignatedContact contact) -> DesignatedContact.Type.REGISTRANT.equals(contact.type);
@@ -30,7 +30,6 @@ import com.google.common.base.MoreObjects;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.googlecode.objectify.Key;
import google.registry.model.EppResource;
import google.registry.model.ImmutableObject;
import google.registry.model.contact.ContactResource;
@@ -72,18 +71,17 @@ public class DomainCommand {
T cloneAndLinkReferences(DateTime now) throws InvalidReferencesException;
}
/** The fields on "chgType" from {@link "http://tools.ietf.org/html/rfc5731"}. */
/** The fields on "chgType" from <a href="http://tools.ietf.org/html/rfc5731">RFC5731</a>. */
@XmlTransient
public static class DomainCreateOrChange<B extends DomainBase.Builder>
extends ImmutableObject implements ResourceCreateOrChange<B> {
public static class DomainCreateOrChange<B extends DomainBase.Builder> extends ImmutableObject
implements ResourceCreateOrChange<B> {
/** The contactId of the registrant who registered this domain. */
@XmlElement(name = "registrant")
String registrantContactId;
/** A resolved key to the registrant who registered this domain. */
@XmlTransient
Key<ContactResource> registrant;
@XmlTransient VKey<ContactResource> registrant;
/** Authorization info (aka transfer secret) of the domain. */
DomainAuthInfo authInfo;
@@ -93,7 +91,7 @@ public class DomainCommand {
}
@Nullable
public Key<ContactResource> getRegistrant() {
public VKey<ContactResource> getRegistrant() {
return registrant;
}
@@ -103,19 +101,20 @@ public class DomainCommand {
}
/**
* A create command for a {@link DomainBase}, mapping "createType" from
* {@link "http://tools.ietf.org/html/rfc5731"}.
* A create command for a {@link DomainBase}, mapping "createType" from <a
* href="http://tools.ietf.org/html/rfc5731">RFC5731</a>.
*/
@XmlRootElement
@XmlType(propOrder = {
"fullyQualifiedDomainName",
"period",
"nameserverFullyQualifiedHostNames",
"registrantContactId",
"foreignKeyedDesignatedContacts",
"authInfo"})
public static class Create
extends DomainCreateOrChange<DomainBase.Builder>
@XmlType(
propOrder = {
"fullyQualifiedDomainName",
"period",
"nameserverFullyQualifiedHostNames",
"registrantContactId",
"foreignKeyedDesignatedContacts",
"authInfo"
})
public static class Create extends DomainCreateOrChange<DomainBase.Builder>
implements CreateOrUpdate<Create> {
/** Fully qualified domain name, which serves as a unique identifier for this domain. */
@@ -128,8 +127,7 @@ public class DomainCommand {
Set<String> nameserverFullyQualifiedHostNames;
/** Resolved keys to hosts that are the nameservers for the domain. */
@XmlTransient
Set<Key<HostResource>> nameservers;
@XmlTransient Set<VKey<HostResource>> nameservers;
/** Foreign keyed associated contacts for the domain (other than registrant). */
@XmlElement(name = "contact")
@@ -159,7 +157,7 @@ public class DomainCommand {
return nullToEmptyImmutableCopy(nameserverFullyQualifiedHostNames);
}
public ImmutableSet<Key<HostResource>> getNameservers() {
public ImmutableSet<VKey<HostResource>> getNameservers() {
return nullToEmptyImmutableCopy(nameservers);
}
@@ -189,7 +187,7 @@ public class DomainCommand {
now);
for (DesignatedContact contact : contacts) {
if (DesignatedContact.Type.REGISTRANT.equals(contact.getType())) {
clone.registrant = contact.getContactKey().getOfyKey();
clone.registrant = contact.getContactKey();
clone.contacts = forceEmptyToNull(difference(contacts, contact));
break;
}
@@ -353,8 +351,7 @@ public class DomainCommand {
Set<String> nameserverFullyQualifiedHostNames;
/** Resolved keys to hosts that are the nameservers for the domain. */
@XmlTransient
Set<Key<HostResource>> nameservers;
@XmlTransient Set<VKey<HostResource>> nameservers;
/** Foreign keyed associated contacts for the domain (other than registrant). */
@XmlElement(name = "contact")
@@ -368,7 +365,7 @@ public class DomainCommand {
return nullSafeImmutableCopy(nameserverFullyQualifiedHostNames);
}
public ImmutableSet<Key<HostResource>> getNameservers() {
public ImmutableSet<VKey<HostResource>> getNameservers() {
return nullToEmptyImmutableCopy(nameservers);
}
@@ -418,7 +415,7 @@ public class DomainCommand {
}
}
private static Set<Key<HostResource>> linkHosts(
private static Set<VKey<HostResource>> linkHosts(
Set<String> fullyQualifiedHostNames, DateTime now) throws InvalidReferencesException {
if (fullyQualifiedHostNames == null) {
return null;
@@ -436,20 +433,18 @@ public class DomainCommand {
for (ForeignKeyedDesignatedContact contact : contacts) {
foreignKeys.add(contact.contactId);
}
ImmutableMap<String, Key<ContactResource>> loadedContacts =
ImmutableMap<String, VKey<ContactResource>> loadedContacts =
loadByForeignKeysCached(foreignKeys.build(), ContactResource.class, now);
ImmutableSet.Builder<DesignatedContact> linkedContacts = new ImmutableSet.Builder<>();
for (ForeignKeyedDesignatedContact contact : contacts) {
linkedContacts.add(
DesignatedContact.create(
contact.type,
VKey.createOfy(ContactResource.class, loadedContacts.get(contact.contactId))));
DesignatedContact.create(contact.type, loadedContacts.get(contact.contactId)));
}
return linkedContacts.build();
}
/** Loads keys to cached EPP resources by their foreign keys. */
private static <T extends EppResource> ImmutableMap<String, Key<T>> loadByForeignKeysCached(
private static <T extends EppResource> ImmutableMap<String, VKey<T>> loadByForeignKeysCached(
final Set<String> foreignKeys, final Class<T> clazz, final DateTime now)
throws InvalidReferencesException {
Map<String, ForeignKeyIndex<T>> fkis = ForeignKeyIndex.loadCached(clazz, foreignKeys, now);
@@ -20,7 +20,7 @@ import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlEnumValue;
import javax.xml.bind.annotation.XmlValue;
/** The "periodType" from {@link "http://tools.ietf.org/html/rfc5731"}. */
/** The "periodType" from <a href="http://tools.ietf.org/html/rfc5731">RFC5731</a>. */
@Embed
@javax.persistence.Embeddable
public class Period extends ImmutableObject {
@@ -29,11 +29,13 @@ import org.joda.time.DateTime;
* An individual price check item in version 0.12 of the fee extension on domain check commands.
* Items look like:
*
* <pre>{@code
* <fee:command name="renew" phase="sunrise" subphase="hello">
* <fee:period unit="y">1</fee:period>
* <fee:class>premium</fee:class>
* <fee:date>2017-05-17T13:22:21.0Z</fee:date>
* </fee:command>
* }</pre>
*
* In a change from previous versions of the extension, items do not contain domain names; instead,
* the names from the non-extension check element are used.
@@ -95,9 +95,6 @@ public class LaunchNotice extends ImmutableObject {
/**
* Validate the checksum of the notice against the domain label.
*
* @throws IllegalArgumentException
* @throws InvalidChecksumException
*/
public void validate(String domainLabel) throws InvalidChecksumException {
// According to http://tools.ietf.org/html/draft-lozano-tmch-func-spec-08#section-6.3, a TCNID
@@ -49,9 +49,9 @@ public enum GracePeriodStatus implements EppEnum {
AUTO_RENEW("autoRenewPeriod"),
/**
* This status value is used to describe a domain for which a <delete> command has been received,
* but the domain has not yet been purged because an opportunity exists to restore the domain and
* abort the deletion process.
* This status value is used to describe a domain for which a &lt;delete&gt; command has been
* received, but the domain has not yet been purged because an opportunity exists to restore the
* domain and abort the deletion process.
*/
REDEMPTION("redemptionPeriod"),
@@ -45,6 +45,8 @@ import google.registry.model.common.TimedTransitionProperty;
import google.registry.model.common.TimedTransitionProperty.TimeMapper;
import google.registry.model.common.TimedTransitionProperty.TimedTransition;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.VKey;
import google.registry.persistence.WithStringVKey;
import java.util.Optional;
import java.util.Set;
import javax.annotation.Nullable;
@@ -53,6 +55,7 @@ import org.joda.time.DateTime;
/** An entity representing an allocation token. */
@ReportedOn
@Entity
@WithStringVKey
public class AllocationToken extends BackupGroupRoot implements Buildable {
// Promotions should only move forward, and ENDED / CANCELLED are terminal states.
@@ -179,6 +182,10 @@ public class AllocationToken extends BackupGroupRoot implements Buildable {
return tokenStatusTransitions;
}
public VKey<AllocationToken> createVKey() {
return VKey.createOfy(getClass(), Key.create(this));
}
@Override
public Builder asBuilder() {
return new Builder(clone(this));
@@ -25,7 +25,7 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
* An allocation token extension that may be present on EPP domain commands.
*
* @see <a href="https://tools.ietf.org/html/draft-ietf-regext-allocation-token-04">the IETF
* draft</a> for full details.
* draft</a>
*/
@XmlRootElement(name = "allocationToken")
public class AllocationTokenExtension extends ImmutableObject implements CommandExtension {
@@ -44,8 +44,9 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
/**
* Container for generic street address.
*
* <p>This is the "addrType" type from {@link "http://tools.ietf.org/html/rfc5733"}. It also matches
* the "addrType" type from {@link "http://tools.ietf.org/html/draft-lozano-tmch-smd"}.
* <p>This is the "addrType" type from <a href="http://tools.ietf.org/html/rfc5733">RFC5733</a>. It
* also matches the "addrType" type from <a
* href="http://tools.ietf.org/html/draft-lozano-tmch-smd">Mark and Signed Mark Objects Mapping</a>.
*
* @see google.registry.model.contact.ContactAddress
* @see google.registry.model.mark.MarkAddress
@@ -29,8 +29,9 @@ import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
/**
* Container for generic E164 phone number.
*
* <p>This is the "e164" type from {@link "http://tools.ietf.org/html/rfc5733"}. It also matches the
* "e164Type" type from {@link "http://tools.ietf.org/html/draft-lozano-tmch-smd"}.
* <p>This is the "e164" type from <a href="http://tools.ietf.org/html/rfc5733">RFC5733</a>. It also
* matches the "e164Type" type from <a href="http://tools.ietf.org/html/draft-lozano-tmch-smd">Mark
* and Signed Mark Objects Mapping</a>
*
* <blockquote>
*
@@ -24,7 +24,7 @@ import javax.xml.bind.annotation.XmlElementWrapper;
import org.joda.time.DateTime;
/**
* A greeting, defined in {@link "http://tools.ietf.org/html/rfc5730"}.
* A greeting, defined in <a href="http://tools.ietf.org/html/rfc5730">RFC5730</a>.
*
* <p>It would be nice to make this a singleton, but we need the {@link #svDate} field to stay
* current.
@@ -32,7 +32,7 @@ import javax.xml.bind.annotation.XmlType;
/** A collection of {@link HostResource} commands. */
public class HostCommand {
/** The fields on "chgType" from {@link "http://tools.ietf.org/html/rfc5732"}. */
/** The fields on "chgType" from <a href="http://tools.ietf.org/html/rfc5732">RFC5732</a>. */
@XmlTransient
abstract static class HostCreateOrChange extends AbstractSingleResourceCommand
implements ResourceCreateOrChange<HostResource.Builder> {
@@ -42,13 +42,13 @@ public class HostCommand {
}
/**
* A create command for a {@link HostResource}, mapping "createType" from
* {@link "http://tools.ietf.org/html/rfc5732"}.
* A create command for a {@link HostResource}, mapping "createType" from <a
* href="http://tools.ietf.org/html/rfc5732">RFC5732</a>.
*/
@XmlType(propOrder = {"targetId", "inetAddresses" })
@XmlType(propOrder = {"targetId", "inetAddresses"})
@XmlRootElement
public static class Create
extends HostCreateOrChange implements ResourceCreateOrChange<HostResource.Builder> {
public static class Create extends HostCreateOrChange
implements ResourceCreateOrChange<HostResource.Builder> {
/** IP Addresses for this host. Can be null if this is an external host. */
@XmlElement(name = "addr")
Set<InetAddress> inetAddresses;
@@ -76,7 +76,7 @@ public class HostResource extends EppResource
@Index
@IgnoreSave(IfNull.class)
@DoNotHydrate
Key<DomainBase> superordinateDomain;
VKey<DomainBase> superordinateDomain;
/**
* The time that this resource was last transferred.
@@ -98,7 +98,7 @@ public class HostResource extends EppResource
return fullyQualifiedHostName;
}
public Key<DomainBase> getSuperordinateDomain() {
public VKey<DomainBase> getSuperordinateDomain() {
return superordinateDomain;
}
@@ -123,7 +123,9 @@ public class HostResource extends EppResource
return fullyQualifiedHostName;
}
public VKey<HostResource> createKey() {
@Override
public VKey<HostResource> createVKey() {
// TODO(mmuller): create symmetric keys if we can ever reload both sides.
return VKey.createOfy(HostResource.class, Key.create(this));
}
@@ -143,18 +145,18 @@ public class HostResource extends EppResource
*
* <p>If the host is not subordinate the domain can be null and we just return last transfer time.
*
* @param superordinateDomain the loaded superordinate domain, which must match the key in
* the {@link #superordinateDomain} field. Passing it as a parameter allows the caller to
* control the degree of consistency used to load it.
* @param superordinateDomain the loaded superordinate domain, which must match the key in the
* {@link #superordinateDomain} field. Passing it as a parameter allows the caller to control
* the degree of consistency used to load it.
*/
public DateTime computeLastTransferTime(@Nullable DomainBase superordinateDomain) {
public DateTime computeLastTransferTime(@Nullable DomainBase superordinateDomain) {
if (!isSubordinate()) {
checkArgument(superordinateDomain == null);
return getLastTransferTime();
}
checkArgument(
superordinateDomain != null
&& Key.create(superordinateDomain).equals(getSuperordinateDomain()));
&& superordinateDomain.createVKey().equals(getSuperordinateDomain()));
DateTime lastSuperordinateChange =
Optional.ofNullable(getLastSuperordinateChange()).orElse(getCreationTime());
DateTime lastTransferOfCurrentSuperordinate =
@@ -205,7 +207,7 @@ public class HostResource extends EppResource
difference(getInstance().getInetAddresses(), inetAddresses)));
}
public Builder setSuperordinateDomain(Key<DomainBase> superordinateDomain) {
public Builder setSuperordinateDomain(VKey<DomainBase> superordinateDomain) {
getInstance().superordinateDomain = superordinateDomain;
return this;
}
@@ -43,6 +43,7 @@ import google.registry.model.annotations.ReportedOn;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.DomainBase;
import google.registry.model.host.HostResource;
import google.registry.persistence.VKey;
import google.registry.util.NonFinalForTesting;
import java.util.Map;
import java.util.Optional;
@@ -99,7 +100,7 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
* <p>This field holds a key to the only referenced resource. It is named "topReference" for
* historical reasons.
*/
Key<E> topReference;
VKey<E> topReference;
public String getForeignKey() {
return foreignKey;
@@ -109,7 +110,7 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
return deletionTime;
}
public Key<E> getResourceKey() {
public VKey<E> getResourceKey() {
return topReference;
}
@@ -125,7 +126,7 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
@SuppressWarnings("unchecked")
Class<E> resourceClass = (Class<E>) resource.getClass();
ForeignKeyIndex<E> instance = instantiate(mapToFkiClass(resourceClass));
instance.topReference = Key.create(resource);
instance.topReference = (VKey<E>) resource.createVKey();
instance.foreignKey = resource.getForeignKey();
instance.deletionTime = deletionTime;
return instance;
@@ -141,18 +142,18 @@ public abstract class ForeignKeyIndex<E extends EppResource> extends BackupGroup
/**
* Loads a {@link Key} to an {@link EppResource} from Datastore by foreign key.
*
* <p>Returns null if no foreign key index with this foreign key was ever created, or if the
* most recently created foreign key index was deleted before time "now". This method does not
* actually check that the referenced resource actually exists. However, for normal epp resources,
* it is safe to assume that the referenced resource exists if the foreign key index does.
* <p>Returns null if no foreign key index with this foreign key was ever created, or if the most
* recently created foreign key index was deleted before time "now". This method does not actually
* check that the referenced resource actually exists. However, for normal epp resources, it is
* safe to assume that the referenced resource exists if the foreign key index does.
*
* @param clazz the resource type to load
* @param foreignKey id to match
* @param now the current logical time to use when checking for soft deletion of the foreign key
* index
* index
*/
@Nullable
public static <E extends EppResource> Key<E> loadAndGetKey(
public static <E extends EppResource> VKey<E> loadAndGetKey(
Class<E> clazz, String foreignKey, DateTime now) {
ForeignKeyIndex<E> index = load(clazz, foreignKey, now);
return (index == null) ? null : index.getResourceKey();
@@ -14,6 +14,7 @@
package google.registry.model.ofy;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
@@ -157,7 +158,7 @@ public class DatastoreTransactionManager implements TransactionManager {
@Override
public <T> ImmutableList<T> load(Iterable<VKey<T>> keys) {
Iterator<Key<T>> iter =
StreamSupport.stream(keys.spliterator(), false).map(key -> key.getOfyKey()).iterator();
StreamSupport.stream(keys.spliterator(), false).map(VKey::getOfyKey).iterator();
// The lambda argument to keys() effectively converts Iterator -> Iterable.
return ImmutableList.copyOf(getOfy().load().keys(() -> iter).values());
@@ -170,7 +171,18 @@ public class DatastoreTransactionManager implements TransactionManager {
}
@Override
public <T> void delete(VKey<T> key) {
public void delete(VKey<?> key) {
getOfy().delete().key(key.getOfyKey()).now();
}
@Override
public void delete(Iterable<? extends VKey<?>> vKeys) {
// We have to create a list to work around the wildcard capture issue here.
// See https://docs.oracle.com/javase/tutorial/java/generics/capture.html
ImmutableList<Key<?>> list =
StreamSupport.stream(vKeys.spliterator(), false)
.map(VKey::getOfyKey)
.collect(toImmutableList());
getOfy().delete().keys(list).now();
}
}
@@ -34,10 +34,9 @@ import com.googlecode.objectify.annotation.EntitySubclass;
import com.googlecode.objectify.impl.translate.TranslatorFactory;
import com.googlecode.objectify.impl.translate.opt.joda.MoneyStringTranslatorFactory;
import google.registry.config.RegistryEnvironment;
import google.registry.model.Buildable;
import google.registry.model.EntityClasses;
import google.registry.model.ImmutableObject;
import google.registry.model.contact.ContactResource;
import google.registry.model.host.HostResource;
import google.registry.model.translators.BloomFilterOfStringTranslatorFactory;
import google.registry.model.translators.CidrAddressBlockTranslatorFactory;
import google.registry.model.translators.CommitLogRevisionsTranslatorFactory;
@@ -131,7 +130,7 @@ public class ObjectifyService {
new InetAddressTranslatorFactory(),
new MoneyStringTranslatorFactory(),
new ReadableInstantUtcTranslatorFactory(),
new VKeyTranslatorFactory(HostResource.class, ContactResource.class),
new VKeyTranslatorFactory(),
new UpdateAutoTimestampTranslatorFactory())) {
factory().getTranslators().add(translatorFactory);
}
@@ -169,10 +168,16 @@ public class ObjectifyService {
}
com.googlecode.objectify.ObjectifyService.register(clazz);
// Autogenerated ids make the commit log code very difficult since we won't always be able
// to create a key for an entity immediately when requesting a save. Disallow that here.
checkState(
!factory().getMetadata(clazz).getKeyMetadata().isIdGeneratable(),
"Can't register %s: Autogenerated ids (@Id on a Long) are not supported.", kind);
// to create a key for an entity immediately when requesting a save. So, we require such
// entities to implement google.registry.model.Buildable as its build() function allocates the
// id to the entity.
if (factory().getMetadata(clazz).getKeyMetadata().isIdGeneratable()) {
checkState(
Buildable.class.isAssignableFrom(clazz),
"Can't register %s: Entity with autogenerated ids (@Id on a Long) must implement"
+ " google.registry.model.Buildable.",
kind);
}
}
}
@@ -19,6 +19,7 @@ import com.googlecode.objectify.annotation.Embed;
import google.registry.model.ImmutableObject;
import google.registry.model.eppcommon.Trid;
import google.registry.model.eppoutput.EppResponse.ResponseData;
import javax.persistence.Embeddable;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
@@ -29,11 +30,12 @@ import org.joda.time.DateTime;
/** The {@link ResponseData} returned when completing a pending action on a domain. */
@XmlTransient
public abstract class PendingActionNotificationResponse
extends ImmutableObject implements ResponseData {
@Embeddable
public class PendingActionNotificationResponse extends ImmutableObject implements ResponseData {
/** The inner name type that contains a name and the result boolean. */
@Embed
@Embeddable
static class NameOrId extends ImmutableObject {
@XmlValue
String value;
@@ -25,6 +25,7 @@ import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.EntitySubclass;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Ignore;
import com.googlecode.objectify.annotation.Index;
import com.googlecode.objectify.annotation.Parent;
import google.registry.model.Buildable;
@@ -39,10 +40,23 @@ import google.registry.model.poll.PendingActionNotificationResponse.DomainPendin
import google.registry.model.poll.PendingActionNotificationResponse.HostPendingActionNotificationResponse;
import google.registry.model.reporting.HistoryEntry;
import google.registry.model.transfer.TransferData.TransferServerApproveEntity;
import google.registry.model.transfer.TransferResponse;
import google.registry.model.transfer.TransferResponse.ContactTransferResponse;
import google.registry.model.transfer.TransferResponse.DomainTransferResponse;
import google.registry.persistence.WithLongVKey;
import java.util.List;
import java.util.Optional;
import javax.persistence.AttributeOverride;
import javax.persistence.AttributeOverrides;
import javax.persistence.Column;
import javax.persistence.DiscriminatorColumn;
import javax.persistence.DiscriminatorValue;
import javax.persistence.Embedded;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Inheritance;
import javax.persistence.InheritanceType;
import javax.persistence.Transient;
import org.joda.time.DateTime;
/**
@@ -62,34 +76,58 @@ import org.joda.time.DateTime;
* <p>Poll messages are identified externally by registrars using the format defined in {@link
* PollMessageExternalKeyConverter}.
*
* @see <a href="https://tools.ietf.org/html/rfc5730#section-2.9.2.3">RFC5730 - EPP - &ltpoll&gt
* @see <a href="https://tools.ietf.org/html/rfc5730#section-2.9.2.3">RFC5730 - EPP - &lt;poll&gt;
* Command</a>
*/
@Entity
@ReportedOn
@ExternalMessagingName("message")
@javax.persistence.Entity
@Inheritance(strategy = InheritanceType.SINGLE_TABLE)
@DiscriminatorColumn(name = "type")
@javax.persistence.Table(
indexes = {
@javax.persistence.Index(columnList = "registrar_id"),
@javax.persistence.Index(columnList = "eventTime")
})
public abstract class PollMessage extends ImmutableObject
implements Buildable, TransferServerApproveEntity {
/** Entity id. */
@Id
long id;
@javax.persistence.Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "poll_message_id")
Long id;
@Parent
@DoNotHydrate
Key<HistoryEntry> parent;
@Parent @DoNotHydrate @Transient Key<HistoryEntry> parent;
/** The registrar that this poll message will be delivered to. */
@Index
@Column(name = "registrar_id", nullable = false)
String clientId;
/** The time when the poll message should be delivered. May be in the future. */
@Index
@Column(nullable = false)
DateTime eventTime;
/** Human readable message that will be returned with this poll message. */
@Column(name = "message")
String msg;
@Ignore String domainRepoId;
@Ignore String contactRepoId;
@Ignore String hostRepoId;
@Ignore Long domainRevisionId;
@Ignore Long contactRevisionId;
@Ignore Long hostRevisionId;
public Key<HistoryEntry> getParentKey() {
return parent;
}
@@ -180,15 +218,78 @@ public abstract class PollMessage extends ImmutableObject
* <p>One-time poll messages are deleted from Datastore once they have been delivered and ACKed.
*/
@EntitySubclass(index = false)
@javax.persistence.Entity
@DiscriminatorValue("ONE_TIME")
@WithLongVKey
public static class OneTime extends PollMessage {
// Response data. Objectify cannot persist a base class type, so we must have a separate field
// to hold every possible derived type of ResponseData that we might store.
@Transient
List<ContactPendingActionNotificationResponse> contactPendingActionNotificationResponses;
List<ContactTransferResponse> contactTransferResponses;
@Transient List<ContactTransferResponse> contactTransferResponses;
@Transient
List<DomainPendingActionNotificationResponse> domainPendingActionNotificationResponses;
List<DomainTransferResponse> domainTransferResponses;
List<HostPendingActionNotificationResponse> hostPendingActionNotificationResponses;
@Transient List<DomainTransferResponse> domainTransferResponses;
@Transient List<HostPendingActionNotificationResponse> hostPendingActionNotificationResponses;
@Ignore
@Embedded
@AttributeOverrides({
@AttributeOverride(
name = "nameOrId.value",
column = @Column(name = "pending_action_response_name_or_id")),
@AttributeOverride(
name = "nameOrId.actionResult",
column = @Column(name = "pending_action_response_action_result")),
@AttributeOverride(
name = "trid.serverTransactionId",
column = @Column(name = "pending_action_response_server_txn_id")),
@AttributeOverride(
name = "trid.clientTransactionId",
column = @Column(name = "pending_action_response_client_txn_id")),
@AttributeOverride(
name = "processedDate",
column = @Column(name = "pending_action_response_processed_date"))
})
PendingActionNotificationResponse pendingActionNotificationResponse;
@Ignore
@Embedded
@AttributeOverrides({
@AttributeOverride(
name = "transferStatus",
column = @Column(name = "transfer_response_transfer_status")),
@AttributeOverride(
name = "gainingClientId",
column = @Column(name = "transfer_response_gaining_registrar_id")),
@AttributeOverride(
name = "transferRequestTime",
column = @Column(name = "transfer_response_transfer_request_time")),
@AttributeOverride(
name = "losingClientId",
column = @Column(name = "transfer_response_losing_registrar_id")),
@AttributeOverride(
name = "pendingTransferExpirationTime",
column = @Column(name = "transfer_response_pending_transfer_expiration_time"))
})
TransferResponse transferResponse;
@Ignore
@Column(name = "transfer_response_domain_name")
String fullyQualifiedDomainName;
@Ignore
@Column(name = "transfer_response_domain_expiration_time")
DateTime extendedRegistrationExpirationTime;
@Ignore
@Column(name = "transfer_response_contact_id")
String contactId;
@Override
public Builder asBuilder() {
@@ -265,9 +366,13 @@ public abstract class PollMessage extends ImmutableObject
* happens.
*/
@EntitySubclass(index = false)
@javax.persistence.Entity
@DiscriminatorValue("AUTORENEW")
@WithLongVKey
public static class Autorenew extends PollMessage {
/** The target id of the autorenew event. */
@Column(name = "autorenew_domain_name")
String targetId;
/** The autorenew recurs annually between {@link #eventTime} and this time. */
@@ -158,8 +158,8 @@ public final class ReservedList
/**
* Gets a ReservedList by name using the caching layer.
*
* @return An Optional<ReservedList> that has a value if a reserved list exists by the given name,
* or absent if not.
* @return An Optional&lt;ReservedList&gt; that has a value if a reserved list exists by the given
* name, or absent if not.
* @throws UncheckedExecutionException if some other error occurs while trying to load the
* ReservedList from the cache or Datastore.
*/
@@ -22,8 +22,8 @@ public final class IcannReportingTypes {
* Represents the set of possible ICANN Monthly Registry Functions Activity Report fields.
*
* <p>Refer to the <a
* href="https://newgtlds.icann.org/sites/default/files/agreements/agreement-approved-09jan14-en.htm#_DV_M278>ICANN
* registry agreement Specification 3 Section 2</a> for details.
* href="https://newgtlds.icann.org/sites/default/files/agreements/agreement-approved-09jan14-en.htm#_DV_M278">
* ICANN registry agreement Specification 3 Section 2</a> for details.
*/
public enum ActivityReportField {
DOMAIN_CHECK("srs-dom-check"),
@@ -16,6 +16,8 @@ package google.registry.model.transfer;
import google.registry.model.Buildable.GenericBuilder;
import google.registry.model.ImmutableObject;
import javax.persistence.EnumType;
import javax.persistence.Enumerated;
import javax.persistence.MappedSuperclass;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlTransient;
@@ -31,6 +33,7 @@ public abstract class BaseTransferObject extends ImmutableObject {
* will always be non-null.
*/
@XmlElement(name = "trStatus")
@Enumerated(EnumType.STRING)
TransferStatus transferStatus;
/** The gaining registrar of the current or last transfer. Can be null if never transferred. */
@@ -17,6 +17,7 @@ package google.registry.model.transfer;
import com.googlecode.objectify.annotation.Embed;
import google.registry.model.EppResource;
import google.registry.model.eppoutput.EppResponse.ResponseData;
import javax.persistence.Embeddable;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlTransient;
@@ -28,7 +29,8 @@ import org.joda.time.DateTime;
* are common to all transfer responses; derived classes add resource specific fields.
*/
@XmlTransient
public abstract class TransferResponse extends BaseTransferObject implements ResponseData {
@Embeddable
public class TransferResponse extends BaseTransferObject implements ResponseData {
/** An adapter to output the XML in response to a transfer command on a domain. */
@Embed
@@ -16,13 +16,12 @@ package google.registry.model.translators;
import static com.google.common.base.Functions.identity;
import static com.google.common.collect.ImmutableMap.toImmutableMap;
import static google.registry.model.EntityClasses.ALL_CLASSES;
import com.google.appengine.api.datastore.Key;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableMap;
import com.googlecode.objectify.annotation.EntitySubclass;
import google.registry.persistence.VKey;
import java.util.List;
import java.util.stream.Stream;
/**
* Translator factory for VKey.
@@ -34,21 +33,16 @@ public class VKeyTranslatorFactory extends AbstractSimpleTranslatorFactory<VKey,
// Class registry allowing us to restore the original class object from the unqualified class
// name, which is all the datastore key gives us.
private final ImmutableMap<String, Class> classRegistry;
// Note that entities annotated with @EntitySubclass are removed because they share the same
// kind of the key with their parent class.
private static final ImmutableMap<String, Class> CLASS_REGISTRY =
ALL_CLASSES.stream()
.filter(clazz -> !clazz.isAnnotationPresent(EntitySubclass.class))
.collect(toImmutableMap(com.googlecode.objectify.Key::getKind, identity()));
;
public VKeyTranslatorFactory(Class... refClasses) {
public VKeyTranslatorFactory() {
super(VKey.class);
// Store a registry of all classes by their unqualified name.
classRegistry =
Stream.of(refClasses)
.collect(
toImmutableMap(
clazz -> {
List<String> nameComponent = Splitter.on('.').splitToList(clazz.getName());
return nameComponent.get(nameComponent.size() - 1);
},
identity()));
}
@Override
@@ -57,14 +51,16 @@ public class VKeyTranslatorFactory extends AbstractSimpleTranslatorFactory<VKey,
@Override
public VKey loadValue(Key datastoreValue) {
// TODO(mmuller): we need to call a method on refClass to also reconstitute the SQL key.
return VKey.createOfy(
classRegistry.get(datastoreValue.getKind()),
com.googlecode.objectify.Key.create(datastoreValue));
return datastoreValue == null
? null
: VKey.createOfy(
CLASS_REGISTRY.get(datastoreValue.getKind()),
com.googlecode.objectify.Key.create(datastoreValue));
}
@Override
public Key saveValue(VKey key) {
return key.getOfyKey().getRaw();
return key == null ? null : key.getOfyKey().getRaw();
}
};
}
@@ -15,7 +15,6 @@
package google.registry.persistence.converter;
import avro.shaded.com.google.common.collect.Maps;
import google.registry.model.common.TimedTransitionProperty;
import google.registry.model.registry.Registry.BillingCostTransition;
import java.util.Map;
import javax.persistence.Converter;
@@ -23,8 +22,8 @@ import org.joda.money.Money;
import org.joda.time.DateTime;
/**
* JPA converter for storing/retrieving {@link TimedTransitionProperty <Money, BillingCostTransition
* >} objects.
* JPA converter for storing/retrieving {@code TimedTransitionProperty<Money,BillingCostTransition>}
* objects.
*/
@Converter(autoApply = true)
public class BillingCostTransitionConverter
@@ -0,0 +1,35 @@
// Copyright 2020 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.persistence.converter;
import google.registry.model.billing.BillingEvent.Flag;
import java.util.Set;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/** JPA {@link AttributeConverter} for storing/retrieving {@link Set}. */
@Converter(autoApply = true)
public class BillingEventFlagSetConverter extends StringSetConverterBase<Flag> {
@Override
String toString(Flag element) {
return element.name();
}
@Override
Flag fromString(String value) {
return Flag.valueOf(value);
}
}
@@ -15,12 +15,11 @@
package google.registry.persistence.converter;
import google.registry.util.CidrAddressBlock;
import java.util.List;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/**
* JPA {@link AttributeConverter} for storing/retrieving {@link List<CidrAddressBlock>} objects.
* JPA {@link AttributeConverter} for storing/retrieving {@code List<CidrAddressBlock>} objects.
* TODO(shicong): Investigate if we can have one converter for any List type
*/
@Converter(autoApply = true)
@@ -21,7 +21,7 @@ import java.util.Map;
import javax.persistence.Converter;
import org.joda.money.CurrencyUnit;
/** JPA converter for storing/retrieving {@link Map <CurrencyUnit, BillingAccountEntry>} objects. */
/** JPA converter for storing/retrieving {@code Map<CurrencyUnit, BillingAccountEntry>} objects. */
@Converter(autoApply = true)
public class CurrencyToBillingConverter
extends StringMapConverterBase<CurrencyUnit, BillingAccountEntry> {
@@ -16,11 +16,10 @@ package google.registry.persistence.converter;
import google.registry.model.contact.Disclose.PostalInfoChoice;
import google.registry.model.contact.PostalInfo;
import java.util.List;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/** JPA {@link AttributeConverter} for storing/retrieving {@link List < PostalInfoChoice >}. */
/** JPA {@link AttributeConverter} for storing/retrieving {@code List<PostalInfoChoice>}. */
@Converter(autoApply = true)
public class PostalInfoChoiceListConverter extends StringListConverterBase<PostalInfoChoice> {
@@ -15,11 +15,10 @@
package google.registry.persistence.converter;
import google.registry.model.registrar.RegistrarContact.Type;
import java.util.Set;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/** JPA {@link AttributeConverter} for storing/retrieving {@link Set<Type>}. */
/** JPA {@link AttributeConverter} for storing/retrieving {@code Set<Type>}. */
@Converter(autoApply = true)
public class RegistrarPocSetConverter extends StringSetConverterBase<Type> {
@Override
@@ -15,11 +15,10 @@
package google.registry.persistence.converter;
import google.registry.model.eppcommon.StatusValue;
import java.util.Set;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/** JPA {@link AttributeConverter} for storing/retrieving {@link Set<StatusValue>}. */
/** JPA {@link AttributeConverter} for storing/retrieving {@code Set<StatusValue>}. */
@Converter(autoApply = true)
public class StatusValueSetConverter extends StringSetConverterBase<StatusValue> {
@@ -37,7 +37,7 @@ import org.hibernate.type.descriptor.sql.SqlTypeDescriptor;
/**
* The {@link JavaTypeDescriptor} and {@link SqlTypeDescriptor} for {@link StringCollection}.
*
* <p>A {@link StringCollection} object is a simple wrapper for a {@link Collection<String>} which
* <p>A {@link StringCollection} object is a simple wrapper for a {@code Collection<String>} which
* can be stored as a string array in the database. The {@link JavaTypeDescriptor} and {@link
* SqlTypeDescriptor} is used by JPA/Hibernate to map between the collection and {@link Array} which
* is the actual type that JDBC uses to read from and write to the database.
@@ -14,11 +14,10 @@
package google.registry.persistence.converter;
import java.util.List;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/** JPA {@link AttributeConverter} for storing/retrieving {@link List<String>}. */
/** JPA {@link AttributeConverter} for storing/retrieving {@code List<String>}. */
@Converter(autoApply = true)
public class StringListConverter extends StringListConverterBase<String> {
@@ -36,7 +36,7 @@ import org.hibernate.type.descriptor.sql.SqlTypeDescriptor;
/**
* The {@link JavaTypeDescriptor} and {@link SqlTypeDescriptor} for {@link StringMap}.
*
* <p>A {@link StringMap} object is a simple wrapper for a {@link Map <String, String>} which can be
* <p>A {@link StringMap} object is a simple wrapper for a {@code Map<String, String>} which can be
* stored in a column with data type of hstore in the database. The {@link JavaTypeDescriptor} and
* {@link SqlTypeDescriptor} is used by JPA/Hibernate to map between the map and hstore which is the
* actual type that JDBC uses to read from and write to the database.
@@ -156,7 +156,7 @@ public class StringMapDescriptor extends AbstractTypeDescriptor<StringMap>
};
}
/** A simple wrapper class for {@link Map<String, String>}. */
/** A simple wrapper class for {@code Map<String, String>}. */
public static class StringMap {
private Map<String, String> map;
@@ -169,7 +169,7 @@ public class StringMapDescriptor extends AbstractTypeDescriptor<StringMap>
return new StringMap(ImmutableMap.copyOf(map));
}
/** Returns the underlying {@link Map<String, String>} object. */
/** Returns the underlying {@code Map<String, String>} object. */
public Map<String, String> getMap() {
return map;
}
@@ -14,11 +14,10 @@
package google.registry.persistence.converter;
import java.util.Set;
import javax.persistence.AttributeConverter;
import javax.persistence.Converter;
/** JPA {@link AttributeConverter} for storing/retrieving {@link Set<String>}. */
/** JPA {@link AttributeConverter} for storing/retrieving {@code Set<String>}. */
@Converter(autoApply = true)
public class StringSetConverter extends StringSetConverterBase<String> {
@@ -15,7 +15,6 @@
package google.registry.persistence.converter;
import com.google.common.collect.Maps;
import google.registry.model.common.TimedTransitionProperty;
import google.registry.model.registry.Registry.TldState;
import google.registry.model.registry.Registry.TldStateTransition;
import java.util.Map;
@@ -23,7 +22,7 @@ import javax.persistence.Converter;
import org.joda.time.DateTime;
/**
* JPA converter for storing/retrieving {@link TimedTransitionProperty<TldState,
* JPA converter for storing/retrieving {@code TimedTransitionProperty<TldState,
* TldStateTransition>} objects.
*/
@Converter(autoApply = true)
@@ -278,7 +278,7 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
.getResultList());
}
private <T> int internalDelete(VKey<T> key) {
private int internalDelete(VKey<?> key) {
checkArgumentNotNull(key, "key must be specified");
assertInTransaction();
EntityType<?> entityType = getEntityType(key.getKind());
@@ -291,10 +291,16 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
}
@Override
public <T> void delete(VKey<T> key) {
public void delete(VKey<?> key) {
internalDelete(key);
}
@Override
public void delete(Iterable<? extends VKey<?>> vKeys) {
checkArgumentNotNull(vKeys, "vKeys must be specified");
vKeys.forEach(this::internalDelete);
}
@Override
public <T> void assertDelete(VKey<T> key) {
if (internalDelete(key) != 1) {
@@ -17,6 +17,7 @@ package google.registry.persistence.transaction;
import com.google.common.collect.ImmutableCollection;
import com.google.common.collect.ImmutableList;
import google.registry.persistence.VKey;
import java.util.NoSuchElementException;
import java.util.Optional;
import java.util.function.Supplier;
import org.joda.time.DateTime;
@@ -114,7 +115,7 @@ public interface TransactionManager {
<T> T load(VKey<T> key);
/**
* Leads the set of entities by their key id.
* Loads the set of entities by their key id.
*
* @throws NoSuchElementException if any of the keys are not found.
*/
@@ -124,5 +125,8 @@ public interface TransactionManager {
<T> ImmutableList<T> loadAll(Class<T> clazz);
/** Deletes the entity by its id. */
<T> void delete(VKey<T> key);
void delete(VKey<?> key);
/** Deletes the set of entities by their key id. */
void delete(Iterable<? extends VKey<?>> keys);
}
@@ -14,6 +14,7 @@
package google.registry.rdap;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.EppResourceUtils.loadByForeignKey;
import static google.registry.model.index.ForeignKeyIndex.loadAndGetKey;
import static google.registry.model.ofy.ObjectifyService.ofy;
@@ -28,10 +29,10 @@ import com.google.common.collect.Streams;
import com.google.common.flogger.FluentLogger;
import com.google.common.net.InetAddresses;
import com.google.common.primitives.Booleans;
import com.googlecode.objectify.Key;
import com.googlecode.objectify.cmd.Query;
import google.registry.model.domain.DomainBase;
import google.registry.model.host.HostResource;
import google.registry.persistence.VKey;
import google.registry.rdap.RdapJsonFormatter.OutputDataType;
import google.registry.rdap.RdapMetrics.EndpointType;
import google.registry.rdap.RdapMetrics.SearchType;
@@ -50,6 +51,7 @@ import java.util.Comparator;
import java.util.List;
import java.util.Optional;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import javax.inject.Inject;
/**
@@ -61,10 +63,9 @@ import javax.inject.Inject;
* (RDAP) Query Format</a>
* @see <a href="http://tools.ietf.org/html/rfc7483">RFC 7483: JSON Responses for the Registration
* Data Access Protocol (RDAP)</a>
*
* TODO(guyben):This isn't required by the RDAP Technical Implementation Guide, and hence should be
* deleted, at least until it's actually required.
*/
// TODO: This isn't required by the RDAP Technical Implementation Guide, and hence should be
// deleted, at least until it's actually required.
@Action(
service = Action.Service.PUBAPI,
path = "/rdap/domains",
@@ -244,7 +245,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
*/
private DomainSearchResponse searchByNameserverLdhName(
final RdapSearchPattern partialStringQuery) {
Iterable<Key<HostResource>> hostKeys = getNameserverRefsByLdhName(partialStringQuery);
Iterable<VKey<HostResource>> hostKeys = getNameserverRefsByLdhName(partialStringQuery);
if (Iterables.isEmpty(hostKeys)) {
metricInformationBuilder.setNumHostsRetrieved(0);
throw new NotFoundException("No matching nameservers found");
@@ -262,7 +263,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
* initial string is not required (e.g. "*.example.tld" is valid), because we can look up the
* domain and just list all of its subordinate hosts.
*/
private Iterable<Key<HostResource>> getNameserverRefsByLdhName(
private Iterable<VKey<HostResource>> getNameserverRefsByLdhName(
final RdapSearchPattern partialStringQuery) {
// Handle queries without a wildcard.
if (!partialStringQuery.getHasWildcard()) {
@@ -293,11 +294,13 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
return query.keys();
return StreamSupport.stream(query.keys().spliterator(), false)
.map(key -> VKey.createOfy(HostResource.class, key))
.collect(toImmutableSet());
}
/** Assembles a list of {@link HostResource} keys by name when the pattern has no wildcard. */
private Iterable<Key<HostResource>> getNameserverRefsByLdhNameWithoutWildcard(
private Iterable<VKey<HostResource>> getNameserverRefsByLdhNameWithoutWildcard(
final RdapSearchPattern partialStringQuery) {
// If we need to check the sponsoring registrar, we need to load the resource rather than just
// the key.
@@ -311,9 +314,9 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
return (!host.isPresent()
|| !desiredRegistrar.get().equals(host.get().getPersistedCurrentSponsorClientId()))
? ImmutableList.of()
: ImmutableList.of(Key.create(host.get()));
: ImmutableList.of(host.get().createVKey());
} else {
Key<HostResource> hostKey =
VKey<HostResource> hostKey =
loadAndGetKey(
HostResource.class,
partialStringQuery.getInitialString(),
@@ -323,7 +326,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
}
/** Assembles a list of {@link HostResource} keys by name using a superordinate domain suffix. */
private Iterable<Key<HostResource>> getNameserverRefsByLdhNameWithSuffix(
private Iterable<VKey<HostResource>> getNameserverRefsByLdhNameWithSuffix(
final RdapSearchPattern partialStringQuery) {
// The suffix must be a domain that we manage. That way, we can look up the domain and search
// through the subordinate hosts. This is more efficient, and lets us permit wildcard searches
@@ -339,7 +342,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
"A suffix in a lookup by nameserver name "
+ "must be a domain defined in the system"));
Optional<String> desiredRegistrar = getDesiredRegistrar();
ImmutableList.Builder<Key<HostResource>> builder = new ImmutableList.Builder<>();
ImmutableList.Builder<VKey<HostResource>> builder = new ImmutableList.Builder<>();
for (String fqhn : ImmutableSortedSet.copyOf(domainBase.getSubordinateHosts())) {
// We can't just check that the host name starts with the initial query string, because
// then the query ns.exam*.example.com would match against nameserver ns.example.com.
@@ -352,10 +355,10 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
shouldIncludeDeleted() ? START_OF_TIME : getRequestTime());
if (host.isPresent()
&& desiredRegistrar.get().equals(host.get().getPersistedCurrentSponsorClientId())) {
builder.add(Key.create(host.get()));
builder.add(host.get().createVKey());
}
} else {
Key<HostResource> hostKey =
VKey<HostResource> hostKey =
loadAndGetKey(
HostResource.class,
fqhn,
@@ -401,7 +404,10 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
if (desiredRegistrar.isPresent()) {
query = query.filter("currentSponsorClientId", desiredRegistrar.get());
}
return searchByNameserverRefs(query.keys());
return searchByNameserverRefs(
StreamSupport.stream(query.keys().spliterator(), false)
.map(key -> VKey.createOfy(HostResource.class, key))
.collect(toImmutableSet()));
}
/**
@@ -410,7 +416,7 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
* <p>This method is called by {@link #searchByNameserverLdhName} and {@link
* #searchByNameserverIp} after they assemble the relevant host keys.
*/
private DomainSearchResponse searchByNameserverRefs(final Iterable<Key<HostResource>> hostKeys) {
private DomainSearchResponse searchByNameserverRefs(final Iterable<VKey<HostResource>> hostKeys) {
// We must break the query up into chunks, because the in operator is limited to 30 subqueries.
// Since it is possible for the same domain to show up more than once in our result list (if
// we do a wildcard nameserver search that returns multiple nameservers used by the same
@@ -421,11 +427,13 @@ public class RdapDomainSearchAction extends RdapSearchActionBase {
ImmutableSortedSet.orderedBy(
Comparator.comparing(DomainBase::getFullyQualifiedDomainName));
int numHostKeysSearched = 0;
for (List<Key<HostResource>> chunk : Iterables.partition(hostKeys, 30)) {
for (List<VKey<HostResource>> chunk : Iterables.partition(hostKeys, 30)) {
numHostKeysSearched += chunk.size();
Query<DomainBase> query = ofy().load()
.type(DomainBase.class)
.filter("nsHosts in", chunk);
Query<DomainBase> query =
ofy()
.load()
.type(DomainBase.class)
.filter("nsHosts in", chunk.stream().map(VKey::getOfyKey).collect(toImmutableSet()));
if (!shouldIncludeDeleted()) {
query = query.filter("deletionTime >", getRequestTime());
// If we are not performing an inequality query, we can filter on the cursor in the query.
@@ -72,10 +72,9 @@ import javax.inject.Inject;
* (RDAP) Query Format</a>
* @see <a href="http://tools.ietf.org/html/rfc7483">RFC 7483: JSON Responses for the Registration
* Data Access Protocol (RDAP)</a>
*
* TODO(guyben):This isn't required by the RDAP Technical Implementation Guide, and hence should be
* deleted, at least until it's actually required.
*/
// TODO: This isn't required by the RDAP Technical Implementation Guide, and hence should be
// deleted, at least until it's actually required.
@Action(
service = Action.Service.PUBAPI,
path = "/rdap/entities",
@@ -433,10 +433,7 @@ public class RdapJsonFormatter {
statuses.add(StatusValue.LINKED);
}
if (hostResource.isSubordinate()
&& ofy()
.load()
.key(hostResource.getSuperordinateDomain())
.now()
&& tm().load(hostResource.getSuperordinateDomain())
.cloneProjectedAtTime(getRequestTime())
.getStatusValues()
.contains(StatusValue.PENDING_TRANSFER)) {
@@ -221,10 +221,9 @@ public abstract class RdapSearchActionBase extends RdapActionBase {
*
* <p>This version handles a list of parameter values, all associated with the same name.
*
* <p>Example: If the original parameters were "a=w&a=x&b=y&c=z", and this method is called with
* parameterName = "b" and parameterValues of "p" and "q", the result will be
* "a=w&a=x&c=z&b=p&b=q". The new values of parameter "b" replace the old ones.
*
* <p>Example: If the original parameters were "a=w&amp;a=x&amp;b=y&amp;c=z", and this method is
* called with parameterName = "b" and parameterValues of "p" and "q", the result will be
* "a=w&amp;a=x&amp;c=z&amp;b=p&amp;b=q". The new values of parameter "b" replace the old ones.
*/
protected String getRequestUrlWithExtraParameter(
String parameterName, List<String> parameterValues) {
@@ -120,9 +120,6 @@ public final class Ghostryde {
/**
* Creates a ghostryde file from an in-memory byte array.
*
* @throws PGPException
* @throws IOException
*/
public static byte[] encode(byte[] data, PGPPublicKey key)
throws IOException, PGPException {
@@ -137,9 +134,6 @@ public final class Ghostryde {
/**
* Deciphers a ghostryde file from an in-memory byte array.
*
* @throws PGPException
* @throws IOException
*/
public static byte[] decode(byte[] data, PGPPrivateKey key)
throws IOException, PGPException {
@@ -17,7 +17,6 @@ package google.registry.rde;
import static com.google.common.base.Preconditions.checkArgument;
import com.google.common.net.InetAddresses;
import com.googlecode.objectify.Key;
import google.registry.model.domain.DomainBase;
import google.registry.model.eppcommon.StatusValue;
import google.registry.model.host.HostResource;
@@ -35,9 +34,8 @@ import org.joda.time.DateTime;
final class HostResourceToXjcConverter {
/** Converts a subordinate {@link HostResource} to {@link XjcRdeHostElement}. */
static XjcRdeHostElement convertSubordinate(
HostResource host, DomainBase superordinateDomain) {
checkArgument(Key.create(superordinateDomain).equals(host.getSuperordinateDomain()));
static XjcRdeHostElement convertSubordinate(HostResource host, DomainBase superordinateDomain) {
checkArgument(superordinateDomain.createVKey().equals(host.getSuperordinateDomain()));
return new XjcRdeHostElement(convertSubordinateHost(host, superordinateDomain));
}
@@ -18,6 +18,7 @@ import static com.google.common.base.Strings.nullToEmpty;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.model.EppResourceUtils.loadAtPointInTime;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.appengine.tools.mapreduce.Mapper;
import com.google.auto.value.AutoValue;
@@ -186,13 +187,16 @@ public final class RdeStagingMapper extends Mapper<EppResource, PendingDeposit,
return result;
} else if (resource instanceof HostResource) {
HostResource host = (HostResource) resource;
result = Optional.of(host.isSubordinate()
? marshaller.marshalSubordinateHost(
host,
// Note that loadAtPointInTime() does cloneProjectedAtTime(watermark) for us.
loadAtPointInTime(
ofy().load().key(host.getSuperordinateDomain()).now(), watermark).now())
: marshaller.marshalExternalHost(host));
result =
Optional.of(
host.isSubordinate()
? marshaller.marshalSubordinateHost(
host,
// Note that loadAtPointInTime() does cloneProjectedAtTime(watermark) for
// us.
loadAtPointInTime(tm().load(host.getSuperordinateDomain()), watermark)
.now())
: marshaller.marshalExternalHost(host));
cache.put(WatermarkModePair.create(watermark, RdeMode.FULL), result);
cache.put(WatermarkModePair.create(watermark, RdeMode.THIN), result);
return result;
@@ -46,9 +46,6 @@ public final class RdeUtil {
/**
* Look at some bytes from {@code xmlInput} to ensure it appears to be a FULL XML deposit and
* then use a regular expression to extract the watermark timestamp which is returned.
*
* @throws IOException
* @throws XmlException
*/
public static DateTime peekWatermark(BufferedInputStream xmlInput)
throws IOException, XmlException {
@@ -34,7 +34,7 @@ import javax.mail.internet.InternetAddress;
import org.joda.time.YearMonth;
/** Utility functions for sending emails involving monthly invoices. */
class BillingEmailUtils {
public class BillingEmailUtils {
private final SendEmailService emailService;
private final YearMonth yearMonth;
@@ -43,8 +43,8 @@ import org.joda.time.YearMonth;
* PublishInvoicesAction} to publish the subsequent output.
*
* <p>This action runs the {@link google.registry.beam.invoicing.InvoicingPipeline} beam template,
* staged at gs://<projectId>-beam/templates/invoicing. The pipeline then generates invoices for the
* month and stores them on GCS.
* staged at gs://&lt;projectId&gt;-beam/templates/invoicing. The pipeline then generates invoices
* for the month and stores them on GCS.
*/
@Action(
service = Action.Service.BACKEND,
@@ -53,7 +53,7 @@ import org.json.JSONException;
* pipeline accordingly.
*
* <p>This calls {@link Spec11EmailUtils#emailSpec11Reports(LocalDate, SoyTemplateInfo, String,
* Set)} on success or {@link Spec11EmailUtils#sendAlertEmail(String, String)} on failure.
* ImmutableSet)} on success or {@link Spec11EmailUtils#sendAlertEmail(String, String)} on failure.
*/
@Action(
service = Action.Service.BACKEND,
@@ -45,9 +45,9 @@ public final class RequestParameters {
*
* <ul>
* <li>/foo?bar=hello hello
* <li>/foo?bar=hello&bar=there hello
* <li>/foo?bar=hello&amp;bar=there hello
* <li>/foo?bar= 400 error (empty)
* <li>/foo?bar=&bar=there 400 error (empty)
* <li>/foo?bar=&amp;bar=there 400 error (empty)
* <li>/foo 400 error (absent)
* </ul>
*
@@ -59,7 +59,7 @@ public class CursorDao {
.transact(() -> jpaTm().getEntityManager().find(Cursor.class, new CursorId(type, scope)));
}
/** If no scope is given, use {@link Cursor.GLOBAL} as the scope. */
/** If no scope is given, use {@link Cursor#GLOBAL} as the scope. */
public static Cursor load(CursorType type) {
checkNotNull(type, "The type of the cursor to load must be specified");
return load(type, Cursor.GLOBAL);
@@ -214,13 +214,13 @@ public final class RegistryLock extends ImmutableObject implements Buildable, Sq
return lockCompletionTimestamp != null && unlockCompletionTimestamp == null;
}
/** Returns true iff the lock was requested >= 1 hour ago and has not been verified. */
/** Returns true iff the lock was requested &gt;= 1 hour ago and has not been verified. */
public boolean isLockRequestExpired(DateTime now) {
return !getLockCompletionTimestamp().isPresent()
&& isBeforeOrAt(getLockRequestTimestamp(), now.minusHours(1));
}
/** Returns true iff the unlock was requested >= 1 hour ago and has not been verified. */
/** Returns true iff the unlock was requested &gt;= 1 hour ago and has not been verified. */
public boolean isUnlockRequestExpired(DateTime now) {
Optional<DateTime> unlockRequestTimestamp = getUnlockRequestTimestamp();
return unlockRequestTimestamp.isPresent()
@@ -42,7 +42,7 @@ public class DriveConnection {
/**
* Creates a folder with the given parent.
*
* @returns the folder id.
* @return the folder id.
*/
public String createFolder(String title, String parentFolderId) throws IOException {
return drive.files()
@@ -58,7 +58,7 @@ public class DriveConnection {
* existing file is the desired behavior, use {@link #createOrUpdateFile(String, MediaType,
* String, byte[])} instead.
*
* @returns the file id.
* @return the file id.
*/
public String createFile(String title, MediaType mimeType, String parentFolderId, byte[] bytes)
throws IOException {
@@ -76,13 +76,10 @@ public class DriveConnection {
*
* @throws IllegalStateException if multiple files with that name exist in the given folder.
* @throws IOException if communication with Google Drive fails for any reason.
* @returns the file id.
* @return the file id.
*/
public String createOrUpdateFile(
String title,
MediaType mimeType,
String parentFolderId,
byte[] bytes) throws IOException {
String title, MediaType mimeType, String parentFolderId, byte[] bytes) throws IOException {
List<String> existingFiles = listFiles(parentFolderId, String.format("title = '%s'", title));
if (existingFiles.size() > 1) {
throw new IllegalStateException(String.format(
@@ -97,10 +94,10 @@ public class DriveConnection {
}
/**
* Updates the file with the given id in place, setting the title, content, and mime type to
* the newly specified values.
* Updates the file with the given id in place, setting the title, content, and mime type to the
* newly specified values.
*
* @returns the file id.
* @return the file id.
*/
public String updateFile(String fileId, String title, MediaType mimeType, byte[] bytes)
throws IOException {
@@ -77,10 +77,6 @@ public class TmchXmlSignature {
*
* @throws GeneralSecurityException for unsupported protocols, certs not signed by the TMCH,
* incorrect keys, and for invalid, old, not-yet-valid or revoked certificates.
* @throws IOException
* @throws MarshalException
* @throws ParserConfigurationException
* @throws SAXException
*/
public void verify(byte[] smdXml)
throws GeneralSecurityException, IOException, MarshalException, ParserConfigurationException,
@@ -18,12 +18,12 @@ import static com.google.common.base.Preconditions.checkState;
import com.google.common.base.Ascii;
import com.google.common.base.Strings;
import com.googlecode.objectify.Key;
import google.registry.model.EppResource;
import google.registry.model.contact.ContactResource;
import google.registry.model.domain.DomainBase;
import google.registry.model.host.HostResource;
import google.registry.model.index.ForeignKeyIndex;
import google.registry.persistence.VKey;
import org.joda.time.DateTime;
/** Container class for static utility methods. */
@@ -41,7 +41,7 @@ class CommandUtilities {
this.clazz = clazz;
}
public Key<? extends EppResource> getKey(String uniqueId, DateTime now) {
public VKey<? extends EppResource> getKey(String uniqueId, DateTime now) {
return ForeignKeyIndex.loadAndGetKey(clazz, uniqueId, now);
}
}
@@ -39,18 +39,14 @@ class CompareDbBackups {
return;
}
ImmutableSet<ComparableEntity> entities1 =
new RecordAccumulator()
.readDirectory(new File(args[0]), DATA_FILE_MATCHER)
.getComparableEntitySet();
ImmutableSet<ComparableEntity> entities2 =
new RecordAccumulator()
.readDirectory(new File(args[1]), DATA_FILE_MATCHER)
.getComparableEntitySet();
ImmutableSet<EntityWrapper> entities1 =
RecordAccumulator.readDirectory(new File(args[0]), DATA_FILE_MATCHER).getEntityWrapperSet();
ImmutableSet<EntityWrapper> entities2 =
RecordAccumulator.readDirectory(new File(args[1]), DATA_FILE_MATCHER).getEntityWrapperSet();
// Calculate the entities added and removed.
SetView<ComparableEntity> added = Sets.difference(entities2, entities1);
SetView<ComparableEntity> removed = Sets.difference(entities1, entities2);
SetView<EntityWrapper> added = Sets.difference(entities2, entities1);
SetView<EntityWrapper> removed = Sets.difference(entities1, entities2);
printHeader(
String.format("First backup: %d records", entities1.size()),
@@ -58,14 +54,14 @@ class CompareDbBackups {
if (!removed.isEmpty()) {
printHeader(removed.size() + " records were removed:");
for (ComparableEntity entity : removed) {
for (EntityWrapper entity : removed) {
System.out.println(entity);
}
}
if (!added.isEmpty()) {
printHeader(added.size() + " records were added:");
for (ComparableEntity entity : added) {
for (EntityWrapper entity : added) {
System.out.println(entity);
}
}
@@ -15,20 +15,32 @@
package google.registry.tools;
import com.google.appengine.api.datastore.Entity;
import com.google.auto.value.AutoValue;
import com.google.common.base.Objects;
/** Wraps {@link Entity} to do hashCode/equals based on both the entity's key and its properties. */
final class ComparableEntity {
/**
* Wraps {@link Entity} for ease of processing in collections.
*
* <p>Note that the {@link #hashCode}/{@link #equals} methods are based on both the entity's key and
* its properties.
*/
final class EntityWrapper {
private static final String TEST_ENTITY_KIND = "TestEntity";
private final Entity entity;
ComparableEntity(Entity entity) {
EntityWrapper(Entity entity) {
this.entity = entity;
}
public Entity getEntity() {
return entity;
}
@Override
public boolean equals(Object that) {
if (that instanceof ComparableEntity) {
ComparableEntity thatEntity = (ComparableEntity) that;
if (that instanceof EntityWrapper) {
EntityWrapper thatEntity = (EntityWrapper) that;
return entity.equals(thatEntity.entity)
&& entity.getProperties().equals(thatEntity.entity.getProperties());
}
@@ -43,6 +55,26 @@ final class ComparableEntity {
@Override
public String toString() {
return "ComparableEntity(" + entity + ")";
return "EntityWrapper(" + entity + ")";
}
public static EntityWrapper from(int id, Property... properties) {
Entity entity = new Entity(TEST_ENTITY_KIND, id);
for (Property prop : properties) {
entity.setProperty(prop.name(), prop.value());
}
return new EntityWrapper(entity);
}
@AutoValue
abstract static class Property {
static Property create(String name, Object value) {
return new AutoValue_EntityWrapper_Property(name, value);
}
abstract String name();
abstract Object value();
}
}
@@ -23,9 +23,9 @@ import static org.joda.time.DateTimeZone.UTC;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.googlecode.objectify.Key;
import google.registry.model.EppResource;
import google.registry.model.reporting.HistoryEntry;
import google.registry.persistence.VKey;
import google.registry.tools.CommandUtilities.ResourceType;
import google.registry.xml.XmlTransformer;
import org.joda.time.DateTime;
@@ -57,7 +57,7 @@ final class GetHistoryEntriesCommand implements CommandWithRemoteApi {
@Override
public void run() {
Key<? extends EppResource> parentKey = null;
VKey<? extends EppResource> parentKey = null;
if (type != null || uniqueId != null) {
checkArgument(
type != null && uniqueId != null,
@@ -66,12 +66,12 @@ final class GetHistoryEntriesCommand implements CommandWithRemoteApi {
checkArgumentNotNull(parentKey, "Invalid resource ID");
}
for (HistoryEntry entry :
(parentKey == null
(parentKey == null
? ofy().load().type(HistoryEntry.class)
: ofy().load().type(HistoryEntry.class).ancestor(parentKey))
.order("modificationTime")
.filter("modificationTime >=", after)
.filter("modificationTime <=", before)) {
: ofy().load().type(HistoryEntry.class).ancestor(parentKey.getOfyKey()))
.order("modificationTime")
.filter("modificationTime >=", after)
.filter("modificationTime <=", before)) {
System.out.printf(
"Client: %s\nTime: %s\nClient TRID: %s\nServer TRID: %s\n%s\n",
entry.getClientId(),
@@ -14,37 +14,101 @@
package google.registry.tools;
import static com.google.common.base.Preconditions.checkState;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.Optional;
/**
* Reads records from a set of LevelDB files and builds a gigantic ImmutableList from them.
* Iterator that incrementally parses binary data in LevelDb format into records.
*
* <p>See <a
* href="https://github.com/google/leveldb/blob/master/doc/log_format.md">log_format.md</a> for the
* leveldb log format specification.</a>
* <p>The input source is automatically closed when all data have been read.
*
* <p>There are several other implementations of this, none of which appeared suitable for our use
* case: <a href="https://github.com/google/leveldb">The original C++ implementation</a>. <a
* href="https://cloud.google.com/appengine/docs/standard/java/javadoc/com/google/appengine/api/files/RecordWriteChannel">
* com.google.appengine.api.files.RecordWriteChannel</a> - Exactly what we need but deprecated. The
* href="https://cloud.google.com/appengine/docs/standard/java/javadoc/com/google/appengine/api/files/RecordReadChannel">
* com.google.appengine.api.files.RecordReadChannel</a> - Exactly what we need but deprecated. The
* referenced replacement: <a
* href="https://github.com/GoogleCloudPlatform/appengine-gcs-client.git">The App Engine GCS
* Client</a> - Does not appear to have any support for working with LevelDB.
* Client</a> - Does not appear to have any support for working with LevelDB. *
*
* <p>See <a
* href="https://github.com/google/leveldb/blob/master/doc/log_format.md">log_format.md</a>
*/
public final class LevelDbLogReader {
public final class LevelDbLogReader implements Iterator<byte[]> {
@VisibleForTesting static final int BLOCK_SIZE = 32 * 1024;
@VisibleForTesting static final int HEADER_SIZE = 7;
private final ByteArrayOutputStream recordContents = new ByteArrayOutputStream();
private final ImmutableList.Builder<byte[]> recordListBuilder = new ImmutableList.Builder<>();
private final LinkedList<byte[]> recordList = Lists.newLinkedList();
private final ByteBuffer byteBuffer = ByteBuffer.allocate(BLOCK_SIZE);
private final ReadableByteChannel channel;
LevelDbLogReader(ReadableByteChannel channel) {
this.channel = channel;
}
@Override
public boolean hasNext() {
while (recordList.isEmpty()) {
try {
Optional<byte[]> block = readFromChannel();
if (!block.isPresent()) {
return false;
}
if (block.get().length != BLOCK_SIZE) {
throw new IllegalStateException("Data size is not multiple of " + BLOCK_SIZE);
}
processBlock(block.get());
} catch (IOException e) {
throw new RuntimeException(e);
}
}
return true;
}
@Override
public byte[] next() {
checkState(hasNext(), "The next() method called on empty iterator.");
return recordList.removeFirst();
}
/**
* Returns the next {@link #BLOCK_SIZE} bytes from the input channel, or {@link
* Optional#empty()} if there is no more data.
*/
// TODO(weiminyu): use ByteBuffer directly.
private Optional<byte[]> readFromChannel() throws IOException {
while (channel.isOpen()) {
int bytesRead = channel.read(byteBuffer);
if (!byteBuffer.hasRemaining() || bytesRead < 0) {
byteBuffer.flip();
if (!byteBuffer.hasRemaining()) {
channel.close();
return Optional.empty();
}
byte[] result = new byte[byteBuffer.remaining()];
byteBuffer.get(result);
byteBuffer.clear();
return Optional.of(result);
}
}
return Optional.empty();
}
/** Read a complete block, which must be exactly 32 KB. */
private void processBlock(byte[] block) {
@@ -63,7 +127,7 @@ public final class LevelDbLogReader {
// If this is the last (or only) chunk in the record, store the full contents into the List.
if (recordHeader.type == ChunkType.FULL || recordHeader.type == ChunkType.LAST) {
recordListBuilder.add(recordContents.toByteArray());
recordList.add(recordContents.toByteArray());
recordContents.reset();
}
@@ -96,40 +160,24 @@ public final class LevelDbLogReader {
return new RecordHeader(checksum, size, ChunkType.fromCode(type));
}
/** Reads all records in the Reader into the record set. */
public void readFrom(InputStream source) throws IOException {
byte[] block = new byte[BLOCK_SIZE];
// read until we have no more.
while (true) {
int amountRead = source.read(block, 0, BLOCK_SIZE);
if (amountRead <= 0) {
break;
}
assert amountRead == BLOCK_SIZE;
processBlock(block);
}
/** Returns a {@link LevelDbLogReader} over a {@link ReadableByteChannel}. */
public static LevelDbLogReader from(ReadableByteChannel channel) {
return new LevelDbLogReader(channel);
}
/** Reads all records from the file specified by "path" into the record set. */
public void readFrom(Path path) throws IOException {
readFrom(Files.newInputStream(path));
/** Returns a {@link LevelDbLogReader} over an {@link InputStream}. */
public static LevelDbLogReader from(InputStream source) {
return new LevelDbLogReader(Channels.newChannel(source));
}
/** Reads all records from the specified file into the record set. */
public void readFrom(String filename) throws IOException {
readFrom(FileSystems.getDefault().getPath(filename));
/** Returns a {@link LevelDbLogReader} over a file specified by {@link Path}. */
public static LevelDbLogReader from(Path path) throws IOException {
return from(Files.newInputStream(path));
}
/**
* Gets the list of records constructed so far.
*
* <p>Note that this does not invalidate the internal state of the object: we return a copy and
* this can be called multiple times.
*/
ImmutableList<byte[]> getRecords() {
return recordListBuilder.build();
/** Returns a {@link LevelDbLogReader} over a file specified by {@code filename}. */
public static LevelDbLogReader from(String filename) throws IOException {
return from(FileSystems.getDefault().getPath(filename));
}
/** Aggregates the fields in a record header. */
@@ -15,42 +15,47 @@
package google.registry.tools;
import com.google.appengine.api.datastore.EntityTranslator;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.storage.onestore.v3.OnestoreEntity.EntityProto;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.function.Predicate;
/** Accumulates Entity records from level db files under a directory hierarchy. */
class RecordAccumulator {
private final LevelDbLogReader reader = new LevelDbLogReader();
private final ImmutableList<byte[]> records;
RecordAccumulator(ImmutableList<byte[]> records) {
this.records = records;
}
/** Recursively reads all records in the directory. */
public final RecordAccumulator readDirectory(File dir, Predicate<File> fileMatcher) {
public static RecordAccumulator readDirectory(File dir, Predicate<File> fileMatcher) {
ImmutableList.Builder<byte[]> builder = new ImmutableList.Builder<>();
for (File child : dir.listFiles()) {
if (child.isDirectory()) {
readDirectory(child, fileMatcher);
builder.addAll(readDirectory(child, fileMatcher).records);
} else if (fileMatcher.test(child)) {
try {
reader.readFrom(new FileInputStream(child));
builder.addAll(LevelDbLogReader.from(child.getPath()));
} catch (IOException e) {
throw new RuntimeException("IOException reading from file: " + child, e);
}
}
}
return this;
return new RecordAccumulator(builder.build());
}
/** Creates an entity set from the current set of raw records. */
ImmutableSet<ComparableEntity> getComparableEntitySet() {
ImmutableSet.Builder<ComparableEntity> builder = new ImmutableSet.Builder<>();
for (byte[] rawRecord : reader.getRecords()) {
/** Creates an {@link EntityWrapper} set from the current set of raw records. */
ImmutableSet<EntityWrapper> getEntityWrapperSet() {
ImmutableSet.Builder<EntityWrapper> builder = new ImmutableSet.Builder<>();
for (byte[] rawRecord : records) {
// Parse the entity proto and create an Entity object from it.
EntityProto proto = new EntityProto();
proto.parseFrom(rawRecord);
ComparableEntity entity = new ComparableEntity(EntityTranslator.createFromPb(proto));
EntityWrapper entity = new EntityWrapper(EntityTranslator.createFromPb(proto));
builder.add(entity);
}
@@ -14,14 +14,14 @@
package google.registry.tools;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import static org.joda.time.DateTimeZone.UTC;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import com.googlecode.objectify.Key;
import google.registry.model.EppResource;
import google.registry.persistence.VKey;
import google.registry.tools.CommandUtilities.ResourceType;
import org.joda.time.DateTime;
@@ -48,12 +48,15 @@ public final class ResaveEppResourceCommand extends MutatingCommand {
@Override
protected void init() {
Key<? extends EppResource> resourceKey = checkArgumentNotNull(
type.getKey(uniqueId, DateTime.now(UTC)),
"Could not find active resource of type %s: %s", type, uniqueId);
VKey<? extends EppResource> resourceKey =
checkArgumentNotNull(
type.getKey(uniqueId, DateTime.now(UTC)),
"Could not find active resource of type %s: %s",
type,
uniqueId);
// Load the resource directly to bypass running cloneProjectedAtTime() automatically, which can
// cause stageEntityChange() to fail due to implicit projection changes.
EppResource resource = ofy().load().key(resourceKey).now();
EppResource resource = tm().load(resourceKey);
stageEntityChange(resource, resource);
}
}
@@ -44,43 +44,45 @@ import javax.annotation.concurrent.Immutable;
* Declarative functional fluent form field converter / validator.
*
* <p>This class is responsible for converting arbitrary data, sent to us by the web browser, into
* validated data structures that the server-side code can use. For example:<pre>
* validated data structures that the server-side code can use. For example:
*
* private enum Gender { MALE, FEMALE }
* <pre>{@code
* private enum Gender { MALE, FEMALE }
*
* private static final FormField<String, String> NAME_FIELD = FormField.named("name")
* .matches("[a-z]+")
* .range(atMost(16))
* .required()
* .build();
* private static final FormField<String, String> NAME_FIELD = FormField.named("name")
* .matches("[a-z]+")
* .range(atMost(16))
* .required()
* .build();
*
* private static final FormField<String, Gender> GENDER_FIELD = FormField.named("gender")
* .asEnum(Gender.class)
* .required()
* .build();
* private static final FormField<String, Gender> GENDER_FIELD = FormField.named("gender")
* .asEnum(Gender.class)
* .required()
* .build();
*
* public Person makePerson(Map<String, String> params) {
* Person.Builder person = new Person.Builder();
* for (String name : NAME_FIELD.extract(params).asSet()) {
* person.setName(name);
* }
* for (Gender name : GENDER_FIELD.extract(params).asSet()) {
* person.setGender(name);
* }
* return person.build();
* }</pre>
* public Person makePerson(Map<String, String> params) {
* Person.Builder person = new Person.Builder();
* for (String name : NAME_FIELD.extract(params).asSet()) {
* person.setName(name);
* }
* for (Gender name : GENDER_FIELD.extract(params).asSet()) {
* person.setGender(name);
* }
* return person.build();
* }
* }</pre>
*
* <p>This class provides <b>full type-safety</b> <i>if and only if</i> you statically initialize
* your FormField objects and write a unit test that causes the class to be loaded.
*
* <h3>Exception Handling</h3>
*
* <p>When values passed to {@link #convert} or {@link #extract} don't meet the contract,
* {@link FormFieldException} will be thrown, which provides the field name and a short error
* message that's safe to pass along to the client.
* <p>When values passed to {@link #convert} or {@link #extract} don't meet the contract, {@link
* FormFieldException} will be thrown, which provides the field name and a short error message
* that's safe to pass along to the client.
*
* <p>You can safely throw {@code FormFieldException} from within your validator functions, and
* the field name will automatically be propagated into the exception object for you.
* <p>You can safely throw {@code FormFieldException} from within your validator functions, and the
* field name will automatically be propagated into the exception object for you.
*
* <p>In situations when you're validating lists or maps, you'll end up with a hierarchical field
* naming structure. For example, if you were validating a list of maps, an error generated by the
@@ -92,23 +94,25 @@ import javax.annotation.concurrent.Immutable;
* <p>You should never assign a partially constructed {@code FormField.Builder} to a variable or
* constant. Instead, you should use {@link #asBuilder()} or {@link #asBuilderNamed(String)}.
*
* <p>Here is an example of how you might go about defining library definitions:<pre>
* <p>Here is an example of how you might go about defining library definitions:
*
* final class FormFields {
* private static final FormField<String, String> COUNTRY_CODE =
* FormField.named("countryCode")
* .range(Range.singleton(2))
* .uppercased()
* .in(ImmutableSet.copyOf(Locale.getISOCountries()))
* .build();
* }
* <pre>{@code
* final class FormFields {
* private static final FormField<String, String> COUNTRY_CODE =
* FormField.named("countryCode")
* .range(Range.singleton(2))
* .uppercased()
* .in(ImmutableSet.copyOf(Locale.getISOCountries()))
* .build();
* }
*
* final class Form {
* private static final FormField<String, String> COUNTRY_CODE_FIELD =
* FormFields.COUNTRY_CODE.asBuilder()
* .required()
* .build();
* }</pre>
* final class Form {
* private static final FormField<String, String> COUNTRY_CODE_FIELD =
* FormFields.COUNTRY_CODE.asBuilder()
* .required()
* .build();
* }
* }</pre>
*
* @param <I> input value type
* @param <O> output value type
@@ -105,9 +105,10 @@ public class SendEmailUtils {
}
}
/** Sends an email from Nomulus to the registrarChangesNotificationEmailAddresses.
/**
* Sends an email from Nomulus to the registrarChangesNotificationEmailAddresses.
*
* <p>See {@link #sendEmail(String, String, ImmutableList<String>)}.
* <p>See {@link #sendEmail(String, String, ImmutableList)}.
*/
public boolean sendEmail(final String subject, String body) {
return sendEmail(subject, body, ImmutableList.of());
@@ -156,7 +156,7 @@ final class DomainWhoisResponse extends WhoisResponseImpl {
// If we refer to a contact that doesn't exist, that's a bug. It means referential integrity
// has somehow been broken. We skip the rest of this contact, but log it to hopefully bring it
// someone's attention.
ContactResource contactResource = EppResource.loadCached(contact.get().getOfyKey());
ContactResource contactResource = EppResource.loadCached(contact.get());
if (contactResource == null) {
logger.atSevere().log(
"(BUG) Broken reference found from domain %s to contact %s",
@@ -16,7 +16,7 @@ package google.registry.whois;
import static com.google.common.base.Preconditions.checkNotNull;
import static com.google.common.base.Preconditions.checkState;
import static google.registry.model.ofy.ObjectifyService.ofy;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import com.google.common.collect.ImmutableList;
import com.google.common.net.InetAddresses;
@@ -49,7 +49,7 @@ final class NameserverWhoisResponse extends WhoisResponseImpl {
HostResource host = hosts.get(i);
String clientId =
host.isSubordinate()
? ofy().load().key(host.getSuperordinateDomain()).now()
? tm().load(host.getSuperordinateDomain())
.cloneProjectedAtTime(getTimestamp())
.getCurrentSponsorClientId()
: host.getPersistedCurrentSponsorClientId();
@@ -58,40 +58,38 @@ import org.joda.time.Duration;
* <p>You can send AJAX requests to our WHOIS API from your <em>very own</em> website using the
* following embed code:
*
* <pre>
* <p>
* <input id="query-input" placeholder="Domain, Nameserver, IP, etc." autofocus>
* <button id="search-button">Lookup</button>
* <p>
* <pre id="whois-results"></pre>
* <script>
* (function() {
* var WHOIS_API_URL = 'https://domain-registry-alpha.appspot.com/whois/';
* function OnKeyPressQueryInput(ev) {
* if (typeof ev == 'undefined' && window.event) {
* ev = window.event;
* }
* if (ev.keyCode == 13) {
* document.getElementById('search-button').click();
* }
* <pre>{@code
* <input id="query-input" placeholder="Domain, Nameserver, IP, etc." autofocus>
* <button id="search-button">Lookup</button>
* <pre id="whois-results"></pre>
* <script>
* (function() {
* var WHOIS_API_URL = 'https://domain-registry-alpha.appspot.com/whois/';
* function OnKeyPressQueryInput(ev) {
* if (typeof ev == 'undefined' && window.event) {
* ev = window.event;
* }
* function OnClickSearchButton() {
* var query = document.getElementById('query-input').value;
* var req = new XMLHttpRequest();
* req.onreadystatechange = function() {
* if (req.readyState == 4) {
* var results = document.getElementById('whois-results');
* results.textContent = req.responseText;
* }
* };
* req.open('GET', WHOIS_API_URL + escape(query), true);
* req.send();
* if (ev.keyCode == 13) {
* document.getElementById('search-button').click();
* }
* document.getElementById('search-button').onclick = OnClickSearchButton;
* document.getElementById('query-input').onkeypress = OnKeyPressQueryInput;
* })();
* </script>
* </pre>
* }
* function OnClickSearchButton() {
* var query = document.getElementById('query-input').value;
* var req = new XMLHttpRequest();
* req.onreadystatechange = function() {
* if (req.readyState == 4) {
* var results = document.getElementById('whois-results');
* results.textContent = req.responseText;
* }
* };
* req.open('GET', WHOIS_API_URL + escape(query), true);
* req.send();
* }
* document.getElementById('search-button').onclick = OnClickSearchButton;
* document.getElementById('query-input').onkeypress = OnKeyPressQueryInput;
* })();
* </script>
* }</pre>
*
* @see WhoisAction
*/
@@ -19,6 +19,9 @@
* Move tests to another (sub)project. This is not a big problem, but feels unnatural.
* Use Hibernate's ServiceRegistry for bootstrapping (not JPA-compliant)
-->
<class>google.registry.model.billing.BillingEvent$Cancellation</class>
<class>google.registry.model.billing.BillingEvent$OneTime</class>
<class>google.registry.model.billing.BillingEvent$Recurring</class>
<class>google.registry.model.contact.ContactResource</class>
<class>google.registry.model.domain.DomainBase</class>
<class>google.registry.model.host.HostResource</class>
@@ -33,9 +36,13 @@
<class>google.registry.schema.tld.ReservedList</class>
<class>google.registry.model.domain.secdns.DelegationSignerData</class>
<class>google.registry.model.domain.GracePeriod</class>
<class>google.registry.model.poll.PollMessage</class>
<class>google.registry.model.poll.PollMessage$OneTime</class>
<class>google.registry.model.poll.PollMessage$Autorenew</class>
<!-- Customized type converters -->
<class>google.registry.persistence.converter.BillingCostTransitionConverter</class>
<class>google.registry.persistence.converter.BillingEventFlagSetConverter</class>
<class>google.registry.persistence.converter.BloomFilterConverter</class>
<class>google.registry.persistence.converter.CidrAddressBlockListConverter</class>
<class>google.registry.persistence.converter.CreateAutoTimestampConverter</class>
@@ -53,6 +60,9 @@
<class>google.registry.persistence.converter.ZonedDateTimeConverter</class>
<!-- Generated converters for VKey -->
<class>google.registry.model.billing.VKeyConverter_BillingEvent</class>
<class>google.registry.model.domain.VKeyConverter_DomainBase</class>
<class>google.registry.model.domain.token.VKeyConverter_AllocationToken</class>
<class>google.registry.model.host.VKeyConverter_HostResource</class>
<class>google.registry.model.contact.VKeyConverter_ContactResource</class>
@@ -55,7 +55,7 @@
{template .resultSuccess}
{@param baseClientId: string} /** The base clientId used for the OT&amp;E setup. */
{@param contactEmail: string} /** The contact's email added to the registrars. */
{@param clientIdToTld: map<string, string>} /** The created registrars->TLD mapping. */
{@param clientIdToTld: map<string, string>} /** The created registrars-&gt;TLD mapping. */
{@param password: string} /** The password given for the created registrars. */
{@param username: string} /** Arbitrary username to display. */
{@param logoutUrl: string} /** Generated URL for logging out of Google. */

Some files were not shown because too many files have changed in this diff Show More