Compare commits

...
Author SHA1 Message Date
Weimin YuandGitHub e42c11051e Download scheduler for BSA (#2209)
* Add BSA download scheduler
2023-11-17 16:15:14 -05:00
Weimin YuandGitHub 85b588b51f Add a disposition header to email attachments (#2223)
This may help with the billing-team with attached invoices.

This is a standard header that should do no harm.
2023-11-16 13:31:12 -05:00
Pavlo TkachandGitHub 572b7101cb Create separate BSA service (#2221) 2023-11-15 18:38:26 -05:00
Weimin YuandGitHub 445825957d Bsa Persistence entity classes (#2205)
* Add persistence model object
2023-11-15 16:43:22 -05:00
Weimin YuandGitHub 7ab76f3573 Pin Flyway tool jar to 9.22.3 (#2222)
Flyway 10+ is not compatible with Java 8.

Rollback this change after we upgrade to Java 11.
2023-11-15 14:48:55 -05:00
Weimin YuandGitHub 9e3c58989a Add an IDN helper (#2217)
* Add an IDN helper

Add a helper that checks the validity of labels in IDNs.
All organizes TLDs according to the IDNs they support.
2023-11-10 19:55:04 -05:00
Lai JiangandGitHub cf9c1ec7c3 Use Java 8 runtime on sandbox and production (#2218)
Java 17 injects unexpected headers to X-Forwarded-For, which causes
issues with validating incoming IP addresses.

This is a partial reversion of #2201. We are still keeping Java 17 in other environment but sandbox and production needs to be able to parse the header to accept incoming EPP connections from registrars. Once we fix it we will re-enable Java 17 in these environment.
2023-11-10 14:39:16 -05:00
Pavlo TkachandGitHub 69ea87be31 Add handler for Console API requests and XSRF token creation and verification (#2211) 2023-11-09 17:51:53 -05:00
Lai JiangandGitHub 779d0c9d37 Add a fallback token verifier (#2216)
This allows us to switch the proxy to a different client ID without
disrupting the service. This is a temporary measure and will be removed
once the switch is complete.
2023-11-09 16:05:14 -05:00
Weimin YuandGitHub 2855944214 Add TLD BSA enroll start date to schema (#2215)
Also adds a placeholder getter in the Tld class, so that it can be
mocked/spied in tests. This way more BSA related code can be submitted
before the schema is deployed to prod.
2023-11-09 13:52:45 -05:00
Ben McIlwainandGitHub 992d1c1349 Reduce the QPS of the refresh DNS for all domains action (#2212)
This also adds a targeted QPS as a parameter in case we need to manually bump it
up (or down) for some reason without having to make code changes and re-deploy.
2023-11-08 13:47:37 -05:00
Pavlo TkachandGitHub f50290ce1d Add static IP connector to crash and alpha configs (#2213) 2023-11-08 13:26:32 -05:00
Pavlo TkachandGitHub e647d4e215 Add retry to cloud build node installation (#2210) 2023-11-06 09:15:36 -05:00
Lai JiangandGitHub 08471242df Refactor transact() related methods. (#2195)
This PR makes a few changes to make it possible to turn on
per-transaction isolation level with minimal disruption:

1) Changed the signatures of transact() and reTransact() methods to allow
passing in lambdas that throw checked exceptions. Previously one has
always to wrap such lambdas in try-and-retrow blocks, which wasn't a
big issue when one can liberally open nested transactions around small
lambdas and keeps the "throwing" part outside the lambda. This becomes a
much bigger hassle when the goal is to eliminate nested transactions and
put as much code as possible within the top-level lambda. As a result,
the transactNoRetry() method now handles checked exceptions by re-throwing
them as runtime exceptions.

2) Changed the name and meaning of the config file field that used to
indicate if per-transaction isolation level is enabled or not. Now it
decides if transact() is called within a transaction, whether to
throw or to log, regardless whether the transaction could have
succeeded based on the isolation override level (if provided). The
flag will initially be set to false and would help us identify all
instances of nested calls and either refactor them or use reTransact()
instead. Once we are fairly certain that no nested calls to transact()
exists, we flip the flag to true and start enforcing this logic.
Eventually the flag will go away and nested calls to transact() will
always throw.

3) Per-transaction isolation level will now always be applied, if an
override is provided. Because currently there should be no actual
use of such feature (except for places where we explicitly use an
override and have ensured no nested transactions exist, like in
RefreshDnsForAllDomainsAction), we do not expect any issues with
conflicting isolation levels, which would resulted in failure.

3) transactNoRetry() is made package private and removed from the
exposed API of JpaTransactionManager. This saves a lot of redundant
methods that do not have a practical use. The only instances where this
method was called outside the package was in the reader of
RegistryJpaIO, which should have no problem with retrying.
2023-11-03 17:43:27 -04:00
Lai JiangandGitHub cd23fea698 Switch to a stronger algorithm for password hashing (#2191)
We have been using SHA256 to hash passwords (for both EPP and registry lock),
which is now considered too weak.

This PR switches to using Scrypt, a memory-hard slow hash function, with
recommended parameters per go/crypto-password-hash.

To ease the transition, when a password is being verified, both Scrypt
and SHA256 are tried. If SHA256 verification is successful, we re-hash
the verified password with Scrypt and replace the stored SHA256 hash
with the new one. This way, as long as a user uses the password once
before the transition period ends (when Scrypt becomes the only valid
algorithm), there would be no need for manual intervention from them.

We will send out notifications to users to remind them of the transition
and urge them to use the password (which should not be a problem with
EPP, but less so with the registry lock). After the transition,
out-of-band reset for EPP password, or remove-and-add on the console for
registry lock password, would be required for the hashes that have not
been re-saved.

Note that the re-save logic is not present for console user's registry
lock password, as there is no production data for console users yet.
Only legacy GAE user's password requires re-save.
2023-11-03 17:29:01 -04:00
Ben McIlwainandGitHub ba54208dad Also load domains for domain checks of type renew/transfer (#2207)
The domains (and their associated billing recurrences) were already being loaded
to check restores, but they also now need to be loaded for renews and transfers
as well, as the billing renewal behavior on the recurrence could be modifying
the relevant renew price that should be shown. (The renew price is used for
transfers as well.)

See https://buganizer.corp.google.com/issues/306212810
2023-11-03 14:33:34 -04:00
111 changed files with 5205 additions and 2638 deletions
+1
View File
@@ -347,6 +347,7 @@ subprojects {
def services = [':services:default',
':services:backend',
':services:bsa',
':services:tools',
':services:pubapi']
@@ -209,7 +209,7 @@ public final class RegistryJpaIO {
@ProcessElement
public void processElement(OutputReceiver<T> outputReceiver) {
tm().transactNoRetry(
tm().transact(
() -> {
query.stream().map(resultMapper::apply).forEach(outputReceiver::output);
});
@@ -0,0 +1,21 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa;
/** Identifiers of the BSA lists with blocking labels. */
public enum BlockList {
BLOCK,
BLOCK_PLUS;
}
@@ -0,0 +1,46 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa;
/** The processing stages of a download. */
public enum DownloadStage {
/** Downloads BSA block list files. */
DOWNLOAD,
/** Generates block list diffs with the previous download. */
MAKE_DIFF,
/** Applies the label diffs to the database tables. */
APPLY_DIFF,
/**
* Makes a REST API call to BSA endpoint, declaring that processing starts for new orders in the
* diffs.
*/
START_UPLOADING,
/** Makes a REST API call to BSA endpoint, sending the domains that cannot be blocked. */
UPLOAD_DOMAINS_IN_USE,
/** Makes a REST API call to BSA endpoint, declaring the completion of order processing. */
FINISH_UPLOADING,
/** The terminal stage after processing succeeds. */
DONE,
/**
* The terminal stage indicating that the downloads are discarded because their checksums are the
* same as that of the previous download.
*/
NOP,
/**
* The terminal stage indicating that the downloads are not processed because their BSA-generated
* checksums do not match those calculated by us.
*/
CHECKSUMS_NOT_MATCH;
}
@@ -0,0 +1,108 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static com.google.common.collect.Maps.transformValues;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableMultimap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
import com.google.common.collect.Sets.SetView;
import google.registry.model.tld.Tld;
import google.registry.model.tld.Tld.TldType;
import google.registry.model.tld.Tlds;
import google.registry.tldconfig.idn.IdnLabelValidator;
import google.registry.tldconfig.idn.IdnTableEnum;
import google.registry.util.Clock;
import javax.inject.Inject;
import org.joda.time.DateTime;
/**
* Checks labels' validity wrt Idns in TLDs enrolled with BSA.
*
* <p>Each instance takes a snapshot of the TLDs at instantiation time, and should be limited to the
* Request scope.
*/
public class IdnChecker {
private static final IdnLabelValidator IDN_LABEL_VALIDATOR = new IdnLabelValidator();
private final ImmutableMap<IdnTableEnum, ImmutableSet<Tld>> idnToTlds;
private final ImmutableSet<Tld> allTlds;
@Inject
IdnChecker(Clock clock) {
this.idnToTlds = getIdnToTldMap(clock.nowUtc());
allTlds = idnToTlds.values().stream().flatMap(ImmutableSet::stream).collect(toImmutableSet());
}
// TODO(11/30/2023): Remove below when new Tld schema is deployed and the `getBsaEnrollStartTime`
// method is no longer hardcoded.
@VisibleForTesting
IdnChecker(ImmutableMap<IdnTableEnum, ImmutableSet<Tld>> idnToTlds) {
this.idnToTlds = idnToTlds;
allTlds = idnToTlds.values().stream().flatMap(ImmutableSet::stream).collect(toImmutableSet());
}
/** Returns all IDNs in which the {@code label} is valid. */
ImmutableSet<IdnTableEnum> getAllValidIdns(String label) {
return idnToTlds.keySet().stream()
.filter(idnTable -> idnTable.getTable().isValidLabel(label))
.collect(toImmutableSet());
}
/**
* Returns the TLDs that support at least one IDN in the {@code idnTables}.
*
* @param idnTables String names of {@link IdnTableEnum} values
*/
public ImmutableSet<Tld> getSupportingTlds(ImmutableSet<String> idnTables) {
return idnTables.stream()
.map(IdnTableEnum::valueOf)
.filter(idnToTlds::containsKey)
.map(idnToTlds::get)
.flatMap(ImmutableSet::stream)
.collect(toImmutableSet());
}
/**
* Returns the TLDs that do not support any IDN in the {@code idnTables}.
*
* @param idnTables String names of {@link IdnTableEnum} values
*/
public SetView<Tld> getForbiddingTlds(ImmutableSet<String> idnTables) {
return Sets.difference(allTlds, getSupportingTlds(idnTables));
}
private static boolean isEnrolledWithBsa(Tld tld, DateTime now) {
DateTime enrollTime = tld.getBsaEnrollStartTime();
return enrollTime != null && enrollTime.isBefore(now);
}
private static ImmutableMap<IdnTableEnum, ImmutableSet<Tld>> getIdnToTldMap(DateTime now) {
ImmutableMultimap.Builder<IdnTableEnum, Tld> idnToTldMap = new ImmutableMultimap.Builder();
Tlds.getTldEntitiesOfType(TldType.REAL).stream()
.filter(tld -> isEnrolledWithBsa(tld, now))
.forEach(
tld -> {
for (IdnTableEnum idn : IDN_LABEL_VALIDATOR.getIdnTablesForTld(tld)) {
idnToTldMap.put(idn, tld);
}
});
return ImmutableMap.copyOf(transformValues(idnToTldMap.build().asMap(), ImmutableSet::copyOf));
}
}
@@ -0,0 +1,45 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa;
import static javax.servlet.http.HttpServletResponse.SC_OK;
import google.registry.request.Action;
import google.registry.request.Action.Service;
import google.registry.request.Response;
import google.registry.request.auth.Auth;
import javax.inject.Inject;
@Action(
service = Service.BSA,
path = PlaceholderAction.PATH,
method = Action.Method.GET,
auth = Auth.AUTH_API_ADMIN)
public class PlaceholderAction implements Runnable {
private final Response response;
static final String PATH = "/_dr/task/bsaDownload";
@Inject
public PlaceholderAction(Response response) {
this.response = response;
}
@Override
public void run() {
response.setStatus(SC_OK);
response.setPayload("Hello World");
}
}
@@ -0,0 +1,102 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import com.google.common.base.Objects;
import google.registry.bsa.persistence.BsaDomainInUse.BsaDomainInUseId;
import google.registry.model.CreateAutoTimestamp;
import google.registry.persistence.VKey;
import java.io.Serializable;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.EnumType;
import javax.persistence.Enumerated;
import javax.persistence.Id;
import javax.persistence.IdClass;
/** A domain matching a BSA label but is in use (registered or reserved), so cannot be blocked. */
@Entity
@IdClass(BsaDomainInUseId.class)
public class BsaDomainInUse {
@Id String label;
@Id String tld;
@Column(nullable = false)
@Enumerated(EnumType.STRING)
Reason reason;
/**
* Creation time of this record, which is the most recent time when the domain was detected to be
* in use wrt BSA. It may be during the processing of a download, or during some other job that
* refreshes the state.
*
* <p>This field is for information only.
*/
@SuppressWarnings("unused")
@Column(nullable = false)
CreateAutoTimestamp createTime = CreateAutoTimestamp.create(null);
// For Hibernate
BsaDomainInUse() {}
public BsaDomainInUse(String label, String tld, Reason reason) {
this.label = label;
this.tld = tld;
this.reason = reason;
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof BsaDomainInUse)) {
return false;
}
BsaDomainInUse that = (BsaDomainInUse) o;
return Objects.equal(label, that.label)
&& Objects.equal(tld, that.tld)
&& reason == that.reason
&& Objects.equal(createTime, that.createTime);
}
@Override
public int hashCode() {
return Objects.hashCode(label, tld, reason, createTime);
}
enum Reason {
REGISTERED,
RESERVED;
}
static class BsaDomainInUseId implements Serializable {
private String label;
private String tld;
// For Hibernate
BsaDomainInUseId() {}
BsaDomainInUseId(String label, String tld) {
this.label = label;
this.tld = tld;
}
}
static VKey<BsaDomainInUse> vKey(String label, String tld) {
return VKey.create(BsaDomainInUse.class, new BsaDomainInUseId(label, tld));
}
}
@@ -0,0 +1,131 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import static com.google.common.collect.ImmutableMap.toImmutableMap;
import static google.registry.bsa.DownloadStage.DOWNLOAD;
import com.google.common.base.Joiner;
import com.google.common.base.Objects;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSortedMap;
import google.registry.bsa.BlockList;
import google.registry.bsa.DownloadStage;
import google.registry.model.CreateAutoTimestamp;
import google.registry.model.UpdateAutoTimestamp;
import google.registry.persistence.VKey;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.EnumType;
import javax.persistence.Enumerated;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Index;
import javax.persistence.Table;
import org.joda.time.DateTime;
/** Records of ongoing and completed download jobs. */
@Entity
@Table(indexes = {@Index(columnList = "creationTime")})
public class BsaDownload {
private static final Joiner CSV_JOINER = Joiner.on(',');
private static final Splitter CSV_SPLITTER = Splitter.on(',');
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
Long jobId;
@Column(nullable = false)
CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);
@Column(nullable = false)
UpdateAutoTimestamp updateTime = UpdateAutoTimestamp.create(null);
@Column(nullable = false)
String blockListChecksums = "";
@Column(nullable = false)
@Enumerated(EnumType.STRING)
DownloadStage stage = DOWNLOAD;
BsaDownload() {}
long getJobId() {
return jobId;
}
DateTime getCreationTime() {
return creationTime.getTimestamp();
}
/**
* Returns the starting time of this job as a string, which can be used as folder name on GCS when
* storing download data.
*/
public String getJobName() {
return getCreationTime().toString();
}
public DownloadStage getStage() {
return this.stage;
}
BsaDownload setStage(DownloadStage stage) {
this.stage = stage;
return this;
}
BsaDownload setChecksums(ImmutableMap<BlockList, String> checksums) {
blockListChecksums =
CSV_JOINER.withKeyValueSeparator("=").join(ImmutableSortedMap.copyOf(checksums));
return this;
}
ImmutableMap<BlockList, String> getChecksums() {
if (blockListChecksums.isEmpty()) {
return ImmutableMap.of();
}
return CSV_SPLITTER.withKeyValueSeparator('=').split(blockListChecksums).entrySet().stream()
.collect(
toImmutableMap(entry -> BlockList.valueOf(entry.getKey()), entry -> entry.getValue()));
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof BsaDownload)) {
return false;
}
BsaDownload that = (BsaDownload) o;
return Objects.equal(creationTime, that.creationTime)
&& Objects.equal(updateTime, that.updateTime)
&& Objects.equal(blockListChecksums, that.blockListChecksums)
&& stage == that.stage;
}
@Override
public int hashCode() {
return Objects.hashCode(creationTime, updateTime, blockListChecksums, stage);
}
static VKey<BsaDownload> vKey(long jobId) {
return VKey.create(BsaDownload.class, Long.valueOf(jobId));
}
}
@@ -0,0 +1,79 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import com.google.common.base.Objects;
import google.registry.persistence.VKey;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import org.joda.time.DateTime;
/**
* Specifies a second-level TLD name that should be blocked from registration in all TLDs except by
* the label's owner.
*
* <p>The label is valid (wrt IDN) in at least one TLD.
*/
@Entity
public final class BsaLabel {
@Id String label;
/**
* Creation time of this label. This field is for human use, and should give the name of the GCS
* folder that contains the downloaded BSA data.
*
* <p>See {@link BsaDownload#getCreationTime} and {@link BsaDownload#getJobName} for more
* information.
*/
@SuppressWarnings("unused")
@Column(nullable = false)
DateTime creationTime;
// For Hibernate.
BsaLabel() {}
BsaLabel(String label, DateTime creationTime) {
this.label = label;
this.creationTime = creationTime;
}
/** Returns the label to be blocked. */
public String getLabel() {
return label;
}
@Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (!(o instanceof BsaLabel)) {
return false;
}
BsaLabel label1 = (BsaLabel) o;
return Objects.equal(label, label1.label) && Objects.equal(creationTime, label1.creationTime);
}
@Override
public int hashCode() {
return Objects.hashCode(label, creationTime);
}
static VKey<BsaLabel> vKey(String label) {
return VKey.create(BsaLabel.class, label);
}
}
@@ -0,0 +1,73 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import com.google.auto.value.AutoValue;
import com.google.common.collect.ImmutableMap;
import google.registry.bsa.BlockList;
import google.registry.bsa.DownloadStage;
import java.util.Optional;
/** Information needed when handling a download from BSA. */
@AutoValue
public abstract class DownloadSchedule {
abstract long jobId();
public abstract String jobName();
public abstract DownloadStage stage();
/** The most recent job that ended in the {@code DONE} stage. */
public abstract Optional<CompletedJob> latestCompleted();
/**
* Returns true if download should be processed even if the checksums show that it has not changed
* from the previous one.
*/
abstract boolean alwaysDownload();
static DownloadSchedule of(BsaDownload currentJob) {
return new AutoValue_DownloadSchedule(
currentJob.getJobId(),
currentJob.getJobName(),
currentJob.getStage(),
Optional.empty(),
/* alwaysDownload= */ true);
}
static DownloadSchedule of(
BsaDownload currentJob, CompletedJob latestCompleted, boolean alwaysDownload) {
return new AutoValue_DownloadSchedule(
currentJob.getJobId(),
currentJob.getJobName(),
currentJob.getStage(),
Optional.of(latestCompleted),
/* alwaysDownload= */ alwaysDownload);
}
/** Information about a completed BSA download job. */
@AutoValue
public abstract static class CompletedJob {
public abstract String jobName();
public abstract ImmutableMap<BlockList, String> checksums();
static CompletedJob of(BsaDownload completedJob) {
return new AutoValue_DownloadSchedule_CompletedJob(
completedJob.getJobName(), completedJob.getChecksums());
}
}
}
@@ -0,0 +1,131 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import static com.google.common.base.Verify.verify;
import static google.registry.bsa.DownloadStage.CHECKSUMS_NOT_MATCH;
import static google.registry.bsa.DownloadStage.DONE;
import static google.registry.bsa.DownloadStage.NOP;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static org.joda.time.Duration.standardSeconds;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import google.registry.bsa.persistence.DownloadSchedule.CompletedJob;
import google.registry.util.Clock;
import java.util.Optional;
import javax.inject.Inject;
import org.joda.time.Duration;
/**
* Assigns work for each cron invocation of the BSA Download job.
*
* <p>The download job is invoked at a divisible fraction of the desired data freshness to
* accommodate potential retries. E.g., for 30-minute data freshness with up to two retries on
* error, the cron schedule for the job should be set to 10 minutes.
*
* <p>The processing of each BSA download progresses through multiple stages as described in {@code
* DownloadStage} until it reaches one of the terminal stages. Each stage is check-pointed on
* completion, therefore if an invocation fails mid-process, the next invocation will skip the
* completed stages. No new downloads will start as long as the most recent one is still being
* processed.
*
* <p>When a new download is scheduled, the block list checksums from the most recent completed job
* is included. If the new checksums match the previous ones, the download may be skipped and the
* job should terminate in the {@code NOP} stage. However, if the checksums have stayed unchanged
* for longer than the user-provided {@code maxNopInterval}, the download will be processed.
*
* <p>The BSA downloads contains server-provided checksums. If they do not match the checksums
* generated on Nomulus' side, the download is skipped and the job should terminate in the {@code
* CHECKSUMS_NOT_MATCH} stage.
*/
public final class DownloadScheduler {
/** Allows a new download to proceed if the cron job fires a little early due to NTP drift. */
private static final Duration CRON_JITTER = standardSeconds(5);
private final Duration downloadInterval;
private final Duration maxNopInterval;
private final Clock clock;
@Inject
DownloadScheduler(Duration downloadInterval, Duration maxNopInterval, Clock clock) {
this.downloadInterval = downloadInterval;
this.maxNopInterval = maxNopInterval;
this.clock = clock;
}
/**
* Returns a {@link DownloadSchedule} instance that describes the work to be performed by an
* invocation of the download action, if applicable; or {@link Optional#empty} when there is
* nothing to do.
*/
public Optional<DownloadSchedule> schedule() {
return tm().transact(
() -> {
ImmutableList<BsaDownload> recentJobs = loadRecentProcessedJobs();
if (recentJobs.isEmpty()) {
// No jobs initiated ever.
return Optional.of(scheduleNewJob(Optional.empty()));
}
BsaDownload mostRecent = recentJobs.get(0);
if (mostRecent.getStage().equals(DONE)) {
return isTimeAgain(mostRecent, downloadInterval)
? Optional.of(scheduleNewJob(Optional.of(mostRecent)))
: Optional.empty();
} else if (recentJobs.size() == 1) {
// First job ever, still in progress
return Optional.of(DownloadSchedule.of(recentJobs.get(0)));
} else {
// Job in progress, with completed previous jobs.
BsaDownload prev = recentJobs.get(1);
verify(prev.getStage().equals(DONE), "Unexpectedly found two ongoing jobs.");
return Optional.of(
DownloadSchedule.of(
mostRecent,
CompletedJob.of(prev),
isTimeAgain(mostRecent, maxNopInterval)));
}
});
}
private boolean isTimeAgain(BsaDownload mostRecent, Duration interval) {
return mostRecent.getCreationTime().plus(interval).minus(CRON_JITTER).isBefore(clock.nowUtc());
}
/**
* Adds a new {@link BsaDownload} to the database and returns a {@link DownloadSchedule} for it.
*/
private DownloadSchedule scheduleNewJob(Optional<BsaDownload> prevJob) {
BsaDownload job = new BsaDownload();
tm().insert(job);
return prevJob
.map(
prev ->
DownloadSchedule.of(job, CompletedJob.of(prev), isTimeAgain(prev, maxNopInterval)))
.orElseGet(() -> DownloadSchedule.of(job));
}
@VisibleForTesting
ImmutableList<BsaDownload> loadRecentProcessedJobs() {
return ImmutableList.copyOf(
tm().getEntityManager()
.createQuery(
"FROM BsaDownload WHERE stage NOT IN :nop_stages ORDER BY creationTime DESC")
.setParameter("nop_stages", ImmutableList.of(CHECKSUMS_NOT_MATCH, NOP))
.setMaxResults(2)
.getResultList());
}
}
@@ -1192,6 +1192,12 @@ public final class RegistryConfig {
return config.auth.oauthClientId;
}
@Provides
@Config("fallbackOauthClientId")
public static String provideFallbackOauthClientId(RegistryConfigSettings config) {
return config.auth.fallbackOauthClientId;
}
/**
* Provides the OAuth scopes required for accessing Google APIs using the default credential.
*/
@@ -1452,6 +1458,15 @@ public final class RegistryConfig {
return makeUrl(CONFIG_SETTINGS.get().gcpProject.backendServiceUrl);
}
/**
* Returns the address of the Nomulus app bsa HTTP server.
*
* <p>This is used by the {@code nomulus} tool to connect to the App Engine remote API.
*/
public static URL getBsaServer() {
return makeUrl(CONFIG_SETTINGS.get().gcpProject.bsaServiceUrl);
}
/**
* Returns the address of the Nomulus app tools HTTP server.
*
@@ -1548,9 +1563,9 @@ public final class RegistryConfig {
return CONFIG_SETTINGS.get().hibernate.connectionIsolation;
}
/** Returns true if per-transaction isolation level is enabled. */
public static boolean getHibernatePerTransactionIsolationEnabled() {
return CONFIG_SETTINGS.get().hibernate.perTransactionIsolation;
/** Returns true if nested calls to {@code tm().transact()} are allowed. */
public static boolean getHibernateAllowNestedTransactions() {
return CONFIG_SETTINGS.get().hibernate.allowNestedTransactions;
}
/** Returns true if hibernate.show_sql is enabled. */
@@ -53,6 +53,7 @@ public class RegistryConfigSettings {
public boolean isLocal;
public String defaultServiceUrl;
public String backendServiceUrl;
public String bsaServiceUrl;
public String toolsServiceUrl;
public String pubapiServiceUrl;
}
@@ -61,6 +62,7 @@ public class RegistryConfigSettings {
public static class Auth {
public List<String> allowedServiceAccountEmails;
public String oauthClientId;
public String fallbackOauthClientId;
}
/** Configuration options for accessing Google APIs. */
@@ -113,7 +115,7 @@ public class RegistryConfigSettings {
/** Configuration for Hibernate. */
public static class Hibernate {
public boolean perTransactionIsolation;
public boolean allowNestedTransactions;
public String connectionIsolation;
public String logSqlQueries;
public String hikariConnectionTimeout;
@@ -20,9 +20,11 @@ gcpProject:
# URLs of the services for the project.
defaultServiceUrl: https://default.example.com
backendServiceUrl: https://backend.example.com
bsaServiceUrl: https://bsa.example.com
toolsServiceUrl: https://tools.example.com
pubapiServiceUrl: https://pubapi.example.com
gSuite:
# Publicly accessible domain name of the running G Suite instance.
domainName: domain-registry.example
@@ -189,11 +191,13 @@ registryPolicy:
sunriseDomainCreateDiscount: 0.15
hibernate:
# Make it possible to specify the isolation level for each transaction. If set
# to true, nested transactions will throw an exception. If set to false, a
# transaction with the isolation override specified will still execute at the
# default level (specified below).
perTransactionIsolation: true
# If set to false, calls to tm().transact() cannot be nested. If set to true,
# nested calls to tm().transact() are allowed, as long as they do not specify
# a transaction isolation level override. These nested transactions should
# either be refactored to non-nested transactions, or changed to
# tm().reTransact(), which explicitly allows nested transactions, but does not
# allow setting an isolation level override.
allowNestedTransactions: true
# Make 'SERIALIZABLE' the default isolation level to ensure correctness.
#
@@ -319,6 +323,10 @@ auth:
# the same as this one.
oauthClientId: iap-oauth-clientid
# Same as above, but serve as a fallback, so we can switch the client ID of
# the proxy without downtime.
fallbackOauthClientId: fallback-oauth-clientid
credentialOAuth:
# OAuth scopes required for accessing Google APIs using the default
# credential.
@@ -0,0 +1,31 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<service>bsa</service>
<app-engine-apis>true</app-engine-apis>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
<max-instances>100</max-instances>
<idle-timeout>10m</idle-timeout>
</basic-scaling>
<system-properties>
<property name="java.util.logging.config.file"
value="WEB-INF/logging.properties"/>
<property name="google.registry.environment"
value="alpha"/>
</system-properties>
<!-- Enable external traffic to go through VPC, required for static ip -->
<vpc-access-connector>
<name>projects/domain-registry-alpha/locations/us-central1/connectors/appengine-connector</name>
<egress-setting>all-traffic</egress-setting>
</vpc-access-connector>
<static-files>
<include path="/*.html" expiration="1m"/>
</static-files>
</appengine-web-app>
@@ -31,6 +31,12 @@ encoding="UTF-8"?>
<context-root>backend</context-root>
</web>
</module>
<module>
<web>
<web-uri>bsa</web-uri>
<context-root>bsa</context-root>
</web>
</module>
<module>
<web>
<web-uri>tools</web-uri>
@@ -0,0 +1,17 @@
# A default java.util.logging configuration.
# (All App Engine logging is through java.util.logging by default).
#
# To use this configuration, copy it into your application's WEB-INF
# folder and add the following to your appengine-web.xml:
#
# <system-properties>
# <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
# </system-properties>
#
# Set the default logging level for all loggers to INFO.
.level = INFO
# Turn off logging in Hibernate classes for misleading ERROR-level logs
org.hibernate.engine.jdbc.batch.internal.BatchingBatch.level=OFF
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.level=OFF
@@ -0,0 +1,70 @@
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" version="2.5"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
<!-- Servlets -->
<!-- Servlet for injected backends actions -->
<servlet>
<display-name>BsaServlet</display-name>
<servlet-name>bsa-servlet</servlet-name>
<servlet-class>google.registry.module.bsa.BsaServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<!-- Test action -->
<servlet-mapping>
<servlet-name>backend-servlet</servlet-name>
<url-pattern>/_dr/task/bsa</url-pattern>
</servlet-mapping>
<!-- Security config -->
<security-constraint>
<web-resource-collection>
<web-resource-name>Internal</web-resource-name>
<description>
Admin-only internal section. Requests for paths covered by the URL patterns below will be
checked for a logged-in user account that's allowed to access the AppEngine admin console
(NOTE: this includes Editor/Viewer permissions in addition to Owner and the new IAM
App Engine Admin role. See https://cloud.google.com/appengine/docs/java/access-control
specifically the "Access handlers that have a login:admin restriction" line.)
TODO(b/28219927): lift some of these restrictions so that we can allow OAuth authentication
for endpoints that need to be accessed by open-source automated processes.
</description>
<!-- Internal AppEngine endpoints. The '_ah' is short for app hosting. -->
<url-pattern>/_ah/*</url-pattern>
<!-- Registrar console (should not be available on non-default module). -->
<url-pattern>/registrar*</url-pattern>
<!-- Verbatim JavaScript sources (only visible to admins for debugging). -->
<url-pattern>/assets/sources/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
<!-- Repeated here since catch-all rule below is not inherited. -->
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
<!-- Require TLS on all requests. -->
<security-constraint>
<web-resource-collection>
<web-resource-name>Secure</web-resource-name>
<description>
Require encryption for all paths. http URLs will be redirected to https.
</description>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
</security-constraint>
</web-app>
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<service>bsa</service>
<app-engine-apis>true</app-engine-apis>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
<max-instances>10</max-instances>
<idle-timeout>10m</idle-timeout>
</basic-scaling>
<system-properties>
<property name="java.util.logging.config.file"
value="WEB-INF/logging.properties"/>
<property name="google.registry.environment"
value="crash"/>
</system-properties>
<!-- Enable external traffic to go through VPC, required for static ip -->
<vpc-access-connector>
<name>projects/domain-registry-crash/locations/us-central1/connectors/appengine-connector</name>
<egress-setting>all-traffic</egress-setting>
</vpc-access-connector>
<static-files>
<include path="/*.html" expiration="1m"/>
</static-files>
</appengine-web-app>
@@ -0,0 +1,28 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<service>bsa</service>
<app-engine-apis>true</app-engine-apis>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
<max-instances>10</max-instances>
<idle-timeout>10m</idle-timeout>
</basic-scaling>
<system-properties>
<property name="java.util.logging.config.file"
value="WEB-INF/logging.properties"/>
<property name="google.registry.environment"
value="local"/>
<property name="appengine.generated.dir"
value="/tmp/domain-registry-appengine-generated/local/"/>
</system-properties>
<static-files>
<include path="/*.html">
<http-header name="Cache-Control" value="max-age=0,must-revalidate" />
</include>
</static-files>
</appengine-web-app>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>backend</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<basic-scaling>
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java8</runtime>
<service>bsa</service>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<basic-scaling>
<max-instances>100</max-instances>
<idle-timeout>10m</idle-timeout>
</basic-scaling>
<system-properties>
<property name="java.util.logging.config.file"
value="WEB-INF/logging.properties"/>
<property name="google.registry.environment"
value="production"/>
</system-properties>
<static-files>
<include path="/*.html" expiration="1d"/>
</static-files>
<!-- Prevent uncaught servlet errors from leaking a stack trace. -->
<static-error-handlers>
<handler file="error.html"/>
</static-error-handlers>
</appengine-web-app>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>default</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<manual-scaling>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>pubapi</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<manual-scaling>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>tools</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<basic-scaling>
@@ -0,0 +1,35 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<service>bsa</service>
<app-engine-apis>true</app-engine-apis>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
<max-instances>10</max-instances>
<idle-timeout>10m</idle-timeout>
</basic-scaling>
<system-properties>
<property name="java.util.logging.config.file"
value="WEB-INF/logging.properties"/>
<property name="google.registry.environment"
value="qa"/>
</system-properties>
<static-files>
<include path="/*.html" expiration="1h"/>
</static-files>
<!-- Enable external traffic to go through VPC, required for static ip -->
<vpc-access-connector>
<name>projects/domain-registry-qa/locations/us-central1/connectors/appengine-connector</name>
<egress-setting>all-traffic</egress-setting>
</vpc-access-connector>
<!-- Prevent uncaught servlet errors from leaking a stack trace. -->
<static-error-handlers>
<handler file="error.html"/>
</static-error-handlers>
</appengine-web-app>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>backend</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
@@ -22,12 +23,6 @@
<include path="/*.html" expiration="1d"/>
</static-files>
<!-- Enable external traffic to go through VPC, required for static ip -->
<vpc-access-connector>
<name>projects/domain-registry-sandbox/locations/us-central1/connectors/appengine-connector</name>
<egress-setting>all-traffic</egress-setting>
</vpc-access-connector>
<!-- Prevent uncaught servlet errors from leaking a stack trace. -->
<static-error-handlers>
<handler file="error.html"/>
@@ -0,0 +1,36 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java8</runtime>
<service>bsa</service>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
<max-instances>100</max-instances>
<idle-timeout>10m</idle-timeout>
</basic-scaling>
<system-properties>
<property name="java.util.logging.config.file"
value="WEB-INF/logging.properties"/>
<property name="google.registry.environment"
value="sandbox"/>
</system-properties>
<static-files>
<include path="/*.html" expiration="1d"/>
</static-files>
<!-- Enable external traffic to go through VPC, required for static ip -->
<vpc-access-connector>
<name>projects/domain-registry-sandbox/locations/us-central1/connectors/appengine-connector</name>
<egress-setting>all-traffic</egress-setting>
</vpc-access-connector>
<!-- Prevent uncaught servlet errors from leaking a stack trace. -->
<static-error-handlers>
<handler file="error.html"/>
</static-error-handlers>
</appengine-web-app>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>default</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<manual-scaling>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>pubapi</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4_1G</instance-class>
<manual-scaling>
@@ -1,9 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
<runtime>java17</runtime>
<runtime>java8</runtime>
<service>tools</service>
<app-engine-apis>true</app-engine-apis>
<!--app-engine-apis>true</app-engine-apis-->
<threadsafe>true</threadsafe>
<sessions-enabled>true</sessions-enabled>
<instance-class>B4</instance-class>
<basic-scaling>
@@ -66,7 +66,6 @@ import google.registry.model.domain.DomainCommand.Check;
import google.registry.model.domain.fee.FeeCheckCommandExtension;
import google.registry.model.domain.fee.FeeCheckCommandExtensionItem;
import google.registry.model.domain.fee.FeeCheckResponseExtensionItem;
import google.registry.model.domain.fee.FeeQueryCommandExtensionItem.CommandName;
import google.registry.model.domain.fee06.FeeCheckCommandExtensionV06;
import google.registry.model.domain.launch.LaunchCheckExtension;
import google.registry.model.domain.token.AllocationToken;
@@ -272,7 +271,7 @@ public final class DomainCheckFlow implements TransactionalFlow {
ImmutableList.Builder<FeeCheckResponseExtensionItem> responseItems =
new ImmutableList.Builder<>();
ImmutableMap<String, Domain> domainObjs =
loadDomainsForRestoreChecks(feeCheck, domainNames, existingDomains);
loadDomainsForChecks(feeCheck, domainNames, existingDomains);
ImmutableMap<String, BillingRecurrence> recurrences = loadRecurrencesForDomains(domainObjs);
for (FeeCheckCommandExtensionItem feeCheckItem : feeCheck.getItems()) {
@@ -335,17 +334,20 @@ public final class DomainCheckFlow implements TransactionalFlow {
}
/**
* Loads and returns all existing domains that are having restore fees checked.
* Loads and returns all existing domains that are having restore/renew/transfer fees checked.
*
* <p>This is necessary so that we can check their expiration dates to determine if a one-year
* renewal is part of the cost of a restore.
* <p>These need to be loaded for renews and transfers because there could be a relevant {@link
* google.registry.model.billing.BillingBase.RenewalPriceBehavior} on the {@link
* BillingRecurrence} affecting the price. They also need to be loaded for restores so that we can
* check their expiration dates to determine if a one-year renewal is part of the cost of a
* restore.
*
* <p>This may be resource-intensive for large checks of many restore fees, but those are
* comparatively rare, and we are at least using an in-memory cache. Also, this will get a lot
* nicer in Cloud SQL when we can SELECT just the fields we want rather than having to load the
* entire entity.
*/
private ImmutableMap<String, Domain> loadDomainsForRestoreChecks(
private ImmutableMap<String, Domain> loadDomainsForChecks(
FeeCheckCommandExtension<?, ?> feeCheck,
ImmutableMap<String, InternetDomainName> domainNames,
ImmutableMap<String, VKey<Domain>> existingDomains) {
@@ -354,18 +356,18 @@ public final class DomainCheckFlow implements TransactionalFlow {
// The V06 fee extension supports specifying the command fees to check on a per-domain basis.
restoreCheckDomains =
feeCheck.getItems().stream()
.filter(fc -> fc.getCommandName() == CommandName.RESTORE)
.filter(fc -> fc.getCommandName().shouldLoadDomainForCheck())
.map(FeeCheckCommandExtensionItem::getDomainName)
.distinct()
.collect(toImmutableList());
} else if (feeCheck.getItems().stream()
.anyMatch(fc -> fc.getCommandName() == CommandName.RESTORE)) {
.anyMatch(fc -> fc.getCommandName().shouldLoadDomainForCheck())) {
// The more recent fee extension versions support specifying the command fees to check only on
// the overall domain check, not per-domain.
restoreCheckDomains = ImmutableList.copyOf(domainNames.keySet());
} else {
// Fall-through case for more recent fee extension versions when the restore fee isn't being
// checked.
// Fall-through case for more recent fee extension versions when the restore/renew/transfer
// fees aren't being checked.
restoreCheckDomains = ImmutableList.of();
}
@@ -28,11 +28,17 @@ import google.registry.flows.EppException.CommandUseErrorException;
import google.registry.flows.EppException.ParameterValuePolicyErrorException;
import google.registry.flows.EppException.UnimplementedExtensionException;
import google.registry.flows.EppException.UnimplementedObjectServiceException;
import google.registry.flows.EppException.UnimplementedProtocolVersionException;
import google.registry.flows.ExtensionManager;
import google.registry.flows.FlowModule.RegistrarId;
import google.registry.flows.FlowUtils.GenericXmlSyntaxErrorException;
import google.registry.flows.MutatingFlow;
import google.registry.flows.SessionMetadata;
import google.registry.flows.TlsCredentials.BadRegistrarCertificateException;
import google.registry.flows.TlsCredentials.BadRegistrarIpAddressException;
import google.registry.flows.TlsCredentials.MissingRegistrarCertificateException;
import google.registry.flows.TransportCredentials;
import google.registry.flows.TransportCredentials.BadRegistrarPasswordException;
import google.registry.model.eppcommon.ProtocolDefinition;
import google.registry.model.eppcommon.ProtocolDefinition.ServiceExtension;
import google.registry.model.eppinput.EppInput;
@@ -41,6 +47,7 @@ import google.registry.model.eppinput.EppInput.Options;
import google.registry.model.eppinput.EppInput.Services;
import google.registry.model.eppoutput.EppResponse;
import google.registry.model.registrar.Registrar;
import google.registry.util.PasswordUtils.HashAlgorithm;
import java.util.Optional;
import java.util.Set;
import javax.inject.Inject;
@@ -48,14 +55,14 @@ import javax.inject.Inject;
/**
* An EPP flow for login.
*
* @error {@link google.registry.flows.EppException.UnimplementedExtensionException}
* @error {@link google.registry.flows.EppException.UnimplementedObjectServiceException}
* @error {@link google.registry.flows.EppException.UnimplementedProtocolVersionException}
* @error {@link google.registry.flows.FlowUtils.GenericXmlSyntaxErrorException}
* @error {@link google.registry.flows.TlsCredentials.BadRegistrarCertificateException}
* @error {@link google.registry.flows.TlsCredentials.BadRegistrarIpAddressException}
* @error {@link google.registry.flows.TlsCredentials.MissingRegistrarCertificateException}
* @error {@link google.registry.flows.TransportCredentials.BadRegistrarPasswordException}
* @error {@link UnimplementedExtensionException}
* @error {@link UnimplementedObjectServiceException}
* @error {@link UnimplementedProtocolVersionException}
* @error {@link GenericXmlSyntaxErrorException}
* @error {@link BadRegistrarCertificateException}
* @error {@link BadRegistrarIpAddressException}
* @error {@link MissingRegistrarCertificateException}
* @error {@link BadRegistrarPasswordException}
* @error {@link LoginFlow.AlreadyLoggedInException}
* @error {@link BadRegistrarIdException}
* @error {@link LoginFlow.TooManyFailedLoginsException}
@@ -134,13 +141,24 @@ public class LoginFlow implements MutatingFlow {
if (!registrar.get().isLive()) {
throw new RegistrarAccountNotActiveException();
}
if (login.getNewPassword().isPresent()) {
if (login.getNewPassword().isPresent()
|| registrar.get().getCurrentHashAlgorithm(login.getPassword()).orElse(null)
!= HashAlgorithm.SCRYPT) {
String newPassword =
login
.getNewPassword()
.orElseGet(
() -> {
logger.atInfo().log("Rehashing existing registrar password with Scrypt");
return login.getPassword();
});
// Load fresh from database (bypassing the cache) to ensure we don't save stale data.
Optional<Registrar> freshRegistrar = Registrar.loadByRegistrarId(login.getClientId());
if (!freshRegistrar.isPresent()) {
throw new BadRegistrarIdException(login.getClientId());
}
tm().put(freshRegistrar.get().asBuilder().setPassword(login.getNewPassword().get()).build());
tm().put(freshRegistrar.get().asBuilder().setPassword(newPassword).build());
}
// We are in!
@@ -152,35 +170,35 @@ public class LoginFlow implements MutatingFlow {
/** Registrar with this ID could not be found. */
static class BadRegistrarIdException extends AuthenticationErrorException {
public BadRegistrarIdException(String registrarId) {
BadRegistrarIdException(String registrarId) {
super("Registrar with this ID could not be found: " + registrarId);
}
}
/** Registrar login failed too many times. */
static class TooManyFailedLoginsException extends AuthenticationErrorClosingConnectionException {
public TooManyFailedLoginsException() {
TooManyFailedLoginsException() {
super("Registrar login failed too many times");
}
}
/** Registrar account is not active. */
static class RegistrarAccountNotActiveException extends AuthorizationErrorException {
public RegistrarAccountNotActiveException() {
RegistrarAccountNotActiveException() {
super("Registrar account is not active");
}
}
/** Registrar is already logged in. */
static class AlreadyLoggedInException extends CommandUseErrorException {
public AlreadyLoggedInException() {
AlreadyLoggedInException() {
super("Registrar is already logged in");
}
}
/** Specified language is not supported. */
static class UnsupportedLanguageException extends ParameterValuePolicyErrorException {
public UnsupportedLanguageException() {
UnsupportedLanguageException() {
super("Specified language is not supported");
}
}
@@ -138,6 +138,7 @@ public final class GmailClient {
BodyPart attachmentPart = new MimeBodyPart();
attachmentPart.setContent(attachment.content(), attachment.contentType().toString());
attachmentPart.setFileName(attachment.filename());
attachmentPart.setDisposition(MimeBodyPart.ATTACHMENT);
multipart.addBodyPart(attachmentPart);
}
msg.addRecipients(RecipientType.BCC, toArray(emailMessage.bccs(), Address.class));
@@ -25,6 +25,7 @@ import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import google.registry.model.Buildable;
import google.registry.model.UpdateAutoTimestampEntity;
import google.registry.persistence.VKey;
import google.registry.util.PasswordUtils;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
@@ -84,8 +85,9 @@ public class User extends UpdateAutoTimestampEntity implements Buildable {
|| isNullOrEmpty(registryLockPasswordHash)) {
return false;
}
return hashPassword(registryLockPassword, registryLockPasswordSalt)
.equals(registryLockPasswordHash);
return PasswordUtils.verifyPassword(
registryLockPassword, registryLockPasswordHash, registryLockPasswordSalt)
.isPresent();
}
/**
@@ -154,9 +156,9 @@ public class User extends UpdateAutoTimestampEntity implements Buildable {
!getInstance().hasRegistryLockPassword(), "User already has a password, remove it first");
checkArgument(
!isNullOrEmpty(registryLockPassword), "Registry lock password was null or empty");
getInstance().registryLockPasswordSalt = base64().encode(SALT_SUPPLIER.get());
getInstance().registryLockPasswordHash =
hashPassword(registryLockPassword, getInstance().registryLockPasswordSalt);
byte[] salt = SALT_SUPPLIER.get();
getInstance().registryLockPasswordSalt = base64().encode(salt);
getInstance().registryLockPasswordHash = hashPassword(registryLockPassword, salt);
return this;
}
}
@@ -34,12 +34,14 @@ public abstract class FeeQueryCommandExtensionItem extends ImmutableObject {
/** The name of a command that might have an associated fee. */
public enum CommandName {
UNKNOWN,
CREATE,
RENEW,
TRANSFER,
RESTORE,
UPDATE;
UNKNOWN(false),
CREATE(false),
RENEW(true),
TRANSFER(true),
RESTORE(true),
UPDATE(false);
private final boolean loadDomainForCheck;
public static CommandName parseKnownCommand(String string) {
try {
@@ -52,6 +54,14 @@ public abstract class FeeQueryCommandExtensionItem extends ImmutableObject {
+ " UPDATE");
}
}
CommandName(boolean loadDomainForCheck) {
this.loadDomainForCheck = loadDomainForCheck;
}
public boolean shouldLoadDomainForCheck() {
return this.loadDomainForCheck;
}
}
/** The default validity period (if not specified) is 1 year for all operations. */
@@ -60,6 +60,8 @@ import google.registry.model.tld.Tld;
import google.registry.model.tld.Tld.TldType;
import google.registry.persistence.VKey;
import google.registry.util.CidrAddressBlock;
import google.registry.util.PasswordUtils;
import google.registry.util.PasswordUtils.HashAlgorithm;
import java.security.cert.CertificateParsingException;
import java.util.Comparator;
import java.util.List;
@@ -97,7 +99,7 @@ import org.joda.time.DateTime;
column = @Column(nullable = false, name = "lastUpdateTime"))
public class Registrar extends UpdateAutoTimestampEntity implements Buildable, Jsonifiable {
/** Represents the type of a registrar entity. */
/** Represents the type of registrar entity. */
public enum Type {
/** A real-world, third-party registrar. Should have non-null IANA and billing account IDs. */
REAL(Objects::nonNull),
@@ -376,7 +378,7 @@ public class Registrar extends UpdateAutoTimestampEntity implements Buildable, J
*/
@Expose String icannReferralEmail;
/** Id of the folder in drive used to publish information for this registrar. */
/** ID of the folder in drive used to publish information for this registrar. */
@Expose String driveFolderId;
// Metadata.
@@ -639,7 +641,11 @@ public class Registrar extends UpdateAutoTimestampEntity implements Buildable, J
}
public boolean verifyPassword(String password) {
return hashPassword(password, salt).equals(passwordHash);
return getCurrentHashAlgorithm(password).isPresent();
}
public Optional<HashAlgorithm> getCurrentHashAlgorithm(String password) {
return PasswordUtils.verifyPassword(password, passwordHash, salt);
}
public String getPhonePasscode() {
@@ -861,8 +867,9 @@ public class Registrar extends UpdateAutoTimestampEntity implements Buildable, J
checkArgument(
Range.closed(6, 16).contains(nullToEmpty(password).length()),
"Password must be 6-16 characters long.");
getInstance().salt = base64().encode(SALT_SUPPLIER.get());
getInstance().passwordHash = hashPassword(password, getInstance().salt);
byte[] salt = SALT_SUPPLIER.get();
getInstance().salt = base64().encode(salt);
getInstance().passwordHash = hashPassword(password, salt);
return this;
}
@@ -37,6 +37,8 @@ import google.registry.model.Jsonifiable;
import google.registry.model.UnsafeSerializable;
import google.registry.model.registrar.RegistrarPoc.RegistrarPocId;
import google.registry.persistence.VKey;
import google.registry.util.PasswordUtils;
import google.registry.util.PasswordUtils.HashAlgorithm;
import java.io.Serializable;
import java.util.Map;
import java.util.Optional;
@@ -240,8 +242,12 @@ public class RegistrarPoc extends ImmutableObject implements Jsonifiable, Unsafe
|| isNullOrEmpty(registryLockPasswordHash)) {
return false;
}
return hashPassword(registryLockPassword, registryLockPasswordSalt)
.equals(registryLockPasswordHash);
return getCurrentHashAlgorithm(registryLockPassword).isPresent();
}
public Optional<HashAlgorithm> getCurrentHashAlgorithm(String registryLockPassword) {
return PasswordUtils.verifyPassword(
registryLockPassword, registryLockPasswordHash, registryLockPasswordSalt);
}
/**
@@ -436,9 +442,9 @@ public class RegistrarPoc extends ImmutableObject implements Jsonifiable, Unsafe
"Not allowed to set registry lock password for this contact");
checkArgument(
!isNullOrEmpty(registryLockPassword), "Registry lock password was null or empty");
getInstance().registryLockPasswordSalt = base64().encode(SALT_SUPPLIER.get());
getInstance().registryLockPasswordHash =
hashPassword(registryLockPassword, getInstance().registryLockPasswordSalt);
byte[] salt = SALT_SUPPLIER.get();
getInstance().registryLockPasswordSalt = base64().encode(salt);
getInstance().registryLockPasswordHash = hashPassword(registryLockPassword, salt);
getInstance().allowedToSetRegistryLockPassword = false;
return this;
}
@@ -25,9 +25,10 @@ import com.google.common.base.Strings;
import com.google.common.flogger.FluentLogger;
import google.registry.model.ImmutableObject;
import google.registry.persistence.VKey;
import google.registry.persistence.transaction.TransactionManager.ThrowingRunnable;
import java.io.Serializable;
import java.util.Optional;
import java.util.function.Supplier;
import java.util.concurrent.Callable;
import javax.annotation.Nullable;
import javax.persistence.Column;
import javax.persistence.Entity;
@@ -184,7 +185,7 @@ public class Lock extends ImmutableObject implements Serializable {
public static Optional<Lock> acquire(
String resourceName, @Nullable String tld, Duration leaseLength) {
String scope = tld != null ? tld : GLOBAL;
Supplier<AcquireResult> lockAcquirer =
Callable<AcquireResult> lockAcquirer =
() -> {
DateTime now = tm().getTransactionTime();
@@ -221,7 +222,7 @@ public class Lock extends ImmutableObject implements Serializable {
/** Release the lock. */
public void release() {
// Just use the default clock because we aren't actually doing anything that will use the clock.
Supplier<Void> lockReleaser =
ThrowingRunnable lockReleaser =
() -> {
// To release a lock, check that no one else has already obtained it and if not
// delete it. If the lock in the database was different, then this lock is gone already;
@@ -246,7 +247,6 @@ public class Lock extends ImmutableObject implements Serializable {
logger.atInfo().log(
"Not deleting lock: %s - someone else has it: %s", lockId, loadedLock);
}
return null;
};
tm().transact(lockReleaser);
}
@@ -550,6 +550,10 @@ public class Tld extends ImmutableObject implements Buildable, UnsafeSerializabl
@JsonSerialize(using = SortedEnumSetSerializer.class)
Set<IdnTableEnum> idnTables;
// TODO(11/30/2023): uncomment below two lines
// /** The start time of this TLD's enrollment in the BSA program, if applicable. */
// @JsonIgnore @Nullable DateTime bsaEnrollStartTime;
public String getTldStr() {
return tldStr;
}
@@ -569,6 +573,15 @@ public class Tld extends ImmutableObject implements Buildable, UnsafeSerializabl
return tldType;
}
/** Returns the time when this TLD was enrolled in the Brand Safety Alliance (BSA) program. */
@JsonIgnore // Annotation can be removed once we add the field and annotate it.
@Nullable
public DateTime getBsaEnrollStartTime() {
// TODO(11/30/2023): uncomment below.
// return this.bsaEnrollStartTime;
return null;
}
/** Retrieve whether invoicing is enabled. */
public boolean isInvoicingEnabled() {
return invoicingEnabled;
@@ -939,6 +952,7 @@ public class Tld extends ImmutableObject implements Buildable, UnsafeSerializabl
}
public Builder setReservedListsByName(Set<String> reservedListNames) {
// TODO(b/309175133): forbid if enrolled with BSA
checkArgument(reservedListNames != null, "reservedListNames must not be null");
ImmutableSet.Builder<ReservedList> builder = new ImmutableSet.Builder<>();
for (String reservedListName : reservedListNames) {
@@ -958,6 +972,7 @@ public class Tld extends ImmutableObject implements Buildable, UnsafeSerializabl
}
public Builder setReservedLists(Set<ReservedList> reservedLists) {
// TODO(b/309175133): forbid if enrolled with BSA
checkArgumentNotNull(reservedLists, "reservedLists must not be null");
ImmutableSet.Builder<String> nameBuilder = new ImmutableSet.Builder<>();
for (ReservedList reservedList : reservedLists) {
@@ -1076,6 +1091,7 @@ public class Tld extends ImmutableObject implements Buildable, UnsafeSerializabl
}
public Builder setIdnTables(ImmutableSet<IdnTableEnum> idnTables) {
// TODO(b/309175133): forbid if enrolled with BSA.
getInstance().idnTables = idnTables;
return this;
}
@@ -1085,6 +1101,13 @@ public class Tld extends ImmutableObject implements Buildable, UnsafeSerializabl
return this;
}
public Builder setBsaEnrollStartTime(DateTime enrollTime) {
// TODO(b/309175133): forbid if enrolled with BSA
// TODO(11/30/2023): uncomment below line
// getInstance().bsaEnrollStartTime = enrollTime;
return this;
}
@Override
public Tld build() {
final Tld instance = getInstance();
@@ -0,0 +1,46 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.module.bsa;
import com.google.monitoring.metrics.MetricReporter;
import dagger.Component;
import dagger.Lazy;
import google.registry.config.CredentialModule;
import google.registry.config.RegistryConfig.ConfigModule;
import google.registry.module.bsa.BsaRequestComponent.BsaRequestComponentModule;
import google.registry.monitoring.whitebox.StackdriverModule;
import google.registry.request.Modules.GsonModule;
import google.registry.request.Modules.UserServiceModule;
import google.registry.request.auth.AuthModule;
import google.registry.util.UtilsModule;
import javax.inject.Singleton;
@Singleton
@Component(
modules = {
AuthModule.class,
UtilsModule.class,
UserServiceModule.class,
GsonModule.class,
ConfigModule.class,
StackdriverModule.class,
CredentialModule.class,
BsaRequestComponentModule.class
})
interface BsaComponent {
BsaRequestHandler requestHandler();
Lazy<MetricReporter> metricReporter();
}
@@ -0,0 +1,45 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.module.bsa;
import dagger.Module;
import dagger.Subcomponent;
import google.registry.bsa.PlaceholderAction;
import google.registry.request.RequestComponentBuilder;
import google.registry.request.RequestModule;
import google.registry.request.RequestScope;
@RequestScope
@Subcomponent(
modules = {
RequestModule.class,
})
interface BsaRequestComponent {
PlaceholderAction bsaAction();
@Subcomponent.Builder
abstract class Builder implements RequestComponentBuilder<BsaRequestComponent> {
@Override
public abstract Builder requestModule(RequestModule requestModule);
@Override
public abstract BsaRequestComponent build();
}
@Module(subcomponents = BsaRequestComponent.class)
class BsaRequestComponentModule {}
}
@@ -0,0 +1,29 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.module.bsa;
import google.registry.request.RequestHandler;
import google.registry.request.auth.RequestAuthenticator;
import javax.inject.Inject;
import javax.inject.Provider;
public class BsaRequestHandler extends RequestHandler<BsaRequestComponent> {
@Inject
public BsaRequestHandler(
Provider<BsaRequestComponent.Builder> componentBuilderProvider,
RequestAuthenticator requestAuthenticator) {
super(componentBuilderProvider, requestAuthenticator);
}
}
@@ -0,0 +1,30 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.module.bsa;
import com.google.monitoring.metrics.MetricReporter;
import dagger.Lazy;
import google.registry.module.ServletBase;
public final class BsaServlet extends ServletBase {
private static final BsaComponent component = DaggerBsaComponent.create();
private static final BsaRequestHandler requestHandler = component.requestHandler();
private static final Lazy<MetricReporter> metricReporter = component.metricReporter();
public BsaServlet() {
super(requestHandler, metricReporter);
}
}
@@ -0,0 +1,16 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
@javax.annotation.ParametersAreNonnullByDefault
package google.registry.module.bsa;
@@ -39,7 +39,7 @@ import javax.persistence.PersistenceException;
* <p>See the {@code logging.properties} files in the {@code env} package for the specific Hibernate
* classes that have logging suppressed.
*/
class DatabaseException extends PersistenceException {
public class DatabaseException extends PersistenceException {
private transient String cachedMessage;
@@ -62,7 +62,7 @@ class DatabaseException extends PersistenceException {
* <p>If the {@code original Throwable} has at least one {@link SQLException} in its chain of
* causes, a {@link DatabaseException} is thrown; otherwise this does nothing.
*/
static void tryWrapAndThrow(Throwable original) {
static void throwIfSqlException(Throwable original) {
Throwable t = original;
do {
if (t instanceof SQLException) {
@@ -16,7 +16,6 @@ package google.registry.persistence.transaction;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.VKey;
import java.util.function.Supplier;
import javax.persistence.EntityManager;
import javax.persistence.Query;
import javax.persistence.TypedQuery;
@@ -62,24 +61,6 @@ public interface JpaTransactionManager extends TransactionManager {
*/
Query query(String sqlString);
/** Executes the work in a transaction with no retries and returns the result. */
<T> T transactNoRetry(Supplier<T> work);
/**
* Executes the work in a transaction at the given {@link TransactionIsolationLevel} with no
* retries and returns the result.
*/
<T> T transactNoRetry(Supplier<T> work, TransactionIsolationLevel isolationLevel);
/** Executes the work in a transaction with no retries. */
void transactNoRetry(Runnable work);
/**
* Executes the work in a transaction at the given {@link TransactionIsolationLevel} with no
* retries.
*/
void transactNoRetry(Runnable work, TransactionIsolationLevel isolationLevel);
/** Deletes the entity by its id, throws exception if the entity is not deleted. */
<T> void assertDelete(VKey<T> key);
@@ -103,7 +84,4 @@ public interface JpaTransactionManager extends TransactionManager {
/** Return the {@link TransactionIsolationLevel} used in the current transaction. */
TransactionIsolationLevel getCurrentTransactionIsolationLevel();
/** Asserts that the current transaction runs at the given level. */
void assertTransactionIsolationLevel(TransactionIsolationLevel expectedLevel);
}
@@ -15,11 +15,12 @@
package google.registry.persistence.transaction;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Throwables.throwIfUnchecked;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static com.google.common.collect.ImmutableMap.toImmutableMap;
import static com.google.common.collect.ImmutableSet.toImmutableSet;
import static google.registry.config.RegistryConfig.getHibernatePerTransactionIsolationEnabled;
import static google.registry.persistence.transaction.DatabaseException.tryWrapAndThrow;
import static google.registry.config.RegistryConfig.getHibernateAllowNestedTransactions;
import static google.registry.persistence.transaction.DatabaseException.throwIfSqlException;
import static google.registry.util.PreconditionsUtils.checkArgumentNotNull;
import static java.util.AbstractMap.SimpleEntry;
import static java.util.stream.Collectors.joining;
@@ -31,6 +32,7 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Streams;
import com.google.common.flogger.FluentLogger;
import com.google.common.flogger.StackSize;
import google.registry.model.ImmutableObject;
import google.registry.persistence.JpaRetries;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
@@ -52,7 +54,7 @@ import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Optional;
import java.util.Set;
import java.util.function.Supplier;
import java.util.concurrent.Callable;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import javax.annotation.Nullable;
@@ -76,6 +78,9 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private static final Retrier retrier = new Retrier(new SystemSleeper(), 3);
private static final String NESTED_TRANSACTION_MESSAGE =
"Nested transaction detected. Try refactoring to avoid nested transactions. If unachievable,"
+ " use reTransact() in nested transactions";
// EntityManagerFactory is thread safe.
private final EntityManagerFactory emf;
@@ -138,21 +143,23 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
}
@Override
public void assertTransactionIsolationLevel(TransactionIsolationLevel expectedLevel) {
assertInTransaction();
TransactionIsolationLevel currentLevel = getCurrentTransactionIsolationLevel();
if (currentLevel != expectedLevel) {
throw new IllegalStateException(
String.format(
"Current transaction isolation level (%s) is not as expected (%s)",
currentLevel, expectedLevel));
public <T> T reTransact(Callable<T> work) {
// This prevents inner transaction from retrying, thus avoiding a cascade retry effect.
if (inTransaction()) {
return transactNoRetry(work, null);
}
return retrier.callWithRetry(
() -> transactNoRetry(work, null), JpaRetries::isFailedTxnRetriable);
}
@Override
public <T> T transact(Supplier<T> work, TransactionIsolationLevel isolationLevel) {
// This prevents inner transaction from retrying, thus avoiding a cascade retry effect.
public <T> T transact(Callable<T> work, TransactionIsolationLevel isolationLevel) {
if (inTransaction()) {
if (!getHibernateAllowNestedTransactions()) {
throw new IllegalStateException(NESTED_TRANSACTION_MESSAGE);
}
logger.atWarning().withStackTrace(StackSize.MEDIUM).log(NESTED_TRANSACTION_MESSAGE);
// This prevents inner transaction from retrying, thus avoiding a cascade retry effect.
return transactNoRetry(work, isolationLevel);
}
return retrier.callWithRetry(
@@ -160,30 +167,32 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
}
@Override
public <T> T reTransact(Supplier<T> work) {
return transact(work);
}
@Override
public <T> T transact(Supplier<T> work) {
public <T> T transact(Callable<T> work) {
return transact(work, null);
}
@Override
public <T> T transactNoRetry(
Supplier<T> work, @Nullable TransactionIsolationLevel isolationLevel) {
Callable<T> work, @Nullable TransactionIsolationLevel isolationLevel) {
if (inTransaction()) {
if (isolationLevel != null && getHibernatePerTransactionIsolationEnabled()) {
TransactionIsolationLevel enclosingLevel = getCurrentTransactionIsolationLevel();
if (isolationLevel != enclosingLevel) {
throw new IllegalStateException(
String.format(
"Isolation level conflict detected in nested transactions.\n"
+ "Enclosing transaction: %s\nCurrent transaction: %s",
enclosingLevel, isolationLevel));
}
// This check will no longer be necessary when the transact() method always throws
// inside a nested transaction, as the only way to pass a non-null isolation level
// is by calling the transact() method (and its variants), which would have already
// thrown before calling transactNoRetry() when inside a nested transaction.
//
// For now, we still need it, so we don't accidentally call a nested transact() with an
// isolation level override. This buys us time to detect nested transact() calls and either
// remove them or change the call site to reTransact().
if (isolationLevel != null) {
throw new IllegalStateException(
"Transaction isolation level cannot be specified for nested transactions");
}
try {
return work.call();
} catch (Exception e) {
throwIfSqlException(e);
throwIfUnchecked(e);
throw new RuntimeException(e);
}
return work.get();
}
TransactionInfo txnInfo = transactionInfo.get();
txnInfo.entityManager = emf.createEntityManager();
@@ -191,43 +200,36 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
try {
txn.begin();
txnInfo.start(clock);
if (isolationLevel != null) {
if (getHibernatePerTransactionIsolationEnabled()) {
getEntityManager()
.createNativeQuery(
String.format("SET TRANSACTION ISOLATION LEVEL %s", isolationLevel.getMode()))
.executeUpdate();
logger.atInfo().log("Running transaction at %s", isolationLevel);
} else {
logger.atWarning().log(
"Per-transaction isolation level disabled, but %s was requested", isolationLevel);
}
if (isolationLevel != null && isolationLevel != getDefaultTransactionIsolationLevel()) {
getEntityManager()
.createNativeQuery(
String.format("SET TRANSACTION ISOLATION LEVEL %s", isolationLevel.getMode()))
.executeUpdate();
logger.atInfo().log(
"Overriding transaction isolation level from %s to %s",
getDefaultTransactionIsolationLevel(), isolationLevel);
}
T result = work.get();
T result = work.call();
txn.commit();
return result;
} catch (RuntimeException | Error e) {
// Error is unchecked!
} catch (Throwable e) {
// Catch a Throwable here so even Errors would lead to a rollback.
try {
txn.rollback();
logger.atWarning().log("Error during transaction; transaction rolled back.");
} catch (Throwable rollbackException) {
} catch (Exception rollbackException) {
logger.atSevere().withCause(rollbackException).log("Rollback failed; suppressing error.");
}
tryWrapAndThrow(e);
throw e;
throwIfSqlException(e);
throwIfUnchecked(e);
throw new RuntimeException(e);
} finally {
txnInfo.clear();
}
}
@Override
public <T> T transactNoRetry(Supplier<T> work) {
return transactNoRetry(work, null);
}
@Override
public void transact(Runnable work, TransactionIsolationLevel isolationLevel) {
public void transact(ThrowingRunnable work, TransactionIsolationLevel isolationLevel) {
transact(
() -> {
work.run();
@@ -237,28 +239,17 @@ public class JpaTransactionManagerImpl implements JpaTransactionManager {
}
@Override
public void reTransact(Runnable work) {
transact(work);
}
@Override
public void transact(Runnable work) {
public void transact(ThrowingRunnable work) {
transact(work, null);
}
@Override
public void transactNoRetry(Runnable work, TransactionIsolationLevel isolationLevel) {
transactNoRetry(
public void reTransact(ThrowingRunnable work) {
reTransact(
() -> {
work.run();
return null;
},
isolationLevel);
}
@Override
public void transactNoRetry(Runnable work) {
transactNoRetry(work, null);
});
}
@Override
@@ -22,7 +22,7 @@ import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.VKey;
import java.util.NoSuchElementException;
import java.util.Optional;
import java.util.function.Supplier;
import java.util.concurrent.Callable;
import java.util.stream.Stream;
import org.joda.time.DateTime;
@@ -48,51 +48,51 @@ public interface TransactionManager {
void assertInTransaction();
/** Executes the work in a transaction and returns the result. */
<T> T transact(Supplier<T> work);
<T> T transact(Callable<T> work);
/**
* Executes the work in a transaction at the given {@link TransactionIsolationLevel} and returns
* the result.
*/
<T> T transact(Supplier<T> work, TransactionIsolationLevel isolationLevel);
<T> T transact(Callable<T> work, TransactionIsolationLevel isolationLevel);
/**
* Executes the work in a (potentially wrapped) transaction and returns the result.
*
* <p>Calls to this method are typically going to be in inner functions, that are called either as
* top-level transactions themselves or are nested inside of larger transactions (e.g. a
* top-level transactions themselves or are nested inside larger transactions (e.g. a
* transactional flow). Invocations of reTransact must be vetted to occur in both situations and
* with such complexity that it is not trivial to refactor out the nested transaction calls. New
* code should be written in such a way as to avoid requiring reTransact in the first place.
*
* <p>In the future we will be enforcing that {@link #transact(Supplier)} calls be top-level only,
* <p>In the future we will be enforcing that {@link #transact(Callable)} calls be top-level only,
* with reTransact calls being the only ones that can potentially be an inner nested transaction
* (which is a noop). Note that, as this can be a nested inner exception, there is no overload
* provided to specify a (potentially conflicting) transaction isolation level.
*/
<T> T reTransact(Supplier<T> work);
<T> T reTransact(Callable<T> work);
/** Executes the work in a transaction. */
void transact(Runnable work);
void transact(ThrowingRunnable work);
/** Executes the work in a transaction at the given {@link TransactionIsolationLevel}. */
void transact(Runnable work, TransactionIsolationLevel isolationLevel);
void transact(ThrowingRunnable work, TransactionIsolationLevel isolationLevel);
/**
* Executes the work in a (potentially wrapped) transaction and returns the result.
*
* <p>Calls to this method are typically going to be in inner functions, that are called either as
* top-level transactions themselves or are nested inside of larger transactions (e.g. a
* top-level transactions themselves or are nested inside larger transactions (e.g. a
* transactional flow). Invocations of reTransact must be vetted to occur in both situations and
* with such complexity that it is not trivial to refactor out the nested transaction calls. New
* code should be written in such a way as to avoid requiring reTransact in the first place.
*
* <p>In the future we will be enforcing that {@link #transact(Runnable)} calls be top-level only,
* with reTransact calls being the only ones that can potentially be an inner nested transaction
* (which is a noop). Note that, as this can be a nested inner exception, there is no overload *
* provided to specify a (potentially conflicting) transaction isolation level.
* <p>In the future we will be enforcing that {@link #transact(ThrowingRunnable)} calls be
* top-level only, with reTransact calls being the only ones that can potentially be an inner
* nested transaction (which is a noop). Note that, as this can be a nested inner exception, there
* is no overload provided to specify a (potentially conflicting) transaction isolation level.
*/
void reTransact(Runnable work);
void reTransact(ThrowingRunnable work);
/** Returns the time associated with the start of this particular transaction attempt. */
DateTime getTransactionTime();
@@ -216,4 +216,15 @@ public interface TransactionManager {
/** Returns a QueryComposer which can be used to perform queries against the current database. */
<T> QueryComposer<T> createQueryComposer(Class<T> entity);
/**
* A runnable that allows for checked exceptions to be thrown.
*
* <p>This makes it easier to write lambdas without having to worry about wrapping and re-throwing
* checked excpetions as unchecked ones.
*/
@FunctionalInterface
interface ThrowingRunnable {
void run() throws Exception;
}
}
@@ -30,11 +30,13 @@ public @interface Action {
/** App Engine services supported by the request processor. */
enum Service {
BSA("bsa"),
DEFAULT("default"),
TOOLS("tools"),
BACKEND("backend"),
PUBAPI("pubapi");
private final String serviceId;
Service(String serviceId) {
@@ -15,6 +15,7 @@
package google.registry.request;
import com.google.common.net.MediaType;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletResponse;
import org.joda.time.DateTime;
@@ -51,4 +52,11 @@ public interface Response {
* @see HttpServletResponse#setDateHeader(String, long)
*/
void setDateHeader(String header, DateTime timestamp);
/**
* Adds a cookie to the response
*
* @see HttpServletResponse#addCookie(Cookie)
*/
void addCookie(Cookie cookie);
}
@@ -17,6 +17,7 @@ package google.registry.request;
import com.google.common.net.MediaType;
import java.io.IOException;
import javax.inject.Inject;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletResponse;
import org.joda.time.DateTime;
@@ -58,4 +59,9 @@ public final class ResponseImpl implements Response {
public void setDateHeader(String header, DateTime timestamp) {
rsp.setDateHeader(header, timestamp.getMillis());
}
@Override
public void addCookie(Cookie cookie) {
rsp.addCookie(cookie);
}
}
@@ -55,6 +55,9 @@ public class AuthModule {
@Qualifier
@interface RegularOidc {}
@Qualifier
@interface RegularOidcFallback {}
@Provides
@IapOidc
@Singleton
@@ -71,6 +74,14 @@ public class AuthModule {
return TokenVerifier.newBuilder().setAudience(clientId).setIssuer(REGULAR_ISSUER_URL).build();
}
@Provides
@RegularOidcFallback
@Singleton
TokenVerifier provideFallbackRegularTokenVerifier(
@Config("fallbackOauthClientId") String clientId) {
return TokenVerifier.newBuilder().setAudience(clientId).setIssuer(REGULAR_ISSUER_URL).build();
}
@Provides
@IapOidc
@Singleton
@@ -25,6 +25,7 @@ import google.registry.model.console.User;
import google.registry.model.console.UserDao;
import google.registry.request.auth.AuthModule.IapOidc;
import google.registry.request.auth.AuthModule.RegularOidc;
import google.registry.request.auth.AuthModule.RegularOidcFallback;
import google.registry.request.auth.AuthSettings.AuthLevel;
import java.util.Optional;
import javax.annotation.Nullable;
@@ -53,6 +54,8 @@ public abstract class OidcTokenAuthenticationMechanism implements Authentication
protected final TokenVerifier tokenVerifier;
protected final Optional<TokenVerifier> fallbackTokenVerifier;
protected final TokenExtractor tokenExtractor;
private final ImmutableSet<String> serviceAccountEmails;
@@ -60,9 +63,11 @@ public abstract class OidcTokenAuthenticationMechanism implements Authentication
protected OidcTokenAuthenticationMechanism(
ImmutableSet<String> serviceAccountEmails,
TokenVerifier tokenVerifier,
@Nullable TokenVerifier fallbackTokenVerifier,
TokenExtractor tokenExtractor) {
this.serviceAccountEmails = serviceAccountEmails;
this.tokenVerifier = tokenVerifier;
this.fallbackTokenVerifier = Optional.ofNullable(fallbackTokenVerifier);
this.tokenExtractor = tokenExtractor;
}
@@ -77,7 +82,7 @@ public abstract class OidcTokenAuthenticationMechanism implements Authentication
if (rawIdToken == null) {
return AuthResult.NOT_AUTHENTICATED;
}
JsonWebSignature token;
JsonWebSignature token = null;
try {
token = tokenVerifier.verify(rawIdToken);
} catch (Exception e) {
@@ -86,8 +91,25 @@ public abstract class OidcTokenAuthenticationMechanism implements Authentication
RegistryEnvironment.get().equals(RegistryEnvironment.PRODUCTION)
? "Raw token redacted in prod"
: rawIdToken);
return AuthResult.NOT_AUTHENTICATED;
}
if (token == null) {
if (fallbackTokenVerifier.isPresent()) {
try {
token = fallbackTokenVerifier.get().verify(rawIdToken);
} catch (Exception e) {
logger.atInfo().withCause(e).log(
"Failed OIDC fallback verification attempt:\n%s",
RegistryEnvironment.get().equals(RegistryEnvironment.PRODUCTION)
? "Raw token redacted in prod"
: rawIdToken);
return AuthResult.NOT_AUTHENTICATED;
}
} else {
return AuthResult.NOT_AUTHENTICATED;
}
}
String email = (String) token.getPayload().get("email");
if (email == null) {
logger.atWarning().log("No email address from the OIDC token:\n%s", token.getPayload());
@@ -141,7 +163,7 @@ public abstract class OidcTokenAuthenticationMechanism implements Authentication
@Config("allowedServiceAccountEmails") ImmutableSet<String> serviceAccountEmails,
@IapOidc TokenVerifier tokenVerifier,
@IapOidc TokenExtractor tokenExtractor) {
super(serviceAccountEmails, tokenVerifier, tokenExtractor);
super(serviceAccountEmails, tokenVerifier, null, tokenExtractor);
}
}
@@ -161,8 +183,9 @@ public abstract class OidcTokenAuthenticationMechanism implements Authentication
protected RegularOidcAuthenticationMechanism(
@Config("allowedServiceAccountEmails") ImmutableSet<String> serviceAccountEmails,
@RegularOidc TokenVerifier tokenVerifier,
@RegularOidcFallback TokenVerifier fallbackTokenVerifier,
@RegularOidc TokenExtractor tokenExtractor) {
super(serviceAccountEmails, tokenVerifier, tokenExtractor);
super(serviceAccountEmails, tokenVerifier, fallbackTokenVerifier, tokenExtractor);
}
}
}
@@ -34,7 +34,7 @@ import org.joda.time.Duration;
/** Helper class for generating and validate XSRF tokens. */
public final class XsrfTokenManager {
/** HTTP header used for transmitting XSRF tokens. */
/** HTTP header or cookie name used for transmitting XSRF tokens. */
public static final String X_CSRF_TOKEN = "X-CSRF-Token";
/** POST parameter used for transmitting XSRF tokens. */
@@ -38,9 +38,7 @@ public final class IdnLabelValidator {
public Optional<String> findValidIdnTableForTld(String label, String tldStr) {
String unicodeString = Idn.toUnicode(label);
Tld tld = Tld.get(tldStr); // uses the cache
ImmutableSet<IdnTableEnum> idnTablesForTld = tld.getIdnTables();
ImmutableSet<IdnTableEnum> idnTables =
idnTablesForTld.isEmpty() ? DEFAULT_IDN_TABLES : idnTablesForTld;
ImmutableSet<IdnTableEnum> idnTables = getIdnTablesForTld(tld);
for (IdnTableEnum idnTable : idnTables) {
if (idnTable.getTable().isValidLabel(unicodeString)) {
return Optional.of(idnTable.getTable().getName());
@@ -48,4 +46,10 @@ public final class IdnLabelValidator {
}
return Optional.empty();
}
/** Returns the names of the IDN tables supported by a {@code tld}. */
public ImmutableSet<IdnTableEnum> getIdnTablesForTld(Tld tld) {
ImmutableSet<IdnTableEnum> idnTablesForTld = tld.getIdnTables();
return idnTablesForTld.isEmpty() ? DEFAULT_IDN_TABLES : idnTablesForTld;
}
}
@@ -84,7 +84,7 @@ public final class IdnTable {
* Returns true if the given label is valid for this IDN table. A label is considered valid if all
* of its codepoints are in the IDN table.
*/
boolean isValidLabel(String label) {
public boolean isValidLabel(String label) {
final int length = label.length();
for (int i = 0; i < length; ) {
int codepoint = label.codePointAt(i);
@@ -170,6 +170,8 @@ public class ServiceConnection {
return RegistryConfig.getToolsServer();
case BACKEND:
return RegistryConfig.getBackendServer();
case BSA:
return RegistryConfig.getBsaServer();
case PUBAPI:
return RegistryConfig.getPubapiServer();
}
@@ -58,13 +58,25 @@ public class RefreshDnsForAllDomainsAction implements Runnable {
private static final FluentLogger logger = FluentLogger.forEnclosingClass();
/** The number of DNS updates to enqueue per transaction. */
private static final int DEFAULT_BATCH_SIZE = 250;
/**
* The default number of DNS updates it is safe to execute per minute.
*
* <p>This is mostly a guess based on existing system performance, but the point is to be on the
* safe side and not cause contention with ongoing DNS updates from clients.
*/
private static final int DEFAULT_REFRESH_QPS = 7;
private final Response response;
private final ImmutableSet<String> tlds;
// Recommended value for batch size is between 200 and 500
private final int batchSize;
private final int refreshQps;
private final Random random;
@Inject
@@ -72,10 +84,12 @@ public class RefreshDnsForAllDomainsAction implements Runnable {
Response response,
@Parameter(PARAM_TLDS) ImmutableSet<String> tlds,
@Parameter("batchSize") Optional<Integer> batchSize,
@Parameter("refreshQps") Optional<Integer> refreshQps,
Random random) {
this.response = response;
this.tlds = tlds;
this.batchSize = batchSize.orElse(DEFAULT_BATCH_SIZE);
this.refreshQps = refreshQps.orElse(DEFAULT_REFRESH_QPS);
this.random = random;
}
@@ -83,7 +97,7 @@ public class RefreshDnsForAllDomainsAction implements Runnable {
public void run() {
assertTldsExist(tlds);
checkArgument(batchSize > 0, "Must specify a positive number for batch size");
int smearMinutes = tm().transact(this::calculateSmearMinutes, TRANSACTION_REPEATABLE_READ);
Duration smear = tm().transact(this::calculateSmear, TRANSACTION_REPEATABLE_READ);
ImmutableList<String> domainsBatch;
@Nullable String lastInPreviousBatch = null;
@@ -91,17 +105,16 @@ public class RefreshDnsForAllDomainsAction implements Runnable {
Optional<String> lastInPreviousBatchOpt = Optional.ofNullable(lastInPreviousBatch);
domainsBatch =
tm().transact(
() -> refreshBatch(lastInPreviousBatchOpt, smearMinutes),
TRANSACTION_REPEATABLE_READ);
() -> refreshBatch(lastInPreviousBatchOpt, smear), TRANSACTION_REPEATABLE_READ);
lastInPreviousBatch = domainsBatch.isEmpty() ? null : getLast(domainsBatch);
} while (domainsBatch.size() == batchSize);
}
/**
* Calculates the number of smear minutes to enqueue refreshes so that the DNS queue does not get
* Calculates the smear duration to enqueue refreshes so that the DNS queue does not get
* overloaded.
*/
private int calculateSmearMinutes() {
private Duration calculateSmear() {
Long activeDomains =
tm().query(
"SELECT COUNT(*) FROM Domain WHERE tld IN (:tlds) AND deletionTime = :endOfTime",
@@ -109,7 +122,7 @@ public class RefreshDnsForAllDomainsAction implements Runnable {
.setParameter("tlds", tlds)
.setParameter("endOfTime", END_OF_TIME)
.getSingleResult();
return Math.max(activeDomains.intValue() / 1000, 1);
return Duration.standardSeconds(Math.max(activeDomains / refreshQps, 1));
}
private ImmutableList<String> getBatch(Optional<String> lastInPreviousBatch) {
@@ -127,11 +140,12 @@ public class RefreshDnsForAllDomainsAction implements Runnable {
}
@VisibleForTesting
ImmutableList<String> refreshBatch(Optional<String> lastInPreviousBatch, int smearMinutes) {
ImmutableList<String> refreshBatch(Optional<String> lastInPreviousBatch, Duration smear) {
ImmutableList<String> domainBatch = getBatch(lastInPreviousBatch);
try {
// Smear the task execution time over the next N minutes.
requestDomainDnsRefresh(domainBatch, Duration.standardMinutes(random.nextInt(smearMinutes)));
// Smear the task execution time over the next N seconds.
requestDomainDnsRefresh(
domainBatch, Duration.standardSeconds(random.nextInt((int) smear.getStandardSeconds())));
} catch (Throwable t) {
logger.atSevere().withCause(t).log("Error while enqueuing DNS refresh batch");
response.setStatus(HttpStatus.SC_OK);
@@ -81,4 +81,10 @@ public class ToolsServerModule {
static Optional<Integer> provideBatchSize(HttpServletRequest req) {
return extractOptionalIntParameter(req, "batchSize");
}
@Provides
@Parameter("refreshQps")
static Optional<Integer> provideRefreshQps(HttpServletRequest req) {
return extractOptionalIntParameter(req, "refreshQps");
}
}
@@ -0,0 +1,72 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.ui.server.console;
import static google.registry.request.Action.Method.GET;
import com.google.api.client.http.HttpStatusCodes;
import google.registry.model.console.User;
import google.registry.security.XsrfTokenManager;
import google.registry.ui.server.registrar.ConsoleApiParams;
import java.util.Arrays;
import java.util.Optional;
import javax.servlet.http.Cookie;
/** Base class for handling Console API requests */
public abstract class ConsoleApiAction implements Runnable {
protected ConsoleApiParams consoleApiParams;
public ConsoleApiAction(ConsoleApiParams consoleApiParams) {
this.consoleApiParams = consoleApiParams;
}
@Override
public final void run() {
// Shouldn't be even possible because of Auth annotations on the various implementing classes
if (!consoleApiParams.authResult().userAuthInfo().get().consoleUser().isPresent()) {
consoleApiParams.response().setStatus(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
return;
}
User user = consoleApiParams.authResult().userAuthInfo().get().consoleUser().get();
if (consoleApiParams.request().getMethod().equals(GET.toString())) {
getHandler(user);
} else {
if (verifyXSRF()) {
postHandler(user);
}
}
}
protected void postHandler(User user) {
throw new UnsupportedOperationException("Console API POST handler not implemented");
}
protected void getHandler(User user) {
throw new UnsupportedOperationException("Console API GET handler not implemented");
}
private boolean verifyXSRF() {
Optional<Cookie> maybeCookie =
Arrays.stream(consoleApiParams.request().getCookies())
.filter(c -> XsrfTokenManager.X_CSRF_TOKEN.equals(c.getName()))
.findFirst();
if (!maybeCookie.isPresent()
|| !consoleApiParams.xsrfTokenManager().validateToken(maybeCookie.get().getValue())) {
consoleApiParams.response().setStatus(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
return false;
}
return true;
}
}
@@ -21,12 +21,10 @@ import com.google.common.collect.ImmutableMap;
import google.registry.config.RegistryConfig.Config;
import google.registry.model.console.User;
import google.registry.request.Action;
import google.registry.request.Response;
import google.registry.request.auth.Auth;
import google.registry.request.auth.AuthResult;
import google.registry.request.auth.UserAuthInfo;
import google.registry.ui.server.registrar.JsonGetAction;
import google.registry.ui.server.registrar.ConsoleApiParams;
import javax.inject.Inject;
import javax.servlet.http.Cookie;
import org.json.JSONObject;
@Action(
@@ -34,12 +32,10 @@ import org.json.JSONObject;
path = ConsoleUserDataAction.PATH,
method = {GET},
auth = Auth.AUTH_PUBLIC_LOGGED_IN)
public class ConsoleUserDataAction implements JsonGetAction {
public class ConsoleUserDataAction extends ConsoleApiAction {
public static final String PATH = "/console-api/userdata";
private final AuthResult authResult;
private final Response response;
private final String productName;
private final String supportPhoneNumber;
private final String supportEmail;
@@ -47,14 +43,12 @@ public class ConsoleUserDataAction implements JsonGetAction {
@Inject
public ConsoleUserDataAction(
AuthResult authResult,
Response response,
ConsoleApiParams consoleApiParams,
@Config("productName") String productName,
@Config("supportEmail") String supportEmail,
@Config("supportPhoneNumber") String supportPhoneNumber,
@Config("technicalDocsUrl") String technicalDocsUrl) {
this.response = response;
this.authResult = authResult;
super(consoleApiParams);
this.productName = productName;
this.supportEmail = supportEmail;
this.supportPhoneNumber = supportPhoneNumber;
@@ -62,13 +56,15 @@ public class ConsoleUserDataAction implements JsonGetAction {
}
@Override
public void run() {
UserAuthInfo authInfo = authResult.userAuthInfo().get();
if (!authInfo.consoleUser().isPresent()) {
response.setStatus(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
return;
}
User user = authInfo.consoleUser().get();
protected void getHandler(User user) {
// As this is a first GET request we use it as an opportunity to set a XSRF cookie
// for angular to read - https://angular.io/guide/http-security-xsrf-protection
Cookie xsrfCookie =
new Cookie(
consoleApiParams.xsrfTokenManager().X_CSRF_TOKEN,
consoleApiParams.xsrfTokenManager().generateToken(user.getEmailAddress()));
xsrfCookie.setSecure(true);
consoleApiParams.response().addCookie(xsrfCookie);
JSONObject json =
new JSONObject(
@@ -90,7 +86,7 @@ public class ConsoleUserDataAction implements JsonGetAction {
// Is used by UI to construct a link to registry resources
"technicalDocsUrl", technicalDocsUrl));
response.setPayload(json.toString());
response.setStatus(HttpStatusCodes.STATUS_CODE_OK);
consoleApiParams.response().setPayload(json.toString());
consoleApiParams.response().setStatus(HttpStatusCodes.STATUS_CODE_OK);
}
}
@@ -0,0 +1,41 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.ui.server.registrar;
import com.google.auto.value.AutoValue;
import google.registry.request.Response;
import google.registry.request.auth.AuthResult;
import google.registry.security.XsrfTokenManager;
import javax.servlet.http.HttpServletRequest;
/** Groups necessary dependencies for Console API actions * */
@AutoValue
public abstract class ConsoleApiParams {
public static ConsoleApiParams create(
HttpServletRequest request,
Response response,
AuthResult authResult,
XsrfTokenManager xsrfTokenManager) {
return new AutoValue_ConsoleApiParams(request, response, authResult, xsrfTokenManager);
}
public abstract HttpServletRequest request();
public abstract Response response();
public abstract AuthResult authResult();
public abstract XsrfTokenManager xsrfTokenManager();
}
@@ -28,6 +28,10 @@ import google.registry.model.registrar.Registrar;
import google.registry.model.registrar.RegistrarPoc;
import google.registry.request.OptionalJsonPayload;
import google.registry.request.Parameter;
import google.registry.request.RequestScope;
import google.registry.request.Response;
import google.registry.request.auth.AuthResult;
import google.registry.security.XsrfTokenManager;
import java.util.Optional;
import javax.servlet.http.HttpServletRequest;
import org.joda.time.DateTime;
@@ -35,9 +39,18 @@ import org.joda.time.DateTime;
/** Dagger module for the Registrar Console parameters. */
@Module
public final class RegistrarConsoleModule {
static final String PARAM_CLIENT_ID = "clientId";
@Provides
@RequestScope
ConsoleApiParams provideConsoleApiParams(
HttpServletRequest request,
Response response,
AuthResult authResult,
XsrfTokenManager xsrfTokenManager) {
return ConsoleApiParams.create(request, response, authResult, xsrfTokenManager);
}
@Provides
@Parameter(PARAM_CLIENT_ID)
static Optional<String> provideOptionalClientId(HttpServletRequest req) {
@@ -47,6 +47,7 @@ import google.registry.request.auth.UserAuthInfo;
import google.registry.security.JsonResponseHelper;
import google.registry.tools.DomainLockUtils;
import google.registry.util.EmailMessage;
import google.registry.util.PasswordUtils.HashAlgorithm;
import java.net.URISyntaxException;
import java.util.Map;
import java.util.Optional;
@@ -126,6 +127,7 @@ public class RegistryLockPostAction implements Runnable, JsonActionRunner.JsonAc
.userAuthInfo()
.orElseThrow(() -> new ForbiddenException("User is not logged in"));
// TODO: Move this line to the transaction below during nested transaction refactoring.
String userEmail = verifyPasswordAndGetEmail(userAuthInfo, postInput);
tm().transact(
() -> {
@@ -208,6 +210,7 @@ public class RegistryLockPostAction implements Runnable, JsonActionRunner.JsonAc
throws RegistrarAccessDeniedException {
// Verify that the user can access the registrar, that the user has
// registry lock enabled, and that the user provided a correct password
Registrar registrar =
getRegistrarAndVerifyLockAccess(registrarAccessor, postInput.registrarId, false);
RegistrarPoc registrarPoc =
@@ -220,6 +223,19 @@ public class RegistryLockPostAction implements Runnable, JsonActionRunner.JsonAc
checkArgument(
registrarPoc.verifyRegistryLockPassword(postInput.password),
"Incorrect registry lock password for contact");
if (registrarPoc.getCurrentHashAlgorithm(postInput.password).orElse(null)
!= HashAlgorithm.SCRYPT) {
logger.atInfo().log("Rehashing existing registry lock password with Scrypt.");
tm().transact(
() -> {
tm().update(
tm().loadByEntity(registrarPoc)
.asBuilder()
.setAllowedToSetRegistryLockPassword(true)
.setRegistryLockPassword(postInput.password)
.build());
});
}
return registrarPoc
.getRegistryLockEmailAddress()
.orElseThrow(
@@ -38,6 +38,9 @@
<mapping-file>META-INF/orm.xml</mapping-file>
<class>google.registry.bsa.persistence.BsaDownload</class>
<class>google.registry.bsa.persistence.BsaLabel</class>
<class>google.registry.bsa.persistence.BsaDomainInUse</class>
<class>google.registry.model.billing.BillingCancellation</class>
<class>google.registry.model.billing.BillingEvent</class>
<class>google.registry.model.billing.BillingRecurrence</class>
@@ -0,0 +1,94 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.tldconfig.idn.IdnTableEnum.EXTENDED_LATIN;
import static google.registry.tldconfig.idn.IdnTableEnum.JA;
import static google.registry.tldconfig.idn.IdnTableEnum.UNCONFUSABLE_LATIN;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import google.registry.model.tld.Tld;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
@ExtendWith(MockitoExtension.class)
public class IdnCheckerTest {
@Mock Tld jaonly;
@Mock Tld jandelatin;
@Mock Tld strictlatin;
IdnChecker idnChecker;
@BeforeEach
void setup() {
idnChecker =
new IdnChecker(
ImmutableMap.of(
JA,
ImmutableSet.of(jandelatin, jaonly),
EXTENDED_LATIN,
ImmutableSet.of(jandelatin),
UNCONFUSABLE_LATIN,
ImmutableSet.of(strictlatin)));
}
@Test
void getAllValidIdns_allTlds() {
assertThat(idnChecker.getAllValidIdns("all"))
.containsExactly(EXTENDED_LATIN, JA, UNCONFUSABLE_LATIN);
}
@Test
void getAllValidIdns_notJa() {
assertThat(idnChecker.getAllValidIdns("à")).containsExactly(EXTENDED_LATIN, UNCONFUSABLE_LATIN);
}
@Test
void getAllValidIdns_extendedLatinOnly() {
assertThat(idnChecker.getAllValidIdns("á")).containsExactly(EXTENDED_LATIN);
}
@Test
void getAllValidIdns_jaOnly() {
assertThat(idnChecker.getAllValidIdns("")).containsExactly(JA);
}
@Test
void getAllValidIdns_none() {
assertThat(idnChecker.getAllValidIdns("д")).isEmpty();
}
@Test
void getSupportingTlds_singleTld_success() {
assertThat(idnChecker.getSupportingTlds(ImmutableSet.of("EXTENDED_LATIN")))
.containsExactly(jandelatin);
}
@Test
void getSupportingTlds_multiTld_success() {
assertThat(idnChecker.getSupportingTlds(ImmutableSet.of("JA")))
.containsExactly(jandelatin, jaonly);
}
@Test
void getForbiddingTlds_success() {
assertThat(idnChecker.getForbiddingTlds(ImmutableSet.of("JA"))).containsExactly(strictlatin);
}
}
@@ -0,0 +1,74 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static org.joda.time.DateTimeZone.UTC;
import static org.junit.jupiter.api.Assertions.assertThrows;
import google.registry.bsa.persistence.BsaDomainInUse.Reason;
import google.registry.persistence.transaction.DatabaseException;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaIntegrationWithCoverageExtension;
import google.registry.testing.FakeClock;
import org.joda.time.DateTime;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Unit tests for {@link BsaDomainInUse}. */
public class BsaDomainInUseTest {
protected FakeClock fakeClock = new FakeClock(DateTime.now(UTC));
@RegisterExtension
final JpaIntegrationWithCoverageExtension jpa =
new JpaTestExtensions.Builder().withClock(fakeClock).buildIntegrationWithCoverageExtension();
@Test
void persist() {
tm().transact(() -> tm().put(new BsaLabel("label", fakeClock.nowUtc())));
tm().transact(() -> tm().put(new BsaDomainInUse("label", "tld", Reason.REGISTERED)));
BsaDomainInUse persisted =
tm().transact(() -> tm().loadByKey(BsaDomainInUse.vKey("label", "tld")));
assertThat(persisted.label).isEqualTo("label");
assertThat(persisted.tld).isEqualTo("tld");
assertThat(persisted.reason).isEqualTo(Reason.REGISTERED);
}
@Test
void cascadeDeletion() {
tm().transact(() -> tm().put(new BsaLabel("label", fakeClock.nowUtc())));
tm().transact(() -> tm().put(new BsaDomainInUse("label", "tld", Reason.REGISTERED)));
assertThat(tm().transact(() -> tm().loadByKeyIfPresent(BsaDomainInUse.vKey("label", "tld"))))
.isPresent();
tm().transact(() -> tm().delete(BsaLabel.vKey("label")));
assertThat(tm().transact(() -> tm().loadByKeyIfPresent(BsaDomainInUse.vKey("label", "tld"))))
.isEmpty();
}
@Test
void insertDomainWithoutLabel() {
assertThat(
assertThrows(
DatabaseException.class,
() ->
tm().transact(
() -> tm().put(new BsaDomainInUse("label", "tld", Reason.REGISTERED)))))
.hasMessageThat()
.contains("violates foreign key constraint");
}
}
@@ -0,0 +1,66 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.bsa.BlockList.BLOCK;
import static google.registry.bsa.BlockList.BLOCK_PLUS;
import static google.registry.bsa.DownloadStage.DOWNLOAD;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static org.joda.time.DateTimeZone.UTC;
import com.google.common.collect.ImmutableMap;
import google.registry.bsa.BlockList;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaIntegrationWithCoverageExtension;
import google.registry.testing.FakeClock;
import org.joda.time.DateTime;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Unit test for {@link BsaDownload}. */
public class BsaDownloadTest {
protected FakeClock fakeClock = new FakeClock(DateTime.now(UTC));
@RegisterExtension
final JpaIntegrationWithCoverageExtension jpa =
new JpaTestExtensions.Builder().withClock(fakeClock).buildIntegrationWithCoverageExtension();
@Test
void saveJob() {
BsaDownload persisted = tm().transact(() -> tm().getEntityManager().merge(new BsaDownload()));
assertThat(persisted.jobId).isNotNull();
assertThat(persisted.creationTime.getTimestamp()).isEqualTo(fakeClock.nowUtc());
assertThat(persisted.stage).isEqualTo(DOWNLOAD);
}
@Test
void loadJobByKey() {
BsaDownload persisted = tm().transact(() -> tm().getEntityManager().merge(new BsaDownload()));
assertThat(tm().transact(() -> tm().loadByKey(BsaDownload.vKey(persisted.jobId))))
.isEqualTo(persisted);
}
@Test
void checksums() {
BsaDownload job = new BsaDownload();
assertThat(job.getChecksums()).isEmpty();
ImmutableMap<BlockList, String> checksums = ImmutableMap.of(BLOCK, "a", BLOCK_PLUS, "b");
job.setChecksums(checksums);
assertThat(job.getChecksums()).isEqualTo(checksums);
assertThat(job.blockListChecksums).isEqualTo("BLOCK=a,BLOCK_PLUS=b");
}
}
@@ -0,0 +1,44 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static org.joda.time.DateTimeZone.UTC;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaIntegrationWithCoverageExtension;
import google.registry.testing.FakeClock;
import org.joda.time.DateTime;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Unit tests for {@link BsaLabel}. */
public class BsaLabelTest {
protected FakeClock fakeClock = new FakeClock(DateTime.now(UTC));
@RegisterExtension
final JpaIntegrationWithCoverageExtension jpa =
new JpaTestExtensions.Builder().withClock(fakeClock).buildIntegrationWithCoverageExtension();
@Test
void persist() {
tm().transact(() -> tm().put(new BsaLabel("label", fakeClock.nowUtc())));
BsaLabel persisted = tm().transact(() -> tm().loadByKey(BsaLabel.vKey("label")));
assertThat(persisted.getLabel()).isEqualTo("label");
assertThat(persisted.creationTime).isEqualTo(fakeClock.nowUtc());
}
}
@@ -0,0 +1,210 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.bsa.persistence;
import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.bsa.DownloadStage.CHECKSUMS_NOT_MATCH;
import static google.registry.bsa.DownloadStage.DONE;
import static google.registry.bsa.DownloadStage.DOWNLOAD;
import static google.registry.bsa.DownloadStage.MAKE_DIFF;
import static google.registry.bsa.DownloadStage.NOP;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static org.joda.time.DateTimeZone.UTC;
import static org.joda.time.Duration.standardDays;
import static org.joda.time.Duration.standardMinutes;
import static org.joda.time.Duration.standardSeconds;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import google.registry.bsa.BlockList;
import google.registry.bsa.DownloadStage;
import google.registry.bsa.persistence.DownloadSchedule.CompletedJob;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaIntegrationWithCoverageExtension;
import google.registry.testing.FakeClock;
import java.util.Optional;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
/** Unit tests for {@link DownloadScheduler} */
public class DownloadSchedulerTest {
static final Duration DOWNLOAD_INTERVAL = standardMinutes(30);
static final Duration MAX_NOP_INTERVAL = standardDays(1);
protected FakeClock fakeClock = new FakeClock(DateTime.now(UTC));
@RegisterExtension
final JpaIntegrationWithCoverageExtension jpa =
new JpaTestExtensions.Builder().withClock(fakeClock).buildIntegrationWithCoverageExtension();
private DownloadScheduler scheduler;
@BeforeEach
void setup() {
scheduler = new DownloadScheduler(DOWNLOAD_INTERVAL, MAX_NOP_INTERVAL, fakeClock);
}
@AfterEach
void dbCheck() {
ImmutableSet<DownloadStage> terminalStages = ImmutableSet.of(DONE, NOP, CHECKSUMS_NOT_MATCH);
assertThat(
tm().transact(
() ->
tm().getEntityManager()
.createQuery("FROM BsaDownload", BsaDownload.class)
.getResultStream()
.filter(job -> !terminalStages.contains(job.getStage()))
.count()))
.isAtMost(1);
}
@Test
void firstJobEver() {
Optional<DownloadSchedule> scheduleOptional = scheduler.schedule();
assertThat(scheduleOptional).isPresent();
DownloadSchedule schedule = scheduleOptional.get();
assertThat(schedule.latestCompleted()).isEmpty();
assertThat(schedule.jobName()).isEqualTo(fakeClock.nowUtc().toString());
assertThat(schedule.stage()).isEqualTo(DownloadStage.DOWNLOAD);
assertThat(schedule.alwaysDownload()).isTrue();
}
@Test
void oneInProgressJob() {
BsaDownload inProgressJob = insertOneJobAndAdvanceClock(MAKE_DIFF);
Optional<DownloadSchedule> scheduleOptional = scheduler.schedule();
assertThat(scheduleOptional).isPresent();
DownloadSchedule schedule = scheduleOptional.get();
assertThat(schedule.jobId()).isEqualTo(inProgressJob.jobId);
assertThat(schedule.jobName()).isEqualTo(inProgressJob.getJobName());
assertThat(schedule.stage()).isEqualTo(MAKE_DIFF);
assertThat(schedule.latestCompleted()).isEmpty();
assertThat(schedule.alwaysDownload()).isTrue();
}
@Test
void oneInProgressJobOneCompletedJob() {
BsaDownload completed = insertOneJobAndAdvanceClock(DONE);
BsaDownload inProgressJob = insertOneJobAndAdvanceClock(MAKE_DIFF);
Optional<DownloadSchedule> scheduleOptional = scheduler.schedule();
assertThat(scheduleOptional).isPresent();
DownloadSchedule schedule = scheduleOptional.get();
assertThat(schedule.jobId()).isEqualTo(inProgressJob.jobId);
assertThat(schedule.jobName()).isEqualTo(inProgressJob.getJobName());
assertThat(schedule.stage()).isEqualTo(MAKE_DIFF);
assertThat(schedule.alwaysDownload()).isFalse();
assertThat(schedule.latestCompleted()).isPresent();
CompletedJob lastCompleted = schedule.latestCompleted().get();
assertThat(lastCompleted.jobName()).isEqualTo(completed.getJobName());
assertThat(lastCompleted.checksums()).isEqualTo(completed.getChecksums());
}
@Test
void doneJob_noNewSchedule() {
insertOneJobAndAdvanceClock(DONE);
assertThat(scheduler.schedule()).isEmpty();
}
@Test
void doneJob_newSchedule() {
BsaDownload completed = insertOneJobAndAdvanceClock(DONE);
fakeClock.advanceBy(DOWNLOAD_INTERVAL);
Optional<DownloadSchedule> scheduleOptional = scheduler.schedule();
assertThat(scheduleOptional).isPresent();
DownloadSchedule schedule = scheduleOptional.get();
assertThat(schedule.stage()).isEqualTo(DOWNLOAD);
assertThat(schedule.alwaysDownload()).isFalse();
assertThat(schedule.latestCompleted()).isPresent();
CompletedJob completedJob = schedule.latestCompleted().get();
assertThat(completedJob.jobName()).isEqualTo(completed.getJobName());
assertThat(completedJob.checksums()).isEqualTo(completedJob.checksums());
}
@Test
void doneJob_newSchedule_alwaysDownload() {
insertOneJobAndAdvanceClock(DONE);
fakeClock.advanceBy(MAX_NOP_INTERVAL);
Optional<DownloadSchedule> scheduleOptional = scheduler.schedule();
assertThat(scheduleOptional).isPresent();
DownloadSchedule schedule = scheduleOptional.get();
assertThat(schedule.alwaysDownload()).isTrue();
}
@Test
void doneJob_cronEarlyWithJitter_newSchedule() {
insertOneJobAndAdvanceClock(DONE);
fakeClock.advanceBy(DOWNLOAD_INTERVAL.minus(standardSeconds(5)));
assertThat(scheduler.schedule()).isPresent();
}
@Test
void doneJob_cronEarlyMoreThanJitter_newSchedule() {
insertOneJobAndAdvanceClock(DONE);
fakeClock.advanceBy(DOWNLOAD_INTERVAL.minus(standardSeconds(6)));
assertThat(scheduler.schedule()).isEmpty();
}
@Test
void loadRecentProcessedJobs_noneExists() {
assertThat(tm().transact(() -> scheduler.loadRecentProcessedJobs())).isEmpty();
}
@Test
void loadRecentProcessedJobs_nopJobsOnly() {
insertOneJobAndAdvanceClock(DownloadStage.NOP);
insertOneJobAndAdvanceClock(DownloadStage.CHECKSUMS_NOT_MATCH);
assertThat(tm().transact(() -> scheduler.loadRecentProcessedJobs())).isEmpty();
}
@Test
void loadRecentProcessedJobs_oneInProgressJob() {
BsaDownload job = insertOneJobAndAdvanceClock(MAKE_DIFF);
assertThat(tm().transact(() -> scheduler.loadRecentProcessedJobs())).containsExactly(job);
}
@Test
void loadRecentProcessedJobs_oneDoneJob() {
BsaDownload job = insertOneJobAndAdvanceClock(DONE);
assertThat(tm().transact(() -> scheduler.loadRecentProcessedJobs())).containsExactly(job);
}
@Test
void loadRecentProcessedJobs_multipleJobs() {
insertOneJobAndAdvanceClock(DownloadStage.DONE);
insertOneJobAndAdvanceClock(DownloadStage.DONE);
BsaDownload completed = insertOneJobAndAdvanceClock(DownloadStage.DONE);
insertOneJobAndAdvanceClock(DownloadStage.NOP);
insertOneJobAndAdvanceClock(DownloadStage.CHECKSUMS_NOT_MATCH);
BsaDownload inprogress = insertOneJobAndAdvanceClock(DownloadStage.APPLY_DIFF);
assertThat(tm().transact(() -> scheduler.loadRecentProcessedJobs()))
.containsExactly(inprogress, completed)
.inOrder();
}
private BsaDownload insertOneJobAndAdvanceClock(DownloadStage stage) {
BsaDownload job = new BsaDownload();
job.setStage(stage);
job.setChecksums(ImmutableMap.of(BlockList.BLOCK, "1", BlockList.BLOCK_PLUS, "2"));
tm().transact(() -> tm().insert(job));
fakeClock.advanceOneMilli();
return job;
}
}
@@ -31,7 +31,6 @@ import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.ImmutableSortedMap;
import com.google.common.testing.TestLogHandler;
import google.registry.config.RegistryConfig;
import google.registry.flows.certs.CertificateChecker;
import google.registry.model.eppcommon.Trid;
import google.registry.model.eppoutput.EppOutput.ResponseOrGreeting;
@@ -89,8 +88,8 @@ class FlowRunnerTest {
@Override
public ResponseOrGreeting run() {
tm().assertTransactionIsolationLevel(
isolationLevel.orElse(tm().getDefaultTransactionIsolationLevel()));
assertThat(tm().getCurrentTransactionIsolationLevel())
.isEqualTo(isolationLevel.orElse(tm().getDefaultTransactionIsolationLevel()));
return mock(EppResponse.class);
}
}
@@ -136,10 +135,8 @@ class FlowRunnerTest {
Optional.of(TransactionIsolationLevel.TRANSACTION_READ_UNCOMMITTED);
flowRunner.flowClass = TestTransactionalFlow.class;
flowRunner.flowProvider = () -> new TestTransactionalFlow(flowRunner.isolationLevelOverride);
if (RegistryConfig.getHibernatePerTransactionIsolationEnabled()) {
flowRunner.run(eppMetricBuilder);
assertThat(eppMetricBuilder.build().getCommandName()).hasValue("TestTransactional");
}
flowRunner.run(eppMetricBuilder);
assertThat(eppMetricBuilder.build().getCommandName()).hasValue("TestTransactional");
}
@Test
@@ -1072,6 +1072,30 @@ class DomainCheckFlowTest extends ResourceCheckFlowTestCase<DomainCheckFlow, Dom
ImmutableMap.of("RENEWPRICE", "11.00")));
}
@Test
void testFeeExtension_existingPremiumDomain_withNonPremiumRenewalBehavior_renewPriceOnly()
throws Exception {
createTld("example");
persistBillingRecurrenceForDomain(persistActiveDomain("rich.example"), NONPREMIUM, null);
setEppInput("domain_check_fee_premium_v06_renew_only.xml");
runFlowAssertResponse(
loadFile(
"domain_check_fee_response_domain_exists_v06_renew_only.xml",
ImmutableMap.of("RENEWPRICE", "11.00")));
}
@Test
void testFeeExtension_existingPremiumDomain_withNonPremiumRenewalBehavior_transferPriceOnly()
throws Exception {
createTld("example");
persistBillingRecurrenceForDomain(persistActiveDomain("rich.example"), NONPREMIUM, null);
setEppInput("domain_check_fee_premium_v06_transfer_only.xml");
runFlowAssertResponse(
loadFile(
"domain_check_fee_response_domain_exists_v06_transfer_only.xml",
ImmutableMap.of("RENEWPRICE", "11.00")));
}
@Test
void testFeeExtension_existingPremiumDomain_withSpecifiedRenewalBehavior() throws Exception {
createTld("example");
@@ -1205,6 +1229,18 @@ class DomainCheckFlowTest extends ResourceCheckFlowTestCase<DomainCheckFlow, Dom
runFlowAssertResponse(loadFile("domain_check_fee_premium_response_v12.xml"));
}
@Test
void testFeeExtension_premiumLabels_v12_specifiedPriceRenewal_renewPriceOnly() throws Exception {
createTld("example");
persistBillingRecurrenceForDomain(
persistActiveDomain("rich.example"), SPECIFIED, Money.of(USD, new BigDecimal("27.74")));
setEppInput("domain_check_fee_premium_v12_renew_only.xml");
runFlowAssertResponse(
loadFile(
"domain_check_fee_premium_response_v12_renew_only.xml",
ImmutableMap.of("RENEWPRICE", "27.74")));
}
@Test
void testFeeExtension_premiumLabels_doesNotApplyDefaultToken_v12() throws Exception {
createTld("example");
@@ -14,11 +14,15 @@
package google.registry.flows.session;
import static com.google.common.io.BaseEncoding.base64;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.deleteResource;
import static google.registry.testing.DatabaseHelper.loadRegistrar;
import static google.registry.testing.DatabaseHelper.persistResource;
import static google.registry.testing.EppExceptionSubject.assertAboutEppExceptions;
import static google.registry.util.PasswordUtils.HashAlgorithm.SCRYPT;
import static google.registry.util.PasswordUtils.HashAlgorithm.SHA256;
import static org.junit.jupiter.api.Assertions.assertThrows;
import com.google.common.collect.ImmutableMap;
@@ -38,6 +42,7 @@ import google.registry.model.eppoutput.EppOutput;
import google.registry.model.registrar.Registrar;
import google.registry.model.registrar.Registrar.State;
import google.registry.testing.DatabaseHelper;
import google.registry.util.PasswordUtils;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -186,6 +191,33 @@ public abstract class LoginFlowTestCase extends FlowTestCase<LoginFlow> {
doFailingTest("login_valid.xml", RegistrarAccountNotActiveException.class);
}
@Test
void testSuccess_sha256Password() throws Exception {
String password = "foo-BAR2";
tm().transact(
() -> {
// The salt is not exposed by Registrar (nor should it be), so we query it
// directly.
String encodedSalt =
tm().query("SELECT salt FROM Registrar WHERE registrarId = :id", String.class)
.setParameter("id", registrar.getRegistrarId())
.getSingleResult();
byte[] salt = base64().decode(encodedSalt);
String newHash = PasswordUtils.hashPassword(password, salt, SHA256);
// Set password directly, as the Java method would have used Scrypt.
tm().query("UPDATE Registrar SET passwordHash = :hash WHERE registrarId = :id")
.setParameter("id", registrar.getRegistrarId())
.setParameter("hash", newHash)
.executeUpdate();
});
assertThat(loadRegistrar("NewRegistrar").getCurrentHashAlgorithm(password).get())
.isEqualTo(SHA256);
doSuccessfulTest("login_valid.xml");
// Verifies that after successfully login, the password is re-hased with Scrypt.
assertThat(loadRegistrar("NewRegistrar").getCurrentHashAlgorithm(password).get())
.isEqualTo(SCRYPT);
}
@Test
void testFailure_incorrectPassword() {
persistResource(getRegistrarBuilder().setPassword("diff password").build());
@@ -138,6 +138,7 @@ public class GmailClientTest {
assertThat(attachment.getContentType()).startsWith(CSV_UTF_8.toString());
assertThat(attachment.getContentType()).endsWith("name=filename");
assertThat(attachment.getContent()).isEqualTo("foo,bar\nbaz,qux");
assertThat(attachment.getDisposition()).isEqualTo("attachment");
}
@Test
@@ -0,0 +1,36 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.module.bsa;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.junit.jupiter.api.Test;
class BsaServletTest {
private final HttpServletRequest req = mock(HttpServletRequest.class);
private final HttpServletResponse rsp = mock(HttpServletResponse.class);
@Test
void testService_unknownPath_returnsNotFound() throws Exception {
when(req.getMethod()).thenReturn("GET");
when(req.getRequestURI()).thenReturn("/lol");
new BsaServlet().service(req, rsp);
verify(rsp).sendError(404);
}
}
@@ -18,7 +18,7 @@ import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.persistence.transaction.DatabaseException.getSqlError;
import static google.registry.persistence.transaction.DatabaseException.getSqlExceptionDetails;
import static google.registry.persistence.transaction.DatabaseException.tryWrapAndThrow;
import static google.registry.persistence.transaction.DatabaseException.throwIfSqlException;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
@@ -99,13 +99,13 @@ public class DatabaseExceptionTest {
@Test
void tryWrapAndThrow_notSQLException() {
RuntimeException orig = new RuntimeException(new Exception());
tryWrapAndThrow(orig);
throwIfSqlException(orig);
}
@Test
void tryWrapAndThrow_hasSQLException() {
Throwable orig = new Throwable(new SQLException());
assertThrows(DatabaseException.class, () -> tryWrapAndThrow(orig));
assertThrows(DatabaseException.class, () -> throwIfSqlException(orig));
}
@Test
@@ -14,6 +14,7 @@
package google.registry.persistence.transaction;
import static com.google.common.base.Preconditions.checkState;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.persistence.PersistenceModule.TransactionIsolationLevel.TRANSACTION_READ_COMMITTED;
@@ -28,6 +29,7 @@ import static google.registry.testing.TestDataHelper.fileClassPath;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mockStatic;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
@@ -36,6 +38,7 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import google.registry.config.RegistryConfig;
import google.registry.model.ImmutableObject;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.VKey;
import google.registry.persistence.transaction.JpaTestExtensions.JpaUnitTestExtension;
import google.registry.testing.DatabaseHelper;
@@ -43,7 +46,6 @@ import google.registry.testing.FakeClock;
import java.io.Serializable;
import java.math.BigInteger;
import java.util.NoSuchElementException;
import java.util.function.Supplier;
import javax.persistence.Entity;
import javax.persistence.EntityManager;
import javax.persistence.Id;
@@ -53,6 +55,8 @@ import javax.persistence.PersistenceException;
import javax.persistence.RollbackException;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.junit.jupiter.api.function.Executable;
import org.mockito.MockedStatic;
/**
* Unit tests for SQL only APIs defined in {@link JpaTransactionManagerImpl}. Note that the tests
@@ -94,7 +98,7 @@ class JpaTransactionManagerImplTest {
insertPerson(10);
insertCompany("Foo");
insertCompany("Bar");
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
});
assertPersonCount(1);
assertPersonExist(10);
@@ -105,145 +109,98 @@ class JpaTransactionManagerImplTest {
@Test
void transact_setIsolationLevel() {
// If not specified, run at the default isolation level.
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
RegistryConfig.getHibernatePerTransactionIsolationEnabled()
? TRANSACTION_READ_UNCOMMITTED
: tm().getDefaultTransactionIsolationLevel());
return null;
},
() -> assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel()),
null);
tm().transact(
() -> assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED),
TRANSACTION_READ_UNCOMMITTED);
// Make sure that we can start a new transaction on the same thread with a different isolation
// level.
// Make sure that we can start a new transaction on the same thread at a different level.
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
RegistryConfig.getHibernatePerTransactionIsolationEnabled()
? TRANSACTION_REPEATABLE_READ
: tm().getDefaultTransactionIsolationLevel());
return null;
},
() -> assertTransactionIsolationLevel(TRANSACTION_REPEATABLE_READ),
TRANSACTION_REPEATABLE_READ);
}
@Test
void transact_nestedTransactions_perTransactionIsolationLevelEnabled() {
if (!RegistryConfig.getHibernatePerTransactionIsolationEnabled()) {
return;
}
// Nested transactions allowed (both at the default isolation level).
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
void transact_nestedTransactions_disabled() {
try (MockedStatic<RegistryConfig> config = mockStatic(RegistryConfig.class)) {
config.when(RegistryConfig::getHibernateAllowNestedTransactions).thenReturn(false);
// transact() not allowed in nested transactions.
IllegalStateException thrown =
assertThrows(
IllegalStateException.class,
() ->
tm().transact(
() -> {
assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
});
});
// Nested transactions allowed (enclosed transaction does not have an override, using the
// enclosing transaction's level).
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED);
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED);
});
},
TRANSACTION_READ_UNCOMMITTED);
// Nested transactions allowed (Both have the same override).
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(TRANSACTION_REPEATABLE_READ);
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(TRANSACTION_REPEATABLE_READ);
},
TRANSACTION_REPEATABLE_READ);
},
TRANSACTION_REPEATABLE_READ);
// Nested transactions disallowed (enclosed transaction has an override that conflicts from the
// default).
IllegalStateException e =
assertThrows(
IllegalStateException.class,
() ->
tm().transact(
() -> {
tm().transact(() -> {}, TRANSACTION_READ_COMMITTED);
}));
assertThat(e).hasMessageThat().contains("conflict detected");
// Nested transactions disallowed (conflicting overrides).
e =
assertThrows(
IllegalStateException.class,
() ->
tm().transact(
() -> {
tm().transact(() -> {}, TRANSACTION_READ_COMMITTED);
},
TRANSACTION_REPEATABLE_READ));
assertThat(e).hasMessageThat().contains("conflict detected");
tm().transact(() -> null);
}));
assertThat(thrown).hasMessageThat().contains("Nested transaction detected");
// reTransact() allowed in nested transactions.
tm().transact(
() -> {
assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().reTransact(
() ->
assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel()));
});
// reTransact() respects enclosing transaction's isolation level.
tm().transact(
() -> {
assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED);
tm().reTransact(
() -> assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED));
},
TRANSACTION_READ_UNCOMMITTED);
}
}
@Test
void transact_nestedTransactions_perTransactionIsolationLevelDisabled() {
if (RegistryConfig.getHibernatePerTransactionIsolationEnabled()) {
return;
void transact_nestedTransactions_enabled() {
try (MockedStatic<RegistryConfig> config = mockStatic(RegistryConfig.class)) {
config.when(RegistryConfig::getHibernateAllowNestedTransactions).thenReturn(true);
// transact() allowed in nested transactions.
tm().transact(
() -> {
assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().reTransact(
() ->
assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel()));
});
// transact() not allowed in nested transactions if isolation level is specified.
IllegalStateException thrown =
assertThrows(
IllegalStateException.class,
() ->
tm().transact(
() -> {
assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
tm().transact(() -> null, TRANSACTION_READ_COMMITTED);
}));
assertThat(thrown).hasMessageThat().contains("cannot be specified");
// reTransact() allowed in nested transactions.
tm().transact(
() -> {
assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().reTransact(
() ->
assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel()));
});
// reTransact() respects enclosing transaction's isolation level.
tm().transact(
() -> {
assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED);
tm().reTransact(
() -> assertTransactionIsolationLevel(TRANSACTION_READ_UNCOMMITTED));
},
TRANSACTION_READ_UNCOMMITTED);
}
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
});
});
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
});
},
TRANSACTION_READ_UNCOMMITTED);
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
},
TRANSACTION_READ_UNCOMMITTED);
});
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
},
TRANSACTION_READ_UNCOMMITTED);
},
TRANSACTION_READ_UNCOMMITTED);
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(tm().getDefaultTransactionIsolationLevel());
tm().transact(
() -> {
tm().assertTransactionIsolationLevel(
tm().getDefaultTransactionIsolationLevel());
},
TRANSACTION_READ_COMMITTED);
},
TRANSACTION_READ_UNCOMMITTED);
}
@Test
@@ -299,32 +256,55 @@ class JpaTransactionManagerImplTest {
OptimisticLockException.class,
() -> spyJpaTm.transact(() -> spyJpaTm.delete(theEntityKey)));
verify(spyJpaTm, times(3)).delete(theEntityKey);
Supplier<Runnable> supplier =
() -> {
Runnable work = () -> spyJpaTm.delete(theEntityKey);
work.run();
return null;
};
assertThrows(OptimisticLockException.class, () -> spyJpaTm.transact(supplier));
assertThrows(
OptimisticLockException.class,
() -> spyJpaTm.transact(() -> spyJpaTm.delete(theEntityKey)));
verify(spyJpaTm, times(6)).delete(theEntityKey);
}
@Test
void transactNoRetry_doesNotRetryOptimisticLockException() {
JpaTransactionManager spyJpaTm = spy(tm());
doThrow(OptimisticLockException.class).when(spyJpaTm).delete(any(VKey.class));
spyJpaTm.transactNoRetry(() -> spyJpaTm.insert(theEntity));
assertThrows(
OptimisticLockException.class,
() -> spyJpaTm.transactNoRetry(() -> spyJpaTm.delete(theEntityKey)));
verify(spyJpaTm, times(1)).delete(theEntityKey);
Supplier<Runnable> supplier =
void transactNoRetry_nested() {
JpaTransactionManagerImpl tm = (JpaTransactionManagerImpl) tm();
// Calling transactNoRetry() without an isolation level override inside a transaction is fine.
tm.transact(
() -> {
Runnable work = () -> spyJpaTm.delete(theEntityKey);
work.run();
tm.transactNoRetry(
() -> {
assertTransactionIsolationLevel(tm.getDefaultTransactionIsolationLevel());
return null;
},
null);
});
// Calling transactNoRetry() with an isolation level override inside a transaction is not
// allowed.
IllegalStateException thrown =
assertThrows(
IllegalStateException.class,
() -> tm.transact(() -> tm.transactNoRetry(() -> null, TRANSACTION_READ_UNCOMMITTED)));
assertThat(thrown).hasMessageThat().contains("cannot be specified");
}
@Test
void transactNoRetry_doesNotRetryOptimisticLockException() {
JpaTransactionManagerImpl spyJpaTm = spy((JpaTransactionManagerImpl) tm());
doThrow(OptimisticLockException.class).when(spyJpaTm).delete(any(VKey.class));
spyJpaTm.transactNoRetry(
() -> {
spyJpaTm.insert(theEntity);
return null;
};
assertThrows(OptimisticLockException.class, () -> spyJpaTm.transactNoRetry(supplier));
},
null);
Executable transaction =
() ->
spyJpaTm.transactNoRetry(
() -> {
spyJpaTm.delete(theEntityKey);
return null;
},
null);
assertThrows(OptimisticLockException.class, transaction);
verify(spyJpaTm, times(1)).delete(theEntityKey);
assertThrows(OptimisticLockException.class, transaction);
verify(spyJpaTm, times(2)).delete(theEntityKey);
}
@@ -338,13 +318,8 @@ class JpaTransactionManagerImplTest {
assertThrows(
RuntimeException.class, () -> spyJpaTm.transact(() -> spyJpaTm.delete(theEntityKey)));
verify(spyJpaTm, times(3)).delete(theEntityKey);
Supplier<Runnable> supplier =
() -> {
Runnable work = () -> spyJpaTm.delete(theEntityKey);
work.run();
return null;
};
assertThrows(RuntimeException.class, () -> spyJpaTm.transact(supplier));
assertThrows(
RuntimeException.class, () -> spyJpaTm.transact(() -> spyJpaTm.delete(theEntityKey)));
verify(spyJpaTm, times(6)).delete(theEntityKey);
}
@@ -740,20 +715,13 @@ class JpaTransactionManagerImplTest {
doThrow(OptimisticLockException.class).when(spyJpaTm).delete(any(VKey.class));
spyJpaTm.transact(() -> spyJpaTm.insert(theEntity));
Supplier<Runnable> supplier =
() -> {
Runnable work = () -> spyJpaTm.delete(theEntityKey);
work.run();
return null;
};
assertThrows(
OptimisticLockException.class,
() ->
spyJpaTm.transact(
() -> {
spyJpaTm.exists(theEntity);
spyJpaTm.transact(supplier);
spyJpaTm.transact(() -> spyJpaTm.delete(theEntityKey));
}));
verify(spyJpaTm, times(3)).exists(theEntity);
@@ -814,6 +782,16 @@ class JpaTransactionManagerImplTest {
assertCompanyCount(0);
}
private static void assertTransactionIsolationLevel(TransactionIsolationLevel expectedLevel) {
tm().assertInTransaction();
TransactionIsolationLevel currentLevel = tm().getCurrentTransactionIsolationLevel();
checkState(
currentLevel == expectedLevel,
"Current transaction isolation level (%s) is not as expected (%s)",
currentLevel,
expectedLevel);
}
private static int countTable(String tableName) {
return tm().transact(
() -> {
@@ -14,6 +14,9 @@
package google.registry.persistence.transaction;
import static com.google.common.base.Throwables.throwIfUnchecked;
import static google.registry.persistence.transaction.DatabaseException.throwIfSqlException;
import com.google.common.collect.ImmutableCollection;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
@@ -21,7 +24,7 @@ import google.registry.model.ImmutableObject;
import google.registry.persistence.PersistenceModule.TransactionIsolationLevel;
import google.registry.persistence.VKey;
import java.util.Optional;
import java.util.function.Supplier;
import java.util.concurrent.Callable;
import java.util.stream.Stream;
import javax.persistence.EntityManager;
import javax.persistence.Query;
@@ -60,11 +63,6 @@ public class ReplicaSimulatingJpaTransactionManager implements JpaTransactionMan
return delegate.getCurrentTransactionIsolationLevel();
}
@Override
public void assertTransactionIsolationLevel(TransactionIsolationLevel expectedLevel) {
delegate.assertTransactionIsolationLevel(expectedLevel);
}
@Override
public EntityManager getStandaloneEntityManager() {
return delegate.getStandaloneEntityManager();
@@ -101,9 +99,15 @@ public class ReplicaSimulatingJpaTransactionManager implements JpaTransactionMan
}
@Override
public <T> T transact(Supplier<T> work, TransactionIsolationLevel isolationLevel) {
if (delegate.inTransaction()) {
return work.get();
public <T> T transact(Callable<T> work, TransactionIsolationLevel isolationLevel) {
if (inTransaction()) {
try {
return work.call();
} catch (Exception e) {
throwIfSqlException(e);
throwIfUnchecked(e);
throw new RuntimeException(e);
}
}
return delegate.transact(
() -> {
@@ -111,33 +115,23 @@ public class ReplicaSimulatingJpaTransactionManager implements JpaTransactionMan
.getEntityManager()
.createNativeQuery("SET TRANSACTION READ ONLY")
.executeUpdate();
return work.get();
return work.call();
},
isolationLevel);
}
@Override
public <T> T reTransact(Supplier<T> work) {
public <T> T reTransact(Callable<T> work) {
return transact(work);
}
@Override
public <T> T transact(Supplier<T> work) {
public <T> T transact(Callable<T> work) {
return transact(work, null);
}
@Override
public <T> T transactNoRetry(Supplier<T> work, TransactionIsolationLevel isolationLevel) {
return transact(work, isolationLevel);
}
@Override
public <T> T transactNoRetry(Supplier<T> work) {
return transactNoRetry(work, null);
}
@Override
public void transact(Runnable work, TransactionIsolationLevel isolationLevel) {
public void transact(ThrowingRunnable work, TransactionIsolationLevel isolationLevel) {
transact(
() -> {
work.run();
@@ -147,25 +141,15 @@ public class ReplicaSimulatingJpaTransactionManager implements JpaTransactionMan
}
@Override
public void reTransact(Runnable work) {
public void reTransact(ThrowingRunnable work) {
transact(work);
}
@Override
public void transact(Runnable work) {
public void transact(ThrowingRunnable work) {
transact(work, null);
}
@Override
public void transactNoRetry(Runnable work, TransactionIsolationLevel isolationLevel) {
transact(work, isolationLevel);
}
@Override
public void transactNoRetry(Runnable work) {
transactNoRetry(work, null);
}
@Override
public DateTime getTransactionTime() {
return delegate.getTransactionTime();
@@ -19,7 +19,6 @@ import static com.google.common.truth.Truth.assertThat;
import static google.registry.request.auth.AuthModule.BEARER_PREFIX;
import static google.registry.request.auth.AuthModule.IAP_HEADER_NAME;
import static google.registry.testing.DatabaseHelper.insertInDb;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
@@ -70,7 +69,7 @@ public class OidcTokenAuthenticationMechanismTest {
private AuthResult authResult;
private OidcTokenAuthenticationMechanism authenticationMechanism =
new OidcTokenAuthenticationMechanism(serviceAccounts, tokenVerifier, e -> rawToken) {};
new OidcTokenAuthenticationMechanism(serviceAccounts, tokenVerifier, null, e -> rawToken) {};
@RegisterExtension
public final JpaTestExtensions.JpaUnitTestExtension jpaExtension =
@@ -78,7 +77,7 @@ public class OidcTokenAuthenticationMechanismTest {
@BeforeEach
void beforeEach() throws Exception {
when(tokenVerifier.verify(eq(rawToken))).thenReturn(jwt);
when(tokenVerifier.verify(rawToken)).thenReturn(jwt);
payload.setEmail(email);
payload.setSubject(gaiaId);
insertInDb(user);
@@ -98,18 +97,30 @@ public class OidcTokenAuthenticationMechanismTest {
@Test
void testAuthenticate_noTokenFromRequest() {
authenticationMechanism =
new OidcTokenAuthenticationMechanism(serviceAccounts, tokenVerifier, e -> null) {};
new OidcTokenAuthenticationMechanism(serviceAccounts, tokenVerifier, null, e -> null) {};
authResult = authenticationMechanism.authenticate(request);
assertThat(authResult).isEqualTo(AuthResult.NOT_AUTHENTICATED);
}
@Test
void testAuthenticate_invalidToken() throws Exception {
when(tokenVerifier.verify(eq(rawToken))).thenThrow(new VerificationException("Bad token"));
when(tokenVerifier.verify(rawToken)).thenThrow(new VerificationException("Bad token"));
authResult = authenticationMechanism.authenticate(request);
assertThat(authResult).isEqualTo(AuthResult.NOT_AUTHENTICATED);
}
@Test
void testAuthenticate_fallbackVerifier() throws Exception {
TokenVerifier fallbackVerifier = mock(TokenVerifier.class);
when(tokenVerifier.verify(rawToken)).thenThrow(new VerificationException("Bad token"));
when(fallbackVerifier.verify(rawToken)).thenReturn(jwt);
authenticationMechanism =
new OidcTokenAuthenticationMechanism(
serviceAccounts, tokenVerifier, fallbackVerifier, e -> rawToken) {};
authResult = authenticationMechanism.authenticate(request);
assertThat(authResult.isAuthenticated()).isEqualTo(true);
}
@Test
void testAuthenticate_noEmailAddress() throws Exception {
payload.setEmail(null);
@@ -223,5 +234,12 @@ public class OidcTokenAuthenticationMechanismTest {
String provideOauthClientId() {
return "client-id";
}
@Provides
@Singleton
@Config("fallbackOauthClientId")
String provideFallbackOauthClientId() {
return "fallback-client-id";
}
}
}
@@ -16,6 +16,9 @@ package google.registry.schema.integration;
import static com.google.common.truth.Truth.assert_;
import google.registry.bsa.persistence.BsaDomainInUseTest;
import google.registry.bsa.persistence.BsaDownloadTest;
import google.registry.bsa.persistence.BsaLabelTest;
import google.registry.model.billing.BillingBaseTest;
import google.registry.model.common.CursorTest;
import google.registry.model.common.DnsRefreshRequestTest;
@@ -82,6 +85,9 @@ import org.junit.runner.RunWith;
BeforeSuiteTest.class,
AllocationTokenTest.class,
BillingBaseTest.class,
BsaDomainInUseTest.class,
BsaDownloadTest.class,
BsaLabelTest.class,
BulkPricingPackageTest.class,
ClaimsListDaoTest.class,
ContactHistoryTest.class,
@@ -0,0 +1,46 @@
// Copyright 2023 The Nomulus Authors. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package google.registry.testing;
import static org.mockito.Mockito.mock;
import com.google.appengine.api.users.UserService;
import google.registry.request.auth.AuthResult;
import google.registry.request.auth.UserAuthInfo;
import google.registry.security.XsrfTokenManager;
import google.registry.ui.server.registrar.ConsoleApiParams;
import java.util.Optional;
import javax.servlet.http.HttpServletRequest;
import org.joda.time.DateTime;
public final class FakeConsoleApiParams {
public static ConsoleApiParams get(Optional<AuthResult> maybeAuthResult) {
AuthResult authResult =
maybeAuthResult.orElseGet(
() ->
AuthResult.createUser(
UserAuthInfo.create(
new com.google.appengine.api.users.User(
"JohnDoe@theregistrar.com", "theregistrar.com"),
false)));
return ConsoleApiParams.create(
mock(HttpServletRequest.class),
new FakeResponse(),
authResult,
new XsrfTokenManager(
new FakeClock(DateTime.parse("2020-02-02T01:23:45Z")), mock(UserService.class)));
}
}
@@ -22,8 +22,10 @@ import static java.util.Collections.unmodifiableMap;
import com.google.common.base.Throwables;
import com.google.common.net.MediaType;
import google.registry.request.Response;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
import javax.servlet.http.Cookie;
import org.joda.time.DateTime;
/** Fake implementation of {@link Response} for testing. */
@@ -36,6 +38,8 @@ public final class FakeResponse implements Response {
private boolean wasMutuallyExclusiveResponseSet;
private String lastResponseStackTrace;
private ArrayList<Cookie> cookies = new ArrayList<>();
public int getStatus() {
return status;
}
@@ -83,6 +87,15 @@ public final class FakeResponse implements Response {
headers.put(checkNotNull(header), checkNotNull(timestamp));
}
@Override
public void addCookie(Cookie cookie) {
cookies.add(cookie);
}
public ArrayList<Cookie> getCookies() {
return cookies;
}
private void checkResponsePerformedOnce() {
checkState(
!wasMutuallyExclusiveResponseSet,
@@ -14,11 +14,14 @@
package google.registry.tldconfig.idn;
import static com.google.common.truth.Truth.assertThat;
import static com.google.common.truth.Truth8.assertThat;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.persistResource;
import com.google.common.collect.ImmutableSet;
import google.registry.model.tld.Tld;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaIntegrationTestExtension;
import org.junit.jupiter.api.Test;
@@ -118,4 +121,24 @@ class IdnLabelValidatorTest {
// Extended Latin shouldn't include Japanese characters
assertThat(idnLabelValidator.findValidIdnTableForTld("みんな", "tld")).isEmpty();
}
@Test
void testGetIdnTablesForTld_custom() {
persistResource(
createTld("tld")
.asBuilder()
.setIdnTables(ImmutableSet.of(IdnTableEnum.EXTENDED_LATIN))
.build());
Tld tld = tm().transact(() -> tm().loadByKey(Tld.createVKey("tld")));
assertThat(idnLabelValidator.getIdnTablesForTld(tld))
.containsExactly(IdnTableEnum.EXTENDED_LATIN);
}
@Test
void testGetIdnTablesForTld_default() {
persistResource(createTld("tld").asBuilder().build());
Tld tld = tm().transact(() -> tm().loadByKey(Tld.createVKey("tld")));
assertThat(idnLabelValidator.getIdnTablesForTld(tld))
.containsExactly(IdnTableEnum.EXTENDED_LATIN, IdnTableEnum.JA);
}
}
@@ -16,6 +16,7 @@ package google.registry.tools;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.model.EntityYamlUtils.createObjectMapper;
import static google.registry.model.domain.token.AllocationToken.TokenType.DEFAULT_PROMO;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.persistPremiumList;
import static google.registry.testing.DatabaseHelper.persistResource;
@@ -47,6 +48,8 @@ import google.registry.model.tld.label.PremiumListDao;
import java.io.File;
import java.util.logging.Logger;
import org.joda.money.Money;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
@@ -98,6 +101,20 @@ public class ConfigureTldCommandTest extends CommandTestCase<ConfigureTldCommand
assertThat(updatedTld.getCreateBillingCost()).isEqualTo(Money.of(USD, 25));
testTldConfiguredSuccessfully(updatedTld, "tld.yaml");
assertThat(updatedTld.getBreakglassMode()).isFalse();
assertThat(tld.getBsaEnrollStartTime()).isNull();
}
@Test
void testSuccess_updateTld_bsaTimeUnaffected() throws Exception {
Tld tld = createTld("tld");
DateTime bsaStartTime = DateTime.now(DateTimeZone.UTC);
tm().transact(() -> tm().put(tld.asBuilder().setBsaEnrollStartTime(bsaStartTime).build()));
File tldFile = tmpDir.resolve("tld.yaml").toFile();
Files.asCharSink(tldFile, UTF_8).write(loadFile(getClass(), "tld.yaml"));
runCommandForced("--input=" + tldFile);
// TODO(11/30/2023): uncomment below two lines
// Tld updatedTld = Tld.get("tld");
// assertThat(tld.getBsaEnrollStartTime()).isEqualTo(bsaStartTime);
}
@Test
@@ -35,6 +35,7 @@ import google.registry.testing.FakeResponse;
import java.util.Optional;
import java.util.Random;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -55,7 +56,7 @@ public class RefreshDnsForAllDomainsActionTest {
createTld("bar");
action =
new RefreshDnsForAllDomainsAction(
response, ImmutableSet.of("bar"), Optional.of(10), new Random());
response, ImmutableSet.of("bar"), Optional.of(10), Optional.empty(), new Random());
}
@Test
@@ -74,9 +75,9 @@ public class RefreshDnsForAllDomainsActionTest {
// Set batch size to 1 since each batch will be enqueud at the same time
action =
new RefreshDnsForAllDomainsAction(
response, ImmutableSet.of("bar"), Optional.of(1), new Random());
tm().transact(() -> action.refreshBatch(Optional.empty(), 1000));
tm().transact(() -> action.refreshBatch(Optional.empty(), 1000));
response, ImmutableSet.of("bar"), Optional.of(1), Optional.of(7), new Random());
tm().transact(() -> action.refreshBatch(Optional.empty(), Duration.standardMinutes(1000)));
tm().transact(() -> action.refreshBatch(Optional.empty(), Duration.standardMinutes(1000)));
ImmutableList<DnsRefreshRequest> refreshRequests =
tm().transact(
() ->
@@ -19,6 +19,7 @@ import static com.google.common.truth.Truth.assertWithMessage;
import com.google.common.collect.ImmutableSet;
import google.registry.request.Action;
import google.registry.request.JsonActionRunner;
import google.registry.ui.server.console.ConsoleApiAction;
import google.registry.ui.server.registrar.HtmlAction;
import google.registry.ui.server.registrar.JsonGetAction;
import io.github.classgraph.ClassGraph;
@@ -34,6 +35,7 @@ final class ActionMembershipTest {
// 1. Extending HtmlAction to signal that we are serving an HTML page
// 2. Extending JsonAction to show that we are serving JSON POST requests
// 3. Extending JsonGetAction to serve JSON GET requests
// 4. Extending ConsoleApiAction to serve JSON requests
ImmutableSet.Builder<String> failingClasses = new ImmutableSet.Builder<>();
try (ScanResult scanResult =
new ClassGraph().enableAnnotationInfo().whitelistPackages("google.registry.ui").scan()) {
@@ -41,7 +43,8 @@ final class ActionMembershipTest {
.getClassesWithAnnotation(Action.class.getName())
.forEach(
classInfo -> {
if (!classInfo.extendsSuperclass(HtmlAction.class.getName())
if (!classInfo.extendsSuperclass(ConsoleApiAction.class.getName())
&& !classInfo.extendsSuperclass(HtmlAction.class.getName())
&& !classInfo.implementsInterface(JsonActionRunner.JsonAction.class.getName())
&& !classInfo.implementsInterface(JsonGetAction.class.getName())) {
failingClasses.add(classInfo.getName());
@@ -14,7 +14,9 @@
package google.registry.ui.server.console;
import static com.google.common.collect.ImmutableList.toImmutableList;
import static com.google.common.truth.Truth.assertThat;
import static org.mockito.Mockito.when;
import com.google.api.client.http.HttpStatusCodes;
import com.google.gson.Gson;
@@ -22,12 +24,18 @@ import google.registry.model.console.GlobalRole;
import google.registry.model.console.User;
import google.registry.model.console.UserRoles;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.request.Action;
import google.registry.request.RequestModule;
import google.registry.request.auth.AuthResult;
import google.registry.request.auth.UserAuthInfo;
import google.registry.testing.FakeConsoleApiParams;
import google.registry.testing.FakeResponse;
import google.registry.ui.server.registrar.ConsoleApiParams;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Map;
import java.util.Optional;
import javax.servlet.http.Cookie;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
@@ -35,12 +43,31 @@ import org.junit.jupiter.api.extension.RegisterExtension;
class ConsoleUserDataActionTest {
private static final Gson GSON = RequestModule.provideGson();
private FakeResponse response = new FakeResponse();
private ConsoleApiParams consoleApiParams;
@RegisterExtension
final JpaTestExtensions.JpaIntegrationTestExtension jpa =
new JpaTestExtensions.Builder().buildIntegrationTestExtension();
@Test
void testSuccess_hasXSRFCookie() throws IOException {
User user =
new User.Builder()
.setEmailAddress("email@email.com")
.setUserRoles(new UserRoles.Builder().setGlobalRole(GlobalRole.FTE).build())
.build();
AuthResult authResult = AuthResult.createUser(UserAuthInfo.create(user));
ConsoleUserDataAction action =
createAction(
Optional.of(FakeConsoleApiParams.get(Optional.of(authResult))), Action.Method.GET);
action.run();
ArrayList<Cookie> cookies = ((FakeResponse) consoleApiParams.response()).getCookies();
assertThat(cookies.stream().map(cookie -> cookie.getName()).collect(toImmutableList()))
.containsExactly("X-CSRF-Token");
}
@Test
void testSuccess_getContactInfo() throws IOException {
User user =
@@ -49,10 +76,15 @@ class ConsoleUserDataActionTest {
.setUserRoles(new UserRoles.Builder().setGlobalRole(GlobalRole.FTE).build())
.build();
ConsoleUserDataAction action = createAction(AuthResult.createUser(UserAuthInfo.create(user)));
AuthResult authResult = AuthResult.createUser(UserAuthInfo.create(user));
ConsoleUserDataAction action =
createAction(
Optional.of(FakeConsoleApiParams.get(Optional.of(authResult))), Action.Method.GET);
action.run();
assertThat(response.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_OK);
Map jsonObject = GSON.fromJson(response.getPayload(), Map.class);
assertThat(((FakeResponse) consoleApiParams.response()).getStatus())
.isEqualTo(HttpStatusCodes.STATUS_CODE_OK);
Map jsonObject =
GSON.fromJson(((FakeResponse) consoleApiParams.response()).getPayload(), Map.class);
assertThat(jsonObject)
.containsExactly(
"isAdmin",
@@ -71,19 +103,18 @@ class ConsoleUserDataActionTest {
@Test
void testFailure_notAConsoleUser() throws IOException {
ConsoleUserDataAction action =
createAction(
AuthResult.createUser(
UserAuthInfo.create(
new com.google.appengine.api.users.User(
"JohnDoe@theregistrar.com", "theregistrar.com"),
false)));
ConsoleUserDataAction action = createAction(Optional.empty(), Action.Method.GET);
action.run();
assertThat(response.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
assertThat(((FakeResponse) consoleApiParams.response()).getStatus())
.isEqualTo(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
}
private ConsoleUserDataAction createAction(AuthResult authResult) throws IOException {
private ConsoleUserDataAction createAction(
Optional<ConsoleApiParams> maybeConsoleApiParams, Action.Method method) throws IOException {
consoleApiParams =
maybeConsoleApiParams.orElseGet(() -> FakeConsoleApiParams.get(Optional.empty()));
when(consoleApiParams.request().getMethod()).thenReturn(method.toString());
return new ConsoleUserDataAction(
authResult, response, "Nomulus", "support@example.com", "+1 (212) 867 5309", "test");
consoleApiParams, "Nomulus", "support@example.com", "+1 (212) 867 5309", "test");
}
}
@@ -15,8 +15,10 @@
package google.registry.ui.server.registrar;
import static com.google.common.collect.ImmutableSetMultimap.toImmutableSetMultimap;
import static com.google.common.io.BaseEncoding.base64;
import static com.google.common.truth.Truth.assertThat;
import static google.registry.model.EppResourceUtils.loadByForeignKey;
import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
import static google.registry.testing.DatabaseHelper.createTld;
import static google.registry.testing.DatabaseHelper.loadRegistrar;
import static google.registry.testing.DatabaseHelper.persistResource;
@@ -25,6 +27,8 @@ import static google.registry.testing.SqlHelper.getRegistryLockByVerificationCod
import static google.registry.testing.SqlHelper.saveRegistryLock;
import static google.registry.tools.LockOrUnlockDomainCommand.REGISTRY_LOCK_STATUSES;
import static google.registry.ui.server.registrar.RegistryLockGetActionTest.userFromRegistrarPoc;
import static google.registry.util.PasswordUtils.HashAlgorithm.SCRYPT;
import static google.registry.util.PasswordUtils.HashAlgorithm.SHA256;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;
@@ -38,6 +42,9 @@ import google.registry.model.console.RegistrarRole;
import google.registry.model.console.UserRoles;
import google.registry.model.domain.Domain;
import google.registry.model.domain.RegistryLock;
import google.registry.model.registrar.RegistrarPoc;
import google.registry.model.registrar.RegistrarPoc.RegistrarPocId;
import google.registry.persistence.VKey;
import google.registry.persistence.transaction.JpaTestExtensions;
import google.registry.persistence.transaction.JpaTestExtensions.JpaIntegrationTestExtension;
import google.registry.persistence.transaction.JpaTransactionManagerExtension;
@@ -54,6 +61,7 @@ import google.registry.testing.DeterministicStringGenerator;
import google.registry.testing.FakeClock;
import google.registry.tools.DomainLockUtils;
import google.registry.util.EmailMessage;
import google.registry.util.PasswordUtils;
import google.registry.util.StringGenerator.Alphabets;
import java.util.Map;
import java.util.Optional;
@@ -123,6 +131,48 @@ final class RegistryLockPostActionTest {
assertSuccess(response, "lock", "Marla.Singer.RegistryLock@crr.com");
}
@Test
void testSuccess_lock_sha256Password() throws Exception {
tm().transact(
() -> {
// The salt is not exposed by RegistrarPoc (nor should it be), so we query
// it directly.
String encodedSalt =
tm().query(
"SELECT registryLockPasswordSalt FROM RegistrarPoc "
+ "WHERE emailAddress = :email "
+ "AND registrarId = :registrarId",
String.class)
.setParameter("email", "Marla.Singer@crr.com")
.setParameter("registrarId", "TheRegistrar")
.getSingleResult();
byte[] salt = base64().decode(encodedSalt);
String newHash = PasswordUtils.hashPassword("hi", salt, SHA256);
// Set password directly, as the Java method would have used Scrypt.
tm().query("UPDATE RegistrarPoc SET registryLockPasswordHash = :hash")
.setParameter("hash", newHash)
.executeUpdate();
});
RegistrarPoc registrarPoc =
tm().transact(
() ->
tm().loadByKey(
VKey.create(
RegistrarPoc.class,
new RegistrarPocId("Marla.Singer@crr.com", "TheRegistrar"))));
assertThat(registrarPoc.getCurrentHashAlgorithm("hi").get()).isEqualTo(SHA256);
Map<String, ?> response = action.handleJsonRequest(lockRequest());
RegistrarPoc updatedRegistrarPoc =
tm().transact(
() ->
tm().loadByKey(
VKey.create(
RegistrarPoc.class,
new RegistrarPocId("Marla.Singer@crr.com", "TheRegistrar"))));
assertThat(updatedRegistrarPoc.getCurrentHashAlgorithm("hi").get()).isEqualTo(SCRYPT);
assertSuccess(response, "lock", "Marla.Singer.RegistryLock@crr.com");
}
@Test
void testSuccess_unlock() throws Exception {
saveRegistryLock(createLock().asBuilder().setLockCompletionTime(clock.nowUtc()).build());
@@ -0,0 +1,33 @@
domain_check_fee_premium_response_v12.xml<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
<response>
<result code="1000">
<msg>Command completed successfully</msg>
</result>
<resData>
<domain:chkData xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<domain:cd>
<domain:name avail="false">rich.example</domain:name>
<domain:reason>In use</domain:reason>
</domain:cd>
</domain:chkData>
</resData>
<extension>
<fee:chkData xmlns:fee="urn:ietf:params:xml:ns:fee-0.12"
xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<fee:cd>
<fee:object>
<domain:name>rich.example</domain:name>
</fee:object>
<fee:command name="renew">
<fee:period unit="y">1</fee:period>
<fee:fee description="renew">%RENEWPRICE%</fee:fee>
</fee:command>
</fee:cd>
</fee:chkData>
</extension>
<trID>
<clTRID>ABC-12345</clTRID>
<svTRID>server-trid</svTRID>
</trID>
</response>
</epp>
@@ -0,0 +1,18 @@
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
<command>
<check>
<domain:check xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<domain:name>rich.example</domain:name>
</domain:check>
</check>
<extension>
<fee:check xmlns:fee="urn:ietf:params:xml:ns:fee-0.6">
<fee:domain>
<fee:name>rich.example</fee:name>
<fee:command>renew</fee:command>
</fee:domain>
</fee:check>
</extension>
<clTRID>ABC-12345</clTRID>
</command>
</epp>
@@ -0,0 +1,18 @@
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
<command>
<check>
<domain:check xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<domain:name>rich.example</domain:name>
</domain:check>
</check>
<extension>
<fee:check xmlns:fee="urn:ietf:params:xml:ns:fee-0.6">
<fee:domain>
<fee:name>rich.example</fee:name>
<fee:command>transfer</fee:command>
</fee:domain>
</fee:check>
</extension>
<clTRID>ABC-12345</clTRID>
</command>
</epp>
@@ -0,0 +1,15 @@
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
<command>
<check>
<domain:check xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<domain:name>rich.example</domain:name>
</domain:check>
</check>
<extension>
<fee:check xmlns:fee="urn:ietf:params:xml:ns:fee-0.12">
<fee:command name="renew" />
</fee:check>
</extension>
<clTRID>ABC-12345</clTRID>
</command>
</epp>
@@ -0,0 +1,30 @@
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
<response>
<result code="1000">
<msg>Command completed successfully</msg>
</result>
<resData>
<domain:chkData xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<domain:cd>
<domain:name avail="0">rich.example</domain:name>
<domain:reason>In use</domain:reason>
</domain:cd>
</domain:chkData>
</resData>
<extension>
<fee:chkData xmlns:fee="urn:ietf:params:xml:ns:fee-0.6">
<fee:cd xmlns:fee="urn:ietf:params:xml:ns:fee-0.6">
<fee:name>rich.example</fee:name>
<fee:currency>USD</fee:currency>
<fee:command>renew</fee:command>
<fee:period unit="y">1</fee:period>
<fee:fee description="renew">%RENEWPRICE%</fee:fee>
</fee:cd>
</fee:chkData>
</extension>
<trID>
<clTRID>ABC-12345</clTRID>
<svTRID>server-trid</svTRID>
</trID>
</response>
</epp>
@@ -0,0 +1,30 @@
<epp xmlns="urn:ietf:params:xml:ns:epp-1.0">
<response>
<result code="1000">
<msg>Command completed successfully</msg>
</result>
<resData>
<domain:chkData xmlns:domain="urn:ietf:params:xml:ns:domain-1.0">
<domain:cd>
<domain:name avail="0">rich.example</domain:name>
<domain:reason>In use</domain:reason>
</domain:cd>
</domain:chkData>
</resData>
<extension>
<fee:chkData xmlns:fee="urn:ietf:params:xml:ns:fee-0.6">
<fee:cd xmlns:fee="urn:ietf:params:xml:ns:fee-0.6">
<fee:name>rich.example</fee:name>
<fee:currency>USD</fee:currency>
<fee:command>transfer</fee:command>
<fee:period unit="y">1</fee:period>
<fee:fee description="renew">%RENEWPRICE%</fee:fee>
</fee:cd>
</fee:chkData>
</extension>
<trID>
<clTRID>ABC-12345</clTRID>
<svTRID>server-trid</svTRID>
</trID>
</response>
</epp>
@@ -261,11 +261,11 @@ td.section {
</tr>
<tr>
<td class="property_name">generated on</td>
<td class="property_value">2023-11-02 18:26:18.901466</td>
<td class="property_value">2023-11-09 01:49:59.861801</td>
</tr>
<tr>
<td class="property_name">last flyway file</td>
<td id="lastFlywayFile" class="property_value">V149__add_bsa_domain_in_use_table.sql</td>
<td id="lastFlywayFile" class="property_value">V150__add_tld_bsa_enroll_date.sql</td>
</tr>
</tbody>
</table>
@@ -285,7 +285,7 @@ td.section {
generated on
</text>
<text text-anchor="start" x="3835.5" y="-10.8" font-family="Helvetica,sans-Serif" font-size="14.00">
2023-11-02 18:26:18.901466
2023-11-09 01:49:59.861801
</text>
<polygon fill="none" stroke="#888888" points="3748,-4 3748,-44 4013,-44 4013,-4 3748,-4" /> <!-- allocationtoken_a08ccbef -->
<g id="node1" class="node">
File diff suppressed because it is too large Load Diff
+1
View File
@@ -147,3 +147,4 @@ V146__last_update_time_via_epp.sql
V147__drop_gaia_id_from_user.sql
V148__add_bsa_download_and_label_tables.sql
V149__add_bsa_domain_in_use_table.sql
V150__add_tld_bsa_enroll_date.sql
@@ -0,0 +1,15 @@
-- Copyright 2023 The Nomulus Authors. All Rights Reserved.
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
ALTER TABLE public."Tld" ADD COLUMN bsa_enroll_start_time timestamptz;
@@ -85,6 +85,29 @@
primary key (billing_recurrence_id)
);
create table "BsaDomainInUse" (
label text not null,
tld text not null,
creation_time timestamptz not null,
reason text not null,
primary key (label, tld)
);
create table "BsaDownload" (
job_id bigserial not null,
block_list_checksums text not null,
creation_time timestamptz not null,
stage text not null,
update_timestamp timestamptz,
primary key (job_id)
);
create table "BsaLabel" (
label text not null,
creation_time timestamptz not null,
primary key (label)
);
create table "ClaimsEntry" (
revision_id int8 not null,
domain_label text not null,
@@ -785,6 +808,7 @@ create index IDXoqttafcywwdn41um6kwlt0n8b on "BillingRecurrence" (domain_repo_id
create index IDXp3usbtvk0v1m14i5tdp4xnxgc on "BillingRecurrence" (recurrence_end_time);
create index IDXp0pxi708hlu4n40qhbtihge8x on "BillingRecurrence" (recurrence_last_expansion);
create index IDXjny8wuot75b5e6p38r47wdawu on "BillingRecurrence" (recurrence_time_of_year);
create index IDXj874kw19bgdnkxo1rue45jwlw on "BsaDownload" (creation_time);
create index IDX3y752kr9uh4kh6uig54vemx0l on "Contact" (creation_time);
create index IDXtm415d6fe1rr35stm33s5mg18 on "Contact" (current_sponsor_registrar_id);
create index IDXn1f711wicdnooa2mqb7g1m55o on "Contact" (deletion_time);

Some files were not shown because too many files have changed in this diff Show More