Change default beam job region (#1937 )

For reasons that I cannot explain, the same expand recurring billing event pipeline would fail in us-east1 but succeed in us-central1. See: https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-02-09_14_52_24-162498476138221714;graphView=0?project=domain-registry https://pantheon.corp.google.com/dataflow/jobs/us-east1/2023-02-09_14_26_07-4564782062878841960;graphView=1?project=domain-registry Also improved how the accuracy of the metrics: It is observed that both counters are consistently higher for the same start and end times when running in dry run mode. There is no way to test for consistency when not running in dry run, for obviously reasons. I can make the recurrings in scope counter consistent by not updating it in a side-effect-causing transaction, but there is no way around the other counter. It can only be trusted when running in dry run mode, unfortunately.
Add a frontend endpoint for retrieving a domain in JSON form (#1916 )
2026-07-12 19:12:35 +00:00 · 2023-02-13 15:57:32 -05:00 · 2023-02-09 15:09:42 -05:00 · 2023-02-09 14:50:35 -05:00 · 2023-02-08 16:10:34 -05:00 · 2023-02-08 09:59:47 -05:00
35 changed files with 638 additions and 99 deletions
@@ -138,9 +138,15 @@ public class ExpandRecurringBillingEventsPipeline implements Serializable {
  private final boolean isDryRun;
  private final boolean advanceCursor;
  private final Counter recurringsInScopeCounter =
-      Metrics.counter("ExpandBilling", "RecurringsInScope");
-  private final Counter expandedOneTimeCounter =
-      Metrics.counter("ExpandBilling", "ExpandedOneTime");
+      Metrics.counter("ExpandBilling", "Recurrings in scope for expansion");
+  // Note that this counter is only accurate when running in dry run mode. Because SQL persistence
+  // is a side effect and not idempotent, a transaction to save OneTimes could be successful but the
+  // transform that contains it could be still be retried, rolling back the counter increment. The
+  // same transform, when retried, would skip the already expanded OneTime, causing this counter to
+  // be lower than it should be when not in dry run mode.
+  // See: https://beam.apache.org/documentation/programming-guide/#user-code-idempotence
+  private final Counter oneTimesToExpandCounter =
+      Metrics.counter("ExpandBilling", "OneTimes that would be expanded");

  ExpandRecurringBillingEventsPipeline(
      ExpandRecurringBillingEventsPipelineOptions options, Clock clock) {
@@ -195,6 +201,7 @@ public class ExpandRecurringBillingEventsPipeline implements Serializable {
                    endTime.minusYears(1)),
                true,
                (BigInteger id) -> {
+                  recurringsInScopeCounter.inc();
                  // Note that because all elements are mapped to the same dummy key, the next
                  // batching transform will effectively be serial. This however does not matter for
                  // our use case because the elements were obtained from a SQL read query, which
@@ -271,13 +278,9 @@ public class ExpandRecurringBillingEventsPipeline implements Serializable {
    } catch (IllegalArgumentException e) {
      return;
    }
-
-    recurringsInScopeCounter.inc();
    Domain domain = tm().loadByKey(Domain.createVKey(recurring.getDomainRepoId()));
    Registry tld = Registry.get(domain.getTld());

-
-
    // Find the times for which the OneTime billing event are already created, making this expansion
    // idempotent. There is no need to match to the domain repo ID as the cancellation matching
    // billing event itself can only be for a single domain.
@@ -301,7 +304,7 @@ public class ExpandRecurringBillingEventsPipeline implements Serializable {
    // Create new OneTime and DomainHistory for EventTimes that needs to be expanded.
    for (DateTime eventTime : eventTimesToExpand) {
      recurrenceLastExpansionTime = latestOf(recurrenceLastExpansionTime, eventTime);
-      expandedOneTimeCounter.inc();
+      oneTimesToExpandCounter.inc();
      DateTime billingTime = eventTime.plus(tld.getAutoRenewGracePeriodLength());
      // Note that the DomainHistory is created as of transaction time, as opposed to event time.
      // This might be counterintuitive because other DomainHistories are created at the time
@@ -427,7 +427,7 @@ misc:

 beam:
  # The default region to run Apache Beam (Cloud Dataflow) jobs in.
-  defaultJobRegion: us-east1
+  defaultJobRegion: us-central1
  # The GCE machine type to use when a job is CPU-intensive (e. g. RDE). Be sure
  # to check the VM CPU quota for the job region. In a massively parallel
  # pipeline this quota can be easily reached and needs to be raised, otherwise
@@ -94,6 +94,12 @@
    <url-pattern>/registry-lock-verify</url-pattern>
  </servlet-mapping>

+  <!-- Registrar console endpoints -->
+  <servlet-mapping>
+    <servlet-name>frontend-servlet</servlet-name>
+    <url-pattern>/console-api/*</url-pattern>
+  </servlet-mapping>
+
  <!-- Security config -->
  <security-constraint>
    <web-resource-collection>
@@ -16,6 +16,7 @@ package google.registry.model;

 import static google.registry.persistence.transaction.TransactionManagerFactory.tm;

+import com.google.gson.annotations.Expose;
 import javax.annotation.Nullable;
 import javax.persistence.Column;
 import javax.persistence.Embeddable;
@@ -28,6 +29,7 @@ import org.joda.time.DateTime;
 public class CreateAutoTimestamp extends ImmutableObject implements UnsafeSerializable {

  @Column(nullable = false)
+  @Expose
  DateTime creationTime;

  @PrePersist
@@ -31,6 +31,7 @@ import com.github.benmanes.caffeine.cache.LoadingCache;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
+import com.google.gson.annotations.Expose;
 import google.registry.config.RegistryConfig;
 import google.registry.dns.RefreshDnsAction;
 import google.registry.model.eppcommon.StatusValue;
@@ -67,7 +68,7 @@ public abstract class EppResource extends UpdateAutoTimestampEntity implements B
   *
   * @see <a href="https://tools.ietf.org/html/rfc5730">RFC 5730</a>
   */
-  @Transient String repoId;
+  @Expose @Transient String repoId;

  /**
   * The ID of the registrar that is currently sponsoring this resource.
@@ -75,7 +76,7 @@ public abstract class EppResource extends UpdateAutoTimestampEntity implements B
   * <p>This can be null in the case of pre-Registry-3.0-migration history objects with null
   * resource fields.
   */
-  String currentSponsorRegistrarId;
+  @Expose String currentSponsorRegistrarId;

  /**
   * The ID of the registrar that created this resource.
@@ -83,7 +84,7 @@ public abstract class EppResource extends UpdateAutoTimestampEntity implements B
   * <p>This can be null in the case of pre-Registry-3.0-migration history objects with null
   * resource fields.
   */
-  String creationRegistrarId;
+  @Expose String creationRegistrarId;

  /**
   * The ID of the registrar that last updated this resource.
@@ -92,7 +93,7 @@ public abstract class EppResource extends UpdateAutoTimestampEntity implements B
   * edits; it only includes EPP-visible modifications such as {@literal <update>}. Can be null if
   * the resource has never been modified.
   */
-  String lastEppUpdateRegistrarId;
+  @Expose String lastEppUpdateRegistrarId;

  /**
   * The time when this resource was created.
@@ -106,6 +107,7 @@ public abstract class EppResource extends UpdateAutoTimestampEntity implements B
   */
  // Need to override the default non-null column attribute.
  @AttributeOverride(name = "creationTime", column = @Column)
+  @Expose
  CreateAutoTimestamp creationTime = CreateAutoTimestamp.create(null);

  /**
@@ -130,10 +132,10 @@ public abstract class EppResource extends UpdateAutoTimestampEntity implements B
   * edits; it only includes EPP-visible modifications such as {@literal <update>}. Can be null if
   * the resource has never been modified.
   */
-  DateTime lastEppUpdateTime;
+  @Expose DateTime lastEppUpdateTime;

  /** Status values associated with this resource. */
-  Set<StatusValue> statuses;
+  @Expose Set<StatusValue> statuses;

  /**
   * When this domain/host's DNS was requested to be refreshed, or null if its DNS is up-to-date.
@@ -40,6 +40,7 @@ import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.ImmutableSortedSet;
 import com.google.common.collect.Ordering;
 import com.google.common.collect.Sets;
+import com.google.gson.annotations.Expose;
 import google.registry.flows.ResourceFlowUtils;
 import google.registry.model.EppResource;
 import google.registry.model.EppResource.ResourceWithTransferData;
@@ -121,20 +122,20 @@ public class DomainBase extends EppResource
   *
   * @invariant domainName == domainName.toLowerCase(Locale.ENGLISH)
   */
-  String domainName;
+  @Expose String domainName;

  /** The top level domain this is under, de-normalized from {@link #domainName}. */
  String tld;

  /** References to hosts that are the nameservers for the domain. */
-  @Transient Set<VKey<Host>> nsHosts;
+  @Expose @Transient Set<VKey<Host>> nsHosts;

  /** Contacts. */
-  VKey<Contact> adminContact;
+  @Expose VKey<Contact> adminContact;

-  VKey<Contact> billingContact;
-  VKey<Contact> techContact;
-  VKey<Contact> registrantContact;
+  @Expose VKey<Contact> billingContact;
+  @Expose VKey<Contact> techContact;
+  @Expose VKey<Contact> registrantContact;

  /** Authorization info (aka transfer secret) of the domain. */
  @Embedded
@@ -175,10 +176,10 @@ public class DomainBase extends EppResource
  String idnTableName;

  /** Fully qualified host names of this domain's active subordinate hosts. */
-  Set<String> subordinateHosts;
+  @Expose Set<String> subordinateHosts;

  /** When this domain's registration will expire. */
-  DateTime registrationExpirationTime;
+  @Expose DateTime registrationExpirationTime;

  /**
   * The poll message associated with this domain being deleted.
@@ -230,7 +231,7 @@ public class DomainBase extends EppResource
   *
   * <p>Can be null if the resource has never been transferred.
   */
-  DateTime lastTransferTime;
+  @Expose DateTime lastTransferTime;

  /**
   * When the domain's autorenewal status will expire.
@@ -25,6 +25,7 @@ import google.registry.monitoring.whitebox.WhiteboxModule;
 import google.registry.request.RequestComponentBuilder;
 import google.registry.request.RequestModule;
 import google.registry.request.RequestScope;
+import google.registry.ui.server.console.ConsoleDomainGetAction;
 import google.registry.ui.server.registrar.ConsoleOteSetupAction;
 import google.registry.ui.server.registrar.ConsoleRegistrarCreatorAction;
 import google.registry.ui.server.registrar.ConsoleUiAction;
@@ -61,6 +62,8 @@ interface FrontendRequestComponent {

  RegistryLockVerifyAction registryLockVerifyAction();

+  ConsoleDomainGetAction consoleDomainGetAction();
+
  @Subcomponent.Builder
  abstract class Builder implements RequestComponentBuilder<FrontendRequestComponent> {
    @Override public abstract Builder requestModule(RequestModule requestModule);
@@ -22,6 +22,7 @@ import com.google.common.base.Joiner;
 import com.google.common.base.Splitter;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
+import com.google.gson.annotations.Expose;
 import google.registry.model.EppResource;
 import google.registry.model.ImmutableObject;
 import google.registry.model.contact.Contact;
@@ -52,7 +53,7 @@ public class VKey<T> extends ImmutableObject implements Serializable {
          .collect(toImmutableMap(Class::getSimpleName, identity()));

  // The primary key for the referenced entity.
-  Serializable key;
+  @Expose Serializable key;

  Class<? extends T> kind;

@@ -0,0 +1,91 @@
+// Copyright 2023 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.ui.server.console;
+
+import static google.registry.persistence.transaction.TransactionManagerFactory.tm;
+
+import com.google.api.client.http.HttpStatusCodes;
+import com.google.gson.Gson;
+import google.registry.model.EppResourceUtils;
+import google.registry.model.console.ConsolePermission;
+import google.registry.model.console.User;
+import google.registry.model.domain.Domain;
+import google.registry.request.Action;
+import google.registry.request.Parameter;
+import google.registry.request.Response;
+import google.registry.request.auth.Auth;
+import google.registry.request.auth.AuthResult;
+import google.registry.request.auth.UserAuthInfo;
+import google.registry.ui.server.registrar.JsonGetAction;
+import java.util.Optional;
+import javax.inject.Inject;
+
+/** Returns a JSON representation of a domain to the registrar console. */
+@Action(
+    service = Action.Service.DEFAULT,
+    path = ConsoleDomainGetAction.PATH,
+    auth = Auth.AUTH_PUBLIC_LOGGED_IN)
+public class ConsoleDomainGetAction implements JsonGetAction {
+
+  public static final String PATH = "/console-api/domain";
+
+  private final AuthResult authResult;
+  private final Response response;
+  private final Gson gson;
+  private final String paramDomain;
+
+  @Inject
+  public ConsoleDomainGetAction(
+      AuthResult authResult,
+      Response response,
+      Gson gson,
+      @Parameter("domain") String paramDomain) {
+    this.authResult = authResult;
+    this.response = response;
+    this.gson = gson;
+    this.paramDomain = paramDomain;
+  }
+
+  @Override
+  public void run() {
+    if (!authResult.isAuthenticated() || !authResult.userAuthInfo().isPresent()) {
+      response.setStatus(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
+      return;
+    }
+    UserAuthInfo authInfo = authResult.userAuthInfo().get();
+    if (!authInfo.consoleUser().isPresent()) {
+      response.setStatus(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
+      return;
+    }
+    User user = authInfo.consoleUser().get();
+    Optional<Domain> possibleDomain =
+        tm().transact(
+                () ->
+                    EppResourceUtils.loadByForeignKeyCached(
+                        Domain.class, paramDomain, tm().getTransactionTime()));
+    if (!possibleDomain.isPresent()) {
+      response.setStatus(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
+      return;
+    }
+    Domain domain = possibleDomain.get();
+    if (!user.getUserRoles()
+        .hasPermission(domain.getCurrentSponsorRegistrarId(), ConsolePermission.DOWNLOAD_DOMAINS)) {
+      response.setStatus(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
+      return;
+    }
+    response.setStatus(HttpStatusCodes.STATUS_CODE_OK);
+    response.setPayload(gson.toJson(domain));
+  }
+}
@@ -156,4 +156,10 @@ public final class RegistrarConsoleModule {
  static Boolean provideIsLock(HttpServletRequest req) {
    return extractBooleanParameter(req, "isLock");
  }
+
+  @Provides
+  @Parameter("domain")
+  static String provideDomain(HttpServletRequest req) {
+    return extractRequiredParameter(req, "domain");
+  }
 }
@@ -23,12 +23,6 @@
      "helpText": "The exclusive upper bound of the operation window, in ISO 8601 format.",
      "is_optional": false
    },
-    {
-      "name": "shard",
-      "label": "The exclusive upper bound of the operation window.",
-      "helpText": "The exclusive upper bound of the operation window, in ISO 8601 format.",
-      "is_optional": true
-    },
    {
      "name": "isDryRun",
      "label": "Whether this job is a dry run.",
@@ -0,0 +1,139 @@
+// Copyright 2023 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.ui.server.console;
+
+import static com.google.common.truth.Truth.assertThat;
+import static google.registry.testing.DatabaseHelper.createTld;
+import static org.mockito.Mockito.mock;
+
+import com.google.api.client.http.HttpStatusCodes;
+import com.google.common.collect.ImmutableMap;
+import com.google.gson.Gson;
+import google.registry.model.console.RegistrarRole;
+import google.registry.model.console.User;
+import google.registry.model.console.UserRoles;
+import google.registry.persistence.transaction.JpaTestExtensions;
+import google.registry.request.auth.AuthLevel;
+import google.registry.request.auth.AuthResult;
+import google.registry.request.auth.UserAuthInfo;
+import google.registry.testing.DatabaseHelper;
+import google.registry.testing.FakeResponse;
+import google.registry.util.UtilsModule;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+/** Tests for {@link google.registry.ui.server.console.ConsoleDomainGetAction}. */
+public class ConsoleDomainGetActionTest {
+
+  private static final Gson GSON = UtilsModule.provideGson();
+  private static final FakeResponse RESPONSE = new FakeResponse();
+
+  @RegisterExtension
+  final JpaTestExtensions.JpaIntegrationTestExtension jpa =
+      new JpaTestExtensions.Builder().buildIntegrationTestExtension();
+
+  @BeforeEach
+  void beforeEach() {
+    createTld("tld");
+    DatabaseHelper.persistActiveDomain("exists.tld");
+  }
+
+  @Test
+  void testSuccess_fullJsonRepresentation() {
+    ConsoleDomainGetAction action =
+        createAction(
+            AuthResult.create(
+                AuthLevel.USER,
+                UserAuthInfo.create(
+                    createUser(
+                        new UserRoles.Builder()
+                            .setRegistrarRoles(
+                                ImmutableMap.of("TheRegistrar", RegistrarRole.ACCOUNT_MANAGER))
+                            .build()))),
+            "exists.tld");
+    action.run();
+    assertThat(RESPONSE.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_OK);
+    assertThat(RESPONSE.getPayload())
+        .isEqualTo(
+            "{\"domainName\":\"exists.tld\",\"adminContact\":{\"key\":\"3-ROID\"},\"techContact\":"
+                + "{\"key\":\"3-ROID\"},\"registrantContact\":{\"key\":\"3-ROID\"},\"registrationExpirationTime\":"
+                + "\"294247-01-10T04:00:54.775Z\",\"lastTransferTime\":\"null\",\"repoId\":\"2-TLD\","
+                + "\"currentSponsorRegistrarId\":\"TheRegistrar\",\"creationRegistrarId\":\"TheRegistrar\","
+                + "\"creationTime\":{\"creationTime\":\"1970-01-01T00:00:00.000Z\"},\"lastEppUpdateTime\":\"null\","
+                + "\"statuses\":[\"INACTIVE\"]}");
+  }
+
+  @Test
+  void testFailure_emptyAuth() {
+    ConsoleDomainGetAction action = createAction(AuthResult.NOT_AUTHENTICATED, "exists.tld");
+    action.run();
+    assertThat(RESPONSE.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
+  }
+
+  @Test
+  void testFailure_appAuth() {
+    ConsoleDomainGetAction action = createAction(AuthResult.create(AuthLevel.APP), "exists.tld");
+    action.run();
+    assertThat(RESPONSE.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
+  }
+
+  @Test
+  void testFailure_wrongTypeOfUser() {
+    ConsoleDomainGetAction action =
+        createAction(
+            AuthResult.create(
+                AuthLevel.USER,
+                UserAuthInfo.create(mock(com.google.appengine.api.users.User.class), false)),
+            "exists.tld");
+    action.run();
+    assertThat(RESPONSE.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_UNAUTHORIZED);
+  }
+
+  @Test
+  void testFailure_noAccessToRegistrar() {
+    ConsoleDomainGetAction action =
+        createAction(
+            AuthResult.create(
+                AuthLevel.USER, UserAuthInfo.create(createUser(new UserRoles.Builder().build()))),
+            "exists.tld");
+    action.run();
+    assertThat(RESPONSE.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
+  }
+
+  @Test
+  void testFailure_nonexistentDomain() {
+    ConsoleDomainGetAction action =
+        createAction(
+            AuthResult.create(
+                AuthLevel.USER,
+                UserAuthInfo.create(createUser(new UserRoles.Builder().setIsAdmin(true).build()))),
+            "nonexistent.tld");
+    action.run();
+    assertThat(RESPONSE.getStatus()).isEqualTo(HttpStatusCodes.STATUS_CODE_NOT_FOUND);
+  }
+
+  private User createUser(UserRoles userRoles) {
+    return new User.Builder()
+        .setEmailAddress("email@email.com")
+        .setGaiaId("gaiaId")
+        .setUserRoles(userRoles)
+        .build();
+  }
+
+  private ConsoleDomainGetAction createAction(AuthResult authResult, String domain) {
+    return new ConsoleDomainGetAction(authResult, RESPONSE, GSON, domain);
+  }
+}
@@ -1,5 +1,6 @@
 PATH                  CLASS                         METHODS  OK AUTH_METHODS        MIN  USER_POLICY
 /_dr/epp              EppTlsAction                  POST     n  INTERNAL,API        APP  PUBLIC
+/console-api/domain   ConsoleDomainGetAction        GET      n  API,LEGACY          USER PUBLIC
 /registrar            ConsoleUiAction               GET      n  INTERNAL,API,LEGACY NONE PUBLIC
 /registrar-create     ConsoleRegistrarCreatorAction POST,GET n  INTERNAL,API,LEGACY NONE PUBLIC
 /registrar-ote-setup  ConsoleOteSetupAction         POST,GET n  INTERNAL,API,LEGACY NONE PUBLIC
@@ -261,11 +261,11 @@ td.section {
    </tr>
    <tr>
     <td class="property_name">generated on</td>
-     <td class="property_value">2023-01-20 19:50:48.623695</td>
+     <td class="property_value">2023-02-08 17:40:28.930421</td>
    </tr>
    <tr>
     <td class="property_name">last flyway file</td>
-     <td id="lastFlywayFile" class="property_value">V134__drop_not_null_request_id_lock_table.sql</td>
+     <td id="lastFlywayFile" class="property_value">V135__null_gaia_id_user.sql</td>
    </tr>
   </tbody>
  </table>
@@ -284,7 +284,7 @@ td.section {
     generated on
    </text>
    <text text-anchor="start" x="3835.5" y="-10.8" font-family="Helvetica,sans-Serif" font-size="14.00">
-     2023-01-20 19:50:48.623695
+     2023-02-08 17:40:28.930421
    </text>
    <polygon fill="none" stroke="#888888" points="3748,-4 3748,-44 4013,-44 4013,-4 3748,-4" /> <!-- allocationtoken_a08ccbef -->
    <g id="node1" class="node">
@@ -3220,7 +3220,7 @@ td.section {
     <text text-anchor="start" x="3815.5" y="-3015.8" font-family="Helvetica,sans-Serif" font-size="14.00"> 
     </text>
     <text text-anchor="start" x="3823.5" y="-3015.8" font-family="Helvetica,sans-Serif" font-size="14.00">
-      text not null
+      text
     </text>
     <polygon fill="none" stroke="#888888" points="3719,-3009.5 3719,-3106.5 3932,-3106.5 3932,-3009.5 3719,-3009.5" /> 
    </g> 
@@ -7067,7 +7067,7 @@ td.section {
    <tr>
     <td class="spacer"></td>
     <td class="minwidth">gaia_id</td>
-     <td class="minwidth">text not null</td>
+     <td class="minwidth">text</td>
    </tr>
    <tr>
     <td colspan="3"></td>
@@ -261,11 +261,11 @@ td.section {
    </tr>
    <tr>
     <td class="property_name">generated on</td>
-     <td class="property_value">2023-01-20 19:50:45.218406</td>
+     <td class="property_value">2023-02-08 17:40:26.84563</td>
    </tr>
    <tr>
     <td class="property_name">last flyway file</td>
-     <td id="lastFlywayFile" class="property_value">V134__drop_not_null_request_id_lock_table.sql</td>
+     <td id="lastFlywayFile" class="property_value">V135__null_gaia_id_user.sql</td>
    </tr>
   </tbody>
  </table>
@@ -274,19 +274,19 @@ td.section {
  <svg viewbox="0.00 0.00 4713.00 7236.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" id="erDiagram" style="overflow: hidden; width: 100%; height: 800px"><g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 7232)">
    <title>SchemaCrawler_Diagram</title>
    <polygon fill="white" stroke="transparent" points="-4,4 -4,-7232 4709,-7232 4709,4 -4,4" />
-    <text text-anchor="start" x="4436.5" y="-29.8" font-family="Helvetica,sans-Serif" font-size="14.00">
+    <text text-anchor="start" x="4444.5" y="-29.8" font-family="Helvetica,sans-Serif" font-size="14.00">
     generated by
    </text>
-    <text text-anchor="start" x="4519.5" y="-29.8" font-family="Helvetica,sans-Serif" font-size="14.00">
+    <text text-anchor="start" x="4527.5" y="-29.8" font-family="Helvetica,sans-Serif" font-size="14.00">
     SchemaCrawler 16.10.1
    </text>
-    <text text-anchor="start" x="4435.5" y="-10.8" font-family="Helvetica,sans-Serif" font-size="14.00">
+    <text text-anchor="start" x="4443.5" y="-10.8" font-family="Helvetica,sans-Serif" font-size="14.00">
     generated on
    </text>
-    <text text-anchor="start" x="4519.5" y="-10.8" font-family="Helvetica,sans-Serif" font-size="14.00">
-     2023-01-20 19:50:45.218406
+    <text text-anchor="start" x="4527.5" y="-10.8" font-family="Helvetica,sans-Serif" font-size="14.00">
+     2023-02-08 17:40:26.84563
    </text>
-    <polygon fill="none" stroke="#888888" points="4432,-4 4432,-44 4697,-44 4697,-4 4432,-4" /> <!-- allocationtoken_a08ccbef -->
+    <polygon fill="none" stroke="#888888" points="4440,-4 4440,-44 4697,-44 4697,-4 4440,-4" /> <!-- allocationtoken_a08ccbef -->
    <g id="node1" class="node">
     <title>allocationtoken_a08ccbef</title>
     <polygon fill="#ebcef2" stroke="transparent" points="1380.5,-1056.5 1380.5,-1075.5 1579.5,-1075.5 1579.5,-1056.5 1380.5,-1056.5" />
@@ -6564,7 +6564,7 @@ td.section {
     <text text-anchor="start" x="4502" y="-7133.3" font-family="Helvetica,sans-Serif" font-size="14.00"> 
     </text>
     <text text-anchor="start" x="4510" y="-7133.3" font-family="Helvetica,sans-Serif" font-size="14.00">
-      text not null
+      text
     </text>
     <text text-anchor="start" x="4320" y="-7114.3" font-family="Helvetica,sans-Serif" font-size="14.00">
      registry_lock_password_hash
@@ -14351,7 +14351,7 @@ td.section {
    <tr>
     <td class="spacer"></td>
     <td class="minwidth">gaia_id</td>
-     <td class="minwidth">text not null</td>
+     <td class="minwidth">text</td>
    </tr>
    <tr>
     <td class="spacer"></td>
@@ -132,3 +132,4 @@ V131__add_default_promo_tokens_to_registry.sql
 V132__drop_gae_user_id_from_registrar_poc.sql
 V133__add_pull_queue_replace_columns.sql
 V134__drop_not_null_request_id_lock_table.sql
+V135__null_gaia_id_user.sql
@@ -0,0 +1,15 @@
+-- Copyright 2023 The Nomulus Authors. All Rights Reserved.
+--
+-- Licensed under the Apache License, Version 2.0 (the "License");
+-- you may not use this file except in compliance with the License.
+-- You may obtain a copy of the License at
+--
+--     http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+ALTER TABLE "User" ALTER COLUMN gaia_id DROP NOT NULL;
@@ -1085,7 +1085,7 @@ CREATE TABLE public."TmchCrl" (
 CREATE TABLE public."User" (
    id bigint NOT NULL,
    email_address text NOT NULL,
-    gaia_id text NOT NULL,
+    gaia_id text,
    registry_lock_password_hash text,
    registry_lock_password_salt text,
    global_role text NOT NULL,
@@ -16,6 +16,7 @@ package google.registry.proxy;

 import static google.registry.util.ResourceUtils.readResourceBytes;

+import com.google.auth.oauth2.GoogleCredentials;
 import com.google.common.collect.ImmutableList;
 import dagger.Module;
 import dagger.Provides;
@@ -43,6 +44,7 @@ import io.netty.handler.timeout.ReadTimeoutHandler;
 import java.io.IOException;
 import java.security.PrivateKey;
 import java.security.cert.X509Certificate;
+import java.util.Optional;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.function.Supplier;
@@ -147,12 +149,18 @@ public final class EppProtocolModule {

  @Provides
  static EppServiceHandler provideEppServiceHandler(
-      @Named("accessToken") Supplier<String> accessTokenSupplier,
+      Supplier<GoogleCredentials> refreshedCredentialsSupplier,
+      @Named("iapClientId") Optional<String> iapClientId,
      @Named("hello") byte[] helloBytes,
      FrontendMetrics metrics,
      ProxyConfig config) {
    return new EppServiceHandler(
-        config.epp.relayHost, config.epp.relayPath, accessTokenSupplier, helloBytes, metrics);
+        config.epp.relayHost,
+        config.epp.relayPath,
+        refreshedCredentialsSupplier,
+        iapClientId,
+        helloBytes,
+        metrics);
  }

  @Singleton
@@ -40,6 +40,7 @@ public class ProxyConfig {
  private static final String CUSTOM_CONFIG_FORMATTER = "config/proxy-config-%s.yaml";

  public String projectId;
+  public String iapClientId;
  public List<String> gcpScopes;
  public int serverCertificateCacheSeconds;
  public Gcs gcs;
@@ -157,6 +157,13 @@ public class ProxyModule {
    return this;
  }

+  @Provides
+  @Named("iapClientId")
+  @Singleton
+  Optional<String> provideIapClientId(ProxyConfig config) {
+    return Optional.ofNullable(config.iapClientId);
+  }
+
  @Provides
  @WhoisProtocol
  int provideWhoisPort(ProxyConfig config) {
@@ -207,7 +214,7 @@ public class ProxyModule {

  @Singleton
  @Provides
-  static GoogleCredentialsBundle provideCredential(ProxyConfig config) {
+  static GoogleCredentialsBundle provideCredentialsBundle(ProxyConfig config) {
    try {
      GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
      if (credentials.createScopedRequired()) {
@@ -219,19 +226,19 @@ public class ProxyModule {
    }
  }

-  /** Access token supplier that auto refreshes 1 minute before expiry. */
+  /** Provides a set of credentials that auto refreshes 1 minute before expiry. */
  @Singleton
  @Provides
-  @Named("accessToken")
-  static Supplier<String> provideAccessTokenSupplier(GoogleCredentialsBundle credentialsBundle) {
+  static Supplier<GoogleCredentials> provideRefreshedCredentialsSupplier(
+      GoogleCredentialsBundle credentialsBundle) {
    return () -> {
      GoogleCredentials credentials = credentialsBundle.getGoogleCredentials();
      try {
        credentials.refreshIfExpired();
      } catch (IOException e) {
-        throw new RuntimeException("Cannot refresh access token.", e);
+        throw new RuntimeException("Cannot refresh credentials.", e);
      }
-      return credentials.getAccessToken().getTokenValue();
+      return credentials;
    };
  }

@@ -14,6 +14,7 @@

 package google.registry.proxy;

+import com.google.auth.oauth2.GoogleCredentials;
 import com.google.common.collect.ImmutableList;
 import dagger.Module;
 import dagger.Provides;
@@ -34,6 +35,7 @@ import google.registry.util.Clock;
 import io.netty.channel.ChannelHandler;
 import io.netty.handler.codec.LineBasedFrameDecoder;
 import io.netty.handler.timeout.ReadTimeoutHandler;
+import java.util.Optional;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.function.Supplier;
@@ -91,10 +93,15 @@ public class WhoisProtocolModule {
  @Provides
  static WhoisServiceHandler provideWhoisServiceHandler(
      ProxyConfig config,
-      @Named("accessToken") Supplier<String> accessTokenSupplier,
+      Supplier<GoogleCredentials> refreshedCredentialsSupplier,
+      @Named("iapClientId") Optional<String> iapClientId,
      FrontendMetrics metrics) {
    return new WhoisServiceHandler(
-        config.whois.relayHost, config.whois.relayPath, accessTokenSupplier, metrics);
+        config.whois.relayHost,
+        config.whois.relayPath,
+        refreshedCredentialsSupplier,
+        iapClientId,
+        metrics);
  }

  @Provides
@@ -8,6 +8,9 @@
 # GCP project ID
 projectId: your-gcp-project-id

+# IAP client ID, if IAP is enabled for this project
+iapClientId: null
+
 # OAuth scope that the GoogleCredential will be constructed with. This list
 # should include all service scopes that the proxy depends on.
 gcpScopes:
@@ -20,6 +20,7 @@ import static google.registry.networking.handler.SslServerInitializer.CLIENT_CER
 import static google.registry.proxy.handler.ProxyProtocolHandler.REMOTE_ADDRESS_KEY;
 import static google.registry.util.X509Utils.getCertificateHash;

+import com.google.auth.oauth2.GoogleCredentials;
 import com.google.common.flogger.FluentLogger;
 import google.registry.proxy.metric.FrontendMetrics;
 import google.registry.util.ProxyHttpHeaders;
@@ -36,6 +37,7 @@ import io.netty.handler.ssl.SslHandshakeCompletionEvent;
 import io.netty.util.AttributeKey;
 import io.netty.util.concurrent.Promise;
 import java.security.cert.X509Certificate;
+import java.util.Optional;
 import java.util.function.Supplier;

 /** Handler that processes EPP protocol logic. */
@@ -60,10 +62,11 @@ public class EppServiceHandler extends HttpsRelayServiceHandler {
  public EppServiceHandler(
      String relayHost,
      String relayPath,
-      Supplier<String> accessTokenSupplier,
+      Supplier<GoogleCredentials> refreshedCredentialsSupplier,
+      Optional<String> iapClientId,
      byte[] helloBytes,
      FrontendMetrics metrics) {
-    super(relayHost, relayPath, accessTokenSupplier, metrics);
+    super(relayHost, relayPath, refreshedCredentialsSupplier, iapClientId, metrics);
    this.helloBytes = helloBytes;
  }

@@ -16,7 +16,12 @@ package google.registry.proxy.handler;

 import static java.nio.charset.StandardCharsets.UTF_8;

+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.auth.oauth2.IdToken;
+import com.google.auth.oauth2.IdTokenProvider;
+import com.google.auth.oauth2.IdTokenProvider.Option;
 import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.flogger.FluentLogger;
 import google.registry.proxy.metric.FrontendMetrics;
@@ -37,9 +42,11 @@ import io.netty.handler.codec.http.cookie.ClientCookieDecoder;
 import io.netty.handler.codec.http.cookie.ClientCookieEncoder;
 import io.netty.handler.codec.http.cookie.Cookie;
 import io.netty.handler.timeout.ReadTimeoutException;
+import java.io.IOException;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.Optional;
 import java.util.function.Supplier;
 import javax.net.ssl.SSLHandshakeException;

@@ -72,18 +79,21 @@ public abstract class HttpsRelayServiceHandler extends ByteToMessageCodec<FullHt
  private final Map<String, Cookie> cookieStore = new LinkedHashMap<>();
  private final String relayHost;
  private final String relayPath;
-  private final Supplier<String> accessTokenSupplier;
+  private final Supplier<GoogleCredentials> refreshedCredentialsSupplier;
+  private final Optional<String> iapClientId;

  protected final FrontendMetrics metrics;

  HttpsRelayServiceHandler(
      String relayHost,
      String relayPath,
-      Supplier<String> accessTokenSupplier,
+      Supplier<GoogleCredentials> refreshedCredentialsSupplier,
+      Optional<String> iapClientId,
      FrontendMetrics metrics) {
    this.relayHost = relayHost;
    this.relayPath = relayPath;
-    this.accessTokenSupplier = accessTokenSupplier;
+    this.refreshedCredentialsSupplier = refreshedCredentialsSupplier;
+    this.iapClientId = iapClientId;
    this.metrics = metrics;
  }

@@ -91,19 +101,37 @@ public abstract class HttpsRelayServiceHandler extends ByteToMessageCodec<FullHt
   * Construct the {@link FullHttpRequest}.
   *
   * <p>This default method creates a bare-bone {@link FullHttpRequest} that may need to be
-   * modified, e. g. adding headers specific for each protocol.
+   * modified, e.g. adding headers specific for each protocol.
   *
   * @param byteBuf inbound message.
   */
  protected FullHttpRequest decodeFullHttpRequest(ByteBuf byteBuf) {
    FullHttpRequest request =
        new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, relayPath);
+    GoogleCredentials credentials = refreshedCredentialsSupplier.get();
    request
        .headers()
        .set(HttpHeaderNames.USER_AGENT, "Proxy")
        .set(HttpHeaderNames.HOST, relayHost)
-        .set(HttpHeaderNames.AUTHORIZATION, "Bearer " + accessTokenSupplier.get())
+        .set(
+            HttpHeaderNames.AUTHORIZATION, "Bearer " + credentials.getAccessToken().getTokenValue())
        .setInt(HttpHeaderNames.CONTENT_LENGTH, byteBuf.readableBytes());
+    // Set the Proxy-Authorization header if using IAP
+    if (iapClientId.isPresent()) {
+      IdTokenProvider idTokenProvider = (IdTokenProvider) credentials;
+      try {
+        // Note: we use Option.FORMAT_FULL to make sure the JWT we receive contains the email
+        // address (as is required by IAP)
+        IdToken idToken =
+            idTokenProvider.idTokenWithAudience(
+                iapClientId.get(), ImmutableList.of(Option.FORMAT_FULL));
+        request
+            .headers()
+            .set(HttpHeaderNames.PROXY_AUTHORIZATION, "Bearer " + idToken.getTokenValue());
+      } catch (IOException e) {
+        logger.atSevere().withCause(e).log("Error when attempting to retrieve IAP ID token");
+      }
+    }
    request.content().writeBytes(byteBuf);
    return request;
  }
@@ -16,6 +16,7 @@ package google.registry.proxy.handler;

 import static com.google.common.base.Preconditions.checkArgument;

+import com.google.auth.oauth2.GoogleCredentials;
 import google.registry.proxy.metric.FrontendMetrics;
 import io.netty.buffer.ByteBuf;
 import io.netty.channel.ChannelFutureListener;
@@ -25,6 +26,7 @@ import io.netty.handler.codec.http.FullHttpRequest;
 import io.netty.handler.codec.http.HttpHeaderNames;
 import io.netty.handler.codec.http.HttpHeaderValues;
 import io.netty.handler.codec.http.HttpResponse;
+import java.util.Optional;
 import java.util.function.Supplier;

 /** Handler that processes WHOIS protocol logic. */
@@ -33,9 +35,10 @@ public final class WhoisServiceHandler extends HttpsRelayServiceHandler {
  public WhoisServiceHandler(
      String relayHost,
      String relayPath,
-      Supplier<String> accessTokenSupplier,
+      Supplier<GoogleCredentials> refreshedCredentialsSupplier,
+      Optional<String> iapClientId,
      FrontendMetrics metrics) {
-    super(relayHost, relayPath, accessTokenSupplier, metrics);
+    super(relayHost, relayPath, refreshedCredentialsSupplier, iapClientId, metrics);
  }

  @Override
@@ -36,6 +36,7 @@ import io.netty.handler.codec.http.HttpResponseStatus;
 import io.netty.handler.codec.http.cookie.Cookie;
 import io.netty.handler.codec.http.cookie.DefaultCookie;
 import io.netty.util.concurrent.Promise;
+import java.io.IOException;
 import java.security.cert.X509Certificate;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -96,14 +97,15 @@ class EppProtocolModuleTest extends ProtocolModuleTest {
    return buffer;
  }

-  private FullHttpRequest makeEppHttpRequest(byte[] content, Cookie... cookies) {
+  private FullHttpRequest makeEppHttpRequest(byte[] content, Cookie... cookies) throws IOException {
    return TestUtils.makeEppHttpRequest(
        new String(content, UTF_8),
        PROXY_CONFIG.epp.relayHost,
        PROXY_CONFIG.epp.relayPath,
-        TestModule.provideFakeAccessToken().get(),
+        TestModule.provideFakeCredentials().get(),
        getCertificateHash(certificate),
        CLIENT_ADDRESS,
+        TestModule.provideIapClientId(),
        cookies);
  }

@@ -17,7 +17,14 @@ package google.registry.proxy;
 import static com.google.common.collect.ImmutableList.toImmutableList;
 import static google.registry.proxy.ProxyConfig.Environment.LOCAL;
 import static google.registry.proxy.ProxyConfig.getProxyConfig;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;

+import com.google.auth.oauth2.AccessToken;
+import com.google.auth.oauth2.ComputeEngineCredentials;
+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.auth.oauth2.IdToken;
+import com.google.auth.oauth2.IdTokenProvider.Option;
 import com.google.common.base.Suppliers;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
@@ -52,7 +59,9 @@ import io.netty.channel.embedded.EmbeddedChannel;
 import io.netty.handler.logging.LoggingHandler;
 import io.netty.handler.ssl.SslProvider;
 import io.netty.handler.timeout.ReadTimeoutHandler;
+import java.io.IOException;
 import java.time.Duration;
+import java.util.Optional;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
@@ -223,7 +232,7 @@ public abstract class ProtocolModuleTest {
   * should be provided in the respective {@code ProtocolModule} instead.
   */
  @Module
-  static class TestModule {
+  public static class TestModule {

    /**
     * A fake clock that is explicitly provided. Users can construct a module with a controller
@@ -235,6 +244,12 @@ public abstract class ProtocolModuleTest {
      this.fakeClock = fakeClock;
    }

+    @Provides
+    @Named("iapClientId")
+    public static Optional<String> provideIapClientId() {
+      return Optional.of("iapClientId");
+    }
+
    @Singleton
    @Provides
    static ProxyConfig provideProxyConfig() {
@@ -249,9 +264,19 @@ public abstract class ProtocolModuleTest {

    @Singleton
    @Provides
-    @Named("accessToken")
-    static Supplier<String> provideFakeAccessToken() {
-      return Suppliers.ofInstance("fake.test.token");
+    static Supplier<GoogleCredentials> provideFakeCredentials() {
+      ComputeEngineCredentials mockCredentials = mock(ComputeEngineCredentials.class);
+      when(mockCredentials.getAccessToken()).thenReturn(new AccessToken("fake.test.token", null));
+      IdToken mockIdToken = mock(IdToken.class);
+      when(mockIdToken.getTokenValue()).thenReturn("fake.test.id.token");
+      try {
+        when(mockCredentials.idTokenWithAudience(
+                "iapClientId", ImmutableList.of(Option.FORMAT_FULL)))
+            .thenReturn(mockIdToken);
+      } catch (IOException e) {
+        throw new RuntimeException(e);
+      }
+      return Suppliers.ofInstance(mockCredentials);
    }

    @Singleton
@@ -17,6 +17,10 @@ package google.registry.proxy;
 import static com.google.common.truth.Truth.assertThat;
 import static java.nio.charset.StandardCharsets.US_ASCII;

+import com.google.auth.oauth2.GoogleCredentials;
+import com.google.auth.oauth2.IdTokenProvider;
+import com.google.auth.oauth2.IdTokenProvider.Option;
+import com.google.common.collect.ImmutableList;
 import google.registry.util.ProxyHttpHeaders;
 import io.netty.buffer.ByteBuf;
 import io.netty.buffer.Unpooled;
@@ -34,6 +38,8 @@ import io.netty.handler.codec.http.HttpVersion;
 import io.netty.handler.codec.http.cookie.ClientCookieEncoder;
 import io.netty.handler.codec.http.cookie.Cookie;
 import io.netty.handler.codec.http.cookie.ServerCookieEncoder;
+import java.io.IOException;
+import java.util.Optional;

 /** Utility class for various helper methods used in testing. */
 public class TestUtils {
@@ -71,13 +77,19 @@ public class TestUtils {
  }

  public static FullHttpRequest makeWhoisHttpRequest(
-      String content, String host, String path, String accessToken) {
+      String content,
+      String host,
+      String path,
+      GoogleCredentials credentials,
+      Optional<String> iapClientId)
+      throws IOException {
    FullHttpRequest request = makeHttpPostRequest(content, host, path);
    request
        .headers()
-        .set("authorization", "Bearer " + accessToken)
+        .set("authorization", "Bearer " + credentials.getAccessToken().getTokenValue())
        .set(HttpHeaderNames.CONTENT_TYPE, "text/plain")
        .set("accept", "text/plain");
+    maybeSetProxyAuthForIap(request, credentials, iapClientId);
    return request;
  }

@@ -85,18 +97,21 @@ public class TestUtils {
      String content,
      String host,
      String path,
-      String accessToken,
+      GoogleCredentials credentials,
      String sslClientCertificateHash,
      String clientAddress,
-      Cookie... cookies) {
+      Optional<String> iapClientId,
+      Cookie... cookies)
+      throws IOException {
    FullHttpRequest request = makeHttpPostRequest(content, host, path);
    request
        .headers()
-        .set("authorization", "Bearer " + accessToken)
+        .set("authorization", "Bearer " + credentials.getAccessToken().getTokenValue())
        .set(HttpHeaderNames.CONTENT_TYPE, "application/epp+xml")
        .set("accept", "application/epp+xml")
        .set(ProxyHttpHeaders.CERTIFICATE_HASH, sslClientCertificateHash)
        .set(ProxyHttpHeaders.IP_ADDRESS, clientAddress);
+    maybeSetProxyAuthForIap(request, credentials, iapClientId);
    if (cookies.length != 0) {
      request.headers().set("cookie", ClientCookieEncoder.STRICT.encode(cookies));
    }
@@ -146,4 +161,16 @@ public class TestUtils {
  public static void assertHttpRequestEquivalent(HttpRequest req1, HttpRequest req2) {
    assertHttpMessageEquivalent(req1, req2);
  }
+
+  private static void maybeSetProxyAuthForIap(
+      FullHttpRequest request, GoogleCredentials credentials, Optional<String> iapClientId)
+      throws IOException {
+    if (iapClientId.isPresent()) {
+      String idTokenValue =
+          ((IdTokenProvider) credentials)
+              .idTokenWithAudience(iapClientId.get(), ImmutableList.of(Option.FORMAT_FULL))
+              .getTokenValue();
+      request.headers().set("proxy-authorization", "Bearer " + idTokenValue);
+    }
+  }
 }
@@ -41,7 +41,7 @@ class WhoisProtocolModuleTest extends ProtocolModuleTest {
  }

  @Test
-  void testSuccess_singleFrameInboundMessage() {
+  void testSuccess_singleFrameInboundMessage() throws Exception {
    String inputString = "test.tld\r\n";
    // Inbound message processed and passed along.
    assertThat(channel.writeInbound(Unpooled.wrappedBuffer(inputString.getBytes(US_ASCII))))
@@ -53,7 +53,8 @@ class WhoisProtocolModuleTest extends ProtocolModuleTest {
            "test.tld",
            PROXY_CONFIG.whois.relayHost,
            PROXY_CONFIG.whois.relayPath,
-            TestModule.provideFakeAccessToken().get());
+            TestModule.provideFakeCredentials().get(),
+            TestModule.provideIapClientId());
    assertThat(actualRequest).isEqualTo(expectedRequest);
    assertThat(channel.isActive()).isTrue();
    // Nothing more to read.
@@ -70,7 +71,7 @@ class WhoisProtocolModuleTest extends ProtocolModuleTest {
  }

  @Test
-  void testSuccess_multiFrameInboundMessage() {
+  void testSuccess_multiFrameInboundMessage() throws Exception {
    String frame1 = "test";
    String frame2 = "1.tld";
    String frame3 = "\r\nte";
@@ -88,7 +89,8 @@ class WhoisProtocolModuleTest extends ProtocolModuleTest {
            "test1.tld",
            PROXY_CONFIG.whois.relayHost,
            PROXY_CONFIG.whois.relayPath,
-            TestModule.provideFakeAccessToken().get());
+            TestModule.provideFakeCredentials().get(),
+            TestModule.provideIapClientId());
    assertThat(actualRequest1).isEqualTo(expectedRequest1);
    // No more message at this point.
    assertThat((Object) channel.readInbound()).isNull();
@@ -102,7 +104,8 @@ class WhoisProtocolModuleTest extends ProtocolModuleTest {
            "test2.tld",
            PROXY_CONFIG.whois.relayHost,
            PROXY_CONFIG.whois.relayPath,
-            TestModule.provideFakeAccessToken().get());
+            TestModule.provideFakeCredentials().get(),
+            TestModule.provideIapClientId());
    assertThat(actualRequest2).isEqualTo(expectedRequest2);
    // The third message is not complete yet.
    assertThat(channel.isActive()).isTrue();
@@ -25,8 +25,14 @@ import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;

+import com.google.auth.oauth2.AccessToken;
+import com.google.auth.oauth2.ComputeEngineCredentials;
+import com.google.auth.oauth2.IdToken;
+import com.google.auth.oauth2.IdTokenProvider.Option;
 import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableList;
 import google.registry.proxy.TestUtils;
 import google.registry.proxy.handler.HttpsRelayServiceHandler.NonOkHttpResponseException;
 import google.registry.proxy.metric.FrontendMetrics;
@@ -44,7 +50,9 @@ import io.netty.handler.codec.http.HttpResponseStatus;
 import io.netty.handler.codec.http.cookie.Cookie;
 import io.netty.handler.codec.http.cookie.DefaultCookie;
 import io.netty.util.concurrent.Promise;
+import java.io.IOException;
 import java.security.cert.X509Certificate;
+import java.util.Optional;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;

@@ -59,9 +67,12 @@ class EppServiceHandlerTest {

  private static final String RELAY_HOST = "registry.example.tld";
  private static final String RELAY_PATH = "/epp";
-  private static final String ACCESS_TOKEN = "this.access.token";
  private static final String CLIENT_ADDRESS = "epp.client.tld";
  private static final String PROTOCOL = "epp";
+  private static final String IAP_CLIENT_ID = "iapClientId";
+
+  private static final ComputeEngineCredentials mockCredentials =
+      mock(ComputeEngineCredentials.class);

  private X509Certificate clientCertificate;

@@ -69,7 +80,12 @@ class EppServiceHandlerTest {

  private final EppServiceHandler eppServiceHandler =
      new EppServiceHandler(
-          RELAY_HOST, RELAY_PATH, () -> ACCESS_TOKEN, HELLO.getBytes(UTF_8), metrics);
+          RELAY_HOST,
+          RELAY_PATH,
+          () -> mockCredentials,
+          Optional.of(IAP_CLIENT_ID),
+          HELLO.getBytes(UTF_8),
+          metrics);

  private EmbeddedChannel channel;

@@ -79,7 +95,7 @@ class EppServiceHandlerTest {
        channel.attr(CLIENT_CERTIFICATE_PROMISE_KEY).get().setSuccess(certificate);
  }

-  private void setHandshakeSuccess() throws Exception {
+  private void setHandshakeSuccess() {
    setHandshakeSuccess(channel, clientCertificate);
  }

@@ -91,23 +107,29 @@ class EppServiceHandlerTest {
            .setFailure(new Exception("Handshake Failure"));
  }

-  private void setHandshakeFailure() throws Exception {
+  private void setHandshakeFailure() {
    setHandshakeFailure(channel);
  }

-  private FullHttpRequest makeEppHttpRequest(String content, Cookie... cookies) {
+  private FullHttpRequest makeEppHttpRequest(String content, Cookie... cookies) throws IOException {
    return TestUtils.makeEppHttpRequest(
        content,
        RELAY_HOST,
        RELAY_PATH,
-        ACCESS_TOKEN,
+        mockCredentials,
        getCertificateHash(clientCertificate),
        CLIENT_ADDRESS,
+        Optional.of(IAP_CLIENT_ID),
        cookies);
  }

  @BeforeEach
  void beforeEach() throws Exception {
+    when(mockCredentials.getAccessToken()).thenReturn(new AccessToken("this.access.token", null));
+    IdToken mockIdToken = mock(IdToken.class);
+    when(mockIdToken.getTokenValue()).thenReturn("fake.test.id.token");
+    when(mockCredentials.idTokenWithAudience(IAP_CLIENT_ID, ImmutableList.of(Option.FORMAT_FULL)))
+        .thenReturn(mockIdToken);
    clientCertificate = SelfSignedCaCertificate.create().cert();
    channel = setUpNewChannel(eppServiceHandler);
  }
@@ -140,10 +162,15 @@ class EppServiceHandlerTest {
    String certHash = getCertificateHash(clientCertificate);
    assertThat(channel.isActive()).isTrue();

-    // Setup the second channel.
+    // Set up the second channel.
    EppServiceHandler eppServiceHandler2 =
        new EppServiceHandler(
-            RELAY_HOST, RELAY_PATH, () -> ACCESS_TOKEN, HELLO.getBytes(UTF_8), metrics);
+            RELAY_HOST,
+            RELAY_PATH,
+            () -> mockCredentials,
+            Optional.empty(),
+            HELLO.getBytes(UTF_8),
+            metrics);
    EmbeddedChannel channel2 = setUpNewChannel(eppServiceHandler2);
    setHandshakeSuccess(channel2, clientCertificate);

@@ -160,10 +187,15 @@ class EppServiceHandlerTest {
    String certHash = getCertificateHash(clientCertificate);
    assertThat(channel.isActive()).isTrue();

-    // Setup the second channel.
+    // Set up the second channel.
    EppServiceHandler eppServiceHandler2 =
        new EppServiceHandler(
-            RELAY_HOST, RELAY_PATH, () -> ACCESS_TOKEN, HELLO.getBytes(UTF_8), metrics);
+            RELAY_HOST,
+            RELAY_PATH,
+            () -> mockCredentials,
+            Optional.empty(),
+            HELLO.getBytes(UTF_8),
+            metrics);
    EmbeddedChannel channel2 = setUpNewChannel(eppServiceHandler2);
    X509Certificate clientCertificate2 = SelfSignedCaCertificate.create().cert();
    setHandshakeSuccess(channel2, clientCertificate2);
@@ -326,4 +358,38 @@ class EppServiceHandlerTest {
    assertThat((Object) channel.readOutbound()).isNull();
    assertThat(channel.isActive()).isTrue();
  }
+
+  @Test
+  void testSuccess_withoutIapClientId() throws Exception {
+    // Without an IAP client ID configured, we shouldn't include the proxy-authorization header
+    EppServiceHandler nonIapServiceHandler =
+        new EppServiceHandler(
+            RELAY_HOST,
+            RELAY_PATH,
+            () -> mockCredentials,
+            Optional.empty(),
+            HELLO.getBytes(UTF_8),
+            metrics);
+    channel = setUpNewChannel(nonIapServiceHandler);
+
+    setHandshakeSuccess();
+    // First inbound message is hello.
+    channel.readInbound();
+    String content = "<epp>stuff</epp>";
+    channel.writeInbound(Unpooled.wrappedBuffer(content.getBytes(UTF_8)));
+    FullHttpRequest request = channel.readInbound();
+    assertThat(request)
+        .isEqualTo(
+            TestUtils.makeEppHttpRequest(
+                content,
+                RELAY_HOST,
+                RELAY_PATH,
+                mockCredentials,
+                getCertificateHash(clientCertificate),
+                CLIENT_ADDRESS,
+                Optional.empty()));
+    // Nothing further to pass to the next handler.
+    assertThat((Object) channel.readInbound()).isNull();
+    assertThat(channel.isActive()).isTrue();
+  }
 }
@@ -22,8 +22,14 @@ import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;

+import com.google.auth.oauth2.AccessToken;
+import com.google.auth.oauth2.ComputeEngineCredentials;
+import com.google.auth.oauth2.IdToken;
+import com.google.auth.oauth2.IdTokenProvider.Option;
 import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableList;
 import google.registry.proxy.handler.HttpsRelayServiceHandler.NonOkHttpResponseException;
 import google.registry.proxy.metric.FrontendMetrics;
 import io.netty.buffer.ByteBuf;
@@ -34,6 +40,7 @@ import io.netty.handler.codec.EncoderException;
 import io.netty.handler.codec.http.FullHttpRequest;
 import io.netty.handler.codec.http.FullHttpResponse;
 import io.netty.handler.codec.http.HttpResponseStatus;
+import java.util.Optional;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;

@@ -43,18 +50,26 @@ class WhoisServiceHandlerTest {
  private static final String RELAY_HOST = "www.example.tld";
  private static final String RELAY_PATH = "/test";
  private static final String QUERY_CONTENT = "test.tld";
-  private static final String ACCESS_TOKEN = "this.access.token";
  private static final String PROTOCOL = "whois";
  private static final String CLIENT_HASH = "none";
+  private static final String IAP_CLIENT_ID = "iapClientId";

+  private static final ComputeEngineCredentials mockCredentials =
+      mock(ComputeEngineCredentials.class);
  private final FrontendMetrics metrics = mock(FrontendMetrics.class);

  private final WhoisServiceHandler whoisServiceHandler =
-      new WhoisServiceHandler(RELAY_HOST, RELAY_PATH, () -> ACCESS_TOKEN, metrics);
+      new WhoisServiceHandler(
+          RELAY_HOST, RELAY_PATH, () -> mockCredentials, Optional.of(IAP_CLIENT_ID), metrics);
  private EmbeddedChannel channel;

  @BeforeEach
-  void beforeEach() {
+  void beforeEach() throws Exception {
+    when(mockCredentials.getAccessToken()).thenReturn(new AccessToken("this.access.token", null));
+    IdToken mockIdToken = mock(IdToken.class);
+    when(mockIdToken.getTokenValue()).thenReturn("fake.test.id.token");
+    when(mockCredentials.idTokenWithAudience(IAP_CLIENT_ID, ImmutableList.of(Option.FORMAT_FULL)))
+        .thenReturn(mockIdToken);
    // Need to reset metrics for each test method, since they are static fields on the class and
    // shared between each run.
    channel = new EmbeddedChannel(whoisServiceHandler);
@@ -74,7 +89,8 @@ class WhoisServiceHandlerTest {

    // Setup second channel.
    WhoisServiceHandler whoisServiceHandler2 =
-        new WhoisServiceHandler(RELAY_HOST, RELAY_PATH, () -> ACCESS_TOKEN, metrics);
+        new WhoisServiceHandler(
+            RELAY_HOST, RELAY_PATH, () -> mockCredentials, Optional.empty(), metrics);
    EmbeddedChannel channel2 =
        // We need a new channel id so that it has a different hash code.
        // This only is needed for EmbeddedChannel because it has a dummy hash code implementation.
@@ -85,10 +101,11 @@ class WhoisServiceHandlerTest {
  }

  @Test
-  void testSuccess_fireInboundHttpRequest() {
+  void testSuccess_fireInboundHttpRequest() throws Exception {
    ByteBuf inputBuffer = Unpooled.wrappedBuffer(QUERY_CONTENT.getBytes(US_ASCII));
    FullHttpRequest expectedRequest =
-        makeWhoisHttpRequest(QUERY_CONTENT, RELAY_HOST, RELAY_PATH, ACCESS_TOKEN);
+        makeWhoisHttpRequest(
+            QUERY_CONTENT, RELAY_HOST, RELAY_PATH, mockCredentials, Optional.of(IAP_CLIENT_ID));
    // Input data passed to next handler
    assertThat(channel.writeInbound(inputBuffer)).isTrue();
    FullHttpRequest inputRequest = channel.readInbound();
@@ -111,6 +128,27 @@ class WhoisServiceHandlerTest {
    assertThat(channel.isActive()).isFalse();
  }

+  @Test
+  void testSuccess_withoutIapClientId() throws Exception {
+    // Without an IAP client ID configured, we shouldn't include the proxy-authorization header
+    WhoisServiceHandler nonIapHandler =
+        new WhoisServiceHandler(
+            RELAY_HOST, RELAY_PATH, () -> mockCredentials, Optional.empty(), metrics);
+    channel = new EmbeddedChannel(nonIapHandler);
+
+    ByteBuf inputBuffer = Unpooled.wrappedBuffer(QUERY_CONTENT.getBytes(US_ASCII));
+    FullHttpRequest expectedRequest =
+        makeWhoisHttpRequest(
+            QUERY_CONTENT, RELAY_HOST, RELAY_PATH, mockCredentials, Optional.empty());
+    // Input data passed to next handler
+    assertThat(channel.writeInbound(inputBuffer)).isTrue();
+    FullHttpRequest inputRequest = channel.readInbound();
+    assertThat(inputRequest).isEqualTo(expectedRequest);
+    // The channel is still open, and nothing else is to be read from it.
+    assertThat((Object) channel.readInbound()).isNull();
+    assertThat(channel.isActive()).isTrue();
+  }
+
  @Test
  void testFailure_OutboundHttpResponseNotOK() {
    String outputString = "line1\r\nline2\r\n";
@@ -31,7 +31,7 @@ RUN apt-get update -y \
    && locale-gen en_US.UTF-8 \
    && apt-get install apt-utils -y \
    && apt-get upgrade -y \
-    && apt-get install openjdk-11-jdk-headless curl -y
+    && apt-get install openjdk-11-jdk-headless curl procps -y

 # Get netstat, used for checking Cloud SQL proxy readiness.
 RUN apt-get install net-tools
@@ -0,0 +1,41 @@
+// Copyright 2023 The Nomulus Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package google.registry.util;
+
+import com.google.gson.TypeAdapter;
+import com.google.gson.stream.JsonReader;
+import com.google.gson.stream.JsonWriter;
+import java.io.IOException;
+import java.util.Objects;
+import org.joda.time.DateTime;
+import org.joda.time.format.ISODateTimeFormat;
+
+/** GSON type adapter for Joda {@link DateTime} objects. */
+public class DateTimeTypeAdapter extends TypeAdapter<DateTime> {
+
+  @Override
+  public void write(JsonWriter out, DateTime value) throws IOException {
+    out.value(Objects.toString(value));
+  }
+
+  @Override
+  public DateTime read(JsonReader in) throws IOException {
+    String stringValue = in.nextString();
+    if (stringValue.equals("null")) {
+      return null;
+    }
+    return ISODateTimeFormat.dateTime().withZoneUTC().parseDateTime(stringValue);
+  }
+}
@@ -16,6 +16,8 @@ package google.registry.util;

 import com.google.appengine.api.modules.ModulesService;
 import com.google.appengine.api.modules.ModulesServiceFactory;
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
 import dagger.Binds;
 import dagger.Module;
 import dagger.Provides;
@@ -25,6 +27,7 @@ import java.security.SecureRandom;
 import java.util.Random;
 import javax.inject.Named;
 import javax.inject.Singleton;
+import org.joda.time.DateTime;

 /** Dagger module to provide instances of various utils classes. */
@Module
@@ -83,4 +86,13 @@ public abstract class UtilsModule {
  public static StringGenerator provideDigitsOnlyStringGenerator(SecureRandom secureRandom) {
    return new RandomStringGenerator(StringGenerator.Alphabets.DIGITS_ONLY, secureRandom);
  }
+
+  @Singleton
+  @Provides
+  public static Gson provideGson() {
+    return new GsonBuilder()
+        .registerTypeAdapter(DateTime.class, new DateTimeTypeAdapter())
+        .excludeFieldsWithoutExposeAnnotation()
+        .create();
+  }
 }
Author	SHA1	Message	Date
Lai Jiang	ff39a4a763	Change default beam job region (#1937 ) For reasons that I cannot explain, the same expand recurring billing event pipeline would fail in us-east1 but succeed in us-central1. See: https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-02-09_14_52_24-162498476138221714;graphView=0?project=domain-registry https://pantheon.corp.google.com/dataflow/jobs/us-east1/2023-02-09_14_26_07-4564782062878841960;graphView=1?project=domain-registry Also improved how the accuracy of the metrics: It is observed that both counters are consistently higher for the same start and end times when running in dry run mode. There is no way to test for consistency when not running in dry run, for obviously reasons. I can make the recurrings in scope counter consistent by not updating it in a side-effect-causing transaction, but there is no way around the other counter. It can only be trusted when running in dry run mode, unfortunately.	2023-02-13 15:57:32 -05:00
gbrodman	b1cd8c5a6f	Add a frontend endpoint for retrieving a domain in JSON form (#1916 ) We might (likely will) modify some of the fiddly bits around this (maybe the GSON serialization, where we do the actual authorization, etc) but this should be a decent basic shell structure for endpoints that the new registrar console can call to retrieve JSON results.	2023-02-09 15:09:42 -05:00
gbrodman	28c7bc3085	Generate and use an IAP-enabled ID token in the proxy (#1926 ) This is only generated and used if "iapClientId" is set in the proxy config. If so, we use code similar to https://cloud.google.com/iap/docs/authentication-howto#obtaining_an_oidc_token_for_the_default_service_account to generate an ID token that is valid for IAP. We set the token on the Proxy-Authorization header so that we can keep using the pre-existing access token as well -- IAP allows for us to use either the Authorization header or the Proxy-Authorization header.	2023-02-09 14:50:35 -05:00
gbrodman	f36d22f4b1	Allow null GAIA IDs for User objects (#1933 ) We were under the mistaken impression before that there was a reliable way to, out-of-band, get a GAIA ID for a particular email address. Unfortunately, that isn't the case (at least, not in a scalable way or one that support agents could use). As a result, we have to allow null GAIA IDs in the database. When we (or the support team) create new users, we will only specify the email address and not the GAIA ID. Then, when the user logs in for the first time, we will have the GAIA ID from the provided ID token, and we can populate it then.	2023-02-08 16:10:34 -05:00
Lai Jiang	ef3ce79b8a	Install procps in schema-deployer image (#1934 ) It turns out this one uses pgrep and pkill as well, go figure... <!-- Reviewable:start --> - - - This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1934) <!-- Reviewable:end -->	2023-02-08 09:59:47 -05:00