mirrors/nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus synced 2026-09-20 06:54:43 +00:00
Code Issues Releases Wiki Activity

Update more of the documentation (#2974)

Browse Source 
We should be at least at a "good enough" state after this -- I'm sure
there are many updates we could make that would improve the
documentation but this is definitely much improved from before and
should hopefully be good enough to get people started.
This commit is contained in:
gbrodman
2026-03-03 20:25:30 +00:00
committed by GitHub
parent 25fcef8a5b
commit 72016b1e5f
6 changed files with 461 additions and 598 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+95 -161
View File
@@ -1,153 +1,97 @@
# Architecture
This document contains information on the overall architecture of Nomulus on
[Google Cloud Platform](https://cloud.google.com/). It covers the App Engine
architecture as well as other Cloud Platform services used by Nomulus.
[Google Cloud Platform](https://cloud.google.com/).
## App Engine
Nomulus was originally built for App Engine, but the modern architecture now
uses Google Kubernetes Engine (GKE) for better flexibility and control over
networking, running as a series of Java-based microservices within GKE pods.
[Google App Engine](https://cloud.google.com/appengine/) is a cloud computing
platform that runs web applications in the form of servlets. Nomulus consists of
Java servlets that process web requests. These servlets use other features
provided by App Engine, including task queues and cron jobs, as explained
below.
In addition, because GKE (and standard HTTP load balancers) typically handle
HTTP(s) traffic, Nomulus uses a custom proxy to handle raw TCP traffic required
for EPP (Port 700). This proxy can run as a GKE sidecar or a standalone cluster.
For more information on the proxy, see [the proxy setup guide](proxy-setup.md).
### Services
### Workloads
Nomulus contains three [App Engine
services](https://cloud.google.com/appengine/docs/python/an-overview-of-app-engine),
which were previously called modules in earlier versions of App Engine. The
services are: default (also called front-end), backend, and tools. Each service
runs independently in a lot of ways, including that they can be upgraded
individually, their log outputs are separate, and their servers and configured
scaling are separate as well.
Nomulus contains four Kubernetes
[workloads](https://kubernetes.io/docs/concepts/workloads/). Each workload is
fairly independent as one would expect, including scaling.
Once you have your app deployed and running, the default service can be accessed
at `https://project-id.appspot.com`, substituting whatever your App Engine app
is named for "project-id". Note that that is the URL for the production instance
of your app; other environments will have the environment name appended with a
hyphen in the hostname, e.g. `https://project-id-sandbox.appspot.com`.
The four workloads are referred to as `frontend`, `backend`, `console`, and
`pubapi`.
The URL for the backend service is `https://backend-dot-project-id.appspot.com`
and the URL for the tools service is `https://tools-dot-project-id.appspot.com`.
The reason that the dot is escaped rather than forming subdomains is because the
SSL certificate for `appspot.com` is only valid for `*.appspot.com` (no double
wild-cards).
Each workload's URL is created by prefixing the name of the workload to the base
domain, e.g. `https://pubapi.mydomain.example`. Requests to each workload are
all handled by the
[RegistryServlet](https://github.com/google/nomulus/blob/master/core/src/main/java/google/registry/module/RegistryServlet.java)
#### Default service
#### Frontend workload
The default service is responsible for all registrar-facing
The frontend workload is responsible for all registrar-facing
[EPP](https://en.wikipedia.org/wiki/Extensible_Provisioning_Protocol) command
traffic, all user-facing WHOIS and RDAP traffic, and the admin and registrar web
consoles, and is thus the most important service. If the service has any
problems and goes down or stops servicing requests in a timely manner, it will
begin to impact users immediately. Requests to the default service are handled
by the `FrontendServlet`, which provides all of the endpoints exposed in
`FrontendRequestComponent`.
traffic. If the workload has any problems or goes down, it will begin to impact
users immediately.
#### Backend service
#### PubApi workload
The backend service is responsible for executing all regularly scheduled
background tasks (using cron) as well as all asynchronous tasks. Requests to the
backend service are handled by the `BackendServlet`, which provides all of the
endpoints exposed in `BackendRequestComponent`. These include tasks for
generating/exporting RDE, syncing the trademark list from TMDB, exporting
backups, writing out DNS updates, handling asynchronous contact and host
deletions, writing out commit logs, exporting metrics to BigQuery, and many
more. Issues in the backend service will not immediately be apparent to end
users, but the longer it is down, the more obvious it will become that
user-visible tasks such as DNS and deletion are not being handled in a timely
manner.
The PubApi (Public API) workload is responsible for all public traffic to the
registry. In practice, this primarily consists of RDAP traffic. This is split
into a separate workload so that public users (without authentication) will have
a harder time impacting intra-registry or registrar-registry actions.
The backend service is also where scheduled and automatically invoked MapReduces
run, which includes some of the aforementioned tasks such as RDE and
asynchronous resource deletion. Consequently, the backend service should be
sized to support not just the normal ongoing DNS load but also the load incurred
by MapReduces, both scheduled (such as RDE) and on-demand (asynchronous
contact/host deletion).
#### Backend workload
#### BSA service
The backend workload is responsible for executing all regularly scheduled
background tasks (using cron) as well as all asynchronous tasks. These include
tasks for generating/exporting RDE, syncing the trademark list from TMDB,
exporting backups, writing out DNS updates, syncing BSA data,
generating/exporting ICANN activity data, and many more. Issues in the backend
workload will not immediately be apparent to end users, but the longer it is
down, the more obvious it will become that user-visible tasks such as DNS and
deletion are not being handled in a timely manner.
The bsa service is responsible for business logic behind Nomulus and BSA
functionality. Requests to the backend service are handled by the `BsaServlet`,
which provides all of the endpoints exposed in `BsaRequestComponent`. These
include tasks for downloading, processing and uploading BSA data.
The backend workload is also where scheduled and automatically-invoked BEAM
pipelines run, which includes some of the aforementioned tasks such as RDE.
Consequently, the backend workload should be sized to support not just the
normal ongoing DNS load but also the load incurred by BEAM pipelines, both
scheduled (such as RDE) and on-demand (started by registry employees).
The backend workload also supports handling of manually-performed actions using
the `nomulus` command-line tool, which provides administrative-level
functionality for developers and tech support employees of the registry.
#### Tools service
### Cloud Tasks queues
The tools service is responsible for servicing requests from the `nomulus`
command line tool, which provides administrative-level functionality for
developers and tech support employees of the registry. It is thus the least
critical of the three services. Requests to the tools service are handled by the
`ToolsServlet`, which provides all of the endpoints exposed in
`ToolsRequestComponent`. Some example functionality that this service provides
includes the server-side code to update premium lists, run EPP commands from the
tool, and manually modify contacts/hosts/domains/and other resources. Problems
with the tools service are not visible to users.
The tools service also runs ad-hoc MapReduces, like those invoked via `nomulus`
tool subcommands like `generate_zone_files` and by manually hitting URLs under
https://tools-dot-project-id.appspot.com, like
`/_dr/task/refreshDnsForAllDomains`.
### Task queues
App Engine [task
queues](https://cloud.google.com/appengine/docs/java/taskqueue/) provide an
GCP's [Cloud Tasks](https://docs.cloud.google.com/tasks/docs) provides an
asynchronous way to enqueue tasks and then execute them on some kind of
schedule. There are two types of queues, push queues and pull queues. Tasks in
push queues are always executing up to some throttlable limit. Tasks in pull
queues remain there until the queue is polled by code that is running for some
other reason. Essentially, push queues run their own tasks while pull queues
just enqueue data that is used by something else. Many other parts of App Engine
are implemented using task queues. For example, [App Engine
cron](https://cloud.google.com/appengine/docs/java/config/cron) adds tasks to
push queues at regularly scheduled intervals, and the [MapReduce
framework](https://cloud.google.com/appengine/docs/java/dataprocessing/) adds
tasks for each phase of the MapReduce algorithm.
schedule. Task queues are essential because by nature, GKE architecture does not
support long-running background processes, and so queues are thus the
fundamental building block that allows asynchronous and background execution of
code that is not in response to incoming web requests.
Nomulus uses a particular pattern of paired push/pull queues that is worth
explaining in detail. Push queues are essential because App Engine's
architecture does not support long-running background processes, and so push
queues are thus the fundamental building block that allows asynchronous and
background execution of code that is not in response to incoming web requests.
However, they also have limitations in that they do not allow batch processing
or grouping. That's where the pull queue comes in. Regularly scheduled tasks in
the push queue will, upon execution, poll the corresponding pull queue for a
specified number of tasks and execute them in a batch. This allows the code to
execute in the background while taking advantage of batch processing.
The task queues used by Nomulus are configured in the `cloud-tasks-queue.xml`
file. Note that many push queues have a direct one-to-one correspondence with
entries in `cloud-scheduler-tasks-ENVIRONMENT.xml` because they need to be
fanned-out on a per-TLD or other basis (see the Cron section below for more
explanation). The exact queue that a given cron task will use is passed as the
query string parameter "queue" in the url specification for the cron task.
The task queues used by Nomulus are configured in the `cloud-tasks-queue.xml`
file. Note that many push queues have a direct one-to-one correspondence with
entries in `cloud-scheduler-tasks.xml` because they need to be fanned-out on a
per-TLD or other basis (see the Cron section below for more explanation).
The exact queue that a given cron task will use is passed as the query string
parameter "queue" in the url specification for the cron task.
Here are the task queues in use by the system. All are push queues unless
explicitly marked as otherwise.
Here are the task queues in use by the system:
* `brda` -- Queue for tasks to upload weekly Bulk Registration Data Access
(BRDA) files to a location where they are available to ICANN. The
`RdeStagingReducer` (part of the RDE MapReduce) creates these tasks at the
end of generating an RDE dump.
* `dns-pull` -- A pull queue to enqueue DNS modifications. Cron regularly runs
`ReadDnsQueueAction`, which drains the queue, batches modifications by TLD,
and writes the batches to `dns-publish` to be published to the configured
`DnsWriter` for the TLD.
(BRDA) files to a location where they are available to ICANN. The RDE
pipeline creates these tasks at the end of generating an RDE dump.
* `dns-publish` -- Queue for batches of DNS updates to be pushed to DNS
writers.
* `lordn-claims` and `lordn-sunrise` -- Pull queues for handling LORDN
exports. Tasks are enqueued synchronously during EPP commands depending on
whether the domain name in question has a claims notice ID.
* `dns-refresh` -- Queues for reading and fanning out DNS refresh requests,
using the `DnsRefreshRequest` SQL table as the source of data
* `marksdb` -- Queue for tasks to verify that an upload to NORDN was
successfully received and verified. These tasks are enqueued by
`NordnUploadAction` following an upload and are executed by
`NordnVerifyAction`.
* `nordn` -- Cron queue used for NORDN exporting. Tasks are executed by
`NordnUploadAction`, which pulls LORDN data from the `lordn-claims` and
`lordn-sunrise` pull queues (above).
`NordnUploadAction`
* `rde-report` -- Queue for tasks to upload RDE reports to ICANN following
successful upload of full RDE files to the escrow provider. Tasks are
enqueued by `RdeUploadAction` and executed by `RdeReportAction`.
@@ -157,28 +101,25 @@ explicitly marked as otherwise.
* `retryable-cron-tasks` -- Catch-all cron queue for various cron tasks that
run infrequently, such as exporting reserved terms.
* `sheet` -- Queue for tasks to sync registrar updates to a Google Sheets
spreadsheet. Tasks are enqueued by `RegistrarServlet` when changes are made
to registrar fields and are executed by `SyncRegistrarsSheetAction`.
spreadsheet, done by `SyncRegistrarsSheetAction`.
### Cron jobs
### Scheduled cron jobs
Nomulus uses App Engine [cron
jobs](https://cloud.google.com/appengine/docs/java/config/cron) to run periodic
scheduled actions. These actions run as frequently as once per minute (in the
case of syncing DNS updates) or as infrequently as once per month (in the case
of RDE exports). Cron tasks are specified in `cron.xml` files, with one per
environment. There are more tasks that run in Production than in other
environments because tasks like uploading RDE dumps are only done for the live
system. Cron tasks execute on the `backend` service.
Nomulus uses [Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs) to
run periodic scheduled actions. These actions run as frequently as once per
minute (in the case of syncing DNS updates) or as infrequently as once per month
(in the case of RDE exports). Cron tasks are specified in
`cloud-scheduler-tasks-{ENVIRONMENT}.xml` files, with one per environment. There
are more tasks that run in Production than in other environments because tasks
like uploading RDE dumps are only done for the live system.
Most cron tasks use the `TldFanoutAction` which is accessed via the
`/_dr/cron/fanout` URL path. This action, which is run by the BackendServlet on
the backend service, fans out a given cron task for each TLD that exists in the
registry system, using the queue that is specified in the `cron.xml` entry.
Because some tasks may be computationally intensive and could risk spiking
system latency if all start executing immediately at the same time, there is a
`jitterSeconds` parameter that spreads out tasks over the given number of
seconds. This is used with DNS updates and commit log deletion.
`/_dr/cron/fanout` URL path. This action fans out a given cron task for each TLD
that exists in the registry system, using the queue that is specified in the XML
entry. Because some tasks may be computationally intensive and could risk
spiking system latency if all start executing immediately at the same time,
there is a `jitterSeconds` parameter that spreads out tasks over the given
number of seconds. This is used with DNS updates and commit log deletion.
The reason the `TldFanoutAction` exists is that a lot of tasks need to be done
separately for each TLD, such as RDE exports and NORDN uploads. It's simpler to
@@ -192,8 +133,7 @@ tasks retry in the face of transient errors.
The full list of URL parameters to `TldFanoutAction` that can be specified in
cron.xml is:
* `endpoint` -- The path of the action that should be executed (see
`web.xml`).
* `endpoint` -- The path of the action that should be executed
* `queue` -- The cron queue to enqueue tasks in.
* `forEachRealTld` -- Specifies that the task should be run in each TLD of
type `REAL`. This can be combined with `forEachTestTld`.
@@ -218,14 +158,14 @@ Each environment is thus completely independent.
The different environments are specified in `RegistryEnvironment`. Most
correspond to a separate App Engine app except for `UNITTEST` and `LOCAL`, which
by their nature do not use real environments running in the cloud. The
recommended naming scheme for the App Engine apps that has the best possible
compatibility with the codebase and thus requires the least configuration is to
pick a name for the production app and then suffix it for the other
environments. E.g., if the production app is to be named 'registry-platform',
then the sandbox app would be named 'registry-platform-sandbox'.
recommended project naming scheme that has the best possible compatibility with
the codebase and thus requires the least configuration is to pick a name for the
production app and then suffix it for the other environments. E.g., if the
production app is to be named 'registry-platform', then the sandbox app would be
named 'registry-platform-sandbox'.
The full list of environments supported out-of-the-box, in descending order from
real to not, is:
real to not-real, is:
* `PRODUCTION` -- The real production environment that is actually running
live TLDs. Since Nomulus is a shared registry platform, there need only ever
@@ -270,28 +210,28 @@ of experience running a production registry using this codebase.
## Cloud SQL
To be filled.
Nomulus uses [GCP Cloud SQL](https://cloud.google.com/sql) (Postgres) to store
information. For more information, see the
[DB project README file.](../db/README.md)
## Cloud Storage buckets
Nomulus uses [Cloud Storage](https://cloud.google.com/storage/) for bulk storage
of large flat files that aren't suitable for Cloud SQL. These files include
backups, RDE exports, and reports. Each bucket name must be unique across all of
Google Cloud Storage, so we use the common recommended pattern of prefixing all
buckets with the name of the App Engine app (which is itself globally unique).
Most of the bucket names are configurable, but the defaults are as follows, with
PROJECT standing in as a placeholder for the App Engine app name:
of large flat files that aren't suitable for SQL. These files include backups,
RDE exports, and reports. Each bucket name must be unique across all of Google
Cloud Storage, so we use the common recommended pattern of prefixing all buckets
with the name of the project (which is itself globally unique). Most of the
bucket names are configurable, but the most important / relevant defaults are:
* `PROJECT-billing` -- Monthly invoice files for each registrar.
* `PROJECT-commits` -- Daily exports of commit logs that are needed for
potentially performing a restore.
* `PROJECT-bsa` -- BSA data and output
* `PROJECT-domain-lists` -- Daily exports of all registered domain names per
TLD.
* `PROJECT-gcs-logs` -- This bucket is used at Google to store the GCS access
logs and storage data. This bucket is not required by the Registry system,
but can provide useful logging information. For instructions on setup, see
the [Cloud Storage
documentation](https://cloud.google.com/storage/docs/access-logs).
the
[Cloud Storage documentation](https://cloud.google.com/storage/docs/access-logs).
* `PROJECT-icann-brda` -- This bucket contains the weekly ICANN BRDA files.
There is no lifecycle expiration; we keep a history of all the files. This
bucket must exist for the BRDA process to function.
@@ -301,9 +241,3 @@ PROJECT standing in as a placeholder for the App Engine app name:
regularly uploaded to the escrow provider. Lifecycle is set to 90 days. The
bucket must exist.
* `PROJECT-reporting` -- Contains monthly ICANN reporting files.
* `PROJECT.appspot.com` -- Temporary MapReduce files are stored here. By
default, the App Engine MapReduce library places its temporary files in a
bucket named {project}.appspot.com. This bucket must exist. To keep
temporary files from building up, a 90-day or 180-day lifecycle should be
applied to the bucket, depending on how long you want to be able to go back
and debug MapReduce problems.