mirrors/nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus synced 2026-08-15 11:46:08 +00:00
Code Issues Releases Wiki Activity

Update more of the documentation (#2974)

Browse Source 
We should be at least at a "good enough" state after this -- I'm sure
there are many updates we could make that would improve the
documentation but this is definitely much improved from before and
should hopefully be good enough to get people started.
This commit is contained in:
gbrodman
2026-03-03 20:25:30 +00:00
committed by GitHub
parent 25fcef8a5b
commit 72016b1e5f
6 changed files with 461 additions and 598 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/install.md
+74 -66
View File
@@ -6,48 +6,41 @@ This document covers the steps necessary to download, build, and deploy Nomulus.
You will need the following programs installed on your local machine:
* A recent version of the [Java 11 JDK][java-jdk11].
* [Google App Engine SDK for Java][app-engine-sdk], and configure aliases to the `gcloud` and `appcfg.sh` utilities (
you'll use them a lot).
* [Git](https://git-scm.com/) version control system.
* Docker (confirm with `docker info` no permission issues, use `sudo groupadd docker` for sudoless docker).
* Python version 3.7 or newer.
* gnupg2 (e.g. in run `sudo apt install gnupg2` in Debian-like Linuxes)
* A recent version of the [Java 21 JDK][java-jdk21].
* The [Google Cloud CLI](https://docs.cloud.google.com/sdk/docs/install-sdk)
(configure an alias to the `gcloud`utility, because you'll use it a lot)
* [Git](https://git-scm.com/) version control system.
* Docker (confirm with `docker info` no permission issues, use `sudo groupadd
docker` for sudoless docker).
* Python version 3.7 or newer.
* gnupg2 (e.g. in run `sudo apt install gnupg2` in Debian-like Linuxes)
**Note:** App Engine does not yet support Java 9. Also, the instructions in this
document have only been tested on Linux. They might work with some alterations
on other operating systems.
**Note:** The instructions in this document have only been tested on Linux. They
might work with some alterations on other operating systems.
## Download the codebase
Start off by using git to download the latest version from the [Nomulus GitHub
page](https://github.com/google/nomulus). You may checkout any of the daily
tagged versions (e.g. `nomulus-20200629-RC00`), but in general it is also
safe to simply checkout from HEAD:
Start off by using git to download the latest version from the
[Nomulus GitHub page](https://github.com/google/nomulus). You may check out any
of the daily tagged versions (e.g. `nomulus-20260101-RC00`), but in general it
is also safe to simply check out from HEAD:
```shell
$ git clone git@github.com:google/nomulus.git
Cloning into 'nomulus'...
[ .. snip .. ]
$ cd nomulus
$ ls
apiserving CONTRIBUTORS java LICENSE scripts
AUTHORS docs javascript python third_party
CONTRIBUTING.md google javatests README.md WORKSPACE
```
Most of the directory tree is organized into gradle sub-projects (see
`settings.gradle` for details). The following other top-level directories are
Most of the directory tree is organized into gradle subprojects (see
`settings.gradle` for details). The following other top-level directories are
also defined:
* `buildSrc` -- Gradle extensions specific to our local build and release
methodology.
* `config` -- Tools for build and code hygiene.
* `docs` -- The documentation (including this install guide)
* `gradle` -- Configuration and code managed by the gradle build system.
* `gradle` -- Configuration and code managed by the Gradle build system.
* `integration` -- Testing scripts for SQL changes.
* `java-format` -- The Google java formatter and wrapper scripts to use it
incrementally.
* `python` -- Some Python reporting scripts
* `release` -- Configuration for our continuous integration process.
## Build the codebase
@@ -56,34 +49,29 @@ The first step is to build the project, and verify that this completes
successfully. This will also download and install dependencies.
```shell
$ ./nom_build build
$ ./gradlew build
Starting a Gradle Daemon (subsequent builds will be faster)
Plugins: Using default repo...
> Configure project :buildSrc
Java dependencies: Using Maven central...
[ .. snip .. ]
```
The `nom_build` script is just a wrapper around `gradlew`. Its main
additional value is that it formalizes the various properties used in the
build as command-line flags.
The "build" command builds all the code and runs all the tests. This will take a
while.
The "build" command builds all of the code and runs all of the tests. This
will take a while.
## Create and configure a GCP project
## Create an App Engine project
First, [create an
application](https://cloud.google.com/appengine/docs/java/quickstart) on Google
Cloud Platform. Make sure to choose a good Project ID, as it will be used
repeatedly in a large number of places. If your company is named Acme, then a
good Project ID for your production environment would be "acme-registry". Keep
First,
[create an application](https://cloud.google.com/appengine/docs/java/quickstart)
on Google Cloud Platform. Make sure to choose a good Project ID, as it will be
used repeatedly in a large number of places. If your company is named Acme, then
a good Project ID for your production environment would be "acme-registry". Keep
in mind that project IDs for non-production environments should be suffixed with
the name of the environment (see the [Architecture
documentation](./architecture.md) for more details). For the purposes of this
example we'll deploy to the "alpha" environment, which is used for developer
testing. The Project ID will thus be `acme-registry-alpha`.
the name of the environment (see the
[Architecture documentation](./architecture.md) for more details). For the
purposes of this example we'll deploy to the "alpha" environment, which is used
for developer testing. The Project ID will thus be `acme-registry-alpha`.
Now log in using the command-line Google Cloud Platform SDK and set the default
project to be this one that was newly created:
@@ -96,6 +84,17 @@ You are now logged in as [user@email.tld].
$ gcloud config set project acme-registry-alpha
```
And make sure the required APIs are enabled in the project:
```shell
$ gcloud services enable \
container.googleapis.com \
artifactregistry.googleapis.com \
sqladmin.googleapis.com \
secretmanager.googleapis.com \
compute.googleapis.com
```
Now modify `projects.gradle` with the name of your new project:
<pre>
@@ -106,42 +105,51 @@ rootProject.ext.projects = ['production': 'your-production-project',
'crash' : 'your-crash-project']
</pre>
Next follow the steps in [configuration](./configuration.md) to configure the
complete system or, alternately, read on for an initial deploy in which case
you'll need to deploy again after configuration.
#### Create GKE Clusters
## Deploy the code to App Engine
We recommend Standard clusters with Workload Identity enabled to allow pods to
securely access Cloud SQL and Secret Manager. Feel free to adjust the numbers
and sizing as desired.
AppEngine deployment with gradle is straightforward:
```shell
$ gcloud container clusters create nomulus-cluster \
--region=$REGION \
--workload-pool=$PROJECT_ID.svc.id.goog \
--num-nodes=3 \
--enable-ip-alias
$ gcloud container clusters create proxy-cluster \
--region=$REGION \
--workload-pool=$PROJECT_ID.svc.id.goog \
--num-nodes=3 \
--enable-ip-alias
```
$ ./nom_build appengineDeploy --environment=alpha
Then create an artifact repository: `shell $ gcloud artifacts repositories
create nomulus-repo \ --repository-format=docker \ --location=$REGION \
--description="Nomulus Docker images"`
To verify successful deployment, visit
https://acme-registry-alpha.appspot.com/registrar in your browser (adjusting
appropriately for the project ID that you actually used). If the project
deployed successfully, you'll see a "You need permission" page indicating that
you need to configure the system and grant access to your Google account. It's
time to go to the next step, configuration.
See the files and documentation in the `release/` folder for more information on
the release process. You will likely need to customize the internal build
process for your own setup, including internal repository management, builds,
and where Nomulus is deployed.
Configuration is handled by editing code, rebuilding the project, and deploying
again. See the [configuration guide](./configuration.md) for more details.
Once you have completed basic configuration (including most critically the
project ID, client id and secret in your copy of the `nomulus-config-*.yaml`
files), you can rebuild and start using the `nomulus` tool to create test
entities in your newly deployed system. See the [first steps tutorial](./first-steps-tutorial.md)
again. See the [configuration guide](./configuration.md) for more details. Once
you have completed basic configuration (including most critically the project
ID, client id and secret in your copy of the `nomulus-config-*.yaml` files), you
can rebuild and start using the `nomulus` tool to create test entities in your
newly deployed system. See the [first steps tutorial](./first-steps-tutorial.md)
for more information.
[app-engine-sdk]: https://cloud.google.com/appengine/docs/java/download
[java-jdk11]: https://www.oracle.com/java/technologies/javase-downloads.html
[java-jdk21]: https://www.oracle.com/java/technologies/javase-downloads.html
## Deploy the BEAM Pipelines
## Deploy the Beam Pipelines
Nomulus is in the middle of migrating all pipelines to use flex-template. For
pipelines already based on flex-template, deployment in the testing environments
Deployment of the Beam pipelines to Cloud Dataflow in the testing environments
(alpha and crash) can be done using the following command:
```shell
./nom_build :core:stageBeamPipelines --environment=alpha
./gradlew :core:stageBeamPipelines -Penvironment=alpha
```
Pipeline deployment in other environments are through CloudBuild. Please refer