Lai Jiang 65c8769c68 Refactor RDE pipeline (#1427) ...

The original RDE pipeline was a direct translation of the App Engine
MapReduce logic. It turned out to be too slow (taking more than a day to
run) due to the way it finds the most recent history entry.

This PR overhauled the pipeline by using embedded EPP resource entities
inside history entries (only available in SQL) and finding the most
recent entries using the SQL engine. It cuts the time done to ~2h.

Note that there are quota limits on the CPU cores and external IP
addresses for a given GCP region inside a project, which will need to
accommodate the resource requirements for the pipeline. More details are
provided in comments.

Also merged the update cursor stage and enqueue next action stage in
RdeIO so that they can be done within a transaction, same as how
MapReduce handles them.


<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1427)
<!-- Reviewable:end -->

2021-11-23 11:29:00 -05:00

..

gradle/dependency-locks

Implement a util class to manage push queues using Cloud Tasks API (#1290)

2021-08-24 21:13:54 -04:00

src

Refactor RDE pipeline (#1427)

2021-11-23 11:29:00 -05:00

build.gradle

Make Nomulus compile on macOS (#1421)

2021-11-15 11:35:48 -05:00

Dockerfile

…