e2e: Add multi-version support to generator (#9933)

* ci: Add missing group to manual E2E workflow

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Improve help for multi-version CLI param

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Rename multiversion to multiVersion

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Extract generator params into struct

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Add support for more than 2 versions in multi-version tests

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Add generator logic to extract latest release tag from Git repo

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Cater for "dev-" prefixes in tags

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Add sophisticated weighted version parsing to generator

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Add note to E2E readme about multi-version testing

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Expand on Docker image comment in readme

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Print node version weights when generating testnets

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* ci: Add manual E2E workflow to run multi-version tests

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Replace "HEAD" keyword with "local"

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Update readme to reflect "HEAD" -> "local" keyword change and expand on examples

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Clarify comment relating to Docker image versions

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Apply suggestions from code review

Co-authored-by: Sergio Mena <sergio@informal.systems>

* e2e: Skip invalid tags instead of erroring in generator

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Add comment in version selection code

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* ci: Replace "HEAD" with "local" keyword in multiversion workflow

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* e2e: Use error format specifier when returning wrapped error

Signed-off-by: Thane Thomson <connect@thanethomson.com>

Signed-off-by: Thane Thomson <connect@thanethomson.com>
Co-authored-by: Sergio Mena <sergio@informal.systems>

.github/workflows/e2e-manual-multiversion.yml

@@ -0,0 +1,38 @@
+# Manually run the nightly E2E tests for a particular branch, but test with
+# multiple versions.
+name: e2e-manual-multiversion
+on:
+  workflow_dispatch:
+
+jobs:
+  e2e-manual-multiversion-test:
+    # Run parallel jobs for the listed testnet groups (must match the
+    # ./build/generator -g flag)
+    strategy:
+      fail-fast: false
+      matrix:
+        group: ['00', '01', '02', '03', '04']
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/setup-go@v3
+        with:
+          go-version: '1.18'
+
+      - uses: actions/checkout@v3
+
+      - name: Build
+        working-directory: test/e2e
+        # Run make jobs in parallel, since we can't run steps in parallel.
+        run: make -j2 docker generator runner tests
+
+      - name: Generate testnets
+        working-directory: test/e2e
+        # When changing -g, also change the matrix groups above
+        # Generate multi-version tests with double the quantity of E2E nodes
+        # based on the current branch as compared to the latest version.
+        run: ./build/generator -g 5 -m "latest:1,local:2" -d networks/nightly/
+
+      - name: Run ${{ matrix.p2p }} p2p testnets
+        working-directory: test/e2e
+        run: ./run-multiple.sh networks/nightly/*-group${{ matrix.group }}-*.toml

.github/workflows/e2e-manual.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        group: ['00', '01', '02', '03']
+        group: ['00', '01', '02', '03', '04']
     runs-on: ubuntu-latest
     timeout-minutes: 60
     steps: