This commit relaxes the upper bounds on several dependencies that were previously hardcapped to specific versions:
- com.google.protobuf to [3.25.5,) and [3.17.3,)
- org.apache.beam to [2.72.0,)
- io.github.ss-bhatt to [1.0.0,)
- io.protostuff to [1.8.0,)
- redis.clients:jedis to [7.4.1,)
- org.junit.jupiter and org.junit.platform to [5.6.2,) and [1.6.2,)
- org.jcommander to [2.0,)
- org.jline to [3.0,)
- jakarta.servlet to [6.0,)
Upgrading to the modern versions of jline introduced a breaking change where DefaultParser().parse(line, line.length()) strips trailing spaces when using the default ParseContext.UNSPECIFIED. This caused the autocompletion to misbehave and tests to fail. This commit fixes ShellCommandTest.java by explicitly passing ParseContext.COMPLETE when parsing test strings to perfectly mimic the real-world JLine completion context.
Additionally, SqlIntegrationTestSuite was migrated to JUnit 5's @Suite annotation, fixing a NoClassDefFoundError introduced by uncapping the JUnit Platform dependencies, and the test suite was re-integrated into the standard :build lifecycle.
The following dependencies remain explicitly capped:
1. Hibernate & Jakarta Persistence (Blocked by -Werror):
These are held back because newer Jakarta Persistence versions deprecate executeUpdate(), setMaxResults(), and getResultStream() on Query.
- org.hibernate.orm:hibernate-core:7.3.4.Final
- org.hibernate.orm:hibernate-hikaricp:7.3.4.Final
- org.hibernate.orm:hibernate-ant:7.3.4.Final
- jakarta.persistence:jakarta.persistence-api:[3.2.0,4.0.0)
2. Netty (Blocked by abandoned v5):
Netty 5.0.0 was an experimental release abandoned in 2015. We explicitly cap beneath 5.0.0 so Gradle doesn't resolve dead-end alphas.
- io.netty:netty-codec-http:[4.1.59.Final, 5.0.0)!!
- io.netty:netty-codec:[4.1.59.Final, 5.0.0)!!
- io.netty:netty-common:[4.1.59.Final, 5.0.0)!!
- io.netty:netty-handler:[4.1.59.Final, 5.0.0)!!
- io.netty:netty-transport:[4.1.59.Final, 5.0.0)!!
- io.netty:netty-buffer:[4.1.59.Final, 5.0.0)!!
3. Google API Services:
Capped beneath their respective unstable beta/v1b4 versions:
- com.google.apis:google-api-services-dataflow:[v1b3-rev20240430-2.0.0, v1b4)!!
- com.google.apis:google-api-services-dns:[v1-rev20240419-2.0.0, v2beta)
The lockfiles have been fully regenerated and all test suites ran successfully against the latest available transitive versions.
- Relaxed Google Cloud and gRPC upper bounds to dynamic versions in dependencies.gradle.
- Relaxed JUnit testing framework bounds to < 6.0.0 and < 2.0.0.
- Aligned beam-runners-core-construction-java to 2.72.0!!
- Regenerated all gradle lockfiles across the project.
- Updated GEMINI.md and MEMORY.md with the optimized lockfile/dependency update workflow.
- Fixed presubmit errors in check_diff.py (added Apache 2.0 license header and fixed syntax).
- Replace deprecated Soy templates for EPP XML with JAXB models and a refined Fluent DSL.
- Migrate Spec11 and administrative emails to FreeMarker with HTML auto-escaping.
- Remove Soy compiler, Gradle tasks, and library dependencies.
- Address PR feedback regarding shadowing, version locking, and security warnings.
- Enhance tests with comprehensive XML equality assertions using Java 15 text blocks.
- Improve Javadocs and maintain strict temporal consistency using java.time.
FreeMarker replaces Soy for email templating, providing native HTML auto-escaping and allowing the removal of the complex 'soyToJava' compilation step from the build process. This significantly simplifies the build system and reduces maintenance overhead. For EPP XML, migrating to JAXB allows tool-generated commands to use the same model classes as the server-side EPP flows. This ensures that tool-generated XML is always schema-compliant and eliminates the risk of divergence between tool templates and actual server-side implementation. This unified approach provides compile-time type safety and improves developer ergonomics via a refined fluent DSL.
The base ImmutableObject class now provides a public clone() override that correctly resets the cached hashCode to null. This centralizes the custom cloning logic previously handled by a static helper and ensures that all subclasses—including the newly added JAXB models—satisfy CodeQL security requirements without needing redundant per-class overrides. The legacy static clone(T) helper has been updated to delegate to this instance method to maintain compatibility and architectural consistency.
This completes the exhaustive refactoring of foundational temporal types from Joda-Time to the native java.time API across the entire codebase.
- Replaced org.joda.time.DateTime, Instant, LocalDate, and Duration with java.time equivalents.
- Audited and updated Clock implementations (FakeClock, SystemClock). Added nowMillis(), nowDate(), and nowDateTime() to eliminate repetitive conversions and maintain parallel naming.
- Replaced ZonedDateTime with OffsetDateTime globally per go/avoid-zdt. OffsetDateTime is a better fit as we use a hardcoded ZoneOffset.UTC throughout the system, making geographical time zone rules (like daylight saving time) irrelevant and preventing serialization ambiguities. Added a presubmit check.
- Completely removed all transitional bridge methods from DateTimeUtils and deleted obsolete converters (e.g., DateTimeConverter).
- Updated testing infrastructure, Apache Beam pipelines, custom JCommander parameters, and networking modules to solely rely on java.time primitives.
- Retained the lone necessary org.joda.time.Instant usage in SafeBrowsingTransforms required by the Apache Beam API.
- Cleared Gradle lockfiles and removed the joda-time dependency entirely from the build configuration.
The fix for https://hibernate.atlassian.net/browse/HHH-20276 was
included in 7.3.4 and will be included in version 8 in the future. 8.x
is still in alpha though so we don't want to use it yet.
Error-prone introduced many more checks in Java 25. We fixed a few
and suppressed most. A follow-up bug is opened to clean this up.
An ai agent should be able to clean up most of it.
This PR is created with gemini-cli. Summary of experience:
* The good: AI caught most compatibility issues, and with permission,
suppressed them through compiler flags and errorprone options.
It also caught many versio references in scripts.
* Where it didn't shine:
- It did not find and update the target version spec in the custome
VKey annotation processor source file.
- It did not flag eclipse-temurin:21 docker image for upgrade.
- When running into failure, its first instinct is to disable checks
e.g., -Werror instead of fixing them.
* More Gradle 9 preparations
Fix additional compatibility warnings after upgrading to Gradle 8.14.3
from 8.13.
* More Gradle 9 compatibility fix
More fixes after upgrading Gradle from 8.13 to 8.14.3.
Upgraded the gradle-license-report plugin, and handled config leaking
issues.
Researched using gemini web and manually applied the fixes. Gemini-cli
could not find the right solution.
This ended up being wayyyy more complicated than expected due to
issues with Hibernate, various dependencies having conflicts with the
proto dependency version, and other breaking changes.
Notes:
- Hibernate 7 switches up the user type / converter system and for us,
this means we must be / want to be more explicit with how we convert
and store things. For example, we need to add Postgres types to @Column
definitions.
- Hibernate 7.3 has an issue with generic MappedSuperclasses -- we have
issues with BaseDomainLabelList. I'll investigate that, but for now
let's stick with 7.2.x
- H7 is more strict with annotations and prevents us from storing mapped
superclasses embedded within other objects. This kinda makes sense but
makes the History objects a bit more difficult. We had to add "concrete"
embeddable DomainBase and HostBase objects that we can store/retrieve
from the DB.
- We convert some of the calls to "Query" to "TypedQuery" -- in
Hibernate 8 / JPA 4.0 these will be super-deprecated and we'll need to
shift everything over, so this is necessary.
- You aren't supposed to put callback listeners on embedded entities
(because it can be not obvious what's happening). We don't like that,
so we add our own annotations that are processed recursively for
embedded entities, so we get things like the update / create
timestamps.
- Hibernate doesn't allow for multiple converters to be auto-applied to
the same "type" and it counts all VKey converters as one type.
Unfortunately, this means we have to explicitly mark each one.
- A bunch of other dependency changes were required to keep from having
the proto 3/4 conflict
I obtained access to an IBM s390x VM so I thought I'd see how multi-arch
Nomulus is.
Our main application is in Java so it is already multi-arch, but several
tests use docker images that are by default x64. Luckily postgres has an
s390x port, but selenium does not. So I had to disable Screenshot tests
when the arch is not amd64.
* Add log traces to Nomulus service on GKE
Add request-scope log traces to Nomulus on GKE which, unlike
AppEngine and Cloud Run etc, does not generate traces for hosted
applications. This change only affects the GKE image. It does not affect
the AppEngine services.
Log traces are added to Nomulus-generated logs in request-processing
threads. Forked threads are not covered yet. The single relevant use
case (TimeLimiter) will be addressed in a followup PR.
The main change is in the logging configuration:
* Use gcp-cloud-logging's LoggingHandler
* Add gcp-cloud-logging's TraceLoggingEnhancer to the handler.
* Set a thread-local trace id through the TraceLoggingEnhancer in
ServletBase on request's entry and clear it on completion.
Also removed an unused class (`RequestLogId`).
* CR
* CR
Note that Dagger currently doesn't work with the Jakarta namespace and
we have to cap the jakarta inject package version below 2.0 so that it
sill provides classes in the old namespace.
Make the necessary changes for the code base to compile with JDK 21.
Other changes:
1. Upgraded testcontainer version and the SQL image version (to be the
same as what we use in Cloud SQL). This led to some schema changes and
also changed the order of results in some test queries (for the
better I think, as the new order appears to be alphabetical).
2. Remove dependency on Truth8, which is deprecated.
3. Enable parallel Gradle task execution and greatly increased the
number of parallel tests in standardTest. Removed outcastTest.
This PR makes it possible to build the Nomulus code base using Java 17.
Building with Java 11 continue to be possible and the resulting bytecodes are
still at Java 8 level. Also upgraded Gradle to 8.5.
There are several necessary changes to make this happen:
1. Some Gradle plugins need to be upgraded to support Java 17, notably
errorprone. As a result, a lot more "errors" were caught and corrected.
2. All test code are now built and run at Java 8 level. Previously it was left
undefined (which defaults to the version of the compiler) and had led to
situations where we inadvertently called Java 8+ features in production that
are not caught by tests. The change also made the java8compatibility subproject
obsolete, which is therefore removed.
3. Removed the docs subproject. Its main use is to generate flows.md, but it
relies heavily on Java internal APIs that have changed significant with each
version. Upgrading to Java 11 required extensive refactoring of the code there,
and Java 17 again removed many APIs that were used. I don't think it is worth
the maintenance effort just to have a tool to generate flows.md which no one
actually reads.
4. Capped a few GCP dependencies because the latest version depends on
grpc-java >= 1.59.0, which includes a runtime incompatibility
(https://github.com/grpc/grpc-java/releases/tag/v1.59.0).
2.25.0 contains a breaking change that made HttpStorageOptions not
serializeable, which breaks RDE as it needs to access GCS from Beam.
2.22.6 was the last version that was used before the Gradle upgrade.
Also had to downgrade google-cloud-nio to pass the tests.
For some inexplicable reason, I had to manually add
guava-listenablefuture as
testRuntimeClasspath/runtimeClasspath/deploy_jar dependencies to the
networking, docs and prober subprojects' lock files, as running
`gradle test --write-locks` would NOT add them and succeed; but without
`--write-locks`, running the corresponding tests would fail.
See: b/294378137.
This includes removing (hopefully temporarily) the gradle-lint plugin as
it is incompatible with various Gradle versions (see
https://github.com/nebula-plugins/gradle-lint-plugin/issues/393). This
is somewhat unfortunate since the plugin is useful for removing unused
dependencies, though with the relatively small amount of Gradle code we
write hopefully it will not be missed much. If Nebula changes their
code to be compatible with Gradle 8+, we can re-add it easily.
This upgrade means we can remove the code added in 342051e1.
The Java code will be added in a followup PR.
Also fixed tests failing due to org.json upgrade: decimal whole numbers
no longer have their fractional parts removed, so currency value strings
must end with ".00" instead of ".0".
This is an 'easy' upgrade that requires a minor change in
common/build.gradle and the removal of an unnecessary import in buildSrc.
Gradle 7.4 and above has breaking changes that break the latest nebula lint plugin. We may have to wait a while.
Some retriers are no longer needed because transactions are
automatically retried by the JPA transaction manager when there's a
transient exception.
<!-- Reviewable:start -->
- - -
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/google/nomulus/1874)
<!-- Reviewable:end -->
* Fix Gradle dependency version pinning
In Gradle 7, version labels require '!!' at the end to be free from
any forced upgrade.
Hibernate min version needs to be advanced past 5.6.12, which is buggy.
Upgraded most dependencies to the latest version.
* Upgrade App Engine Standard to Java 17 w/ bundled APIs
Note that this doesn't yet upgrade our actual Gradle scripts to use a more
recent of Java (that will happen separately); this solely affects the GAE
instances.
I followed the instructions here:
https://cloud.google.com/appengine/docs/standard/java-gen2/services/access
And note that I removed threadsafe true from appengine's XML config because
that doesn't do anything anymore and was just throwing errors (the new
instances handle multiple requests in parallel by default, no configuration
necessary).
* Convert to gradle 7.
* More fixes, regenerated lockfiles.
* Update lockfiles for dependency update.
* Fix show_upgrade_diff for new lockfile format
* Add property for allowInsecureProtocol
Allow us to override the restriction against use of plain HTTP for
communication to dependency repositories. We need this to be able to use a
local proxy for dependency gathering.
* Checking in missing gradle.lockfile
Ben McIlwain
gbrodman
Weimin Yu
Lai Jiang
Michael Muller
