Pass through the local containers directory (it cannot
be bind-mounted to /var/lib/containers since podman checks
the path hasn't changed) with overrides to the paths. This
allows containers to be created inside the dbuild container,
so we can enlist pre-packaged software (such as opensearch)
in test.py. If the container images are already downloaded
in the host, they won't be downloaded again.
It turns out that the container ecosystem doesn't support
nested network namespaces well, so we configure the outer
container to use host networking for the inner containers.
It's useful anyway.
The frozen toolchain now installs podman and buildah so
there's something to actually drive those nested containers.
We disable weak dnf dependencies to avoid installing qemu.
The frozen toolchain is regenerated with optimized clang from
https://devpkg.scylladb.com/clang/clang-19.1.7-Fedora-41-aarch64.tar.gzhttps://devpkg.scylladb.com/clang/clang-19.1.7-Fedora-41-x86_64.tar.gzClosesscylladb/scylladb#24020
Add new dependency pytest-xdist to the toolchain. This will allow executing
boost and unit tests from pytest in parallel, reducing the time
needed for the run.
Closesscylladb/scylladb#21222
* tools/java e505a6d3bb...5b0e274f12 (1):
> Merge 'build.xml: install and use java-11 when building' from Kefu Chai
Updates to clang 18.1.8 + LLVM patch to match Fedora 40.
New optimized clang build generated and stored in
https://devpkg.scylladb.com/clang/clang-18.1.8-x86_64.tar.gzhttps://devpkg.scylladb.com/clang/clang-18.1.8-aarch64.tar.gz
Due to the loss of the jmx submodule, we no longer install java-11-openjdk.
We add it in install-dependencies.sh here to compensate, pending a better
solution.
tools/java submodule updated to remove build failure where Java 8
was selected instead of Java 11.
The scylla_gdb test suite was disabled due to a regression in gdb 15,
which is brought in by the toolchain update [1].
[1] https://github.com/scylladb/scylladb/issues/20741.
Add allure-pytest pip dependency to be able to use it for generating the allure report later.
Main benefits of the allure report:
1. Group test failures
2. Possibility to attach log files to she test itself
3. Timeline of test run
4. Test description on the report
5. Search by test name or tag
[avi: regenerate toolchain]
Closesscylladb/scylladb#19335
As part of code coverage we need some additional packages in order to
being able to process the code coverage data and being able to provide
some meaningful information in logs.
Here we add the following packages:
fedora packages:
----------------
lcov - A package of utilities to manipulate lcov traces and generate
coverage html reports
fedora python3 packages:
------------------------
The following packages are added into fedora_packages and not the
python3_packages since we don't need them to be packaged into
scylla-python3 package but we only require them for the build
environment.
python3-unidiff - A python library for working with patch files, this is
required in order to generate "patch coverage" reports.
python3-humanfriendly - A python library to format some quantities into
a human readable strings (time spans, sizes, etc...)
we use it to print meaningful logs that tracks
the volume and time it takes to process coverage
data so we can better debug and optimize it in the
future.
python3-jinja3 - This is a template based generator that will eventually
will allow to consolidate and rearrange several reports into one so we
can publish a single report "site" for all of the coverage information.
For example, include both, coverage report as well as
patch report in a tab based site.
pip packages:
-------------
treelib - A tree data structure that supports also pretty printing of
the tree data. We use it to log the coverage processing steps in
order to have debugging capabilities in the future.
Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>
Closesscylladb/scylladb#16330
[avi: regenerate toolchain]
Closesscylladb/scylladb#16357
Update node_exporter to 1.7.0.
The previous version (1.6.1) was flagged by security scanners (such as
Trivy) with HIGH-severity CVE-2023-39325. 1.7.0 release fixed that
problem.
[Botond: regenerate frozen toolchain]
Fixes#16085Closesscylladb/scylladb#16086Closesscylladb/scylladb#16090
* seastar 576ee47d...bab1625c (13):
> build: s/{dpdk_libs}/${dpdk_libs}/
> build: build with dpdk v23.07
> scripts: Fix escaping of regexes in addr2line
> linux-aio: print more specific error when setup_aio fails
> linux-aio: correct the error message raised when io_setup() fails
> build: reenable -Warray-bound compiling option
> build: error out if find_program() fails
> build: enable systemtap only if it is available
> build: check if libucontext is necessary for using ucontext functions
> smp: reference correct variable when fetch_or()
> build: use target_compile_definitions() for adding -D...
> http/client: pass tls_options to tls::connect()
> Merge 'build, process: avoid using stdout or stderr as C++ identifiers' from Kefu Chai
Frozen toolchain regenerated for new Seastar depdendencies.
configure.py adjusted for new Seastar arch names.
Closesscylladb/scylladb#15476
This refreshes clang to 16.0.6 and libstdc++ to 13.1.1.
compiler-rt, libasan, and libubsan are added to install-dependencies.sh
since they are no longer pulled in as depdendencies.
Closes#13730
After the addition of the rust-std-static-wasm32-wasi target, we're
able to compile the Rust programs to Wasm binaries. However, we're still
only able to handle the Wasm UDFs in the Text format, so we need a tool
to translate the .wasm files to .wat. Additionally, the .wasm files
generated by default are unnecessarily large, which can be helped
using wasm-opt and wasm-strip.
The tool for translating wasm to wat (wasm2wat), and the tool for
stripping the wasm binaries (wasm-strip) are included in the `wabt`
package, and the optimization tool (wasm-opt) is included in the
`binaryen` package. Both packages are added to install-dependencies.sh
Closes#13282
[avi: regenerate frozen toolchain]
Closes#13605
In the future, when testing WASM UDFs, we will only store the Rust
source codes of them, and compile them to WASM. To be able to
do that, we need rust standard library for the wasm32-wasi target,
which is available as an RPM called rust-std-static-wasm32-wasi.
Closes#12896
[avi: regenerate toolchain]
Closes#13258
It will be needed by S3 driver to parse multipart-upload messages from
server
refs: #12523
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
Closes#13158
[avi: regenerate toolchain]
Closes#13192
These two are static binaries, so no need in yum/apt-installing them with dependencies.
Just download with curl and put them into /urs/local/bin with X-bit set.
This is needed for future object-storage work in order to run unit tests against minio.
refs: #12523
Signed-off-by: Pavel Emelyanov <xemul@scylladb.com>
[avi: regenerate frozen toolchain]
Closes#13064Closes#13099
This lets us carry fewer things and rely on the distribution
for maintenance.
The frozen toolchain is updated. Incidental updates include clang 15.0.6,
and pytest that doesn't need workarounds.
Closes#12397
Now that our toolchain is based on Fedora 37, we can rely on its
libdeflate rather than have to carry our own in a submodule.
Frozen toolchain is regenerated. As a side effect clang is updated
from 15.0.0 to 15.0.4.
Closes#12000
To fix CVE-2022-24675, we need to a binary compiled in <= golang 1.18.1.
Only released version which compiled <= golang 1.18.1 is node_exporter
1.4.0, so we need to update to it.
See scylladb/scylla-enterprise#2317
Closes#11400
[avi: regenerated frozen toolchain]
Closes#11879
Using traceback_with_variables module, generate more detail traceback
with variables into debug log.
This will help fixing bugs which is hard to reproduce.
Closes#10472
[avi: regenerate frozen toolchain]
The main reason for adding rust dependency to scylla is the
wasmtime library, which is written in rust. Although there
exist c++ bindings, they don't expose all of its features,
so we want to do that ourselves using rust's cxx.
Signed-off-by: Wojciech Mitros <wojciech.mitros@scylladb.com>
[avi: update toolchain]
[avi: remove example, saving for a follow-on]
Pass --pip-packages option to tools/python3/reloc/build_reloc.sh,
add scylla-driver to relocatable python3 which required for
fix_system_distributed_tables.py.
[avi: regenrate toolchain]
Ref #9040
If the wasmtime library is available for download, it will be
set up by install-dependencies and prepared for linking.
Closes#9151
[avi: regenerate toolchain, which also updates clang to 12.0.1]
Download node_exporter in frozen image to prepare adding node_exporter
to relocatable pacakge.
Related #2190Closes#7765
[avi: updated toolchain, x86_64/aarch64/s390x]
