Commit Graph

14 Commits

Author SHA1 Message Date
Takuya ASADA
d6165bc1c3 dist/debian/python3: drop dependency on pystache
Same as 287d6e5, we need to drop pystache from package build script
since Fedora 32 dropped it.
2020-06-22 14:03:13 +03:00
Avi Kivity
5b92a6d9e4 build: drop __pycache__ directories from python3 relocatable package
Recently ./reloc/build_deb.sh started failing with

dpkg-source: info: using source format '1.0'
dpkg-source: info: building scylla-python3 using existing scylla-python3_3.8.3-0.20200604.77dfa4f15.orig.tar.gz
dpkg-source: info: building scylla-python3 in scylla-python3_3.8.3-0.20200604.77dfa4f15-1.diff.gz
dpkg-source: error: cannot represent change to scylla-python3/lib64/python3.8/site-packages/urllib3/packages/backports/__pycache__/__init__.cpython-38.pyc:
dpkg-source: error:   new version is plain file
dpkg-source: error:   old version is symlink to /usr/lib/python3.8/site-packages/__pycache__/six.cpython-38.pyc
dpkg-source: error: unrepresentable changes to source
dpkg-buildpackage: error: dpkg-source -b . subprocess returned exit status 1
debuild: fatal error at line 1182:

Those files are not in fact symlinks, so it's clear that dpkg is confused
about something. Rather than debug dpkg, however, it's easier to just
drop __pycache__ directories. These hold the result of bytecode
compilation and are therefore optional, as Python will compile the sources
if the cache is not populated.

Fixes #6584.
2020-06-04 13:04:34 +03:00
Takuya ASADA
536ab4ebe4 reloc-pkg: move all files under project name directory
To make unified relocatable package easily, we may want to merge tarballs to single tarball like this:
zcat *.tar.gz | gzip -c > scylla-unified.tar.xz
But it's not possible with current relocatable package format, since there are multiple files conflicts, install.sh, SCYLLA-*-FILE, dist/, README.md, etc..

To support this, we need to archive everything in the directory when building relocatable package.

This is modifying relocatable package format, we need to provide a way to
detect the format version.
To do this, we added a new file ".relocatable_package_version" on the top of the
archive, and set version number "2" to the file.

Fixes #6315
2020-06-03 09:52:44 +03:00
Takuya ASADA
cfa8885ae1 reloc/python3: add install.sh on python relocatable package
To support nonroot installation on scylla-python3, add install.sh on
scylla-python3 relocatable package.
2019-09-03 20:06:30 +09:00
Takuya ASADA
25112408a7 dist/debian: support relocatable python3 on Debian variants
Unlike CentOS, Debian variants has python3 package on official repository,
so we don't have to use relocatable python3 on these distributions.
However, official python3 version is different on each distribution, we may
have issue because of that.
Also, our scripts and packaging implementation are becoming presuppose
existence of relocatable python3, it is causing issue on Debian
variants.

Switching to relocatable python3 on Debian variants avoid these issues,
it will easier to manage Scylla python3 environments accross multiple
distributions.

Fixes #4495

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190531112707.20082-1-syuu@scylladb.com>
2019-06-02 14:59:43 +03:00
Avi Kivity
7a0c6cd583 Revert "dist/debian: support relocatable python3 on Debian variants"
This reverts commit 4d119cbd6d. It breaks build_deb.sh:

18:39:56 +	seastar/scripts/perftune.py seastar/scripts/seastar-addr2line seastar/scripts/perftune.py
18:39:56 Traceback (most recent call last):
18:39:56   File "./relocate_python_scripts.py", line 116, in <module>
18:39:56     fixup_scripts(archive, args.scripts)
18:39:56   File "./relocate_python_scripts.py", line 104, in fixup_scripts
18:39:56     fixup_script(output, script)
18:39:56   File "./relocate_python_scripts.py", line 79, in fixup_script
18:39:56     orig_stat = os.stat(script)
18:39:56 FileNotFoundError: [Errno 2] No such file or directory: '/data/jenkins/workspace/scylla-master/unified-deb/scylla/build/debian/scylla-package/+'
18:39:56 make[1]: *** [debian/rules:19: override_dh_auto_install] Error 1
2019-05-29 13:58:41 +03:00
Takuya ASADA
4d119cbd6d dist/debian: support relocatable python3 on Debian variants
Unlike CentOS, Debian variants has python3 package on official repository,
so we don't have to use relocatable python3 on these distributions.
However, official python3 version is different on each distribution, we may
have issue because of that.
Also, our scripts and packaging implementation are becoming presuppose
existence of relocatable python3, it is causing issue on Debian
variants.

Switching to relocatable python3 on Debian variants avoid these issues,
it will easier to manage Scylla python3 environments accross multiple
distributions.

Fixes #4495

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190526105138.677-1-syuu@scylladb.com>
2019-05-26 13:56:30 +03:00
Takuya ASADA
4b08a3f906 reloc/python3: add license files on relocatable python3 package
It's better to have license files on our python3 distribution.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190516094329.13273-1-syuu@scylladb.com>
2019-05-19 18:30:19 +03:00
Takuya ASADA
64c2aa8f9b reloc/python3: add missing SCYLLA-PRODUCT-FILE to python3 relocatable package
Since 214c74a, we need SCYLLA-PRODUCT-FILE on relocatable package so add
it on python3 package as well.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190422085620.22486-1-syuu@scylladb.com>
2019-04-22 13:56:38 +03:00
Takuya ASADA
e3a5ac2945 reloc: run fix_sharedlib() only on application/x-sharedlib and application/x-pie-executable
We need to prevent to run fix_sharedlib() on non-ELF files.

Fixes #4415

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190409114941.28276-1-syuu@scylladb.com>
2019-04-09 14:54:54 +03:00
Glauber Costa
2305cc88f3 relocatable python: Be more permissive with mime type checking
Fedora28 python magic used to return a x-sharedlib mime type for .so files.
Fedora29 changed that to x-pie-executable, so the libraries are no longer
relocated.

Let's be more permissive and relocate everything that starts with application/.

Fixes #4396

Signed-off-by: Glauber Costa <glauber@scylladb.com>
Message-Id: <20190404140929.7119-1-glauber@scylladb.com>
2019-04-07 09:26:56 +03:00
Takuya ASADA
dc5cec4194 reloc/python3: archive package version number on build_reloc.sh
Instead of getting python3 version number on build_rpm.sh, archive
version number when generating python3 relocatable package.
2019-03-24 00:27:24 +09:00
Takuya ASADA
4fed4fecf6 reloc/python3: archive rpm build script in the relocatable package, build rpm using the script
Since we archive rpm/deb build script on relocatable package and build
rpm/deb using the script, so align python relocatable package too.

Also added SCYLLA-RELOCATABLE-FILE, SCYLLA-RELEASE-FILE and SCYLLA-VERSION-FILE
since these files are required for relocatable package.
2019-03-24 00:27:16 +09:00
Takuya ASADA
a515324732 reloc: move relocatable python build scripts to reloc/python3 and dist/redhat/python3
To make easier to find build scripts and keep script filename simpler,
move them to python3 directory.
2019-03-24 00:25:50 +09:00