Compare commits

...

14 Commits

Author SHA1 Message Date
Levente Polyak
cb6484fe45 Version 20200213 2020-02-13 01:53:45 +01:00
Levente Polyak
bcb1b4a163 offload-build: handle user specific makepkg.conf on the remote host
This ensures we take user specific config values for PKGDEST into
account when printing the package list. This is required as devtools
archbuild_cmd puts packages potentially into the user defined PKGDEST
which the package list would otherwise miss.
2020-02-12 00:05:00 +01:00
Levente Polyak
57fb44b976 offload-build: take makepkg.conf from devtools for appropriate packagelist
This fixes an issue with the usage of makepkg --packagelist to get the
produced artifacts filenames according to the PKGEXT used in devtools'
makepkg.conf instead of the one defined in pacman.

One goal we want to preserve is that devtools configuration should be
self contained and not require any editing of non owned files like
the host /etc/makepkg.conf to produce expected results.

Additionally modify the archbuild_cmd override for multilib builds to
use an independent variable and not fiddle with the actual arch
variable to select the appropriate cmd.
2020-02-11 01:42:17 +01:00
Levente Polyak
9b4d8ae930 offload-build: convert to in-prog so we can perform pre-processing 2020-02-10 23:02:41 +01:00
Levente Polyak
4c206ab549 makerepropkg: take makepkg.conf from devtools pkgdatadir location
This ensures we use the same configuration for reproducing packages as
we use for building them via devtools.
One example of why we care about this are the COMPRESS* settings that
may differ from the guest's pacman shipped makepkg.conf that affect the
reproducibility of packages.
2020-02-10 23:02:16 +01:00
Eli Schwartz
4c08847bfa makerepropkg: correctly reproduce a pkgfile with any compression type
We don't want the default PKGEXT in the current version of devtools, we
want the PKGEXT we *know* the input file used.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-27 18:54:26 +01:00
Levente Polyak
74d7a70915 Version 20191227 2019-12-27 16:06:30 +01:00
Robin Broda
5eacb475cd makerepopkg: fix zstd extension (.zstd -> .zst) 2019-12-27 16:05:14 +01:00
Robin Broda
bcda211dd8 makepkg.conf: change default compression method to zstd 2019-12-27 16:04:56 +01:00
Levente Polyak
d4546c59ce Version 20191212 2019-12-12 01:01:44 +01:00
Levente Polyak
d6866e0544 lib: support arbitrary compressed package files
We need to modify the matching of valid package files to support formats
like zstd. Let's try to use an eager approach instead of a simple
whitelist in order to be functional for arbitrary formats that may be
introduced in the future without the need to adjust any code.

Allow any single fragment word as compression type but filter out known
non-package content like signature files.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-08 20:59:11 +01:00
Eli Schwartz
8edad22616 offload-build: return failure if no package files were found
This means that the remote command died at some stage earlier than the
printing of created files.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2019-12-05 23:34:23 +01:00
Levente Polyak
f20435643f makechrootpkg: sync databases for checkpkg off-site
Use pacman's --dbpath feature to sync fresh databases inside an isolated
location and split up the database sync and package location calls to
remove the need of weird grep calls.

It isn't nice of makechrootpkg to modify the host database state just by
building packages. No foreign program shall automatically modify
the host database other than by the explicit will of a system
maintainer, which is the major reason this changes get incorporated.

However, there is certain indoctrinated believe that using -Sy is
the prime evil. In fact it has been declared as a social rule to a
technical problem of not getting into potential partial upgrade states.
This is not a proper loophole less solution as there are multiple ways
and use cases that lead to such a state, like aborting a -Syu on the
prompt for whatever reason, what really matters is that it is not a
technically bullet proof solution to solve the problem.

Databases shall have the freedom to be as up to date as databases or
their owner wishes, allowing querying on latest database state without
fear. The only loophole-less contract that _really_ is from importance
is always using -Su instead of plain -S to install packages. Installing
packages is what actually brings one into a potential partial upgrade
state and by using -Su an outstanding upgrade is forced when installing
a new package. This properly solves all edge cases in a technical
manner instead of declaring people who abort the prompt of -Syu to be
the problem. In fact, using this simple contract allows whatever system
maintenance workflow a host owner wants to follow, which may still be to
always use -Syu and deal with system upgrades explicitly instead of the
time when installing new packages, but the -Su contract is the real safe
guard to guarantee no edge case can ever slip in. This magically also
opens up the freedom to people who wish to use -Sy to simply query on up
to date data as the currently indoctrinated "never do -Sy" stone plates
not only are not rock solid in technical terms but also make certain use
cases simply impossible and hence cripple the functionality without at
the very least being fully loophole free.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-05 22:50:11 +01:00
Levente Polyak
8d99df602d makechrootpkg: double quote array expansions for checkpkg locations
Avoid re-splitting remotepkg elements used for checkpkg conditions.
2019-11-30 13:44:02 +01:00
7 changed files with 37 additions and 17 deletions

1
.gitignore vendored
View File

@@ -11,6 +11,7 @@ lddd
makechrootpkg
makerepropkg
mkarchroot
offload-build
rebuildpkgs
zsh_completion
find-libdeps

View File

@@ -1,4 +1,4 @@
V=20191130
V=20200213
PREFIX = /usr/local
MANDIR = $(PREFIX)/share/man
@@ -17,12 +17,12 @@ IN_PROGS = \
makerepropkg \
mkarchroot \
makechrootpkg \
offload-build \
rebuildpkgs \
sogrep
BINPROGS = \
$(IN_PROGS) \
offload-build \
$(IN_PROGS)
CONFIGFILES = \
makepkg-x86_64.conf \

View File

@@ -142,7 +142,7 @@ find_cached_package() {
for dir in "${searchdirs[@]}"; do
[[ -d $dir ]] || continue
for pkg in "$dir"/*.pkg.tar?(.?z); do
for pkg in "$dir"/*.pkg.tar?(.!(sig|*.*)); do
[[ -f $pkg ]] || continue
# avoid adding duplicates of the same inode
@@ -152,7 +152,7 @@ find_cached_package() {
# split apart package filename into parts
pkgbasename=${pkg##*/}
pkgbasename=${pkgbasename%.pkg.tar?(.?z)}
pkgbasename=${pkgbasename%.pkg.tar*}
arch=${pkgbasename##*-}
pkgbasename=${pkgbasename%-"$arch"}

View File

@@ -387,17 +387,28 @@ if (( ret != 0 )); then
else
if (( run_checkpkg )); then
msg "Running checkpkg"
remotepkgs=($(pacman -Syddp --logfile /dev/null "${pkgnames[@]}"|grep '://'))
# sync off-site databases for up-to-date queries
trap 'rm -rf $dbpath; cleanup' EXIT INT TERM QUIT
dbpath=$(mktemp -d --tmpdir makechrootpkg-database.XXXXXXXXXX)
mkdir -p "$dbpath"
pacman -Sy --dbpath "$dbpath" --logfile /dev/null
# query current package locations
remotepkgs=($(pacman -Sddp --dbpath "$dbpath" --logfile /dev/null "${pkgnames[@]}"))
if (( $? )); then
warning "Skipped checkpkg due to missing repo packages"
exit 0
fi
for remotepkg in ${remotepkgs[@]}; do
# download package files if any non-local location exists
for remotepkg in "${remotepkgs[@]}"; do
[[ $remotepkg == file://* ]] && continue
msg2 "Downloading current versions"
pacman --noconfirm -Swdd --logfile /dev/null "${pkgnames[@]}"
pacman --noconfirm -Swdd --dbpath "$dbpath" --logfile /dev/null "${pkgnames[@]}"
break
done
msg2 "Checking packages"
sudo -u "$makepkg_user" checkpkg --rmdir --warn
fi

View File

@@ -132,7 +132,7 @@ DBGSRCDIR="/usr/src/debug"
COMPRESSGZ=(gzip -c -f -n)
COMPRESSBZ2=(bzip2 -c -f)
COMPRESSXZ=(xz -c -z -)
COMPRESSZST=(zstd -c -z -q -)
COMPRESSZST=(zstd -c -T0 --ultra -20 -)
COMPRESSLRZ=(lrzip -q)
COMPRESSLZO=(lzop -q)
COMPRESSZ=(compress -c -f)
@@ -143,7 +143,7 @@ COMPRESSLZ=(lzip -c -f)
# EXTENSION DEFAULTS
#########################################################################
#
PKGEXT='.pkg.tar.xz'
PKGEXT='.pkg.tar.zst'
SRCEXT='.src.tar.gz'
# vim: set ft=sh ts=2 sw=2 et:

View File

@@ -59,7 +59,7 @@ get_pkgfile() {
local pkgname=${pkgfilebase%-*-*-*}
local pkgfile ext
for ext in .xz .zstd ''; do
for ext in .xz .zst ''; do
pkgfile=${pkgfilebase}.pkg.tar${ext}
for c in "${cache_dirs[@]}"; do
@@ -137,6 +137,7 @@ parse_buildinfo < <(bsdtar -xOqf "${pkgfile}" .BUILDINFO)
export SOURCE_DATE_EPOCH="${buildinfo[builddate]}"
PACKAGER="${buildinfo[packager]}"
BUILDDIR="${buildinfo[builddir]}"
PKGEXT=${pkgfile#${pkgfile%.pkg.tar*}}
# nuke and restore reproducible testenv
for copy in "${buildroot}"/*/; do
@@ -152,15 +153,14 @@ for fname in "${installed[@]}"; do
exit 1
fi
done
printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1
printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -M @pkgdatadir@/makepkg-x86_64.conf -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1
# use makechrootpkg to prep the build directory
makechrootpkg -r "${buildroot}" -l "${chroot}" -- --packagelist || exit 1
# set detected makepkg.conf options
{
for var in PACKAGER BUILDDIR; do
for var in PACKAGER BUILDDIR PKGEXT; do
printf '%s=%s\n' "${var}" "${!var@Q}"
done
printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}"

View File

@@ -74,11 +74,12 @@ while (( $# )); do
done
# multilib must be handled specially
archbuild_arch="${arch}"
if [[ $repo = multilib* ]]; then
arch=
archbuild_arch=
fi
archbuild_cmd=("${repo}${arch:+-$arch}-build" "$@")
archbuild_cmd=("${repo}${archbuild_arch:+-$archbuild_arch}-build" "$@")
trap 'rm -rf $SRCPKGDEST' EXIT INT TERM QUIT
@@ -108,7 +109,12 @@ mapfile -t files < <(
printf "%s\n" "" "-> build complete" &&
printf "\t%s\n" "$temp"/*
} >&2 &&
makepkg --packagelist
makepkg_user_config="${XDG_CONFIG_HOME:-$HOME/.config}/pacman/makepkg.conf" &&
makepkg_config="/usr/share/devtools/makepkg-'"${arch}"'.conf" &&
if [[ -f /usr/share/devtools/makepkg-'"${repo}"'-'"${arch}"'.conf ]]; then
makepkg_config="/usr/share/devtools/makepkg-'"${repo}"'-'"${arch}"'.conf"
fi &&
makepkg --config <(cat "${makepkg_user_config}" "${makepkg_config}" 2>/dev/null) --packagelist
')
@@ -116,4 +122,6 @@ if (( ${#files[@]} )); then
printf '%s\n' '' '-> copying files...'
load_makepkg_config
scp "${files[@]/#/$server:}" "${PKGDEST:-${PWD}}/"
else
exit 1
fi