Compare commits

..

37 Commits

Author SHA1 Message Date
bartus
6517c8d9c0 Canonicalize array initialization 2021-05-17 15:40:01 +02:00
bartus
6b2cb91b7d Drop pacutils fallback, use pacman-conf 2021-05-17 15:40:00 +02:00
bartus
2d0854f204 Rewrite cache_dirs logic
* initialize cache_dirs with one of:
 1. value provided in command line -c switch
 2. pacman.conf provided in command line -C switch
 3. container pacman.conf file
* find first writable cache_dir, bind it rw, remove from cache_dirs
* extract host mirrors, find file:// ones, append to cache_dirs
* extract server url, find file:// ones, append to cache_dirs
* concatenate cache_dirs removing duplicates, bind them ro to container
* check if any extra cache_dirs was defined, append to
  container pacman.conf as new [options] section.

Fix #57 #58 #59
2021-05-17 15:38:43 +02:00
Levente Polyak
90ba07d9be Version 20210202 2021-02-02 00:51:57 +01:00
Levente Polyak
8c26438df8 makechrootpkg: only expose failed build logs instead of products
After further followups always moving all products isn't actually
desired as they can theoretically be broken in various ways if
arch-nspawn exists non successful.

However, as we would like to always preserve the produced log files we
instead split out the logfiles into an own function and call that for
unsuccessful buils.

Fixes 4f305aa3
2021-01-31 23:44:11 +01:00
Christian Hesse
a49bcf2097 doc: update default build host
Commit 09e169b741 changed the default
build host from dragon.archlinux.org to build.archlinux.org - match
in documentation.

Signed-off-by: Christian Hesse <mail@eworm.de>
2021-01-31 02:16:44 +01:00
Eli Schwartz
0883f45b3a makerepropkg: allow specifying the package in pacman -S format
We now accept:

1) # nothing

    in which case we'll use the PKGBUILD to retrieve...

2) name, or repo/name

    in which case we'll use pacman to cache the package and retrieve...

3) a filename

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-01-27 00:56:48 +01:00
Eli Schwartz
99c2020d47 makerepropkg: do fast cache lookups
Teach get_pkgfile to call itself in local-only mode and find a cached
file no matter what its extension is. Avoids repetitively trying to curl
random files, fail with 404 errors, and proceed to discover a cache hit
under a different file extension.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-01-27 00:56:48 +01:00
Christian Hesse
00f7a6a415 makepkg.conf: make rsync use new-style compression in makepkg.conf
Our rsync package is no longer built with bundled zlib, so old-style
compression is no longer supported.

https://www.archlinux.org/news/rsync-compatibility/

Signed-off-by: Christian Hesse <mail@eworm.de>
2021-01-26 20:56:46 +01:00
bartoszek
4f305aa316 makechrootpkg: Expose failed build logs/products
Since move_products() function is fairly robust we can make it run for
failed build also to expose logs for packages that fails in build(),
prepare() or package(). It also exposes partially packaged split
packages if they fail in latter package_xxx().
2021-01-26 20:43:29 +01:00
Levente Polyak
09e169b741 offload-build: change default host to build.archlinux.org 2021-01-26 20:40:47 +01:00
Jelle van der Waa
360a7611a8 doc: add devtools.7 man page
Signed-off-by: Jelle van der Waa <jelle@vdwaa.nl>
2021-01-26 20:39:06 +01:00
Konstantin Gizdov
d507db9490 offload-build: respect SRCEXT from makepkg.conf 2021-01-26 20:38:53 +01:00
Levente Polyak
aff81d34fd Version 20200407 2020-04-07 19:46:10 +02:00
Levente Polyak
5b1123e11f offload-build: retrieve the PKGBUILD in case makepkg changes it via pkgver
Building a package may change the PKGBUILD during update_pkgver. Let's
retrieve the PKGBUILD after building to ensure we have the very same
file as the one we used to build the package. Otherwise this may lead to
the inability to distribute the package during commitpkg in case the
expected and the actual hashsum mismatch.
2020-03-09 23:27:03 +01:00
Levente Polyak
a0f79fcce0 makerepropkg: prioritize downloading .zst packages over legacy format
First try a .zst location before falling back to legacy variants. This
should slightly speed up downloading of dependencies, especially over
time as .zst packages are or will be the dominant format.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-03-09 23:27:03 +01:00
Eli Schwartz
94b0413e13 arch-nspawn: fix up host_mirrors cachedir handling
This was incorrectly implemented in commit
0067176529, which added the host_mirrors
root directory as a cachedir, when we actually want to use the pool/*
subdirectories (the same ones installed on the build server's
/etc/pacman.conf).

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:27 +01:00
Eli Schwartz
e963b6da9e makechrootpkg: use the chroot database to find checkpkg packages
We don't want to check against the current version known to the host
system, because that will be incorrect in a wide variety of situations,
including:
- the build host hasn't done a full system upgrade yet
- we're building against staging, and want to see the delta between
  different staging versions
- we're building against extra, but the host runs testing which carries
  changes we don't want to visualize right now
- the chroot has a configured database not available to the host, and
  the package is only available there

Essentially, it's rarely 100% correct to run checkpkg on the host, but
we already have a database we *know* is correct, and that is the one we
just built the package against. So let's use that.

This also fixes a bug in the current logic, where in order to try
downloading fresh databases, we work in a non-cached temporary working
database to download the package files, but then let checkpkg default to
comparing packages in the system database. Since we are explicitly
trying to compare against packages that differ from the host's pacman
database, we need to pass the package files as options to checkpkg,
using the additional modes added in commit c14338c0fe

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:19 +01:00
Eli Schwartz
b2cbb8628e checkpkg: fix support for http:// url comparisons
Broken in commit c14338c0fe.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:14 +01:00
Eli Schwartz
6d273f79c3 zsh-completion: update for new makerepropkg options
Thanks to anthraxx for the guidance.

Original-patch-by: Levente Polyak <anthraxx@archlinux.org>
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:05 +01:00
Eli Schwartz
51842a1676 makerepropkg: support checking multiple split packages
By specifying multiple package files, we assume they are all from the
same PKGBUILD, and try to check them all against the produced artifacts.
Since the buildinfo should be comparable for all of them, we simply use
the first one passed on the command line.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:49:54 +01:00
Eli Schwartz
53fe5c67a1 makerepropkg: add support to check unreproducible packages using diffoscope
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:49:48 +01:00
Eli Schwartz
21d9984acc makerepropkg: fix wonky indent
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:49:33 +01:00
Levente Polyak
cb6484fe45 Version 20200213 2020-02-13 01:53:45 +01:00
Levente Polyak
bcb1b4a163 offload-build: handle user specific makepkg.conf on the remote host
This ensures we take user specific config values for PKGDEST into
account when printing the package list. This is required as devtools
archbuild_cmd puts packages potentially into the user defined PKGDEST
which the package list would otherwise miss.
2020-02-12 00:05:00 +01:00
Levente Polyak
57fb44b976 offload-build: take makepkg.conf from devtools for appropriate packagelist
This fixes an issue with the usage of makepkg --packagelist to get the
produced artifacts filenames according to the PKGEXT used in devtools'
makepkg.conf instead of the one defined in pacman.

One goal we want to preserve is that devtools configuration should be
self contained and not require any editing of non owned files like
the host /etc/makepkg.conf to produce expected results.

Additionally modify the archbuild_cmd override for multilib builds to
use an independent variable and not fiddle with the actual arch
variable to select the appropriate cmd.
2020-02-11 01:42:17 +01:00
Levente Polyak
9b4d8ae930 offload-build: convert to in-prog so we can perform pre-processing 2020-02-10 23:02:41 +01:00
Levente Polyak
4c206ab549 makerepropkg: take makepkg.conf from devtools pkgdatadir location
This ensures we use the same configuration for reproducing packages as
we use for building them via devtools.
One example of why we care about this are the COMPRESS* settings that
may differ from the guest's pacman shipped makepkg.conf that affect the
reproducibility of packages.
2020-02-10 23:02:16 +01:00
Eli Schwartz
4c08847bfa makerepropkg: correctly reproduce a pkgfile with any compression type
We don't want the default PKGEXT in the current version of devtools, we
want the PKGEXT we *know* the input file used.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-27 18:54:26 +01:00
Levente Polyak
74d7a70915 Version 20191227 2019-12-27 16:06:30 +01:00
Robin Broda
5eacb475cd makerepopkg: fix zstd extension (.zstd -> .zst) 2019-12-27 16:05:14 +01:00
Robin Broda
bcda211dd8 makepkg.conf: change default compression method to zstd 2019-12-27 16:04:56 +01:00
Levente Polyak
d4546c59ce Version 20191212 2019-12-12 01:01:44 +01:00
Levente Polyak
d6866e0544 lib: support arbitrary compressed package files
We need to modify the matching of valid package files to support formats
like zstd. Let's try to use an eager approach instead of a simple
whitelist in order to be functional for arbitrary formats that may be
introduced in the future without the need to adjust any code.

Allow any single fragment word as compression type but filter out known
non-package content like signature files.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-08 20:59:11 +01:00
Eli Schwartz
8edad22616 offload-build: return failure if no package files were found
This means that the remote command died at some stage earlier than the
printing of created files.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2019-12-05 23:34:23 +01:00
Levente Polyak
f20435643f makechrootpkg: sync databases for checkpkg off-site
Use pacman's --dbpath feature to sync fresh databases inside an isolated
location and split up the database sync and package location calls to
remove the need of weird grep calls.

It isn't nice of makechrootpkg to modify the host database state just by
building packages. No foreign program shall automatically modify
the host database other than by the explicit will of a system
maintainer, which is the major reason this changes get incorporated.

However, there is certain indoctrinated believe that using -Sy is
the prime evil. In fact it has been declared as a social rule to a
technical problem of not getting into potential partial upgrade states.
This is not a proper loophole less solution as there are multiple ways
and use cases that lead to such a state, like aborting a -Syu on the
prompt for whatever reason, what really matters is that it is not a
technically bullet proof solution to solve the problem.

Databases shall have the freedom to be as up to date as databases or
their owner wishes, allowing querying on latest database state without
fear. The only loophole-less contract that _really_ is from importance
is always using -Su instead of plain -S to install packages. Installing
packages is what actually brings one into a potential partial upgrade
state and by using -Su an outstanding upgrade is forced when installing
a new package. This properly solves all edge cases in a technical
manner instead of declaring people who abort the prompt of -Syu to be
the problem. In fact, using this simple contract allows whatever system
maintenance workflow a host owner wants to follow, which may still be to
always use -Syu and deal with system upgrades explicitly instead of the
time when installing new packages, but the -Su contract is the real safe
guard to guarantee no edge case can ever slip in. This magically also
opens up the freedom to people who wish to use -Sy to simply query on up
to date data as the currently indoctrinated "never do -Sy" stone plates
not only are not rock solid in technical terms but also make certain use
cases simply impossible and hence cripple the functionality without at
the very least being fully loophole free.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-05 22:50:11 +01:00
Levente Polyak
8d99df602d makechrootpkg: double quote array expansions for checkpkg locations
Avoid re-splitting remotepkg elements used for checkpkg conditions.
2019-11-30 13:44:02 +01:00
13 changed files with 258 additions and 98 deletions

2
.gitignore vendored
View File

@@ -11,6 +11,7 @@ lddd
makechrootpkg
makerepropkg
mkarchroot
offload-build
rebuildpkgs
zsh_completion
find-libdeps
@@ -18,3 +19,4 @@ crossrepomove
arch-nspawn
sogrep
doc/*.1
doc/*.7

View File

@@ -1,4 +1,4 @@
V=20191130
V=20210202
PREFIX = /usr/local
MANDIR = $(PREFIX)/share/man
@@ -17,12 +17,12 @@ IN_PROGS = \
makerepropkg \
mkarchroot \
makechrootpkg \
offload-build \
rebuildpkgs \
sogrep
BINPROGS = \
$(IN_PROGS) \
offload-build \
$(IN_PROGS)
CONFIGFILES = \
makepkg-x86_64.conf \
@@ -75,7 +75,8 @@ MANS = \
doc/makerepropkg.1 \
doc/mkarchroot.1 \
doc/find-libdeps.1 \
doc/find-libprovides.1
doc/find-libprovides.1 \
doc/devtools.7
all: $(BINPROGS) bash_completion zsh_completion man

View File

@@ -57,39 +57,51 @@ shift 1
[[ -z $working_dir ]] && die 'Please specify a working directory.'
pacconf_cmd=$(command -v pacman-conf || command -v pacconf)
if (( ${#cache_dirs[@]} == 0 )); then
mapfile -t cache_dirs < <($pacconf_cmd --config "${pac_conf:-$working_dir/etc/pacman.conf}" CacheDir)
mapfile -t cache_dirs < <(pacman-conf --config "${pac_conf:-$working_dir/etc/pacman.conf}" CacheDir)
fi
# shellcheck disable=2016
host_mirrors=($($pacconf_cmd --repo extra Server 2> /dev/null | sed -r 's#(.*/)extra/os/.*#\1$repo/os/$arch#'))
for host_mirror in "${host_mirrors[@]}"; do
if [[ $host_mirror == *file://* ]]; then
host_mirror=$(echo "$host_mirror" | sed -r 's#file://(/.*)/\$repo/os/\$arch#\1#g')
in_array "$host_mirror" "${cache_dirs[@]}" || cache_dirs+=("$host_mirror")
# find first writable cache, bind it to container in rw mode, remove it from cache_dirs
for cache_dir in "${cache_dirs[@]}"; do
if [[ -d "$cache_dir" && -w "$cache_dir" ]]; then
mount_args+=("--bind=$cache_dir")
mapfile -t cache_dirs < <(printf "%s\n" "${cache_dirs[@]}" | grep -vFx "$cache_dir")
break
fi
# this could print an warning to the user that no host caches would be used
done
while read -r line; do
mapfile -t lines < <($pacconf_cmd --config "${pac_conf:-$working_dir/etc/pacman.conf}" \
--repo $line Server | sed -r 's#(.*/)[^/]+/os/.+#\1#')
for line in "${lines[@]}"; do
if [[ $line = file://* ]]; then
line=${line#file://}
in_array "$line" "${cache_dirs[@]}" || cache_dirs+=("$line")
fi
# shellcheck disable=2016
mapfile -t host_mirrors < <(pacman-conf --repo extra Server 2> /dev/null | sed -r 's#(.*/)extra/os/.*#\1$repo/os/$arch#')
# extract local host mirrors (Server=file://...)
mapfile -t host_mirrors_local < <(printf "%s\n" "${host_mirrors[@]}" | grep -Po "(?<=file://).*(?=\/\$repo)")
for host_mirror in "${host_mirrors_local[@]}"; do
for m in "$host_mirror"/pool/*/; do
cache_dirs+=("$m")
done
done < <($pacconf_cmd --config "${pac_conf:-$working_dir/etc/pacman.conf}" --repo-list)
mount_args+=("--bind=${cache_dirs[0]//:/\\:}")
for cache_dir in "${cache_dirs[@]:1}"; do
mount_args+=("--bind-ro=${cache_dir//:/\\:}")
done
mapfile -t repos < <(pacman-conf --config "${pac_conf:-$working_dir/etc/pacman.conf}" --repo-list)
declare -a servers
for repo in "${repos[@]}"; do
mapfile -t -O "${#servers[@]}" servers < <( \
pacman-conf --config "${pac_conf:-$working_dir/etc/pacman.conf}" --repo "$repo" Server | \
grep -Po "(?<=file://).*$" | sed -r -e "s#\$repo#$repo#" -e 's#/$arch##' )
done
cache_dirs+=("${servers[@]}")
# rest of cache_dirs are getting bind ro therefor we don't care about ordering and can use sort -u for concatenation
if (( ${#cache_dirs[@]} != 0 )); then
mapfile -t cache_dirs < <(printf '%s\n' "${cache_dirs[@]}" | sort -u)
for cache_dir in "${cache_dirs[@]}"; do
mount_args+=("--bind-ro=${cache_dir//:/\\:}")
done
fi
# {{{ functions
copy_hostconf () {
unshare --fork --pid gpg --homedir "$working_dir"/etc/pacman.d/gnupg/ --no-permission-warning --quiet --batch --import --import-options import-local-sigs "$(pacman-conf GpgDir)"/pubring.gpg >/dev/null 2>&1
@@ -100,13 +112,18 @@ copy_hostconf () {
[[ -n $pac_conf ]] && cp "$pac_conf" "$working_dir/etc/pacman.conf"
[[ -n $makepkg_conf ]] && cp "$makepkg_conf" "$working_dir/etc/makepkg.conf"
# check if any extra cache_dirs was added
mapfile -t container_cache_dirs < <(pacman-conf --config "$working_dir/etc/pacman.conf" CacheDir)
mapfile -t extra_cache_dirs < <(comm -13 <(printf "%s\n" "${container_cache_dirs[@]}"|sort) <(printf "%s\n" "${cache_dirs[@]}"|sort))
if (( ${#extra_cache_dirs[@]} != 0 )); then
echo -e "\n[options]\nCacheDir = ${extra_cache_dirs[*]}" >> "$working_dir/etc/pacman.conf"
fi
local file
for file in "${files[@]}"; do
mkdir -p "$(dirname "$working_dir$file")"
cp -T "$file" "$working_dir$file"
done
sed -r "s|^#?\\s*CacheDir.+|CacheDir = ${cache_dirs[*]}|g" -i "$working_dir/etc/pacman.conf"
}
# }}}

View File

@@ -95,8 +95,10 @@ for _pkgname in "${pkgname[@]}"; do
if (( $# )); then
case $1 in
*://*)
pkgurl=$1 ;;
/*|*/*)
pkgurl=file://$(readlink -m "$1") ;;
pkgurl=$(readlink -m "$1") ;;
*.pkg.tar*)
pkgurl=$1 ;;
'')

46
doc/devtools.7.asciidoc Normal file
View File

@@ -0,0 +1,46 @@
devtools(7)
===========
Name
----
devtools - Developer tools for the Arch Linux distribution
Description
-----------
Devtools contains tools for package maintenance in Arch Linux. The toolset
varies from tools for building packages in a clean chroot ('mkarchroot',...),
packaging related tools for sonames ('sogrep', 'lddd') and tools for
repository management such as ('archco', 'extra2community')
Programs
--------
The list below gives a short overview; see the respective documentation
for details.
linkman:checkpkg[1]
Compare the current build pakcage with the repository version
linkman:find-libdeps[1]
Find soname dependencies for a package
linkman:find-libprovides[1]
Find soname's which are provided by a package
linkman:lddd[1]
Find broken library links on your system
linkman:makerepropkg[1]
Rebuild a package to see if it is reproducible
linkman:mkarchroot[1]
Creates an arch chroot in a specified location with a specified set of
packages
linkman:offload-build[1]
Build a PKGBUILD on a remote server using makechrootpkg
linkman:sogrep[1]
Find packages using a linked to a given shared library
include::footer.asciidoc[]

View File

@@ -7,12 +7,12 @@ makerepropkg - Rebuild a package to see if it is reproducible
Synopsis
--------
makerepropkg [OPTIONS] <package_file>
makerepropkg [OPTIONS] [<package_file|pkgname>...]
Description
-----------
Given the path to a built pacman package, attempt to rebuild it using the
Given the path to a built pacman package(s), attempt to rebuild it using the
PKGBUILD in the current directory. The package will be built in an environment
as closely matching the environment of the initial package as possible, by
building up a chroot to match the information exposed in the package's
@@ -20,12 +20,28 @@ linkman:BUILDINFO[5] manifest. On success, the resulting package will be
compared to the input package, and makerepropkg will report whether the
artifacts are identical.
When given multiple packages, additional package files are assumed to be split
packages and will be treated as additional artifacts to compare during the
verification step.
A valid target(s) for pacman -S can be specified instead, and makerepropkg will
download it to the cache if needed. This is mostly useful to specify which
repository to retrieve from. If no positional arguments are specified, the
targets will be sourced from the PKGBUILD.
In either case, the package name will be converted to a filename from the
cache, and makerepropkg will proceed as though this filename was initially
specified.
This implements a verifier for pacman/libalpm packages in accordance with the
link:https://reproducible-builds.org/[Reproducible Builds] project.
Options
-------
*-d*::
If packages are not reproducible, compare them using diffoscope.
*-c*::
Set the pacman cache directory.

View File

@@ -28,7 +28,7 @@ Options
architecture officially supported by Arch Linux.
*-s, --server* <hostname>::
Offload to a specific build server. The default is dragon.archlinux.org
Offload to a specific build server. The default is build.archlinux.org
which is used as part of the build toolchain for the official Arch Linux
repos.

View File

@@ -142,7 +142,7 @@ find_cached_package() {
for dir in "${searchdirs[@]}"; do
[[ -d $dir ]] || continue
for pkg in "$dir"/*.pkg.tar?(.?z); do
for pkg in "$dir"/*.pkg.tar?(.!(sig|*.*)); do
[[ -f $pkg ]] || continue
# avoid adding duplicates of the same inode
@@ -152,7 +152,7 @@ find_cached_package() {
# split apart package filename into parts
pkgbasename=${pkg##*/}
pkgbasename=${pkgbasename%.pkg.tar?(.?z)}
pkgbasename=${pkgbasename%.pkg.tar*}
arch=${pkgbasename##*-}
pkgbasename=${pkgbasename%-"$arch"}

View File

@@ -242,6 +242,15 @@ download_sources() {
die "Could not download sources."
}
move_logfiles() {
local l
for l in "$copydir"/logdest/*; do
[[ $l == */logpipe.* ]] && continue
chown "$src_owner" "$l"
mv "$l" "$LOGDEST"
done
}
move_products() {
local pkgfile
for pkgfile in "$copydir"/pkgdest/*; do
@@ -254,12 +263,7 @@ move_products() {
fi
done
local l
for l in "$copydir"/logdest/*; do
[[ $l == */logpipe.* ]] && continue
chown "$src_owner" "$l"
mv "$l" "$LOGDEST"
done
move_logfiles
for s in "$copydir"/srcpkgdest/*; do
chown "$src_owner" "$s"
@@ -366,14 +370,11 @@ if arch-nspawn "$copydir" \
"${bindmounts_ro[@]}" "${bindmounts_rw[@]}" \
/chrootbuild "${makepkg_args[@]}"
then
pkgnames=()
for pkgfile in "$copydir"/pkgdest/*; do
pkgfile=${pkgfile##*/};
pkgnames+=("${pkgfile%-*-*-*}");
done
mapfile -t pkgnames < <(sudo -u "$makepkg_user" bash -c 'source PKGBUILD; printf "%s\n" "${pkgname[@]}"')
move_products
else
(( ret += 1 ))
move_logfiles
fi
(( temp_chroot )) && delete_chroot "$copydir" "$copy"
@@ -387,19 +388,30 @@ if (( ret != 0 )); then
else
if (( run_checkpkg )); then
msg "Running checkpkg"
remotepkgs=($(pacman -Syddp --logfile /dev/null "${pkgnames[@]}"|grep '://'))
if (( $? )); then
mapfile -t remotepkgs < <(pacman --config "$copydir"/etc/pacman.conf \
--dbpath "$copydir"/var/lib/pacman \
-Sddp "${pkgnames[@]}")
if ! wait $!; then
warning "Skipped checkpkg due to missing repo packages"
exit 0
fi
for remotepkg in ${remotepkgs[@]}; do
[[ $remotepkg == file://* ]] && continue
msg2 "Downloading current versions"
pacman --noconfirm -Swdd --logfile /dev/null "${pkgnames[@]}"
break
# download package files if any non-local location exists
for remotepkg in "${remotepkgs[@]}"; do
if [[ $remotepkg != file://* ]]; then
msg2 "Downloading current versions"
arch-nspawn "$copydir" pacman --noconfirm -Swdd "${pkgnames[@]}"
mapfile -t remotepkgs < <(pacman --config "$copydir"/etc/pacman.conf \
--dbpath "$copydir"/var/lib/pacman \
-Sddp "${pkgnames[@]}")
break
fi
done
msg2 "Checking packages"
sudo -u "$makepkg_user" checkpkg --rmdir --warn
sudo -u "$makepkg_user" checkpkg --rmdir --warn "${remotepkgs[@]/#file:\/\//}"
fi
true
fi

View File

@@ -15,7 +15,7 @@ DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u'
'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u'
'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u'
'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u'
'rsync::/usr/bin/rsync --no-motd -z %u %o'
'rsync::/usr/bin/rsync --no-motd -zz %u %o'
'scp::/usr/bin/scp -C %u %o')
# Other common tools:
@@ -132,7 +132,7 @@ DBGSRCDIR="/usr/src/debug"
COMPRESSGZ=(gzip -c -f -n)
COMPRESSBZ2=(bzip2 -c -f)
COMPRESSXZ=(xz -c -z -)
COMPRESSZST=(zstd -c -z -q -)
COMPRESSZST=(zstd -c -T0 --ultra -20 -)
COMPRESSLRZ=(lrzip -q)
COMPRESSLZO=(lzop -q)
COMPRESSZ=(compress -c -f)
@@ -143,7 +143,7 @@ COMPRESSLZ=(lzip -c -f)
# EXTENSION DEFAULTS
#########################################################################
#
PKGEXT='.pkg.tar.xz'
PKGEXT='.pkg.tar.zst'
SRCEXT='.src.tar.gz'
# vim: set ft=sh ts=2 sw=2 et:

View File

@@ -29,6 +29,7 @@ declare -a buildenv buildopts installed installpkgs
archiveurl='https://archive.archlinux.org/packages'
buildroot=/var/lib/archbuild/reproducible
chroot=testenv
diffoscope=0
parse_buildinfo() {
local line var val
@@ -56,10 +57,16 @@ parse_buildinfo() {
get_pkgfile() {
local cdir=${cache_dirs[0]}
local pkgfilebase=${1}
local mode=${2}
local pkgname=${pkgfilebase%-*-*-*}
local pkgfile ext
for ext in .xz .zstd ''; do
# try without downloading
if [[ ${mode} != localonly ]] && get_pkgfile "${pkgfilebase}" localonly; then
return 0
fi
for ext in .zst .xz ''; do
pkgfile=${pkgfilebase}.pkg.tar${ext}
for c in "${cache_dirs[@]}"; do
@@ -71,6 +78,9 @@ get_pkgfile() {
for f in "${pkgfile}" "${pkgfile}.sig"; do
if [[ ! -f "${cdir}/${f}" ]]; then
if [[ ${mode} = localonly ]]; then
continue 2
fi
msg2 "retrieving '%s'..." "${f}" >&2
curl -Llf -# -o "${cdir}/${f}" "${archiveurl}/${pkgname:0:1}/${pkgname}/${f}" || continue 2
fi
@@ -94,34 +104,65 @@ package, including the .PKGINFO as well as the buildinfo.
For more details see https://reproducible-builds.org/
OPTIONS
-d Run diffoscope if the package is unreproducible
-c <dir> Set pacman cache
-M <file> Location of a makepkg config file
-h Show this usage message
__EOF__
}
while getopts 'M:c:h' arg; do
case "$arg" in
M) archroot_args+=(-M "$OPTARG") ;;
c) cache_dirs+=("$OPTARG") ;;
h) usage; exit 0 ;;
*|?) usage; exit 1 ;;
esac
while getopts 'dM:c:h' arg; do
case "$arg" in
d) diffoscope=1 ;;
M) archroot_args+=(-M "$OPTARG") ;;
c) cache_dirs+=("$OPTARG") ;;
h) usage; exit 0 ;;
*|?) usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))
check_root
if [[ -n $1 ]]; then
pkgfile="$1"
if ! bsdtar -tqf "${pkgfile}" .BUILDINFO >/dev/null 2>&1; then
error "file is not a valid pacman package: '%s'" "${pkgfile}"
[[ -f PKGBUILD ]] || { error "No PKGBUILD in current directory."; exit 1; }
# without arguments, get list of packages from PKGBUILD
if [[ -z $1 ]]; then
mapfile -t pkgnames < <(source PKGBUILD; pacman -Sddp --print-format '%r/%n' "${pkgname[@]}")
wait $! || {
error "No package file specified and failed to retrieve package names from './PKGBUILD'."
plain "Try '${BASH_SOURCE[0]##*/} -h' for more information." >&2
exit 1
}
msg "Reproducing all pkgnames listed in ./PKGBUILD"
set -- "${pkgnames[@]}"
fi
# check each package to see if it's a file, and if not, try to download it
# using pacman -Sw, and get the filename from there
splitpkgs=()
for p in "$@"; do
if [[ -f ${p} ]]; then
splitpkgs+=("${p}")
else
pkgfile_remote=$(pacman -Sddp "${p}" 2>/dev/null) || { error "package name '%s' not in repos" "${p}"; exit 1; }
pkgfile=${pkgfile_remote#file://}
if [[ ! -f ${pkgfile} ]]; then
msg "Downloading package '%s' into pacman's cache" "${pkgfile}"
sudo pacman -Swdd --noconfirm --logfile /dev/null "${p}" || exit 1
pkgfile_remote=$(pacman -Sddp "${p}" 2>/dev/null)
pkgfile="${pkgfile_remote#file://}"
fi
splitpkgs+=("${pkgfile}")
fi
done
for f in "${splitpkgs[@]}"; do
if ! bsdtar -tqf "${f}" .BUILDINFO >/dev/null 2>&1; then
error "file is not a valid pacman package: '%s'" "${f}"
exit 1
fi
else
error "no package file specified. Try '${BASH_SOURCE[0]##*/} -h' for more information. "
exit 1
fi
done
if (( ${#cache_dirs[@]} == 0 )); then
mapfile -t cache_dirs < <(pacman-conf CacheDir)
@@ -133,10 +174,11 @@ load_makepkg_config
HOME=${ORIG_HOME}
[[ -d ${SRCDEST} ]] || SRCDEST=${PWD}
parse_buildinfo < <(bsdtar -xOqf "${pkgfile}" .BUILDINFO)
parse_buildinfo < <(bsdtar -xOqf "${splitpkgs[0]}" .BUILDINFO)
export SOURCE_DATE_EPOCH="${buildinfo[builddate]}"
PACKAGER="${buildinfo[packager]}"
BUILDDIR="${buildinfo[builddir]}"
PKGEXT=${splitpkgs[0]#${splitpkgs[0]%.pkg.tar*}}
# nuke and restore reproducible testenv
for copy in "${buildroot}"/*/; do
@@ -152,15 +194,14 @@ for fname in "${installed[@]}"; do
exit 1
fi
done
printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1
printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -M @pkgdatadir@/makepkg-x86_64.conf -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1
# use makechrootpkg to prep the build directory
makechrootpkg -r "${buildroot}" -l "${chroot}" -- --packagelist || exit 1
# set detected makepkg.conf options
{
for var in PACKAGER BUILDDIR; do
for var in PACKAGER BUILDDIR PKGEXT; do
printf '%s=%s\n' "${var}" "${!var@Q}"
done
printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}"
@@ -173,18 +214,26 @@ arch-nspawn "${buildroot}/${chroot}" \
--bind="${PWD}:/startdir" \
--bind="${SRCDEST}:/srcdest" \
/chrootbuild -C --noconfirm --log --holdver --skipinteg
ret=$?
if (( $? == 0 )); then
if (( ${ret} == 0 )); then
msg2 "built succeeded! built packages can be found in ${buildroot}/${chroot}/pkgdest"
msg "comparing artifacts..."
if cmp -s "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}"; then
msg2 "Package successfully reproduced!"
exit 0
else
warning "Package is not reproducible. :("
sha256sum "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}"
fi
for pkgfile in "${splitpkgs[@]}"; do
comparefiles=("${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}")
if cmp -s "${comparefiles[@]}"; then
msg2 "Package '%s' successfully reproduced!" "${pkgfile}"
else
ret=1
warning "Package '%s' is not reproducible. :(" "${pkgfile}"
sha256sum "${comparefiles[@]}"
if (( diffoscope )); then
diffoscope "${comparefiles[@]}"
fi
fi
done
fi
# the package either failed to build, or was unreproducible
exit 1
# return failure from chrootbuild, or the reproducibility status
exit ${ret}

View File

@@ -24,7 +24,7 @@ source /usr/share/makepkg/util/config.sh
# global defaults suitable for use by Arch staff
repo=extra
arch=x86_64
server=dragon.archlinux.org
server=build.archlinux.org
die() { printf "error: $1\n" "${@:2}"; exit 1; }
@@ -74,18 +74,23 @@ while (( $# )); do
done
# multilib must be handled specially
archbuild_arch="${arch}"
if [[ $repo = multilib* ]]; then
arch=
archbuild_arch=
fi
archbuild_cmd=("${repo}${arch:+-$arch}-build" "$@")
archbuild_cmd=("${repo}${archbuild_arch:+-$archbuild_arch}-build" "$@")
trap 'rm -rf $SRCPKGDEST' EXIT INT TERM QUIT
trap 'rm -rf $TEMPDIR' EXIT INT TERM QUIT
# Load makepkg.conf variables to be available
load_makepkg_config
# Use a source-only tarball as an intermediate to transfer files. This
# guarantees the checksums are okay, and guarantees that all needed files are
# transferred, including local sources, install scripts, and changelogs.
export SRCPKGDEST=$(mktemp -d)
export TEMPDIR=$(mktemp -d --tmpdir offload-build.XXXXXXXXXX)
export SRCPKGDEST=${TEMPDIR}
makepkg --source || die "unable to make source package"
# Temporary cosmetic workaround makepkg if SRCDEST is set somewhere else
@@ -96,7 +101,7 @@ mapfile -t files < <(
# This is sort of bash golfing but it allows running a mildly complex
# command over ssh with a single connection.
# shellcheck disable=SC2145
cat "$SRCPKGDEST"/*.src.tar.gz |
cat "$SRCPKGDEST"/*"$SRCEXT" |
ssh $server '
temp="${XDG_CACHE_HOME:-$HOME/.cache}/offload-build" &&
mkdir -p "$temp" &&
@@ -108,12 +113,21 @@ mapfile -t files < <(
printf "%s\n" "" "-> build complete" &&
printf "\t%s\n" "$temp"/*
} >&2 &&
makepkg --packagelist
makepkg_user_config="${XDG_CONFIG_HOME:-$HOME/.config}/pacman/makepkg.conf" &&
makepkg_config="/usr/share/devtools/makepkg-'"${arch}"'.conf" &&
if [[ -f /usr/share/devtools/makepkg-'"${repo}"'-'"${arch}"'.conf ]]; then
makepkg_config="/usr/share/devtools/makepkg-'"${repo}"'-'"${arch}"'.conf"
fi &&
makepkg --config <(cat "${makepkg_user_config}" "${makepkg_config}" 2>/dev/null) --packagelist &&
printf "%s\n" "${temp}/PKGBUILD"
')
if (( ${#files[@]} )); then
printf '%s\n' '' '-> copying files...'
load_makepkg_config
scp "${files[@]/#/$server:}" "${PKGDEST:-${PWD}}/"
scp "${files[@]/#/$server:}" "${TEMPDIR}/"
mv "${TEMPDIR}"/*.pkg.tar* "${PKGDEST:-${PWD}}/"
mv "${TEMPDIR}/PKGBUILD" "${PWD}/"
else
exit 1
fi

View File

@@ -94,10 +94,11 @@ _offload_build_args=(
)
_makerepropkg_args=(
'-d[Run diffoscope if the package is unreproducible]'
'-c[Set pacman cache]:pacman_cache:_files -/'
'-M[Location of a makepkg config file]:makepkg_config:_files -g "*.conf(.)"'
'-h[Display usage]'
'1:working_dir:_files -g "*.pkg.tar.*(.)"'
'*:working_dir:_files -g "*.pkg.tar.*(.)"'
)
_devtools_completions_all_packages() {