Compare commits

...

44 Commits

Author SHA1 Message Date
Caleb Maclennan
82caeab9e2 makechrootpkg don't run interactive shell 2021-09-20 12:46:11 +03:00
yoursweetie
9028302ac7 doc: Add arch-nspawn man page 2021-09-08 22:13:36 +02:00
Jelle van der Waa
a9bf4789b3 Sort man page links in man 7 devtools 2021-09-07 22:37:25 +02:00
tsuibin
fd17f53cef doc: Add archbuild man page 2021-09-07 22:23:32 +02:00
yoursweetie
80e8c1fc70 doc: Add makechrootpkg man page 2021-09-07 20:55:42 +02:00
Felix Yan
6535ac9b99 Fix CI and switch to GitHub CI (#68) 2021-09-04 22:19:59 +08:00
bartus
c5c5dbc64f Drop pacutils:pacconf fallback
As of pacman:5.2 `pacman-conf` obsoletes `pacconf`
2021-07-17 23:10:30 +02:00
Levente Polyak
0a0e66a784 feat: support exposing buildtool to aid reproducible builds configs
This helps to map the correct build tool configs that are required to
reproduce a specific package and have the appropriate *FLAGS etc.
2021-07-17 22:56:11 +02:00
Levente Polyak
60e96c9d4b makerepropkg: avoid competing redirects into the same makepkg.conf file 2021-07-17 22:45:21 +02:00
Erich Eckner
30ed6920c7 allow to call setarch with a different value than $CARCH
Introduce setarch-aliases.d/ which gets installed inside
/usr/share/devtools. This allows to assign aliases which map one CARCH
to a different name which gets provided as argument for setarch. This is
necessary on archlinuxarm ("armv6h" -> "armv6l", "armv7h" -> "armv7l")
and allows for more fine-grained architectures (e.g. archlinux32 has
"i686" and "pentium4", which differ in the required cpu capabilities).

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2021-07-17 22:35:04 +02:00
Chih-Hsuan Yen
4602659068 zsh-completion: complete makechrootpkg arguments after *-build -- 2021-07-01 20:25:38 +02:00
Levente Polyak
43d58212c5 pacman.conf: enable ParallelDownloads and NoProgressBar
The progress bar feature creates noisy log files, especially with
parallel downloads. Lets disable the progress bar and instead use
parallel downloads.
2021-06-19 22:26:25 +02:00
Levente Polyak
6a628af422 pacman.conf: enable VerbosePkgLists for easier log consumption 2021-06-19 21:18:37 +02:00
Levente Polyak
0635f0c5ec sync pacman.conf with the pacman 6.0.0-3 package
This effectively does nothing but keep the commented options in sync to
not deviate from the original file for no reason.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2021-06-17 00:05:06 +02:00
Levente Polyak
04af0374f3 sync makepkg.conf with the pacman 6.0.0-3 package
The only effective difference is that -Wp,-D_GLIBCXX_ASSERTIONS is now
only defined for CXXFLAGS as of pacman 6.0.0-1

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2021-06-17 00:04:52 +02:00
Eli Schwartz
54e03641a3 sync makepkg.conf with the pacman package
Aside for comments, this implements the buildflags RFC from:

https://lists.archlinux.org/pipermail/arch-dev-public/2021-March/030374.html
https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/3
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-06-16 23:42:23 +02:00
Evangelos Foutras
385b47e56e sogrep: refresh if any database is over a day old
Outdated sogrep cache has already bit us once in the past. Finding one
or more databases older than a day is a good indication that a refresh
is in order, so do that automatically.
2021-06-16 23:37:10 +02:00
Evangelos Foutras
9d39abbefe sogrep: store unextracted *.links.tar.gz databases
Extracting these databases is painfully slow on HDDs (especially laptop
ones). There shouldn't be a drawback to keeping the tarballs around and
extracting them to a temporary directory (usually tmpfs) to parse them.

The implemented update logic tries to avoid redownloading unchanged dbs.
2021-06-16 23:37:06 +02:00
Levente Polyak
90ba07d9be Version 20210202 2021-02-02 00:51:57 +01:00
Levente Polyak
8c26438df8 makechrootpkg: only expose failed build logs instead of products
After further followups always moving all products isn't actually
desired as they can theoretically be broken in various ways if
arch-nspawn exists non successful.

However, as we would like to always preserve the produced log files we
instead split out the logfiles into an own function and call that for
unsuccessful buils.

Fixes 4f305aa3
2021-01-31 23:44:11 +01:00
Christian Hesse
a49bcf2097 doc: update default build host
Commit 09e169b741 changed the default
build host from dragon.archlinux.org to build.archlinux.org - match
in documentation.

Signed-off-by: Christian Hesse <mail@eworm.de>
2021-01-31 02:16:44 +01:00
Eli Schwartz
0883f45b3a makerepropkg: allow specifying the package in pacman -S format
We now accept:

1) # nothing

    in which case we'll use the PKGBUILD to retrieve...

2) name, or repo/name

    in which case we'll use pacman to cache the package and retrieve...

3) a filename

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-01-27 00:56:48 +01:00
Eli Schwartz
99c2020d47 makerepropkg: do fast cache lookups
Teach get_pkgfile to call itself in local-only mode and find a cached
file no matter what its extension is. Avoids repetitively trying to curl
random files, fail with 404 errors, and proceed to discover a cache hit
under a different file extension.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-01-27 00:56:48 +01:00
Christian Hesse
00f7a6a415 makepkg.conf: make rsync use new-style compression in makepkg.conf
Our rsync package is no longer built with bundled zlib, so old-style
compression is no longer supported.

https://www.archlinux.org/news/rsync-compatibility/

Signed-off-by: Christian Hesse <mail@eworm.de>
2021-01-26 20:56:46 +01:00
bartoszek
4f305aa316 makechrootpkg: Expose failed build logs/products
Since move_products() function is fairly robust we can make it run for
failed build also to expose logs for packages that fails in build(),
prepare() or package(). It also exposes partially packaged split
packages if they fail in latter package_xxx().
2021-01-26 20:43:29 +01:00
Levente Polyak
09e169b741 offload-build: change default host to build.archlinux.org 2021-01-26 20:40:47 +01:00
Jelle van der Waa
360a7611a8 doc: add devtools.7 man page
Signed-off-by: Jelle van der Waa <jelle@vdwaa.nl>
2021-01-26 20:39:06 +01:00
Konstantin Gizdov
d507db9490 offload-build: respect SRCEXT from makepkg.conf 2021-01-26 20:38:53 +01:00
Levente Polyak
aff81d34fd Version 20200407 2020-04-07 19:46:10 +02:00
Levente Polyak
5b1123e11f offload-build: retrieve the PKGBUILD in case makepkg changes it via pkgver
Building a package may change the PKGBUILD during update_pkgver. Let's
retrieve the PKGBUILD after building to ensure we have the very same
file as the one we used to build the package. Otherwise this may lead to
the inability to distribute the package during commitpkg in case the
expected and the actual hashsum mismatch.
2020-03-09 23:27:03 +01:00
Levente Polyak
a0f79fcce0 makerepropkg: prioritize downloading .zst packages over legacy format
First try a .zst location before falling back to legacy variants. This
should slightly speed up downloading of dependencies, especially over
time as .zst packages are or will be the dominant format.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-03-09 23:27:03 +01:00
Eli Schwartz
94b0413e13 arch-nspawn: fix up host_mirrors cachedir handling
This was incorrectly implemented in commit
0067176529, which added the host_mirrors
root directory as a cachedir, when we actually want to use the pool/*
subdirectories (the same ones installed on the build server's
/etc/pacman.conf).

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:27 +01:00
Eli Schwartz
e963b6da9e makechrootpkg: use the chroot database to find checkpkg packages
We don't want to check against the current version known to the host
system, because that will be incorrect in a wide variety of situations,
including:
- the build host hasn't done a full system upgrade yet
- we're building against staging, and want to see the delta between
  different staging versions
- we're building against extra, but the host runs testing which carries
  changes we don't want to visualize right now
- the chroot has a configured database not available to the host, and
  the package is only available there

Essentially, it's rarely 100% correct to run checkpkg on the host, but
we already have a database we *know* is correct, and that is the one we
just built the package against. So let's use that.

This also fixes a bug in the current logic, where in order to try
downloading fresh databases, we work in a non-cached temporary working
database to download the package files, but then let checkpkg default to
comparing packages in the system database. Since we are explicitly
trying to compare against packages that differ from the host's pacman
database, we need to pass the package files as options to checkpkg,
using the additional modes added in commit c14338c0fe

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:19 +01:00
Eli Schwartz
b2cbb8628e checkpkg: fix support for http:// url comparisons
Broken in commit c14338c0fe.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:14 +01:00
Eli Schwartz
6d273f79c3 zsh-completion: update for new makerepropkg options
Thanks to anthraxx for the guidance.

Original-patch-by: Levente Polyak <anthraxx@archlinux.org>
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:50:05 +01:00
Eli Schwartz
51842a1676 makerepropkg: support checking multiple split packages
By specifying multiple package files, we assume they are all from the
same PKGBUILD, and try to check them all against the produced artifacts.
Since the buildinfo should be comparable for all of them, we simply use
the first one passed on the command line.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:49:54 +01:00
Eli Schwartz
53fe5c67a1 makerepropkg: add support to check unreproducible packages using diffoscope
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:49:48 +01:00
Eli Schwartz
21d9984acc makerepropkg: fix wonky indent
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2020-02-27 14:49:33 +01:00
Levente Polyak
cb6484fe45 Version 20200213 2020-02-13 01:53:45 +01:00
Levente Polyak
bcb1b4a163 offload-build: handle user specific makepkg.conf on the remote host
This ensures we take user specific config values for PKGDEST into
account when printing the package list. This is required as devtools
archbuild_cmd puts packages potentially into the user defined PKGDEST
which the package list would otherwise miss.
2020-02-12 00:05:00 +01:00
Levente Polyak
57fb44b976 offload-build: take makepkg.conf from devtools for appropriate packagelist
This fixes an issue with the usage of makepkg --packagelist to get the
produced artifacts filenames according to the PKGEXT used in devtools'
makepkg.conf instead of the one defined in pacman.

One goal we want to preserve is that devtools configuration should be
self contained and not require any editing of non owned files like
the host /etc/makepkg.conf to produce expected results.

Additionally modify the archbuild_cmd override for multilib builds to
use an independent variable and not fiddle with the actual arch
variable to select the appropriate cmd.
2020-02-11 01:42:17 +01:00
Levente Polyak
9b4d8ae930 offload-build: convert to in-prog so we can perform pre-processing 2020-02-10 23:02:41 +01:00
Levente Polyak
4c206ab549 makerepropkg: take makepkg.conf from devtools pkgdatadir location
This ensures we use the same configuration for reproducing packages as
we use for building them via devtools.
One example of why we care about this are the COMPRESS* settings that
may differ from the guest's pacman shipped makepkg.conf that affect the
reproducibility of packages.
2020-02-10 23:02:16 +01:00
Eli Schwartz
4c08847bfa makerepropkg: correctly reproduce a pkgfile with any compression type
We don't want the default PKGEXT in the current version of devtools, we
want the PKGEXT we *know* the input file used.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>
2019-12-27 18:54:26 +01:00
29 changed files with 517 additions and 148 deletions

15
.github/workflows/test.yml vendored Normal file
View File

@@ -0,0 +1,15 @@
on: push
jobs:
test:
runs-on: ubuntu-latest
container: archlinux/archlinux:latest
steps:
- name: Install dependencies
run: pacman -Syu --noconfirm m4 make openssh subversion rsync arch-install-scripts git bzr mercurial diffutils asciidoc shellcheck
- uses: actions/checkout@v2
- name: Run tests
run: |
make PREFIX=/usr
make PREFIX=/usr DESTDIR="$(mktemp -d)" install
make check || true
SHELLCHECK_OPTS="-S error" make check

2
.gitignore vendored
View File

@@ -11,6 +11,7 @@ lddd
makechrootpkg makechrootpkg
makerepropkg makerepropkg
mkarchroot mkarchroot
offload-build
rebuildpkgs rebuildpkgs
zsh_completion zsh_completion
find-libdeps find-libdeps
@@ -18,3 +19,4 @@ crossrepomove
arch-nspawn arch-nspawn
sogrep sogrep
doc/*.1 doc/*.1
doc/*.7

View File

@@ -1,28 +0,0 @@
language: shell
sudo: required
services:
- docker
archlinux:
packages:
- openssh
- subversion
- rsync
- arch-install-scripts
- git
- bzr
- mercurial
- diffutils
- asciidoc
- shellcheck
script:
- sudo pacman -Syu --noconfirm --needed "${CONFIG_PACKAGES[@]}"
- make PREFIX=/usr
- make PREFIX=/usr DESTDIR="$(mktemp -d)" install
- make check || true
- SHELLCHECK_OPTS="-S error" make check
script: 'curl -s https://raw.githubusercontent.com/mikkeloscar/arch-travis/master/arch-travis.sh | bash'
# vim: ft=yaml ts=2 sw=2 et:

View File

@@ -1,4 +1,4 @@
V=20191227 V=20210202
PREFIX = /usr/local PREFIX = /usr/local
MANDIR = $(PREFIX)/share/man MANDIR = $(PREFIX)/share/man
@@ -17,12 +17,12 @@ IN_PROGS = \
makerepropkg \ makerepropkg \
mkarchroot \ mkarchroot \
makechrootpkg \ makechrootpkg \
offload-build \
rebuildpkgs \ rebuildpkgs \
sogrep sogrep
BINPROGS = \ BINPROGS = \
$(IN_PROGS) \ $(IN_PROGS)
offload-build \
CONFIGFILES = \ CONFIGFILES = \
makepkg-x86_64.conf \ makepkg-x86_64.conf \
@@ -35,6 +35,8 @@ CONFIGFILES = \
pacman-kde-unstable.conf \ pacman-kde-unstable.conf \
pacman-gnome-unstable.conf pacman-gnome-unstable.conf
SETARCH_ALIASES = \
COMMITPKG_LINKS = \ COMMITPKG_LINKS = \
extrapkg \ extrapkg \
testingpkg \ testingpkg \
@@ -68,6 +70,9 @@ BASHCOMPLETION_LINKS = \
MANS = \ MANS = \
doc/archbuild.1 \
doc/arch-nspawn.1 \
doc/makechrootpkg.1 \
doc/lddd.1 \ doc/lddd.1 \
doc/checkpkg.1 \ doc/checkpkg.1 \
doc/offload-build.1 \ doc/offload-build.1 \
@@ -75,7 +80,8 @@ MANS = \
doc/makerepropkg.1 \ doc/makerepropkg.1 \
doc/mkarchroot.1 \ doc/mkarchroot.1 \
doc/find-libdeps.1 \ doc/find-libdeps.1 \
doc/find-libprovides.1 doc/find-libprovides.1 \
doc/devtools.7
all: $(BINPROGS) bash_completion zsh_completion man all: $(BINPROGS) bash_completion zsh_completion man
@@ -86,7 +92,7 @@ edit = sed -e "s|@pkgdatadir[@]|$(PREFIX)/share/devtools|g"
%: %.in Makefile lib/common.sh %: %.in Makefile lib/common.sh
@echo "GEN $@" @echo "GEN $@"
@$(RM) "$@" @$(RM) "$@"
@{ echo -n 'm4_changequote([[[,]]])'; cat $@.in; } | m4 -P | $(edit) >$@ @{ echo -n 'm4_changequote([[[,]]])'; cat $@.in; } | m4 -P --define=m4_devtools_version=$V | $(edit) >$@
@chmod a-w "$@" @chmod a-w "$@"
@chmod +x "$@" @chmod +x "$@"
@bash -O extglob -n "$@" @bash -O extglob -n "$@"
@@ -94,16 +100,17 @@ edit = sed -e "s|@pkgdatadir[@]|$(PREFIX)/share/devtools|g"
$(MANS): doc/asciidoc.conf doc/footer.asciidoc $(MANS): doc/asciidoc.conf doc/footer.asciidoc
doc/%: doc/%.asciidoc doc/%: doc/%.asciidoc
a2x --no-xmllint --asciidoc-opts="-f doc/asciidoc.conf" -d manpage -f manpage -D doc $< a2x --no-xmllint --asciidoc-opts="-f doc/asciidoc.conf" -d manpage -f manpage -D doc -a pkgdatadir=$(PREFIX)/share/devtools $<
clean: clean:
rm -f $(IN_PROGS) bash_completion zsh_completion $(MANS) rm -f $(IN_PROGS) bash_completion zsh_completion $(MANS)
install: install:
install -dm0755 $(DESTDIR)$(PREFIX)/bin install -dm0755 $(DESTDIR)$(PREFIX)/bin
install -dm0755 $(DESTDIR)$(PREFIX)/share/devtools install -dm0755 $(DESTDIR)$(PREFIX)/share/devtools/setarch-aliases.d
install -m0755 ${BINPROGS} $(DESTDIR)$(PREFIX)/bin install -m0755 ${BINPROGS} $(DESTDIR)$(PREFIX)/bin
install -m0644 ${CONFIGFILES} $(DESTDIR)$(PREFIX)/share/devtools install -m0644 ${CONFIGFILES} $(DESTDIR)$(PREFIX)/share/devtools
for a in ${SETARCH_ALIASES}; do install -m0644 setarch-aliases.d/$$a $(DESTDIR)$(PREFIX)/share/devtools/setarch-aliases.d; done
for l in ${COMMITPKG_LINKS}; do ln -sf commitpkg $(DESTDIR)$(PREFIX)/bin/$$l; done for l in ${COMMITPKG_LINKS}; do ln -sf commitpkg $(DESTDIR)$(PREFIX)/bin/$$l; done
for l in ${ARCHBUILD_LINKS}; do ln -sf archbuild $(DESTDIR)$(PREFIX)/bin/$$l; done for l in ${ARCHBUILD_LINKS}; do ln -sf archbuild $(DESTDIR)$(PREFIX)/bin/$$l; done
for l in ${CROSSREPOMOVE_LINKS}; do ln -sf crossrepomove $(DESTDIR)$(PREFIX)/bin/$$l; done for l in ${CROSSREPOMOVE_LINKS}; do ln -sf crossrepomove $(DESTDIR)$(PREFIX)/bin/$$l; done
@@ -119,6 +126,7 @@ install:
uninstall: uninstall:
for f in ${BINPROGS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$f; done for f in ${BINPROGS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$f; done
for f in ${CONFIGFILES}; do rm -f $(DESTDIR)$(PREFIX)/share/devtools/$$f; done for f in ${CONFIGFILES}; do rm -f $(DESTDIR)$(PREFIX)/share/devtools/$$f; done
for f in ${SETARCH_ALIASES}; do rm -f $(DESTDIR)$(PREFIX)/share/devtools/setarch-aliases.d/$$f; done
for l in ${COMMITPKG_LINKS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$l; done for l in ${COMMITPKG_LINKS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$l; done
for l in ${ARCHBUILD_LINKS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$l; done for l in ${ARCHBUILD_LINKS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$l; done
for l in ${CROSSREPOMOVE_LINKS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$l; done for l in ${CROSSREPOMOVE_LINKS}; do rm -f $(DESTDIR)$(PREFIX)/bin/$$l; done

View File

@@ -57,24 +57,24 @@ shift 1
[[ -z $working_dir ]] && die 'Please specify a working directory.' [[ -z $working_dir ]] && die 'Please specify a working directory.'
pacconf_cmd=$(command -v pacman-conf || command -v pacconf)
if (( ${#cache_dirs[@]} == 0 )); then if (( ${#cache_dirs[@]} == 0 )); then
mapfile -t cache_dirs < <($pacconf_cmd --config "${pac_conf:-$working_dir/etc/pacman.conf}" CacheDir) mapfile -t cache_dirs < <(pacman-conf --config "${pac_conf:-$working_dir/etc/pacman.conf}" CacheDir)
fi fi
# shellcheck disable=2016 # shellcheck disable=2016
host_mirrors=($($pacconf_cmd --repo extra Server 2> /dev/null | sed -r 's#(.*/)extra/os/.*#\1$repo/os/$arch#')) host_mirrors=($(pacman-conf --repo extra Server 2> /dev/null | sed -r 's#(.*/)extra/os/.*#\1$repo/os/$arch#'))
for host_mirror in "${host_mirrors[@]}"; do for host_mirror in "${host_mirrors[@]}"; do
if [[ $host_mirror == *file://* ]]; then if [[ $host_mirror == *file://* ]]; then
host_mirror=$(echo "$host_mirror" | sed -r 's#file://(/.*)/\$repo/os/\$arch#\1#g') host_mirror=$(echo "$host_mirror" | sed -r 's#file://(/.*)/\$repo/os/\$arch#\1#g')
in_array "$host_mirror" "${cache_dirs[@]}" || cache_dirs+=("$host_mirror") for m in "$host_mirror"/pool/*/; do
in_array "$m" "${cache_dirs[@]}" || cache_dirs+=("$m")
done
fi fi
done done
while read -r line; do while read -r line; do
mapfile -t lines < <($pacconf_cmd --config "${pac_conf:-$working_dir/etc/pacman.conf}" \ mapfile -t lines < <(pacman-conf --config "${pac_conf:-$working_dir/etc/pacman.conf}" \
--repo $line Server | sed -r 's#(.*/)[^/]+/os/.+#\1#') --repo $line Server | sed -r 's#(.*/)[^/]+/os/.+#\1#')
for line in "${lines[@]}"; do for line in "${lines[@]}"; do
if [[ $line = file://* ]]; then if [[ $line = file://* ]]; then
@@ -82,7 +82,7 @@ while read -r line; do
in_array "$line" "${cache_dirs[@]}" || cache_dirs+=("$line") in_array "$line" "${cache_dirs[@]}" || cache_dirs+=("$line")
fi fi
done done
done < <($pacconf_cmd --config "${pac_conf:-$working_dir/etc/pacman.conf}" --repo-list) done < <(pacman-conf --config "${pac_conf:-$working_dir/etc/pacman.conf}" --repo-list)
mount_args+=("--bind=${cache_dirs[0]//:/\\:}") mount_args+=("--bind=${cache_dirs[0]//:/\\:}")
@@ -124,8 +124,13 @@ copy_hostconf
eval "$(grep -a '^CARCH=' "$working_dir/etc/makepkg.conf")" eval "$(grep -a '^CARCH=' "$working_dir/etc/makepkg.conf")"
[[ -z $nosetarch ]] || unset CARCH [[ -z $nosetarch ]] || unset CARCH
if [[ -f "@pkgdatadir@/setarch-aliases.d/${CARCH}" ]]; then
read -r set_arch < "@pkgdatadir@/setarch-aliases.d/${CARCH}"
else
set_arch="${CARCH}"
fi
exec ${CARCH:+setarch "$CARCH"} systemd-nspawn -q \ exec ${CARCH:+setarch "$set_arch"} systemd-nspawn -q \
-D "$working_dir" \ -D "$working_dir" \
-E "PATH=/usr/local/sbin:/usr/local/bin:/usr/bin" \ -E "PATH=/usr/local/sbin:/usr/local/bin:/usr/bin" \
--register=no --keep-unit --as-pid2 \ --register=no --keep-unit --as-pid2 \

View File

@@ -17,6 +17,11 @@ else
repo=${tag%-*} repo=${tag%-*}
arch=${tag##*-} arch=${tag##*-}
fi fi
if [[ -f "@pkgdatadir@/setarch-aliases.d/${arch}" ]]; then
read -r set_arch < "@pkgdatadir@/setarch-aliases.d/${arch}"
else
set_arch="${arch}"
fi
chroots='/var/lib/archbuild' chroots='/var/lib/archbuild'
clean_first=false clean_first=false
@@ -69,7 +74,7 @@ if ${clean_first} || [[ ! -d "${chroots}/${repo}-${arch}" ]]; then
rm -rf --one-file-system "${chroots}/${repo}-${arch}" rm -rf --one-file-system "${chroots}/${repo}-${arch}"
(umask 0022; mkdir -p "${chroots}/${repo}-${arch}") (umask 0022; mkdir -p "${chroots}/${repo}-${arch}")
setarch "${arch}" mkarchroot \ setarch "${set_arch}" mkarchroot \
-C "${pacman_config}" \ -C "${pacman_config}" \
-M "${makepkg_config}" \ -M "${makepkg_config}" \
"${chroots}/${repo}-${arch}/root" \ "${chroots}/${repo}-${arch}/root" \

View File

@@ -95,8 +95,10 @@ for _pkgname in "${pkgname[@]}"; do
if (( $# )); then if (( $# )); then
case $1 in case $1 in
*://*)
pkgurl=$1 ;;
/*|*/*) /*|*/*)
pkgurl=file://$(readlink -m "$1") ;; pkgurl=$(readlink -m "$1") ;;
*.pkg.tar*) *.pkg.tar*)
pkgurl=$1 ;; pkgurl=$1 ;;
'') '')

View File

@@ -0,0 +1,40 @@
arch-nspawn(1)
==============
Name
----
arch-nspawn - Run a command or OS in a light-weight namespace container
Synopsis
--------
arch-nspawn [options] working-dir [systemd-nspawn arguments]
Description
-----------
'arch-nspawn' is a wrapper around systemd-nspawn to run command or OS in a
namespace container such as a directory including base utilities of a OS.
It is used to build package(s) in given clean and defined environment.
Options
-------
*-C* <file>::
Location of a pacman config file
*-M* <file>::
Location of a makepkg config file
*-c* <dir>::
Set pacman cache, if no directory is specified the passed pacman.conf's cachedir is used with a fallback to '/etc/pacman.conf'
*-f* <file>::
Copy file from the host to the chroot
*-s*::
Do not run setarch
*-h*::
Show this usage message
include::footer.asciidoc[]

47
doc/archbuild.1.asciidoc Normal file
View File

@@ -0,0 +1,47 @@
archbuild(1)
============
Name
----
archbuild - a script to build an Arch Linux package inside a clean chroot.
Synopsis
--------
archbuild [options] -- [makechrootpkg args]
Description
-----------
'archbuild' is a script to build an Arch Linux package. archbuild is part of devtools but should only be used via one of the included symlinks:
* extra-x86_64-build
* gnome-unstable-x86_64-build
* kde-unstable-x86_64-build
* multilib-build
* multilib-staging-build
* multilib-testing-build
* staging-x86_64-build
* testing-x86_64-build
The symlink used to run it will be inspected by archbuild, to determine which target you want it to use. It will load the available pacman configuration from 'pacman-reponame-arch.conf' with a fallback to 'pacman-reponame.conf' from {pkgdatadir}. The makepkg configuration is loaded from 'makepkg-repo-arch.conf' with a fallback to 'makepkg-reponame.conf' from {pkgdatadir}.
Options
-------
*-h*::
Output command line options.
*-c*::
Recreate the chroot before building.
*-r* <dir>::
Create chroots in this directory.
See Also
--------
linkman:devtools[7]
include::footer.asciidoc[]

55
doc/devtools.7.asciidoc Normal file
View File

@@ -0,0 +1,55 @@
devtools(7)
===========
Name
----
devtools - Developer tools for the Arch Linux distribution
Description
-----------
Devtools contains tools for package maintenance in Arch Linux. The toolset
varies from tools for building packages in a clean chroot ('mkarchroot',...),
packaging related tools for sonames ('sogrep', 'lddd') and tools for
repository management such as ('archco', 'extra2community')
Programs
--------
The list below gives a short overview; see the respective documentation
for details.
linkman:archbuild[1]
Build an Arch Linux package inside a clean chroot
linkman:arch-nspawn[1]
Run a command or OS in a light-weight namespace container
linkman:checkpkg[1]
Compare the current build pakcage with the repository version
linkman:find-libdeps[1]
Find soname dependencies for a package
linkman:find-libprovides[1]
Find soname's which are provided by a package
linkman:lddd[1]
Find broken library links on your system
linkman:mkarchroot[1]
Creates an arch chroot in a specified location with a specified set of
packages
linkman:makechrootpkg[1]
Build a PKGBUILD in a given chroot environment
linkman:makerepropkg[1]
Rebuild a package to see if it is reproducible
linkman:offload-build[1]
Build a PKGBUILD on a remote server using makechrootpkg
linkman:sogrep[1]
Find packages using a linked to a given shared library
include::footer.asciidoc[]

View File

@@ -0,0 +1,76 @@
makechrootpkg(1)
================
Name
----
makechrootpkg - Build a PKGBUILD in a given chroot environment
Synopsis
--------
makechrootpkg [OPTIONS] -r <chrootdir> [--] [makepkg args]
Description
-----------
Run this script in a directory containing a PKGBUILD to build a package
inside a clean chroot. Arguments passed to this script after the
end-of-options marker (--) will be passed to makepkg.
The chroot dir consists of the following directories:
<chrootdir>/{root, copy} but only "root" is required
by default. The working copy will be created as needed
The chroot "root" directory must be created via the following
command:
mkarchroot <chrootdir>/root base-devel
This script reads {SRC,SRCPKG,PKG,LOG}DEST, MAKEFLAGS and PACKAGER
from makepkg.conf(5), if those variables are not part of the
environment.
Default makepkg args: --syncdeps --noconfirm --log --holdver --skipinteg
Options
-------
*-h*::
Show this usage message
*-c*::
Clean the chroot before building
*-d* <dir>::
Bind directory into build chroot as read-write
*-D* <dir>::
Bind directory into build chroot as read-only
*-u*::
Update the working copy of the chroot before building
This is useful for rebuilds without dirtying the pristine
chroot
*-r* <dir>::
The chroot dir to use
*-I* <pkg>::
Install a package into the working copy of the chroot
*-l* <copy>::
The directory to use as the working copy of the chroot
Useful for maintaining multiple copies
Default: $USER
*-n*::
Run namcap on the build package
*-C*::
Run checkpkg on the build package
*-T*::
Build in a temporary directory
*-U*::
Run makepkg as a specified user
include::footer.asciidoc[]

View File

@@ -7,12 +7,12 @@ makerepropkg - Rebuild a package to see if it is reproducible
Synopsis Synopsis
-------- --------
makerepropkg [OPTIONS] <package_file> makerepropkg [OPTIONS] [<package_file|pkgname>...]
Description Description
----------- -----------
Given the path to a built pacman package, attempt to rebuild it using the Given the path to a built pacman package(s), attempt to rebuild it using the
PKGBUILD in the current directory. The package will be built in an environment PKGBUILD in the current directory. The package will be built in an environment
as closely matching the environment of the initial package as possible, by as closely matching the environment of the initial package as possible, by
building up a chroot to match the information exposed in the package's building up a chroot to match the information exposed in the package's
@@ -20,12 +20,28 @@ linkman:BUILDINFO[5] manifest. On success, the resulting package will be
compared to the input package, and makerepropkg will report whether the compared to the input package, and makerepropkg will report whether the
artifacts are identical. artifacts are identical.
When given multiple packages, additional package files are assumed to be split
packages and will be treated as additional artifacts to compare during the
verification step.
A valid target(s) for pacman -S can be specified instead, and makerepropkg will
download it to the cache if needed. This is mostly useful to specify which
repository to retrieve from. If no positional arguments are specified, the
targets will be sourced from the PKGBUILD.
In either case, the package name will be converted to a filename from the
cache, and makerepropkg will proceed as though this filename was initially
specified.
This implements a verifier for pacman/libalpm packages in accordance with the This implements a verifier for pacman/libalpm packages in accordance with the
link:https://reproducible-builds.org/[Reproducible Builds] project. link:https://reproducible-builds.org/[Reproducible Builds] project.
Options Options
------- -------
*-d*::
If packages are not reproducible, compare them using diffoscope.
*-c*:: *-c*::
Set the pacman cache directory. Set the pacman cache directory.

View File

@@ -28,7 +28,7 @@ Options
architecture officially supported by Arch Linux. architecture officially supported by Arch Linux.
*-s, --server* <hostname>:: *-s, --server* <hostname>::
Offload to a specific build server. The default is dragon.archlinux.org Offload to a specific build server. The default is build.archlinux.org
which is used as part of the build toolchain for the official Arch Linux which is used as part of the build toolchain for the official Arch Linux
repos. repos.

View File

@@ -14,6 +14,10 @@ $_INCLUDE_COMMON_SH
# Avoid any encoding problems # Avoid any encoding problems
export LANG=C export LANG=C
# Set buildtool properties
export BUILDTOOL=devtools
export BUILDTOOLVER=m4_devtools_version
shopt -s extglob shopt -s extglob
# check if messages are to be printed using color # check if messages are to be printed using color

View File

@@ -190,6 +190,8 @@ EOF
printf '#!/bin/bash\n' printf '#!/bin/bash\n'
declare -f _chrootbuild declare -f _chrootbuild
declare -p SOURCE_DATE_EPOCH 2>/dev/null || true declare -p SOURCE_DATE_EPOCH 2>/dev/null || true
declare -p BUILDTOOL 2>/dev/null
declare -p BUILDTOOLVER 2>/dev/null
printf '_chrootbuild "$@" || exit\n' printf '_chrootbuild "$@" || exit\n'
if (( run_namcap )); then if (( run_namcap )); then
@@ -213,7 +215,10 @@ _chrootbuild() {
# use "$" in arguments to commands with "sudo -i". ${foo} or # use "$" in arguments to commands with "sudo -i". ${foo} or
# ${1} is OK, but $foo or $1 isn't. # ${1} is OK, but $foo or $1 isn't.
# https://bugzilla.sudo.ws/show_bug.cgi?id=765 # https://bugzilla.sudo.ws/show_bug.cgi?id=765
sudo --preserve-env=SOURCE_DATE_EPOCH -iu builduser bash -c 'cd /startdir; makepkg "$@"' -bash "$@" sudo --preserve-env=SOURCE_DATE_EPOCH \
--preserve-env=BUILDTOOL \
--preserve-env=BUILDTOOLVER \
-u builduser bash -c 'cd /startdir; makepkg "$@"' -bash "$@"
ret=$? ret=$?
case $ret in case $ret in
0|14) 0|14)
@@ -242,6 +247,15 @@ download_sources() {
die "Could not download sources." die "Could not download sources."
} }
move_logfiles() {
local l
for l in "$copydir"/logdest/*; do
[[ $l == */logpipe.* ]] && continue
chown "$src_owner" "$l"
mv "$l" "$LOGDEST"
done
}
move_products() { move_products() {
local pkgfile local pkgfile
for pkgfile in "$copydir"/pkgdest/*; do for pkgfile in "$copydir"/pkgdest/*; do
@@ -254,12 +268,7 @@ move_products() {
fi fi
done done
local l move_logfiles
for l in "$copydir"/logdest/*; do
[[ $l == */logpipe.* ]] && continue
chown "$src_owner" "$l"
mv "$l" "$LOGDEST"
done
for s in "$copydir"/srcpkgdest/*; do for s in "$copydir"/srcpkgdest/*; do
chown "$src_owner" "$s" chown "$src_owner" "$s"
@@ -294,7 +303,7 @@ done
[[ -n $makepkg_user && -z $(id -u "$makepkg_user") ]] && die 'Invalid makepkg user.' [[ -n $makepkg_user && -z $(id -u "$makepkg_user") ]] && die 'Invalid makepkg user.'
makepkg_user=${makepkg_user:-${SUDO_USER:-$USER}} makepkg_user=${makepkg_user:-${SUDO_USER:-$USER}}
check_root SOURCE_DATE_EPOCH,GNUPGHOME,SRCDEST,SRCPKGDEST,PKGDEST,LOGDEST,MAKEFLAGS,PACKAGER check_root SOURCE_DATE_EPOCH,BUILDTOOL,BUILDTOOLVER,GNUPGHOME,SRCDEST,SRCPKGDEST,PKGDEST,LOGDEST,MAKEFLAGS,PACKAGER
# Canonicalize chrootdir, getting rid of trailing / # Canonicalize chrootdir, getting rid of trailing /
chrootdir=$(readlink -e "$passeddir") chrootdir=$(readlink -e "$passeddir")
@@ -366,14 +375,11 @@ if arch-nspawn "$copydir" \
"${bindmounts_ro[@]}" "${bindmounts_rw[@]}" \ "${bindmounts_ro[@]}" "${bindmounts_rw[@]}" \
/chrootbuild "${makepkg_args[@]}" /chrootbuild "${makepkg_args[@]}"
then then
pkgnames=() mapfile -t pkgnames < <(sudo -u "$makepkg_user" bash -c 'source PKGBUILD; printf "%s\n" "${pkgname[@]}"')
for pkgfile in "$copydir"/pkgdest/*; do
pkgfile=${pkgfile##*/};
pkgnames+=("${pkgfile%-*-*-*}");
done
move_products move_products
else else
(( ret += 1 )) (( ret += 1 ))
move_logfiles
fi fi
(( temp_chroot )) && delete_chroot "$copydir" "$copy" (( temp_chroot )) && delete_chroot "$copydir" "$copy"
@@ -388,29 +394,29 @@ else
if (( run_checkpkg )); then if (( run_checkpkg )); then
msg "Running checkpkg" msg "Running checkpkg"
# sync off-site databases for up-to-date queries mapfile -t remotepkgs < <(pacman --config "$copydir"/etc/pacman.conf \
trap 'rm -rf $dbpath; cleanup' EXIT INT TERM QUIT --dbpath "$copydir"/var/lib/pacman \
dbpath=$(mktemp -d --tmpdir makechrootpkg-database.XXXXXXXXXX) -Sddp "${pkgnames[@]}")
mkdir -p "$dbpath"
pacman -Sy --dbpath "$dbpath" --logfile /dev/null
# query current package locations if ! wait $!; then
remotepkgs=($(pacman -Sddp --dbpath "$dbpath" --logfile /dev/null "${pkgnames[@]}"))
if (( $? )); then
warning "Skipped checkpkg due to missing repo packages" warning "Skipped checkpkg due to missing repo packages"
exit 0 exit 0
fi fi
# download package files if any non-local location exists # download package files if any non-local location exists
for remotepkg in "${remotepkgs[@]}"; do for remotepkg in "${remotepkgs[@]}"; do
[[ $remotepkg == file://* ]] && continue if [[ $remotepkg != file://* ]]; then
msg2 "Downloading current versions" msg2 "Downloading current versions"
pacman --noconfirm -Swdd --dbpath "$dbpath" --logfile /dev/null "${pkgnames[@]}" arch-nspawn "$copydir" pacman --noconfirm -Swdd "${pkgnames[@]}"
break mapfile -t remotepkgs < <(pacman --config "$copydir"/etc/pacman.conf \
--dbpath "$copydir"/var/lib/pacman \
-Sddp "${pkgnames[@]}")
break
fi
done done
msg2 "Checking packages" msg2 "Checking packages"
sudo -u "$makepkg_user" checkpkg --rmdir --warn sudo -u "$makepkg_user" checkpkg --rmdir --warn "${remotepkgs[@]/#file:\/\//}"
fi fi
true true
fi fi

View File

@@ -26,6 +26,7 @@ DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u'
#-- The package required by makepkg to download VCS sources #-- The package required by makepkg to download VCS sources
# Format: 'protocol::package' # Format: 'protocol::package'
VCSCLIENTS=('bzr::bzr' VCSCLIENTS=('bzr::bzr'
'fossil::fossil'
'git::git' 'git::git'
'hg::mercurial' 'hg::mercurial'
'svn::subversion') 'svn::subversion')
@@ -38,21 +39,25 @@ CARCH="x86_64"
CHOST="x86_64-pc-linux-gnu" CHOST="x86_64-pc-linux-gnu"
#-- Compiler and Linker Flags #-- Compiler and Linker Flags
CPPFLAGS="-D_FORTIFY_SOURCE=2" #CPPFLAGS=""
CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt" CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \
CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt" -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security \
-fstack-clash-protection -fcf-protection"
CXXFLAGS="$CFLAGS -Wp,-D_GLIBCXX_ASSERTIONS"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now"
#RUSTFLAGS="-C opt-level=2"
#-- Make Flags: change this for DistCC/SMP systems #-- Make Flags: change this for DistCC/SMP systems
#MAKEFLAGS="-j2" #MAKEFLAGS="-j2"
#-- Debugging flags #-- Debugging flags
DEBUG_CFLAGS="-g -fvar-tracking-assignments" DEBUG_CFLAGS="-g -fvar-tracking-assignments"
DEBUG_CXXFLAGS="-g -fvar-tracking-assignments" DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"
#DEBUG_RUSTFLAGS="-C debuginfo=2"
######################################################################### #########################################################################
# BUILD ENVIRONMENT # BUILD ENVIRONMENT
######################################################################### #########################################################################
# #
# Defaults: BUILDENV=(!distcc !color !ccache check !sign) # Makepkg defaults: BUILDENV=(!distcc !color !ccache check !sign)
# A negated environment option will do the opposite of the comments below. # A negated environment option will do the opposite of the comments below.
# #
#-- distcc: Use the Distributed C/C++/ObjC compiler #-- distcc: Use the Distributed C/C++/ObjC compiler
@@ -75,7 +80,7 @@ BUILDENV=(!distcc color !ccache check !sign)
# These are default values for the options=() settings # These are default values for the options=() settings
######################################################################### #########################################################################
# #
# Default: OPTIONS=(!strip docs libtool staticlibs emptydirs !zipman !purge !debug) # Makepkg defaults: OPTIONS=(!strip docs libtool staticlibs emptydirs !zipman !purge !debug !lto)
# A negated option will do the opposite of the comments below. # A negated option will do the opposite of the comments below.
# #
#-- strip: Strip symbols from binaries/libraries #-- strip: Strip symbols from binaries/libraries
@@ -86,11 +91,12 @@ BUILDENV=(!distcc color !ccache check !sign)
#-- zipman: Compress manual (man and info) pages in MAN_DIRS with gzip #-- zipman: Compress manual (man and info) pages in MAN_DIRS with gzip
#-- purge: Remove files specified by PURGE_TARGETS #-- purge: Remove files specified by PURGE_TARGETS
#-- debug: Add debugging flags as specified in DEBUG_* variables #-- debug: Add debugging flags as specified in DEBUG_* variables
#-- lto: Add compile flags for building with link time optimization
# #
OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !debug) OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !debug !lto)
#-- File integrity checks to use. Valid: md5, sha1, sha256, sha384, sha512 #-- File integrity checks to use. Valid: md5, sha1, sha224, sha256, sha384, sha512, b2
INTEGRITY_CHECK=(md5) INTEGRITY_CHECK=(sha256)
#-- Options to be used when stripping binaries. See `man strip' for details. #-- Options to be used when stripping binaries. See `man strip' for details.
STRIP_BINARIES="--strip-all" STRIP_BINARIES="--strip-all"
#-- Options to be used when stripping shared libraries. See `man strip' for details. #-- Options to be used when stripping shared libraries. See `man strip' for details.
@@ -146,4 +152,10 @@ COMPRESSLZ=(lzip -c -f)
PKGEXT='.pkg.tar.zst' PKGEXT='.pkg.tar.zst'
SRCEXT='.src.tar.gz' SRCEXT='.src.tar.gz'
#########################################################################
# OTHER
#########################################################################
#
#-- Command used to run pacman as root, instead of trying sudo and su
#PACMAN_AUTH=()
# vim: set ft=sh ts=2 sw=2 et: # vim: set ft=sh ts=2 sw=2 et:

View File

@@ -29,6 +29,7 @@ declare -a buildenv buildopts installed installpkgs
archiveurl='https://archive.archlinux.org/packages' archiveurl='https://archive.archlinux.org/packages'
buildroot=/var/lib/archbuild/reproducible buildroot=/var/lib/archbuild/reproducible
chroot=testenv chroot=testenv
diffoscope=0
parse_buildinfo() { parse_buildinfo() {
local line var val local line var val
@@ -56,10 +57,16 @@ parse_buildinfo() {
get_pkgfile() { get_pkgfile() {
local cdir=${cache_dirs[0]} local cdir=${cache_dirs[0]}
local pkgfilebase=${1} local pkgfilebase=${1}
local mode=${2}
local pkgname=${pkgfilebase%-*-*-*} local pkgname=${pkgfilebase%-*-*-*}
local pkgfile ext local pkgfile ext
for ext in .xz .zst ''; do # try without downloading
if [[ ${mode} != localonly ]] && get_pkgfile "${pkgfilebase}" localonly; then
return 0
fi
for ext in .zst .xz ''; do
pkgfile=${pkgfilebase}.pkg.tar${ext} pkgfile=${pkgfilebase}.pkg.tar${ext}
for c in "${cache_dirs[@]}"; do for c in "${cache_dirs[@]}"; do
@@ -71,6 +78,9 @@ get_pkgfile() {
for f in "${pkgfile}" "${pkgfile}.sig"; do for f in "${pkgfile}" "${pkgfile}.sig"; do
if [[ ! -f "${cdir}/${f}" ]]; then if [[ ! -f "${cdir}/${f}" ]]; then
if [[ ${mode} = localonly ]]; then
continue 2
fi
msg2 "retrieving '%s'..." "${f}" >&2 msg2 "retrieving '%s'..." "${f}" >&2
curl -Llf -# -o "${cdir}/${f}" "${archiveurl}/${pkgname:0:1}/${pkgname}/${f}" || continue 2 curl -Llf -# -o "${cdir}/${f}" "${archiveurl}/${pkgname:0:1}/${pkgname}/${f}" || continue 2
fi fi
@@ -94,34 +104,65 @@ package, including the .PKGINFO as well as the buildinfo.
For more details see https://reproducible-builds.org/ For more details see https://reproducible-builds.org/
OPTIONS OPTIONS
-d Run diffoscope if the package is unreproducible
-c <dir> Set pacman cache -c <dir> Set pacman cache
-M <file> Location of a makepkg config file -M <file> Location of a makepkg config file
-h Show this usage message -h Show this usage message
__EOF__ __EOF__
} }
while getopts 'M:c:h' arg; do while getopts 'dM:c:h' arg; do
case "$arg" in case "$arg" in
M) archroot_args+=(-M "$OPTARG") ;; d) diffoscope=1 ;;
c) cache_dirs+=("$OPTARG") ;; M) archroot_args+=(-M "$OPTARG") ;;
h) usage; exit 0 ;; c) cache_dirs+=("$OPTARG") ;;
*|?) usage; exit 1 ;; h) usage; exit 0 ;;
esac *|?) usage; exit 1 ;;
esac
done done
shift $((OPTIND - 1)) shift $((OPTIND - 1))
check_root check_root
if [[ -n $1 ]]; then [[ -f PKGBUILD ]] || { error "No PKGBUILD in current directory."; exit 1; }
pkgfile="$1"
if ! bsdtar -tqf "${pkgfile}" .BUILDINFO >/dev/null 2>&1; then # without arguments, get list of packages from PKGBUILD
error "file is not a valid pacman package: '%s'" "${pkgfile}" if [[ -z $1 ]]; then
mapfile -t pkgnames < <(source PKGBUILD; pacman -Sddp --print-format '%r/%n' "${pkgname[@]}")
wait $! || {
error "No package file specified and failed to retrieve package names from './PKGBUILD'."
plain "Try '${BASH_SOURCE[0]##*/} -h' for more information." >&2
exit 1
}
msg "Reproducing all pkgnames listed in ./PKGBUILD"
set -- "${pkgnames[@]}"
fi
# check each package to see if it's a file, and if not, try to download it
# using pacman -Sw, and get the filename from there
splitpkgs=()
for p in "$@"; do
if [[ -f ${p} ]]; then
splitpkgs+=("${p}")
else
pkgfile_remote=$(pacman -Sddp "${p}" 2>/dev/null) || { error "package name '%s' not in repos" "${p}"; exit 1; }
pkgfile=${pkgfile_remote#file://}
if [[ ! -f ${pkgfile} ]]; then
msg "Downloading package '%s' into pacman's cache" "${pkgfile}"
sudo pacman -Swdd --noconfirm --logfile /dev/null "${p}" || exit 1
pkgfile_remote=$(pacman -Sddp "${p}" 2>/dev/null)
pkgfile="${pkgfile_remote#file://}"
fi
splitpkgs+=("${pkgfile}")
fi
done
for f in "${splitpkgs[@]}"; do
if ! bsdtar -tqf "${f}" .BUILDINFO >/dev/null 2>&1; then
error "file is not a valid pacman package: '%s'" "${f}"
exit 1 exit 1
fi fi
else done
error "no package file specified. Try '${BASH_SOURCE[0]##*/} -h' for more information. "
exit 1
fi
if (( ${#cache_dirs[@]} == 0 )); then if (( ${#cache_dirs[@]} == 0 )); then
mapfile -t cache_dirs < <(pacman-conf CacheDir) mapfile -t cache_dirs < <(pacman-conf CacheDir)
@@ -133,10 +174,13 @@ load_makepkg_config
HOME=${ORIG_HOME} HOME=${ORIG_HOME}
[[ -d ${SRCDEST} ]] || SRCDEST=${PWD} [[ -d ${SRCDEST} ]] || SRCDEST=${PWD}
parse_buildinfo < <(bsdtar -xOqf "${pkgfile}" .BUILDINFO) parse_buildinfo < <(bsdtar -xOqf "${splitpkgs[0]}" .BUILDINFO)
export SOURCE_DATE_EPOCH="${buildinfo[builddate]}" export SOURCE_DATE_EPOCH="${buildinfo[builddate]}"
PACKAGER="${buildinfo[packager]}" PACKAGER="${buildinfo[packager]}"
BUILDDIR="${buildinfo[builddir]}" BUILDDIR="${buildinfo[builddir]}"
BUILDTOOL="${buildinfo[buildtool]}"
BUILDTOOLVER="${buildinfo[buildtoolver]}"
PKGEXT=${splitpkgs[0]#${splitpkgs[0]%.pkg.tar*}}
# nuke and restore reproducible testenv # nuke and restore reproducible testenv
for copy in "${buildroot}"/*/; do for copy in "${buildroot}"/*/; do
@@ -152,20 +196,19 @@ for fname in "${installed[@]}"; do
exit 1 exit 1
fi fi
done done
printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1 printf '%s\n' "${allpkgfiles[@]}" | mkarchroot -M @pkgdatadir@/makepkg-x86_64.conf -U "${archroot_args[@]}" "${buildroot}"/root - || exit 1
# use makechrootpkg to prep the build directory # use makechrootpkg to prep the build directory
makechrootpkg -r "${buildroot}" -l "${chroot}" -- --packagelist || exit 1 makechrootpkg -r "${buildroot}" -l "${chroot}" -- --packagelist || exit 1
# set detected makepkg.conf options # set detected makepkg.conf options
{ {
for var in PACKAGER BUILDDIR; do for var in PACKAGER BUILDDIR BUILDTOOL BUILDTOOLVER PKGEXT; do
printf '%s=%s\n' "${var}" "${!var@Q}" printf '%s=%s\n' "${var}" "${!var@Q}"
done done
printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}" printf 'OPTIONS=(%s)\n' "${buildopts[*]@Q}"
printf 'BUILDENV=(%s)\n' "${buildenv[*]@Q}" printf 'BUILDENV=(%s)\n' "${buildenv[*]@Q}"
} >> "${buildroot}/${chroot}"/etc/makepkg.conf >> "${buildroot}/${chroot}"/etc/makepkg.conf } >> "${buildroot}/${chroot}"/etc/makepkg.conf
install -d -o "${SUDO_UID:-$UID}" -g "$(id -g "${SUDO_UID:-$UID}")" "${buildroot}/${chroot}/${BUILDDIR}" install -d -o "${SUDO_UID:-$UID}" -g "$(id -g "${SUDO_UID:-$UID}")" "${buildroot}/${chroot}/${BUILDDIR}"
# kick off the build # kick off the build
@@ -173,18 +216,26 @@ arch-nspawn "${buildroot}/${chroot}" \
--bind="${PWD}:/startdir" \ --bind="${PWD}:/startdir" \
--bind="${SRCDEST}:/srcdest" \ --bind="${SRCDEST}:/srcdest" \
/chrootbuild -C --noconfirm --log --holdver --skipinteg /chrootbuild -C --noconfirm --log --holdver --skipinteg
ret=$?
if (( $? == 0 )); then if (( ${ret} == 0 )); then
msg2 "built succeeded! built packages can be found in ${buildroot}/${chroot}/pkgdest" msg2 "built succeeded! built packages can be found in ${buildroot}/${chroot}/pkgdest"
msg "comparing artifacts..." msg "comparing artifacts..."
if cmp -s "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}"; then
msg2 "Package successfully reproduced!" for pkgfile in "${splitpkgs[@]}"; do
exit 0 comparefiles=("${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}")
else if cmp -s "${comparefiles[@]}"; then
warning "Package is not reproducible. :(" msg2 "Package '%s' successfully reproduced!" "${pkgfile}"
sha256sum "${pkgfile}" "${buildroot}/${chroot}/pkgdest/${pkgfile##*/}" else
fi ret=1
warning "Package '%s' is not reproducible. :(" "${pkgfile}"
sha256sum "${comparefiles[@]}"
if (( diffoscope )); then
diffoscope "${comparefiles[@]}"
fi
fi
done
fi fi
# the package either failed to build, or was unreproducible # return failure from chrootbuild, or the reproducibility status
exit 1 exit ${ret}

View File

@@ -63,10 +63,9 @@ shift 1
[[ -z $working_dir ]] && die 'Please specify a working directory.' [[ -z $working_dir ]] && die 'Please specify a working directory.'
pacconf_cmd=$(command -v pacman-conf || command -v pacconf)
if (( ${#cache_dirs[@]} == 0 )); then if (( ${#cache_dirs[@]} == 0 )); then
mapfile -t cache_dirs < <($pacconf_cmd CacheDir) mapfile -t cache_dirs < <(pacman-conf CacheDir)
fi fi
umask 0022 umask 0022

View File

@@ -24,7 +24,7 @@ source /usr/share/makepkg/util/config.sh
# global defaults suitable for use by Arch staff # global defaults suitable for use by Arch staff
repo=extra repo=extra
arch=x86_64 arch=x86_64
server=dragon.archlinux.org server=build.archlinux.org
die() { printf "error: $1\n" "${@:2}"; exit 1; } die() { printf "error: $1\n" "${@:2}"; exit 1; }
@@ -74,18 +74,23 @@ while (( $# )); do
done done
# multilib must be handled specially # multilib must be handled specially
archbuild_arch="${arch}"
if [[ $repo = multilib* ]]; then if [[ $repo = multilib* ]]; then
arch= archbuild_arch=
fi fi
archbuild_cmd=("${repo}${arch:+-$arch}-build" "$@") archbuild_cmd=("${repo}${archbuild_arch:+-$archbuild_arch}-build" "$@")
trap 'rm -rf $SRCPKGDEST' EXIT INT TERM QUIT trap 'rm -rf $TEMPDIR' EXIT INT TERM QUIT
# Load makepkg.conf variables to be available
load_makepkg_config
# Use a source-only tarball as an intermediate to transfer files. This # Use a source-only tarball as an intermediate to transfer files. This
# guarantees the checksums are okay, and guarantees that all needed files are # guarantees the checksums are okay, and guarantees that all needed files are
# transferred, including local sources, install scripts, and changelogs. # transferred, including local sources, install scripts, and changelogs.
export SRCPKGDEST=$(mktemp -d) export TEMPDIR=$(mktemp -d --tmpdir offload-build.XXXXXXXXXX)
export SRCPKGDEST=${TEMPDIR}
makepkg --source || die "unable to make source package" makepkg --source || die "unable to make source package"
# Temporary cosmetic workaround makepkg if SRCDEST is set somewhere else # Temporary cosmetic workaround makepkg if SRCDEST is set somewhere else
@@ -96,7 +101,7 @@ mapfile -t files < <(
# This is sort of bash golfing but it allows running a mildly complex # This is sort of bash golfing but it allows running a mildly complex
# command over ssh with a single connection. # command over ssh with a single connection.
# shellcheck disable=SC2145 # shellcheck disable=SC2145
cat "$SRCPKGDEST"/*.src.tar.gz | cat "$SRCPKGDEST"/*"$SRCEXT" |
ssh $server ' ssh $server '
temp="${XDG_CACHE_HOME:-$HOME/.cache}/offload-build" && temp="${XDG_CACHE_HOME:-$HOME/.cache}/offload-build" &&
mkdir -p "$temp" && mkdir -p "$temp" &&
@@ -108,14 +113,21 @@ mapfile -t files < <(
printf "%s\n" "" "-> build complete" && printf "%s\n" "" "-> build complete" &&
printf "\t%s\n" "$temp"/* printf "\t%s\n" "$temp"/*
} >&2 && } >&2 &&
makepkg --packagelist makepkg_user_config="${XDG_CONFIG_HOME:-$HOME/.config}/pacman/makepkg.conf" &&
makepkg_config="/usr/share/devtools/makepkg-'"${arch}"'.conf" &&
if [[ -f /usr/share/devtools/makepkg-'"${repo}"'-'"${arch}"'.conf ]]; then
makepkg_config="/usr/share/devtools/makepkg-'"${repo}"'-'"${arch}"'.conf"
fi &&
makepkg --config <(cat "${makepkg_user_config}" "${makepkg_config}" 2>/dev/null) --packagelist &&
printf "%s\n" "${temp}/PKGBUILD"
') ')
if (( ${#files[@]} )); then if (( ${#files[@]} )); then
printf '%s\n' '' '-> copying files...' printf '%s\n' '' '-> copying files...'
load_makepkg_config scp "${files[@]/#/$server:}" "${TEMPDIR}/"
scp "${files[@]/#/$server:}" "${PKGDEST:-${PWD}}/" mv "${TEMPDIR}"/*.pkg.tar* "${PKGDEST:-${PWD}}/"
mv "${TEMPDIR}/PKGBUILD" "${PWD}/"
else else
exit 1 exit 1
fi fi

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -31,10 +31,11 @@ Architecture = auto
# Misc options # Misc options
#UseSyslog #UseSyslog
#Color #Color
#TotalDownload NoProgressBar
# We cannot check disk space from within a chroot environment # We cannot check disk space from within a chroot environment
#CheckSpace #CheckSpace
#VerbosePkgLists VerbosePkgLists
ParallelDownloads = 5
# By default, pacman accepts packages signed by keys that its local keyring # By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages. # trusts (see pacman-key and its man page), as well as unsigned packages.

View File

@@ -18,6 +18,8 @@
# along with this program. If not, see <https://www.gnu.org/licenses/>. # along with this program. If not, see <https://www.gnu.org/licenses/>.
# #
m4_include(lib/common.sh)
# globals # globals
: ${SOLINKS_MIRROR:="https://mirror.pkgbuild.com"} : ${SOLINKS_MIRROR:="https://mirror.pkgbuild.com"}
: ${SOCACHE_DIR:="${XDG_CACHE_HOME:-${HOME}/.cache}/sogrep"} : ${SOCACHE_DIR:="${XDG_CACHE_HOME:-${HOME}/.cache}/sogrep"}
@@ -39,13 +41,37 @@ recache() {
for repo in "${_repos[@]}"; do for repo in "${_repos[@]}"; do
for arch in "${arches[@]}"; do for arch in "${arches[@]}"; do
# delete extracted tarballs from previous sogrep versions
rm -rf "${SOCACHE_DIR}/${arch}/${repo}" rm -rf "${SOCACHE_DIR}/${arch}/${repo}"
mkdir -p "${SOCACHE_DIR}/${arch}/${repo}"
curl -L "$verbosity" "${SOLINKS_MIRROR}/${repo}/os/${arch}/${repo}.links.tar.gz" | bsdtar -xf - -C "${SOCACHE_DIR}/${arch}/${repo}" # fetch repo links database if newer than our cached copy
local dbpath=${SOCACHE_DIR}/${arch}/${repo}.links.tar.gz
mkdir -p "${dbpath%/*}"
(( VERBOSE )) && echo "Fetching ${repo}.links.tar.gz..."
curl -LR "${verbosity}" -o "${dbpath}" -z "${dbpath}" \
"${SOLINKS_MIRROR}/${repo}/os/${arch}/${repo}.links.tar.gz"
done done
done done
} }
is_outdated_cache() {
local repo arch
# links databases are generated at about the same time every day; we should
# attempt to check for new database files if any of them are over a day old
for repo in "${_repos[@]}"; do
for arch in "${arches[@]}"; do
local dbpath=${SOCACHE_DIR}/${arch}/${repo}.links.tar.gz
if [[ ! -f ${dbpath} ]] || [[ $(find "${dbpath}" -mtime +0) ]]; then
return 0
fi
done
done
return 1
}
search() { search() {
local repo=$1 arch lib=$2 srepos=("${_repos[@]}") local repo=$1 arch lib=$2 srepos=("${_repos[@]}")
@@ -58,15 +84,20 @@ search() {
srepos=("${repo}") srepos=("${repo}")
fi fi
setup_workdir
for arch in "${arches[@]}"; do for arch in "${arches[@]}"; do
for repo in "${srepos[@]}"; do for repo in "${srepos[@]}"; do
local prefix= local prefix=
(( VERBOSE && ${#srepos[@]} > 1 )) && prefix=${repo}/ (( VERBOSE && ${#srepos[@]} > 1 )) && prefix=${repo}/
db=${SOCACHE_DIR}/${arch}/${repo}/ local db=${SOCACHE_DIR}/${arch}/${repo}.links.tar.gz
if [[ -d ${db} ]]; then if [[ -f ${db} ]]; then
local extracted=${WORKDIR}/${arch}/${repo}
mkdir -p "${extracted}"
bsdtar -C "${extracted}" -xf "${db}"
while read -rd '' pkg; do while read -rd '' pkg; do
read -r match read -r match
pkg=${pkg#${db}} pkg=${pkg#${extracted}/}
pkg="${prefix}${pkg%-*-*/links}" pkg="${prefix}${pkg%-*-*/links}"
if (( VERBOSE )); then if (( VERBOSE )); then
@@ -74,7 +105,7 @@ search() {
else else
printf '%s\n' "${pkg}" printf '%s\n' "${pkg}"
fi fi
done < <(grep -rZ "${lib}" "${db}") | sort -u done < <(grep -rZ "${lib}" "${extracted}") | sort -u
fi fi
done done
done | resort done | resort
@@ -139,7 +170,8 @@ if ! (( ( REFRESH && $# == 0 ) || $# == 2 )); then
exit 1 exit 1
fi fi
if (( REFRESH )) || [[ ! -d ${SOCACHE_DIR} ]]; then # trigger a refresh if requested explicitly or the cached dbs might be outdated
if (( REFRESH )) || [[ ! -d ${SOCACHE_DIR} ]] || is_outdated_cache; then
recache recache
(( $# == 2 )) || exit 0 (( $# == 2 )) || exit 0
fi fi

View File

@@ -10,6 +10,7 @@ _archbuild_args=(
'-c[Recreate the chroot before building]' '-c[Recreate the chroot before building]'
'-r[Create chroots in this directory]:base_dir:_files -/' '-r[Create chroots in this directory]:base_dir:_files -/'
'-h[Display usage]' '-h[Display usage]'
'--[Introduce makechrootpkg options]:*::makechrootpkg options:= _dispatch makechrootpkg makechrootpkg'
) )
_archco_args=( _archco_args=(
@@ -94,10 +95,11 @@ _offload_build_args=(
) )
_makerepropkg_args=( _makerepropkg_args=(
'-d[Run diffoscope if the package is unreproducible]'
'-c[Set pacman cache]:pacman_cache:_files -/' '-c[Set pacman cache]:pacman_cache:_files -/'
'-M[Location of a makepkg config file]:makepkg_config:_files -g "*.conf(.)"' '-M[Location of a makepkg config file]:makepkg_config:_files -g "*.conf(.)"'
'-h[Display usage]' '-h[Display usage]'
'1:working_dir:_files -g "*.pkg.tar.*(.)"' '*:working_dir:_files -g "*.pkg.tar.*(.)"'
) )
_devtools_completions_all_packages() { _devtools_completions_all_packages() {