OILS / regtest / aports.md View on Github | oils.pub

281 lines, 186 significant
1regtest/aports
2==============
3
4`regtest/aports` tests OSH with Alpine Linux packages, in `aports/main`. It's
5organized as a series of "task files", which you can see with:
6
7 $ regtest/aports-test.sh count-lines # ~2000 lines including debug code
8
9As usual, there are notes at the top of each task file. Here is an overview.
10
11## Set Up Machine to avoid `sudo` prompts
12
13Add a file to `/etc/sudoers.d` so that `sudo` doesn't ask for a password after
14a given time period. Otherwise building the second shard may get "stuck".
15
16This is how I did it manually:
17
18 $ sudo visudo -f /etc/sudoers.d/no-timeout
19
20The result should be:
21
22 $ sudo cat /etc/sudoers.d/no-timeout
23 Defaults:andy timestamp_timeout=-1
24
25The -1 value means it's cached forever.
26
27## Note on Directory Structure
28
29The following scripts assume a directory structure like this:
30
31```
32*this can be any path*/
33 oils-for-unix/
34 oils/ # dir where this git repo is cloned
35 alpinelinux/ # created by scripts below
36```
37
38This layout mimics the GitHub URL namespace. If the layout on your machine differs, you might run into problems.
39
40## Set up Alpine chroot - `he.oils.pub`
41
42The first step is in `regtest/aports-setup.sh`:
43
44 $ regtest/aports-setup.sh remove-chroot # optional; for a CLEAN build
45
46 $ regtest/aports-setup.sh fetch-all # git clone, download Oils from CI
47 $ regtest/aports-setup.sh prepare-all # make a chroot
48
49This usually happens on the **build server**, e.g. `he.oils.pub`.
50
51 $ regtest/aports-setup.sh unpack-distfiles # optional: requires _chroot/distfiles.tar
52
53The file `_chroot/distfiles.tar` may contain ~6 GB of
54`$CHROOT_DIR/var/cache/distfiles`. Keeping the tarball saves the ~30 minutes
55it takes to download all the `.tar.gz` source tarballs.
56
57## Build aports Packages - `he.oils.pub`
58
59If you didn't unpack `_chroot/distfiles.tar`, run:
60
61 $ regtest/aports-run.sh fetch-packages '.*'
62
63I do this separately, because contacting hundreds of servers is inherently reliable.
64
65The results are not that consistent, so we divide the ~1640 `APKBUILD` files
66into 17 *shards*. You can run two shards like this:
67
68 $ export APORTS_EPOCH=2025-08-07-fix # directory name, and .wwz name
69
70 $ regtest/aports-run.sh build-many-shards-overlayfs shard5 shard6
71
72This is the normal way to run all 17 shards (using bash brace expansion):
73
74 $ regtest/aports-run.sh build-many-shards-overlayfs shard{0..16}
75
76But this is how I run it right now, due to flakiness:
77
78 # weird order!
79 $ regtest/aports-run.sh build-many-shards-overlayfs shard{10..16} shard{0..5}
80
81 # Now BLOW AWAY CHROOT, to work around errors
82 $ regtest/aports-setup.sh remove-chroot
83 $ regtest/aports-setup.sh prepare-chroot
84 $ regtest/aports-setup.sh unpack-distfiles
85
86 # Run remaining shards
87 $ regtest/aports-run.sh build-many-shards-overlayfs shard{6..9}
88
89(This was discovered empirically; we should remove this workaround eventually.)
90
91
92## Make Reports with Tables
93
94### Credentials
95
96You will need these credentials:
97
98- to rsync from the `he.oils.pub` server
99 - I think each person should have their own account
100- to `scp` to `.wwz` and `.html` to the `op.oils.pub` server
101 - ask for SSH key; give user name
102
103### Sync and Preview - local machine
104
105You can sync results while the build is running:
106
107 $ regtest/aports-html.sh sync-results # rsync from he.oils.pub to _tmp/aports-report
108
109This creates a structure like:
110
111 _tmp/aports-report/
112 2025-08-07-fix/
113 shard10/
114 shard11/
115
116And then make a partial report:
117
118 $ regtest/aports-html.sh write-all-reports _tmp/aports-report/2025-08-07-fix
119
120Now look at this file in your browser:
121
122 _tmp/aports-report/2025-08-07-fix/diff_merged.html # 17 shards merged
123
124
125### Checking for Flakiness
126
127The `aports` build can be flaky for a couple reasons, which are currently
128unexplained:
129
1301. The `abuild builddeps` step fails
1311. "cannot create executable" or "cannot compile programs" errors.
132 - Associated with "PHDR segment not covered".
133
134Both of these errors happen with the baseline build, not only with OSH.
135
136---
137
138So right now, I periodically sync the results to my local machine, and check
139the results with:
140
141 $ regtest/aports-debug.sh grep-c-bug-2
142 $ regtest/aports-debug.sh grep-phdr-bug-2
143 $ regtest/aports-debug.sh grep-b-bug-2
144
145If there are too many results, the chroot may have "crapped out".
146
147TODO: we can fix this by:
148
149- running under `podman` (`aports-container.sh`)
150- running under a VM
151
152### Publish Reports - `op.oils.pub`
153
154After verifying the output of `write-all-reports`, add a line to the markdown
155in `aports-html.sh published-html`.
156
157Then share the results:
158
159 $ regtest/aports-html.sh make-wwz _tmp/aports-report/2025-08-07-fix
160
161 $ regtest/aports-html.sh deploy-wwz-op _tmp/aports-report/2025-08-07-fix.wwz
162
163Now visit
164
165- <https://op.oils.pub/aports-build/published.html>
166
167which will link to:
168
169- <https://op.oils.pub/aports-build/2025-08-07-fix.wwz/_tmp/aports-report/2025-08-07-fix/diff-merged.html>
170
171### Add reports to `published.html`
172
173If the results look good, add a line to the markdown in `regtest/aports-html.sh
174published-html`, and then run:
175
176 $ regtest/aports-html.sh deploy-published
177
178And then visit:
179
180- <https://op.oils.pub/aports-build/published.html>
181
182## Other Instructions
183
184### Reproducing a single package build failure
185
186You can reproduce build failures on your own machine. Do the same steps you did on `he.oils.pub`:
187
188 $ regtest/aports-setup.sh fetch-all
189 $ regtest/aports-setup.sh prepare-all
190
191 $ regtest/aports-setup.sh unpack-distfiles # optional
192
193And then ONE of these commands:
194
195 $ regtest/aports-run.sh set-baseline # normal Alpine config
196 $ regtest/aports-run.sh set-osh-as-sh # replace /bin/sh with OSH
197
198And then
199
200 # 7zip is the PKG_FILTER
201 # $config is either 'baseline' or 'osh-as-sh'
202 $ regtest/aports-run.sh build-packages '7zip' $config
203
204Then look at the logs in
205
206 _chroot/aports-build/
207 /home/udu/oils/
208 _tmp/aports-guest/baseline/
209 7zip.log.txt
210
211### Get a Shell in the Chroot
212
213 $ regtest/aports-run.sh enter-rootfs-user # as unprivileged 'udu' user
214
215 $ regtest/aports-run.sh enter-rootfs # as root user
216
217## TODO
218
219- Running under podman could be more reliable
220 - `regtest/aports-container.sh` shows that podman is able to run `abuild
221 rootbld` -> `bwrap`, if it's passed the `--privileged` flag
222
223## Related Links
224
225- [Updating-Causes-of-Aports-Failures](https://github.com/oils-for-unix/oils/wiki/Updating-Causes-of-Aports-Failures) (wiki)
226
227## Appendix: Dir Structure
228
229```
230he.oils.pub/
231 ~/git/oils-for-unix/oils/
232 _chroot/
233 aports-build/
234 enter-chroot # you can run this, passing -u udu
235 home/udu/ # user name is 'udu'
236 oils/ # a subset of the Oils repo
237 build/
238 py.sh
239 _tmp/
240 aports-guest/
241 baseline/
242 7zip.log.txt
243 7zip.task.tsv
244 osh-as-sh/
245 osh-as-bash/
246 osh-as-sh.overlay/
247 layer/
248 merged/
249 enter-chroot # you can run this, passing -u udu
250 work/
251 _tmp/aports-build/
252 2025-08-07-fix/ # $APORTS_EPOCH
253 shard0/
254 baseline/
255 tasks.tsv # concatenated .task.tsv
256 log/
257 7zip.log.txt
258 abridged-log/ # tail -n 1000 ont he log
259 gcc.log.txt
260 shard1/
261 ...
262
263localhost/
264 ~/git/oils-for-unix/oils/
265 _tmp/aports-report/ # destination for sync-results
266 2025-08-07-fix/ # $APORTS_EPOCH
267 diff-merged.html
268 shard0/
269 baseline/
270 index.html # from tasks.tsv
271 tasks.tsv
272 log/
273 7zip.log.txt
274 abridged-log/ # tail -n 1000 ont he log
275 gcc.log.txt
276 osh-as-sh/ # from tasks.tsv
277 tasks.tsv
278 log/
279 abridged-log/
280 index.html
281```