| 1 | regtest/aports
|
| 2 | ==============
|
| 3 |
|
| 4 | `regtest/aports` tests OSH with Alpine Linux packages, in `aports/main`. It's
|
| 5 | organized as a series of "task files", which you can see with:
|
| 6 |
|
| 7 | $ regtest/aports-test.sh count-lines # ~2000 lines including debug code
|
| 8 |
|
| 9 | As usual, there are notes at the top of each task file. Here is an overview.
|
| 10 |
|
| 11 | ## Set Up Machine to avoid `sudo` prompts
|
| 12 |
|
| 13 | Add a file to `/etc/sudoers.d` so that `sudo` doesn't ask for a password after
|
| 14 | a given time period. Otherwise building the second shard may get "stuck".
|
| 15 |
|
| 16 | This is how I did it manually:
|
| 17 |
|
| 18 | $ sudo visudo -f /etc/sudoers.d/no-timeout
|
| 19 |
|
| 20 | The result should be:
|
| 21 |
|
| 22 | $ sudo cat /etc/sudoers.d/no-timeout
|
| 23 | Defaults:andy timestamp_timeout=-1
|
| 24 |
|
| 25 | The -1 value means it's cached forever.
|
| 26 |
|
| 27 | ## Note on Directory Structure
|
| 28 |
|
| 29 | The following scripts assume a directory structure like this:
|
| 30 |
|
| 31 | ```
|
| 32 | *this can be any path*/
|
| 33 | oils-for-unix/
|
| 34 | oils/ # dir where this git repo is cloned
|
| 35 | alpinelinux/ # created by scripts below
|
| 36 | ```
|
| 37 |
|
| 38 | This layout mimics the GitHub URL namespace. If the layout on your machine differs, you might run into problems.
|
| 39 |
|
| 40 | ## Set up Alpine chroot - `he.oils.pub`
|
| 41 |
|
| 42 | The first step is in `regtest/aports-setup.sh`:
|
| 43 |
|
| 44 | $ regtest/aports-setup.sh remove-chroot # optional; for a CLEAN build
|
| 45 |
|
| 46 | $ regtest/aports-setup.sh fetch-all # git clone, download Oils from CI
|
| 47 | $ regtest/aports-setup.sh prepare-all # make a chroot
|
| 48 |
|
| 49 | This usually happens on the **build server**, e.g. `he.oils.pub`.
|
| 50 |
|
| 51 | # optional: requires _chroot/distfiles-{main,community}.tar
|
| 52 | $ regtest/aports-setup.sh unpack-distfiles main
|
| 53 |
|
| 54 | The file `_chroot/distfiles.tar` may contain ~6 GB of
|
| 55 | `$CHROOT_DIR/var/cache/distfiles`. Keeping the tarball saves the ~30 minutes
|
| 56 | it takes to download all the `.tar.gz` source tarballs.
|
| 57 |
|
| 58 | ## Build aports Packages - `he.oils.pub`
|
| 59 |
|
| 60 | If you didn't unpack `_chroot/distfiles.tar`, run:
|
| 61 |
|
| 62 | $ regtest/aports-run.sh fetch-packages '.*'
|
| 63 |
|
| 64 | This fetches package source tarballs from hundreds of servers, which is
|
| 65 | inherently **unreliable**. That's why it's a separate step.
|
| 66 |
|
| 67 | To avoid this step next time, do
|
| 68 |
|
| 69 | $ regtest/aports-setup.sh make-distfiles-tar main
|
| 70 |
|
| 71 | Currently we divide the ~1600 `APKBUILD` files into 16 *shards*. You can run
|
| 72 | two shards like this:
|
| 73 |
|
| 74 | $ export APORTS_EPOCH=2025-08-07-fix # directory name, and .wwz name
|
| 75 |
|
| 76 | $ regtest/aports-run.sh build-many-shards-overlayfs shard5 shard6
|
| 77 |
|
| 78 | This is the normal way to run all 16 shards (using bash brace expansion):
|
| 79 |
|
| 80 | $ regtest/aports-run.sh build-many-shards-overlayfs shard{0..15}
|
| 81 |
|
| 82 | ## Make Reports with Tables
|
| 83 |
|
| 84 | ### Credentials
|
| 85 |
|
| 86 | You will need these credentials:
|
| 87 |
|
| 88 | - to rsync from the `he.oils.pub` server
|
| 89 | - I think each person should have their own account
|
| 90 | - to `scp` to `.wwz` and `.html` to the `op.oils.pub` server
|
| 91 | - ask for SSH key; give user name
|
| 92 |
|
| 93 | ### Sync and Preview - local machine
|
| 94 |
|
| 95 | You can sync results while the build is running:
|
| 96 |
|
| 97 | $ regtest/aports-html.sh sync-results # rsync from he.oils.pub to _tmp/aports-report
|
| 98 |
|
| 99 | This creates a structure like:
|
| 100 |
|
| 101 | _tmp/aports-report/
|
| 102 | 2025-08-07-fix/
|
| 103 | shard10/
|
| 104 | shard11/
|
| 105 |
|
| 106 | And then make a partial report:
|
| 107 |
|
| 108 | $ regtest/aports-html.sh write-all-reports _tmp/aports-report/2025-08-07-fix
|
| 109 |
|
| 110 | Now look at this file in your browser:
|
| 111 |
|
| 112 | _tmp/aports-report/2025-08-07-fix/diff_merged.html # 17 shards merged
|
| 113 |
|
| 114 |
|
| 115 | ### Checking for Flakiness
|
| 116 |
|
| 117 | The `aports` build can be flaky for a couple reasons, which are currently
|
| 118 | unexplained:
|
| 119 |
|
| 120 | 1. The `abuild builddeps` step fails
|
| 121 | 1. "cannot create executable" or "cannot compile programs" errors.
|
| 122 | - Associated with "PHDR segment not covered".
|
| 123 |
|
| 124 | Both of these errors happen with the baseline build, not only with OSH.
|
| 125 |
|
| 126 | ---
|
| 127 |
|
| 128 | So right now, I periodically sync the results to my local machine, and check
|
| 129 | the results with:
|
| 130 |
|
| 131 | $ regtest/aports-debug.sh grep-c-bug-2
|
| 132 | $ regtest/aports-debug.sh grep-phdr-bug-2
|
| 133 | $ regtest/aports-debug.sh grep-b-bug-2
|
| 134 |
|
| 135 | If there are too many results, the chroot may have "crapped out".
|
| 136 |
|
| 137 | TODO: we can fix this by:
|
| 138 |
|
| 139 | - running under `podman` (`aports-container.sh`)
|
| 140 | - running under a VM
|
| 141 |
|
| 142 | ### Publish Reports - `op.oils.pub`
|
| 143 |
|
| 144 | After verifying the output of `write-all-reports`, add a line to the markdown
|
| 145 | in `aports-html.sh published-html`.
|
| 146 |
|
| 147 | Then share the results:
|
| 148 |
|
| 149 | $ regtest/aports-html.sh make-wwz _tmp/aports-report/2025-08-07-fix
|
| 150 |
|
| 151 | $ regtest/aports-html.sh deploy-wwz-op _tmp/aports-report/2025-08-07-fix.wwz
|
| 152 |
|
| 153 | Now visit
|
| 154 |
|
| 155 | - <https://op.oils.pub/aports-build/published.html>
|
| 156 |
|
| 157 | which will link to:
|
| 158 |
|
| 159 | - <https://op.oils.pub/aports-build/2025-08-07-fix.wwz/_tmp/aports-report/2025-08-07-fix/diff-merged.html>
|
| 160 |
|
| 161 | ### Add reports to `published.html`
|
| 162 |
|
| 163 | If the results look good, add a line to the markdown in `regtest/aports-html.sh
|
| 164 | published-html`, and then run:
|
| 165 |
|
| 166 | $ regtest/aports-html.sh deploy-published
|
| 167 |
|
| 168 | And then visit:
|
| 169 |
|
| 170 | - <https://op.oils.pub/aports-build/published.html>
|
| 171 |
|
| 172 | ## Other Instructions
|
| 173 |
|
| 174 | ### Reproducing a single package build failure
|
| 175 |
|
| 176 | You can reproduce build failures on your own machine. Do the same steps you did on `he.oils.pub`:
|
| 177 |
|
| 178 | $ regtest/aports-setup.sh fetch-all
|
| 179 | $ regtest/aports-setup.sh prepare-all
|
| 180 |
|
| 181 | $ regtest/aports-setup.sh unpack-distfiles # optional
|
| 182 |
|
| 183 | And then ONE of these commands:
|
| 184 |
|
| 185 | $ regtest/aports-run.sh set-baseline # normal Alpine config
|
| 186 | $ regtest/aports-run.sh set-osh-as-sh # replace /bin/sh with OSH
|
| 187 |
|
| 188 | And then
|
| 189 |
|
| 190 | # 7zip is the PKG_FILTER
|
| 191 | # $config is either 'baseline' or 'osh-as-sh'
|
| 192 | $ regtest/aports-run.sh build-packages '7zip' $config
|
| 193 |
|
| 194 | Then look at the logs in
|
| 195 |
|
| 196 | _chroot/aports-build/
|
| 197 | /home/udu/oils/
|
| 198 | _tmp/aports-guest/baseline/
|
| 199 | 7zip.log.txt
|
| 200 |
|
| 201 | ### Get a Shell in the Chroot
|
| 202 |
|
| 203 | $ regtest/aports-run.sh enter-rootfs-user # as unprivileged 'udu' user
|
| 204 |
|
| 205 | $ regtest/aports-run.sh enter-rootfs # as root user
|
| 206 |
|
| 207 | ## TODO
|
| 208 |
|
| 209 | - Running under podman could be more reliable
|
| 210 | - `regtest/aports-container.sh` shows that podman is able to run `abuild
|
| 211 | rootbld` -> `bwrap`, if it's passed the `--privileged` flag
|
| 212 |
|
| 213 | ## Related Links
|
| 214 |
|
| 215 | - [Updating-Causes-of-Aports-Failures](https://github.com/oils-for-unix/oils/wiki/Updating-Causes-of-Aports-Failures) (wiki)
|
| 216 |
|
| 217 | ## Appendix: Dir Structure
|
| 218 |
|
| 219 | ```
|
| 220 | he.oils.pub/
|
| 221 | ~/git/oils-for-unix/oils/
|
| 222 | _chroot/
|
| 223 | aports-build/
|
| 224 | enter-chroot # you can run this, passing -u udu
|
| 225 | home/udu/ # user name is 'udu'
|
| 226 | oils/ # a subset of the Oils repo
|
| 227 | build/
|
| 228 | py.sh
|
| 229 | _tmp/
|
| 230 | aports-guest/
|
| 231 | baseline/
|
| 232 | 7zip.log.txt
|
| 233 | 7zip.task.tsv
|
| 234 | osh-as-sh/
|
| 235 | osh-as-bash/
|
| 236 | osh-as-sh.overlay/
|
| 237 | layer/
|
| 238 | merged/
|
| 239 | enter-chroot # you can run this, passing -u udu
|
| 240 | work/
|
| 241 | _tmp/aports-build/
|
| 242 | 2025-08-07-fix/ # $APORTS_EPOCH
|
| 243 | shard0/
|
| 244 | baseline/
|
| 245 | tasks.tsv # concatenated .task.tsv
|
| 246 | log/
|
| 247 | 7zip.log.txt
|
| 248 | abridged-log/ # tail -n 1000 ont he log
|
| 249 | gcc.log.txt
|
| 250 | shard1/
|
| 251 | ...
|
| 252 |
|
| 253 | localhost/
|
| 254 | ~/git/oils-for-unix/oils/
|
| 255 | _tmp/aports-report/ # destination for sync-results
|
| 256 | 2025-08-07-fix/ # $APORTS_EPOCH
|
| 257 | diff-merged.html
|
| 258 | shard0/
|
| 259 | baseline/
|
| 260 | index.html # from tasks.tsv
|
| 261 | tasks.tsv
|
| 262 | log/
|
| 263 | 7zip.log.txt
|
| 264 | abridged-log/ # tail -n 1000 ont he log
|
| 265 | gcc.log.txt
|
| 266 | osh-as-sh/ # from tasks.tsv
|
| 267 | tasks.tsv
|
| 268 | log/
|
| 269 | abridged-log/
|
| 270 | index.html
|
| 271 | ```
|