Dev mode — no Cloudflare Access header present. Authenticated as dev@local.
EPL
dev@local

Lab 02 · Day 1, Session 2

ImageBuilder Firmware

Duration: 75 minutes

Not started
Hardware profile:
IP: 192.168.8.1 Password: goodlife
Environment:

Lab 02 — ImageBuilder Firmware

Duration: 75 minutes

You met the Mango on stock GL.iNet firmware in Lab 01. This lab replaces that firmware with a custom build: a sysupgrade .bin produced from OpenWrt’s ImageBuilder and pinned to 23.05.3 / ramips / mt76x8.

The engineering exercise is package selection under a hard 16 MB NOR flash ceiling. Tailscale, cloudflared, and python3 do not fit in NOR alongside a kernel and bootloader. You confirm that by reading the package list, build the image, and verify the resulting .bin against the size constraint. Lab 03 will move those heavy packages onto a USB ExtRoot.

Reproducibility. The build sets SOURCE_DATE_EPOCH=1714694400 to strip squashfs timestamps. Two students building from the same commit on the same day produce identical SHA256 sums. “Is your firmware identical to mine?” becomes a one-line check.

Loading component-tree diagram…

Where each step runs

This lab runs entirely from the operator console (Codespace, local Dev Container, or Minimal local; any path with Docker available). Flashing the produced .bin to the Mango is Lab 03, not here. Lab 02 ends with a verified artifact on disk.

StepOperator console (any of: Codespace, Local Dev, Minimal local)
1. Inspect contractyes
2. Build sysupgrade (ImageBuilder)yes
3. Verify size + manifestyes
4. Compare to upstream rootfsyes
Validationyes

Learning objectives

  • Understand the ImageBuilder model: pre-compiled packages assembled into a firmware image without a full source build.
  • Read the PACKAGES list in build-drop-mango.sh as the engineering document. Each addition or removal has a NOR-flash cost.
  • Use SOURCE_DATE_EPOCH to produce bit-identical artifacts across hosts.
  • Read build-manifest.json as a reproducibility audit trail.
  • Verify a squashfs size constraint programmatically. Know what to cut when it fails.

Pre-state

Before starting this lab confirm:

# Lab 01 is complete: SSH to the Mango works.
# (Skip this check if you are running Lab 02 ahead of bench access; the
# build itself does not need the Mango to be reachable.)
ssh -o ConnectTimeout=5 root@192.168.8.1 'echo ok'   # stock GL.iNet; use 192.168.1.1 for pure-OpenWrt

# Docker is reachable from your operator console (this is the inner DinD
# daemon in Codespace / Local Dev Container, or your host docker for
# Minimal local).
docker info | grep 'Server Version'

# Pull the OpenWrt 23.05.3 ImageBuilder image (~800 MB; first run only).
docker pull openwrt/imagebuilder:ramips-mt76x8-23.05.3

# The lab's build inputs are checked into the repo. Confirm they exist.
ls courses/engagement-platform-labs/labs/Makefile
ls courses/engagement-platform-labs/labs/lab02-imagebuilder-firmware/build-drop-mango.sh
ls courses/engagement-platform-labs/labs/shared/files-mango/etc/banner
ls courses/engagement-platform-labs/labs/shared/build-manifest.schema.json

If any of these are missing, run git status. Restore with git checkout HEAD -- <path>.

Optional pre-warm for Local Dev Container. Running make engagement-platform from labs/ builds the Debian operator-console Dev Container image and writes a manifest. This is orthogonal to ImageBuilder (it’s a normal Docker build, not an OpenWrt firmware build) but it is the same image VS Code triggers when you “Reopen in Container,” so pre-running it warms the cache. Codespace students skip this; the Codespace builds the image on launch.


Walkthrough

Spend five minutes reading the four files that define what goes into the firmware. The rest of the lab makes more sense once the package list is in your head.

# All commands run from the course root:
# cd courses/engagement-platform-labs

# The build script. The PACKAGES variable is the engineering document.
cat labs/lab02-imagebuilder-firmware/build-drop-mango.sh

# The shared files overlay baked into the squashfs:
cat labs/shared/files-mango/etc/banner
cat labs/shared/files-mango/etc/uci-defaults/99-enroll.sh.template
# Note: template placeholders ({{WORKER_URL}} etc.) are NOT substituted
# here. Lab 12 substitutes real secrets and rebuilds.

# The Makefile glue. `drop-mango` is the target you run in Step 2.
cat labs/Makefile

Read the PACKAGES list deliberately. The list is short and intentional:

  • Base system: base-files, busybox, fstools, libc, mtd, netifd, opkg, uci. The minimum you need for an OpenWrt boot.
  • Crypto + transport: ca-bundle, ca-certificates, dropbear, libustream-mbedtls, urandom-seed, urngd. The Mango speaks TLS to the Cloudflare Worker and accepts SSH from the tailnet operator.
  • ExtRoot prerequisites: block-mount, kmod-usb-storage, kmod-usb3, kmod-fs-ext4, e2fsprogs. These let Lab 03 mount a USB drive as the filesystem overlay. Without them in NOR, you couldn’t grow past 16 MB.
  • Minimal tooling: curl, jsonfilter, tcpdump-mini. Just enough to talk to a Worker and capture a packet.

Read the negative entries (lines starting with -):

  • -luci, -luci-base, -luci-mod-admin-full, -luci-theme-bootstrap. No web UI. The drop device is operated over SSH and Tailscale, not a browser.
  • -dnsmasq, -firewall4, -nftables, -kmod-nft-offload. No router-on-the-edge role. The Mango connects only to the tailnet; it does not provide DNS, NAT, or firewalling for downstream clients.
  • -wpad-*, -odhcpd-ipv6only, -ppp*. Wi-Fi authenticator and IPv6/PPP daemons that ship in the default profile but are not used in the drop role.

Background reading. OpenWrt’s ImageBuilder docs describe the command-line flags this script is wrapping (make image PROFILE= PACKAGES= FILES=). The package selection guide is helpful if you want to extend the list later.

The build runs inside the ImageBuilder Docker container. That image ships with the cross-compilation toolchain and pre-built packages for ramips/mt76x8, so you do not need a native MIPS toolchain on your laptop. The docker-compose.yml file in labs/ wires the container up; the Makefile target invokes it.

cd courses/engagement-platform-labs/labs

# The canonical entry point. Idempotent.
make drop-mango

The Makefile expands to:

docker compose run --rm imagebuilder \
    /labs/lab02-imagebuilder-firmware/build-drop-mango.sh

Expected output (truncated):

>>> drop-mango build
    PROFILE=glinet_gl-mt300n-v2
    FILES_DIR=/labs/shared/files-mango
    SOURCE_DATE_EPOCH=1714694400
    OUTPUT_DIR=/labs/output
...
>>> built: bin/targets/ramips/mt76x8/openwrt-23.05.3-ramips-mt76x8-glinet_gl-mt300n-v2-squashfs-sysupgrade.bin
    size: <N> bytes
    sha256: <...>
    squashfs rootfs: <M> bytes
>>> artifacts in /labs/output/

First-run timing. ImageBuilder downloads ~150 MB of package metadata on the first invocation (the OpenWrt 23.05.3 feed for ramips/mt76x8). Subsequent rebuilds reuse the container’s package cache. Allow 3–5 minutes the first time; under a minute thereafter.

If the build fails because the package list pushes the squashfs past the ceiling, the script exits with:

ERROR: squashfs rootfs N bytes exceeds ceiling 13631488
       trim the PACKAGES list or move heavy packages to ExtRoot (Lab 03)

This is intentional. The constraint is the lesson.

# Confirm the artifact exists.
ls -lh labs/output/
# Expected:
#   build-manifest.json
#   openwrt-23.05.3-ramips-mt76x8-glinet_gl-mt300n-v2-squashfs-sysupgrade.bin
#   ... and possibly the matching factory.bin and .manifest from ImageBuilder

# Locate the sysupgrade .bin and report its size in MB.
BIN=$(ls labs/output/*glinet_gl-mt300n-v2*sysupgrade.bin 2>/dev/null | head -1)
echo "$BIN"
wc -c < "$BIN" | awk '{printf "%.2f MB\n", $1/1024/1024}'
# Should be well under 16 MB. A typical clean build lands around 5–7 MB.

The 16 MB ceiling is the Mango’s NOR chip size. The build script enforces a tighter 13 MB cap on the rootfs alone, leaving headroom for U-Boot (~256 KB), the kernel (~1.5 MB), and a small reserved region. See the GL-MT300N-V2 device page for the partition layout.

# Read the build manifest. Required fields: role, openwrt_version, created_at.
# Optional but useful: openwrt_target, openwrt_profile, package_list_sha256,
# image_sha256, source_date_epoch.
cat labs/output/build-manifest.json

The manifest is the audit trail for “this exact firmware was built from this exact source on this exact day.” package_list_sha256 is the SHA of the resolved opkg package list (sorted, newline-delimited). When two students compare builds:

  • If the image_sha256 matches: identical bytes. Confidence high.
  • If package_list_sha256 matches but image_sha256 differs: the package list was the same, but a feed served different package metadata between the two builds. Investigate the feed mirror.
  • If package_list_sha256 differs: someone edited the PACKAGES list.

Compare with a peer:

sha256sum labs/output/*sysupgrade.bin

Two students who clone the same commit and build on the same day should see the same hash, byte for byte.

You met the upstream OpenWrt rootfs in Lab 01 Step 4: the openwrt/rootfs:x86-64-23.05.3 sibling launched from the operator console. That image is the unconstrained OpenWrt: ~127 packages, busybox + opkg + dropbear + a kernel’s worth of modules + the standard OpenWrt service plumbing, with no flash limit because it lives on a container filesystem.

Your Mango sysupgrade is the constrained OpenWrt: a deliberately narrower package list selected to fit the 16 MB NOR chip. The interesting question is what survived the cut.

# 1) Read the package manifest ImageBuilder produced. Sorted, line-delimited:
ls labs/output/*glinet_gl-mt300n-v2*.manifest
cat labs/output/*glinet_gl-mt300n-v2*.manifest | sort

# 2) Count it.
wc -l labs/output/*glinet_gl-mt300n-v2*.manifest
# A clean drop-mango build lands around 90 packages (verified 2026-05-04
# at 93 lines). The PACKAGES list is small (~30 named entries), but opkg's
# dependency resolution pulls in libc, libgcc, mbedtls, kernel modules,
# wireless-regdb, and similar transitive deps, so the resolved manifest is
# roughly 3x the named-package count.

# 3) From inside the OpenWrt rootfs sibling (Lab 01 Step 4b), get its
#    package count for comparison:
docker exec ep-devcontainer opkg list-installed | wc -l
# ~127 packages on the upstream x86_64 rootfs.

The point is not that the Mango image is smaller than the upstream rootfs. The point is that the Mango image carries a different kind of software:

CategoryUpstream rootfs (sibling)Mango sysupgrade (this build)
Base + opkgyesyes
LuCI web UIyesno (drop role)
dnsmasq, firewall4yesno (not a router)
ExtRoot prereqsno (irrelevant on x86)yes (USB grow path)
Tailscale / cloudflarednono (installed in Lab 03)

Lab 03 takes this firmware, flashes it (sysupgrade -n), mounts a USB drive as ExtRoot, and installs Tailscale and cloudflared on the overlay. That sequence is only possible because Lab 02 reserved the block-mount + kmod-usb-storage + kmod-fs-ext4 + e2fsprogs slots in NOR.


Post-state

When this lab is complete you should be able to answer yes to all of the following:

  • labs/output/ contains a *-glinet_gl-mt300n-v2-squashfs-sysupgrade.bin file produced by the latest build.
  • wc -c on that file reports a size below the 16 MB NOR ceiling (16,777,216 bytes).
  • labs/output/build-manifest.json exists with role: "drop-mango", openwrt_version: "23.05.3", and a recent created_at timestamp.
  • validate.sh exits 0.
  • You can name three packages that are present in the Mango image and explain what each enables (canonical answer: block-mount for ExtRoot, dropbear for SSH, ca-bundle for outbound TLS).
  • You can name three packages that are deliberately absent from the Mango image and explain why (canonical answer: luci because no web UI is needed in the drop role; dnsmasq because the Mango is not a router; tailscale because it doesn’t fit in NOR and lives on ExtRoot in Lab 03).

Validation

Three validation paths. Pick whichever matches your setup. All three call the same validate.sh and post the result to the same backend; only the delivery channel differs.

Path 1: epl CLI (recommended from the operator console).

epl validate lab02-imagebuilder-firmware

The validator reads labs/output/build-manifest.json and the *-sysupgrade.bin file; no Mango access required. Re-run as many times as you like; only the most recent result is recorded.

Path 2: direct script.

bash courses/engagement-platform-labs/labs/lab02-imagebuilder-firmware/validate.sh

Or via the Makefile:

cd courses/engagement-platform-labs/labs
make validate-lab02-imagebuilder-firmware

Path 3: paste output into the widget.

Run validate.sh anywhere, copy the full output, paste it into the widget below.

What the script checks:

  1. labs/output/build-manifest.json exists.
  2. The manifest contains the required fields (role, openwrt_version, created_at).
  3. The manifest validates against labs/shared/build-manifest.schema.json when python3 is available. (Skipped with a [SKIP] note otherwise.)
  4. A *glinet_gl-mt300n-v2*sysupgrade.bin file exists in labs/output/.
  5. That .bin is smaller than 16 MB (the NOR chip size).
Validate Output Paste your validate.sh output below

Take-home extension

See take-home/lab02-mt3000-build/ for the parallel exercise on the cancelled MT3000 hardware (mediatek/filogic, glinet_gl-mt3000). The MT3000 carries eMMC, so its “drop firmware” has no 16 MB constraint. The contrast becomes a different question: what does the MT3000 carry that the Mango cannot, and what mission profile would use both?

The take-home directory is a stub during the May 2026 cohort and will be expanded post-workshop.


Primitives and drop-in substitutes

OpenWrt’s ImageBuilder is one implementation of a wider primitive: declarative firmware assembly. You name a target, a profile, a package list, and a file overlay; the toolchain produces a flashable artifact with a known size and a reproducible hash. The package selection is the engineering exercise; the toolchain that performs the assembly is interchangeable.

The table below maps that primitive to its closest equivalents across the embedded and systems-Linux landscape. Each tool accepts a declarative description of what should be in an image and outputs a bootable artifact. The variables are: what the “declaration” looks like, how the output is constrained, and who the upstream maintainer is.

NameModelLicenseWhen to consider it
OpenWrt full source build (make image)Source-from-scratch; full toolchain compiles every package from CGPL-2.0When ImageBuilder’s pre-compiled feed is missing a patch you need; when developing OpenWrt itself
Yocto Project / OpenEmbeddedLayered recipes assembled by BitBake; each layer can override any earlier layerMIT (framework); various (generated binaries)Industrial and commercial embedded products; vendors supply BSP layers for their SoCs
BuildrootDeclarative defconfig + Makefile; no package manager in the final imageGPL-2.0Smallest possible custom Linux; dev-board and appliance firmware with no runtime package management
Alpine Linux + apk + initramfsapk packages assembled into a custom rootfs via mkimage/mkinitfsMIT (apk-tools)x86 / ARM appliance images; container base images where musl libc is acceptable
Debian debootstrap + custom kerneldebootstrap extracts a base, then on-host apt installs packages into a chrootGPL/variousTargets with glibc and no flash constraint; common for SBCs (Raspberry Pi, BeagleBone)
mkosiDeclarative spec file (.conf) builds systemd-native images; outputs raw, GPT, or OCILGPL-2.1+Targets running systemd; the operator console Dev Container is a natural fit
OpenWrt-derivative distros (DD-WRT, FriendlyWrt, Padavan)OpenWrt fork with vendor-specific patches and package setsGPL-2.0 (mostly)When the vendor’s tree carries drivers or features that mainline OpenWrt lacks for specific hardware
package feed / sources


assembly tool (ImageBuilder / Yocto / Buildroot / mkosi / debootstrap)

        ├── size constraint applied (NOR ceiling, eMMC budget, container layer limit)


image artifact  ──►  SHA256 hash  ──►  reproducibility check

Take-home prompt. Pick a target on your shelf (RPi 4, an old PC with a 32 GB SSD, a dev board) and rebuild the equivalent of the drop-mango package list using one of the tools above. Compare the resulting artifact size, the hash-reproducibility properties, and the wall-clock time to first usable byte.


Further reading

ImageBuilder and reproducibility:

Hardware-specific:

Workshop-internal:

  • Lab 03: Overlay Deployment. Flashes the .bin you built here onto the Mango, mounts a USB drive as ExtRoot, and installs Tailscale on the overlay.
  • Lab 12: Drop Device. Re-runs this build with enrollment secrets baked into the firmware so the device self-registers on first boot.

View this page’s source: labs/lab02-imagebuilder-firmware/README.mdx


Troubleshooting

make drop-mango: docker compose run fails (imagebuilder service not found)

The docker-compose.yml must be present in courses/engagement-platform-labs/labs/. Verify:

ls courses/engagement-platform-labs/labs/docker-compose.yml
docker compose -f courses/engagement-platform-labs/labs/docker-compose.yml config

If the file is missing, check git status. It may be untracked or deleted.

opkg download errors during build (mirror temporarily unavailable)

ImageBuilder fetches package metadata from downloads.openwrt.org on first invocation. If a feed is temporarily unavailable, the build exits non-zero. Retry:

make drop-mango

If retries also fail, check the OpenWrt mirror status at downloads.openwrt.org. Corporate networks sometimes block outbound on port 80; switching to a mobile hotspot for the first build is a fast workaround.

squashfs ceiling exceeded; build-drop-mango.sh exits with ERROR

You modified PACKAGES and pushed the rootfs past the 13 MB cap. Identify the heaviest packages in the staged image:

docker compose run --rm imagebuilder \
    find /home/buildbot/openwrt-imagebuilder-23.05.3-ramips-mt76x8.Linux-x86_64/bin/targets \
         -name '*.ipk' | xargs ls -lS 2>/dev/null | head -20

Candidate packages to remove first: luci-* (large), nginx*, any extra kmod-* for unused subsystems. Do not remove block-mount, kmod-usb-storage, kmod-fs-ext4, or e2fsprogs. They are required for ExtRoot in Lab 03.

build-manifest.json validation skipped (python3 not found)

validate.sh uses python3 to validate the manifest against the JSON schema. If your host lacks Python 3, the validator emits a [SKIP] notice and continues without that check. The other four assertions still run.

To run the schema check, either install python3 on the host (apt install python3 / brew install python3) or run validation from inside the operator console (which has python3 baked in):

bash courses/engagement-platform-labs/labs/lab02-imagebuilder-firmware/validate.sh
SHA256 differs from a peer’s build, but PACKAGES is identical

Most often this means the OpenWrt package feed served slightly different package metadata between the two builds. The package_list_sha256 in build-manifest.json will agree, but image_sha256 will not.

Solutions, in order of effort:

  1. Re-run both builds at the same time so they hit the same feed snapshot.
  2. Pin a feed mirror by editing /etc/opkg/distfeeds.conf inside the ImageBuilder container.
  3. Accept the drift and rely on package_list_sha256 as the reproducibility check; this is what the OpenWrt project itself does for non-strict reproducibility.
Validate output paster — available in Wave 2D (ValidateOutputPaster lab="lab02")
Downloadable artifacts for lab02 — served from R2 after Wave 3B deployment