Environment:

Lab 03 — Overlay Deployment

Duration: 60 minutes

The Mango has 16 MB of NOR flash. A kernel, bootloader, and base system together consume most of that. Tailscale, cloudflared, and python3 do not fit in the remaining space. ExtRoot is the OpenWrt solution: mount a USB drive as the /overlay filesystem so every opkg install lands on USB rather than NOR.

This lab takes the sysupgrade .bin built in Lab 02, flashes it onto the Mango, then sets up ExtRoot using UCI fstab. By the end, tailscale, cloudflared, and python3-light are installed and reachable from the Mango shell. Lab 05 puts Tailscale to work; Lab 06 does the same for cloudflared. Both rely on what this lab puts in place.

Order matters here. Step 5 (ExtRoot configuration) must complete and survive a reboot before Step 7 (package installation). If you install packages before ExtRoot is active, they land on NOR and the device may run out of space. Do not skip the reboot in Step 5.

Where each step runs

This lab is marked codespaces: "partial" because flashing, USB partitioning, and reboot monitoring need physical access to the bench. Once the Mango is on the tailnet (Lab 05), the SSH steps in Steps 5 through 8 can be driven from any operator console. For this lab, the tailnet is not yet established, so all SSH steps originate from your laptop.

Step	Bench laptop required	Operator console (local Dev Container or Codespace)
1. Locate artifact	no	yes (artifact lives in the operator console)
2. Flash sysupgrade	yes (Mango is on local LAN)	no
3. Identify USB	yes	no
4. Format ext4	yes	no
5. Configure ExtRoot	yes	no
6. Verify ExtRoot	yes	no
7. Install packages	yes	no
8. Check disk layout	yes	no
Validation	yes (validate.sh SSHes to Mango)	no (until Lab 05)

Codespace users: keep a laptop terminal open for all Mango SSH steps. The Codespace cannot reach your Mango’s local LAN directly. Step 1 (locating the artifact) is the only step that runs inside the Codespace. Once Lab 05 connects the Mango to the tailnet, subsequent Mango SSH work can move into the Codespace. For this lab, use the laptop.

Learning objectives

Flash a custom OpenWrt sysupgrade image from a running OS using sysupgrade -n.
Understand the OpenWrt overlay filesystem model: read-only squashfs (NOR) plus a writable upper layer.
Configure ExtRoot via uci fstab so the upper layer lives on a USB drive rather than in NOR.
Install packages that exceed the NOR budget onto the USB overlay.
Know where Tailscale and cloudflared state persists on the Mango, and why it survives a power cycle.

Pre-state

Before starting this lab confirm all of the following.

Artifacts and software:

# Lab 02 produced the sysupgrade .bin. Confirm it exists.
ls courses/engagement-platform-labs/labs/output/*glinet_gl-mt300n-v2*sysupgrade*.bin

# SSH client on your laptop
command -v ssh

# scp (usually bundled with openssh-client)
command -v scp

Hardware on the bench:

Mango powered on, Ethernet to laptop LAN (black) port.
USB drive (16 GB or larger), not yet plugged into the Mango.
The Lab 02 sysupgrade .bin produced at labs/output/openwrt-23.05.3-drop-v1-ramips-mt76x8-glinet_gl-mt300n-v2-squashfs-sysupgrade.bin. The drop-v1 substring comes from IMAGE_NAME=drop-v1 in build-drop-mango.sh; Lab 12 reuses the convention with drop-v1-sealed.

Mango is reachable:

# If your Mango is still on stock GL.iNet firmware from Lab 01:
ping -c 3 192.168.8.1

# If it was already flashed to pure upstream OpenWrt during Lab 01/02:
ping -c 3 192.168.1.1

After Lab 03’s sysupgrade (Step 2), the Mango will boot pure OpenWrt 23.05.3 at 192.168.1.1. Substitute that IP for 192.168.8.1 in all commands once the flash is complete.

Override for validate.sh. The validator defaults to MANGO_HOST=192.168.8.1. After the sysupgrade the Mango listens at 192.168.1.1. Run validation as: MANGO_HOST=192.168.1.1 ./validate.sh

Walkthrough

The sysupgrade .bin was produced by make drop-mango in Lab 02 and written to labs/output/. Confirm it is present before copying to the Mango.

# From the course root (or from inside the operator console):
ls courses/engagement-platform-labs/labs/output/
# Look for:
#   openwrt-23.05.3-drop-v1-ramips-mt76x8-glinet_gl-mt300n-v2-squashfs-sysupgrade.bin

If the file is absent, rebuild it:

cd courses/engagement-platform-labs/labs
make drop-mango

Record the path for the next step:

BIN=$(ls courses/engagement-platform-labs/labs/output/*glinet_gl-mt300n-v2*sysupgrade*.bin 2>/dev/null | head -1)
echo "$BIN"

The labs/output/ directory is inside the Codespace filesystem. You can copy the .bin to your laptop via the VS Code Explorer’s right-click “Download” option, or via scp from the Codespace to your laptop. The flash step (Step 2) must run on your laptop.

The labs/output/ directory is bind-mounted from your laptop into the Dev Container, so the .bin is accessible at the same path on both sides. You can run the scp in Step 2 directly from your laptop using the host path.

The labs/output/ directory is on your laptop’s filesystem. The scp in Step 2 runs directly from your laptop shell.

Flash the Lab 02 image onto the Mango using sysupgrade -n. The -n flag discards the existing overlay (desired for a clean base; any settings from Lab 01 or prior experimentation are removed). All commands run from your laptop.

Bench-only step. The Mango is on your local LAN. Run these commands from your laptop terminal, not from the operator console.

Method A: scp + sysupgrade (standard path)

# Copy the image to Mango RAM. /tmp is a ramdisk; it survives while the
# Mango is running but is wiped on reboot.
# `-O` forces legacy SCP transfer; OpenSSH 9.0+ defaults to SFTP, which
# the Mango's dropbear does not provide.
scp -O "$BIN" root@192.168.8.1:/tmp/sysupgrade.bin  # stock GL.iNet; pure-OpenWrt is 192.168.1.1

# Verify the copy. Both checksums must match before flashing.
sha256sum "$BIN"
ssh root@192.168.8.1 'sha256sum /tmp/sysupgrade.bin'  # stock GL.iNet; pure-OpenWrt is 192.168.1.1

# Flash. The SSH session will drop when sysupgrade kills sshd.
# That is expected behavior; it confirms flashing started.
ssh root@192.168.8.1 'sysupgrade -n /tmp/sysupgrade.bin'  # stock GL.iNet; pure-OpenWrt is 192.168.1.1
# Wait 90 seconds for flash + reboot to complete.

Why scp -O? OpenSSH 9.0+ uses SFTP for scp by default. The Mango’s dropbear does not ship sftp-server, so the unflagged scp fails with ash: /usr/libexec/sftp-server: not found. The -O flag forces the legacy SCP protocol which dropbear handles natively.

After the flash: laptop static IP

The Lab 02 firmware deliberately omits dnsmasq, firewall4, and nftables because the Mango is a drop device, not a router. With dnsmasq absent, the Mango does not run a DHCP server on its LAN port, so your laptop cannot auto-configure when you replug after the flash. Set a static IP on your laptop’s wired interface before the next step:

# Linux (replace enp2s0 with your interface name; check with `ip -o link`)
sudo nmcli device disconnect enp2s0
sudo ip addr add 192.168.1.2/24 dev enp2s0
sudo ip link set enp2s0 up

# macOS (replace en0 with your active wired interface; check with `ifconfig`)
sudo ifconfig en0 192.168.1.2 netmask 255.255.255.0

# Windows (PowerShell as Administrator; replace alias with your wired adapter)
New-NetIPAddress -InterfaceAlias "Ethernet" -IPAddress 192.168.1.2 -PrefixLength 24

Confirm the laptop can now reach the Mango:

ping -c 2 192.168.1.1

After the reboot, the Mango runs pure OpenWrt 23.05.3 at 192.168.1.1 (upstream default; no GL.iNet overlay). The root password is empty on first boot. Set one immediately or install your SSH public key:

ssh root@192.168.1.1   # empty password on first connect

# Option A: set a password
passwd

# Option B: install your SSH public key (then disable password auth)
mkdir -p /etc/dropbear && chmod 700 /etc/dropbear
cat >> /etc/dropbear/authorized_keys << 'EOF'
<paste your ~/.ssh/id_ed25519.pub here>
EOF
chmod 600 /etc/dropbear/authorized_keys

Confirm the firmware version:

ssh root@192.168.1.1 'cat /etc/openwrt_release'
# DISTRIB_RELEASE='23.05.3'
# DISTRIB_TARGET='ramips/mt76x8'

Method B: U-Boot HTTP recovery (fallback)

Use this if Method A fails because the Mango is unresponsive over SSH. This is the same u-boot_mod HTTP recovery path exercised in Lab 01 Step 7.

Power off the Mango.
Set your laptop’s Ethernet interface to a static 192.168.1.2/24.
Press and hold the reset button.
While holding, plug power back in. Hold for at least 5 seconds.
Serial will print HTTP server is starting at IP: 192.168.1.1 when ready.

Upload the image:

curl -F "firmware=@${BIN}" http://192.168.1.1/

Wait for the Mango to reboot (LED returns to solid white, roughly 90 seconds).
Restore your laptop’s Ethernet to DHCP and verify SSH at 192.168.1.1.

No TFTP on this hardware. The Mango uses u-boot_mod HTTP recovery, not TFTP. Do not set up a TFTP server. See Lab 01 Step 7 for the full explanation.

Plug the USB drive into the Mango’s USB-A port, then SSH in and run block detect:

ssh root@192.168.1.1 'block detect'

Expected output (excerpt):

config 'device'
    option name 'sda'

config 'mount'
    option device '/dev/sda1'
    option target '/mnt/sda1'

If a partition appears (/dev/sda1), the drive already has a partition table. Proceed to Step 4. If only the raw device (/dev/sda) appears with no partition, create one now:

ssh root@192.168.1.1 '
fdisk /dev/sda <<EOF
o
n
p
1


w
EOF
'

Refresh the block detection and confirm /dev/sda1 is listed:

ssh root@192.168.1.1 'block detect'

Why ext4, not FAT32 or exFAT. OpenWrt’s block-mount requires a Linux filesystem to handle POSIX permissions and symlinks in the overlay. FAT32 and exFAT do not support these, so mkfs.ext4 is the correct choice for ExtRoot. The kmod-fs-ext4 and e2fsprogs packages were included in the Lab 02 NOR image specifically for this step.

e2fsprogs is present in the NOR image from Lab 02. Format the partition with a label for easy identification:

ssh root@192.168.1.1 'mkfs.ext4 -L extroot /dev/sda1'

Expected output (abbreviated):

mke2fs 1.46.x (...)
Creating filesystem with 3932160 4k blocks ...
Writing superblocks and filesystem accounting information: done

If mkfs.ext4 is not found, the Lab 02 build did not include e2fsprogs. Stop and rebuild:

cd courses/engagement-platform-labs/labs
make drop-mango
# Then repeat Step 2 (flash) and return here.

This is the critical step. The UCI fstab configuration tells block-mount to mount /dev/sda1 as /overlay at boot, replacing the NOR overlay with the USB filesystem. Without this, the NOR’s 16 MB ceiling remains in force.

Reference: OpenWrt ExtRoot configuration

SSH into the Mango and run the following sequence:

ssh root@192.168.1.1

Once on the Mango shell:

# Mount the USB temporarily so we can copy the current overlay contents.
# This preserves any UCI configuration already written to the NOR overlay.
mkdir -p /mnt/usb
mount /dev/sda1 /mnt/usb

# Copy current overlay to USB
cp -ar /overlay/. /mnt/usb/
umount /mnt/usb

# Read the ext4 UUID for the fstab entry. UUID-based mounts survive
# device enumeration changes (e.g., if a second USB device is added).
eval "$(block info /dev/sda1 | grep -o 'UUID=\S*')"
echo "UUID: $UUID"

# Write the fstab entry
uci set fstab.extroot=mount
uci set fstab.extroot.uuid="$UUID"
uci set fstab.extroot.target="/overlay"
uci set fstab.extroot.enabled="1"
uci commit fstab

# Verify
uci show fstab

Expected output from uci show fstab:

fstab.extroot=mount
fstab.extroot.uuid='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
fstab.extroot.target='/overlay'
fstab.extroot.enabled='1'

Exit the shell and reboot:

exit
ssh root@192.168.1.1 'reboot'
# Wait 60 seconds.

Why UUID and not /dev/sda1 directly. If the Mango ever enumerates a second USB device before the ExtRoot drive, the block device letter can shift (sda becomes sdb). A UUID mount is unaffected by enumeration order.

After reboot, confirm the USB is mounted as /overlay:

ssh root@192.168.1.1 'df -h /overlay'

Expected output:

Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                14.9G     28.0K     14.1G   0% /overlay

The filesystem source must be /dev/sda1 (or whichever block device your USB drive enumerated as), not overlayfs:/overlay. The latter indicates NOR is still the overlay and ExtRoot did not take effect.

Also check the mount table:

ssh root@192.168.1.1 'mount | grep overlay'
# Must include a line like:
# /dev/sda1 on /overlay type ext4 (rw,relatime,...)

If ExtRoot is not active after reboot, see the Troubleshooting section below. The most common cause is a UUID mismatch between the fstab entry and the filesystem created in Step 4.

With /overlay on the USB drive, the available package space is limited only by the drive size. The two packages that could not fit in NOR are tailscale and python3-light. You install them offline from a .ipk bundle staged on your laptop.

Why offline (and not `opkg update && opkg install ...`)?

The Mango’s drop firmware excludes dnsmasq, firewall4, and nftables because the Mango is a drop device, not a router. As a result, the Mango has no DHCP client on its LAN side, no NAT, and (with only the LAN cable plugged in) no internet. Running opkg update on the Mango at this point fails with Network unreachable.

Setting up internet for the Mango means either:

A second Ethernet cable from the Mango’s WAN port to your home router, OR
Configuring your laptop as a NAT gateway (different setup on Linux, macOS, and Windows; brittle in classroom networks).

Both paths add support friction. The cleaner approach is to stage the .ipk files on your laptop ahead of time and copy them to the Mango. Practical wins:

No internet on the Mango required. Your laptop already has internet (that is how you cloned the repo); the Mango never needs to talk to the OpenWrt feeds directly.
OS-agnostic. Every laptop has Python and curl; per-OS NAT setup does not.
Reproducible. The same script on the same day produces identical bytes. Pinned through opkg’s Filename: index column and recorded with SHA256 sums in the bundle’s manifest.json.
Real-world drop-device pattern. Operators who deploy embedded hardware to constrained networks routinely build an offline package bundle on a staging host, then sneakernet it onto the device. Lab 12 reuses exactly this pattern when it bakes secrets into the Mango image.
Audit-friendly. Every binary that lands on the Mango passes through your laptop, where you can hash, sign, or scan it before deploy. The pattern generalizes: you control what goes on the drop device.

Build the offline bundle

The cache-mango-ipks.py helper in tools/ resolves the tailscale + python3-light + fdisk dependency closure against the OpenWrt 23.05.3 feeds and downloads the .ipk files to labs/output/ipks/.

# From the course root, on the laptop (or operator console):
cd courses/engagement-platform-labs
uv run tools/cache-mango-ipks.py
# Bundle lands in labs/output/ipks/ with a manifest.json

Expected output (sizes will match across students on the same day):

[cache] resolving dependency closure...
  8 packages in closure (excluding NOR baseline)
    tailscale          1.58.2-1
    python3-light      3.11.7-1
    fdisk              2.39-2
    kmod-tun           5.15.150-1
    python3-base       3.11.7-1
    libbz2-1.0         1.0.8-1
    zlib               1.2.13-1
    libpython3-3.11    3.11.7-1
[cache] done: 8 .ipk(s), 12201 KB total

The manifest.json records each package’s name, version, size, and SHA256 so two students can compare bundles in one diff.

Install offline

Copy the bundle to the Mango’s /tmp (a ramdisk; cleaned on reboot, which is what we want for a transient install staging area):

scp -O labs/output/ipks/*.ipk root@192.168.1.1:/tmp/

Then run a single opkg install against the local .ipk paths. opkg resolves the install order from the package metadata; you do not need to install dependencies first:

ssh root@192.168.1.1 'cd /tmp && opkg install *.ipk'

Expected (truncated):

Installing libbz2-1.0 (1.0.8-1) to root...
Installing libpython3-3.11 (3.11.7-1) to root...
Installing python3-base (3.11.7-1) to root...
Installing python3-light (3.11.7-1) to root...
Installing zlib (1.2.13-1) to root...
Installing kmod-tun (5.15.150-1) to root...
Installing tailscale (1.58.2-1) to root...
Configuring libbz2-1.0.
Configuring libpython3-3.11.
Configuring python3-base.
Configuring zlib.
Configuring python3-light.
Configuring kmod-tun.
Configuring tailscale.

Tailscale’s post-install scriptlet may emit harmless bootstrapDNS errors as it tries to phone home for log upload. Those are not install failures and are safe to ignore at this stage.

Clean up the staging copy from the Mango’s ramdisk:

ssh root@192.168.1.1 'rm /tmp/*.ipk'

Why isn’t cloudflared installed on the Mango? Cloudflare does not publish a mipsel release binary (only linux-386 / amd64 / arm / arm64 / armhf), and after the May 2026 architecture pivot the Cloudflare Tunnel terminates at the operator console (Debian, with cloudflared from the Cloudflare apt repo) rather than on the Mango. The Mango reaches the operator console over Tailscale; the public tunnel does not need a Mango-side cloudflared daemon. See Lab 06 for the tunnel architecture.

Verify the installs:

ssh root@192.168.1.1 'tailscale --version'
# 1.58.2
#   go version: go1.21.13

ssh root@192.168.1.1 'python3 --version'
# Python 3.11.7

Both must return non-empty output before proceeding to the Validation section.

Run a full df -h to confirm the complete storage picture:

ssh root@192.168.1.1 'df -h'

Expected output (sizes vary by USB drive):

Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.4M      2.4M         0 100% /rom
tmpfs                    61.6M    308.0K     61.3M   1% /tmp
/dev/sda1                14.9G    120.0M     14.1G   1% /overlay
overlayfs:/overlay       14.9G    120.0M     14.1G   1% /

Key indicators:

/dev/root at /rom is the read-only squashfs from NOR. It reports 100% used because it is a compressed, read-only image with no free space by design.
/dev/sda1 is at /overlay, with gigabytes free.
overlayfs:/overlay at / reports the USB drive’s free space, not NOR’s.

If / still reports only a few MB free, ExtRoot is not active. Return to Step 5.

Also confirm Tailscale state will persist across a power cycle:

ssh root@192.168.1.1 'ls /var/lib/tailscale'
# Should show state files after a tailscale up (Lab 05); for now,
# just confirm the directory is writable on the USB overlay.
ssh root@192.168.1.1 'touch /var/lib/tailscale/.test && rm /var/lib/tailscale/.test && echo writable'

Because /var/lib/tailscale is on the overlayfs, which is backed by USB, Tailscale state survives power cycles. This is the same persistence guarantee a named Docker volume provides to the operator console.

Problem	Mango solution	Operator console solution
Where does Tailscale state persist?	`/var/lib/tailscale` on USB via `/overlay`	`/var/lib/tailscale` on a named Docker volume (`epl-tailscale`)
Why not write to rootfs directly?	NOR read-only squashfs base; overlay is writable but was NOR-constrained	Container restart wipes ephemeral layers
How is the mount configured?	UCI `fstab` entry (this lab)	`devcontainer.json` `mounts` key

Post-state

When this lab is complete you should be able to answer yes to all of the following:

cat /etc/openwrt_release on the Mango shows DISTRIB_RELEASE='23.05.3' and DISTRIB_TARGET='ramips/mt76x8'.
df -h /overlay shows /dev/sda1 (the USB drive), not overlayfs:/overlay.
mount | grep overlay shows type ext4.
/overlay has at least 1 GB free.
tailscale --version on the Mango returns a non-empty version string.
python3 --version on the Mango returns a non-empty version string. (cloudflared is not installed on the Mango; see Step 7 note.)

Validation

Three validation paths. Pick whichever matches your setup. All three call the same validate.sh and post the result to the same backend; only the delivery channel differs.

Path 1: epl CLI (recommended from the operator console once the tailnet is live; see note below).

epl validate lab03-overlay-deployment

The validator SSHes to the Mango at MANGO_HOST. In this lab the Mango is not yet on the tailnet (that is Lab 05), so the operator console cannot reach it directly. Use Path 2 or 3 from your laptop until Lab 05 is complete.

Path 2: direct script (bench laptop).

MANGO_HOST=192.168.1.1 bash courses/engagement-platform-labs/labs/lab03-overlay-deployment/validate.sh

Or via the Makefile:

cd courses/engagement-platform-labs/labs
MANGO_HOST=192.168.1.1 make validate-lab03-overlay-deployment

The script exits 0 on success and prints the first failing assertion on failure.

Path 3: paste output into the widget.

Run validate.sh on your laptop, copy the full output, and paste it into the widget below. Useful if neither the operator console nor the automated path can post to the backend directly.

What the script checks:

SSH into the Mango at $MANGO_HOST succeeds.
/overlay is listed in mount output with type ext4 (ExtRoot is active).
/overlay has at least 1 GB free (confirms USB drive, not NOR ramdisk).
tailscale --version returns a non-empty string.
python3 --version returns a non-empty string.

Default MANGO_HOST. The script defaults to 192.168.8.1 (stock GL.iNet address). After the Lab 03 sysupgrade the Mango is at 192.168.1.1. Always pass MANGO_HOST=192.168.1.1 explicitly when running this lab’s validator.

Take-home extension

See take-home/lab03-mt3000-emmc/ for the MT3000 variant. The MT3000 carries eMMC storage and has no 16 MB constraint; the ExtRoot pattern is unnecessary there. The contrast makes a useful architectural discussion: when does ExtRoot make sense and when does it add complexity for no benefit?

The take-home directory is a stub during the May 2026 cohort and will be expanded post-workshop.

Primitives and drop-in substitutes

OpenWrt’s ExtRoot mounts a USB drive as the filesystem overlay so opkg installs land on USB rather than NOR. The primitive itself is older and more general: a read-only lower layer, a writable upper layer, and a kernel driver that merges them at the VFS layer. Linux has shipped overlayfs since 3.18 (2014); block-mount and UCI fstab are OpenWrt-specific glue that automate the mount. The substitution surface is mostly about how the upper and lower layers are sourced, persisted, and reset.

Overlay structure

+------------------------------+
|   merged view (/)            |  <- what processes see
+------------------------------+
| upper: /mnt/usb (ext4)       |  <- writes land here (this lab)
| lower: /rom    (squashfs)    |  <- read-only, NOR-resident
+------------------------------+
           ^       ^
           |       |
      kernel overlayfs driver

Substitutes

Name	Model	When to consider it
Linux `overlayfs` (raw)	manual mount: `lower=/`, `upper=/mnt/usb`, `work=/mnt/usb-work`, `merged=/overlay`	non-OpenWrt embedded Linux with a small rootfs and a writable partition; what `block-mount` automates
`aufs`	older multi-layer union FS	historical reference only; unmaintained upstream; do not use for new work
Initramfs unpack-then-overlay	initramfs is the lower; tmpfs or block device is the upper; standard on systemd boot	no NOR/NAND constraint but a clean reboot to a known state is desired
btrfs subvolume + snapshot	block-level copy-on-write; immutable base subvolume + writable subvolume	larger storage substrate (SBC, appliance); per-snapshot rollback available
ZFS dataset + snapshot	similar to btrfs; mature on FreeBSD and Linux	servers with substantial storage; not appropriate for 16 MB NOR
OSTree (libostree)	git-style commit tree of filesystems; atomic upgrades	immutable-OS distributions (Fedora Silverblue, Endless OS)
Container image layers	overlay2 driver in Docker or Podman	the closest parallel to this lab: a multi-layer base plus a writable container layer is `overlayfs` under the hood
A/B partition rollback	two full rootfs slots with a boot-time selector	Android, ChromeOS, automotive firmware; not a “grow rootfs” primitive but addresses the same “recover a bad upgrade” problem

Take-home prompt

On your laptop, mount a tmpfs as the upper layer over /usr/share (lower) and use a workdir under /var/tmp. Install a small package via apt while the overlay is active and observe where the writes land. Unmount the overlay and confirm the base /usr/share is unchanged. Minimal setup:

sudo mkdir -p /var/tmp/overlay-upper /var/tmp/overlay-work /mnt/overlay-test
sudo mount -t overlay overlay \
  -o lowerdir=/usr/share,upperdir=/var/tmp/overlay-upper,workdir=/var/tmp/overlay-work \
  /mnt/overlay-test
ls /mnt/overlay-test        # merged view
# write a file and check it appears in /var/tmp/overlay-upper, not /usr/share
sudo umount /mnt/overlay-test
ls /var/tmp/overlay-upper   # writes are here
ls /usr/share/$(ls /var/tmp/overlay-upper | head -1) 2>/dev/null || echo "not in base"

Troubleshooting

ExtRoot not active after reboot (df still shows NOR overlay)

Check whether block-mount found the UUID:

ssh root@192.168.1.1 'block info /dev/sda1'
# The UUID= value here must match what is in fstab.

ssh root@192.168.1.1 'uci show fstab'
# Compare fstab.extroot.uuid against the block info output above.

ssh root@192.168.1.1 'logread | grep -i extroot'
# Look for mount errors or "not found" messages.

The most common cause is a UUID mismatch: if mkfs.ext4 was run after the eval "$(block info ...)" line in Step 5, the UUID in UCI refers to the old filesystem. Re-run the eval line and the uci set fstab.extroot.uuid command with the current UUID, then uci commit fstab and reboot.

Second common cause: the USB drive is not plugged in. Confirm the drive is physically seated in the Mango’s USB-A port.

mkfs.ext4 not found

e2fsprogs is in the canonical PACKAGES list for build-drop-mango.sh. If it is missing, the Lab 02 build did not use the canonical list.

Temporary workaround without rebuilding:

ssh root@192.168.1.1 'opkg update && opkg install e2fsprogs'
# NOR space is tight; this may fail if other packages consumed the free space.
# If it fails, rebuild Lab 02 with the canonical list and reflash.

opkg install tailscale fails (package not found)

Confirm the feed configuration points at 23.05.3:

ssh root@192.168.1.1 'cat /etc/opkg/distfeeds.conf'
# Should reference packages.openwrt.org/releases/23.05.3/mipsel_24kc

If the feed URL is wrong, the Lab 02 firmware used a different release or profile. Rebuild with make drop-mango and reflash.

If the feed is correct but tailscale is absent from the index, check the package search directly:

ssh root@192.168.1.1 'opkg list | grep ^tailscale'

As a fallback, download the .ipk from the feed mirror and install manually:

# https://pkgs.openwrt.org/23.05.3/mipsel_24kc/base/
# Find the tailscale ipk, download it, scp to Mango, then:
ssh root@192.168.1.1 'opkg install /tmp/tailscale*.ipk'

sysupgrade drops SSH connection immediately

Expected. sysupgrade sends SIGTERM to all processes, including sshd, before writing to flash. The connection drop confirms the process started. Wait 90 seconds and reconnect at 192.168.1.1 (not 192.168.8.1; the new firmware uses the upstream OpenWrt default).

SHA256 mismatch after scp to Mango

Do not proceed with flashing if the checksums differ. The transfer was corrupted. Remove the partial file and re-copy:

ssh root@192.168.8.1 'rm /tmp/sysupgrade.bin'       # stock GL.iNet; pure-OpenWrt is 192.168.1.1
scp "$BIN" root@192.168.8.1:/tmp/sysupgrade.bin      # stock GL.iNet; pure-OpenWrt is 192.168.1.1
sha256sum "$BIN"
ssh root@192.168.8.1 'sha256sum /tmp/sysupgrade.bin' # stock GL.iNet; pure-OpenWrt is 192.168.1.1

If the mismatch persists, check available RAM on the Mango (free -m); /tmp is a ramdisk, and a large image on a lightly-loaded device should have room. If /tmp is full, reboot the Mango first to clear it.

Validate output paster — available in Wave 2D (ValidateOutputPaster lab="lab03")

Overlay Deployment

Lab 03 — Overlay Deployment

Where each step runs

Learning objectives

Pre-state

Walkthrough

1. Locate the Lab 02 build artifact

2. Flash the Mango with sysupgrade

After the flash: laptop static IP

3. Identify and partition the USB drive

4. Format the USB as ext4

5. Configure ExtRoot (USB as /overlay)

6. Verify ExtRoot is active

7. Install packages onto ExtRoot

Why offline (and not opkg update && opkg install ...)?