Testing

Every pass/fail test is a twister test; all but the S3 variant of the integration test also run end to end under twister (that one is deployed by hand, see below). Four tiers:

Tier

Where

Runs on

Covers

Unit

tests/unit/

host (native_sim/native/64), no board

Portable code as ztest suites: the 802.15.4 PHY (ieee154_phy), the driver’s signal processing (ieee154_dsp), link RS(255,231) (link_rs) and Hamming(128,120) (link_hamming), the stream app’s I/Q packing (iq_pack), the diagnostic-bus ring’s box16 and CIC decimator (diag_dsp, bit-exact against naive references). With IEEE154_TABLES set, ieee154_phy and ieee154_dsp also print their sweep tables (scripts/unit-tables.sh).

Integration

tests/integration/

one real board

ztest on the radio itself. esp_sdr_rx_api: the receive API’s contract (error codes, bounds, one real capture per supported rate); on the S3, C6 and C5. esp_sdr_diag_api: the C5’s gapless diagnostic-bus ring (argument contract, a 10 s run’s byte rate and integrity, every step of the decimation ladder, stop and restart). Need no antenna, peer or host tool and never transmit. esp_sdr_replay_api: the C5’s DAC-engine replay (argument contract, pass time at 80 and 40 MS/s, repeat count, restarts, receive afterwards); plays all-zero words, so only the LO leaks out: keep the board on its coax link.

Regression

tests/regression/

two real boards

pytest through twister’s multi-board harness. ieee802154_link: the C6 native radio and the S3 software radio exchange apps/wpan test frames both ways on channels 11, 15, 20 and 25, plus ACK requests. Guards against the two bugs that only ever showed as lost frames between real boards (doc/hardware-quirks.md). lo_tuning: the S3 sends tones to the C6 at overlapping LO frequencies, through the 5/6 divider and directly tuned; guards the transmit divider and -ERANGE. stream_gapless: the C6 and the C5 on a coax link stream a tone to each other through apps/osmosdr_serial (receive by the CPU-paced estimator or by the C5’s diagnostic-bus ring); no gap may be found in the recording.

Smoke

apps/*/sample.yaml

build, and one board

Every app builds for its boards; apps/capture also boots on the S3 and the C6 and must print esp-sdr rx: ready (console harness); apps/multiapp boots into an app from its file on the FAT volume (pytest, sample.esp_sdr.multiapp.hil).

Continuous integration

.github/workflows/build.yml runs on every push and pull request. It builds every app and test for each board it allows (xiao_esp32s3/esp32s3/procpu and xiao_esp32c6/esp32c6/hpcore, one job each) and builds and runs the unit tests on native_sim and native_sim/native/64. A GitHub runner has no boards, so the integration and regression tests and the console smoke test are only built there; run them on the desk as below. The twister reports of each job are kept as artifacts.

Git hooks

.pre-commit-config.yaml runs the same checks locally with pre-commit (in the nix shell); pre-commit install once sets up both hooks. Each check runs only when its files are staged or pushed:

Hook

Check

When

svd2md

the register pages match doc/registers/*.svd

commit

svdmerge

those SVDs merge into Espressif’s

commit

soapy

the SoapySDR module builds (in build-soapy/)

commit

docs

Doxygen and Sphinx with warnings as errors

push

unit-tests

twister on native_sim/native/64 (the shell has no 32-bit glibc)

push

build-boards

twister --build-only for the boards

by hand

pre-commit run --hook-stage manual <hook> --all-files runs one on demand (build-boards only runs this way), and SKIP=<hook>[,<hook>] git commit (or git push) skips some, for example offline: docs fetches Zephyr’s intersphinx inventory and svdmerge downloads Espressif’s SVDs into .cache/esp-sdr-zephyr/svd in the west workspace the first time.

Unit tests

esp-sdr-zephyr/scripts/run-unit-tests.sh
esp-sdr-zephyr/scripts/run-unit-tests.sh -s unit.esp_sdr.link_rs    # one suite

The script runs twister on native_sim/native/64 with NSI_OPT=-O2. Plain native_sim is 32-bit and this nix shell has no 32-bit glibc. Without an optimization level, glibc’s _FORTIFY_SOURCE check warns inside native_sim’s own runtime code, and -Werror makes that fatal.

tests/unit/link_rs builds apps/link/src/rs.c with -U__ZEPHYR__, so its host branch is compiled instead of the ESP-IDF one.

Sweep tables

ieee154_phy and ieee154_dsp each hold one more test case that is skipped unless the environment variable IEEE154_TABLES is set (to anything): the PHY’s packet error rate over SNR and carrier offset (test_per_table), and the decoded fraction of the driver’s chain through each decimator over SNR (test_decimator_table; with the variable set, test_cfo_estimator_accuracy also prints its carrier offset estimates). These are the tables doc/ieee802154.md quotes, and what tuning the PHY or a decimator needs.

esp-sdr-zephyr/scripts/unit-tables.sh          # both
esp-sdr-zephyr/scripts/unit-tables.sh phy      # the PHY error rate table
esp-sdr-zephyr/scripts/unit-tables.sh dsp      # the decimator table

The script runs the two tests with IEEE154_TABLES=1 and prints their logs. On an idle machine each table takes between half a minute and a minute, plus the build; concurrent builds can triple that, hence the generous twister timeouts of these ids (300 s and 900 s). The table cases also assert what the documentation claims (phy: no frame lost from 6 dB SNR up, none twice, none with wrong content; dsp: each decimator column reads 1.000 from 2 dB above the SNR where it first does), after printing the whole table, so a regression fails the run and still shows the numbers. Without the variable the cases are skipped and the unit tests stay at a few seconds.

Integration tests (one board)

Each test has one id per board: integration.esp_sdr.rx_api.c6, .c5 and .s3; integration.esp_sdr.diag_api.c5 (the SoCs with a gapless diagnostic-bus ring; another SoC adds its id and boards/<board>.conf to that test); integration.esp_sdr.replay_api.c5 (the same for the DAC-engine replay). The C5 and the C6 both have a USB-Serial-JTAG console, deployed the same way:

west twister -p waveshare_esp32c5_zero/esp32c5/hpcore -T esp-sdr-zephyr/tests/integration/esp_sdr_diag_api \
    --device-testing --flash-before \
    --device-serial-pty "python3 .claude/skills/esp32s3-console/s3pty.py <by-id port>" \
    --west-flash="--esp-device=<by-id port>" --outdir <dir> --runtime-artifact-cleanup pass

esp_sdr_diag_api runs about 25 s and drives 8 pads as outputs (toggling at up to 40 MHz): nothing may be wired to the C5’s GPIO 0, 1, 3, 4, 5, 7, 10 and 23. Its checks are the ring’s contract, not a signal: every byte of 40 MB/s arrives for the time of the run (to 2 %), no block is skipped, the interrupt never falls behind, the byte count follows the system timer (a byte lost per ring wrap would be -17 ppm; the bound is 5 ppm), the consumer’s cost stays under the 6 cycles per byte the CPU has at 240 MHz, the receiver’s noise reaches the pads (the ring is not a constant), and each step of the decimation ladder (250 to 15.625 kS/s) has the right rate, a gapless index and the right count.

esp_sdr_replay_api runs about 10 s. It times the engine against the CPU cycle counter: a pass of n words takes n x 3 cycles at 80 MS/s and n x 6 at 40 MS/s (240 MHz CPU) plus a start of under 400 cycles, and so do 255 passes in a row, which shows the passes follow back to back. Pointing the 40 MS/s bit at bit 15 instead of 17 fails the two 40 MS/s cases (146363 cycles for 96000).

  • C6: flashed directly with twister’s device testing. Opening the C6’s USB-Serial-JTAG port resets the chip, and the suite finishes before a plain --device-serial attach after the flash would see it: open the console after the flash so the reset reruns the suite (--flash-before), or use a --device-serial-pty script that resets the chip once it is attached.

  • S3: the board normally runs an app that owns the USB OTG port, so esptool cannot reset it into the ROM bootloader. The S3 variant therefore builds with sysbuild and MCUboot and has its own CDC-ACM shell and DFU. Deploy it with west dfu -d <build dir> -s <usb serial> and read its CDC-ACM console. The suite sleeps 10 s after boot so the host can enumerate the port first: CDC-ACM does not replay output printed before a reader attaches.

multiapp desk check (one board)

sample.esp_sdr.multiapp.hil builds the multiapp host with hello as its autostart app (apps/multiapp/hil.conf). scripts/twister-flash.py puts the image on the S3 with west dfu and, because the build has a FAT volume image, the volume with west dfu --storage; apps/multiapp/pytest/test_multiapp.py then checks that every app’s file is on the volume and equals its built-in copy, that the boot ran hello from its file, and that hello’s commands answer, its PSRAM atomics included. From the repository root, with the S3 attached:

west twister -T apps/multiapp -s sample.esp_sdr.multiapp.hil --device-testing \
    --hardware-map tests/regression/hardware-map.yaml \
    --flash-command "$PWD/scripts/twister-flash.py"

About a minute, flashing included. It leaves the test image on the board: west dfu and west dfu --storage of a normal multiapp build afterwards.

Regression tests (two boards)

Three scenarios, all pytest through twister’s multi-board harness, each with its own pair of images:

  • tests/regression/ieee802154_link: 802.15.4 frames between the C6’s native radio and the S3’s software radio.

  • tests/regression/lo_tuning: tones from the S3 to the C6 at overlapping LO frequencies.

  • tests/regression/stream_gapless: a tone streamed between the C6 and the C5 through a coax link, both ways, with no gap in the recording.

Each has three ids: regression.esp_sdr.<test>.c6 and .s3 only build the two images (build_only); regression.esp_sdr.<test> runs pytest/ on both boards (required_devices and required_applications). stream_gapless has the C6 and two C5 images (.c5: the receive paced by the CPU through the IQ estimator, .c5_diag: from the diagnostic-bus ring) and a scenario for each: regression.esp_sdr.stream_gapless.c6_c5 and .c6_c5_diag.

Run them with both boards attached:

esp-sdr-zephyr/scripts/run-regression-tests.sh -O <output dir>

For each scenario it builds the two images (the scenario id builds none of its own), puts them on the boards and runs the tests, a few minutes in all; -s regression.esp_sdr.lo_tuning picks one. A relative -O is relative to the repository root. What is specific to a desk is in tests/regression/hardware-map.yaml (change the serial numbers for another desk), apart from the radio levels of lo_tuning (below). Four pieces make twister cope with these two boards:

  • scripts/twister-flash.py, passed as --flash-command. Twister calls it once per board with that board’s build directory and its id from the map. An MCUboot image (the S3) goes on with west dfu -s <id>, anything else with west flash to the port of that id. Twister’s own west flash (esptool) cannot reset an S3 whose app owns the USB OTG port. Both boards flash at once. Afterwards /tmp/esp-sdr-tty/<id> points at the board’s port, found by serial number alone; the map names the S3’s port by this link, because its /dev/serial/by-id name carries the USB product string of whichever scenario’s image is on it.

  • flash_before: true in the map: the console is opened after the flash, because the C6 resets when its port is opened and the S3’s port goes away and comes back during the DFU update.

  • pytest_dut_scope: module and a generous timeout in each scenario: both boards are flashed once for all tests, and the flashing counts against the timeout.

  • tests/regression/conftest.py presses Ctrl-C on each shell before twister looks for the prompt. Twister presses Enter with a bare LF, which a Zephyr shell swallows after a CR (process_nl() in subsys/shell/shell.c), and west dfu ends its lines with CR LF, so the S3 would never show a prompt. Pytest loads this conftest for both scenarios because twister runs it from its own working directory (the repository root with the script), which with no pytest ini file in the tree becomes pytest’s rootdir.

Twister compiles with -Werror, a plain west build does not. With CDC-ACM as console and shell, usbd_cdc_acm.c turns its own logging off with a #warning, which is fatal under twister. Every S3 app with a CDC-ACM shell therefore sets CONFIG_USBD_CDC_ACM_LOG_LEVEL_OFF=y itself.

stream_gapless

tests/regression/stream_gapless builds apps/osmosdr_serial from its own directory, as ieee802154_link does apps/wpan (diag.conf of the app selects the ring for the .c5_diag image). It needs the C6 and a C5 on a coax link with a 30 dB attenuator between their U.FL ports, 2437 MHz. The pytest asks twister for the flashed boards, lets go of their consoles (the stream is binary) and drives them with apps/osmosdr_serial/tools/esdr_serial.py.

For each direction (C6 to C5, C5 to C6) and for the receive rates 250 and 31.25 kS/s, one board transmits a tone at 125 kS/s; after 15 s (the transmitter’s tone drifts a few ppm while it warms up) the other records 10 s. What passes:

  • a usable link: the tone at least 30 dB over the median bin and inside the band;

  • the samples the stream owes (98 %), no index jump, no overflow flag, no lost sample counted by the device (rx_overflows, rx_ring_lost_pairs);

  • apps/osmosdr_serial/tools/gapcheck.py finds no gap in the tone: it tracks the tone segment by segment, demodulates, averages blocks of 64 samples and looks for a spike in the block phase (k lost samples are k * 360 * f / fs degrees) or a dip of the amplitude (zero-filled samples). Phase noise of the blocks is about 1 degree at the bench levels, the threshold 30 degrees;

  • the transmitter played 99.5 % of what it was given (tx_underruns) and the host fed it in real time.

The levels are for this desk: C5 receive gain index 40 (about 10 dB below full scale; from index 48 up the C5 clips), C6 receive index 30, transmit step 2. Override them with ESP_SDR_STREAM_<C5|C6>_RX_GAIN and ESP_SDR_STREAM_<C5|C6>_TX_STEP, and the recording time with ESP_SDR_STREAM_CAPTURE_S. The diagnostic-bus scenario was checked to fail on the bug it guards: serving the USB FIFO once per 8 KiB ring block instead of every 2 KiB failed c6_to_c5 at 250 kS/s with 1.55 M of 2.5 M samples, 758 index jumps and 761 receiver overflows, while the other three streams passed. A failure carries the evidence in one line: sample count, flags, index jumps, the tone’s frequency and level, and the device’s counters. gapcheck.py also runs by hand on any recording (esdr_serial.py <port> rx --out x.npy, then gapcheck.py x.npy --rate 250000; --selftest checks the detector itself). Another SoC pair is another entry in the pytest’s CHIPS and the gains, and another set of ids in testcase.yaml.

lo_tuning

tests/regression/lo_tuning is a small firmware of its own for both boards, with a lo shell command: lo f tunes and prints esp_sdr_set_freq()’s return code, lo tone on/off loops a tone on the S3 (loop playback, no gaps), and lo rx captures 8192 samples at 80 MS/s on the C6 and prints the strongest tone and the level at an expected offset. The C6 does the analysis itself: a Welch average of eight 1024-point FFTs, each bin compared with the median of the bins within 1 MHz, so the C6’s uneven noise floor does not count as a tone. Every filter is open (receive and transmit low-pass code 0). The C6 receives at fixed gain index 40 on its U.FL port and the S3 transmits at its weakest step: on this desk the tones arrive 35 to 51 dB above the floor without clipping. Another RF path (an attenuator, antennas) may need other RX_GAIN and TX_STEP in pytest/test_lo_tuning.py. Before any other check the scenario measures the link at 2412 MHz, a Wi-Fi channel tuned the vendor’s way, and stops with a “no usable link” message if the tone is missing there.

A tone sent at TX LO + 3 MHz must show up at (TX LO - RX LO) + 3 MHz, at least 20 dB above the local floor, the strongest one in the capture, within 150 kHz (the two crystals plus an FFT bin). Equal LOs alone would miss two PLLs that both land on the same wrong frequency, so each frequency is also checked with the boards 10 MHz apart, both ways round. The scenario checks:

  • 1900 to 2610 MHz, in the divider band (PLL at 6/5) and directly tuned. Together with the pairs around 2210 MHz they keep the PLL between 2210 and 2651 MHz, the envelope upstream esp-sdr qualified on all chips.

  • The switch at 2209/2210 MHz: divider on one board, direct tuning on the other.

  • A 2000 MHz transmission is absent at 2400 MHz + 3 MHz, where it went out before the transmit path selected the divider.

  • With the transmitter silent, nothing at the offset (the detector itself).

  • -ERANGE on both boards at 1000, 1700, 3000 and 5000 MHz.

  • No -ERANGE means the PLL locks. The S3’s window edges are found by tuning MHz by MHz, from 1880 MHz down and from 2610 MHz up, and must exist. At the last 3 MHz inside each end, wherever both boards tune without -ERANGE (the C6 8 MHz further in, or 16 if it reports -ERANGE itself), the tone must arrive; at least 2 of the 3 per end must tune. -ERANGE itself is conservative (esp_sdr.h) and is not checked for being reported too often.

Other tools

Measuring tools, not pass/fail gates (the host tables are in the unit tests, above):

  • apps/rfprobe and its tools/rfprobe.py: LO pair sweeps with a tone between two boards, raw PLL sweeps, PLL register scans, analog register dumps and diffs, the receive noise floor.

  • apps/wpan/tools/interop.py and iq_dump.py: two-board interop runs and raw captures.