Crash reports with Embedded Linux
Collect Linux kernel panic and oops records from pstore and send them to Spotflow after reboot.
spotflowd collects kernel panic and oops backtraces from Linux pstore and uploads them as Spotflow crash reports.
Pstore uses a persistent backend, such as ramoops reserved RAM, EFI variables, or ACPI ERST, to retain records across reboots.
These reports contain kernel log/backtrace records, not full memory dumps or userspace process core dumps. Do not assume the MCU crash-analysis features, such as inspecting thread locals and registers from an ELF core dump, apply to these records.
Prerequisites
- Install and configure
spotflowdusing the quickstart or Yocto guide. - Enable pstore and a suitable backend in your kernel and board configuration. For ramoops, this typically includes
CONFIG_PSTORE,CONFIG_PSTORE_RAM, and a board-specific reserved RAM region. - Mount pstore at
/sys/fs/pstoreand verify that records survive a test crash and reboot.
The daemon and Yocto layer do not configure your kernel or reserve RAM. Consult the kernel's ramoops documentation for board setup.
Enable Crash Collection
For a Yocto image, add this setting before building:
PACKAGECONFIG:append:pn-spotflowd = " crashdump"This enables collection, selects root, and configures service access to pstore and the systemd-pstore archive.
For other installations, enable collection in the existing configuration:
[crashdump]
enabled = true
paths = ["/sys/fs/pstore", "/var/lib/systemd/pstore"]The archive path covers systems where systemd-pstore.service moves records out of /sys/fs/pstore before the daemon sees them.
Grant Service Access
The current pstore collector requires root. Both the Yocto crashdump option and the override below run the entire daemon as root, including its network handling, not just crash collection. This increases the impact of a security vulnerability; systemd hardening does not eliminate that risk. Enable it only after reviewing this privilege increase for your deployment. For logging and metrics without crash collection, keep the standard systemd service running as spotflow.
Skip this step if the Yocto crashdump option already configured your service.
For the systemd service installed by the quickstart, create a drop-in:
sudo systemctl edit spotflowd[Service]
User=root
Group=root
ReadWritePaths=-/sys/fs/pstore -/var/lib/systemd/pstoreThese paths extend the unit's existing writable state directory. The - prefix permits missing pstore directories without failing service startup.
If you use other crash paths, grant access to those as well.
sudo systemctl daemon-reload
sudo systemctl restart spotflowd
sudo spotflowd statusVerify Capture After Reboot
Use your board's controlled kernel-crash test procedure on a disposable test device, then reboot it.
A stored record might be named dmesg-ramoops-0 or console-ramoops-0.
A kernel panic stops the entire device and can lose data. Do not trigger test panics on production devices or systems with unsaved data.
The daemon scans on startup and periodically afterwards. On systemd devices, inspect capture logs with:
sudo journalctl -u spotflowd -b
sudo spotflowd statusAfter the device reconnects, open the Spotflow web application and filter crash reports by device ID. Successfully queued records may already have been removed from pstore, so an empty directory alone does not indicate a failure.
Storage and Delivery
Crash records are not copied into the log spool. Pstore, or the configured archive directory, holds records while the device is offline.
The daemon splits each record into CORE_DUMP_CHUNK messages and publishes them with MQTT QoS 0.
By default, a record is deleted once all chunks have entered the local MQTT queue, not after confirmed receipt by Spotflow.
A disconnect, crash, or shutdown can therefore lose queued crash data.
Set delete_after_capture = false if you need to retain the original record for separate archival; local deduplication still prevents ordinary re-sending of already queued records.
Deduplication state is stored at <storage.state_dir>/crashdump/crashdump_state.json.
The daemon includes the kernel release string (uname -r) as a best-effort build ID, not a GNU ELF build ID.
Configuration Options
Option under [crashdump] | Default | Description |
|---|---|---|
enabled | false | Enable collection |
paths | ["/sys/fs/pstore"] | Directories to scan |
kinds | ["dmesg", "console"] | Filename prefixes to collect |
delete_after_capture | true | Delete records after all chunks are queued locally |
poll_interval_secs | 300 | Periodic rescan interval; startup also triggers a scan |
max_bytes | 262144 | Per-record byte limit; larger records are truncated |
chunk_bytes | 8192 | Raw bytes per crash-dump message |