# Bluetooth Low Energy (/fundamentals/bluetooth-low-energy) Bluetooth Low Energy support uses a gateway architecture. A leaf device exposes the Spotflow Observability Service over BLE, and a gateway connects to that service, reads device context, receives framed messages, and forwards them to Spotflow over MQTT. This page describes the BLE communication format: roles, GATT service layout, stream characteristics, message types, and frame encoding. ## Roles [#roles] | Role | Device | | -------------- | ----------- | | BLE central | Gateway | | BLE peripheral | Leaf device | | GATT client | Gateway | | GATT server | Leaf device | The leaf device is the source of telemetry. The gateway bridges the BLE stream to Spotflow MQTT topics. ## GATT service [#gatt-service] The Spotflow BLE protocol uses the following UUID layout: | Item | UUID | | ------------ | -------------------------------------- | | Service UUID | `26530001-81E5-4861-82AE-2C92E6887922` | The service exposes these characteristics: | Name | UUID fragment | Properties | Description | | ---------------- | -------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------- | | Capabilities | `26530002-81E5-4861-82AE-2C92E6887922` | `READ` | Protocol version. The initial protocol version is `0x01`. | | Device ID | `26530003-81E5-4861-82AE-2C92E6887922` | `READ` | Device ID the gateway uses when connecting the device to Spotflow. | | Session Metadata | `26530004-81E5-4861-82AE-2C92E6887922` | `READ` | CBOR-encoded Session Metadata message. The gateway forwards this payload to the `ingest-cbor` MQTT topic. | | TX Stream | `26530005-81E5-4861-82AE-2C92E6887922` | `NOTIFY` | Device-to-gateway stream for telemetry and reported configuration messages. | | RX Stream | `26530006-81E5-4861-82AE-2C92E6887922` | `WRITE without response` | Gateway-to-device stream for desired configuration messages. | ## Streams [#streams] The protocol uses one stream in each direction: | Stream | Direction | BLE property | Used for | | --------- | ----------------- | ------------------------ | ------------------------------------ | | TX Stream | Device to gateway | `NOTIFY` | Telemetry and reported configuration | | RX Stream | Gateway to device | `WRITE without response` | Desired configuration | The gateway controls device-to-gateway traffic through the Client Characteristic Configuration Descriptor (CCCD) on the TX Stream characteristic. When the gateway enables notifications in the CCCD, the device may send TX Stream notifications. When notifications are disabled, the device must stop sending TX Stream notifications. Both streams use the same frame format. Framing is required because BLE messages may be larger than the negotiated MTU, which can be as low as 23 bytes. Because there is a single stream in each direction, multiplexing is handled by the message type in each frame rather than by separate GATT characteristics. ## Message types [#message-types] The first byte of each frame identifies the message type. The receiver reads this byte before parsing the rest of the frame. | Message type | Byte identifier | Fragmented | Allowed in | | ------------------------ | --------------- | ---------- | ---------- | | `ACK` | `0x00` | No | TX/RX | | `NACK` | `0x01` | No | TX/RX | | `TELEMETRY` | `0x02` | Yes | TX | | `REPORTED_CONFIGURATION` | `0x03` | Yes | TX | | `DESIRED_CONFIGURATION` | `0x04` | Yes | RX | `TELEMETRY` messages carry Spotflow telemetry payloads, including core dump chunks, metrics, and logs. When multiple telemetry payloads are pending, the device sends them in this priority order: core dump chunks first, then metrics, then logs. `ACK` and `NACK` are defined by the frame format, but application-level acknowledgement is not currently implemented. ## Frame format [#frame-format] Every frame starts with the message type byte: ```text Byte 0: Message type Byte 1+: Data ``` Fragmented message types use the following frame layouts. First fragment: ```text Byte 0: Message type Byte 1: Flags Byte 2: Message sequence number, uint8 Bytes 3-4: Total message byte length, uint16 little-endian Bytes 5+: Fragment data ``` Continuation fragment: ```text Byte 0: Message type Byte 1: Flags Byte 2: Message sequence number, uint8 Bytes 3+: Fragment data ``` The pair of message type and sequence number identifies one fragmented message within a rolling window of 256 messages of that type. ### Flags [#flags] | Bit | Name | Meaning | | --- | ----------- | ----------------------------------------------------------------------------------------------- | | 0 | `IS_FIRST` | First or only fragment of a message. | | 1 | `IS_LAST` | Last or only fragment of a message. | | 2 | `NEEDS_ACK` | Reserved for application-level acknowledgement. ACK/NACK handling is not currently implemented. | | 3-7 | Reserved | Set to `0`. Receivers must not depend on these bits. | ## Gateway flow [#gateway-flow] A gateway connects a BLE leaf device to Spotflow with this flow: 1. Scan for a device advertising the Spotflow GATT service. 2. Connect to the device. 3. Read the Capabilities characteristic and check protocol compatibility. 4. Read the Device ID characteristic. 5. Read the Session Metadata characteristic. 6. Establish an MQTT connection to Spotflow using the device ID and the gateway-provided ingest key. 7. Publish the Session Metadata payload to the `ingest-cbor` topic. 8. Enable notifications on the TX Stream characteristic. 9. Forward `TELEMETRY` messages from the TX Stream to the `ingest-cbor` MQTT topic. 10. Forward `REPORTED_CONFIGURATION` messages from the TX Stream to the `config-cbor-d2c` MQTT topic. 11. Forward MQTT messages from the `config-cbor-c2d` topic to the RX Stream as `DESIRED_CONFIGURATION` messages. The gateway forwards complete message payloads between BLE and MQTT. It does not need to interpret Spotflow telemetry payloads to route them. ## Delivery guarantees [#delivery-guarantees] The current BLE stream uses `NOTIFY` for device-to-gateway data and `WRITE without response` for gateway-to-device data. This provides best-effort delivery at the application level. The frame format defines `ACK`, `NACK`, and the `NEEDS_ACK` flag for application-level acknowledgement, but that acknowledgement flow is not currently implemented. ## Sample [#sample] The Spotflow device SDK includes a Zephyr sample that demonstrates the BLE transport: [zephyr/samples/ble](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/samples/ble). ## Use the Spotflow Web App as a BLE Gateway [#use-the-spotflow-web-app-as-a-ble-gateway] For development and testing, the Spotflow Web App can act as a BLE gateway. It connects to a BLE leaf device and forwards telemetry to Spotflow over MQTT. Open the BLE Gateway page from the user menu in the top-right corner of the application. "BLE gateway link" On the BLE Gateway page, select the [ingest key](https://docs.spotflow.io/fundamentals/device-authorization) to use for the MQTT connection. Then click **Scan for BLE devices** to find nearby BLE leaf devices advertising the Spotflow GATT service. Select a device from the list to connect. Once connected, the browser forwards telemetry from the BLE device to Spotflow over MQTT. This setup is intended for development and testing because the browser Bluetooth API provides limited control over the BLE stack. # Device authorization (/fundamentals/device-authorization) Devices must be properly authorized to send telemetry data to Spotflow. This guide explains how to use ingest keys for secure device authentication and authorization. ## What are Ingest Keys? [#what-are-ingest-keys] When a device connects to Spotflow, it must present its device id and a valid ingest key. If the key is valid, the device is authorized to send telemetry data. Ingest keys contain `sf_ikv1_` as a prefix for easy identification, they may look like this: ``` sf_ikv1_abcd1234567890abcdef1234567890abcdef1234567890abcdef ``` Ingest keys are sensitive credentials. Treat them like passwords and never expose them in public repositories, logs, or unsecured locations. If an ingest key is compromised, it can allow foreign devices to send data to your Spotflow workspace. ## How to Manage Ingest Keys [#how-to-manage-ingest-keys] Depending on your security requirements and operational needs, you can use a single key for multiple devices or assign unique keys to each device. ### Single Key for Multiple Devices [#single-key-for-multiple-devices] **Use Case**: Development environments, small device fleets, or when operational simplicity is prioritized. **Benefits**: Simplified key management and easier deployment. **Considerations**: If compromised, affects all devices using the key. ### Unique Key per Device [#unique-key-per-device] **Use Case**: Production environments, high-security applications, or when fine-grained access control is required. **Benefits**: Enhanced security isolation and granular access revocation. **Considerations**: Requires secure key provisioning process. ### Hybrid Approach [#hybrid-approach] You might also use a hybrid approach where you group devices by environment, product line, or security zone (e.g., development, staging, production). ## Key revocation [#key-revocation] If you need to revoke an ingest key, you can do so from the Spotflow web app. This will immediately prevent any device using that key from sending data, and it will disconnect any existing connections using that key. ## Learn more [#learn-more] # Gateway & relay architectures (/fundamentals/device-source-id) In a direct-connection setup, the device that transmits a message to Spotflow is also the device that generated the message. In gateway and relay architectures, however, one device connects to the platform on behalf of others, forwarding their telemetry without those devices having a direct internet connection. The **Source Device ID** and **Transport Route** fields allow you to preserve the identity of the originating device throughout the forwarding chain, so that logs, metrics, and crash reports are correctly attributed in Spotflow regardless of how they were delivered. ## Source Device ID [#source-device-id] The `sourceDeviceId` field identifies the original device that generated a message. It is included in the message payload by the forwarding device. | Value | Meaning | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Not specified | The connected device is the source, no forwarding is taking place. | | A string | A single source device whose message is being forwarded by the connected device. | | An array of strings | A multi-hop chain, each element is a device in the forwarding path, ordered from source to the device just before the connected one. | **Constraints:** * Maximum of 4 components in the route (including the connected device). * Each device ID must be alphanumeric and may contain hyphens and underscores, up to 128 characters. * Empty strings and whitespace-only values are rejected. ## Transport Route [#transport-route] The **transport route** is the complete path a message took from its origin to the platform. Spotflow computes it automatically by appending the connected device's ID to the `sourceDeviceId` array. | `sourceDeviceId` in message | Connected device | Resolved transport route | Attributed device ID | | --------------------------- | ---------------- | ----------------------------------------- | -------------------- | | *(absent)* | `gateway-01` | `["gateway-01"]` | `gateway-01` | | `"sensor-01"` | `gateway-01` | `["sensor-01", "gateway-01"]` | `sensor-01` | | `["sensor-01", "relay-01"]` | `gateway-01` | `["sensor-01", "relay-01", "gateway-01"]` | `sensor-01` | The **attributed device ID**, used for querying and display in Spotflow, is always the **first element** of the transport route: the original source device. All devices appearing in the transport route are automatically registered in your Spotflow workspace. You do not need to provision gateway or relay devices separately. ## Use Cases [#use-cases] ### Gateway Architecture [#gateway-architecture] The most common pattern: resource-constrained devices (sensors, actuators) communicate over a short-range protocol such as BLE or Zigbee and rely on a gateway to forward their telemetry to Spotflow over the internet. The gateway connects to Spotflow and publishes messages on behalf of each sensor, setting `sourceDeviceId` to the originating sensor's ID. Spotflow attributes the data to the sensor, not the gateway. ### Multi-Hop Relay [#multi-hop-relay] For topologies with intermediate relay nodes between the source device and the internet-connected gateway: The gateway publishes the message with `sourceDeviceId: ["sensor-01", "relay-01"]`. Spotflow appends the gateway's ID to produce the full transport route `["sensor-01", "relay-01", "gateway-01"]` and attributes the data to `sensor-01`. ## Protocol [#protocol] The `sourceDeviceId` field is supported in log, metric, and crash report messages. Pass a string for a single forwarding hop, or an array of strings for a multi-hop route: ```json { "body": "Temperature reading: 23.5C", "sourceDeviceId": "sensor-01" } ``` ```json { "body": "Temperature reading: 23.5C", "sourceDeviceId": ["sensor-01", "relay-01"] } ``` For the full protocol reference for each signal type, including both JSON and CBOR formats, see: * [Logging — Transport Protocol](/fundamentals/monitoring/logging#transport-protocol) * [Metrics — Transport Protocol](/fundamentals/monitoring/metrics#transport-protocol) * [Crash Reports — Transport Protocol](/fundamentals/monitoring/crash-reports#transport-protocol) ## Learn more [#learn-more] # Firmware management (/fundamentals/firmware-management) Firmware management is a feature that allows to organize one or more firmwares, their versions and associated symbol files and devices, designed specifically for embedded devices. "List of firmwares in the web application." * The concept of `Firmware` represents set of software components deployed to a fleet of devices (typically as single executable binary, e.g. an ELF file). * Firmware can have one or more `Firmware Versions`. * The firmware versions also have associated `Symbol Files`, which are used to analyze crash reports from devices running the firmware. When you create a firmware, you also define its `slug`, which devices use to identify the firmware during Over-the-Air (OTA) updates. Once the firmware is used in an OTA deployment, the slug can no longer be changed. You can also upload firmware images to firmware version for OTA updates. See [Deploy Over-the-Air (OTA) Updates](/using-spotflow/guides/ota). ## Firmware targeting single platform [#firmware-targeting-single-platform] If your firmware is fairly simple (in the terms of builds), we recommend creating single firmware (e.g. *Air monitor x8 firmware*) and create new version (e.g. *v1.2.4*) with symbol file for each new build. If your workflow includes creating many pre-prelease build, we recommend also creating one `pre-release`, `alpha`, `dev` or similar version and assign all symbol files from pre-release builds to this version. Upon release, you can promote the pre-release symbol files to a release version or upload new ones, depending on your workflow. ## Firmware targeting multiple platforms (hardware or OS) [#firmware-targeting-multiple-platforms-hardware-or-os] If your firmware needs to be build for multiple platforms (e.g. different boards or operating systems), you choose one of the following strategies: ### 1. Stick to simple, single-firmware management (described above) and distinguish between platforms via versions [#1-stick-to-simple-single-firmware-management-described-above-and-distinguish-between-platforms-via-versions] In this case, you can create separate versions for each platform by appending the platform name to the version (e.g. *v1.2.4-arm*, *v1.2.4-riscv*). Same goes for the pre-release build, where `pre-release` version can be split into `pre-release-arm`, `pre-release-riscv` and so on. This strategy is suitable for small to moderate number of target platforms, managed by the same team with the versioning scheme is (almost) the same for all of them. ### 2. Create multiple firmwares, one for each platform [#2-create-multiple-firmwares-one-for-each-platform] In this case, you can create separate firmwares for each platform (e.g. *Air monitor x8 firmware - Zephyr*, *Air monitor x8 firmware - FreeRTOS*). In each firmware, you can use the simple versioning schema (e.g. *v1.2.4*) or complex one to incorporate one or more targeting dimensions (e.g. *v1.2.4-arm*, *v1.2.4-riscv*). This strategy is suitable for moderate or large number of target platforms, managed by the multiple teams with possibly different versioning schemes. ## Other scenarios [#other-scenarios] If none of the strategies above fits your scenario or you are having other issues, please open a Feature request or let us know via email [hello@spotflow.io](mailto:hello@spotflow.io) or our Discord. We will be happy to work with you to incorporate necessary changes to the platform or find other suitable solution. ## Show firmware versions running on devices [#show-firmware-versions-running-on-devices] Spotflow links devices to firmware versions automatically using Build IDs. The [Spotflow Device Module](/guides/zephyr/crash-reports-zephyr) for Zephyr RTOS automatically publishes the [Build ID](/fundamentals/monitoring/crash-reports#build-ids) of the device firmware and sends it to Spotflow upon connection. The [Devices](https://app.spotflow.io/devices) page then shows each device's reported Build ID, and also the firmware and its version if the Build ID matches any uploaded Symbol File. This helps you understand which devices are running which versions of firmware and investigate issues with specific releases. "Device page showing devices with their firmware versions." On the [Events](https://app.spotflow.io/events) page, every log or crash report that the device sends shows the Build ID, firmware, and firmware version of the device at the time it was published. "Log detail showing Build ID, firmware, and version." # Alerts (/fundamentals/monitoring/alerts) This page walks you through key aspects of Spotflow alerting, from defining alert rules over device metrics, through understanding how evaluations and alert resolution work, to configuring email notification targets. ## Device Metrics Integration [#device-metrics-integration] Spotflow allows you to define alerting rules based on metrics collected from your embedded devices. [Guide: Gathering Metrics with Zephyr or Nordic nRF Connect SDK](/guides/zephyr/metrics-zephyr) The Spotflow device module can collect system metrics automatically and send them to Spotflow without additional instrumentation. It also provides functions for registering and reporting custom application metrics. [Guide: Gathering Metrics with MQTT](/guides/mqtt/metrics-mqtt) For devices running other platforms or when you cannot use the Spotflow device module, integration is also possible via the standard MQTT interface. The Spotflow platform exposes a scalable MQTT broker accessible from anywhere on the internet that can be used to ingest metrics. See [Transport Protocol](#transport-protocol) section for details. ## Alert Rules [#alert-rules] Alert rules define the conditions under which you want to be notified about certain events or changes in your embedded device fleet. You can create alert rules based on [System Metrics](/fundamentals/monitoring/metrics#system-metrics) (e.g. CPU usage, memory usage) or [Custom Application Metrics](/fundamentals/monitoring/metrics#custom-metrics) (e.g. button presses, battery level, operation duration). ### Query [#query] The core of an alert rule is a query that selects the relevant metric data. The query can either return: * **A single time series**. E.g. average CPU usage across all devices or for a specific device. * **Multiple time series**. E.g. CPU usage for each individual device. ### Condition [#condition] The condition defines the criteria that trigger an alert. An alert is triggered when any of the returned time series meet the specified condition. For example, if the query is grouped by device ID, an alert is triggered when any one device exceeds the threshold. Spotflow supports two types of conditions: * **Threshold**: Alert when a metric crosses a fixed threshold value. For example, CPU usage above 90%. * **Percentual Change**: Alert when a metric changes by a percentage over time. For example, battery voltage drops by 20%. Optionally, you can also configure an alert to trigger when no data is received, which is useful for detecting a device going offline or failing to report a specific metric. ### Intervals Configuration [#intervals-configuration] Alert rules are evaluated at regular intervals, which can be configured when creating the alert rule. You can specify: * **Evaluation Interval**: How often the alert rule is evaluated. For example, every 5 minutes. * **Evaluation Window**: The time range of data that is considered for evaluation. For example, the last 1 hour. * **First Evaluation At**: When the first evaluation should happen. For example, immediately, at the start of the next hour or at midnight. ## Alert [#alert] When the condition of an alert rule is met, an alert is triggered. The alert contains information about the time it was triggered, the metric values that caused the trigger, and any relevant metadata (e.g. device ID). You can also assign custom tags to the alerts to filter them in the alert list. ### Alert Resolution [#alert-resolution] When an alert is triggered, it remains active until the condition is no longer met. For example, if the CPU usage remains above 90%, the alert stays active. Once the CPU usage drops below 90%, the alert will resolve. ## Notification Targets [#notification-targets] When an alert is triggered or resolved, Spotflow can send email notifications to ensure that the right people are informed. A Notification Target is a group of email addresses that can be reused across multiple alert rules. For example, you can have a target for the operations team and use it in all critical alert rules. If you wish to use a different notification method, please open a Feature request or let us know via email [hello@spotflow.io](mailto:hello@spotflow.io) or our Discord. We will be happy to work with you to incorporate necessary changes to the platform or find other suitable solution. ## Learn more [#learn-more] # Crash reports & core dumps (/fundamentals/monitoring/crash-reports) This page walks you through all the aspects related to Spotflow crash reports, from integrating it into devices, through understanding the transport protocol, to analyzing and troubleshooting crash reports in the web application. ## Getting Started with Device Integration [#getting-started-with-device-integration] [Guide: Crash Reports with Zephyr or Nordic nRF Connect SDK](/guides/zephyr/crash-reports-zephyr) Spotflow offers native integration for devices running Zephyr and Nordic nRF Connect SDK through a lightweight software module. This module integrates seamlessly with your existing core dump infrastructure - simply add it as a dependency and let Zephyr create the core dump and call Spotflow's fatal error handler. [Guide: Crash Reports with ESP-IDF](/guides/esp-idf/crash-reports-esp-idf) Spotflow offers native integration for devices running ESP-IDF through a lightweight software module. The module uses ESP-IDF's built-in core dump functionality, stores crash data in a flash partition, and uploads it after reboot. [Guide: Crash Reports with MQTT](/guides/mqtt/crash-reports-mqtt) For devices running other platforms or when you cannot use Spotflow device module, integration is also possible via standard MQTT interface. Spotflow platform exposes scalable MQTT broker accessible anywhere from the Internet that can be used to ingest core dumps. See [Transport Protocol](#transport-protocol) section for details. ## Analyze Crash Reports in the Web Application [#analyze-crash-reports-in-the-web-application] Once your device is integrated, you can analyze crash reports and core dumps in the web application. You can list the crash reports in the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter crash reports by their content, device ID and other metadata. ### Automatic Analysis [#automatic-analysis] We automatically analyse each crash report to give you a quick insight into the issue. For precise results, we extract data directly from the core dump (stack traces, register values, etc.), decompile the firmware binary, and review relevant documentation. Our proprietary AI agent then investigates the crash, leveraging all available data. The result is available in the detail view of each crash report. The full analysis includes a detailed description of the root cause and suggestions for fixing the issue. Also, for complete transparency, we include references to all documentation pages used, allowing you to explore further. We also provide all the raw data extracted from the core dump, so you can dive deeper into the issue if needed. This includes: * Stack traces of one or more threads. * Register values and local variables for individual stack frames. If a register value is available, it can be also casted to several types. * State of global variables at the time of the crash. ## Transport Protocol [#transport-protocol] Spotflow uses an optimized transport protocol based on TCP, MQTT and TLS and two serialization formats (CBOR and JSON). ### MQTT over TLS [#mqtt-over-tls] All log transmission uses **MQTT over TLS (MQTTS)** for secure, reliable communication: * **TLS Version**: 1.2 or higher * **Certificate**: [Let's Encrypt ISRG Root X1](https://letsencrypt.org/certificates/#root-cas) (pre-installed on many systems or included in Spotflow SDKs) * **Authentication**: Devices authenticate using [ingest keys](/fundamentals/device-authorization) as MQTT passwords and their unique device IDs as MQTT usernames. ### Serialization format [#serialization-format] Core dump is a binary file which exact format of the core dump depends on the operating system, hardware and toolchain used to generate it. Some operating systems define their own custom core dump format (e.g. [Zephyr RTOS](https://docs.zephyrproject.org/latest/services/debugging/coredump.html)) while others are building on top of the well-known formats, mainly ELF, by setting `ET_CORE` ELF file identifiers and adding one or more ELF sections (e.g. [Linux](https://www.man7.org/linux/man-pages/man5/core.5.html), [NetBSD](https://man.netbsd.org/core.5)). Spotflow is agnostic to the exact core dump format and provides basic support for core dumps in any format. However, there is advanced support for some formats, including Zephyr RTOS and ESP-IDF. For the specifically supported formats, various information such as stack traces, register values etc are extracted from the core dumps and further analysed. No matter the format, the core dumps are always sent in one or more chunks via MQTT messages with CBOR or JSON formatted content: * **CBOR-based**: recommended for memory and bandwidth efficiency. Used by Spotflow Zephyr and ESP-IDF modules. * **JSON-based**: recommended for simplicity and interoperability, especially for custom integrations over MQTT. When building a custom MQTT integration, JSON core dump chunk payloads can be used even if your device-side SDK uses CBOR by default. Publish core dump chunk messages to the `ingest-json` topic using the following JSON schema: ```json { // Must be set to CORE_DUMP_CHUNK. "messageType": "CORE_DUMP_CHUNK", // Identifier of the core dump unique within a scope of device and last 7-days. "coreDumpId": 123, // Zero-based index of the chunk in the sequence. "chunkOrdinal": 1, // Base64-encoded chunk data. "content": "WkUCAAMABQADAAAAQQIARAADAAAAAAAAAElTKgAAAAAAuMIFE...", // (Optional) Flag indicating if this is the last chunk. "isLastChunk": false, // (Optional) Identifier of the ELF file build for linking with symbols. "buildId": "build-123", // (Optional) Operating system indicator, currently only "Zephyr" or empty one is supported. "os": "Zephyr", // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` Publish core dump chunk messages to the `ingest-cbor` topic using the following CDDL schema: ```CDDL core-dump-chunk = { 0 => 2, ; messageType: must be set to 2 = "CORE_DUMP_CHUNK". 9 => uint, ; coreDumpId: identifier of the core dump unique within a scope of device and last 7-days. 10 => uint, ; chunkOrdinal: zero-based index of the chunk in the sequence. 11 => bstr, ; content: chunk data. ? 12 => bool, ; isLastChunk (optional): flag indicating if this is the last chunk. ? 14 => tstr, ; buildId (optional): identifier of the ELF file build for linking with symbols. ? 15 => tstr, ; os (optional): operating system indicator, currently only "Zephyr" or empty one is supported. ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } ``` ## Build IDs [#build-ids] Depending on the core dump format, some information (such as register names and values) can be obtained directly from the core dump files without any other context. However, to decode information such as stack traces, local variables names and values, global variables names and values, the core dump needs to be linked with the exact version of an executable binary that produced the core dump (more precisely, to the debugging symbols for that binary). Sometimes, These debugging symbols are part of the executable binary itself (e.g. the ELF files support this) and sometimes they are stored in separate symbol files. To facilitate this linking, the concept of **Build ID** is widely used. This ID represents a specific version of the executable binary and associated debugging symbols. When the build ID is included in the core dump metadata and there is a repository repository of debugging symbols available, these two can be automatically linked together. See [Firmware Management](/fundamentals/firmware-management) for details about managing debugging symbols in Spotflow. From the implementation point of view, the build ID is generated by hashing relevant parts of the executable binary (e.g. ELF sections that influence the runtime behavior such as code or global data, excluding sections like debugging symbols) Spotflow is agnostic to the exact build ID generation algorithm as long as it is deterministic and consistent. When using [Spotflow Device Module](/guides/zephyr/crash-reports-zephyr) for Zephyr RTOS, the build ID is automatically generated (using algorithm similar to [GNU Build ID](https://grok.com/share/c2hhcmQtMw%3D%3D_b1d1cef1-8147-4f60-8318-6dd6f3165595)) and embedded into the ELF file during build. When using [Spotflow Device Module](/guides/esp-idf/crash-reports-esp-idf) for ESP-IDF, the build ID is derived from ESP-IDF's application ELF SHA-256 value. ## Device ID [#device-id] Ideally, each physical device should use its own unique device ID for core dump chunks ingestion (possibly even its own [Ingest Key](/fundamentals/device-authorization)). However, we know that mistakes in device provisioning and configuration can happen so our transmission protocol is robust to multiple devices using the same device ID at the same time. In such cases, the devices will still to send core dump chunks to Spotflow without disruption, given that they use unique core dump IDs. ## At least once delivery [#at-least-once-delivery] Once a core dump chunk message is ingested by our MQTT broker, we guarantee that the message will be processed and made available for querying. No data loss is tolerated after acceptance. However, depending on the MQTT QoS level used for publishing, some messages might get lost before they are ingested by the broker. ## Learn more [#learn-more] # Dashboards (/fundamentals/monitoring/dashboards) ## Overview Dashboard [#overview-dashboard] The Overview Dashboard is the fastest way to understand what’s happening across your entire device fleet. It’s a good first stop when you want a fleet-level picture before drilling into a specific firmware version or device. It helps you answer questions like: * **Are devices connecting as expected?** * **Are crashes increasing, and is it tied to a rollout?** * **Which firmware versions are active in the field right now?** * **Did something change in logs (volume or severity)?** Use it when you’re monitoring day-to-day operations, validating a rollout, or responding to a report like “devices started rebooting” or “telemetry dropped”. The dashboard is split into two sections: **Workspace Vitals** and **Fleet Stability**. ### Workspace Vitals [#workspace-vitals] **Workspace Vitals** provides a high-level snapshot of your fleet’s current state and how it has evolved over time. This section helps you quickly spot changes that might indicate an incident or an unexpected effect of a firmware update. It also lets you easily drill down to investigate the devices that need your attention. This section shows: * **Devices connected within last 7 days**: how many unique devices have connected recently * **Devices with crashes**: how many devices have experienced firmware crashes * **Firmware version distribution**: which firmware versions are currently running across your fleet * **Log volume by severity**: how much logging you’re receiving, broken down by severity ### Fleet Stability [#fleet-stability] The **Fleet Stability** section focuses on fleet stability and firmware crashes. Here you can see: * **Crash-free hours**: how reliability changes over time * **Crash rate by firmware version**: which firmware versions are most crash-prone * **Reboot and crash causes**: what’s behind restarts and crashes across the fleet Together, these insights help you quickly pinpoint problematic firmware, assess overall stability, and decide where to focus your troubleshooting or rollout efforts. ## Device Dashboard [#device-dashboard] The Device Dashboard is your go-to place for understanding how a single device behaves in the real world over time. **All data shown in the dashboard is computed from metrics automatically collected by our device module, no extra instrumentation or manual setup required.** To populate these views from Zephyr devices, enable system metrics as described in [Gathering metrics with Zephyr](/guides/zephyr/metrics-zephyr). It helps you answer questions like: * **Is this device stable?** * **Did a recent firmware update introduce crashes?** * **Is the device communicating reliably with the cloud?** * **Are we running close to CPU or memory limits?** The dashboard is organized into three sections: **Device Vitals**, **Connectivity & Traffic**, and **Resource Usage** sections. ### Device Vitals [#device-vitals] Device Vitals provides a high-level snapshot of the device’s overall status. This is usually the first place to look when something feels “off” with a device. This section includes: * **Uptime**: how long the device has been running without restarting * **Crashes over time**: number of crashes the device has experienced * **Crashes by firmware version**: useful when validating new releases * **Reboot reasons**: whether restarts are expected (e.g. user initiated) or unexpected (e.g. crashes) ### Connectivity & Traffic [#connectivity--traffic] The **Connectivity & Traffic** section shows how the device communicates with Spotflow and the network. You can track: * **Sent and received bytes over time**: to understand traffic patterns and data volume * **Logs sent vs. logs dropped**: the device might drop logs if network connectivity is unavailable and the device buffers are full ### Resource Usage [#resource-usage] The **Resource Usage** section helps you understand how close the device is to its hardware limits. You can monitor: * **CPU utilization over time**: to spot performance bottlenecks or spikes * **Heap and stack usage**: to detect memory leaks or insufficient memory allocation ## Custom Dashboards & Product Analytics [#custom-dashboards--product-analytics] Apart from existing dashboards mentioned above, you can also create custom dashboards to monitor **specific aspects** of your device fleet or track **product usage** and **user behavior** through custom application metrics. You can create one or more custom dashboards, each containing a set of widgets visualizing relevant metrics. When adding a **widget**, you can choose between: * Any existing Spotflow-premade widgets described above. * Custom widget. Custom widgets allow you to specify: * Tailored **data query**, based on Spotflow-provided or custom metrics including **filters** and **grouping** based on built-in or custom labels. * Several visualization options such as chart type and units. See [Create Custom Dashboard](/using-spotflow/guides/custom-dashboards) guide for step-by-step instructions. ## Learn more [#learn-more] # Logging (/fundamentals/monitoring/logging) This page walks you through all the aspects related to Spotflow logging, from integrating it into devices, through understanding the transport protocol, to analyzing and troubleshooting logs in the web application. ## Getting Started with Device Integration [#getting-started-with-device-integration] [Guide: Zephyr or Nordic nRF Connect SDK](/guides/zephyr/logging-zephyr) Spotflow offers native integration for devices running Zephyr and Nordic nRF Connect SDK through a lightweight software module. This module integrates seamlessly with your existing logging infrastructure - simply add it as a dependency and use the standard logging macros you're already familiar with. [Guide: ESP-IDF](/guides/esp-idf/logging-esp-idf) Spotflow natively supports logging within devices running ESP-IDF through a lightweight software module. This module integrates seamlessly with your existing logging infrastructure - simply add it as a dependency and use the standard logging macros you're already familiar with. [Guide: MQTT](/guides/mqtt/logging-mqtt) For devices running other platforms or when you cannot use Spotflow device module, integration is also possible via standard MQTT interface. Spotflow platform exposes scalable MQTT broker accessible anywhere from the Internet that can be used to ingest logs. See [Transport Protocol](#transport-protocol) section for details. ## Transport Protocol [#transport-protocol] Spotflow uses an optimized transport protocol based on TCP, MQTT and TLS and two serialization formats (CBOR and JSON). ### MQTT over TLS [#mqtt-over-tls] All log transmission uses **MQTT over TLS (MQTTS)** for secure, reliable communication: * **TLS Version**: 1.2 or higher * **Certificate**: [Let's Encrypt ISRG Root X1](https://letsencrypt.org/certificates/#root-cas) (pre-installed on many systems or included in Spotflow SDKs) * **Authentication**: Devices authenticate using [ingest keys](/fundamentals/device-authorization) as MQTT passwords and their unique device IDs as MQTT usernames. ### Serialization format [#serialization-format] There are two protocol flavors for log ingestion: * **CBOR-based**: recommended for memory and bandwidth efficiency. Used by Spotflow Zephyr and ESP-IDF modules. * **JSON-based**: recommended for simplicity and interoperability, especially for custom integrations over MQTT. When building a custom MQTT integration, JSON log payloads can be used even if your device-side SDK uses CBOR by default. Publish log messages to the `ingest-json` topic using the following JSON schema: ```json { // Fully interpolated log line string (optional when bodyTemplate is used) "body": "SmartLock was Unlocked", // (Optional) printf-like interpolation string for the log line "bodyTemplate": "SmartLock was %s", // (Optional) array of values for interpolation in the bodyTemplate "bodyTemplateValues": ["Unlocked"], // (Recommended) Log severity, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" "severity": "INFO", // (Optional) device uptime when the log was generated (milliseconds since the device booted) "deviceUptimeMs": 23455, // (Optional) time when the log was generated (milliseconds since the UNIX epoch) "deviceTimestampMs": 1748530133808, // (Optional) you can add extra metadata to your logs "labels": { "initiatorKind": "MobileApp", "userId": "1234567890" }, // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` Publish log messages to the `ingest-cbor` topic using the following CDDL schema: ```CDDL log-message = { ? 1 => tstr, ; body: fully interpolated log line string (optional when bodyTemplate is used) ? ( 2 => tstr, ; bodyTemplate (optional): printf-like interpolation string 3 => body-template-values ; bodyTemplateValues (optional): values for interpolation ), ? 4 => severity, ; severity (recommended) ? 5 => labels, ; labels (optional): user-defined key-value pairs for additional context ? 6 => uint, ; deviceUptimeMs (optional): device uptime in milliseconds in range [0, 2^63 - 1] ? 7 => uint, ; deviceTimestampMs (optional): device timestamp in milliseconds in range [0, 2^63 - 1] ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } labels = {* (tstr => (tstr / int / float / bool))} ; Strongly typed key-value pairs ; Integer severity values debug-severity = 30 info-severity = 40 warning-severity = 50 error-severity = 60 critical-severity = 70 severity = (debug-severity / info-severity / warning-severity / error-severity / critical-severity) ; A strongly typed array of values, or an array of byte string representations (big-endian) of the values body-template-values = [ * (tstr / int / float / bool / null) ] / [ * bstr ] ``` ## Device ID [#device-id] Ideally, each physical device should use its own unique device ID for log ingestion (possibly even its own [Ingest Key](/fundamentals/device-authorization)). However, we know that mistakes in device provisioning and configuration can happen so our transmission protocol is robust to multiple devices using the same device ID at the same time. In such cases, the devices will still to send logs to Spotflow without disruption. ## At least once delivery [#at-least-once-delivery] Once a log message is ingested by our MQTT broker, we guarantee that the message will be processed and made available for querying. No data loss is tolerated after acceptance. However, depending on the MQTT QoS level used for publishing, some messages might get lost before they are ingested by the broker. {/* ### Latency, processing guarantees, ordering guarantees & deduplication As majority of observability systems, and for good reasons, Spotflow focuses on high throughput and low latency of log ingestion. However, this comes at a cost of weaker delivery guarantees: * **Latency of few seconds:** After a log message is ingested by our MQTT broker, it is made available for querying in the web application within a few seconds. * **At least once delivery:** Once a log message is ingested by our MQTT broker, we guarantee that the message will be processed and made available for querying. No data loss is tolerated after acceptance. However, depending on the MQTT QoS level used for publishing, some messages might get lost before they are ingested by the broker. * **Ordering**: Upon querying, the log messages are returned in an order defined by ingestion timestamp (with millisecond precision) assigned by our MQTT broker. In rare cases of two messages with identical ingestion timestamps from the same device, the causal order might not be preserved. * **Deduplication**: We do not require publishers to assign unique IDs to each log message. Without unique IDs, there is no reliable way how to deduplicate messages without risking data loss. Therefore, messages are currently not deduplicated in any way. ../../../../components/content-snippets/tradeoffs-let-us-know.mdx */} ## Analyze Logs in the Web Application [#analyze-logs-in-the-web-application] Once your device is integrated and sending logs, you can analyze them in the web application. The main entry point is the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter logs by their content, device ID, severity, and other metadata. You can click on individual log messages to see their details, and it is possible to drill down into specific events by matching their metadata. ## Adjust Device's Minimal Log Severity [#adjust-devices-minimal-log-severity] Adjusting device's minimal log severity requires using Spotflow Device Module. You can choose to set the minimum log severity for the logs that the device sends to Spotflow. This can significantly reduce the device's bandwidth usage and can be configured remotely in the Spotflow web application. Please note that logs generated by the `printk()` function, unlike logging macros such as `LOG_INF` , do not have a severity level assigned and are never filtered out. Select your device on the [Devices](https://app.spotflow.io/devices) page, which gives you a comprehensive view of all connected devices. In the device detail, you can view the current minimum log severity reported by the device and set it to a desired value. ## Learn more [#learn-more] # Metrics (/fundamentals/monitoring/metrics) This page walks you through key aspects of Spotflow metrics, from enabling collection on devices to understanding how metrics are aggregated. If you are interested in visualizing and analyzing the metrics, check out the [Dashboards](/fundamentals/monitoring/dashboards) page. ## Getting Started with Device Integration [#getting-started-with-device-integration] [Guide: Gathering Metrics with Zephyr or Nordic nRF Connect SDK](/guides/zephyr/metrics-zephyr) Spotflow offers native metrics integration for devices running Zephyr and Nordic nRF Connect SDK through a lightweight software module. The module can collect system metrics automatically and send them to Spotflow without additional instrumentation. The module also provides functions for registering and reporting custom application metrics. [Guide: Gathering Metrics with ESP-IDF](/guides/esp-idf/metrics-esp-idf) Spotflow offers native metrics integration for devices running ESP-IDF through the Spotflow device module. The module can collect system metrics automatically and send them to Spotflow without additional instrumentation. The module also provides functions for registering and reporting custom application metrics. [Guide: Gathering Metrics with MQTT](/guides/mqtt/metrics-mqtt) For devices running other platforms or when you cannot use Spotflow device module, integration is also possible via standard MQTT interface. Spotflow platform exposes scalable MQTT broker accessible anywhere from the Internet that can be used to ingest metrics. See [Transport Protocol](#transport-protocol) section for details. ## Transport Protocol [#transport-protocol] Spotflow metrics use an optimized transport protocol based on TCP, MQTT and TLS and two serialization formats (CBOR and JSON). ### MQTT over TLS [#mqtt-over-tls] All log transmission uses **MQTT over TLS (MQTTS)** for secure, reliable communication: * **TLS Version**: 1.2 or higher * **Certificate**: [Let's Encrypt ISRG Root X1](https://letsencrypt.org/certificates/#root-cas) (pre-installed on many systems or included in Spotflow SDKs) * **Authentication**: Devices authenticate using [ingest keys](/fundamentals/device-authorization) as MQTT passwords and their unique device IDs as MQTT usernames. ### Serialization format [#serialization-format] There are two protocol flavors for metric ingestion: * **CBOR-based**: recommended for memory and bandwidth efficiency. Used by Spotflow device modules. * **JSON-based**: recommended for simplicity and interoperability, especially for custom integrations over MQTT. When building a custom MQTT integration, JSON metric payloads can be used even if your device-side SDK uses CBOR by default. Publish metric messages to the `ingest-json` topic using the following JSON schema: ```json { "messageType": "METRIC", "metricName": "cpu_utilization_percent", // Optional for 0/1m/1h/1d metrics, present for aggregated metrics "aggregationInterval": "1m", // Optional labels for dimensional metrics "labels": { "interface": "wlan0" }, // Device uptime when metric sample/window was produced "deviceUptimeMs": 123456, // Sequence number within a metric stream "sequenceNumber": 42, // For aggregated metrics: sum over the window // For 0/no aggregation: raw sample value "sum": 318.7, // Optional overflow marker for integer sums "sumTruncated": false, // Present for aggregated metrics "count": 30, "min": 5.2, "max": 18.1, // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` Publish metric messages to the `ingest-cbor` topic using the following CDDL schema: ```CDDL metric-message = { 0 => 5, ; messageType: metric 21 => tstr, ; metricName ? 22 => agg-interval, ; aggregationInterval (0/1/3/4) ? 5 => labels, ; labels map ? 6 => int, ; deviceUptimeMs ? 13 => uint, ; sequenceNumber 24 => metric-value, ; sum (or raw value for no aggregation) ? 25 => bool, ; sumTruncated (optional) ? 26 => uint, ; count (for aggregated metrics) ? 27 => metric-value, ; min (for aggregated metrics) ? 28 => metric-value, ; max (for aggregated metrics) ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } labels = {* (tstr => tstr)} metric-value = int / float ; Aggregation interval enum values used by device module agg-none = 0 ; 0 agg-1min = 1 ; 1m agg-1hour = 3 ; 1h agg-1day = 4 ; 1d agg-interval = (agg-none / agg-1min / agg-1hour / agg-1day) ``` ## Aggregation Model [#aggregation-model] Metrics can be aggregated before transmission to limit bandwidth usage. Aggregation enables a trade-off between transmission frequency and data volume on one side, and data granularity on the other. In practice, aggregation is useful when devices report frequently but you want stable operational trends instead of every raw sample. For example, one-hour aggregation is often enough for fleet health monitoring, while longer windows can be useful for low-bandwidth deployments. No aggregation is typically best for event-like metrics or short-lived diagnostics where every sample matters. * Supported aggregation windows are: * `0`: no aggregation * `1m`: 1 minute * `1h`: 1 hour * `1d`: 1 day A practical pattern is to choose aggregation based on the tradeoff between metric granularity and available bandwidth. ## System Metrics [#system-metrics] System metrics are primarily useful for: * **Operational health monitoring**: track CPU, heap, and stack pressure before failures happen. * **Connectivity diagnostics**: detect unstable network behavior from TX/RX trends and connection state transitions. * **Stability and reliability analysis**: correlate resets with firmware versions, deployment waves, or environmental conditions. * **Capacity planning**: understand how close devices run to resource limits over time. Together, these metrics provide a baseline observability layer even when application-specific metrics are not yet instrumented. Spotflow supports the following system metrics: * Heap Free Bytes (`heap_free_bytes`) * Heap Allocated Bytes (`heap_allocated_bytes`) * CPU Utilization Percent (`cpu_utilization_percent`) * Thread Stack Free Bytes (`thread_stack_free_bytes`) * Thread Stack Used Percent (`thread_stack_used_percent`) * Network TX Bytes (`network_tx_bytes`) * Network RX Bytes (`network_rx_bytes`) * MQTT Connection State (`connection_mqtt_connected`) * Boot Reset Cause (`boot_reset`) Connection state and reset cause are event-based metrics and are reported on events (state change / boot), not by periodic sampling windows. ### Uptime Metric (Heartbeat) [#uptime-metric-heartbeat] In fleet views, uptime trends can help distinguish isolated device instability from broader rollout or infrastructure issues. Uptime is reported as a dedicated heartbeat metric: * Uptime Milliseconds (`uptime_ms`) This metric is used in the Device Uptime section of [Device Vitals](/fundamentals/monitoring/dashboards#device-vitals). Heartbeat is useful as a lightweight liveness signal: * It confirms that the device is still active and reporting. * It helps detect silent outages where no logs or other telemetry are produced. * It provides context for resets by showing uptime progression between reboot events. ## Custom Metrics [#custom-metrics] In addition to built-in system metrics, Spotflow allows you to define and report custom application metrics relevant to your firmware and use case. You can use them to track specific application events, performance indicators, as well as business KPIs. To visualize custom metrics in Spotflow dashboards, you can create custom charts and add them to your device or fleet dashboards. See [Dashboards](/fundamentals/monitoring/dashboards) for more details. Typical use cases include: * **Sensor readings**: temperature, humidity, pressure, or other physical measurements. * **Application counters**: processed messages, completed tasks, retries. * **Latency tracking**: operation duration, response times, with labels for operation type or method. * **Business events**: button presses, user interactions, error occurrences. Custom metrics use the same transport, aggregation, and encoding pipeline as system metrics. The difference is that you register and report them explicitly from your application code, giving you full control over what is measured and when. To define a custom metric, you need to decide on: * **metricName**: a unique identifier for the metric (e.g., `request_count`). * **aggregationInterval**: the desired aggregation window (e.g., 1 hour). * **labels**: optional string key-value pairs to add dimensions to the metric (e.g., `location: east-14`). Custom metrics support labels for dimensional breakdowns. Each unique combination of label values creates a separate time series with independent aggregation state. For example, a smart lock operation duration metric with `operation` and `method` labels tracks each combination (e.g. unlock/nfc, lock/keypad) separately. When an aggregation window is configured, reported values are accumulated and transmitted as a single aggregated message containing the following statistical values: * **sum**: sum of all sample values in the window. * **count**: number of samples in the window. * **min**: minimum sample value in the window. * **max**: maximum sample value in the window. When using the Spotflow device SDK (e.g. Zephyr or ESP-IDF), these values are computed automatically — you just report raw values and the SDK handles the rest. When using the [MQTT integration](/guides/mqtt/metrics-mqtt) directly, you construct the aggregated payload yourself. If no aggregation is needed, set the aggregation window to `0` and only the `sum` field is required (it represents the raw sample value). System metrics are visualized in the built-in [Device Dashboard](/fundamentals/monitoring/dashboards#device-dashboard). Custom metrics can be visualized in [Custom Dashboards](/fundamentals/monitoring/dashboards#custom-dashboards--product-analytics). For detailed guides on how to define and report custom metrics, see the [Metrics with Zephyr](/guides/zephyr/metrics-zephyr), [Metrics with ESP-IDF](/guides/esp-idf/metrics-esp-idf), and [Metrics with MQTT](/guides/mqtt/metrics-mqtt) guides. ### Product analytics [#product-analytics] While technical metrics are essential for monitoring device health and performance, custom application metrics can also be used to track product usage and user behavior. See [Dashboards / Product Analytics](/fundamentals/monitoring/dashboards#custom-dashboards--product-analytics) for more details. ## Reference Repository Materials [#reference-repository-materials] * [Metrics sample (Zephyr)](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/samples/metrics) * [Metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/metrics/Kconfig) * [System metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/metrics/system/Kconfig) * [Metrics sample (ESP-IDF)](https://github.com/spotflow-io/device-sdk/tree/main/esp_idf/spotflow/device_sdk/examples/metrics) * [ESP-IDF metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/esp_idf/spotflow/device_sdk/Kconfig.metrics) * [ESP-IDF system metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/esp_idf/spotflow/device_sdk/Kconfig.metrics_systems) ## Learn more [#learn-more] # Over-the-air (OTA) updates (/fundamentals/ota) Over-the-air (OTA) updates allow you to remotely update firmware without physical access to the device. ## Device OTA Integration [#device-ota-integration] Spotflow allows you to manage OTA updates for your embedded devices. [Guide: Over-the-air updates with Zephyr](/guides/zephyr/ota-zephyr) [Guide: Over-the-air updates of external MCUs with Zephyr](/guides/zephyr/ota-external-mcu-zephyr) The Spotflow device module can automatically perform OTA updates of your main firmware on Zephyr devices. It also allows you to handle OTA updates of external MCUs connected to your Zephyr device. [Guide: Over-the-air updates with MQTT](/guides/mqtt/ota-mqtt) For devices running other platforms or when you cannot use the Spotflow device module, integration of OTA updates is also possible via the standard MQTT interface. ## Firmwares & Firmware Versions [#firmwares--firmware-versions] [Firmwares are tracked in Spotflow](/fundamentals/firmware-management), and each firmware can have multiple versions. A firmware version represents a specific build that can be deployed to devices. ### Firmware Image [#firmware-image] Each firmware version is associated with a firmware image file, such as a `.bin` file, that contains the binary flashed to devices. ## Deployment Cohort [#deployment-cohort] Deployment cohorts allow you to organize your devices into logical groups for targeted OTA updates. You can assign devices to deployment cohorts by selecting them manually or by choosing devices based on their tags. This enables you to roll out firmware updates to specific subsets of your device fleet. ### Device Tags [#device-tags] Device tags are key-value pairs that can be assigned to devices to categorize and filter them and to move them efficiently between deployment cohorts. For example, you could tag devices with their location, model, or hardware version. This allows you to create deployment cohorts based on these tags and target specific devices for OTA updates. ### Deployments [#deployments] A deployment represents the process of rolling out a specific firmware version to a deployment cohort. When you create a deployment, you select the firmware version to be deployed and the target deployment cohort. #### Main vs Secondary Firmware [#main-vs-secondary-firmware] A deployment can include multiple firmware packages. The firmware that is running the Spotflow module should be marked as main. The update of this firmware is handled automatically by the Spotflow device SDK. You can register custom callbacks within the main MCU firmware to handle the updates of other firmware packages. This setup allows you, for example, to update the firmware of a secondary MCUs like a Wi-Fi module. #### Progress Tracking [#progress-tracking] You can monitor deployment progress, see which devices updated successfully, and identify any devices that encountered issues during the update process. If a deployment is active for a cohort, it is propagated to devices that are already in the cohort and also to devices added to the cohort later. Each device in a deployment goes through the following states: * **Pending** — The device is queued and waiting to receive the update. * **In Progress** — The update manifest was sent to the device and it is installing the update. * **Succeeded** — The device successfully completed the update. * **Failed** — The device encountered an error during the update. * **Cancelled** — The update was cancelled before the device completed it. ## Learn more [#learn-more] # Crash reports with ESP-IDF (/guides/esp-idf/crash-reports-esp-idf) This guide explains how to collect ESP-IDF core dumps using the Spotflow device module and analyze crash reports in the Spotflow web application. ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with ESP-IDF 5.0+ installed, * have established a connection to the internet from the device, * have a Spotflow ingest key. Alternatively, you can follow the [Quickstart: ESP-IDF Integration Guide](/quickstart/esp-idf) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a dependency to your `idf_component.yml` file. ```yaml title="idf_component.yml" dependencies: spotflow: git: https://github.com/spotflow-io/device-sdk.git path: esp_idf/spotflow version: main ``` ## Update Configuration [#update-configuration] Enable Spotflow coredump collection, configure device authentication, and tell ESP-IDF to use a custom partition table: ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_DEVICE_ID="esp-device-001" # Set unique identifier of your device CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" # Set your Spotflow ingest key # Enable Spotflow coredump collection CONFIG_SPOTFLOW_COREDUMP_BACKEND=y # Include a build identifier so Spotflow can link crashes to uploaded symbols CONFIG_SPOTFLOW_USE_BUILD_ID=y # Use a partition table with a coredump partition CONFIG_PARTITION_TABLE_CUSTOM=y CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv" CONFIG_PARTITION_TABLE_FILENAME="partitions.csv" # Recommended: avoid slowing down coredump writes and triggering watchdogs CONFIG_ESP_COREDUMP_LOGS=n ``` The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys on the [ingest keys](https://app.spotflow.io/ingest-keys) page. See details about the [Spotflow-specific options](#kconfig-options) later in this doc. ## Define Core Dump Flash Partition [#define-core-dump-flash-partition] ESP-IDF stores the core dump in a dedicated flash partition before Spotflow uploads it after reboot. Add a `coredump` data partition to your custom partition table: ```csv title="partitions.csv" # Name, Type, SubType, Offset, Size nvs, data, nvs, 0x9000, 0x6000 phy_init, data, phy, 0xf000, 0x1000 factory, app, factory, 0x10000, 0x150000 coredump, data, coredump,, 256K ``` If [Flash Encryption](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/security/flash-encryption.html) is enabled on the device, mark the core dump partition as encrypted: ```csv title="partitions.csv" coredump, data, coredump,, 256K, encrypted ``` ESP-IDF cannot read encrypted core dump partitions directly from flash using `idf.py coredump-info` or `idf.py coredump-debug` . The Spotflow module reads the core dump on the device after reboot, so ESP-IDF decrypts the partition before the data is uploaded. The exact partition layout depends on your firmware size, OTA strategy, target chip, and flash size. Keep the `coredump` partition large enough for the ESP-IDF core dump format you configure. ## Initialize Spotflow [#initialize-spotflow] Include `spotflow.h` and call `spotflow_init()` after your network connection is established. On startup, `spotflow_init()` starts the Spotflow MQTT client and checks whether an ESP-IDF core dump is available for upload. ```c title="main/app_main.c" #include "spotflow.h" void app_main(void) { // Initialize NVS, networking, Wi-Fi, and connect to the internet first. spotflow_init(); } ``` ## Upload ELF File with Symbols [#upload-elf-file-with-symbols] To unlock advanced crash analysis features, upload the ELF file containing debug symbols to Spotflow. The Spotflow ESP-IDF module can include the ESP-IDF application ELF SHA-256 value as the build ID in the uploaded core dump metadata. Spotflow uses that build ID to link the crash report with the matching firmware symbols. See dedicated [Firmware Management](/fundamentals/firmware-management) page for information about managing firmwares and symbol files. ## Wait for or Simulate a Crash [#wait-for-or-simulate-a-crash] ESP-IDF creates the core dump when a fatal error occurs. After the device reboots and reconnects, Spotflow uploads the stored core dump. To simulate a crash, call `esp_system_abort()` from task context: ```c title="main/app_main.c" #include "esp_system.h" static void simulate_crash(void) { esp_system_abort("Deliberate crash for Spotflow coredump test"); } ``` The SDK also includes a complete [coredump example](https://github.com/spotflow-io/device-sdk/tree/main/esp_idf/spotflow/device_sdk/examples/coredump) that triggers a crash from the board's BOOT button. When testing the coredump example, stop OpenOCD/JTAG debugging and connect to the board through the normal USB serial port. An active debugger can change panic, reset, and reconnect behavior and may prevent the normal crash/coredump flow from being observed. Press BOOT only after the application has booted and connected; do not press RESET/EN. On ESP32-C6 boards, the example expects the BOOT button on GPIO9. If your board uses a different button GPIO, update the example's `GPIO_INPUT_IO_0` definition. To create a project from the example, run: ```bash title=">_ ESP-IDF Terminal" idf.py create-project-from-example "spotflow/device_sdk:coredump" ``` ## Analyze Crash Reports in the Web Application [#analyze-crash-reports-in-the-web-application] Once your device is integrated, you can analyze ESP-IDF crash reports and core dumps in the web application. You can list crash reports on the [Events](https://app.spotflow.io/) page, filter them by device ID and metadata, and open the detail view for each crash report. ### Automatic Analysis [#automatic-analysis] Spotflow automatically processes ESP-IDF core dumps and, when a matching ELF file with symbols is available, links the crash report with the uploaded symbols. The crash report detail can show extracted debugging data. This includes: * Stack traces for one or more threads. * Register values for individual stack frames, including type casting where available. * Local variables for individual stack frames. * Global variables captured at the time of the crash. We also provide raw data extracted from the core dump, so you can dive deeper into the issue if needed. AI root-cause analysis is not currently available for ESP-IDF crash reports. ## How the Device Module Works [#how-the-device-module-works] The Spotflow device module integrates with ESP-IDF's native [core dump functionality](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/core_dump.html): * ESP-IDF writes the core dump to the `coredump` flash partition after a fatal error. * On the next boot, `spotflow_init()` checks whether a core dump is present. * If a core dump is found, the module reads its address and size using ESP-IDF APIs. * The module reads the core dump from flash in chunks controlled by `CONFIG_SPOTFLOW_COREDUMPS_CHUNK_SIZE`. * Each chunk is encoded as CBOR and published to Spotflow over MQTT on the `ingest-cbor` topic. * Core dump chunks use MQTT QoS 1 and are prioritized over regular log messages. * After the module has processed the core dump for upload, it erases the stored core dump image. Core dump chunks are sent before buffered logs so crash data is uploaded first after a reboot. ## Build ID and Symbol Linking [#build-id-and-symbol-linking] To provide the best possible crash analysis experience, Spotflow needs debugging symbols associated with the code that produced the crash. When `CONFIG_SPOTFLOW_USE_BUILD_ID=y`, the ESP-IDF module includes a build ID with the uploaded core dump metadata. For ESP-IDF, the build ID is derived from the running application descriptor's `app_elf_sha256` value. Spotflow uses it to link the uploaded core dump with the matching ELF file uploaded through [Firmware Management](/fundamentals/firmware-management). If the build ID is not available, ELF files can still be linked manually to specific core dumps. ## Core Dumps vs. Logging [#core-dumps-vs-logging] Spotflow core dumps and [logging](/guides/esp-idf/logging-esp-idf) are complementary features. After reboot, core dump upload has priority over buffered log messages so the crash report can be reconstructed as soon as possible. For local ESP-IDF coredump storage, keep `CONFIG_ESP_COREDUMP_LOGS=n` unless you have verified that logging during coredump generation does not trigger watchdog timeouts on your target. ## Kconfig Options [#kconfig-options] ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_COREDUMP_BACKEND=y ``` Set `SPOTFLOW_COREDUMP_BACKEND` to `y` to enable Spotflow coredump collection. This also selects ESP-IDF's ELF coredump data format and configures coredumps to be stored in flash. ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_COREDUMPS_CHUNK_SIZE=2048 ``` Use `SPOTFLOW_COREDUMPS_CHUNK_SIZE` to set the size of each core dump chunk uploaded to Spotflow. Larger chunks reduce upload overhead but require more RAM for preparing and sending each MQTT message. ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_MAX_QUEUE_SIZE=2 ``` Use `SPOTFLOW_MAX_QUEUE_SIZE` to set how many core dump chunks can be queued before sending. Approximate queue memory usage is `CONFIG_SPOTFLOW_MAX_QUEUE_SIZE * CONFIG_SPOTFLOW_COREDUMPS_CHUNK_SIZE`, plus encoding overhead. ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_USE_BUILD_ID=y ``` Set `SPOTFLOW_USE_BUILD_ID` to `y` to include the ESP-IDF application build ID in the first uploaded core dump chunk. This enables automatic linking between crash reports and uploaded ELF symbols. ```dotenv title="sdkconfig.defaults" CONFIG_ESP_COREDUMP_LOGS=n ``` We recommend disabling ESP-IDF coredump logs because they can increase the time needed to store the core dump to flash. On some targets, that can trigger the watchdog timer and make the core dump unusable. See [KConfig for core dumps](https://github.com/spotflow-io/device-sdk/blob/main/esp_idf/spotflow/device_sdk/Kconfig.coredump) for details about the Spotflow ESP-IDF coredump options. ## Learn More [#learn-more] # Logging with ESP-IDF (/guides/esp-idf/logging-esp-idf) This guide explains how to enable log collection on devices running ESP-IDF using the Spotflow device module and analyze logs in the Spotflow web application. ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with ESP-IDF 5.0+ installed, * have established a connection to the internet from the device. Alternatively, you can follow the [Quickstart: ESP-IDF Integration Guide](/quickstart/esp-idf) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a dependency to your `idf_component.yml` file. ```yaml title="idf_component.yml" dependencies: spotflow: git: https://github.com/spotflow-io/device-sdk.git path: esp_idf/spotflow version: main ``` ## Update Configuration [#update-configuration] To enable Spotflow logging, you need to configure the Spotflow module first. Add the following lines to your `sdkconfig.defaults` file: ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_DEVICE_ID="esp-device-001" # Set unique identifier of your device CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" # Set your Spotflow ingest key ``` The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys on the [ingest keys](https://app.spotflow.io/ingest-keys) page. ## Use ESP-IDF Logging Library [#use-esp-idf-logging-library] After installing and initializing the Spotflow device module, you can use the standard ESP-IDF logging macros to send the logs: ```c title="src/main.c" #include "esp_log.h" #include "spotflow.h" static const char *TAG = "main_module"; void main(void) { // Initialize the Spotflow logging module spotflow_init(); // Ensure the device has an active internet connection. ESP_LOGI(TAG, "Device has booted up successfully."); for (int i = 1; i <= 5; i++) { int reading = i * 10; ESP_LOGI(TAG, "Sensor reading %d: %d units", i, reading); if (reading > 30) ESP_LOGW(TAG, "Reading is above normal. Value: %d.", reading); } ESP_LOGI(TAG, "Device shutting down."); } ``` ## Analyze Logs in the Web Application [#analyze-logs-in-the-web-application] Once your device is integrated and sending logs, you can analyze them in the web application. The main entry point is the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter logs by their content, device ID, severity, and other metadata. You can click on individual log messages to see their details, and it is possible to drill down into specific events by matching their metadata. ## How the Device Module Works [#how-the-device-module-works] The Spotflow device module integrates with ESP-IDF's native [logging library](https://docs.espressif.com/projects/esp-idf/en/v4.2.4/esp32/api-reference/system/log.html) to provide a familiar development experience. Instead of replacing your existing logging workflow, it extends it by implementing a custom `vprintf`-like logging handler that captures logs from standard ESP-IDF macros and transmits them to Spotflow. The original logging handler is still called, so the previous logging behavior is preserved. **Key benefits of this architecture:** * **Zero code changes**: Continue using `ESP_LOGV`, `ESP_LOGD()`, `ESP_LOGI()`, `ESP_LOGW()`, and `ESP_LOGE()` macros as before. * **Local debugging**: Our device module can run alongside other logging handlers, allowing you to see logs in your local console while also transmitting them to Spotflow. ## Handling Network Interruptions [#handling-network-interruptions] The device module is designed to handle network interruptions gracefully. When the connection is unavailable, **logs are stored in a circular buffer**. That is, when the buffer is full, the oldest logs are overwritten with the new ones. Once the connection is restored, buffered logs are automatically sent to Spotflow. ## Support for Constrained Devices [#support-for-constrained-devices] To minimize the memory footprint, the device module exposes number of configuration options to tune the buffer size and other parameters. See the available [Kconfig options](#kconfig-options) below. ## Reliability of data delivery [#reliability-of-data-delivery] Spotflow is using MQTT as the underlying [transport protocol](/fundamentals/monitoring/logging#transport-protocol) for log ingestion. Most embedded devices running ESP-IDF are significantly resource-constrained which has several implications: 1. To minimize the footprint of the Spotflow device module, **MQTT QoS level 0** is used which has minimal overhead but at the cost of weaker delivery guarantees. Although MQTT itself is TCP-based protocol and TCP guarantees transport-level delivery, with QoS 0, some messages still can be lost in case of client or broker failures and similar issues. This should not be an issue for most observability use-cases. 2. In case of a network interruption or slow down, the device module buffers not yet sent logs in a circular buffer. However, this buffer has limited size and once it is full, the oldest logs are overwritten with the new ones. Size of this buffer can be via [Kconfig options](#kconfig-options). If needed, you can create your [custom integration](/guides/mqtt/logging-mqtt) directly with our MQTT broker to tailor the solution to your specific resource constraints vs. reliability tradeoff. Spotflow MQTT broker supports all QoS levels from 0 to 2 and is designed to handle even high throughput log streams. ## Kconfig options [#kconfig-options] ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_LOG_BACKEND=y ``` To explicitly enable or disable Spotflow logging backend use `SPOTFLOW_LOG_BACKEND`. It is enabled by default. ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_DEBUG_MESSAGE_TERMINAL=y ``` To enable debug messages from the Spotflow device module, enable `SPOTFLOW_DEBUG_MESSAGE_TERMINAL` option. It is disabled by default. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_MESSAGE_QUEUE_SIZE=5 CONFIG_SPOTFLOW_LOG_BUFFER_SIZE=512 CONFIG_SPOTFLOW_CBOR_LOG_MAX_LEN=1024 CONFIG_MQTT_BUFFER_SIZE=1024 ``` You can configure these options to adjust performance and resource usage. See [KConfig for logging](https://github.com/spotflow-io/device-sdk/blob/main/esp_idf/spotflow/device_sdk/Kconfig.logging) for details about all options above. ## Learn more [#learn-more] # Metrics with ESP-IDF (/guides/esp-idf/metrics-esp-idf) This guide explains how to gather metrics from devices running ESP-IDF using the Spotflow device module. You can collect system metrics automatically and report custom application metrics using the same SDK. ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have a development environment with ESP-IDF 5.0+ installed, * have a device configuration that can connect to the internet, * have a Spotflow ingest key. Alternatively, follow the [Quickstart: ESP-IDF Integration Guide](/quickstart/esp-idf) to integrate Spotflow using a sample application. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a dependency to your `idf_component.yml` file. ```yaml title="idf_component.yml" dependencies: spotflow: git: https://github.com/spotflow-io/device-sdk.git path: esp_idf/spotflow version: main ``` ## Enable metrics in Kconfig [#enable-metrics-in-kconfig] Configure the Spotflow device identity and enable metrics in `sdkconfig.defaults`: ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_DEVICE_ID="esp-device-001" # Set unique identifier of your device CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" # Set your Spotflow ingest key # Enable Spotflow metrics collection CONFIG_SPOTFLOW_METRICS=y ``` The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys on the [ingest keys](https://app.spotflow.io/ingest-keys) page. ## Initialize Spotflow [#initialize-spotflow] Include `spotflow.h` and call `spotflow_init()` during application startup. When metrics are enabled, `spotflow_init()` initializes the metrics registry, metrics transport, heartbeat, system metrics, and the Spotflow MQTT client. Metrics can be reported before MQTT is connected. The SDK queues encoded metric messages locally and sends them once the MQTT connection is available. ```c title="main/app_main.c" #include "spotflow.h" void app_main(void) { // Initialize required platform services such as NVS and networking setup. spotflow_init(); } ``` ## Enable system metrics auto-collection [#enable-system-metrics-auto-collection] To collect system telemetry automatically, enable: ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_METRICS_SYSTEM=y ``` The following system metrics are collected: | Metric | How it is collected | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Heap Free Bytes (`heap_free_bytes`) | Sampled with `heap_caps_get_free_size(MALLOC_CAP_DEFAULT)`. | | Heap Allocated Bytes (`heap_allocated_bytes`) | Derived from `heap_caps_get_total_size(MALLOC_CAP_DEFAULT)` minus free heap. | | CPU Utilization Percent (`cpu_utilization_percent`) | Sampled from FreeRTOS idle runtime counters over a one-second window. | | Thread Stack Free Bytes (`thread_stack_free_bytes`) | Sampled per tracked FreeRTOS task. | | Thread Stack Used Percent (`thread_stack_used_percent`) | Derived per tracked FreeRTOS task from stack bounds and current stack pointer. | | Network TX Bytes (`network_tx_bytes`) | Counted per active lwIP network interface by wrapping `linkoutput`. | | Network RX Bytes (`network_rx_bytes`) | Counted per active lwIP network interface by wrapping `input`. | | MQTT Connection State (`connection_mqtt_connected`) | Event-driven and reported when MQTT state changes through `spotflow_metrics_system_report_connection_state(bool)`. | | Boot Reset Cause (`boot_reset`) | Reported once on boot using `esp_reset_reason()`. | | Uptime Milliseconds (`uptime_ms`) | Reported by the metrics heartbeat timer. | By default, stack metrics are collected for all FreeRTOS tasks (`CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_ALL_THREADS=y`). If you need to limit this, disable that option and register only selected tasks with `spotflow_metrics_system_enable_thread_stack(...)`. Check implementation details in the [SDK ESP-IDF system metrics folder](https://github.com/spotflow-io/device-sdk/tree/main/esp_idf/spotflow/device_sdk/src/metrics/system). ## Analyze system metrics in dashboards [#analyze-system-metrics-in-dashboards] After enabling system metrics, open the Device Dashboard to inspect device vitals, resource usage, and connectivity trends: * [Device Dashboard](/fundamentals/monitoring/dashboards#device-dashboard) ## Report custom metrics [#report-custom-metrics] Custom metrics let you track application-specific values such as sensor readings, request latencies, or business counters. The SDK handles aggregation, encoding, and transmission to Spotflow automatically. `CONFIG_SPOTFLOW_METRICS_SYSTEM=y` is not required for custom metrics. You can use custom metrics independently or alongside system metrics. ### Include the metrics header [#include-the-metrics-header] Include the metrics API header in your application source file: ```c title="main/app_main.c" #include "metrics/spotflow_metrics_backend.h" ``` This header provides functions for registering and reporting metrics. It transitively includes the type definitions and registration API. ### Register metrics [#register-metrics] Before reporting values, each metric must be registered with a name and an aggregation interval. Registration returns a handle that is used for all subsequent reports. Metrics can be either **integer** (`int64_t`) or **float** (`float`), and optionally support **labels** for dimensional breakdowns. #### Label-less metrics [#label-less-metrics] A label-less metric tracks a single time series: ```c title="main/app_main.c" static struct spotflow_metric_int *counter_metric; static struct spotflow_metric_float *temperature_metric; int rc; /* Integer metric aggregated over 1 minute */ rc = spotflow_register_metric_int( "app_counter", SPOTFLOW_AGG_INTERVAL_1MIN, &counter_metric); /* Float metric with no aggregation (each value sent immediately) */ rc = spotflow_register_metric_float( "temperature_celsius", SPOTFLOW_AGG_INTERVAL_NONE, &temperature_metric); ``` #### Labeled metrics [#labeled-metrics] A labeled metric tracks multiple time series distinguished by label key-value pairs. At registration, specify the maximum number of unique label combinations (`max_timeseries`) and the maximum number of labels per report (`max_labels`): ```c title="main/app_main.c" static struct spotflow_metric_float *request_duration_metric; rc = spotflow_register_metric_float_with_labels( "http_request_duration_ms", SPOTFLOW_AGG_INTERVAL_1MIN, 18, /* max_timeseries: e.g. 3 endpoints x 2 methods x 3 statuses */ 3, /* max_labels */ &request_duration_metric); ``` Each unique combination of label values is tracked as a separate time series with its own aggregation state. #### Aggregation intervals [#aggregation-intervals] The aggregation interval controls how long values are accumulated before being sent: | Constant | Interval | Behavior | | ----------------------------- | -------------- | ----------------------------------------------- | | `SPOTFLOW_AGG_INTERVAL_NONE` | No aggregation | Each reported value is sent immediately | | `SPOTFLOW_AGG_INTERVAL_1MIN` | 1 minute | Values are aggregated into sum, count, min, max | | `SPOTFLOW_AGG_INTERVAL_1HOUR` | 1 hour | Values are aggregated into sum, count, min, max | | `SPOTFLOW_AGG_INTERVAL_1DAY` | 1 day | Values are aggregated into sum, count, min, max | Metric names are normalized before registration: alphanumeric characters are lowercased, dashes, dots, and spaces are converted to underscores, and other characters are removed. For example, `"My-Metric.Name"` becomes `"my_metric_name"`. ### Report metric values [#report-metric-values] After registration, report values from your application tasks. #### Label-less values [#label-less-values] ```c title="main/app_main.c" /* Report an integer value */ spotflow_report_metric_int(counter_metric, 42); /* Report a float value */ spotflow_report_metric_float(temperature_metric, 23.5f); ``` #### Labeled values [#labeled-values] Attach labels to each report using an array of `struct spotflow_label`: ```c title="main/app_main.c" struct spotflow_label labels[] = { { .key = "endpoint", .value = "/api/users" }, { .key = "method", .value = "GET" }, { .key = "status", .value = "200" } }; spotflow_report_metric_float_with_labels( request_duration_metric, 120.5f, labels, 3); ``` Label keys are stored in 16-byte buffers and values in 32-byte buffers, including the null terminator. Longer keys or values are truncated by the SDK. #### Events [#events] For point-in-time occurrences where you only need to record that something happened, use the event API. An event is equivalent to reporting an integer value of `1` with no aggregation: ```c title="main/app_main.c" static struct spotflow_metric_int *button_pressed_metric; /* Register with no aggregation */ spotflow_register_metric_int( "button_pressed", SPOTFLOW_AGG_INTERVAL_NONE, &button_pressed_metric); /* Report that the event occurred */ spotflow_report_event(button_pressed_metric); ``` Events also support labels: ```c title="main/app_main.c" static struct spotflow_metric_int *app_error_metric; spotflow_register_metric_int_with_labels( "app_error", SPOTFLOW_AGG_INTERVAL_NONE, 10, /* max_timeseries */ 1, /* max_labels */ &app_error_metric); struct spotflow_label error_labels[] = { { .key = "code", .value = "timeout" } }; spotflow_report_event_with_labels(app_error_metric, error_labels, 1); ``` Custom metrics can be visualized in [Custom Dashboards](/using-spotflow/guides/custom-dashboards). ## Analyze custom metrics in dashboards [#analyze-custom-metrics-in-dashboards] After reporting custom metrics, open Custom Dashboards to visualize and analyze your application-specific data: * [Custom Dashboards](/using-spotflow/guides/custom-dashboards) ## Try the metrics example [#try-the-metrics-example] The SDK includes a complete ESP-IDF metrics example that enables system metrics and reports custom application metrics. To create a project from the example, run: ```bash title=">_ ESP-IDF Terminal" idf.py create-project-from-example "spotflow/device_sdk:metrics" ``` Configure Wi-Fi and Spotflow credentials in `sdkconfig.defaults` or through `idf.py menuconfig`, then build and flash the project. ## Tune collection and aggregation [#tune-collection-and-aggregation] ### System metrics [#system-metrics] Use these options to control sampling cadence and the aggregation window before upload: ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=10 CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL=60 ``` * `CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL` defines how often system metrics are sampled, in seconds. * `CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL` defines how long samples are aggregated before sending. * Aggregation interval supports `0` (no aggregation), `60` (1 minute), `3600` (1 hour), and `86400` (1 day). * With aggregation interval `0`, each sample is sent immediately. * Lower collection interval gives finer time resolution but increases local processing overhead. * Higher aggregation interval reduces message frequency and network traffic by combining more samples. ### Custom metrics [#custom-metrics] The following Kconfig options control buffering, encoding, and metric limits: ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_METRICS_QUEUE_SIZE=16 CONFIG_SPOTFLOW_METRICS_CBOR_BUFFER_SIZE=512 CONFIG_SPOTFLOW_METRICS_MAX_REGISTERED=32 CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC=4 ``` * `CONFIG_SPOTFLOW_METRICS_QUEUE_SIZE` controls how many encoded messages are buffered before MQTT transmission. The default is `16`, or `64` when system metrics are also enabled. * `CONFIG_SPOTFLOW_METRICS_CBOR_BUFFER_SIZE` sets the buffer size used for CBOR encoding. * `CONFIG_SPOTFLOW_METRICS_MAX_REGISTERED` limits how many metrics can be registered simultaneously. System and custom metrics share this pool. * `CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC` limits how many label key-value pairs can be attached to a single report. When MQTT is not connected, metric messages remain in the metrics queue until they can be sent. If the queue becomes full, the SDK drops the oldest queued metric and enqueues the newest one. Tune `CONFIG_SPOTFLOW_METRICS_QUEUE_SIZE` for expected offline periods and reporting frequency. ### Heartbeat [#heartbeat] Heartbeat is enabled by default when metrics are enabled. It reports `uptime_ms` periodically and has higher priority than regular metrics. ```dotenv title="sdkconfig.defaults" CONFIG_SPOTFLOW_METRICS_HEARTBEAT=y CONFIG_SPOTFLOW_METRICS_HEARTBEAT_INTERVAL=60 ``` ## Reference Repository Materials [#reference-repository-materials] * [Metrics sample (ESP-IDF)](https://github.com/spotflow-io/device-sdk/tree/main/esp_idf/spotflow/device_sdk/examples/metrics) * [Metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/esp_idf/spotflow/device_sdk/Kconfig.metrics) * [System metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/esp_idf/spotflow/device_sdk/Kconfig.metrics_systems) ## Learn More [#learn-more] # Crash reports with MQTT (/guides/mqtt/crash-reports-mqtt) This guide explains how to send crash reports and core dumps from devices running running a MQTT client and analyze crashes and core dumps in the Spotflow web application. ## Connect to the Spotflow MQTT broker [#connect-to-the-spotflow-mqtt-broker] As the very first step, check the following basic integration guides: * [Quickstart: Connect via MQTT](/quickstart/mqtt) ## Publish core dump in JSON or CBOR format [#publish-core-dump-in-json-or-cbor-format] Core dump first needs to be split into chunks (of up to 900 KiB each) and then each chunk can be sent as a separate MQTT message. Order in which the messages are sent is not critical as long as all chunks have proper **ordinal** and the **last chunk is marked** accordingly. See more general information about core dump formats on the [Fundamentals: Crash reports & core dumps](/fundamentals/monitoring/crash-reports) page. All chunks belonging must have the same **core dump ID** set which is a non-negative integer with range between `0` and `2^63-1` (`LONG_LONG_MAX`). This ID needs to be unique in the scope of a one device (device ID) and last 7 days, not globally. This implies that senders/devices can generate random integers in a stateless manner for this purposes. To enable automatic core dump analysis, specify the **operating system** in the core dump chunk. The **build ID** can be sent either in the core dump chunk or earlier in [Session Metadata](/guides/mqtt/session-metadata) for the same direct MQTT connection. If both are provided, the chunk-level build ID is used. For gateway or relay scenarios, include the build ID in the core dump chunk for the source device. To send core dump chunks in JSON, simply publish MQTT messages into the `ingest-json` topic. The messages should follow the schema below: ```json { // Must be set to CORE_DUMP_CHUNK. "messageType": "CORE_DUMP_CHUNK", // Identifier of the core dump unique within a scope of device and last 7-days. "coreDumpId": 123, // Zero-based index of the chunk in the sequence. "chunkOrdinal": 1, // Base64-encoded chunk data. "content": "WkUCAAMABQADAAAAQQIARAADAAAAAAAAAElTKgAAAAAAuMIFE...", // (Optional) Flag indicating if this is the last chunk. "isLastChunk": false, // (Optional) Identifier of the ELF file build for linking with symbols. "buildId": "build-123", // (Optional) Operating system indicator, currently only "Zephyr" or empty one is supported. "os": "Zephyr", // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` **Tip for Testing:** use a command-line MQTT client to test messages without writing code. Note, that the `publish` command within these CLI clients typically handles the entire sequence for you: it connects, sends the message, and disconnects. See examples in [Quickstart](/quickstart/mqtt#publish-messages). To send core dump chunks serialized in CBOR, simply publish MQTT messages into the `ingest-cbor` topic. The payload should include a chunk following the `core-dump-chunk` schema as defined below: ```CDDL core-dump-chunk = { 0 => 2, ; messageType: must be set to 2 = "CORE_DUMP_CHUNK". 9 => uint, ; coreDumpId: identifier of the core dump unique within a scope of device and last 7-days. 10 => uint, ; chunkOrdinal: zero-based index of the chunk in the sequence. 11 => bstr, ; content: chunk data. ? 12 => bool, ; isLastChunk (optional): flag indicating if this is the last chunk. ? 14 => tstr, ; buildId (optional): identifier of the ELF file build for linking with symbols. ? 15 => tstr, ; os (optional): operating system indicator, currently only "Zephyr" or empty one is supported. ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } ``` ## Analyze Crash Reports in the Web Application [#analyze-crash-reports-in-the-web-application] Once your device is integrated, you can analyze crash reports and core dumps in the web application. You can list the crash reports in the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter crash reports by their content, device ID and other metadata. ### Automatic Analysis [#automatic-analysis] We automatically analyse each crash report to give you a quick insight into the issue. For precise results, we extract data directly from the core dump (stack traces, register values, etc.), decompile the firmware binary, and review relevant documentation. Our proprietary AI agent then investigates the crash, leveraging all available data. The result is available in the detail view of each crash report. The full analysis includes a detailed description of the root cause and suggestions for fixing the issue. Also, for complete transparency, we include references to all documentation pages used, allowing you to explore further. We also provide all the raw data extracted from the core dump, so you can dive deeper into the issue if needed. This includes: * Stack traces of one or more threads. * Register values and local variables for individual stack frames. If a register value is available, it can be also casted to several types. * State of global variables at the time of the crash. ## Learn more [#learn-more] # Logging with MQTT (/guides/mqtt/logging-mqtt) This guide explains how to enable log collection on devices running a MQTT client and analyze logs in the Spotflow web application. ## Connect to the Spotflow MQTT broker [#connect-to-the-spotflow-mqtt-broker] As the very first step, check the following basic integration guide: * [Quickstart: Connect via MQTT](/quickstart/mqtt) ## Publish logs in JSON or CBOR format [#publish-logs-in-json-or-cbor-format] Once your client is connected, you can start sending logs. You can choose to send logs in JSON or CBOR format. Prefer CBOR for efficiency in bandwidth-constrained environments. Use JSON for convenience when slightly larger messages are acceptable. To send logs serialized in JSON, simply publish MQTT messages into the `ingest-json` topic. The messages should follow the schema below: ```json { // Fully interpolated log line string (optional when bodyTemplate is used) "body": "SmartLock was Unlocked", // (Optional) printf-like interpolation string for the log line "bodyTemplate": "SmartLock was %s", // (Optional) array of values for interpolation in the bodyTemplate "bodyTemplateValues": ["Unlocked"], // (Recommended) Log severity, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" "severity": "INFO", // (Optional) device uptime when the log was generated (milliseconds since the device booted) "deviceUptimeMs": 23455, // (Optional) time when the log was generated (milliseconds since the UNIX epoch) "deviceTimestampMs": 1748530133808, // (Optional) you can add extra metadata to your logs "labels": { "initiatorKind": "MobileApp", "userId": "1234567890" }, // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` **Tip for Testing:** use a command-line MQTT client to test messages without writing code. Note, that the `publish` command within these CLI clients typically handles the entire sequence for you: it connects, sends the message, and disconnects. First, download and install the MQTTX CLI from [their website](https://mqttx.app/cli#download). Then, use the following command to send a log message: Bash PowerShell ```bash mqttx pub \ --hostname 'mqtt.spotflow.io' \ --port 8883 \ --topic 'ingest-json' \ --protocol 'mqtts' \ --username 'quickstart_device' \ --password '' \ --message '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' ``` ```powershell mqttx pub ` --hostname 'mqtt.spotflow.io' ` --port 8883 ` --topic 'ingest-json' ` --protocol 'mqtts' ` --username 'quickstart_device' ` --password '' ` --message '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' ``` [Mosquitto](https://mosquitto.org/download/) ships with a `mosquitto_pub` command line MQTT client. Make sure you have `mosquitto_pub` available in your PATH. Then, use the following command to send a log message: Bash PowerShell ```bash mosquitto_pub \ -h 'mqtt.spotflow.io' \ -p 8883 \ -t 'ingest-json' \ -u 'quickstart_device' \ -P '' \ -m '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' \ --capath /etc/ssl/certs ``` ```bash mosquitto_pub ` -h 'mqtt.spotflow.io' ` -p 8883 ` -t 'ingest-json' ` -u 'quickstart_device' ` -P '' ` -m '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' ` --cafile ./isrgrootx1.pem ``` Depending on your operating system and its configuration, you may need to customize the command to point it to a proper location containing the [ISRG Root X1 TLS certificate](https://letsencrypt.org/certificates/#root-cas) via `--capath` or `--cafile` options. To send logs serialized in CBOR, simply publish MQTT messages into the `ingest-cbor` topic. The payload should include a single log following the `log-message` schema as defined below: ```CDDL log-message = { ? 1 => tstr, ; body: fully interpolated log line string (optional when bodyTemplate is used) ? ( 2 => tstr, ; bodyTemplate (optional): printf-like interpolation string 3 => body-template-values ; bodyTemplateValues (optional): values for interpolation ), ? 4 => severity, ; severity (recommended) ? 5 => labels, ; labels (optional): user-defined key-value pairs for additional context ? 6 => uint, ; deviceUptimeMs (optional): device uptime in milliseconds in range [0, 2^63 - 1] ? 7 => uint, ; deviceTimestampMs (optional): device timestamp in milliseconds in range [0, 2^63 - 1] ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } labels = {* (tstr => (tstr / int / float / bool))} ; Strongly typed key-value pairs ; Integer severity values debug-severity = 30 info-severity = 40 warning-severity = 50 error-severity = 60 critical-severity = 70 severity = (debug-severity / info-severity / warning-severity / error-severity / critical-severity) ; A strongly typed array of values, or an array of byte string representations (big-endian) of the values body-template-values = [ * (tstr / int / float / bool / null) ] / [ * bstr ] ``` For logs, [Session Metadata](/guides/mqtt/session-metadata) provides context that applies to every later log in the same MQTT connection. Session labels are merged into log labels, and `deviceUptimeMs` can help calculate timestamps for logs that do not include `deviceTimestampMs`. ## Analyze Logs in the Web Application [#analyze-logs-in-the-web-application] Once your device is integrated and sending logs, you can analyze them in the web application. The main entry point is the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter logs by their content, device ID, severity, and other metadata. You can click on individual log messages to see their details, and it is possible to drill down into specific events by matching their metadata. ## Learn more [#learn-more] # Metrics with MQTT (/guides/mqtt/metrics-mqtt) This guide explains how to send metrics from devices running a MQTT client and analyze them in the Spotflow web application. ## Connect to the Spotflow MQTT broker [#connect-to-the-spotflow-mqtt-broker] As the very first step, check the following basic integration guide: * [Quickstart: Connect via MQTT](/quickstart/mqtt) ## Publish metrics in JSON or CBOR format [#publish-metrics-in-json-or-cbor-format] Metrics can be sent over MQTT using either JSON or CBOR payloads. Publish metric messages to the `ingest-json` topic using the following schema: ```json { "messageType": "METRIC", "metricName": "cpu_utilization_percent", // Optional for 0/1m/1h/1d metrics, present for aggregated metrics "aggregationInterval": "1m", // Optional labels for dimensional metrics "labels": { "interface": "wlan0" }, // Device uptime when metric sample/window was produced "deviceUptimeMs": 123456, // Sequence number within a metric stream "sequenceNumber": 42, // For aggregated metrics: sum over the window // For 0/no aggregation: raw sample value "sum": 318.7, // Optional overflow marker for integer sums "sumTruncated": false, // Present for aggregated metrics "count": 30, "min": 5.2, "max": 18.1, // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` Publish metric messages to the `ingest-cbor` topic using the following schema: ```CDDL metric-message = { 0 => 5, ; messageType: metric 21 => tstr, ; metricName ? 22 => agg-interval, ; aggregationInterval (0/1/3/4) ? 5 => labels, ; labels map ? 6 => int, ; deviceUptimeMs ? 13 => uint, ; sequenceNumber 24 => metric-value, ; sum (or raw value for no aggregation) ? 25 => bool, ; sumTruncated (optional) ? 26 => uint, ; count (for aggregated metrics) ? 27 => metric-value, ; min (for aggregated metrics) ? 28 => metric-value, ; max (for aggregated metrics) ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } labels = {* (tstr => tstr)} metric-value = int / float ; Aggregation interval enum values used by device module agg-none = 0 ; 0 agg-1min = 1 ; 1m agg-1hour = 3 ; 1h agg-1day = 4 ; 1d agg-interval = (agg-none / agg-1min / agg-1hour / agg-1day) ``` For metrics, [Session Metadata](/guides/mqtt/session-metadata) is useful for dimensions that stay the same for the whole MQTT connection, such as firmware channel, board revision, or site. Session labels are merged into metric labels, so you do not need to repeat them on every data point. ## Available system metrics [#available-system-metrics] When using Spotflow system metrics, the following metric names, value data, and labels are used: | Metric name | Value data | Expected labels | | --------------------------- | ----------------------------------------------- | --------------- | | `heap_free_bytes` | Free heap bytes (`int`) | None | | `heap_allocated_bytes` | Allocated heap bytes (`int`) | None | | `cpu_utilization_percent` | CPU utilization in percent (`float`) | None | | `thread_stack_free_bytes` | Free stack bytes per thread (`int`) | `thread` | | `thread_stack_used_percent` | Used stack percentage per thread (`float`) | `thread` | | `network_tx_bytes` | Transmitted bytes per network interface (`int`) | `interface` | | `network_rx_bytes` | Received bytes per network interface (`int`) | `interface` | | `connection_mqtt_connected` | MQTT connection state (`int`, `0` or `1`) | None | | `boot_reset` | Reset event marker (`int`, value `1`) | `reason` | | `uptime_ms` | Device uptime in milliseconds (`int`) | None | System metrics are visualized in the built-in [Device Dashboard](/fundamentals/monitoring/dashboards#device-dashboard). ### Custom metrics [#custom-metrics] You are not limited to the system metrics listed above. Use any `metricName` to track application-specific values such as sensor readings, counters, or latencies. The payload format is the same. For example, an aggregated custom metric with labels: ```json { "messageType": "METRIC", "metricName": "lock_operation_duration_ms", "aggregationInterval": "1m", "labels": { "operation": "unlock", "method": "nfc" }, "deviceUptimeMs": 120000, "sequenceNumber": 5, "sum": 1250.0, "count": 10, "min": 45.2, "max": 310.7 } ``` * `metricName` can be any string. Names are normalized on ingestion (lowercased, special characters replaced with underscores). * `aggregationInterval` is optional. Omit it or set it to `"0"` to send raw values without aggregation. Supported values are `"0"` (no aggregation), `"1m"` (1 minute), `"1h"` (1 hour), and `"1d"` (1 day). * `labels` is optional. Use it to add dimensional breakdowns to your metric. * For non-aggregated metrics, only `sum` is required (it represents the raw value). For aggregated metrics, include `sum`, `count`, `min`, and `max`. System metrics have predefined dashboards (see [Device Dashboard](/fundamentals/monitoring/dashboards#device-dashboard)). For custom metrics, you can [create your own dashboards](/using-spotflow/guides/custom-dashboards) with tailored widgets. ## Analyze metrics in dashboards [#analyze-metrics-in-dashboards] Open the Spotflow dashboards to inspect device and fleet metric trends: * [Dashboards](/fundamentals/monitoring/dashboards) ## Learn more [#learn-more] # Over-the-air (OTA) updates with MQTT (/guides/mqtt/ota-mqtt) This guide explains how a device can participate in Spotflow over-the-air (OTA) updates using MQTT directly, without using the Spotflow device module. For basic concepts and deployment workflows in the web application, see [Over-the-air (OTA) updates](/fundamentals/ota) and [Deploy over-the-air (OTA) updates](/using-spotflow/guides/ota). ## Connect to the Spotflow MQTT broker [#connect-to-the-spotflow-mqtt-broker] As the first step, connect to the Spotflow MQTT broker: * [Quickstart: Connect via MQTT](/quickstart/mqtt) OTA updates use dedicated MQTT topics. Choose JSON or CBOR and use the matching topic names: | Direction | JSON topic | CBOR topic | QoS | | --------------- | -------------- | -------------- | ------ | | Cloud to device | `ota-json-c2d` | `ota-cbor-c2d` | 1 | | Device to cloud | `ota-json-d2c` | `ota-cbor-d2c` | 0 or 1 | Subscribe to the cloud-to-device topic for your chosen format. The broker routes the subscription to the device-specific topic, such as `ota-cbor-c2d/{workspaceId}/{deviceId}`. CBOR payloads use numeric property keys. ## Handle `UPDATE_ARTIFACTS` [#handle-update_artifacts] The cloud sends the message `UPDATE_ARTIFACTS` when the device should perform an OTA update: ```json { "messageType": "UPDATE_ARTIFACTS", "updateAttemptId": 123, "isCanceled": false, "manifest": [ { "artifactType": "FIRMWARE", "slug": "main", "isMain": true, "url": "https://api.spotflow.io/.../content", "secret": "artifact-download-secret", "version": "2.0.0" } ] } ``` ```CDDL update-artifacts = { 0 => 6, ; messageType: UPDATE_ARTIFACTS 32 => uint, ; updateAttemptId, non-zero 64-bit unsigned integer ? 33 => bool, ; isCanceled, defaults to false 34 => [* artifact] } artifact = { 35 => 0, ; artifactType: FIRMWARE 36 => tstr .size (1..32), ; slug ? 37 => bool, ; isMain, defaults to false 38 => tstr .size (1..117), ; url 39 => tstr .size (1..24), ; secret 40 => tstr .size (1..64) ; version } ``` The update attempt is uniquely identified by a non-zero `updateAttemptId`. Each artifact in the manifest contains the following properties: * `artifactType`: Spotflow currently supports only firmware updates. Other kinds of artifacts will be added in the future. * `slug`: The unique identifier of the firmware, such as `sample-firmware`. * `isMain`: Whether the firmware is meant to update the main MCU on the device. The Spotflow device module handles main firmware in a special way, but your custom implementation might not need it. * `url`: The URL of the firmware image to download. * `secret`: Use this secret in the HTTP header `Authorization: OtaSecret ` when downloading the firmware image. * `version`: The firmware version. Your implementation might persist the latest installed version for each firmware slug to avoid unnecessary downloads. ## Publish `UPDATE_RESULTS` [#publish-update_results] After each firmware update, publish the message `UPDATE_RESULTS` to the device-to-cloud topic for your chosen format: Publish JSON `UPDATE_RESULTS` messages to `ota-json-d2c`: ```json { "messageType": "UPDATE_RESULTS", "updateAttemptId": 123, "succeeded": [0], "failed": [], "canceled": [] } ``` If an error prevents the device from handling the attempt before any artifact is processed, send `updateAttemptError` instead of artifact result arrays: ```json { "messageType": "UPDATE_RESULTS", "updateAttemptId": 123, "updateAttemptError": "CANNOT_PARSE_MESSAGE" } ``` The allowed values of `updateAttemptError` are `UNKNOWN_ERROR`, `ARTIFACT_COUNT_EXCEEDED`, `UNKNOWN_ARTIFACT_TYPE`, and `CANNOT_PARSE_MESSAGE`. Publish CBOR `UPDATE_RESULTS` messages to `ota-cbor-d2c`: ```CDDL update-results = { 0 => 9, ; messageType: UPDATE_RESULTS 32 => uint, ; updateAttemptId ? 45 => uint, ; updateAttemptError ? 42 => [* uint], ; succeeded artifact indexes ? 43 => [* uint], ; failed artifact indexes ? 44 => [* uint] ; canceled artifact indexes } ``` If an error prevents the device from handling the attempt before any artifact is processed, set `updateAttemptError` instead of artifact result arrays. The allowed values of `updateAttemptError` are `UNKNOWN_ERROR` (`0`), `ARTIFACT_COUNT_EXCEEDED` (`1`), `UNKNOWN_ARTIFACT_TYPE` (`2`), and `CANNOT_PARSE_MESSAGE` (`3`). The arrays `succeeded`, `failed`, and `canceled` contain the indexes of the firmware updates in the manifest. You can publish one result at a time or publish several known results together. The Spotflow device module processes the firmware updates in the manifest order. If an update fails, the module reports the remaining unprocessed updates as canceled. Your implementation might process the updates in a different order (even in parallel) and have different failure handling logic. Still, it must eventually report the results of all the updates in the manifest. ## Handle `REPORT_UPDATE_RESULTS` [#handle-report_update_results] Because the device is allowed to publish `UPDATE_RESULTS` with QoS 0, this message might be lost in rare cases. To ensure that update results are eventually delivered to the cloud, the cloud can ask the device to resend them. The device should publish an `UPDATE_RESULTS` message with all firmware update results for the current attempt when it receives the message `REPORT_UPDATE_RESULTS`: ```json { "messageType": "REPORT_UPDATE_RESULTS", "updateAttemptId": 123 } ``` ```CDDL report-update-results = { 0 => 8, ; messageType: REPORT_UPDATE_RESULTS 32 => uint ; updateAttemptId } ``` ## (Optional) Handle cancellation [#optional-handle-cancellation] The cloud can request cancellation in two ways: * When the device has already received the update attempt in the current MQTT session, the cloud sends the message `CANCEL_UPDATE`: ```json { "messageType": "CANCEL_UPDATE", "updateAttemptId": 123 } ``` ```CDDL cancel-update = { 0 => 7, ; messageType: CANCEL_UPDATE 32 => uint ; updateAttemptId } ``` * When the device subscribes to the cloud-to-device topic for OTA updates while the update attempt is already canceled, the cloud sends the message `UPDATE_ARTIFACTS` with `isCanceled: true`. This message still contains the full manifest in case the device wants to ignore the cancellation. If your implementation supports cancellation, handle it by publishing `UPDATE_RESULTS` with a `canceled` array containing the indexes of firmware updates that were not completed. Cancellation should be handled on a best-effort basis so that the device is not left in a half-updated state. For example, the Spotflow device module reacts to cancellation only while the first firmware update is still in progress. ## (Optional) Extend session metadata with OTA update attempt ID [#optional-extend-session-metadata-with-ota-update-attempt-id] If your device supports OTA updates, include `lastUpdateAttemptId` in [Session Metadata](/guides/mqtt/session-metadata#over-the-air-ota-update-attempt-metadata). It lets Spotflow detect, on a best-effort basis, when firmware may have changed outside OTA updates. ## Protocol guarantees [#protocol-guarantees] * `UPDATE_ARTIFACTS` must be the first cloud-to-device message for an attempt. `CANCEL_UPDATE` and `REPORT_UPDATE_RESULTS` only make sense after the device has the attempt context. * Only one firmware with `isMain` set to `true` can be present in the manifest. * Until the cloud receives results for all the artifacts or an `updateAttemptError`, it sends the same `UPDATE_ARTIFACTS` message whenever the device subscribes to the cloud-to-device topic for OTA updates. There is an exception to this rule: When you delete the device's deployment cohort before the device finishes the current attempt and put the device in a new cohort, the cloud will not wait for the results of the current attempt before sending the new one. However, your implementation might ignore this edge case if you do not delete deployment cohorts. ## Learn more [#learn-more] # MQTT Session metadata (/guides/mqtt/session-metadata) A Session Metadata message lets a device describe the MQTT session before it starts sending logs, metrics, or crash data. It is optional, but it is useful whenever the same context applies to many messages: firmware build ID, SDK or agent version, device run ID, OS version, hardware revision, or any other labels you want to search and filter by later. Send it once, right after the MQTT connection is established, before the first telemetry message. Spotflow then applies that metadata to subsequent messages in the same session. ## How it works [#how-it-works] Publish a message with `messageType` set to `SESSION_METADATA` to the normal device-to-cloud ingestion topic: * Use `ingest-json` for JSON payloads. * Use `ingest-cbor` for CBOR payloads. The metadata describes the current MQTT connection and device run. Most integrations send it once with all known session-level fields. It applies only to messages sent after it in the same MQTT session. If you send another Session Metadata message later in the same session, the new context replaces the previous one for subsequent messages. You can still set labels on individual log or metric messages. Message-level labels are merged with session-level labels, and message-level labels take precedence when the same key appears in both places. One-shot MQTT CLI commands usually connect, publish one message, and disconnect. To test Session Metadata, use a client or script that can publish metadata and telemetry on the same connection. ## Over-the-air (OTA) update attempt metadata [#over-the-air-ota-update-attempt-metadata] Devices that support over-the-air (OTA) updates should include `lastUpdateAttemptId`. It is the most recent OTA update attempt ID received by the device. After the device receives a new OTA update attempt, it should report that attempt ID in future sessions. If this value is not equal to the last update attempt ID sent to the device, the cloud assumes that the firmware version has changed outside of OTA updates. In that case, if the device is in the *Succeeded* state of an *Active* deployment, the cloud will attempt to update the device again. Valid OTA attempt IDs are non-zero, so send `0` when the device supports OTA but has no record of any received OTA attempt. The cloud distinguishes a missing field from `0`: * Missing means the device does not provide OTA update attempt metadata, so Spotflow does not make any decisions based on it. * `0` means the device is capable of OTA updates but has no remembered attempt. This signals to the cloud that it should attempt to update the device. ## Message schema [#message-schema] Only `messageType` is required. All other fields are optional and can be included when they add useful context for the session. Publish Session Metadata messages to `ingest-json` using the following structure: ```json { // Required: identifies this message as Session Metadata "messageType": "SESSION_METADATA", // (Optional) SDK, MQTT client, or integration that opened the session "mqttAgent": "custom-mqtt-client/1.2.0", // (Optional) firmware build ID as a lowercase 40-character SHA1 hex string "buildId": "184952ef1dde1c12364174d40618f9cca9d1814d", // (Optional) current device run ID, usually changed after each boot "deviceRunId": 42, // (Optional) last OTA update attempt received by the device. // Use 0 when the device supports OTA but has not received any OTA attempt yet. "lastUpdateAttemptId": 123, // (Optional) 64-bit monotonic device uptime in milliseconds. // Used to calculate deviceTimestampMs for later log messages that only provide deviceUptimeMs. "deviceUptimeMs": 1200, // (Optional) IANA time zone name used as a hint for UI defaults and queries "timeZone": "Europe/Prague", // (Optional) labels applied to subsequent messages in this MQTT session. // Keys must be non-empty strings; values can be strings, numbers, or booleans. "labels": { "osVersion": "0.42.0", "hardwareRevision": "rev-b", "firmwareChannel": "staging" } } ``` Publish Session Metadata messages to `ingest-cbor` using the following structure. CBOR payloads can use numeric property keys to reduce payload size: ```CDDL session-metadata-message = { 0 => 1, ; messageType: SESSION_METADATA (required) ? 8 => tstr, ; mqttAgent: SDK, MQTT client, or integration name ? 14 => bstr .size 20, ; buildId: firmware SHA1 bytes ? 5 => labels, ; labels: session-level labels applied to subsequent messages ? 6 => uint, ; deviceUptimeMs: used to calculate timestamps for later log messages ? 20 => tstr, ; timeZone: IANA time zone name ? 30 => uint, ; deviceRunId: current device run ID, usually changed after boot ? 41 => uint ; lastUpdateAttemptId: last OTA attempt received by the device, or 0 if none } labels = {* (tstr => (tstr / int / float / bool))} ; non-empty string keys; string, number, or boolean values ``` ## JSON example [#json-example] ```json { "messageType": "SESSION_METADATA", "mqttAgent": "custom-mqtt-client/1.2.0", "buildId": "184952ef1dde1c12364174d40618f9cca9d1814d", "deviceRunId": 42, "lastUpdateAttemptId": 123, "labels": { "osVersion": "0.42.0", "hardwareRevision": "rev-b", "firmwareChannel": "staging" } } ``` After this message, a log can be much smaller while still carrying the session context: ```json { "body": "Lock motor calibration completed", "severity": "INFO", "labels": { "component": "lock-motor" } } ``` In Spotflow, the log can be filtered by `component=lock-motor` from the log message and by `osVersion=0.42.0`, `hardwareRevision=rev-b`, and `firmwareChannel=staging` from the Session Metadata message. ## Learn more [#learn-more] # Difference from "vanilla" Zephyr (/guides/nordic-nrf-connect/zephyr-differences) Because nRF Connect SDK is based on Zephyr, the following pages in the Zephyr guide apply to nRF Connect SDK as well: * [Logging](/guides/zephyr/logging-zephyr) * [Crash reports](/guides/zephyr/crash-reports-zephyr) (see [Core dump partition configuration](#core-dump-partition-configuration) below) * [Advanced configuration](/guides/zephyr/advanced-config-zephyr) There is only one difference you should be aware of when customizing partitions for core dumps: ## Core dump partition configuration [#core-dump-partition-configuration] Although nRF Connect SDK uses [Partition Manager](https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/scripts/partition_manager/partition_manager.html) to compute flash partitions, the Zephyr core dump module also requires the core dump partition to be defined in the devicetree. Therefore, you need to add the core dump partition **both** to the Partition Manager configuration and to the devicetree. For example, add the following file to the root of your project when building for nRF7002DK without TF-M: ```yaml title="pm_static_nrf7002dk_nrf5340_cpuapp.yml" app: address: 0x0 end_address: 0xec000 region: flash_primary size: 0xec000 coredump_partition: address: 0xf0000 end_address: 0x100000 region: flash_primary size: 0x10000 settings_storage: address: 0xec000 end_address: 0xf0000 region: flash_primary size: 0x4000 ``` Then, add the following devicetree overlay: ```dts title="boards/nrf7002dk_nrf5340_cpuapp.overlay" /delete-node/ &storage_partition; /delete-node/ &tfm_ps_partition; /delete-node/ &tfm_its_partition; /delete-node/ &tfm_otp_partition; &flash0 { partitions { /* Reserve last 64 KiB of flash for core dumps */ coredump_partition: partition@f0000 { label = "coredump-partition"; reg = <0x000f0000 DT_SIZE_K(64)>; }; }; }; ``` Note that only the core dump partition is defined in the devicetree overlay. It's sufficient to define other partitions (such as storage partition) only in the Partition Manager configuration. Ensure that the address and size of the core dump partition in the devicetree overlay and in the Partition Manager configuration match. Check our [core dump sample on GitHub](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/samples/coredumps), for example, to see the [Partition Manager configuration for nRF7002DK with TF-M](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/samples/coredumps/pm_static_nrf7002dk_nrf5340_cpuapp_ns.yml). # Silicon Labs (/guides/silicon-labs) The following tested Silicon Labs boards can connect to Spotflow through a Bluetooth Low Energy gateway or directly over Wi-Fi. Choose the transport that matches your board and application: | Board | Zephyr board target | BLE | Wi-Fi | | ----------------------------- | ------------------- | --- | ----- | | xG24 Explorer Kit (EFR32MG24) | `xg24_ek2703a` | Yes | NA | | SiWx917 Dev Kit (BRD2605A) | `siwx917_dk2605a` | Yes | Yes | The Spotflow Device SDK selects one Spotflow transport per firmware build. BLE sends data through a gateway (e.g the Spotflow Web App) and ingest key is provided by the gateway. Wi-Fi uses MQTT to connect directly to Spotflow and requires an ingest key (a secret used for authentication) to be configured in KConfig that can be found in the Spotflow Web App. ## Bluetooth Low Energy [#bluetooth-low-energy] Use the Spotflow BLE transport to send logs, metrics, and crash reports through a gateway. The board acts as a BLE peripheral, while the gateway connects to Spotflow over MQTT. BLE devices do not store a Spotflow ingest key. The gateway provides the ingest key and forwards data from the board to Spotflow. For development and testing, you can use the Spotflow Web App as the gateway. ### Install prerequisites [#install-prerequisites] Install the dependencies from the Zephyr [Getting Started Guide](https://docs.zephyrproject.org/latest/develop/getting_started/index.html#install-dependencies), including Git and Python 3.10 or later. You also need: * A supported Silicon Labs board and a USB cable * Flashing software supported by the board * A computer with Bluetooth Low Energy support * A Chromium-based browser with Web Bluetooth support to use the Spotflow Web App as a gateway On Windows, use a short workspace path, such as `C:\spotflow-ws`, to avoid exceeding the maximum path length during the build. ### Create a West workspace [#create-a-west-workspace] Create a workspace using the Silicon Labs manifest from the Spotflow Device SDK. The manifest installs Zephyr and the required Silicon Labs HAL. ```bash mkdir spotflow-ws cd spotflow-ws python3 -m venv .venv source .venv/bin/activate pip install west west init \ --manifest-url https://github.com/spotflow-io/device-sdk \ --manifest-file zephyr/manifests/west-zephyr-silabs.yml \ . west update --fetch-opt=--depth=1 --narrow west packages pip --install west sdk install --version 1.0.1 --toolchains arm-zephyr-eabi west blobs fetch hal_silabs ``` ```powershell New-Item -ItemType Directory -Path C:\spotflow-ws Set-Location C:\spotflow-ws python -m venv .venv .\.venv\Scripts\Activate.ps1 python -m pip install west west init ` --manifest-url https://github.com/spotflow-io/device-sdk ` --manifest-file zephyr/manifests/west-zephyr-silabs.yml ` . west update --fetch-opt=--depth=1 --narrow west packages pip --install west sdk install --version 1.0.1 --toolchains arm-zephyr-eabi west blobs fetch hal_silabs ``` ### Configure the BLE sample [#configure-the-ble-sample] Replace the sample device ID in the configuration file for your board with an identifier that is unique within your Spotflow workspace: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_DEVICE_ID="" ``` The sample already enables the BLE transport, logging, metrics, and crash reports: ```dotenv title="prj.conf" CONFIG_SPOTFLOW=y CONFIG_SPOTFLOW_TRANSPORT_BLE=y CONFIG_SPOTFLOW_LOG_BACKEND=y CONFIG_SPOTFLOW_METRICS=y CONFIG_SPOTFLOW_COREDUMPS=y ``` Do not add `CONFIG_SPOTFLOW_INGEST_KEY`. Authentication is handled by the BLE gateway. ### Build and flash the sample [#build-and-flash-the-sample] Run the commands from `/modules/lib/spotflow/zephyr/samples/ble`. ```bash west build --pristine --board xg24_ek2703a . west flash ``` The sample uses a 32 KiB in-memory crash-report backend on this board. The SDK also contains `boards/xg24_ek2703a-internal.overlay`, which configures the radio for 20 dBm output and a 3.3 V PA supply. Do not apply this overlay unless those settings are correct for your exact hardware and permitted in your region. ```bash west build --pristine --board siwx917_dk2605a . west flash ``` The sample uses a 32 KiB in-memory crash-report backend because the network processor controls flash and it cannot be accessed from the fault handler. ### Use the Spotflow Web App as a BLE Gateway [#use-the-spotflow-web-app-as-a-ble-gateway] For development and testing, you can use Spotflow Web App as a BLE gateway. It connects to a BLE leaf device and forwards telemetry to Spotflow over MQTT. In the Spotflow Web Application, open the BLE Gateway page from the user menu in the top-right corner. "BLE gateway link" On the BLE Gateway page, select the **Ingest key** to use for the MQTT connection. Then click **Scan for BLE devices** to find nearby BLE leaf devices advertising the Spotflow GATT service. Select the device advertised as `Spotflow EFR32MG24` or `Spotflow SiLabs SIWX917`. Keep the gateway page open while testing the integration. Select a device from the list to connect. Once connected, the browser forwards telemetry from the BLE device to Spotflow over MQTT. This setup is intended for development and testing because the browser Bluetooth API provides limited control over the BLE stack. The Spotflow Web App has to stay open in the browser to maintain the BLE connection. If you close the browser or navigate away from the page, the BLE connection is lost and the board cannot forward telemetry to Spotflow. Use the button Open Device Events. ### Analyze Logs in the Web Application [#analyze-logs-in-the-web-application] Use the button Open Device Events. You will be forwarded to the device events page in the Spotflow Web App, where you can view logs, metrics, and crash reports from the board. ### Verify telemetry and crash reports [#verify-telemetry-and-crash-reports] The serial console should contain messages similar to: ```text Starting Spotflow BLE sample Bluetooth enabled for Spotflow BLE transport BLE advertising started as ... ``` Press the board button represented by the Zephyr `sw0` devicetree alias to call `k_oops()`. The board restarts and sends the stored crash report after reconnecting to the gateway. The board disconnects from the gateway when it restarts. You must reconnect it manually after a crash to forward the crash report to Spotflow. ### (Recommended) Upload ELF file with symbols [#recommended-upload-elf-file-with-symbols] To unlock advanced crash analysis features, you can upload the ELF file containing debug symbols to Spotflow. The core dump will be automatically linked based on **Build ID** embedded into both ELF file and core dump file automatically by Spotflow. The symbols will be used to decode stack traces and variable names in the Spotflow web application. See dedicated [Firmware Management](/fundamentals/firmware-management) page for information about managing firmwares and symbol files. ### Analyze Crash Reports in the Web Application [#analyze-crash-reports-in-the-web-application] Once your device is integrated, you can analyze crash reports and core dumps in the web application. You can list the crash reports in the Device Events page, which gives you a comprehensive view of all events collected from your devices. There, you can filter crash reports. The full analysis includes a detailed description of the root cause and suggestions for fixing the issue. Also, for complete transparency, we include references to all documentation pages used, allowing you to explore further. ## Wi-Fi [#wi-fi] Use the Spotflow MQTT transport to connect a SiWx917 board directly to Spotflow over Wi-Fi. A BLE gateway is not involved in this workflow. ### Install prerequisites [#install-prerequisites-1] Install the dependencies from the Zephyr [Getting Started Guide](https://docs.zephyrproject.org/latest/develop/getting_started/index.html#install-dependencies). You also need a supported SiWx917 board, flashing software supported by the board, and a 2.4 GHz Wi-Fi network with internet access. ### Create a West workspace [#create-a-west-workspace-1] Run the setup script for the SiWx917 Dev Kit: ```bash source <(curl --proto '=https' --tlsv1.2 -sSf https://downloads.spotflow.io/spotflowup.sh) \ --zephyr --board siwx917_dk2605a ``` ```powershell Invoke-Expression "& { $(Invoke-RestMethod -Uri 'https://downloads.spotflow.io/spotflowup.ps1' -UseBasicParsing) } -zephyr -board siwx917_dk2605a" ``` The script creates a West workspace using the Silicon Labs manifest and installs the required Zephyr SDK and `arm-zephyr-eabi` toolchain. ### Configure Wi-Fi and Spotflow [#configure-wi-fi-and-spotflow] Open `/modules/lib/spotflow/zephyr/samples/logs/prj.conf` and add: ```dotenv title="prj.conf" CONFIG_NET_WIFI_SSID="" CONFIG_NET_WIFI_PASSWORD="" CONFIG_SPOTFLOW_DEVICE_ID="" # Ingest key obtained from https://app.spotflow.io/ingest-keys CONFIG_SPOTFLOW_INGEST_KEY="" ``` The Device ID must be unique within your Spotflow workspace. Copy an ingest key on the [ingest keys page](https://app.spotflow.io/ingest-keys). The logging sample uses the MQTT transport by default. The sample detects the SiWx91x Wi-Fi interface and enables Wi-Fi automatically. ### Build and flash the sample [#build-and-flash-the-sample-1] Run the commands from `/modules/lib/spotflow/zephyr/samples/logs`: ```bash west build --pristine --board siwx917_dk2605a . west flash ``` ### Verify the connection [#verify-the-connection] Open the serial console and confirm that the board connects to the configured Wi-Fi network, obtains an IP address, and establishes an MQTT connection. Expected output includes: ```text Waiting for network... Connecting to SSID: ... Network connectivity established and IP address assigned MQTT connected! ``` The sample sends an informational log every two seconds. Open the device in the Spotflow Web App and confirm that the logs arrive. ## Analyze Logs in the Web Application [#analyze-logs-in-the-web-application-1] Once your device is integrated and sending logs, you can analyze them in the web application. The main entry point is the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter logs by their content, device ID, severity, and other metadata. You can click on individual log messages to see their details, and it is possible to drill down into specific events by matching their metadata. ### Continue Exploring Spotflow Features [#continue-exploring-spotflow-features] You can continue with other Spotflow features, such as metrics and crash reports. See the [Zephyr Crash Reports](/guides/zephyr/crash-reports-zephyr) or [Zephyr Metrics](/guides/zephyr/metrics-zephyr) guides for more information. ## Learn more [#learn-more] # Advanced configuration for Zephyr (/guides/zephyr/advanced-config-zephyr) ## Device ID [#device-id] ### Kconfig option [#kconfig-option] Easiest way to set device ID is by using the `CONFIG_SPOTFLOW_DEVICE_ID` Kconfig option in your `prj.conf` file: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_DEVICE_ID="your-device-id" ``` If setting the device ID via Kconfig works for you, there is no need to investigate other options and we recommend sticking with this approach. ### Setting device ID at runtime [#setting-device-id-at-runtime] However, in more complex scenarios, Kconfig option may not be sufficient, especially when device ID is not known at compile time or needs to be set dynamically. In such cases, define function `spotflow_override_device_id` that returns a string with the device ID. Inside this function, you can implement the logic to determine the device ID at runtime and Spotflow device module will call this function to get the device ID. ```c title="src/main.c" const char* spotflow_override_device_id() { return "my_nrf7002dk_test"; } ``` Although this approach allows to setting device ID dynamically, it most cases the function should contain deterministic logic returning consistent values across device reboots. Random or frequently changing will prevent you from efficiently analyzing device-specific behavior and performance as well as it might lead to increase subscription usage in terms of [number of registered devices](https://spotflow.io/#pricing). ### Hardware Info Interface [#hardware-info-interface] If no device ID is explicitly provided, the Spotflow device module will try to use the Zephyr's [Hardware Info Interface](https://docs.zephyrproject.org/latest/hardware/peripherals/hwinfo.html) to retrieve the device ID via [`hwinfo_get_device_id(...)`](https://docs.zephyrproject.org/latest/doxygen/html/group__hwinfo__interface.html#ga197b58d995c77aae423527d0f8d9ff31) function. ## Session metadata labels [#session-metadata-labels] A [Session Metadata](/guides/mqtt/session-metadata) message describes the current connection and provides context for telemetry sent during that connection. The Spotflow device module sends it automatically when the connection is established. Session metadata labels add context that applies to all logs and metrics sent during a connection, such as a hardware revision, production line, or feature flag. You can use these labels to search, filter, and group telemetry without repeating the same context on individual metrics. Labels attached directly to a metric take precedence when they use the same key as a session metadata label. Include `` and define `spotflow_override_session_metadata_labels()` to return the labels: ```c title="src/main.c" #include #include static const struct spotflow_session_label session_labels[] = { { .key = "hardwareRevision", .type = SPOTFLOW_SESSION_LABEL_STRING, .value.string = "rev-b", }, { .key = "productionLine", .type = SPOTFLOW_SESSION_LABEL_INT, .value.integer = 4, }, { .key = "samplingRatio", .type = SPOTFLOW_SESSION_LABEL_FLOAT, .value.floating = 0.25, }, { .key = "diagnosticsEnabled", .type = SPOTFLOW_SESSION_LABEL_BOOL, .value.boolean = true, }, }; struct spotflow_session_metadata_labels spotflow_override_session_metadata_labels(void) { return (struct spotflow_session_metadata_labels){ .items = session_labels, .count = ARRAY_SIZE(session_labels), }; } ``` The callback must not block or call other Spotflow APIs. The Session Metadata message uses a statically allocated encoding buffer. If your labels do not fit, increase its size in `prj.conf`: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_SESSION_METADATA_BUFFER_SIZE=512 ``` The default is 256 bytes. MQTT supports values up to 4096 bytes, while BLE supports up to 512 bytes. If any label is invalid or the complete message does not fit, the module logs a warning and sends Session Metadata without the application-defined labels. ## Cloud hostname and port [#cloud-hostname-and-port] By default, the Spotflow device module connects to the Spotflow cloud at `mqtt.spotflow.io` on port `8883` which is a standard port used for MQTT over TLS. In case you need to change these settings (e.g. due to using proxy), following Kconfig options are available: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_CLOUD_HOSTNAME="mqtt.spotflow.io" CONFIG_SPOTFLOW_CLOUD_PORT=8883 ``` ## Storing sensitive information in the configuration files [#storing-sensitive-information-in-the-configuration-files] Most of the configuration options are typically not sensitive and thus is it safe and beneficial to version them in the Git or other version control system. However, some options such as [Ingest Key](/fundamentals/device-authorization) or WiFi passwords are sensitive and it is a security issue to store them in the version control. For such cases, we recommend store the sensitive options in a separate [`credentials.conf`](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/samples/coredumps/credentials-sample.conf) file that is not tracked by version control and [merge them](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/samples/coredumps/CMakeLists.txt) into the resulting configuration during build. ## Thread priorities and stack size [#thread-priorities-and-stack-size] Spotflow device module is using a separate thread to send data via MQTT. To modify priority of this thread, enable `SPOTFLOW_MQTT_THREAD_CUSTOM_PRIORITY` option and choose the custom priority via `SPOTFLOW_MQTT_THREAD_PRIORITY`. To modify stack size of this thread, use `SPOTFLOW_PROCESSING_THREAD_STACK_SIZE` option. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_MQTT_THREAD_CUSTOM_PRIORITY=y CONFIG_SPOTFLOW_MQTT_THREAD_PRIORITY=14 CONFIG_SPOTFLOW_PROCESSING_THREAD_STACK_SIZE=2560 ``` ## Build ID [#build-id] To disable generating and embedding build ID, disable `SPOTFLOW_GENERATE_BUILD_ID` option. Disabling build ID will prevent Spotflow from automatically linking logs and core dumps with the uploaded ELF files. However, the ELF files can still be linked manually to specific core dumps. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_GENERATE_BUILD_ID=n ``` ## Internal logging [#internal-logging] To adjust default log level of internal Spotflow device module logging, use the `SPOTFLOW_MODULE_DEFAULT_LOG_LEVEL` option. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_MODULE_DEFAULT_LOG_LEVEL=3 ``` ## Changing minimal log severity from portal [#changing-minimal-log-severity-from-portal] The Spotflow device module allows you to update the minimal severity of sent log messages during runtime by [changing its value in the portal](/guides/zephyr/logging-zephyr#optional-adjust-devices-minimal-log-severity). By default, all log messages are sent, and you can use this option to reduce the log volume when needed. Alternatively, you can use the Kconfig option `SPOTFLOW_DEFAULT_SENT_LOG_LEVEL` to start sending initially, for example, only errors and warnings, and increase the verbosity only when needed. If `CONFIG_SPOTFLOW_SETTINGS=n`, the minimal log severity is not persisted on the device. In this case, the Spotflow device module uses `SPOTFLOW_DEFAULT_SENT_LOG_LEVEL` from the device start and loads the latest value set from the portal after it connects to the Spotflow cloud. As a result, logs might not be filtered in the way you want until the connection is established. Therefore, in order to persist the minimal log severity to the device and use it right from the device start, enable the [Zephyr settings subsystem](https://docs.zephyrproject.org/latest/services/storage/settings/index.html) and one of its backends, such as the [non-volatile storage (NVS)](https://docs.zephyrproject.org/latest/services/storage/nvs/nvs.html) backend: ```dotenv title="prj.conf" CONFIG_FLASH=y CONFIG_FLASH_MAP=y CONFIG_NVS=y CONFIG_SETTINGS=y CONFIG_SETTINGS_NVS=y ``` `CONFIG_SPOTFLOW_SETTINGS=y` will be set by default when `CONFIG_SETTINGS=y` is set. # Crash reports with Zephyr (/guides/zephyr/crash-reports-zephyr) This guide explains how to send crash reports and core dumps from devices running Zephyr RTOS using the Spotflow device module and analyze crashes and core dumps in the Spotflow web application. ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with Zephyr 3.7, 4.1, 4.2, 4.3, or 4.4, * have established a connection to the internet from the device. When creating a new west workspace, make sure to use a supported Zephyr version, for example, using `west init --mr v4.3.0`. By default, west init checks out the main branch, which may contain changes breaking the build. Alternatively, you can follow the [Quickstart: Zephyr Integration Guide](/quickstart/zephyr) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. Before you start integrating your device with Spotflow, make sure you: * have development environment with nRF Connect SDK v3.0.0 or later (v3.2.4 recommended), * have established a connection to the internet from the device. Alternatively, you can follow the [Nordic nRF Connect SDK Integration Guide](/quickstart/nordic-nrf-connect) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a west dependency to your `west.yml` file. ```yaml title="west.yml" manifest: projects: - name: spotflow revision: main path: modules/lib/spotflow url: https://github.com/spotflow-io/device-sdk ``` Then, install the west dependencies using the following command: ```bash west update ``` ## Update Kconfig Configuration [#update-kconfig-configuration] Enable Spotflow device module to collect core dumps and configure Zephyr's core dump subsystem to include all available crash information or only selected parts: ```dotenv title="prj.conf" CONFIG_SPOTFLOW=y # Enable Spotflow Module CONFIG_SPOTFLOW_DEVICE_ID="zephyr-device-001" # Set unique identifier of your device CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" # Set your Spotflow ingest key # Enable Spotflow coredump collection CONFIG_SPOTFLOW_COREDUMPS=y # Configure what should be included in the coredump CONFIG_DEBUG_COREDUMP_THREADS_METADATA=y CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_THREADS=y ``` The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys on the [ingest keys](https://app.spotflow.io/ingest-keys) page. See details about the [Spotflow-specific options](#kconfig-options) later in this doc. See Zephyr docs for details regarding options such as [`DEBUG_COREDUMP_MEMORY_DUMP_*`](https://docs.zephyrproject.org/latest/kconfig.html#!DEBUG_COREDUMP_MEMORY_DUMP) or [`DEBUG_COREDUMP_THREAD*`](https://docs.zephyrproject.org/latest/kconfig.html#!DEBUG_COREDUMP_THREAD). ## Define core dump flash partition [#define-core-dump-flash-partition] When the crash occurs, the core dump file needs to be stored somewhere before it can be uploaded to Spotflow after reboot. We make use of Zephyr core dump subsystem's ability to store the file into a flash partition. ```dts title="boards/nrf7002dk_nrf5340_cpuapp_ns.overlay" &flash0 { /* Although partitions are defined using Partition Manager, Zephyr core dumps require to * have the core dump partition defined here as well. */ partitions { /* Reserve last 64 KiB of flash for core dumps */ coredump_partition: partition@f0000 { label = "coredump-partition"; reg = <0x000f0000 DT_SIZE_K(64)>; }; }; }; ``` This step might vary across target boards. See our fully working [samples on GitHub](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/samples/coredumps) . Refer to [this page](/guides/nordic-nrf-connect/zephyr-differences#core-dump-partition-configuration) if you are using nRF Connect SDK. ## (Recommended) Upload ELF file with symbols [#recommended-upload-elf-file-with-symbols] To unlock advanced crash analysis features, you can upload the ELF file containing debug symbols to Spotflow. The core dump will be automatically linked based on **build id** embedded into both ELF file and core dump file automatically by Spotflow. The symbols will be used to decode stack traces and variable names in the Spotflow web application. See dedicated [Firmware Management](/fundamentals/firmware-management) page for information about managing firmwares and symbol files. ## Wait for (or simulate) a crash [#wait-for-or-simulate-a-crash] Core dump file is automatically created by Zephyr when a fatal error occurs and sent to cloud for analysis by Spotflow device module, immediately after the device reboots. By default, Spotflow device module automatically reboots the device after a fatal error. To simulate the crash, you can use for example Zephyr's [`k_oops`](https://docs.zephyrproject.org/latest/doxygen/html/kernel_8h.html#abde5aa8ca5e64a045b25b88f91370dcd) function which will terminate current thread fatally: ```c static void simulate_crash() { LOG_INF("Simulating crash. Going to oops."); k_oops(); } ``` ## Analyze Crash Reports in the Web Application [#analyze-crash-reports-in-the-web-application] Once your device is integrated, you can analyze crash reports and core dumps in the web application. You can list the crash reports in the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter crash reports by their content, device ID and other metadata. ### Automatic Analysis [#automatic-analysis] We automatically analyse each crash report to give you a quick insight into the issue. For precise results, we extract data directly from the core dump (stack traces, register values, etc.), decompile the firmware binary, and review relevant documentation. Our proprietary AI agent then investigates the crash, leveraging all available data. The result is available in the detail view of each crash report. The full analysis includes a detailed description of the root cause and suggestions for fixing the issue. Also, for complete transparency, we include references to all documentation pages used, allowing you to explore further. We also provide all the raw data extracted from the core dump, so you can dive deeper into the issue if needed. This includes: * Stack traces of one or more threads. * Register values and local variables for individual stack frames. If a register value is available, it can be also casted to several types. * State of global variables at the time of the crash. ## How the Device Module Works [#how-the-device-module-works] The Spotflow device module integrates with Zephyr's native [core dump subsystem](https://docs.zephyrproject.org/latest/services/debugging/coredump.html) to provide a familiar development experience: * When Spotflow device module is configured for collecting core dumps, storing of the core dump files on flash partition is automatically enabled via [`DEBUG_COREDUMP_BACKEND_FLASH_PARTITION`](https://docs.zephyrproject.org/latest/kconfig.html#!DEBUG_COREDUMP_BACKEND_FLASH_PARTITION) Zephyr's Kconfig option. * The core dump file is automatically created on the flash partition when a fatal error occurs. * After each reboot, Spotflow device module checks the presence of a core dump file on the flash partition. If found, it uploads it to Spotflow. * After successful upload, the core dump file is erased from the flash partition. * Only Zephyr-native APIs are used. Core dump file is automatically erased from the flash partition after upload. ### Automatic restart [#automatic-restart] By default, Zephyr RTOS halts the system unconditionally after a fatal error (e.g. by entering an infinite loop, depending on the architecture). This behavior is not suitable for production use and can be overridden by providing custom implementation of fatal error policy handler (the `k_sys_fatal_error_handler` function). Spotflow device module, by default, overrides the Zephyr RTOS's standard `k_sys_fatal_error_handler` implementation in order to automatically reboots the device after a fatal error, allowing the core dump file to be uploaded. Users can customize this behavior by first disabling the Spotflow's handler via Kconfig option [`SPOTFLOW_USE_DEFAULT_REBOOT_HANDLER`](#Kconfig-options) while providing their own implementation of the `k_sys_fatal_error_handler` function at the same time. With Spotflow device module, the device will automatically reboot after a fatal error. ### ELF files, debugging symbols and build ID [#elf-files-debugging-symbols-and-build-id] To provide the best possible crash analysis experience, debugging symbols associated with the code running on devices are needed. Spotflow allows users to upload and manage Zephyr ELF files containing the debugging symbols (see [Firmware Management](/fundamentals/firmware-management) for details). To link ELF file containing debugging symbols with an uploaded core dump, the **build ID** is used. The build ID is a [GNU Build ID](https://grok.com/share/c2hhcmQtMw%3D%3D_b1d1cef1-8147-4f60-8318-6dd6f3165595)-style identifier that uniquely identifies an ELF file by its relevant parts (ELF sections that influence the runtime behavior such as code or global data, excluding sections like debugging symbols, hashed by SHA-1). The build ID is embedded into the Zephyr ELF file automatically by the Spotflow device module during build. More specifically, it is stored in the ELF file as a Zephyr custom [Binary Descriptor](https://docs.zephyrproject.org/latest/services/binary_descriptors/index.html) with ID `0x5f0` and length of 20 bytes. For Zephyr targets that do not support Binary Descriptors, the build ID is stored as a `bindesc_entry_spotflow_build_id` symbol in the ELF file. In case that build ID is not available, the ELF files can still be linked manually to specific core dumps. ### Core dumps vs. logging [#core-dumps-vs-logging] Spotflow core dumps and [logging](/guides/zephyr/logging-zephyr) are complementary features, seamlessly working side by side. After reboot, the core dump file upload has a priority over the logs. ## Kconfig options [#kconfig-options] ```dotenv title="prj.conf" CONFIG_SPOTFLOW_COREDUMPS=n ``` Set `SPOTFLOW_COREDUMPS` option to `y` to enable Spotflow device module to collect core dumps. This also enables automatic restarts (via `SPOTFLOW_USE_DEFAULT_REBOOT_HANDLER` option.) ```dotenv title="prj.conf" CONFIG_SPOTFLOW_COREDUMPS_CHUNK_SIZE=1024 ``` Use `SPOTFLOW_COREDUMPS_CHUNK_SIZE` option to set the size of the chunk of core dump file (in bytes) to be uploaded to Spotflow in one go. Larger chunks are more efficient but require more stable connection for the upload to succeed. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_USE_DEFAULT_REBOOT_HANDLER=y ``` The default behavior of automatic restarts can be disabled by setting `SPOTFLOW_USE_DEFAULT_REBOOT_HANDLER` option to `n` or additionally customized by providing a custom implementation of the `k_sys_fatal_error_handler` function. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_COREDUMPS_PROCESSING_LOG_LEVEL=3 ``` Use the `SPOTFLOW_COREDUMPS_PROCESSING_LOG_LEVEL` option to set the log level of the Spotflow processing backend code for core dumps. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_COREDUMPS_BACKEND_QUEUE_SIZE=16 ``` The `SPOTFLOW_COREDUMPS_BACKEND_QUEUE_SIZE` defines size of the queue used by Spotflow Coredump backend to store coredump chunks before sending them to the cloud. See [KConfig for core dumps](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/coredumps/KConfig) for details about options above. See [Advanced configuration options for Zephyr](/guides/zephyr/advanced-config-zephyr) for additional options not directly related to core dumps. ## Learn more [#learn-more] # Logging with Zephyr (/guides/zephyr/logging-zephyr) This guide explains how to enable log collection on devices running Zephyr RTOS using the Spotflow device module and analyze logs in the Spotflow web application. ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with Zephyr 3.7, 4.1, 4.2, 4.3, or 4.4, * have established a connection to the internet from the device. When creating a new west workspace, make sure to use a supported Zephyr version, for example, using `west init --mr v4.3.0`. By default, west init checks out the main branch, which may contain changes breaking the build. Alternatively, you can follow the [Quickstart: Zephyr Integration Guide](/quickstart/zephyr) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. Before you start integrating your device with Spotflow, make sure you: * have development environment with nRF Connect SDK v3.0.0 or later (v3.2.4 recommended), * have established a connection to the internet from the device. Alternatively, you can follow the [Nordic nRF Connect SDK Integration Guide](/quickstart/nordic-nrf-connect) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a west dependency to your `west.yml` file. ```yaml title="west.yml" manifest: projects: - name: spotflow revision: main path: modules/lib/spotflow url: https://github.com/spotflow-io/device-sdk ``` Then, install the west dependencies using the following command: ```bash west update ``` ## Update Kconfig Configuration [#update-kconfig-configuration] To enable Spotflow logging, you need to add the following lines to your `prj.conf` file: ```dotenv title="prj.conf" CONFIG_LOG=y # Enable Zephyr Logging CONFIG_SPOTFLOW=y # Enable Spotflow Module CONFIG_SPOTFLOW_DEVICE_ID="zephyr-device-001" # Set unique identifier of your device CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" # Set your Spotflow ingest key ``` The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys on the [ingest keys](https://app.spotflow.io/ingest-keys) page. ## Use Zephyr logging macros [#use-zephyr-logging-macros] After installing the Spotflow device module and enabling logging via Kconfig configuration, you can use standard Zephyr logging macros to send the logs: ```c title="src/main.c" #include // Register the logging module LOG_MODULE_REGISTER(main); void main(void) { // Ensure the device has an active internet connection. LOG_INF("Device has booted up successfully."); for (int i = 1; i <= 5; i++) { int reading = i * 10; LOG_INF("Sensor reading %d: %d units", i, reading); if (reading > 30) LOG_WRN("Reading is above normal. Value: %d.", reading); } LOG_INF("Device shutting down."); } ``` ## Analyze Logs in the Web Application [#analyze-logs-in-the-web-application] Once your device is integrated and sending logs, you can analyze them in the web application. The main entry point is the [Events](https://app.spotflow.io/) page, which gives you a comprehensive view of all events collected from your devices. There, you can filter logs by their content, device ID, severity, and other metadata. You can click on individual log messages to see their details, and it is possible to drill down into specific events by matching their metadata. ## (Optional) Adjust Device's Minimal Log Severity [#optional-adjust-devices-minimal-log-severity] You can choose to set the minimum log severity for the logs that the device sends to Spotflow. This can significantly reduce the device's bandwidth usage and can be configured remotely in the Spotflow web application. Please note that logs generated by the `printk()` function, unlike logging macros such as `LOG_INF` , do not have a severity level assigned and are never filtered out. ## Open the Device in the Web Application [#open-the-device-in-the-web-application] Select your device on the [Devices](https://app.spotflow.io/devices) page, which gives you a comprehensive view of all connected devices. In the device detail, you can view the current minimum log severity reported by the device and set it to a desired value. ## Session metadata labels [#session-metadata-labels] To attach custom key-value records to all logs sent during a connection, configure [session metadata labels](/guides/zephyr/advanced-config-zephyr#session-metadata-labels). ## How the Device Module Works [#how-the-device-module-works] The Spotflow device module integrates with Zephyr's native [logging subsystem](https://docs.zephyrproject.org/latest/services/logging/index.html) to provide a familiar development experience. Instead of replacing your existing logging workflow, it extends it by implementing a custom logging backend that captures logs from standard Zephyr macros and transmits them to Spotflow. **Key benefits of this architecture:** * **Zero code changes**: Continue using `LOG_DBG()`, `LOG_INF()`, `LOG_WRN()`, and `LOG_ERR()` macros as before. * **Local debugging**: Our device module can run alongside other logging backends, allowing you to see logs in your local console while also transmitting them to Spotflow. These logs will be consumed by Spotflow and your existing logging backends (such as the built-in UART backend from Zephyr), with no additional code required. ## Handling Network Interruptions [#handling-network-interruptions] The device module is designed to handle network interruptions gracefully. When the connection is unavailable, **logs are stored in a circular buffer**. That is, when the buffer is full, the oldest logs are overwritten with the new ones. Once the connection is restored, buffered logs are automatically sent to Spotflow. ## Support for Constrained Devices [#support-for-constrained-devices] To minimize the memory footprint, the device module exposes number of configuration options to tune the buffer size and other parameters. See the available [Kconfig options](#kconfig-options) below. ## Reliability of data delivery [#reliability-of-data-delivery] Spotflow is using MQTT as the underlying [transport protocol](/fundamentals/monitoring/logging#transport-protocol) for log ingestion. Most embedded devices running Zephyr are significantly resource-constrained which has several implications: 1. To minimize the footprint of the Spotflow device module, **MQTT QoS level 0** is used which has minimal overhead but at the cost of weaker delivery guarantees. Although MQTT itself is TCP-based protocol and TCP guarantees transport-level delivery, with QoS 0, some messages still can be lost in case of client or broker failures and similar issues. This should not be an issue for most observability use-cases. 2. In case of a network interruption or slow down, the device module buffers not yet sent logs in a circular buffer. However, this buffer has limited size and once it is full, the oldest logs are overwritten with the new ones. Size of this buffer can be via [Kconfig options](#kconfig-options). If needed, you can create your [custom integration](/guides/mqtt/logging-mqtt) directly with our MQTT broker to tailor the solution to your specific resource constraints vs. reliability tradeoff. Spotflow MQTT broker supports all QoS levels from 0 to 2 and is designed to handle even high throughput log streams. ## Kconfig options [#kconfig-options] ```dotenv title="prj.conf" CONFIG_SPOTFLOW_LOG_BACKEND=y ``` To explicitly enable or disable Spotflow logging backend (e.g. in cases when only [crash reports](/guides/zephyr/crash-reports-zephyr) are required), use `SPOTFLOW_LOG_BACKEND`. It is enabled by default. ```dotenv title="prj.conf" SPOTFLOW_LOGS_PROCESSING_LOG_LEVEL=3 ``` To configure logging level for the Spotflow backend code, use `SPOTFLOW_LOGS_PROCESSING_LOG_LEVEL` option with one of the following values: * `0 = OFF`, do not write. * `1 = ERROR`, only write LOG\_ERR. * `2 = WARNING`, write LOG\_WRN and lower. * `3 = INFO`, write LOG\_INF and lower. * `4 = DEBUG`, write LOG\_DBG and lower. By default, INFO is used. ```dotenv title="prj.conf" CONFIG_SPOTFLOW_LOG_BACKEND_QUEUE_SIZE=64 CONFIG_SPOTFLOW_LOG_BUFFER_SIZE=512 CONFIG_SPOTFLOW_CBOR_LOG_MAX_LEN=1024 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW=6144 ``` You can configure these options to adjust performance and resource usage. See [KConfig for logging](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/logging/KConfig) for details about all options above. See [Advanced configuration options for Zephyr](/guides/zephyr/advanced-config-zephyr) for additional options that influence not only logging. ## Learn more [#learn-more] # Metrics with Zephyr (/guides/zephyr/metrics-zephyr) This guide explains how to gather metrics from devices running Zephyr RTOS using the Spotflow device module. You can collect system metrics automatically and report custom application metrics using the same SDK. ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with Zephyr 3.7, 4.1, 4.2, 4.3, or 4.4, * have established a connection to the internet from the device. When creating a new west workspace, make sure to use a supported Zephyr version, for example, using `west init --mr v4.3.0`. By default, west init checks out the main branch, which may contain changes breaking the build. Alternatively, you can follow the [Quickstart: Zephyr Integration Guide](/quickstart/zephyr) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. Before you start integrating your device with Spotflow, make sure you: * have development environment with nRF Connect SDK v3.0.0 or later (v3.2.4 recommended), * have established a connection to the internet from the device. Alternatively, you can follow the [Nordic nRF Connect SDK Integration Guide](/quickstart/nordic-nrf-connect) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a west dependency to your `west.yml` file. ```yaml title="west.yml" manifest: projects: - name: spotflow revision: main path: modules/lib/spotflow url: https://github.com/spotflow-io/device-sdk ``` Then, install the west dependencies using the following command: ```bash west update ``` ## Enable metrics in Kconfig [#enable-metrics-in-kconfig] Add the following options to your `prj.conf`: ```dotenv title="prj.conf" CONFIG_SPOTFLOW=y # Enable Spotflow Module CONFIG_SPOTFLOW_DEVICE_ID="zephyr-device-001" # Set unique identifier of your device CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" # Set your Spotflow ingest key # Enable Spotflow metrics collection CONFIG_SPOTFLOW_METRICS=y ``` ## Enable system metrics auto-collection [#enable-system-metrics-auto-collection] To collect system telemetry automatically, enable: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_METRICS_SYSTEM=y ``` The following system metrics are collected: | Metric | How it is collected | | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Heap Free Bytes (`heap_free_bytes`) | Sampled with `sys_heap_runtime_stats_get()` from `_system_heap` and reported from `free_bytes`. | | Heap Allocated Bytes (`heap_allocated_bytes`) | Sampled with `sys_heap_runtime_stats_get()` from `_system_heap` and reported from `allocated_bytes`. | | CPU Utilization Percent (`cpu_utilization_percent`) | Sampled with `cpu_load_get(true)` and converted from per-mille to percent. | | Thread Stack Free Bytes (`thread_stack_free_bytes`) | Sampled per tracked thread with `k_thread_stack_space_get()`. | | Thread Stack Used Percent (`thread_stack_used_percent`) | Derived per tracked thread from `thread->stack_info.size` and `k_thread_stack_space_get()`. | | Network TX Bytes (`network_tx_bytes`) | Sampled per active interface via `net_if_foreach(...)` from `iface->stats.bytes.sent`. | | Network RX Bytes (`network_rx_bytes`) | Sampled per active interface via `net_if_foreach(...)` from `iface->stats.bytes.received`. | | MQTT Connection State (`connection_mqtt_connected`) | Event-driven and reported when MQTT state changes through `spotflow_metrics_system_report_connection_state(bool)`. | | Boot Reset Cause (`boot_reset`) | Reported once on boot using `hwinfo_get_reset_cause()`, then reset cause is cleared with `hwinfo_clear_reset_cause()`. | By default, stack metrics are collected for all threads (`CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_ALL_THREADS=y`). If you need to limit this, disable that option and register only selected threads with `spotflow_metrics_system_enable_thread_stack(...)`. Check implementation details in the [SDK system metrics folder](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/src/metrics/system). ## Analyze system metrics in dashboards [#analyze-system-metrics-in-dashboards] After enabling system metrics, open the Device Dashboard to inspect device vitals, resource usage, and connectivity trends: * [Device Dashboard](/fundamentals/monitoring/dashboards#device-dashboard) ## Report custom metrics [#report-custom-metrics] Custom metrics let you track application-specific values such as sensor readings, request latencies, or business counters. The SDK handles aggregation, encoding, and transmission to Spotflow automatically. `CONFIG_SPOTFLOW_METRICS_SYSTEM=y` is not required for custom metrics. You can use custom metrics independently or alongside system metrics. ### Include the metrics header [#include-the-metrics-header] Include the metrics API header in your application source file: ```c title="src/main.c" #include "metrics/spotflow_metrics_backend.h" ``` This header provides functions for registering and reporting metrics. It transitively includes the type definitions and registration API. ### Register metrics [#register-metrics] Before reporting values, each metric must be registered with a name and an aggregation interval. Registration returns a handle that is used for all subsequent reports. Metrics can be either **integer** (`int64_t`) or **float** (`float`), and optionally support **labels** for dimensional breakdowns. #### Label-less metrics [#label-less-metrics] A label-less metric tracks a single time series: ```c title="src/main.c" static struct spotflow_metric_int *counter_metric; static struct spotflow_metric_float *temperature_metric; int rc; /* Integer metric aggregated over 1 minute */ rc = spotflow_register_metric_int( "app_counter", SPOTFLOW_AGG_INTERVAL_1MIN, &counter_metric); /* Float metric with no aggregation (each value sent immediately) */ rc = spotflow_register_metric_float( "temperature_celsius", SPOTFLOW_AGG_INTERVAL_NONE, &temperature_metric); ``` #### Labeled metrics [#labeled-metrics] A labeled metric tracks multiple time series distinguished by label key-value pairs. At registration, specify the maximum number of unique label combinations (`max_timeseries`) and the maximum number of labels per report (`max_labels`): ```c title="src/main.c" static struct spotflow_metric_float *lock_duration_metric; rc = spotflow_register_metric_float_with_labels( "lock_operation_duration_ms", SPOTFLOW_AGG_INTERVAL_1MIN, 6, /* max_timeseries: e.g. 2 operations x 3 methods */ 2, /* max_labels */ &lock_duration_metric); ``` Each unique combination of label values is tracked as a separate time series with its own aggregation state. For context that remains constant during a connection, use [session metadata labels](/guides/zephyr/advanced-config-zephyr#session-metadata-labels) instead of repeating the labels on every metric report. Labels attached directly to a metric take precedence when they use the same key as a session metadata label. #### Aggregation intervals [#aggregation-intervals] The aggregation interval controls how long values are accumulated before being sent: | Constant | Interval | Behavior | | ----------------------------- | -------------- | ----------------------------------------------- | | `SPOTFLOW_AGG_INTERVAL_NONE` | No aggregation | Each reported value is sent immediately | | `SPOTFLOW_AGG_INTERVAL_1MIN` | 1 minute | Values are aggregated into sum, count, min, max | | `SPOTFLOW_AGG_INTERVAL_1HOUR` | 1 hour | Values are aggregated into sum, count, min, max | | `SPOTFLOW_AGG_INTERVAL_1DAY` | 1 day | Values are aggregated into sum, count, min, max | Metric names are normalized before registration: alphanumeric characters are lowercased, dashes, dots, and spaces are converted to underscores, and other characters are removed. For example, `"My-Metric.Name"` becomes `"my_metric_name"`. ### Report metric values [#report-metric-values] After registration, report values at any time from any thread. #### Label-less values [#label-less-values] ```c title="src/main.c" /* Report an integer value */ spotflow_report_metric_int(counter_metric, 42); /* Report a float value */ spotflow_report_metric_float(temperature_metric, 23.5f); ``` #### Labeled values [#labeled-values] Attach labels to each report using an array of `struct spotflow_label`: ```c title="src/main.c" struct spotflow_label labels[] = { { .key = "operation", .value = "unlock" }, { .key = "method", .value = "nfc" } }; spotflow_report_metric_float_with_labels( lock_duration_metric, 120.5f, labels, 2); ``` Label keys can be up to 15 characters and values up to 31 characters (excluding the null terminator). #### Events [#events] For point-in-time occurrences where you only need to record that something happened, use the event API. An event is equivalent to reporting an integer value of `1` with no aggregation: ```c title="src/main.c" static struct spotflow_metric_int *door_opened_metric; /* Register with no aggregation */ spotflow_register_metric_int( "door_opened", SPOTFLOW_AGG_INTERVAL_NONE, &door_opened_metric); /* Report that the event occurred */ spotflow_report_event(door_opened_metric); ``` Events also support labels: ```c title="src/main.c" static struct spotflow_metric_int *error_metric; spotflow_register_metric_int_with_labels( "app_error", SPOTFLOW_AGG_INTERVAL_NONE, 10, /* max_timeseries */ 1, /* max_labels */ &error_metric); struct spotflow_label error_labels[] = { { .key = "code", .value = "timeout" } }; spotflow_report_event_with_labels(error_metric, error_labels, 1); ``` Custom metrics can be visualized in [Custom Dashboards](/using-spotflow/guides/custom-dashboards). ## Analyze custom metrics in dashboards [#analyze-custom-metrics-in-dashboards] After reporting custom metrics, open Custom Dashboards to visualize and analyze your application-specific data: * [Custom Dashboards](/using-spotflow/guides/custom-dashboards) ## (Optional) Tune collection and aggregation [#optional-tune-collection-and-aggregation] ### System metrics [#system-metrics] Use these options to control sampling cadence and the aggregation window before upload: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=10 CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL=60 ``` * `CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL` defines how often metrics are sampled (in seconds). * `CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL` defines how long samples are aggregated before sending. * Aggregation interval supports `0` (no aggregation), `60` (1 minute), `3600` (1 hour), and `86400` (1 day). * With aggregation interval `0`, each sample is sent immediately. * Lower collection interval gives finer time resolution but increases local processing overhead. * Higher aggregation interval reduces message frequency and network traffic by combining more samples. ### Custom metrics [#custom-metrics] The following Kconfig options control memory allocation and limits for custom metrics: ```dotenv title="prj.conf" CONFIG_SPOTFLOW_METRICS_MAX_REGISTERED=32 # Max number of registered metrics CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC=4 # Max labels per metric report CONFIG_SPOTFLOW_METRICS_QUEUE_SIZE=16 # Message queue size before transmission CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS=8192 # Heap memory budget for custom metrics (bytes) ``` * `CONFIG_SPOTFLOW_METRICS_MAX_REGISTERED` limits how many metrics can be registered simultaneously (both system and custom metrics share this pool). * `CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC` limits how many label key-value pairs can be attached to a single report. * `CONFIG_SPOTFLOW_METRICS_QUEUE_SIZE` controls how many encoded messages are buffered before MQTT transmission. The default is `16`, or `64` when system metrics are also enabled. * `CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS` sets the heap budget for custom metric internals. Each label-less metric uses approximately 310 bytes, while a labeled metric with 4 time series uses approximately 1 KB. ## Reference repository materials [#reference-repository-materials] * [Metrics sample (Zephyr)](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/samples/metrics) * [Metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/metrics/Kconfig) * [System metrics Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/metrics/system/Kconfig) ## Learn more [#learn-more] # Over-the-air (OTA) updates of external MCUs (/guides/zephyr/ota-external-mcu-zephyr) This guide explains how to set up the Spotflow device module to perform over-the-air (OTA) updates of external MCUs connected to your Zephyr device. If you want to automatically update the firmware that runs the module itself (the *main* firmware), see [Over-the-air (OTA) updates with Zephyr](/guides/zephyr/ota-zephyr). To support both scenarios in one application, see the [notes below](#advanced-combine-automatic-and-manual-handling-of-ota-updates). ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with Zephyr 3.7, 4.1, 4.2, 4.3, or 4.4, * have established a connection to the internet from the device. When creating a new west workspace, make sure to use a supported Zephyr version, for example, using `west init --mr v4.3.0`. By default, west init checks out the main branch, which may contain changes breaking the build. Alternatively, you can follow the [Quickstart: Zephyr Integration Guide](/quickstart/zephyr) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. Before you start integrating your device with Spotflow, make sure you: * have development environment with nRF Connect SDK v3.0.0 or later (v3.2.4 recommended), * have established a connection to the internet from the device. Alternatively, you can follow the [Nordic nRF Connect SDK Integration Guide](/quickstart/nordic-nrf-connect) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. OTA updates of external MCUs additionally require: * A persistent Zephyr [settings](https://docs.zephyrproject.org/latest/services/storage/settings/index.html) backend (typically [NVS](https://docs.zephyrproject.org/latest/services/storage/nvs/nvs.html)) so that the update state and results survive reboot. ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a west dependency to your `west.yml` file. ```yaml title="west.yml" manifest: projects: - name: spotflow revision: main path: modules/lib/spotflow url: https://github.com/spotflow-io/device-sdk ``` Then, install the west dependencies using the following command: ```bash west update ``` ## Enable OTA updates in Kconfig [#enable-ota-updates-in-kconfig] Add the following to your `prj.conf`: ```dotenv title="prj.conf" CONFIG_SPOTFLOW=y CONFIG_SPOTFLOW_DEVICE_ID="zephyr-device-001" CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" CONFIG_SPOTFLOW_OTA=y # Automatic main firmware handling is not needed when updating only external MCUs CONFIG_SPOTFLOW_OTA_AUTO_HANDLE_MAIN_FIRMWARE=n # Persistent storage for OTA update attempt state and results CONFIG_SETTINGS=y CONFIG_NVS=y CONFIG_SETTINGS_NVS=y ``` ## Handle firmware updates [#handle-firmware-updates] Implement `spotflow_on_handle_firmware_update()`. The device module calls it for each received firmware update. The recommended way to download the firmware image is to use the Spotflow downloader: ```c #include static SPOTFLOW_DEFINE_DOWNLOADER(external_mcu_downloader); static void write_external_mcu_block(const struct spotflow_artifact_block *block, struct spotflow_downloader *downloader, void *ctx) { /* Write block->data to the external MCU. */ } enum spotflow_ota_result spotflow_on_handle_firmware_update(const struct spotflow_firmware_info *info) { int ret = spotflow_download_artifact(&external_mcu_downloader, info->download_request, write_external_mcu_block, NULL); if (ret < 0) { return SPOTFLOW_OTA_RESULT_FAILED; } /* Verify and activate the firmware on the external MCU. */ return SPOTFLOW_OTA_RESULT_SUCCEEDED; } ``` The `info` parameter contains the details of the firmware to download. Its field `slug` contains the unique identifier of the firmware that you can set when creating the firmware in the portal. You can use slugs to distinguish between firmware updates for different external MCUs. The `download_request` field contains the URL and OTA secret for the firmware image, and you can pass it directly to `spotflow_download_artifact()`. See [Firmware management](/fundamentals/firmware-management) for more details about firmware slugs, versions, symbol files, and more. The implementation of `spotflow_on_handle_firmware_update()` must always return a terminal result: `SPOTFLOW_OTA_RESULT_SUCCEEDED`, `SPOTFLOW_OTA_RESULT_FAILED`, or `SPOTFLOW_OTA_RESULT_CANCELED`. The function runs in the update worker thread, so it can block as needed while your application performs the update. The device module calls `spotflow_on_handle_firmware_update()` for each firmware update that still requires processing. If the device resets before the result is persisted, it can call the function again for the same attempt and firmware after reconnecting; make the handler idempotent or persist application-specific progress. ## Deploy OTA update [#deploy-ota-update] Once the device is online and connected to Spotflow MQTT, create a deployment in the portal as described in [Deploy Over-the-Air (OTA) Updates](/using-spotflow/guides/ota). Your implementation of `spotflow_on_handle_firmware_update()` will be called for each firmware in the deployment, and all results will be reported to the cloud. Because you disabled `CONFIG_SPOTFLOW_OTA_AUTO_HANDLE_MAIN_FIRMWARE` in Kconfig, each firmware update will be passed to your implementation of `spotflow_on_handle_firmware_update()`, regardless of whether it is marked as main. Still, it is a good practice to mark firmware as main only when it runs the Spotflow device module. ## Advanced: Handle cancellation [#advanced-handle-cancellation] When a device is currently processing an update attempt and you stop the deployment or supersede it with a new one, the cloud sends a cancellation request to the device. The Spotflow device module considers the cancellation *actionable* only while the first firmware update in the attempt is still in progress. After that update finishes, there is nothing useful to cancel: * If it failed or was canceled, the module has already canceled all remaining firmware updates. * If it succeeded, the module ignores a later cancellation so that it does not interrupt the remaining update. When the module receives an actionable cancellation, it calls `spotflow_on_update_canceled()`. Your handler can use it to stop ongoing work, for example by canceling the download started by `spotflow_on_handle_firmware_update()`: ```c void spotflow_on_update_canceled(void) { spotflow_cancel_download(&external_mcu_downloader); } ``` Because `spotflow_on_update_canceled()` runs on the system workqueue, do not perform blocking work in it. You can also poll for actionable cancellation using `spotflow_is_update_canceled()`. Report successful cancellation by returning `SPOTFLOW_OTA_RESULT_CANCELED` from `spotflow_on_handle_firmware_update()`: ```c enum spotflow_ota_result spotflow_on_handle_firmware_update(const struct spotflow_firmware_info *info) { int ret = spotflow_download_artifact(&external_mcu_downloader, info->download_request, write_external_mcu_block, NULL); if (spotflow_is_update_canceled()) { return SPOTFLOW_OTA_RESULT_CANCELED; } if (ret < 0) { return SPOTFLOW_OTA_RESULT_FAILED; } /* Verify and activate the firmware on the external MCU. */ return SPOTFLOW_OTA_RESULT_SUCCEEDED; } ``` ## Advanced: Combine automatic and manual handling of OTA updates [#advanced-combine-automatic-and-manual-handling-of-ota-updates] You can combine both [automatic](/guides/zephyr/ota-zephyr) and manual handling of OTA updates in one application. Keep `CONFIG_SPOTFLOW_OTA_AUTO_HANDLE_MAIN_FIRMWARE=y` to let the SDK update the main firmware automatically, and still implement `spotflow_on_handle_firmware_update()` for external MCUs. The Spotflow device module processes firmware updates in the order specified when you create the deployment. If any firmware update fails, the module cancels the remaining firmware updates in the attempt. ## How device module works [#how-device-module-works] See [How device module works](/guides/zephyr/ota-zephyr#how-device-module-works) in the main Zephyr OTA update guide for the details of attempt processing, persistence, threading, security, and configuration. ## Reference repository materials [#reference-repository-materials] * Public API: [`spotflow/ota.h`](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/include/spotflow/ota.h), [`spotflow/downloader.h`](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/include/spotflow/downloader.h) * [Sample for OTA updates](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/samples/ota) * [Kconfig options for OTA updates](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/ota/Kconfig) ## Learn more [#learn-more] # Over-the-air (OTA) updates with Zephyr (/guides/zephyr/ota-zephyr) This guide explains how to set up the Spotflow device module to automatically perform over-the-air (OTA) updates of the firmware that runs the module itself (the *main* firmware). If you want to update firmware on external MCUs connected to your Zephyr device, see [Over-the-air (OTA) updates of external MCUs with Zephyr](/guides/zephyr/ota-external-mcu-zephyr). ## Prerequisites [#prerequisites] Before you start integrating your device with Spotflow, make sure you: * have development environment with Zephyr 3.7, 4.1, 4.2, 4.3, or 4.4, * have established a connection to the internet from the device. When creating a new west workspace, make sure to use a supported Zephyr version, for example, using `west init --mr v4.3.0`. By default, west init checks out the main branch, which may contain changes breaking the build. Alternatively, you can follow the [Quickstart: Zephyr Integration Guide](/quickstart/zephyr) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. Before you start integrating your device with Spotflow, make sure you: * have development environment with nRF Connect SDK v3.0.0 or later (v3.2.4 recommended), * have established a connection to the internet from the device. Alternatively, you can follow the [Nordic nRF Connect SDK Integration Guide](/quickstart/nordic-nrf-connect) to integrate Spotflow using sample application, instead of adding the `spotflow` module to an existing project. OTA updates additionally require: * A persistent Zephyr [settings](https://docs.zephyrproject.org/latest/services/storage/settings/index.html) backend (typically [NVS](https://docs.zephyrproject.org/latest/services/storage/nvs/nvs.html)) so that the update state and results survive reboot. * **MCUboot** enabled through [Zephyr sysbuild](https://docs.zephyrproject.org/latest/build/sysbuild/index.html). * Flash support (`CONFIG_FLASH`, `CONFIG_FLASH_MAP`, `CONFIG_STREAM_FLASH`) for automatic main-firmware handling. * A flash partition layout with an MCUboot secondary (update) slot. See the [MCUboot documentation](https://docs.mcuboot.com/readme-zephyr.html) for details. * Zephyr [binary descriptors](https://docs.zephyrproject.org/latest/services/binary_descriptors/index.html) so that the Spotflow [build ID](/fundamentals/monitoring/crash-reports#build-ids) can be embedded in the firmware image. See the dependencies of [`CONFIG_BINDESC`](https://docs.zephyrproject.org/latest/kconfig.html#CONFIG_BINDESC). ## Install Spotflow Device Module [#install-spotflow-device-module] Add the `spotflow` module as a west dependency to your `west.yml` file. ```yaml title="west.yml" manifest: projects: - name: spotflow revision: main path: modules/lib/spotflow url: https://github.com/spotflow-io/device-sdk ``` Then, install the west dependencies using the following command: ```bash west update ``` ## Enable OTA updates in Kconfig [#enable-ota-updates-in-kconfig] Add the following to your `prj.conf`: ```dotenv title="prj.conf" CONFIG_SPOTFLOW=y CONFIG_SPOTFLOW_DEVICE_ID="zephyr-device-001" CONFIG_SPOTFLOW_INGEST_KEY="{your-ingest-key}" CONFIG_SPOTFLOW_OTA=y # Persistent storage for OTA attempt state and results CONFIG_SETTINGS=y CONFIG_NVS=y CONFIG_SETTINGS_NVS=y # Required by automatic main-firmware handling CONFIG_FLASH=y CONFIG_FLASH_MAP=y CONFIG_STREAM_FLASH=y ``` Additionally, enable the MCUboot bootloader in your `sysbuild.conf` file: ```dotenv title="sysbuild.conf" SB_CONFIG_BOOTLOADER_MCUBOOT=y ``` ## Confirm main firmware image after reboot [#confirm-main-firmware-image-after-reboot] After MCUboot starts the new image in test mode, the image remains unconfirmed. Your application must validate that the new firmware works correctly and then call `spotflow_confirm_main_firmware_image()`: ```c #include #include static void confirm_if_main_firmware_is_valid(void) { struct spotflow_ota_main_firmware_state state; if (spotflow_get_main_firmware_update_state(&state) < 0) { return; } if (state.phase != SPOTFLOW_OTA_PHASE_UNCONFIRMED) { return; } /* Run your own validation first: hardware checks, self-tests, connectivity, etc. */ if (!application_self_test_passed()) { /* Rebooting without confirmation makes MCUboot roll back to the previous image. */ sys_reboot(SYS_REBOOT_COLD); } spotflow_confirm_main_firmware_image(&state); } ``` The call confirms the image with MCUboot and schedules reporting success to Spotflow. If the image is not confirmed before the next reboot, MCUboot rolls back to the previous image and the module reports the main firmware update as failed. ## Build and flash firmware [#build-and-flash-firmware] Build with sysbuild so the firmware image is signed for MCUboot and the bootloader is included in the build: ```bash west build --sysbuild --board --pristine --build-dir build-v1 ``` Flash the first version to the device together with MCUboot: ```bash west flash --build-dir build-v1 ``` Then prepare the version you want to deploy remotely and build it: ```bash west build --sysbuild -b --pristine -d build-v2 ``` ## Deploy OTA update [#deploy-ota-update] Once the device is online and connected to Spotflow MQTT, create a deployment in the portal as described in [Deploy Over-the-Air (OTA) Updates](/using-spotflow/guides/ota). For the firmware image, use the signed image from the second build. It should be located in the following path: ``` /build-v2//zephyr/zephyr.signed.bin ``` The default development key used to sign firmware images is publicly available. You should use a custom key in production. See [Signing Binaries](https://docs.zephyrproject.org/latest/build/signing/index.html) in Zephyr documentation. When you start the deployment, the Spotflow device module will download the firmware image to the secondary slot, schedule the new image to run once on the next reboot (while keeping the current image as a fallback), and reboot. After reboot, MCUboot will swap the images and the device will run the new firmware. After your code calls `spotflow_confirm_main_firmware_image()`, the module makes the image permanent with MCUboot and eventually reports success to Spotflow. If the new firmware is not confirmed before the next reboot, MCUboot will roll back to the previous image and the module will report the update as failed. ## Advanced: Customize main firmware update handling [#advanced-customize-main-firmware-update-handling] Although the device module handles the main firmware update automatically, your application code can observe and influence its progress. The main firmware update can be in one of the following phases: * `NOT_RUNNING`: No main firmware update in progress. * `PENDING_DOWNLOAD`: Download of the main firmware image is ready to start. * `DOWNLOADING`: Downloading the image into the MCUboot update slot. * `PENDING_UPGRADE`: The image has been downloaded and the module is about to perform the MCUboot test upgrade. * `PENDING_REBOOT`: The test upgrade has been requested and the device is about to reboot. * `UNCONFIRMED`: Device rebooted into the new image; awaiting confirmation. ### Progress observation [#progress-observation] Define the function `spotflow_on_main_firmware_update_progressed()` to be informed when the phase changes: ```c void spotflow_on_main_firmware_update_progressed(const struct spotflow_ota_main_firmware_state *state) { /* React to state->phase and state->result. For example, update UI, logs, or LEDs. */ } ``` When the phase changes to `NOT_RUNNING`, the module has finished the current update attempt and `state->result` contains its result. ### Pausing and resuming [#pausing-and-resuming] In many real-world applications, the device might not be ready to update firmware immediately. Because `spotflow_on_main_firmware_update_progressed()` is called on the OTA worker thread for most phases (`PENDING_DOWNLOAD`, `DOWNLOADING`, `PENDING_UPGRADE`, `PENDING_REBOOT`), blocking its execution effectively postpones the update: ```c void spotflow_on_main_firmware_update_progressed(const struct spotflow_ota_main_firmware_state *state) { if (state->phase == SPOTFLOW_OTA_PHASE_PENDING_REBOOT) { LOG_INF("Device will reboot in 10 seconds to finish the firmware update."); k_sleep(K_SECONDS(10)); } } ``` While this approach works for simple cases, it can be cumbersome in more complex scenarios, such as when the device needs confirmation from the user before continuing. You can use `spotflow_pause_main_firmware_update()` and `spotflow_resume_main_firmware_update()` for this purpose: ```c void spotflow_on_main_firmware_update_progressed(const struct spotflow_ota_main_firmware_state *state) { if (state->phase == SPOTFLOW_OTA_PHASE_PENDING_DOWNLOAD) { LOG_INF("New firmware update is ready to be downloaded. Press A to perform the update."); spotflow_pause_main_firmware_update(NULL); } } void on_button_a_pressed(void) { struct spotflow_ota_main_firmware_state state; if (spotflow_get_main_firmware_update_state(&state) < 0) { return; } if (!state.is_paused) { return; } if (spotflow_resume_main_firmware_update(&state) < 0) { return; } LOG_INF("Proceeding with the firmware update."); } ``` Pausing an update effectively puts the OTA worker thread to sleep in its current phase until it is resumed. While pausing an update in `spotflow_on_main_firmware_update_progressed()` is most common, you can do it from any thread. When your application pauses an update in the phase `DOWNLOADING`, the download will stop after the next firmware image block is received. After the update is resumed, the download will resume from the point where it was paused. This feature is useful, for example, when the device does not have stable network connectivity and needs to prioritize other communication when the signal is low. Pausing in `PENDING_REBOOT` postpones only the reboot initiated by the Spotflow device module. The MCUboot test upgrade has already been scheduled. If the device reboots for another reason while paused, the device still boots into the new image and continues with the `UNCONFIRMED` phase. An update **cannot** be paused in the phase `UNCONFIRMED`, because it is not under the Spotflow device module's control when your application confirms the image. ### Aborting [#aborting] In certain cases, you might want to allow your application to completely reject the update instead of just pausing it. You can do this by calling `spotflow_abort_main_firmware_update()`. This aborts the update and reports the main firmware update as failed. The function `spotflow_abort_main_firmware_update()` **cannot** be used in the phases `PENDING_REBOOT` and `UNCONFIRMED`. The only way to fail an update in these phases is to force a rollback by rebooting in the `UNCONFIRMED` phase. ## How device module works [#how-device-module-works] The Spotflow device module communicates with the cloud using CBOR messages over MQTT. See [Over-the-air (OTA) updates with MQTT](/guides/mqtt/ota-mqtt) for the protocol details. In order to store state across reboots, the module persists the results of the latest update attempt, installed versions of each firmware, and main firmware probation information. Each update attempt received by the device module contains one or more firmware updates. The module processes each firmware update as follows: * If the installed version of the firmware already matches the requested version, the module immediately reports success. * If the firmware is not main or `CONFIG_SPOTFLOW_OTA_AUTO_HANDLE_MAIN_FIRMWARE` is disabled, the module passes it to `spotflow_on_handle_firmware_update()` and reports the result to the cloud. [This guide](/guides/zephyr/ota-external-mcu-zephyr) explains how to use this API to handle firmware updates of external MCUs. * If the firmware is main and automatic handling is enabled, the module downloads the image over HTTPS into the MCUboot secondary slot, persists the expected build ID, requests a test upgrade, and reboots. On the next boot, it compares the running image build ID with the expected build ID: * If they match, the module waits in `UNCONFIRMED` until your application confirms the image. After confirmation, the module reports the main firmware update as succeeded. * If they do not match (because of a rollback or an unsuccessful swap), the module reports the main firmware update as failed. * When the firmware update does not succeed, the module cancels the remaining firmware updates in the attempt. The module also automatically informs the cloud about the last OTA update attempt received by the device. This information is sent whenever the device connects to the cloud as part of the [session metadata](/guides/mqtt/session-metadata). The cloud can use this information to repeat a device update attempt if the device was updated manually. See [this section](/guides/mqtt/session-metadata#over-the-air-ota-update-attempt-metadata) for more details. See the [implementation notes in the GitHub repository](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/docs/ota.md) for more details. ### Threading [#threading] The MQTT processing thread handles the communication with the cloud and a dedicated OTA update worker thread handles most of the work, such as downloading firmware images. Callbacks run on the following threads: | Callback | Thread | | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `spotflow_on_handle_firmware_update()` | update worker | | `spotflow_on_main_firmware_update_progressed()` | update worker in most cases; notification about the phases `NOT_RUNNING` and `UNCONFIRMED` might happen on the first thread that calls the public API of OTA updates | | `spotflow_on_update_canceled()` | Zephyr [system workqueue](https://docs.zephyrproject.org/latest/kernel/services/threads/workqueue.html#system-workqueue) | Prefer not to perform blocking work in `spotflow_on_update_canceled()` unless you are sure that it will not negatively impact other work items in the system workqueue. ### Security [#security] Firmware images are downloaded over HTTPS using the same TLS configuration as the Spotflow MQTT connection. Along with the URL, the device receives an **OTA secret** for each firmware image and uses it in the HTTP `Authorization` header. When handling firmware updates manually, do not log their URLs or secrets to reduce the risk of accidental exposure. ## Kconfig options [#kconfig-options] Summary of [OTA Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/ota/Kconfig): | Option | Default | Purpose | | ----------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `CONFIG_SPOTFLOW_OTA` | n | Enable support of Spotflow OTA updates | | `CONFIG_SPOTFLOW_OTA_AUTO_HANDLE_MAIN_FIRMWARE` | y | Automatically handle main firmware OTA updates | | `CONFIG_SPOTFLOW_OTA_LOG_LEVEL` | `CONFIG_SPOTFLOW_MODULE_DEFAULT_LOG_LEVEL` | Log level for OTA update messages | | `CONFIG_SPOTFLOW_OTA_MAX_ARTIFACTS` | 4 | Maximum number of firmware images in an OTA update attempt | | `CONFIG_SPOTFLOW_OTA_THREAD_STACK_SIZE` | 6144 | OTA update worker thread stack (includes TLS/HTTP download) | | `CONFIG_SPOTFLOW_OTA_HTTP_TIMEOUT_MS` | 120000 | Per-attempt time budget; blocking DNS, TCP, and TLS operations use separate timeouts and can exceed it | | `CONFIG_SPOTFLOW_OTA_DOWNLOAD_BUFFER_SIZE` | `CONFIG_IMG_BLOCK_BUF_SIZE` if image management is enabled; otherwise 512 | HTTP receive buffer size | | `CONFIG_SPOTFLOW_OTA_DOWNLOAD_RETRY_INITIAL_DELAY_MS` | 5000 | Initial ceiling for randomized transient download retry delays | | `CONFIG_SPOTFLOW_OTA_DOWNLOAD_RETRY_MAX_DELAY_MS` | 300000 | Maximum ceiling for randomized transient download retry delays | Transient download failures are retried indefinitely using HTTP Range requests. The retry delay ceiling doubles after each failure until it reaches the configured maximum. Each actual delay is randomized between half and all of the current ceiling to avoid synchronized retry traffic from multiple devices. Retries continue until the download succeeds, is canceled, or encounters a non-retryable error. ## Reference repository materials [#reference-repository-materials] * Public API: [`spotflow/ota.h`](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/include/spotflow/ota.h), [`spotflow/downloader.h`](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/include/spotflow/downloader.h) * [OTA sample](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/samples/ota) * [OTA Kconfig options](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/src/ota/Kconfig) ## Learn more [#learn-more] # What is Spotflow? (/) Spotflow is an **observability platform** designed specifically for embedded devices. With Spotflow, you can see what is happening across your whole device fleet, detect problems early, investigate issues, and deploy [firmware updates over-the-air](/using-spotflow/guides/ota). Spotflow gives you [dashboards](/fundamentals/monitoring/dashboards) for a clear view of your devices. It tracks important device metrics out of the box, and you can add [custom metrics](/fundamentals/monitoring/metrics#custom-metrics) directly in your firmware for the parts of your product that matter most. You can also set up [alerts](/fundamentals/monitoring/alerts) so you know when something goes wrong. When you need to investigate a problem, Spotflow lets you look at individual devices in detail. Our SDK collects [logs](/fundamentals/monitoring/logging), [metrics](/fundamentals/monitoring/metrics), and [crash dumps](/fundamentals/monitoring/crash-reports) out of the box, helping you find the root cause faster. We offer first-class integration for Zephyr RTOS, ESP-IDF, and Nordic nRF Connect SDK. Other platforms can be integrated easily using our MQTT-based protocol. Our goal is to provide an all-in-one observability solution for embedded devices. Visit our [roadmap](https://roadmap.spotflow.io/roadmap) to see what we are working on next. ## Components [#components] Spotflow consists of two main components: * **Device Module:** A lightweight library that you integrate into your Zephyr, Nordic nRF Connect SDK, or ESP-IDF firmware. It automatically collects metrics, logs, and core dumps, then sends them to Spotflow’s telemetry backend. It can also perform over-the-air firmware updates when requested. * **Cloud Platform:** A cloud service that communicates with devices over MQTT and provides a web interface for monitoring, analyzing, and managing your device fleet. * **Telemetry Storage and Querying:** A storage and query system built on a highly optimized columnar data store. It lets you search metrics, logs, and crash reports by content and metadata, including device ID, timestamp, firmware version, and more. * **FOTA Management:** A firmware update management system for deploying updates to specific groups of devices. It provides clear visibility into rollout progress and lets you roll back updates if something goes wrong. * **Alerting System:** A rules-based system that monitors incoming metrics and sends email notifications when user-defined conditions are met, such as threshold breaches or rapid changes in reported values. * **Crash Analyzer:** A combination of detailed GDB-based analysis and a purpose-built AI agent. It lets you inspect stack traces, registers, and static and global variables for each crash report. The AI agent analyzes crashes automatically and provides root-cause insights with suggestions for fixing the issue. ## Why choose Spotflow? [#why-choose-spotflow] We are building Spotflow with the goal of minimal friction for setting up while providing deep insights into your embedded devices. To name a few of the key features: * **Minimal setup:** Integrating your device with Spotflow requires just a few lines of code. Then our device module will automatically collect all the necessary telemetry. * **Great visibility into device fleet:** Our web interface provides a clear overview of your entire device fleet, including stability metrics, firmware versions, and more. * **Works when internet connection is not available:** Our device module buffers logs within volatile memory, ensuring that logs are collected even when the device is offline. Once the device is back online, it will automatically send the buffered logs to Spotflow. * **Optimized transport format:** We use a custom transport format based on CBOR. This format is designed to have minimal overhead, ensuring that logs are transmitted efficiently over the network. * **Fine-tunable to work on resource-constrained devices:** You can configure the sizes of the log buffers, allowing you to optimize memory usage based on your device's capabilities. * **Rich querying capabilities:** Our web interface provides powerful querying, allowing you to filter and search logs based on their contents and metadata like device id, timestamp, and more. * **Quick insights into firmware crashes:** Our AI agent automatically analyzes each crash report to provide you with a root cause analysis and suggestions for fixing the issue. ## Supported platforms [#supported-platforms] Spotflow currently aims to provide best-in-class experience for Zephyr RTOS, Nordic nRF Connect SDK, and ESP-IDF. With these platforms, the integration is as simple as adding a few lines of code to your device's firmware. Then you can use the standard logging macros to log messages, which will be automatically collected and sent to Spotflow. Other devices can be integrated using the MQTT protocol. This approach provides the biggest flexibility, allowing you to send logs from any device that supports MQTT. ### Learn how to connect your device [#learn-how-to-connect-your-device] See the quickstart guides below to learn how to connect your specific device to Spotflow. Learn how to integrate your device running Zephyr RTOS with Spotflow. Learn how to integrate your Nordic device running nRF Connect SDK with Spotflow. Learn how to integrate your ESP-IDF device with Spotflow. Learn how to integrate any device with Spotflow using MQTT. # ESP-IDF Integration (/quickstart/esp-idf) ## Prerequisites [#prerequisites] If you don't have ESP-IDF 5.0 or newer already installed, refer to the official documentation for [instructions](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/). ## Clone Spotflow Component [#clone-spotflow-component] Run the following command to clone our device SDK repository together with examples: ## Configure Sample Application [#configure-sample-application] All the following terminal commands are expected to run with access to the ESP-IDF tools. See the official documentation for more details: [Windows](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/windows-setup.html#launching-esp-idf-environment), [Linux and macOS](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/linux-macos-setup.html#step-4-set-up-the-environment-variables). To try out logging to Spotflow, let's configure the sample application. Modify the following options in device-sdk/esp\_idf/spotflow/device\_sdk/examples/logs/sdkconfig.defaults: The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys in the [ingest keys](https://app.spotflow.io/ingest-keys) page. Navigate to the logging sample application within the cloned repository and select the board of your choice as the target: ## Run Sample Application [#run-sample-application] Now, you are ready to build and flash the sample application to the device and start sending logs to Spotflow: To view the logs locally on your console, you can use: The console output should be similar to this one: ## Next Steps [#next-steps] After the basic logging integration works, you can also enable metrics collection for ESP-IDF devices: * [Metrics with ESP-IDF](/guides/esp-idf/metrics-esp-idf) * [ESP-IDF metrics sample](https://github.com/spotflow-io/device-sdk/tree/main/esp_idf/spotflow/device_sdk/examples/metrics) # Connect via MQTT (/quickstart/mqtt) ## Prerequisites [#prerequisites] You can use any MQTT client to connect to Spotflow. Just make sure: * you use MQTT 3.1.1 or higher, * your device have the Let's Encrypt's Certificate Authority [ISRG Root X1 TLS certificate](https://letsencrypt.org/certificates/#root-cas) installed. This root certificate is pre-installed on the majority of modern operating systems. ## Setup the MQTT Connection [#setup-the-mqtt-connection] First, you need to connect your MQTT client to the Spotflow broker. Set your client to use MQTT over TLS (MQTTS) and use the following parameters to connect: ```dotenv BROKER_HOSTNAME="mqtt.spotflow.io" BROKER_PORT="8883" USERNAME="quickstart_device" PASSWORD="{your-ingest-key}" ``` The `USERNAME` will be interpreted as a unique identifier of the device. The credential used as the `PASSWORD` is called an ingest key. You can manage your ingest keys in the [ingest keys](https://app.spotflow.io/ingest-keys) page. If your MQTT client requires it, Client ID can be specified, but it will be ignored by the Spotflow MQTT broker which maintains its own server-generated unique client IDs. ## Publish Logs [#publish-logs] Once your client is connected, you can start sending logs. You can choose to send logs in JSON or CBOR format. Prefer CBOR for efficiency in bandwidth-constrained environments. Use JSON for convenience when slightly larger messages are acceptable. To send logs serialized in JSON, simply publish MQTT messages into the `ingest-json` topic. The messages should follow the schema below: ```json { // Fully interpolated log line string (optional when bodyTemplate is used) "body": "SmartLock was Unlocked", // (Optional) printf-like interpolation string for the log line "bodyTemplate": "SmartLock was %s", // (Optional) array of values for interpolation in the bodyTemplate "bodyTemplateValues": ["Unlocked"], // (Recommended) Log severity, possible values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" "severity": "INFO", // (Optional) device uptime when the log was generated (milliseconds since the device booted) "deviceUptimeMs": 23455, // (Optional) time when the log was generated (milliseconds since the UNIX epoch) "deviceTimestampMs": 1748530133808, // (Optional) you can add extra metadata to your logs "labels": { "initiatorKind": "MobileApp", "userId": "1234567890" }, // (Optional) source device ID for gateway/relay architectures — see Source Device ID // Can be a string (single hop) or an array of strings (multi-hop) "sourceDeviceId": "sensor-01" } ``` **Tip for Testing:** use a command-line MQTT client to test messages without writing code. Note, that the `publish` command within these CLI clients typically handles the entire sequence for you: it connects, sends the message, and disconnects. First, download and install the MQTTX CLI from [their website](https://mqttx.app/cli#download). Then, use the following command to send a log message: Bash PowerShell ```bash mqttx pub \ --hostname 'mqtt.spotflow.io' \ --port 8883 \ --topic 'ingest-json' \ --protocol 'mqtts' \ --username 'quickstart_device' \ --password '' \ --message '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' ``` ```powershell mqttx pub ` --hostname 'mqtt.spotflow.io' ` --port 8883 ` --topic 'ingest-json' ` --protocol 'mqtts' ` --username 'quickstart_device' ` --password '' ` --message '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' ``` [Mosquitto](https://mosquitto.org/download/) ships with a `mosquitto_pub` command line MQTT client. Make sure you have `mosquitto_pub` available in your PATH. Then, use the following command to send a log message: Bash PowerShell ```bash mosquitto_pub \ -h 'mqtt.spotflow.io' \ -p 8883 \ -t 'ingest-json' \ -u 'quickstart_device' \ -P '' \ -m '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' \ --capath /etc/ssl/certs ``` ```bash mosquitto_pub ` -h 'mqtt.spotflow.io' ` -p 8883 ` -t 'ingest-json' ` -u 'quickstart_device' ` -P '' ` -m '{"body": "Quickstart Device Booted Up", "severity": "INFO", "deviceUptimeMs": 10, "labels": {"location": "facility-123"}}' ` --cafile ./isrgrootx1.pem ``` Depending on your operating system and its configuration, you may need to customize the command to point it to a proper location containing the [ISRG Root X1 TLS certificate](https://letsencrypt.org/certificates/#root-cas) via `--capath` or `--cafile` options. To send logs serialized in CBOR, simply publish MQTT messages into the `ingest-cbor` topic. The payload should include a single log following the `log-message` schema as defined below: ```CDDL log-message = { ? 1 => tstr, ; body: fully interpolated log line string (optional when bodyTemplate is used) ? ( 2 => tstr, ; bodyTemplate (optional): printf-like interpolation string 3 => body-template-values ; bodyTemplateValues (optional): values for interpolation ), ? 4 => severity, ; severity (recommended) ? 5 => labels, ; labels (optional): user-defined key-value pairs for additional context ? 6 => uint, ; deviceUptimeMs (optional): device uptime in milliseconds in range [0, 2^63 - 1] ? 7 => uint, ; deviceTimestampMs (optional): device timestamp in milliseconds in range [0, 2^63 - 1] ? 31 => (tstr / [+ tstr]) ; sourceDeviceId (optional): source device for gateway/relay architectures } labels = {* (tstr => (tstr / int / float / bool))} ; Strongly typed key-value pairs ; Integer severity values debug-severity = 30 info-severity = 40 warning-severity = 50 error-severity = 60 critical-severity = 70 severity = (debug-severity / info-severity / warning-severity / error-severity / critical-severity) ; A strongly typed array of values, or an array of byte string representations (big-endian) of the values body-template-values = [ * (tstr / int / float / bool / null) ] / [ * bstr ] ``` If your client keeps the MQTT connection open for multiple messages, you can publish [Session Metadata](/guides/mqtt/session-metadata) before the logs to attach shared context such as firmware build ID or session labels. ## Learn more [#learn-more] # nRF Connect SDK Integration (/quickstart/nordic-nrf-connect) ## Prerequisites [#prerequisites]
See the official [documentation of nRF Connect SDK](https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/installation/install_ncs.html#install_prerequisites) to install the following: * SEGGER J-Link and related tools * nRF Util - either standalone or as a part of nRF Connect for VS Code The workspace setup script requires Python 3.10+ and Git. ## Create West Workspace [#create-west-workspace] The following script will guide you through the creation of a new workspace that contains the Spotflow module and sample applications: Alternatively, you can create the workspace manually by following [these steps](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/ci/README.md#workspace-setup-scripts). ## Configure Sample Application [#configure-sample-application] To try out logging to Spotflow, let's configure the sample application. Fill in the required configuration options in \/{props.board.spotflow_path}/zephyr/samples/logs/prj.conf: Make sure the Ethernet cable is plugged in into the board and the network has internet access. If you’re using Ethernet, make sure the Ethernet cable is plugged in into the board and the network has internet access. The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys in the [ingest keys](https://app.spotflow.io/ingest-keys) page. ## Run Sample Application [#run-sample-application] The following terminal commands are expected to run in the nRF Connect toolchain environment. See the official documentation for more details: [VS Code](https://docs.nordicsemi.com/bundle/nrf-connect-vscode/page/guides/extension_nrfconnect_profile.html), [CLI](https://docs.nordicsemi.com/bundle/nrfutil/page/nrfutil-sdk-manager/guides/sdk_manager_env_launch.html). Now, you are ready to build and flash your application to the device and start sending logs to Spotflow: If you open the UART output from the device, it should be similar to this one: ## Learn more [#learn-more] # Zephyr Integration (/quickstart/zephyr) ## Prerequisites [#prerequisites]
If you don't have Zephyr dependencies already installed, refer to the official [Getting Started Guide](https://docs.zephyrproject.org/latest/develop/getting_started/index.html#install-dependencies) for the instructions on how to install them for your operating system. Some devices might also require specific software for flashing, such as J-Link or OpenOCD. Check the Zephyr documentation of the board {props.board.name} for more details. ## Create West Workspace [#create-west-workspace] The following script will guide you through the creation of a new workspace that contains the Spotflow module and sample applications: Alternatively, you can create the workspace manually by following [these steps](https://github.com/spotflow-io/device-sdk/blob/main/zephyr/ci/README.md#workspace-setup-scripts). ## Configure Sample Application [#configure-sample-application] To try out logging to Spotflow, let's configure the sample application. Fill in the required configuration options in \/{props.board.spotflow_path}/zephyr/samples/logs/prj.conf: Make sure the Ethernet cable is plugged in into the board and the network has internet access. If you’re using Ethernet, make sure the Ethernet cable is plugged in into the board and the network has internet access. The `CONFIG_SPOTFLOW_INGEST_KEY` is a secret key that allows your device to authenticate with Spotflow. You can manage your ingest keys in the [ingest keys](https://app.spotflow.io/ingest-keys) page. When using a custom board, you might need to adjust other configuration options to compile the example. Refer to the [options for specific boards](https://github.com/spotflow-io/device-sdk/tree/main/zephyr/samples/logs/boards) in our device SDK repository for inspiration. ## Run Sample Application [#run-sample-application] Now, you are ready to build and flash your application to the device and start sending logs to Spotflow: If you open the UART output from the device, it should be similar to this one: ## Learn more [#learn-more] # Test (/test) Hello World again! ## Installation [#installation] npm pnpm yarn bun ```bash npm i fumadocs-core fumadocs-ui ``` ```bash pnpm add fumadocs-core fumadocs-ui ``` ```bash yarn add fumadocs-core fumadocs-ui ``` ```bash bun add fumadocs-core fumadocs-ui ``` # Comply with CRA (/using-spotflow/best-practices/comply-with-cra) The EU Cyber Resilience Act (CRA) puts cybersecurity responsibilities on manufacturers throughout a product's lifetime. Spotflow helps you keep track of the software in your devices, respond to vulnerabilities, deliver fixes, and monitor devices in the field. Spotflow can support your CRA compliance work, but it cannot make your product compliant on its own. Your organization is responsible for determining which requirements apply and for meeting them. ## Track software and vulnerabilities [#track-software-and-vulnerabilities] "...identify and document vulnerabilities and components contained in products..." Upload an SBOM for every firmware version to keep a record of its components and dependencies. Spotflow matches these components against known CVEs and rescans existing SBOMs as the CVE database changes. This allows you to track vulnerabilities across your whole device fleet. Assess each finding with VEX-style states to record whether it affects your firmware, and document the response or justification. The audit timeline records who made each decision and why. See [Track security issues across your devices](/using-spotflow/guides/track-security-issues) for the complete workflow. ## Deliver fixes over the air [#deliver-fixes-over-the-air] "...ensure that vulnerabilities can be addressed through security updates, including, where applicable, through automatic security updates..." When a vulnerability needs a fix, Spotflow can deliver a new firmware version over the air without physical access to each device. You can deploy a specific firmware version to a cohort of devices. These cohorts can be defined either as a manually selected list of devices or dynamically using device tags, such as hardware model, region, or end-user consent. Spotflow tracks the version running on each device and reports the deployment status. See [Over-the-air (OTA) updates](/fundamentals/ota) and [Deploy over-the-air (OTA) updates](/using-spotflow/guides/ota) for device integration and deployment guidance. ## Monitor devices in the field [#monitor-devices-in-the-field] "...monitoring relevant internal activity..." Monitoring helps you spot unexpected or suspicious behavior and investigate incidents. Spotflow collects logs, metrics, and crash reports from individual devices and links them to the firmware version running at the time. ### Logs [#logs] Search and filter device logs across the fleet, or inspect the history of a single device. With the Spotflow device module, you can also increase a device's log level from the portal when you need more diagnostic detail. See [Logging](/fundamentals/monitoring/logging) for integration and analysis options. ### Metrics [#metrics] Spotflow device modules report system metrics such as CPU, heap and stack usage, network traffic, connection state, reset causes, and uptime. Add application-specific metrics, visualize them in dashboards, and create alerts for conditions that need attention. See [Metrics](/fundamentals/monitoring/metrics), [Dashboards](/fundamentals/monitoring/dashboards), and [Alert rules](/using-spotflow/guides/alert-rules). ### Crash reports and core dumps [#crash-reports-and-core-dumps] When a device crashes, Spotflow collects its core dump and links it to the exact firmware build. For supported Zephyr and ESP-IDF formats, it also extracts details such as stack traces and register values to help your team diagnose the failure. See [Crash reports & core dumps](/fundamentals/monitoring/crash-reports) for setup and analysis details. ## A practical workflow [#a-practical-workflow] 1. Upload an SBOM whenever you create a firmware version. 2. Review new CVEs and assess whether they affect your firmware. 3. Prioritize the issues, build a fix, and deploy it in controlled cohorts. 4. Monitor logs, metrics, and crash reports for suspicious activity. ## Learn more [#learn-more] # Using Spotflow with Your Existing MQTT Broker (/using-spotflow/best-practices/existing-mqtt-broker) Many IoT deployments already have a cloud MQTT broker (AWS IoT Core, Azure IoT Hub, HiveMQ, or a self-hosted broker) handling communication between devices and backend services. Adding Spotflow observability to such a deployment does not require a second MQTT connection from the device. Opening a separate connection carries real costs on cellular and constrained hardware: additional radio time, extra memory for a second TLS session, and a second set of credentials to provision and rotate. Instead, the [Spotflow device module](https://github.com/spotflow-io/device-sdk) can package telemetry and publish it to dedicated topics on your existing broker. The Spotflow MQTT Subscriber connects to those topics from the cloud side, and your devices, your broker infrastructure, and your existing backend services remain unchanged. This integration path is currently in development. The architecture described here reflects the planned design. If you are evaluating this approach for your deployment or would like to participate in early access, contact us at [hello@spotflow.io](mailto:hello@spotflow.io) or reach out on [Discord](https://discord.gg/yw8rAvGZBx). The device maintains a single MQTT connection to your broker, exactly as it does today. The Spotflow device module publishes telemetry (logs, metrics, and crash reports) to a dedicated topic prefix (for example, `spotflow/`) within that broker. Your application continues publishing to its own topics on the same connection. The two streams share the connection but remain logically separate. On the cloud side, the Spotflow MQTT Subscriber connects to your broker. Most enterprise MQTT brokers support topic-level access control, which means you can grant Spotflow read access to the `spotflow/` prefix only, with no visibility into any other topic. Once the Spotflow MQTT Subscriber receives telemetry, it processes and stores it in the Spotflow platform, and the data becomes available in the Spotflow Portal exactly as it would in a standard Spotflow integration: log search, metric dashboards, crash report analysis, alerts, and firmware management. Once telemetry reaches the Spotflow MQTT Subscriber, it is processed and available in the Spotflow Portal near-realtime. ## Learn More [#learn-more] # Using Spotflow in Mesh Networks (/using-spotflow/best-practices/mesh-networks) Not every device can connect to the internet on its own. In many deployments, only one device (the gateway) has internet access, while other devices nearby communicate with it over a short-range link such as Bluetooth, Zigbee, or a wired bus. These devices without a direct internet connection are called **end devices**. This page explains how to collect logs, metrics, and crash reports from end devices in Spotflow, and what you need to build to make it work. This article covers the basic message forwarding pattern. More advanced scenarios, such as gateway-controlled telemetry streams, log level negotiation between gateway and end device, and on-demand coredump transfer, are not yet covered and are planned for a future release of the Spotflow device module. If you need guidance for a more complex setup, contact us at [hello@spotflow.io](mailto:hello@spotflow.io) or reach out on [Discord](https://discord.gg/7SfqWMv3). ## Architecture [#architecture] The central idea is straightforward: end devices serialize Spotflow messages and pass them over the local link toward a gateway device, optionally through one or more relay nodes. Each relay node forwards raw message bytes over the next local link hop without inspecting or modifying the content. The gateway forwards each message to the Spotflow MQTT broker under its own connection. Spotflow attributes the record to the originating end device, and the gateway's ID appears as the transport route. Spotflow automatically registers every device ID forwarded by a trusted, authenticated gateway, so no pre-registration of end devices is required. The gateway authenticates to Spotflow using its ingest key; end device identities are accepted transitively through that trust. Once the first relayed message arrives, the end device appears in the portal with its own device page, event history, and dashboards. The transport route (the full chain of device IDs from origin to cloud) is recorded on each event. ## How Spotflow Handles Relayed Messages [#how-spotflow-handles-relayed-messages] The mechanism that makes this work is the `sourceDeviceId` field. Every Spotflow message format (logs, metrics, core dump chunks) accepts this optional field. When present, the cloud uses it as the authoritative device ID for that record. The device that actually connected to MQTT (the gateway) is recorded in the transport route instead. | `sourceDeviceId` in message | Gateway connects as | Resolved device ID | Transport route | | --------------------------- | ------------------- | ------------------ | ---------------- | | *(absent)* | `gd1` | `gd1` | `["gd1"]` | | `"ed1"` | `gd1` | `ed1` | `["ed1", "gd1"]` | The resolved device ID is always the original source. This is the ID you will see in the portal, in queries, and in dashboards. **Constraint:** Each device ID must consist of alphanumeric characters, hyphens, underscores, dots, and colons; maximum 64 characters. ### Adding sourceDeviceId to JSON messages [#adding-sourcedeviceid-to-json-messages] For any message type published to the `ingest-json` topic, add the `sourceDeviceId` field at the top level: ```json title="Log message with sourceDeviceId" { "body": "Sensor node booted", "severity": "INFO", "deviceUptimeMs": 500, "sourceDeviceId": "ed1" } ``` ```json title="Metric message with sourceDeviceId" { "messageType": "METRIC", "metricName": "temperature_celsius", "sum": 23.5, "deviceUptimeMs": 60000, "sequenceNumber": 1, "sourceDeviceId": "ed1" } ``` ```json title="Core dump chunk with sourceDeviceId" { "messageType": "CORE_DUMP_CHUNK", "coreDumpId": 987654321, "chunkOrdinal": 0, "content": "WkUCAAMABQAD...", "isLastChunk": false, "buildId": "build-ed1-v1.2.0", "os": "Zephyr", "sourceDeviceId": "ed1" } ``` ### Adding sourceDeviceId to CBOR messages [#adding-sourcedeviceid-to-cbor-messages] For CBOR-encoded messages published to the `ingest-cbor` topic, `sourceDeviceId` uses property key `31`: ``` ; CBOR field reference 31 => tstr ; sourceDeviceId: originating end device ID ``` This applies to all three message types (log messages, metric messages, and core dump chunks). The field is appended to the existing CBOR map alongside the message-type-specific fields. ## Current Limitations [#current-limitations] The Spotflow device module does not yet include a built-in mesh transport layer. There are no SDK APIs for extracting pending messages from an end device's queue or for injecting a relayed message into a gateway's send queue. The architecture described in this article requires you to implement the relay logic in your application firmware. Support for mesh transport is planned for a future SDK release. The sections below describe exactly how to implement this relay logic in your application firmware. ## The Gateway Device [#the-gateway-device] The gateway is the only device in the network with an MQTT connection to Spotflow. It connects using its own ingest key with its device ID as the MQTT username, exactly as a directly connected device would. See the [MQTT quickstart](/quickstart/mqtt) for setup. The gateway's job is to: 1. Receive raw message bytes from end devices over the local link. 2. Reconstruct the complete message from the local link. 3. Publish the bytes as-is to the appropriate MQTT topic (`ingest-json` or `ingest-cbor`). The gateway treats each message as opaque bytes, it does not need to inspect or modify the content, because `sourceDeviceId` is already embedded by the end device. The gateway also collects its own telemetry using the Spotflow device module normally. Its logs, metrics, and core dumps are sent directly without a `sourceDeviceId` field. ## The Relay Node [#the-relay-node] A relay node is an intermediate device that sits between one or more end devices and the gateway. It does not connect to Spotflow directly and does not need an ingest key. Its only responsibility is to forward raw message bytes from end devices toward the gateway over the next local link hop. The relay node's job is to: 1. Receive raw message bytes from end devices over the local link. 2. Forward the bytes as-is to the next hop (another relay node or the gateway). Because `sourceDeviceId` is already embedded in the message by the originating end device, the relay node does not need to inspect or modify the content. Multiple relay hops are supported; the bytes pass through each hop unchanged until they reach the gateway. A relay node may also act as an end device and send its own telemetry to the gateway, in which case it follows the same rules as any other end device. ## The End Device [#the-end-device] The end device's job is to produce serialized Spotflow messages and deliver them to the gateway. For each message: 1. Serialize the message payload in CBOR (preferred for bandwidth efficiency) or JSON, including `sourceDeviceId` set to the end device's own device ID. 2. Transmit to the next hop (a relay node or the gateway directly). The end device does not connect to MQTT and does not need an ingest key. Its device ID in `sourceDeviceId` is the identifier that will appear in the Spotflow portal. ### Device ID consistency [#device-id-consistency] The device ID used in `sourceDeviceId` must be stable across reboots and consistent across all message types. Use a value derived from hardware (a MAC address, chip serial number, or provisioned identifier) rather than a randomly generated value. If the device ID changes, the portal will treat the device as a new entity. ## Message Priority and Queuing [#message-priority-and-queuing] When multiple message types are pending, deliver them in this order: 1. **Core dump chunks**: highest priority; deliver before anything else. 2. **Metrics**: second priority; deliver when no core dump is in progress. 3. **Logs**: lowest priority; deliver when neither core dumps nor metrics are pending. This matches the priority order used internally by the Spotflow device module and ensures that crash data (the most diagnostically critical) reaches the cloud before routine telemetry. For local link transports that support different reliability modes (acknowledged vs. unacknowledged), consider sending core dump chunks as acknowledged messages and logs as unacknowledged. This trades throughput for crash data integrity. ## Core Dump Reliability [#core-dump-reliability] A core dump is split into numbered chunks, all sharing the same `coreDumpId`. Spotflow requires all chunks to arrive (though not necessarily in order) to reconstruct and analyze the dump. Missing chunks result in an incomplete crash report. To maximize reliability: * Use the **acknowledged (reliable)** mode of your local transport for core dump chunk transfers where available. * Store undelivered core dump chunks in **non-volatile storage** on the end device. Core dump data should survive a reboot so it can be retransmitted if the gateway connection was not available at crash time. * Track delivery progress per chunk on the end device. Mark each chunk as delivered only after the gateway confirms receipt at the local link level. ## Network Topologies [#network-topologies] Any topology where one or more gateways forward end device messages to Spotflow is supported, device IDs are stable regardless of which gateway forwards a given message. Relay nodes may be inserted between end devices and the gateway to extend range or bridge network segments; messages pass through each relay hop unchanged. The one constraint is that **each end device must route all its messages through a single gateway.** If the same message reaches Spotflow via two different gateways, it will appear as a duplicate in the portal. Ensuring single-gateway routing per end device is the responsibility of your network and application code. ## Learn More [#learn-more] # Using Spotflow with Resource-Constrained Devices (/using-spotflow/best-practices/resource-constrained-devices) [Spotflow's device module](https://github.com/spotflow-io/device-sdk) is designed to be tunable. Every major subsystem (logging, metrics, coredumps, and the TLS stack) exposes Kconfig options that directly control its RAM allocation, transmission frequency, and CPU footprint. This page walks through the options most relevant to devices with limited resources and explains the trade-offs behind each one. The guidance here targets Zephyr and Nordic nRF Connect SDK. ESP-IDF shares the same underlying principles but uses `sdkconfig.defaults` instead of `prj.conf`. The defaults are intentionally set to cover the widest range of use cases and boards, which means they are higher than most production deployments actually require. This page gives an overview of the options for reducing memory and bandwidth allocations. If you have specific requirements or constraints, reach out via [hello@spotflow.io](mailto:hello@spotflow.io) or [Discord](https://discord.gg/yw8rAvGZBx) and we will help you find the best optimization approach for your setup. ## Memory [#memory] Memory usage is controlled through Kconfig heap additions. Each subsystem declares its own heap pool contribution; enabling a subsystem adds that amount to the system heap. Nothing is allocated silently. The table below shows the default heap contribution for each subsystem: | Subsystem | Kconfig option | Default | | ----------------- | ------------------------------------------------ | ------- | | Logging backend | `HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_LOGGING` | 6 KB | | Custom metrics | `HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS` | 8 KB | | System metrics | `HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS_SYSTEM` | 20 KB | | Coredumps backend | `HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_COREDUMPS` | 18 KB | | Mbed TLS | `MBEDTLS_HEAP_SIZE` | 32 KB | Metrics and coredumps are disabled by default (`CONFIG_SPOTFLOW_METRICS=n`, `CONFIG_SPOTFLOW_COREDUMPS=n`). With only logging enabled, the baseline is roughly **38 KB** (6 KB logging + 32 KB Mbed TLS), plus a 2.5 KB processing thread stack. The Mbed TLS heap dominates the baseline. Of the \~38 KB total, 32 KB is Mbed TLS and \~8.5 KB is Spotflow itself (6 KB logging heap + 2.5 KB thread stack). Mbed TLS uses this heap for the full TLS session: certificate chain verification, and input and output record buffers for the encrypted connection. If your firmware already uses a TLS library, the 32 KB may not be an incremental cost as it may already be accounted for in your memory budget. If you are evaluating footprint against other solutions, compare the numbers without the TLS layer unless the competing solution includes one too. If you are using a different TLS library or need a lower-footprint TLS option, contact us and we can look at adjusting the integration. ### Reducing the logging heap [#reducing-the-logging-heap] The logging heap is sized to hold a queue of CBOR-encoded messages waiting to be sent. Three options control it: | Kconfig option | Default | Effect | | ------------------------------------ | ----------- | ---------------------------------------------------------------------------------- | | `SPOTFLOW_LOG_BACKEND_QUEUE_SIZE` | 16 messages | Depth of the in-RAM log queue | | `SPOTFLOW_LOG_BUFFER_SIZE` | 512 bytes | Raw per-message buffer before CBOR encoding | | `SPOTFLOW_CBOR_LOG_MAX_LEN` | 1024 bytes | CBOR-encoded output buffer per message | | `SPOTFLOW_LOG_INCLUDE_BODY_TEMPLATE` | `y` | Whether to include the original format template alongside the interpolated message | If your log lines are short (under \~100 characters), you can safely reduce `SPOTFLOW_LOG_BUFFER_SIZE` and `SPOTFLOW_CBOR_LOG_MAX_LEN`. Messages that exceed the buffer are dropped. Disabling `SPOTFLOW_LOG_INCLUDE_BODY_TEMPLATE` omits the original format template from each log message, sending only the final interpolated string. This reduces per-message CBOR payload size at the cost of losing the template in the cloud. ```dotenv title="prj.conf — reduced logging heap" # Default: ~6 KB heap CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_LOGGING=6144 # Constrained: ~3 KB heap CONFIG_SPOTFLOW_LOG_BACKEND_QUEUE_SIZE=8 CONFIG_SPOTFLOW_LOG_BUFFER_SIZE=256 CONFIG_SPOTFLOW_CBOR_LOG_MAX_LEN=512 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_LOGGING=3072 ``` ### Reducing the custom metrics heap [#reducing-the-custom-metrics-heap] When you register a labeled metric, two parameters directly determine how much heap it consumes: * **`MAX_LABELS_PER_METRIC`** (Kconfig, compile-time) — the maximum number of label key-value pairs any single metric report will ever carry. Each label slot costs \~48 bytes per timeseries regardless of whether that slot is actually used, because the array is sized at compile time. * **`max_timeseries`** (registration call, per-metric) — the number of unique label combinations the SDK must hold in memory simultaneously. Each unique combination gets its own independent aggregation state (sum, count, min, max) for the full duration of the aggregation window. The heap cost per metric follows: ``` timeseries_size = ~36 + (48 × MAX_LABELS_PER_METRIC) bytes per-metric heap = 80 + (timeseries_size × max_timeseries) ``` `max_timeseries` should equal the number of unique label value combinations you expect at runtime. For example, a metric with a `location` label (3 possible values: `north`, `south`, `east`) and a `channel` label (2 possible values: `a`, `b`) produces 6 combinations, set `max_timeseries=6`. If a new combination appears beyond that limit, the SDK silently drops reports for it until the aggregation window resets. The following table shows how `MAX_LABELS_PER_METRIC` affects timeseries size and the total heap for that 6-combination example: | `MAX_LABELS_PER_METRIC` | Bytes per timeseries | Heap for 6 timeseries | | ----------------------- | -------------------- | --------------------- | | 1 | \~84 bytes | \~590 bytes | | 2 | \~132 bytes | \~875 bytes | | 4 (default) | \~228 bytes | \~1.5 KB | | 8 (max) | \~420 bytes | \~2.6 KB | Set `MAX_LABELS_PER_METRIC` to the highest label count used by any single metric in your application, not to a generous ceiling. If your most label-heavy metric uses 2 labels, set it to 2. System metrics use at most 1 label. ```dotenv title="prj.conf — reduced custom metrics heap" # Default: 4 labels per timeseries, up to 32 metrics CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC=4 CONFIG_SPOTFLOW_METRICS_MAX_REGISTERED=32 # Constrained: trim to actual label usage CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC=2 CONFIG_SPOTFLOW_METRICS_MAX_REGISTERED=8 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS=4096 ``` ### Reducing the system metrics heap [#reducing-the-system-metrics-heap] The system metrics heap is the largest contributor after Mbed TLS. Thread stack monitoring dominates its cost: with the default of tracking up to 32 threads automatically, stack metrics alone account for roughly 15 KB of the 20 KB default. ```dotenv title="prj.conf — reduced system metrics heap" # Default: 20 KB heap, all threads tracked automatically CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS_SYSTEM=20480 CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_ALL_THREADS=y CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_MAX_THREADS=32 # Constrained: ~8 KB heap, only track specific threads manually CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_ALL_THREADS=n CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_MAX_THREADS=4 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS_SYSTEM=8192 ``` When `STACK_ALL_THREADS=n`, register the threads you care about explicitly from application code: ```c title="src/main.c" spotflow_metrics_system_enable_thread_stack(my_critical_thread); ``` ### Reducing the coredumps heap [#reducing-the-coredumps-heap] The coredumps heap is sized to buffer binary chunks before MQTT transmission. Reducing the chunk size and queue depth lowers the heap proportionally: ``` heap ≈ (CHUNK_SIZE × QUEUE_SIZE) + 2 KB ``` ```dotenv title="prj.conf — reduced coredumps heap" # Default: ~18 KB heap CONFIG_SPOTFLOW_COREDUMPS_CHUNK_SIZE=1024 CONFIG_SPOTFLOW_COREDUMPS_BACKEND_QUEUE_SIZE=16 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_COREDUMPS=18432 # Constrained: ~6 KB heap CONFIG_SPOTFLOW_COREDUMPS_CHUNK_SIZE=512 CONFIG_SPOTFLOW_COREDUMPS_BACKEND_QUEUE_SIZE=8 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_COREDUMPS=6144 ``` On boards with limited flash for storing the coredump image before upload, also enable: ```dotenv title="prj.conf" CONFIG_DEBUG_COREDUMP_MEMORY_DUMP_MIN=y ``` ## Bandwidth [#bandwidth] ### Log level filtering [#log-level-filtering] The most direct bandwidth reduction is filtering out low-severity logs before they are serialized and queued. Use `SPOTFLOW_DEFAULT_SENT_LOG_LEVEL` to set the initial filter at boot time: | Value | Level | When to use | | ----- | ------- | -------------------------------------------- | | `1` | ERROR | Production devices: only failures | | `2` | WARNING | Production devices: failures and anomalies | | `3` | INFO | Staging / burn-in: standard operational logs | | `4` | DEBUG | Development: all messages (default) | ```dotenv title="prj.conf — production log filter" # Default: send all log levels (DEBUG and above) CONFIG_SPOTFLOW_DEFAULT_SENT_LOG_LEVEL=4 # Production: send only WARNING and ERROR CONFIG_SPOTFLOW_DEFAULT_SENT_LOG_LEVEL=2 ``` You can also change the log level at runtime from the Spotflow portal without a firmware update (see [Adjust Device's Minimal Log Severity](/guides/zephyr/logging-zephyr#optional-adjust-devices-minimal-log-severity)). On MQTT connect, the device reports its current level; when you change it in the portal, the device applies it immediately. To persist the level across reboots so it takes effect before the first MQTT connection, enable the [Zephyr settings subsystem](https://docs.zephyrproject.org/latest/services/storage/settings/index.html): ```dotenv title="prj.conf" CONFIG_FLASH=y CONFIG_FLASH_MAP=y CONFIG_NVS=y CONFIG_SETTINGS=y CONFIG_SETTINGS_NVS=y # CONFIG_SPOTFLOW_SETTINGS=y is set automatically when CONFIG_SETTINGS=y ``` Also suppress the device module's own internal logs in production builds: ```dotenv title="prj.conf" # 1 = ERROR only (default is 2 = WARNING) CONFIG_SPOTFLOW_MODULE_DEFAULT_LOG_LEVEL=1 ``` ### Metrics aggregation [#metrics-aggregation] Metrics are aggregated on-device before transmission: individual samples are accumulated and sent as a single message containing `sum`, `count`, `min`, and `max` for the window. This means transmission frequency is decoupled from sampling frequency. | Aggregation interval | Constant | Transmissions per metric per day | | -------------------- | ----------------------------- | -------------------------------- | | No aggregation | `SPOTFLOW_AGG_INTERVAL_NONE` | One per sample | | 1 minute | `SPOTFLOW_AGG_INTERVAL_1MIN` | 1440 | | 1 hour | `SPOTFLOW_AGG_INTERVAL_1HOUR` | 24 | | 1 day | `SPOTFLOW_AGG_INTERVAL_1DAY` | 1 | One-hour aggregation is typically sufficient for fleet health monitoring. One-day aggregation is practical for low-bandwidth deployments where only long-term trends matter. No aggregation is appropriate for event-like metrics where each occurrence is individually significant. The default aggregation interval and heartbeat frequency are both configurable: ```dotenv title="prj.conf — reduced metric transmission" # Default: 1-minute aggregation, 60-second heartbeat CONFIG_SPOTFLOW_METRICS_DEFAULT_AGGREGATION_INTERVAL=60 CONFIG_SPOTFLOW_METRICS_HEARTBEAT_INTERVAL=60 # Low-bandwidth: 1-hour aggregation, 1-hour heartbeat CONFIG_SPOTFLOW_METRICS_DEFAULT_AGGREGATION_INTERVAL=3600 CONFIG_SPOTFLOW_METRICS_HEARTBEAT_INTERVAL=3600 ``` System metrics use separate collection and aggregation intervals: ```dotenv title="prj.conf — reduced system metrics transmission" # Default: sample every 10 s, send aggregated every 60 s CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=10 CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL=60 # Low-bandwidth: sample every 60 s, send aggregated every hour CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=60 CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL=3600 ``` ### CBOR encoding [#cbor-encoding] The [Spotflow device module](https://github.com/spotflow-io/device-sdk) always uses CBOR (Concise Binary Object Representation) for all messages — logs, metrics, and coredumps. CBOR uses single-byte field identifiers and encodes values in their native binary form, which produces significantly smaller payloads than text-based formats. ### Dictionary Logging [#dictionary-logging] Dictionary logging support is in development. Standard text logging transmits the full format string with every message. On a constrained link, those strings add up quickly: a single log line with a short message and two numeric arguments can easily exceed 50 bytes on the wire, and a busy firmware may emit hundreds of such messages per second. Dictionary logging eliminates that overhead. Instead of transmitting the format string, the device sends only a compact numeric reference pointing to the string in the build's ELF file. Arguments are encoded in their native binary form rather than converted to text. The host-side parser reconstructs the full human-readable message offline using a build-time dictionary file. String formatting is never performed on the device. The difference in what is actually transmitted over the wire for a single log call: ```text // Text logging — full string sent with every message (~50 bytes) "[INF] sensor_read: temperature=23 pressure=1012" // Dictionary logging — ELF reference + raw binary arguments (~6 bytes) [0x3A2F] 0x00000017 0x000003F4 ``` Beyond bandwidth, dictionary logging also reduces CPU overhead in the logging backend thread by skipping on-device string formatting. ## Power and Radio Activity [#power-and-radio-activity] Longer aggregation windows and a longer heartbeat interval directly reduce the number of MQTT publishes per day, which is the primary factor in radio-on time for cellular and low-power wireless devices. The design choices that reduce radio overhead by default: * **QoS 0**: The device module always uses MQTT QoS 0. No acknowledgment round-trips and no retransmit state machine mean fewer radio transactions per message. * **Aggregation**: A single aggregated MQTT message replaces hundreds of raw samples. With 1-hour aggregation, a metric that is sampled every 10 seconds produces one transmission instead of 360. * **Heartbeat**: The heartbeat interval controls how often the uptime metric is sent. Extending it from the default 60 seconds to 3600 seconds eliminates 59 out of every 60 heartbeat transmissions. The configuration in the [Metrics aggregation](#metrics-aggregation) section above covers all the relevant options. ## CPU Overhead [#cpu-overhead] The MQTT processing runs on a dedicated background thread at `K_LOWEST_APPLICATION_THREAD_PRIO` by default (priority 14). This means application threads preempt it freely, and it only runs when no application work is pending. No additional configuration is needed to achieve this behavior. The main tuning lever for CPU overhead is the system metrics collection interval. Sampling heap, stack, CPU utilization, and network counters has a small but nonzero cost. Increasing the interval reduces that overhead: ```dotenv title="prj.conf" # Default: collect system metrics every 10 seconds CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=10 # Reduced overhead: collect every 60 seconds CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=60 ``` CPU utilization metrics (`SPOTFLOW_METRICS_SYSTEM_CPU`) depend on Zephyr's `CPU_LOAD` subsystem, which is only available on single-core (non-SMP) targets: Cortex-M, RISC-V, and Cortex-A. On SMP builds, disable this metric to avoid a build error. ## Offline Operation [#offline-operation] The logging backend stores messages in a circular queue in RAM. When the device loses connectivity, messages continue to accumulate in the queue. When the connection is restored, the processing thread drains the queue and transmits buffered messages automatically, no application code is required. When the queue is full and the device is still offline, the oldest messages are overwritten. This newest-wins eviction policy preserves the most recent context at the cost of older entries. Queue depth is controlled by `SPOTFLOW_LOG_BACKEND_QUEUE_SIZE`. The log buffer is volatile. If the device reboots while offline, buffered logs are lost. Only coredumps survive reboots, they are stored in a dedicated flash partition and uploaded on the next boot once connectivity is available. Metrics behave similarly: aggregated values are held in RAM within the current aggregation window. If the device reboots mid-window, the partial aggregation is discarded. ## Enabling Only What You Need [#enabling-only-what-you-need] Metrics and coredumps are disabled by default. Only enable subsystems you actively use: | Subsystem | Enable with | Default heap | | -------------- | ---------------------------------------------- | ------------ | | Logging | `CONFIG_SPOTFLOW_LOG_BACKEND=y` (auto-enabled) | 6 KB | | Custom metrics | `CONFIG_SPOTFLOW_METRICS=y` | 8 KB | | System metrics | `CONFIG_SPOTFLOW_METRICS_SYSTEM=y` | 20 KB | | Coredumps | `CONFIG_SPOTFLOW_COREDUMPS=y` | 18 KB | Within system metrics, each metric type can be toggled independently: ```dotenv title="prj.conf — disable unused system metrics" CONFIG_SPOTFLOW_METRICS_SYSTEM_HEAP=y CONFIG_SPOTFLOW_METRICS_SYSTEM_NETWORK=y CONFIG_SPOTFLOW_METRICS_SYSTEM_CPU=y CONFIG_SPOTFLOW_METRICS_SYSTEM_CONNECTION=y CONFIG_SPOTFLOW_METRICS_SYSTEM_RESET_CAUSE=y # Disable stack monitoring if thread count is high and heap is tight CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK=n ``` Disabling `SPOTFLOW_METRICS_SYSTEM_STACK` removes the dominant cost of the system metrics heap (roughly 15 KB of the 20 KB default). Also disable IPv6 if your network stack does not require it, this is a common memory saving in Zephyr samples that applies regardless of Spotflow: ```dotenv title="prj.conf" CONFIG_NET_IPV6=n ``` ## Minimum Footprint Configuration [#minimum-footprint-configuration] The following is a starting-point `prj.conf` for a constrained device using logging only. Adjust queue sizes and log level to match your application's line lengths and production logging requirements. ```dotenv title="prj.conf — minimum footprint (logging only)" # Enable Spotflow CONFIG_SPOTFLOW=y # Reduce logging heap: ~1 KB instead of default 6 KB CONFIG_SPOTFLOW_LOG_BACKEND_QUEUE_SIZE=4 CONFIG_SPOTFLOW_LOG_BUFFER_SIZE=128 CONFIG_SPOTFLOW_CBOR_LOG_MAX_LEN=160 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_LOGGING=1024 # Send INFO and above by default CONFIG_SPOTFLOW_DEFAULT_SENT_LOG_LEVEL=3 # Suppress SDK internal debug output CONFIG_SPOTFLOW_MODULE_DEFAULT_LOG_LEVEL=3 # Omit format template from log messages to reduce payload size CONFIG_SPOTFLOW_LOG_INCLUDE_BODY_TEMPLATE=n # IPv4 only CONFIG_NET_IPV6=n ``` For a device that also uses metrics, extend with: ```dotenv title="prj.conf — metrics additions for constrained devices" CONFIG_SPOTFLOW_METRICS=y CONFIG_SPOTFLOW_METRICS_SYSTEM=y # Aggregate metrics hourly instead of every minute CONFIG_SPOTFLOW_METRICS_DEFAULT_AGGREGATION_INTERVAL=3600 CONFIG_SPOTFLOW_METRICS_HEARTBEAT_INTERVAL=3600 # Reduce system metrics heap by limiting thread tracking CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_ALL_THREADS=n CONFIG_SPOTFLOW_METRICS_SYSTEM_STACK_MAX_THREADS=4 CONFIG_HEAP_MEM_POOL_ADD_SIZE_SPOTFLOW_METRICS_SYSTEM=8192 # Sample system metrics less frequently CONFIG_SPOTFLOW_METRICS_SYSTEM_COLLECTION_INTERVAL=60 CONFIG_SPOTFLOW_METRICS_SYSTEM_AGGREGATION_INTERVAL=3600 # Trim label slots to actual usage (saves ~48 bytes per timeseries per unused label) CONFIG_SPOTFLOW_METRICS_MAX_LABELS_PER_METRIC=1 ``` ## Learn more [#learn-more] # Set up alerts (/using-spotflow/guides/alert-rules) ## Create Alert Rule [#create-alert-rule] Create your alert rule by clicking `+ Create Alert Rule` on the [Alert Rules](https://app.spotflow.io/alerting/alert-rules) page. ## Select Alert Type [#select-alert-type] Choose how you want to define the condition that will trigger the alert. There are two types: * [Threshold](/fundamentals/monitoring/alerts): Alert when a metric crosses a fixed threshold value. * Example: CPU Usage above 90%. * [Percentual Change](/fundamentals/monitoring/alerts): Alert when a metric changes by a percentage over time. * Example: Battery voltage drops by 20%. ## Define Alert Condition and Intervals [#define-alert-condition-and-intervals] In the query builder, define the query that selects the relevant metric data and set the threshold, evaluation interval, and evaluation window for the alert rule. If you want to be notified when a device goes offline or fails to report a specific metric, you can also set the alert to trigger when no data is received. ## Define Alert Notification Targets [#define-alert-notification-targets] Specify a descriptive name for the alert rule and create a new notification target that should receive the alert notifications. Add any email addresses that should receive the notifications and save the alert rule. Alert notifications are sent immediately when an alert is triggered and also when the alert is resolved. If you wish to use a different notification method, please open a Feature request or let us know via email [hello@spotflow.io](mailto:hello@spotflow.io) or our Discord. We will be happy to work with you to incorporate necessary changes to the platform or find other suitable solution. ## See Evaluations and Alerts [#see-evaluations-and-alerts] After creating the alert rule, you can see the last evaluations of the rule. On the [Alerts](https://app.spotflow.io/alerts) page, you can see the history of triggered alerts. You can also click on an individual alert to see more details about it. ## Add Tags to Alerts [#add-tags-to-alerts] You can create and assign custom tags to the alert to make it easier to filter alerts in the alert list. Filter alerts by tags to quickly find relevant alerts in the alert list. ## Learn more [#learn-more] # Create custom dashboard (/using-spotflow/guides/custom-dashboards) ## Create Dashboard [#create-dashboard] Start creating your own dashboard by clicking `+ Create Dashboard` on `Dashboards` in [Spotflow portal](https://app.spotflow.io). Pick a suitable name and, optionally, a longer description. ## Add Premade Widgets [#add-premade-widgets] Add one or more premade widgets. These widgets are pre-configured to show key metrics and insights about your device fleet, such as connectivity, crashes, firmware versions, and more. They provide a great starting point for monitoring your devices without needing to set up custom queries or configurations. You can check existing widgets in [Dashboards](/fundamentals/monitoring/dashboards) documentation or directly in the [Spotflow portal](https://app.spotflow.io). ## Add Custom Widgets [#add-custom-widgets] Add one or more custom widgets. Use GUI builder to configure data queries and visualizations to create tailored views of your device data. Use the builder to select a metric name and optionally filters and grouping based on built-in or custom labels. ## Using the Dashboard [#using-the-dashboard] After adding all required widgets, save the dashboard and start using it to monitor your devices. Configure the desired time range and auto-refresh frequency. Anytime, you can go back and: * Add or remove widgets. * Resize & rename existing widgets. * Create additional dashboards. ## Learn more [#learn-more] # Invite members to your workspace (/using-spotflow/guides/invite-members) ## Invite Members to your Workspace [#invite-members-to-your-workspace] Select the workspace you want to invite members to, click on its name in the top-left corner, and choose `Settings` from the dropdown menu. Under the `Workspace` section, click on `Members`. Enter the email address of the person you want to invite. Use `Add Another` to invite several people at once, then click on `Invite Members`. Below the invite form, the `Member List` tab shows current members and the `Pending Invitations` tab shows people who haven't accepted yet. ## Accept the Invitation [#accept-the-invitation] The invited members receive an email titled *"You've been invited to join \ on Spotflow"*. They open it and click `Accept Invitation`. They are taken to Spotflow to sign in with the invited email address. If they don't have a Spotflow account yet, they can create one using `Sign up` or a social login. After signing in or creating an account, the invited members can accept the invitation to join the workspace by clicking on `Accept`. ## Switch Workspaces [#switch-workspaces] After accepting the invitation, the invited members can switch between the invited workspace and their own by clicking on the workspace name in the top-left corner and selecting the desired workspace from the dropdown menu. ## Remove a Member (Optional) [#remove-a-member-optional] To revoke someone's access, go back to `Settings` → `Members`. On the `Member List` tab, click the trash icon next to the member you want to remove. To cancel an invitation that hasn't been accepted yet, do the same on the `Pending Invitations` tab. ## Learn more [#learn-more] # Deploy Over-the-Air (OTA) updates (/using-spotflow/guides/ota) ## Prerequisite: Device OTA Integration [#prerequisite-device-ota-integration] Spotflow allows you to manage OTA updates for your embedded devices. [Guide: Over-the-air updates with Zephyr](/guides/zephyr/ota-zephyr) [Guide: Over-the-air updates of external MCUs with Zephyr](/guides/zephyr/ota-external-mcu-zephyr) The Spotflow device module can automatically perform OTA updates of your main firmware on Zephyr devices. It also allows you to handle OTA updates of external MCUs connected to your Zephyr device. [Guide: Over-the-air updates with MQTT](/guides/mqtt/ota-mqtt) For devices running other platforms or when you cannot use the Spotflow device module, integration of OTA updates is also possible via the standard MQTT interface. ## Upload the Firmware File [#upload-the-firmware-file] Make sure you have created [firmware and firmware versions](/fundamentals/firmware-management) on the [Firmwares](https://app.spotflow.io/firmwares) page. This step uploads the binary that devices will download during the update. Upload your firmware image by clicking `Upload Firmware Image File` for the selected firmware version. ## Select Which Devices to Update [#select-which-devices-to-update] A deployment cohort is a group of devices that should receive the same OTA deployment. Create a cohort by clicking `+ Create Cohort` on the [Deployment Cohorts](https://app.spotflow.io/ota-updates) page. Enter the cohort name and select the devices that should be included in this group. ## Create a Deployment [#create-a-deployment] Create a new deployment by clicking `New Deployment` within the deployment cohort. Once you start a deployment, it becomes the active deployment for that cohort. The active deployment is propagated to devices that are already in the cohort and also to devices added to the cohort later. Select the firmware and firmware version that should be deployed to the devices in the cohort and click `Continue to review`. If you mark the firmware as main in the UI, the update is handled directly by the Spotflow device SDK. Updates of other firmware packages are handled by callbacks in your device code. Review the deployment details and click `Start Deployment`. ## Monitor Deployment Progress [#monitor-deployment-progress] You will see the newly created deployment as the latest active deployment in the deployment cohort. You can view detailed information about the deployment, including which devices have successfully received the update and which have encountered issues. ## (Optional) Retry Failed Updates [#optional-retry-failed-updates] If some devices have failed to install the update, you can retry the deployment for those devices. ## (Optional) Stop Deployment [#optional-stop-deployment] Click `Stop Deployment` to stop sending the update to devices that have not started it yet. Devices already updating are stopped on a best-effort basis and may finish anyway, while devices that already finished are not affected. ## (Optional) Rollback [#optional-rollback] Click `Rollback Deployment` to create a new deployment with the same firmware version as the previous deployment, effectively rolling back the update. ## Learn more [#learn-more] # Track security issues across your devices (/using-spotflow/guides/track-security-issues) Spotflow lets you upload Software Bills of Materials (SBOMs), which list the software components in a firmware version, then automatically checks their dependencies for known CVEs and tracks their impact across your devices. This gives you a clear view of vulnerable dependencies and their impact on deployed devices, while helping you meet software inventory requirements such as those in the Cyber Resilience Act (CRA). ## How it works [#how-it-works] 1. Upload one or more SBOM files to a firmware version. 2. Spotflow extracts components and dependency information from the files. 3. Spotflow matches the extracted components against known CVEs. 4. As the CVE database is updated, Spotflow re-evaluates each SBOM to detect newly disclosed vulnerabilities in existing firmware releases. ## Open the firmware version [#open-the-firmware-version] Go to [Firmwares](https://app.spotflow.io/firmwares), select a firmware, and open the version you want to scan. ## Upload the SBOM files [#upload-the-sbom-files] Click `Upload SBOM Files`, then select the SBOM files for this firmware version (tag-value SPDX 2 and JSON SPDX 3 SBOMs are supported). Spotflow extracts the components and dependencies from the files and matches them against known CVEs. The Security tab shows a summary of the detected security issues, the number of affected devices, and the associated CVEs. ## Review the security issues [#review-the-security-issues] Filter issues by name, severity, assessment state, affected component, or whether the CVE appears in the CISA Known Exploited Vulnerabilities (KEV) catalog. Select a row in the security issues table to view its details. Each issue includes its severity, the affected dependency, the dependency chain that introduces it, and other relevant details. ## Assess the issues [#assess-the-issues] Each security issue has an assessment state that indicates whether it affects your firmware. New findings start as `Not Assessed`. As your team investigates an issue, set its state to `In Triage`, `Affected`, or `Not Affected`. You can add details to each assessment. Following the Vulnerability Exploitability eXchange (VEX) approach, every `Affected` assessment requires a response, while every `Not Affected` assessment requires a justification selected from a predefined list. You can also ignore an issue when it should not appear in active lists or summaries. The audit timeline records every assessment and ignore action, including who made the change, when, and why. ## Monitor deployed firmware [#monitor-deployed-firmware] Open [Security](https://app.spotflow.io/security) to prioritize remediation work across your fleet. The page lists firmware versions with security issues that are deployed to your devices, along with the number of affected devices. Select the affected-device count to view the devices running that firmware version.