Logging with ESP-IDF

This guide explains how to enable log collection on devices running ESP-IDF using the Spotflow device module and analyze logs in the Spotflow web application.

Install Spotflow Device Module

As the very first step, check the following basic integration guide:

• Quickstart: ESP-IDF Integration Guide

Use ESP-IDF Logging Library

After installing and initializing the Spotflow device module, you can use the standard ESP-IDF logging macros to send the logs:

#include "esp_log.h"
#include "spotflow.h"

static const char *TAG = "main_module";

void main(void)
{
    // Initialize the Spotflow logging module
    spotflow_init();

    // Ensure the device has an active internet connection.

    ESP_LOGI(TAG, "Device has booted up successfully.");

    for (int i = 1; i <= 5; i++) {
        int reading = i * 10;
        ESP_LOGI(TAG, "Sensor reading %d: %d units", i, reading);
        if (reading > 30) ESP_LOGW(TAG, "Reading is above normal. Value: %d.", reading);
    }

    ESP_LOGI(TAG, "Device shutting down.");
}

Analyze Logs in the Web Application

Once your device is integrated and sending logs, you can analyze them in the web application. The main entry point is the Events page, which gives you a comprehensive view of all events collected from your devices. There, you can filter logs by their content, device ID, severity, and other metadata.

The Events page displays all logs with filtering and search capabilities.

You can click on individual log messages to see their details, and it is possible to drill down into specific events by matching their metadata.

It is possible to query logs based on metadata of a particular log message.

How the Device Module Works

The Spotflow device module integrates with ESP-IDF's native logging library to provide a familiar development experience. Instead of replacing your existing logging workflow, it extends it by implementing a custom vprintf-like logging handler that captures logs from standard ESP-IDF macros and transmits them to Spotflow. The original logging handler is still called, so the previous logging behavior is preserved.

Key benefits of this architecture:

• Zero code changes: Continue using ESP_LOGV, ESP_LOGD(), ESP_LOGI(), ESP_LOGW(), and ESP_LOGE() macros as before.
• Local debugging: Our device module can run alongside other logging handlers, allowing you to see logs in your local console while also transmitting them to Spotflow.

Handling Network Interruptions

The device module is designed to handle network interruptions gracefully. When the connection is unavailable, logs are stored in a circular buffer. That is, when the buffer is full, the oldest logs are overwritten with the new ones. Once the connection is restored, buffered logs are automatically sent to Spotflow.

Support for Constrained Devices

To minimize the memory footprint, the device module exposes number of configuration options to tune the buffer size and other parameters. See the available Kconfig options below.

Reliability of data delivery

Spotflow is using MQTT as the underlying transport protocol for log ingestion.

Most embedded devices running ESP-IDF are significantly resource-constrained which has several implications:

1. To minimize the footprint of the Spotflow device module, MQTT QoS level 0 is used which has minimal overhead but at the cost of weaker delivery guarantees. Although MQTT itself is TCP-based protocol and TCP guarantees transport-level delivery, with QoS 0, some messages still can be lost in case of client or broker failures and similar issues. This should not be an issue for most observability use-cases.

2. In case of a network interruption or slow down, the device module buffers not yet sent logs in a circular buffer. However, this buffer has limited size and once it is full, the oldest logs are overwritten with the new ones. Size of this buffer can be via Kconfig options.

If needed, you can create your custom integration directly with our MQTT broker to tailor the solution to your specific resource constraints vs. reliability tradeoff. Spotflow MQTT broker supports all QoS levels from 0 to 2 and is designed to handle even high throughput log streams.

Kconfig options

CONFIG_SPOTFLOW_LOG_BACKEND=y

To explicitly enable or disable Spotflow logging backend use SPOTFLOW_LOG_BACKEND. It is enabled by default.

CONFIG_SPOTFLOW_DEBUG_MESSAGE_TERMINAL=y

To enable debug messages from the Spotflow device module, enable SPOTFLOW_DEBUG_MESSAGE_TERMINAL option. It is disabled by default.

CONFIG_SPOTFLOW_MESSAGE_QUEUE_SIZE=5
CONFIG_SPOTFLOW_LOG_BUFFER_SIZE=512
CONFIG_SPOTFLOW_CBOR_LOG_MAX_LEN=1024
CONFIG_MQTT_BUFFER_SIZE=1024

You can configure these options to adjust performance and resource usage.

See KConfig for logging for details about all options above.

Learn more