Compact logs protocol and format

Compact logs send a format string address and raw argument bytes instead of a formatted log line. Spotflow reads the string from the matching firmware ELF file, decodes the arguments, and formats the message in the cloud. This page is the producer contract for an MQTT integration. If you use the Spotflow Zephyr module, see Logging with Zephyr for setup.

How a compact log becomes readable text in Spotflow.

Before sending compact logs

1. Upload the exact firmware symbol file (the ELF file containing the format strings and referenced constant strings) through Firmware Management.
2. Report that ELF file's Build ID as buildId in Session Metadata before sending logs on each connection.

CBOR fields

Compact logs are only supported in CBOR, not in JSON. Use these fields in place of body or the bodyTemplate / bodyTemplateValues pair and labels in a CBOR log message:

For example, suppose address 134222388 contains Door opened by %s after %u ms and the ELF file specifies little-endian byte order. This CBOR message, shown in diagnostic notation, produces Door opened by Maria after 250 ms using the default argument layout:

{46: 134222388, 47: h'00000000fa000000', 48: {0: "Maria"}, 4: 40}

Bytes 0–3 hold the string pointer slot, supplied by the embedded string at offset 0. Bytes 4–7 (fa 00 00 00) encode the unsigned integer 250; severity 40 means INFO. The format string comes from the ELF file; the user-defined name Maria is sent with the message because it is transient.

Compact labels

You can use the CBOR map compactLabels (CBOR key 49) instead of labels to save bandwidth by sending addresses of constant strings instead of their contents. For example, if 0x08001234 addresses source and 0x08005678 addresses motor_driver, you can create the label map {"source": "motor_driver"} by adding the following key and value to the log message:

{49: {0x08001234: 0x08005678}}

If you want to send a label with integer value, annotate the value with the CBOR tag 15. For example, send the following to create the label map {"motor_driver": 9}:

{49: {0x08005678: 15(9)}}

In general, this is how Spotflow interprets the map:

Keys:

• Unsigned integer: address used to look up a null-terminated string from the ELF file.
• Text string: used as a literal key.

Values:

• Untagged unsigned integer: address used to look up a null-terminated string from the ELF file.
• Unsigned integer tagged with CBOR tag 15: literal integer value.
• Text string, negative integer, float, or boolean: (regardless whether tagged or not) used as a literal value.

If a message contains both compactLabels and labels, the resulting maps are merged together. If a key is present in both maps, the value in labels is used in the result.

Argument layout

The device provides the argument values in the compactBodyTemplateValues byte array. Spotflow determines argument types and order from the format string and decodes them from left to right. Byte order always comes from the ELF file's EI_DATA field, for both integers and floating-point values.

By default, each type has four-byte alignment and the sizes below. These default sizes produce no padding between arguments. The optional custom argument layout overrides sizes and alignments.

String arguments

For %s, Spotflow resolves the string as follows:

1. If compactEmbeddedStrings contains the pointer slot's zero-based byte offset in compactBodyTemplateValues, Spotflow uses that text.
2. Otherwise, Spotflow reads the pointer and retrieves the null-terminated string from the ELF file.

The slot occupies its configured size and alignment even when its pointer is ignored. %p formats the pointer value without looking up a string.

Format strings and constant strings in the ELF file use Latin-1, mapping each byte to the corresponding Unicode code point. Transient strings must be extracted on the device and sent as CBOR text strings in compactEmbeddedStrings; their runtime contents are unavailable in the ELF file.

For lookup in the ELF file, place format strings and referenced constant strings in file-backed SHT_PROGBITS sections with SHF_ALLOC set and SHF_WRITE, SHF_EXECINSTR, and SHF_TLS clear. These flags identify allocated, read-only, non-executable sections outside thread-local storage; section names do not matter. Zephyr's log_strings section is an explicit exception to the allocation, write, and execute flags, so strings retained only in the ELF file remain eligible. It must still be a file-backed, non-TLS SHT_PROGBITS section. Compressed string sections are unsupported.

Use target virtual addresses from the linked, fixed-address executable ELF file. Each string's null terminator must lie within the same eligible section; an address may point into the middle of a string. Writable data, executable sections, zero-initialized memory, debug string tables, and other sections present only in the ELF file are not sources for string lookup. Send string arguments from those locations in compactEmbeddedStrings, and send a format string from an ineligible location as bodyTemplate. Retain .spotflow.compact_logs separately as argument-layout metadata; it is not a string source.

Custom argument layout

The optional .spotflow.compact_logs section in the ELF file describes the sizes and alignments Spotflow uses to decode argument bytes. Include it when your firmware's argument layout differs from the defaults.

The section contains exactly one 24-byte record:

• All numeric fields are single unsigned bytes. Integer and pointer sizes are independently 1–8 bytes; the double size is 2, 4, or 8 bytes.
• Every alignment is independently 1–8 bytes. Signed and unsigned counterparts share a layout.
• The original argument offset is nonnegative. Zero means alignment is relative to the transmitted buffer's start.
• Use a non-allocated SHT_PROGBITS section outside loadable segments so it consumes no device memory. Retain it during linking and any stripping of the ELF file.

Spotflow uses the declared sizes rather than inferring them from the ELF file's 32-bit or 64-bit class. Without this section, it uses the default sizes, four-byte alignment for every type, and original argument offset zero.

Padding

Spotflow expects each argument to begin at the next multiple of its alignment in the original data layout, whose base must satisfy all declared alignments. The original prefix is the data preceding the argument buffer on the device, such as a logging record header; its length is the original argument offset. The transmitted buffer excludes this prefix but retains padding before and between arguments.

This example has an eight-byte original prefix, a four-byte int, and an eight-byte long long. Aligning the latter to eight bytes adds four padding bytes:

Padding between arguments: an eight-byte prefix is omitted from the transmitted buffer.

With a four-byte prefix and a single eight-byte long long aligned to eight bytes, the transmitted buffer starts with four padding bytes:

Padding before the first argument: the prefix is omitted, but the leading padding is transmitted.

Supported representations and formatting

The currently only supported semantics profile (value 1 in the .spotflow.compact_logs section) defines these rules for both default and custom layouts:

• Numbers: eight-bit bytes and two's-complement signed integers. Floating-point arguments use the double layout entry: size 2 selects IEEE 754 binary16 (half precision), 4 selects binary32 (single precision), and 8 selects binary64 (double precision). Mixed-endian floating-point storage is unsupported.
• Promotions: char and short arguments occupy an int slot; float arguments occupy the configured double slot. Integer modifiers hh and h narrow the decoded value to eight and sixteen bits without changing the slot size. Floating-point modifier l also uses the double slot.
• Argument consumption: each dynamic width or precision (*) consumes an int before the corresponding value, %% consumes nothing, and %n consumes a pointer without producing output or writing memory. %n accepts integer length modifiers but no flags, width, or precision.
• Unsupported: long double (L), wide characters and strings (%lc, %ls), and positional arguments.

Malformed or unsupported format strings and fixed width or precision exceeding 2147483647 leave the compact body unresolved.

A missing, duplicate, or invalid layout section selects defaults. Argument bytes must contain all required slots, and referenced strings from the ELF file must be available for a compact body to be decoded. An embedded string is used only when its offset matches a decoded %s pointer slot; entries at other offsets are ignored.