Send Data From Devices

IoT devices can send data to the Spotflow IoT Platform via the Device SDK. Messages from the Devices are organised into logical “pipes” called streams. In those streams, multiple consecutive Messages are grouped into a batch representing Messages from a given timeframe (e.g. day) or your domain event (e.g. one unit of work performed by the machine). You can also configure whether Messages from each batch should be concatenated into a single file.

tip

The Spotflow IoT Platform persists device messages in stream storage and route them to external systems via egress sinks.

See Consume Data In Your Systems for more information on consuming the device data.

Message

A Message represents a standalone unit of information that you can process independently, e.g. JSON object, CSV row, image, or binary data. It originates from a single device and is identified by:

• Message ID - Each message must specify message ID if message ID Autofill isn’t enabled.
• Batch ID - Each message must specify batch ID if batch ID Autofill isn’t enabled.

Stream

When sending a message, your device's software must specify the stream to which the message should be published. A stream acts as a “pipe” that logically groups related messages from one or more devices. All messages from a stream have a specific prefix in stream storage.

One Device can send messages to multiple streams.

Batch

A batch consists of one or more messages with the same batch ID from one device that are somehow related. For example, the batch could contain all temperature readings from some time period (hour, day) or all messages for some domain event recorded by an IoT device. Device can have only one active batch at a time in one stream (see the picture below).

The device sends batches inside one stream sequentially.

Batches have three purposes:

• All messages from one batch are stored within a single folder in the stream storage.
• All messages from on batch can be concatenated into a single file in the stream storage.
• Information about completed batches can be used to trigger additional actions.

Please see the stream storage page for more details on format of stored data and concatenation feature.

Each message must belong to a batch.

Batch completion

By default, batch is considered completed when the device sends a message with a different batch ID when the previous one or a message with explicit complete-batch flag.

Optionally, stream can be configured to complete active batches automatically when some amount of time passes since the last message was received (timeout).

Batch ID and Message ID Autofill

Your device software must provide batch ID and message ID for each message when sending. Alternatively, the stream can specify batch ID or message ID Autofill. The Autofill is used when the device software doesn’t specify batch ID or message ID property.

An autofill is a string that can contain alphanumeric characters, the characters ., _, - and :, and placeholders for the following built-in variables enclosed in curly braces {...}.

Variables for Message ID

1. {dateTime:<format>} - UTC timestamp of the message delivery into the Spotflow IoT Platform. E.g. {dateTime:yyyy-MM-dd_HH:mm:ss} would produce 2023-10-30_18:02:51.
2. {guid} - Randomly generated UUID/GUID (v4). E.g. d57d25db-d9c2-4578-8cf0-724d074102d3.
3. {sequenceId} - When you send a message to the Spotflow IoT Platform, it is assigned a monotonically increasing number which is unique for each device. E.g. 00000000010000000000000180405.

info

Message ID Autofill should contain at least one unique variable: {guid} or {sequenceId} so the platform can distinguish between different messages. Otherwise, the platform will reject the message as a duplicate. Example: msg_{dateTime:yyyy-MM-dd}_{guid}.

Variables for Batch ID

1. {dateTime:<format>} - UTC timestamp of the message delivery into the Spotflow IoT Platform. E.g. {dateTime:yyyy-MM-dd} would produce 2023-10-30.
2. {guid} - Randomly generated UUID/GUID (v4). E.g. d57d25db-d9c2-4578-8cf0-724d074102d3.

Example Autofill

• Batch ID Autofill: {dateTime:yyyy-MM-dd}
• Message ID Autofill: {dateTime:HH:mm:ss}-{sequenceId}

Resulting messages will look like this (<batch-id>/<message-id>):

• 2023-10-30/18:02:51_00000000010000000000000180405
• 2023-10-30/18:02:59_00000000010000000000000180406

Chunking

When dealing with large messages, splitting them into smaller chunks is often more efficient than sending them all at once. This approach is especially helpful in environments with slow and unreliable networks or limited memory resources:

• Smaller chunks have a higher chance of being delivered successfully, thus reducing the amount of data that needs to be retransmitted.
• Devices can stream large files without storing them in memory all at once and for an unnecessary period.

Each chunk is assigned a Chunk ID (unique within the message) alongside the message ID and batch ID, and the last chunk of the message is marked with a special flag. Support for chunking is fully integrated into Device SDK.

When the last chunk is delivered, chunked message is automatically reassembled and published as a single message in the Stream Storage or Egress Sinks.

A message can be split into multiple Chunks.

Stream Group

A stream group is a logical container for one or more streams. You can use it to separate streams for different purposes or different audiences.

Each stream must be associated with a stream group.

Default Stream Group & Stream

In situations where multiple streams and stream groups are not needed, or one stream or stream Group is predominantly used, defaults can be set:

• Stream Group - The default name can be set at the Workspace level.
• Stream - The default name can be set at the stream group level.

When one or both defaults are set, the devices do not have to specify the stream group or stream when sending messages.

Stream Storage & Egress Sinks

Stream storage and egress sinks are two kinds of destinations where the messages from your devices can be stored or routed to.

Stream storage is a built-in storage capable of storing data for longer periods of time. Egress sinks are abstractions over many external storages, message brokers, APIs, and more where the data can be routed to.

For details, please see Consume Data In Your Systems for more information on consuming the data.

Batch Slices

Batch slices are an opt-in concept that allows splitting batches into smaller parts. Such splitting might be handy in more complex scenarios, where a batch has multiple logical parts (slices) that should be clearly separated but stored close to each other and processed simultaneously.

Batch slices are especially useful for streams that use the concatenation feature. Instead of big file for each batch, with batch slices, it is possible to strike the right balance between the number of files and their size, which might allow for optimal postprocessing. You can get more details on the concatenation feature in the stream storage page.

For streams without concatenation, the resulting structure in the stream storage is:

Batch slices used?	File structure
No.	One file per message: /<stream-group>/<stream>/<device-id>/<batch-id>/[file: message-id]
Yes.	One file per message in <batch-slice-id> folder: /<stream-group>/<stream>/<device-id>/<batch-id>/<batch-slice-id>/[file: message-id]

For streams with concatenation, the resulting structure in the stream storage is:

Batch slices used?	File structure
No.	One file per batch: /<stream-group>/<stream>/<device-id>/[file: batch-id]
Yes.	One file per batch slice: /<stream-group>/<stream>/<device-id>/<batch-id>/[file: batch-slice-id]

Batch slices do not require any particular configuration. The only thing needed is that devices attach the batch-slice-id property next to the batch-id property to the individual messages.

Batch slices do not affect batch completion. The completion still happens at the level of batches rather than at the level of slices. Hence, only one batch completion event is emitted regardless of the presence of batch slices.

Each message can belong to a different batch slice.