Send Data From Devices

IoT devices can send data to the Spotflow IoT Platform via the Device SDK. Messages from the Devices are organised into logical “pipes” called Streams. In those Streams, multiple consecutive Messages are grouped into a Batch representing Messages from a given timeframe (e.g. day) or your domain event (e.g. one unit of work performed by the machine). You can also configure whether Messages from each Batch should be concatenated into a single file.

tip

The Spotflow IoT Platform persists Device Messages in Stream Storage and can route the data to your apps.

See Integrate Data With Your Systems for more information on consuming the Device data.

Message

A Message represents a standalone unit of information that you can process independently, e.g. JSON object, CSV row, image, or binary data. It originates from a single Device and is identified by:

• Message ID - Each Message must specify Message ID if Message ID Autofill isn’t enabled.
• Batch ID - Each Message must specify Batch ID if Batch ID Autofill isn’t enabled.

Stream

When sending a Message, your Device's software must specify the Stream to which the Message should be published. A Stream acts as a “pipe” that logically groups related Messages from one or more Devices. All Messages from a Stream have a specific prefix in Stream storage.

One Device can send Messages to multiple Streams.

Batch

A Batch consists of one or more Messages with the same Batch ID from one Device that are somehow related. For example, the Batch could contain all temperature readings from some time period (hour, day) or all Messages for some domain event recorded by an IoT device. Device can have only one active Batch at a time in one Stream (see the picture below).

The Device sends Batches inside one Stream sequentially.

Batches have three purposes:

• All messages from one Batch are stored within a single folder in the Stream Storage.
• All messages from on Batch are concatenated into a single file in the Stream Storage. See for details.
• Information about completed Batches can be used to trigger additional actions.

Each Message must belong to a Batch.

Batch completion

By default, batch is considered completed when the device sends a message with a different Batch ID when the previous one or a message with explicit complete-batch flag.

Optionally, stream can be configured to complete active batches automatically when some amount of time passes since the last message was received (timeout).

Batch ID and Message ID Autofill

Your Device software must provide Batch ID and Message ID for each Message when sending. Alternatively, the Stream can specify Batch ID or Message ID Autofill. The Autofill is used when the Device software doesn’t specify Batch ID or Message ID property.

The Autofill is a pattern that can be composed of the following variables:

1. {dateTime:<format>} - UTC timestamp of the Message delivery into the Spotflow IoT Platform. E.g. {dateTime:yyyy-MM-dd} that would produce 2023-10-30.
2. {uuid} - Universally Unique Identifier (UUID/GUID). E.g. d57d25db-d9c2-4578-8cf0-724d074102d3
3. {sequenceId} - When you send a Message to the Spotflow IoT Platform, it is assigned a monotonically increasing number which is unique for each Device. E.g. 00000000010000000000000180405
4. any arbitrary text

Example Autofill

• Batch ID Autofill: {dateTime:yyyy-MM-dd}
• Message ID Autofill: {dateTime:HH:mm:ss}-{sequenceId}

Resulting Messages will look like this (<batch-id>/<message-id>):

• /messages/2023-10-30/18:02:51_00000000010000000000000180405
• /messages/2023-10-30/18:02:59_00000000010000000000000180406

Stream Group

A Stream Group is a logical container for one or more Streams. You can use it to separate Streams for different purposes or different audiences.

Each stream must be associated with a Stream Group.

Stream Storage

See Stream Storage to understand where is your data stored.

Concatenation

You can configure the Stream to one of 3 options:

1. No concatenation - All Messages result in different files (blobs) in the Stream Storage.
2. Concatenation without new lines - A single file is created for each Batch that contains all Messages combined in the order the Devices sent them to the Spotflow IoT Platform. Messages are also available in separate blobs without concatenation.
3. Concatenation with new lines - A single file is created for each Batch that contains all Messages combined in the order the Devices sent them to the Spotflow IoT Platform with a new line character (\n) between them. Messages are also available in separate blobs without concatenation.

If Stream has any concatenation enabled, the resulting Batch blobs are stored in batches container and have the following structure: batches/<stream-group>/<stream>/<device-id>/<batch-id> (the blob contains all Messages for the Batch)

Batch Slices

Batch Slices are an opt-in concept that allows splitting batches into smaller parts. Such splitting might be handy in more complex scenarios, where a batch has multiple logical parts (slices) that should be clearly separated but stored close to each other and processed simultaneously.

Batch slices are especially useful for streams that use the concatenation feature. Instead of having one file per message or one big file for each batch, with batch slices, it is possible to strike the right balance between the number of files and their size, which might allow for optimal postprocessing.

For streams without concatenation, the resulting structure in the stream storage is:

Batch slices used?	File structure
No.	One file per message: /<stream-group>/<stream>/<device-id>/<batch-id>/[file: message-id]
Yes.	One file per message in <batch-slice-id> folder: /<stream-group>/<stream>/<device-id>/<batch-id>/<batch-slice-id>/[file: message-id]

For streams with concatenation, the resulting structure in the stream storage is:

Batch slices used?	File structure
No.	One file per batch: /<stream-group>/<stream>/<device-id>/[file: batch-id]
Yes.	One file per batch slice: /<stream-group>/<stream>/<device-id>/<batch-id>/[file: batch-slice-id]

Batch slices do not require any particular configuration. The only thing needed is that devices attach the batch-slice-id property next to the batch-id property to the individual messages.

Batch slices do not affect batch completion. The completion still happens at the level of batches rather than at the level of slices. Hence, only one batch completion event is emitted regardless of the presence of batch slices.

Each message can belong in different Batch Slice.