Stream Storage
The Spotflow IoT Platform provides Stream Storage for each Stream Group. It is a dedicated Azure Storage Account used as blob storage for all the Messages from the Streams.
You can provide a connection string to your Azure Storage Account when creating a Stream Group and let the Spotflow IoT Platform to write Messages to the Storage Account.
All messages are stored in messages container as individual blobs with the following structure:
messages/<stream-group>/<stream>/<device-id>/<batch-id>/<message-id>
Messages that failed to be stored for some reason (e.g. invalid Stream, missing Message ID or Batch ID) are persisted in dead-letter container.
Concatenation
You can configure the Stream to one of 3 options:
- No concatenation - All Messages result in different files (blobs) in the Stream Storage.
- Concatenation without new lines - A single file is created for each Batch that contains all Messages combined in the order the Devices sent them to the Spotflow IoT Platform. Messages are also available in separate blobs without concatenation.
- Concatenation with new lines - A single file is created for each Batch that contains all Messages combined in the order the Devices sent them to the Spotflow IoT Platform with a new line character (
\n) between them. Messages are also available in separate blobs without concatenation.
If Stream has any concatenation enabled, the resulting Batch blobs are stored in batches container and have the following structure:
batches/<stream-group>/<stream>/<device-id>/<batch-id> (the blob contains all Messages for the Batch)