Amazon Web Services S3

Amazon Web Services S3 is an object storage service provided by Amazon Web Services (AWS). You can use your AWS S3 Bucket as an Egress Sink and store Messages or Batches from your IoT Devices as objects in your bucket.

tip

See Consume Data in Your Systems page to learn more about configuring Egress Sinks and Routes.

Supported egress events:

Kind	Is supported
Messages	✅
Batch Completions	✅

Configuration

To store Messages or Batch Completions in AWS S3 Bucket, the AWS S3 Egress Sink must be configured with the following parameters:

• Bucket Name
  • Name of the AWS S3 Bucket. For example: my-bucket.
• Region
  • The region where the bucket is located. For example eu-central-1.
• Access Key
  • Access Key credential of IAM User that has permission to access the bucket. See Creating IAM User.
• Secret Access Key
  • Secret Access Key credential of IAM User that has permission to access the bucket. See Creating IAM User.

The associated Egress Route can be customized with the following parameters:

• Target Path Config (optional)
  • This parameter can be used to add a custom prefix to the S3 objects.

Bucket Structure

Messages

If Egress Route Input is configured to Messages, the platform stores objects in the S3 bucket with the following key:

(<path>)/messages/<stream-group>/<stream>/<device-id>/<batch-id>/(<batch-slice-id>)/<message-id>

The object payload contains the payload of the Message.

The S3 object has the following metadata:

Name	Type	Example	Description
x-amz-meta-stream-group-name	string	group-a	Name of the stream group the message was sent into.
x-amz-meta-stream-name	string	telemetry	Name of the stream the message was sent into.
x-amz-meta-device-id	string	robot-125	Id of the device that sent the message.
x-amz-meta-batch-id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
x-amz-meta-batch-slice-id	string	logs	Identifier of batch slice (if it was provided by the device).
x-amz-meta-message-id	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
x-amz-meta-workspace-id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.

Batch Completions

If Egress Route Input is configured to BatchCompletions, the platform stores objects in the S3 bucket with the following key:

(<path>)/batches/<stream-group>/<stream>/<device-id>/<batch-id>

The object payload contains the concatenated Messages of the Batch in concatenated Streams. Non-concatenated Streams contain an empty object only with metadata.

The S3 object has the following metadata:

Name	Type	Example	Description
x-amz-meta-stream-group-name	string	group-a	Name of the stream group the message was sent into.
x-amz-meta-stream-name	string	telemetry	Name of the stream the message was sent into.
x-amz-meta-device-id	string	robot-125	Id of the device that sent the message.
x-amz-meta-batch-id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
x-amz-meta-batch-slice-id	string	logs	Identifier of batch slice (if it was provided by the device).
x-amz-meta-workspace-id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.

Creating IAM User

To create an S3 Egress Sink, you must create an IAM User with permission to create objects in the S3 Bucket and copy the Access Key and Secret Access Key.

Using predefined policy

1

2

3

4

5

6

7

8

9

10

11

12

13

1. Open AWS Console.

2. Select Services → IAM (manage access to AWS resources).

3. Select Access Management → Users.

4. Choose the Create User button.

5. Enter a name for the identity, for example, spotflow-egress-sink-user.

6. Select Attach policies directly → Search for AmazonS3FullAccess and select it.

7. Create the user.

8. Select the created user from the list, for example spotflow-egress-sink-user.

9. Open Security credentials.

10. Click the Create access key.

11. Select Other.

12. Create an Access key.

13. Copy the Access key value and Secret access key value.

Using custom policy (advanced)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

1. Open AWS Console.

2. Select Services → S3.

3. Choose the bucket that you want to store data in.

4. Open the Properties tab and copy Amazon Resource Name (ARN).

5. Select Services → IAM (manage access to AWS resources).

6. Select Access management → Policies.

7. Choose the Create policy button.

8. Switch to JSON editor.

9. Paste the following JSON and replace <REPLACE_WITH_YOUR_S3_BUCKET_ARN> with the ARN you copied in step 4.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:AbortMultipartUpload"
               ],
               "Resource": "<REPLACE_WITH_YOUR_S3_BUCKET_ARN>/*"
           }
       ]
   }

10. Create a name for the policy. For example, spotflow-egress-sink-policy.

11. Select Access management → Users

12. Choose the Create user button.

13. Enter the name for the identity, for example, spotflow-egress-sink-user.

14. Select Attach policies directly → Search for the policy you just created (e.g. spotflow-egress-sink-policy) and select it.

15. Create the user.

16. Select the created user from the list, for example spotflow-egress-sink-user.

17. Open Security credentials.

18. Click the Create access key button.

19. Select Other.

20. Create an Access key.

21. Copy the Access key value and Secret access key value.