Azure Event Hub

Azure Event Hub is a log-based messaging queue that can be used as an Egress Sink.

tip

See Consume Data in Your Systems page to learn more about configuring Egress Sinks and Routes.

Supported egress events:

Kind	Is supported
Messages	✅
Batch Completions	✅

When used as an Egress Sink, each logical Message or Batch Completion is routed to the Event Hub as a physical Event Hub event.

The Azure Event Hub events have two parts: body and properties. The body is a simple byte array, and the properties are a set of key-value pairs. In the context of Egress Sinks, the properties are used to transport the metadata about the Message/Batch Completion Event, such as stream-name. Depending on the Egress Route configuration and size of the routed Message, the body might contain the actual payload of the Message that arrived from a device. The exact format specification can be found below.

Configuration

In order to route a Message or a Batch Completion, the target Event Hub must be configured as an Egress Sink and one or more Streams must have Egress Route targeting that Egress Sink.

To configure the Egress Sink, the following parameters are required:

• Connectiong String

  • Connection string for Event Hub level shared access policy with at least Send permission/claim. The connection string must contain SharedAccessKey and SharedAccessKeyName properties. Property SharedAccessKeySignature is not supported.
  • Example: Endpoint=sb://<event-hub-namespace-hostname>;SharedAccessKeyName=<key-name>;SharedAccessKey=<key>;EntityPath=<event-hub-name>;.

The associated Egress Routes can be customized with the following parameters:

• Disable Payload Embedding: Flag that, when set to true, prevents Message payload to be embedded into the Event Hub event body. This can be used to optimize Event Hub throughput. Default is false which means that the payload is embedded if possible
• Partition Key Pattern: This parameter can be used to customize Event Hub Partition Key for routed events. Please see details below.

Partition Keys

It is important to consider partitioning strategy when using Event Hubs. By default, following partition key is used:

Partition Key = <workspace-id>-<stream-group-name>-<stream-name>-<device-id>

This partition key ensures that all events from the same device are routed to the same partition. This allows to preserve the order of events and efficient caching during processing.

Partition Key Pattern

If the default Partition Key is not suitable for given use case, it can be customized using Partition Key Pattern parameter of the Egress Route. This pattern is a string that can be interpolated with several variables (using the {variableName} syntax). Example of such string is:

Pattern = example_{workspaceId}_{streamName}

Suppose that a Message arrived from Workspace 69f09b3f-ec0d-4b9e-a5ec-87150b935296 and Stream telemetry, the materialized partition key will be:

Partition Key = example_69f09b3f-ec0d-4b9e-a5ec-87150b935296_telemetry

Following variables are available for interpolation:

• {streamGroupName}: Name of the stream group the message was sent into.
• {streamName}: Name of the stream the message was sent into.
• {deviceId}: Id of the device that sent the message.
• {workspaceId}: Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
• {guid}: Randomly generated UUID/GUID (v4). Generated by cryptographically secure pseudo-random number generator (CSPRNG) and formatted as 32 hexadecimal digits (lowercase) separated by hyphens. Generated at the time of materialization.

The resulting partition key must be, at most, 1024 characters long.

Message format

If possible, body of Event Hub event contains embedded payload of the Message that arrived from a device. This is possible only if:

• The payload must be smaller than 900 KiB.
• The associated Egress Routes must not have Disable Payload Embedding flag set to true.

Otherwise, the body of the Event Hub event is empty.

In any case, following properties are available on the Event Hub event:

Name	Type	Example	Description
kind	string	Message	Identifier that allows to distinguish amongst different kinds of events. For messages, this is always set to Message.
stream-group-name	string	group-a	Name of the stream group the message was sent into.
stream-name	string	telemetry	Name of the stream the message was sent into.
device-id	string	robot-125	Id of the device that sent the message.
batch-id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
batch-slice-id	string	logs	Identifier of batch slice (if it was provided by the device).
message-id	string	m00767	Identifier of the message. It is provided by device or auto-filled by the platform (if configured).
is-embedded-payload	boolean	true	Is true when the Message payload is embedded directly, otherwise false. If payload is not embedded, it can be downloaded via payload-sas-uri property.
payload-sas-uri	string		SAS URI to the Message blob in Stream Storage.
workspace-id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.
ingress-enqueued-date-time	string	2023-12-19T11:25:56.1408925+01:00	Time when the Message was ingested by the platform. ISO 8601 format.

Batch completion format

Body of the Event Hub event is always empty and following properties are available:

Name	Type	Example	Description
kind	string	BatchCompletion	Identifier that allows to distinguish amongst different kinds of events. For Batch Completions, this is always set to BatchCompletion.
stream-group-name	string	group-a	Name of the stream group the message was sent into.
stream-name	string	telemetry	Name of the stream the message was sent into.
device-id	string	robot-125	Id of the device that sent the message.
batch-id	string	2023-12-19	Identifier of batch. It is provided by device or auto-filled by the platform (if configured).
blob-sas-uri	string		SAS URI to the blob in the Stream Storage that contains concatenated messages from given batch. Only available when stream has concatenation enabled. Not present when blob-container-sas-uri and blob-directory-path properties are set.
blob-container-sas-uri	string		Container SAS URI that can be used to list and access messages stored as individual blobs. Only available when stream has concatenation disabled. Not present when blob-sas-uri property is set.
blob-directory-path	string	group-a/telemetry/robot-125/2023-12-19	Name prefix for messages (stored as blobs) in the Stream Storage. Only available when stream has concatenation disabled. Not present when blob-sas-uri property is set.
workspace-id	string	69f09b3f-ec0d-4b9e-a5ec-87150b935296	Identifier of the Workspace that originating Device and Stream belong into. Formatted as GUID/UUID with 32 hexadecimal digits (lowercase) separated by hyphens.