Egress Sinks
Egress Sinks are destinations where data from devices can be routed. The platform supports many kinds of such destinations via a unified interface.
You can use egress sinks to:
- Run custom stream processing logic over the device data streams.
- Export the data from the platform to external systems.
- Build real-time applications, dashboards, and alerts.
Check out also the Stream Storage for other ways to consume data from the platform, which are more suitable for batch processing scenarios, or Visualize Data for visualizing the data in the Grafana.
Each stream can be routed to one or more Egress Sinks. The link between a stream and an egress sink is called an Egress Route. Apart from specifying the target Egress Sink, the route can be used to customize routing behavior such as:
- What egress events are routed. Whether individual stream messages or completed batches should be routed.
- How exactly the data should be stored in the target Egress Sink. This is specific to each Egress Sink kind. For example for SQL, the mapping of the message to SQL table columns is defined at the route level.
Supported Egress Sink kinds
The following egress sink kinds are supported:
- Azure Event Hub
- Azure Service Bus
- SQL Database
- Open Telemetry Endpoint
- Databricks Delta Table
- Includes first-class support for Open Telemetry data.
- Amazon S3
- Grafana
- The platform also provides hosted Grafana to visualise your time-series data.
- It is a special Egress Sink that is available for every Workspace out-of-the-blox.
Egress Route
The Egress Route defines to which egress sinks the data from streams are routed into.
The following guarantees are provided:
- One stream can be routed to multiple egress sinks.
- One egress sink can receive data from multiple streams.
- One route is related to exactly one stream and one egress sink.
- Data are routed to egress sinks in the order they are received from devices. Order is maintained within the context of a single stream and device.
- Data are routed to egress sinks after they are written in the stream storage.
Egress Events
There are two types of events you can consume to process data from your devices: Messages Completion Events and Batch Completion Events.
Messages Completion Events
You can receive individual messages that the devices are sending. The events are emitted when a new message is received or when the last chunk of chunked message is received.
The event contains the payload and metadata of individual messages. Exact format of the event depends on the target Egress Sink. See Azure Event Hub for an example.
Batch Completion Events
You can receive notifications about completed batches. The event is emitted when a previously open batch is completed. Depending on the stream concatenation mode.
For streams that are concatenated, the event contains a link to all messages from given batch concatenated into a single blob in the stream storage. For not concatenated streams, there is link to the blob container in the stream Storage and folder (name prefix) for messages belonging to the completed batch. In both cases, additional metadata about the batch are available. Exact format of the event depends on the target egress sink. See Azure Event Hub for an example.