Skip to main content

Tutorial: Receive Data in Your Observability Backend

This tutorial will show you how to route data to an observability backend, such as New Relic, Splunk, Grafana Cloud, DataDog, and Honeycomb. You will:

  • Create a free Grafana Cloud account in case you don't have any observability backend
  • Create an OpenTelemetry Egress Sink
  • Configure a Stream to forward data to the Egress Sink
  • See data sent by your Device in Grafana Cloud

OpenTelemetry Egress Sink Overview

Requirements​

  • If you are not registered to the Spotflow IoT Platform yet, Sign Up.
  • You need to have an existing Stream. A Stream called default-stream in Stream Group default-stream-group is created automatically after you register. Create a new one if you want to use a different Stream.

1. Create Free Grafana Cloud Account​

info

You can skip this step if you already have an existing observability backend.

There are many options when choosing an observability backend and the decision should be made according to your needs. We have published two blog articles to help you understand the caveats of monitoring and observability in the IoT world:

To give you a quick start, we've created a simple tutorial on how to create a free Grafana Cloud account.

1
2
3
4
5
6
7
8

  1. Go to https://grafana.com/ and create a free account.

  2. Sign in using one of the available options.

  3. Fill in the details about your newly created Grafana stack and click Finish Setup. You will be redirected to Grafana UI, but go to Grafana Cloud Portal.

  4. In the Grafana Cloud Portal, navigate to your stack by clicking on its name.

  5. Click the Configure button in OpenTelemetry section.

  6. Here you can find the details of your Grafana OTLP endpoint. You don't need to copy it now, but remember where to find it, you will need it later to configure Spotflow IoT Platform. Click Generate now.

  7. Name the token. The predefined scopes are sufficient, Spotflow IoT Platform requires metrics:write, logs:write and traces:write. Click Create token.

  8. Copy and securely store the token, you won't be able to reveal it later. This is the credential token allowing Spotflow IoT Platform to write metrics, logs and traces to your Grafana. You can close the window now.

2. Route Data to a new OpenTelemetry Egress Sink​

An OpenTelemetry Egress Sink represents an existing observability backend to which the Platform can forward data. Let's create a new one and configure a stream to forward data to it.

1
2
3
4
5
6

  1. Click on Data Flows in the left sidebar.

  2. Click on the Add Route button next to the stream that should be routed to the OTEL Egress Sink.

  3. Choose the Create a New Egress Sink option and click on the Next button.

  4. Fill the name of the Egress Sink, e.g. grafana-cloud. Select OpenTelemetry as the egress sink kind. To configure the connection, you need to provide the URL of the OTLP endpoint, username and password. Click Create Egress Sink and Continue.

  5. The new OpenTelemetry Egress Sink is now created! At this point, it is only needed to create an egress route so the data from your stream are forwarded to this sink. Review the default configuration and optionally customize it. If you want to enrich the telemetry with labels such as deviceId, expand the section Enrich Telemetry with Labels and select the appropriate labels. Click Create Egress Route to finish.

  6. The route to the new OpenTelemetry Egress Sink has been created! You should see an arrow going from your stream to the Egress Sink.

3. Send Data to Platform​

Once the Spotflow IoT Platform is configured as described above, you can start exporting telemetry signals from your Device's program using the Device SDK. There are only two requirements when sending data to an OpenTelemetry Egress Sink:

  1. You must send the signals in the binary Protobuf encoding.
  2. You must set the Batch Slice ID of each message depending on which signal are you exporting. The following values are allowed: logs, metrics, and traces.

Example Using Device SDK to Export Metrics​

Your telemetry data can be exported from your Device's program with the help of the standard OpenTelemetry SDK. Specifically, we recommend creating a custom metric exporter. This allows you to instrument your Device's program in the standard way, and the telemetry signals will be automatically routed to your observability backend.

Obtain Dependencies​

Install the recent Device SDK and OpenTelemetry SDK for Python using the following command:

pip install --upgrade spotflow-device opentelemetry-sdk opentelemetry-exporter-otlp-proto-common

Create Custom OTEL Exporter​

You can create a custom OTEL exporter by implementing the opentelemetry.sdk.metrics.export.MetricExporter abstract class. To start collecting metrics using the custom exporter, you need to create a metric reader and pass it to your MeterProvider. Take a look at the following simple implementation:

send_otel_data.py
import time
from typing import Dict

from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.common.metrics_encoder import encode_metrics
from opentelemetry.proto.metrics.v1.metrics_pb2 import AggregationTemporality
from opentelemetry.sdk.metrics.export import MetricExporter, MetricExportResult, MetricsData, PeriodicExportingMetricReader
from opentelemetry.sdk.metrics.view import Aggregation
from opentelemetry.sdk.metrics import MeterProvider

from spotflow_device import StreamSender, DeviceClient

class SpotflowMetricExporter(MetricExporter):
    def __init__(
        self,
        stream_sender: StreamSender,
        preferred_temporality: Dict[type, AggregationTemporality] = None,
        preferred_aggregation: Dict[type, Aggregation] = None,
    ):
        super().__init__(
            preferred_temporality=preferred_temporality,
            preferred_aggregation=preferred_aggregation,
        )
        self.__stream_sender = stream_sender

    def export(self, metrics_data: MetricsData, timeout_millis: float = 10_000, **kwargs) -> MetricExportResult:
        print("Sending metrics.")
        # Serialize metrics data and send it via StreamSender
        self.__stream_sender.send_message(encode_metrics(metrics_data).SerializeToString(), batch_slice_id="metrics")
        return MetricExportResult.SUCCESS

    def force_flush(self, timeout_millis: float = 10_000) -> bool:
        return True

    def shutdown(self, timeout_millis: float = 30_000, **kwargs) -> None:
        return None

# Connect to the Platform (starts Device Provisioning if the Device is not already registered)
client = DeviceClient.start(device_id="my-device", provisioning_token="<Your Provisioning Token>", db="spotflow.db")

# Create a sender to the stream of your choice
sender = client.create_stream_sender(stream_group = "my-new-stream-group", stream = "my-new-stream")

# Initialize the custom exporter
spotflow_exporter = SpotflowMetricExporter(sender)

# Set up the periodic exporting metric reader
metric_reader = PeriodicExportingMetricReader(spotflow_exporter, export_interval_millis=1000)

# Set up the meter provider with the metric reader
meter_provider = MeterProvider(metric_readers=[metric_reader], shutdown_on_exit=False)

# Register the meter provider globally
metrics.set_meter_provider(meter_provider)

# Get a meter from the provider
meter = meter_provider.get_meter("custom-meter")

# Create a counter instrument
counter = meter.create_counter("counter")

# Record metrics in a loop
for i in range(15):
    counter.add(1)
    time.sleep(1)

Note that there are only two concepts specific to the Spotflow IoT Platform in the code:

  1. The class requires an instance of StreamSender in its constructor. This stream sender must be configured to send messages into the stream that is routed to your OpenTelemetry egress sink.
  2. The export method sends the data via the aforementioned StreamSender instance. Remember, that the telemetry data must be encoded in the binary Protobuf format. In the example this is achieved with the OpenTelemetry SDK. Lastly, it's important to note that the batch_slice_id parameter needs to be set to metrics in this case.

Run Program​

Paste the code above into a new file, or download it. Replace the placeholder <Your Provisioning Token> with your Provisioning Token, enter your Stream Group and Stream, and run the program:

python send_otel_data.py

DeviceClient.start will display the Provisioning Operation ID and Verification Code:

Provisioning operation initialized, waiting for approval.
Operation ID: eb82df7f-63ab-429f-80df-f0ae05f57306
Verification Code: 3cwz1g1c

After you approve the operation in the next step, the Device will start sending metrics to the Platform.

4. Approve Device​

While the code on the Device side is still running, you'll now approve the Provisioning Operation started in the previous step:

1
2
3
4

  1. Expand the link Devices in the left sidebar if it's not already expanded. Open the link Approvals.

  2. The list of Provisioning Operations opens. You should see a row with the Device ID my-device. Click Approve in the column Actions.

  3. The approval dialog window shows the Provisioning Operation details and allows you to override Device registration details. Check the Verification Code, leave the default values and click Approve.

  4. Good job! The Device disappeared from the table because its state changed to Approved, while the current filter shows only the Devices in the state Waiting for Approval.

Finally, if you look at the program output, you should see that the device is registered and started sending metrics:

Provisioning operation initialized, waiting for approval.
Operation ID: eb82df7f-63ab-429f-80df-f0ae05f57306
Verification Code: 3cwz1g1c
Sending metrics.
Sending metrics.
Sending metrics.
Sending metrics.
...

6. See Data in Grafana Cloud​

info

The following text demonstrates how to visualize data in Grafana Cloud. Refer to the documentation of your observability backend if you use a different one from Grafana Cloud.

1
2
3
4
5
6
7

  1. Go to https://grafana.com/ and click My Account (or sign in if you needed).

  2. In the Grafana Cloud Portal, navigate to your stack by clicking on its name.

  3. Click Launch in Grafana section. Grafana UI will open.

  4. Click Explore in the left menu.

  5. Select the Prometheus datasource, in this example it is called grafanacloud-smolatomas-prom.

  6. Select metric counter_total and labels identifying data sent by your Device. Click Run query.

  7. Explore the counter metric sent by your Device through the Platform to Grafana Cloud.

Copyright © 2024 Spotflow, s.r.o.