AWS Timestream

Description

ezvpn-mqtt-awstimestream listens on an MQTT broker and writes data to Amazon Timestream Service.

In IOhubTM architecture, ezvpn-mqtt-awstimestream will subscribe fld/+/r/# topic to the ezvpn-mqtt broker and store data read to Amazon Timestream Service.

AWS Timestream Schema

How to use it

ezvpn-mqtt-awstimestream is a Docker container image pre-configured for communication with ezvpn-mqtt and Amazon Timestream. In order to use it, you first need an Amazon AWS account with Timestream Service enabled. Then you need to install ezvpn-mqtt with a Field container and configure ezvpn-mqtt-awstimestream with AWS credential and required data. All data read from the field will be stored in AWS Timestream automatically.

To create a data credential, you need to have an account on Amazon AWS. Click this link to create the credentials to connect to AWS Timestream Service if you don't have it already.

Create your first database on AWS Timestream and a table where you want to store data. When you create the table you need to specify data retention as well.

AWS Create DB

After database creation, add the Environment variables as required to store data to AWS. Here the result:

AWS Timestream

Timestream Dimension

Using the METADATA variable, the data is sent to AWS Timestream service adding the AWS dimension to the measurement:

{
    "property1": "<value>",
    "property2": "<value>",
    "property3": "<value>"
}

If the variable METADATA is defined, each property will be used as the dimension name; the property value (usually a string) will be used as the dimension value.

In the example below, METADATA value is

{
    "company": "ACME",
    "machine_name": "cutter 23",
    "serial_number": "1234567890"
}

and the dimensions company, machine_name, and serial_number will be sent to Timestream.

Buffer and Batch mode

You can enable buffer using MQTT_WANT_QOS2 and RETRY_ON_ERROR environment variables.

The messages are sent to AWS Timestream service individually. If you have a lot of messages to send you can group them using GROUP_SEND_SIZE functionality.

Environment variables

When you start the AWS Timestream image, you can adjust the instance's configuration by passing one or more environment variables to the docker run command.

MQTT connection

  • MQTT_HOST: MQTT host. Defaults to 127.0.0.1
  • MQTT_PORT: MQTT port. Defaults to 1883
  • MQTT_IN_TOPIC: IOhubTM MQTT topic to subscribe (e.g. fld/modbus/r/#). Defaults to fld/+/r/#. Subscribe to multiple topics can be obtained defining each topic on a separated line.
  • MQTT_WANT_QOS2: ask MQTT to upgrade each incoming message on MQTT_IN_TOPIC to QoS 2 (delivery guaranteed). MQTT will honor the request only if persistence is enabled.

AWS connection

  • AWS_ACCESS_KEY_ID(Required): AWS Access Key.
  • AWS_SECRET_ACCESS_KEY(Required): AWS Secret Access Key.
  • AWS_REGION: AWS Timestream DB region (e.g. us-east-1). Defaults to us-west-2.
  • AWS_DATABASE_NAME(Required): AWS Timestream DB.
  • AWS_TABLE_NAME(Required): AWS Timestream DB Table.
  • METADATA: An object containing additional dimensions to provide to Timestream (protocol is always provided). If present, it must be a JSON object, each property representing a dimension name, with its value.

Retry on error

  • RETRY_ON_ERROR: if true, on AWS communication failure, Timestream write is retried every RETRY_INTERVAL milliseconds, forever. Otherwise the message is discarded. Defaults to false.
  • RETRY_INTERVAL: interval in milliseconds between each write retry to Timestream (if GROUP_SEND_SIZE = 1) or between each queue retry (if GROUP_SEND_SIZE > 1). Defaults to 5000. Used only if RETRY_ON_ERROR is true.

Group send

  • GROUP_SEND_SIZE: send grouped records to Timestream. It represent the max size of the buffer. If buffer is full, with the next incoming record the behavior is as if no buffer exists (the behavior for each incoming message depends upon the RETRY_ON_ERROR environment variable). Used to optimize TCP traffic and with high data volume. Defaults to 1 (no grouping).
  • GROUP_SEND_EVERY: try to empty buffer every GROUP_SEND_EVERY milliseconds. Defaults to 5000. Size the GROUP_SEND_SIZE accordingly to GROUP_SEND_EVERY to avoid accumulating messages on MQTT (if RETRY_ON_ERROR is true) or losing messages (if RETRY_ON_ERROR is false).

Other variables

  • SKIP_RETAIN: if true, discard all incoming MQTT messages with retain flag set to true. Defaults to false.

With SKIP_RETAIN set to false, each client that subscribes to a topic pattern receives the retained message immediately after they subscribe.

The broker stores only one retained message per topic.

Docker container details

Image: us-central1-docker.pkg.dev/ez-shared/iohub/iohub-mqtt-awstimestream

Supported architecture: amd64

Changelog

v1.0.4
  • Smaller image
v1.0.4
  • MQTT_IN_TOPIC accepts multiple topics
v1.0.3
  • MQTT_WANT_QOS2 Environment variable added
  • GROUP_SEND_SIZE Environment variable added
  • GROUP_SEND_EVERY Environment variable added
  • SKIP_RETAIN Environment variable added
  • BUFFERED renamed as RETRY_ON_ERROR
v1.0.2
  • BUFFERED Environment variable added
  • RETRY_INTERVAL Environment variable added
v1.0.1
  • Added METADATA Environment Variable
v1.0.0
  • First Release