File

Description

ezvpn-fld-file is a service (provided as Docker container image) acting as a file system reader/writer, which:

  • reads a modification on the local file system, addition, modification, or deletion of files, and sends files as MQTT messages.
  • subscribes to MQTT topics and writes incoming messages as files to the local file system

File

This container has reading and writing functionalities from/to the local file system, as it happens for the standard field containers reading/writing from/to PLCs, devices, ... Standard mqtt messages ({ "value": <value>, "ts": <timestamp> }) are used, as it happens for any other field container. The content of value is the file content, base64 encoded.

Reading: when a file is created, modified or deleted in the directory under the folder definition in the pollings section, its content is sent to the MQTT broker on topic fld/file/r/folder/subfolder/<filename>.

For example: if you create a file named image.png on the filesystem in the images/foo/bar path, the content of the image.png file is sent as a message to the topic fld/file/r/images/foo/bar/image.png

A deleted file is sent with null as a value.

Writing: when a message is received on the topic fld/file/w/<filename>, a file is written; the written file content with is the mqtt message content, after base64 decoding.

For example: if a message is received on the topic fld/file/w/csv/archive/csvfile1.csv, the container will save the file csvfile1.csv inside the folder csv/archive. Any missing intermediate folder is automatically created.

If you want to delete a file, you have to send an mqtt message containing #.

For example: if you want to delete the file a/b/c.txt, you have to send the message # to the topic fld/file/w/a/b/c.txt.

Local file system

Any folder managed by ezvpn-fld-file is relative to /data. E.g., the topic fld/file/w/csv/archive/csvfile1.csv represents the file /data/csv/archive/csvfile1.csv.

It is not possible to read/write files outside the /data folder.

The /data folder should be mapped on a persistent volume, otherwise its content would be zeroed upon restarts.

How to use it

ezvpn-fld-file is a Docker container image pre-configured to communicate with ezvpn-mqtt. In case you need to use it with your MQTT broker, please refer to the section below regarding customization.

If used with ezvpn-mqtt, only one environment variable must be provided: FLD_CFG.

FLD_CFG is a string representing a JSON object, describing the read/write file system configuration.

In the example below:

  • if a file is sent to the topic fld/file/w/<filename>, it will not be written to the local file system because writable is set to false. With writable set to false, the container is operating one-way, sending file system notifications to MQTT, without listening for incoming write/delete requests.
  • every 5 seconds, every modification to the file system under the directory folder1 (and in every folder below recursively) is sent over mqtt. Intermediate modifications, happening during the five seconds are neglected. Only the situation at the 5 seconds expire is sent.
  • As loadExisting is set to true, at each startup all the files found in the folder1 (and in every folder below recursively) are sent over MQTT.
  • every morning at 8 am in the Los Angeles time zone all the files found in the folder folder2 (and in every folder below recursively) are sent over MQTT.
  • as soon as a modification occurs in folder3 (or in the folders below recursively), it is sent over MQTT.
{
    "writable": false,
    "pollings": [
        {
            "every": "5s",
            "folder": "folder1",
            "loadExisting": true
        },
        {
            "cron": "00 8 * * *",
            "timezone": "America/Los_Angeles",
            "folder": "folder2"
        },
        {
            "immediate": true,
            "folder": "folder3"
        }
    ]
}

File system write detections are debounced for two seconds. A write is considered as complete if no further modification is written within two seconds.

File deletion is instead detected immediately, without the two seconds delay.

The example below is one-way, only listening for incoming files over MQTT. No file is sent over MQTT when the file system gets modified.

{
    "writable": true
}

Mapping configuration (FLD_CFG)

The JSON object has one object and one array field:

{
    "writables": <true or false>,
    "pollings":[]
}

writables (optional)

Defines whether on arrival of a file via MQTT it should be written to the local file system. It defaults to true.

pollings (optional)

pollings represents a list of polling cycles on the file system changes.

Each polling has its independent polling frequency and it can span over different file system folders.

This one below is the structure of a polling object:

{
    "every": "<frequency>",
    "cron": "<frequency time>",
    "timezone": "<timezone>",
    "immediate": <true or false>,
    "folder": <folder to scan>,
    "loadExisting": <true or false>
}

If polling is defined, one of every, cron or immediate fields must be present.

  • every: can be supplied in : human readable time format
  • cron: can be supplied in : cron time format
  • timezone: an optional timezone, see the list, to use for the cron expression. If not defined UTC is used.
  • immediate: if true execute action at each file system change, It defaults to false.
  • folder: folder to listen recursively Required
  • loadExisting: At each startup all the files present in recursive mode in the configured folder will be sent to MQTT if loadExisting is true. It defaults to false.

Environment variables

When you start the ezvpn-fld-file image, you can adjust the instance's configuration by passing one or more environment variables to the docker run command.

General Configuration

  • FLD_CFG: mapping/polling configuration Required
  • MQTT_HOST: MQTT broker address, defaults to 127.0.0.1
  • MQTT_PORT: MQTT broker port, defaults to 1883
  • FROM_FILE_SYSTEM_TOPIC_PREFIX: MQTT topic use to publish values read from devices, defaults to fld/file/r/
  • TO_FILE_SYSTEM_TOPIC_PREFIX: MQTT topic subscribed to get values to write on devices, defaults to fld/file/w/
  • NO_RETAIN: if true values are sent to MQTT_HOST without the retain flag; defaults to false

Docker container details

Image: us-central1-docker.pkg.dev/ez-shared/iohub/iohub-fld-file

Supported architecture: amd64

Changelog

v1.0.0
  • First release