Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch

A lightweight two-stage approach for parsing, preprocessing and storing semi-structured home automation data The full configuration example is available on GitHub. Home automation systems tend to generate a surprisingly heterogeneous collection of logs and measurements. In this project, most data originates from Domoticz, but additional information is produced by ESP8266/D1 devices, alarm scripts, monitoring tools and other IoT components. Some records are plain text, some contain timestamps and metadata, while others contain JSON embedded inside a log line. Rather than forcing every producer into a single format, I use a two-stage processing architecture based on Fluent Bit and Elasticsearch . The complete chain is: IoT / home automation sources | +----------------+----------------+ | | | Domoticz ESP/D1 Other scripts | | | +----------------+----------------+ | Fluent Bit | parsing + preprocessing | v Elasticsearch | ingest pipeline | final JSON decoding | v dom_v3 index | +----------+----------+ | | Kibana Grafana The complete pipeline normally has a latency of less than one minute in the author's installation. 1. Why two processing stages? The main design decision is to avoid making Fluent Bit responsible for every transformation. Fluent Bit performs the first level of processing: reading the various log files; recognizing several recurring line formats; extracting timestamps and metadata; decoding JSON when it is directly available; removing intermediate fields; forwarding the resulting records to Elasticsearch. Elasticsearch then performs the final JSON decoding through an ingest pipeline. This separation keeps the Fluent Bit configuration relatively simple while giving Elasticsearch responsibility for the final document transformation. 2. Fluent Bit parsers The incoming data belongs to several recurring families. Instead of trying to recognize all of them with one very large regular expression, the configuration uses several smaller parsers. A typical Domoticz