It’s not a surprise that year over year the number of connected embedded devices is growing exponentially; from one side many Hardware challenges are being addressed properly, we can find cheap devices that provides low power consumption with good connectivity, improved processors and flexible architectures in general. These are enough reasons to encourage the creation of innovative solutions for different markets such as automotive, medical and home automation within many.
When creating software on top of these new hardware, we often needs to address the next layer of challenges, one of them is data. The operating system and the set of applications that runs on your device often generate events that helps to troubleshoot of monitor general statuses, so collecting this information is a must.
There is one concept that is changing over the years, sometimes data is commonly known as logs, but logs are not longer considered a register as a set of bytes in the file system, logs are now a stream of events that usually can be collected through a serial or network interface.
Early this year at Treasure Data, we started researching the best ways to address all concerns described above for Embedded Software, for short how to solve data collection. I have to recall that data usually comes in different formats, while the Kernel writes to it Kernel messages interface, others applications writes to syslog or push some message to some network service.
Treasure Data solved data collection for servers with the creation of Fluentd, but this tool may not be applicable as a solution for all Embedded devices because of design and resources required, but there is a lot of acquired experience that can be used as the foundation for a new specialized solution, that’s why Fluent Bit have been created.
Along this year, the project have been growing and it’s becoming a great solution for data collection and data routing in general, the following table describe the usual scenarios on how Fluent Bit can collect data:
| modes | description |
|---|---|
| library | Fluent Bit can be build as a shared or static library that can be used by your own program. So you invoke it API and ingest data to it. |
| metrics generator | Fluent Bit auto-generate data from it built-in input metrics: CPU, Memory and Kernel log messages. |
| remote service | This mode means that Fluent Bit runs as a service in a separate hardware and your embedded applications can send data to it. |
In a more descriptive way, Fluent Bit provides input types for data, the current ones available as of v0.5 release are:
| Input Plugin | Description |
|---|---|
| CPU | Calculate CPU usage in the system |
| Memory (metric) | Gather memory usage in the system |
| Kernel log messages | Gather Kernel logs from the Kernel messages interface |
| Serial | Read data from the serial interface |
| STDIN | Read data from the standard input interface |
| MQTT | Listen for MQTT messages over the network. |
| XBee | Listen for messages over a XBee Radio device. |
Once the data is collected, it should go somewhere, that’s when the output plugins provide different options.
| Output Plugin | Description |
|---|---|
| Fluentd | Fluentd is a famous open source data collector for servers, it provides an extensive list of interfaces for collecting and routing data to third party services. |
| Elastic Search | Elastic Search is an open source database to search & analyze Data in Real Time. |
| Treasure Data | Treasure Data provides an Analytics Cloud service for Big Data. |
| STDOUT | Print event records to the standard output. |
The project has reach a good level, flexible enough to behave as a puzzle in terms of which features to include and which not, it allows you to collect and route events easily across the network.
As this year I had the opportunity to attend and speak at a few conferences, I had learn from very experienced Embedded Engineers and end-users who shared their concerns and ideas on how data should be collected, this have been very useful to define current roadmap and general features available today.
Fluent Bit is Open Source and licensed under the terms of Apache License v2, feel free to contribute through our public repository on http://github.com/fluent/fluent-bit, or if you just want to get in touch, join our Slack team at http://slack.fluentd.org.
As said, embedded/IoT data matters, and Fluent Bit take cares of it, spread the word!