FluentCon North America 2021

The Fluent Ecosystem has recently added support for Prometheus and Open Metrics. This hands-on session will give attendees a chance to both learn and try out the new capabilities while being guided by a Fluent maintainer and Prometheus community expert. in this session we will cover the following: * An introduction to Fluent ecosystem and metrics * How to spin up an open source metrics backend (M3DB) as well as visualization platform (Grafana) * Setting up Fluent Bit to collect and send metrics to the backend; Viewing and visualizing data that is sent by Fluent Bit * Question and answer section. This session is suitable for learners of all skill levels, and attendees can expect to leave having a better grasp of how these Cloud Native projects can be used together to strengthen observability.

FluentBit Operator was created by the KubeSphere community to solve several problems:

1. Collect K8s logs through a light-weighted agent like Fluent Bit.

2. Control Fluent Bit via Kubernetes API.

3. Collect logs and then send them to the final destination without having to go through Fluentd.

4. Enable dynamic config reloading for Fluent Bit to reload its config whenever the config changes without restarting the Fluent Bit Pod.

FluentBit Operator has reached its maturity level gradually after two and a half years of iterations, and KubeSphere and Fluent Community collaborated to transfer it to the Fluent organization.

Sometimes Fluentd is still needed as an optional sink to aggregate logs received from Fluent Bit, then filter and forward to more destinations. So FluentBit Operator maintainers are considering adding Fluentd CRDs and rename the project to Fluent Operator.

In this talk, FluentBit Operator maintainers will talk about the architecture and use cases of FluentBit Operator and the design of Fluent Operator.

Fluent Bit has many useful plugins and features but by far its most powerful feature is its support for custom scripts written in a programming language called Lua. Unfortunately, it can also often be one of Fluent Bit's most underutilized features, with most users limiting themselves to minor variations of the sample scripts provided in Fluent Bit's documentation. This is because Lua is not as well-known as other programming languages, and its unfamiliarity can make it seem intimidating to people and prevent them from taking full advantage of its capabilities. This talk aims to help users become more comfortable using Lua outside the bounds of those sample scripts by walking through some novel scenarios, as well as sharing some tips and tricks, to help users take advantage of Lua to unlock Fluent Bit's true potential.

At DigitalOcean App Platform we use Fluent Bit for log processing of internal components and for customer-facing functionality. This session will review how we collect internal logs from dozens of Kubernetes clusters into a centralized logging system utilizing a custom plugin. We will also describe how we dynamically configure Fluent Bit for customer applications' log processing. This presented several operational, scalability, and security considerations. Orchestrating Fluent Bit in a security-restricted environment proved challenging. We also need detailed observability into applications' logging pipeline health and metrics. Different implementation approaches and potential future improvements will be discussed.

For the cloud-native transformation of Couchbase, we had to do something about our logs. We had various logs scattered around from different language binaries (Java, Erlang, C, etc.) and in different formats with none fit for “human consumption”.

If you’ve ever started tearing your hair out over differing log formats then this is the talk for you! If you have ever tried to understand what a developer was thinking when adding a log over multiple lines then this is also the talk for you! We include some specific examples of difficult logs and how we solved them with no change to the log itself.

Pragmatism was essential:

• We did not have the resources to refactor every log statement across all these teams/codebases.
• We needed to support an approach that allowed us to provide an out-of-the box solution but also easy integration with anything already deployed.
• We had a desire to focus on CNCF or at least open source solutions that are loosely coupled to lower the limits for adoption.

A blog post covering more in-depth examples to support this presentation can be found here.

We have seen the impact of data formats, messaging sizes, event rates, and even data sources causing challenges. For the successful operation in a large scale environment, understanding the performance characteristics of Fluentd and designing a proper solution architecture are important. TK has had the opportunities to work with many large institutions with thousands of servers deployed with Fluentd / Fluent Bit and will show how internals of Fluentd / Fluent Bit impact performance as well as benchmark results. We will share tips and tricks on how to optimize and tune parameters for your own deployments based on characteristics of input, filter, buffer and output pipelines. Lastly, we will discuss how to go beyond tuning and how to increase performance with additional compute / multi workers / and deployment architectures.

Understanding threats is the goal of a SIEM, which collects, enriches, and correlates events and threats. Learn how to build an open-source SIEM with Fluentd and Fluent Bit. SIEMs primary challenge is data collection and scale. Security infrastructure generates diverse data. We will cover data sources along with how to parse and security signals. We will provide real-world examples of how these data collection systems are used to bring together security data into an open-source SIEM. Learn how extracting metrics from logs with Fluentd can provide additional data to understand your organization’s security posture. The EFK Stack is very popular for log analytics. This includes the life cycle of collection, indexing, and storing them. Log data is valuable, but use cases for logging are operational for observability and debugging. The security world typically uses other tools, but building it on top of the same logging stack is efficient. SIEM takes a centralized approach to collection, enrichment, and analysis. As we know in today’s environments, we must ultimately federate this work to the edge to reduce data volumes and take action faster. While this is not something that SIEMs do today, it’s something that the future SIEM and technologies like Fluentd will provide.