Log ingestion tools move events from a source—such as a container's stdout, a file, or an OTLP sender—into a receiver that can store and search them. A collector may parse, redact, enrich, batch, and retry along the way. It is not the same thing as the searchable backend. The useful choice starts with the source you must read, the protocol the destination accepts, and what should happen when either side slows or fails.
Fluent Bit, Vector, the OpenTelemetry Collector, Filebeat, and Logstash can all participate in that path, but they are not identical products or a single tier of alternatives. Filebeat is primarily a shipping agent; Logstash is a processing pipeline; the others can run as local agents or intermediate collectors with different component and delivery models. No one tool can recover an event the application never emitted or guarantee storage merely because a send call returned. Their current buffering, acknowledgment, resilience, and queue documentation makes those boundaries explicit.
Map the handoffs before choosing software
A small logging pipeline has four distinct responsibilities:
- Source: the application, container runtime, host journal, file, or network sender creates an event. Decide who owns its format, timestamp, severity, sensitive-data policy, and rotation.
- Local collection: an agent reads the source and tracks its position or receives a protocol message. It may add host, service, or Kubernetes context. If it starts late or loses its cursor, the receiver cannot reconstruct the missing records.
- Transport and processing: one or more collectors parse, redact, route, buffer, and send to a receiver. A queue is a finite handoff, not a permanent archive. Backpressure may pause a source, reject traffic, or drop records depending on the component and settings.
- Destination: a receiver authenticates and accepts the event, then a backend stores it for search and retention. Check whether an HTTP acknowledgment means queued or durably stored; those are different claims.
Many deployments need only a source-side agent and a destination. Add an aggregation tier or external message queue when it solves a measured routing, isolation, or outage problem; it also adds capacity planning, access control, and another failure point. Define a source-to-stored-row test before buying features or comparing throughput claims.
Fluent Bit: compact collection at the edge
Fluent Bit is a practical fit when many hosts or Kubernetes nodes need to tail local files or container output and forward records with modest parsing and metadata enrichment. Its pipeline has inputs, parsers/filters, buffering, routing, and outputs. A local agent avoids making each application depend synchronously on a remote logging endpoint. Review the exact input and output plugins for the source format and destination protocol rather than assuming every pair has the same behavior.
Fluent Bit can buffer in memory or use filesystem-backed chunks. The current buffering guide explains that memory-only buffers are more exposed to a restart; filesystem buffering reduces that risk but consumes disk and still has failure limits. A bounded output queue can discard its oldest chunks when it fills. Its ring-buffer mode deliberately drops older chunks to keep the newest. These are policy choices, not a universal “durable mode.” Size disk and retry behavior for the expected outage, monitor queue pressure and dropped chunks, and confirm what happens when the source is paused or rotates while the destination is unavailable.
Choose Fluent Bit when a lightweight node-level collector plus known parsers and routes meets the requirement. If extensive transformations or strict end-to-end acknowledgments dominate the design, compare those exact components and limits with Vector or a staged pipeline rather than inferring guarantees from a disk-buffer checkbox.
Vector: agent or aggregator with explicit acknowledgment checks
Vector can collect, transform, and route events as an agent or an aggregator. Its source, transform, and sink components should be evaluated as a specific path, because capabilities differ by component. For example, a transformation can intentionally filter an event; that is not a transport failure. Record which drops are policy and which indicate overload.
Vector's guarantees documentation distinguishes best-effort from at-least-once paths. End-to-end acknowledgments require a source that can wait for them and a sink with acknowledgments enabled. For survival across restarts, the relevant sink also needs a disk buffer with an appropriate full-buffer policy. Vector says disk buffering alone is not enough to claim end-to-end delivery, and it does not claim exactly-once delivery. A successful acknowledgment can mean the event was persisted to Vector's disk buffer for later retry, not that the final search backend has already indexed it.
Vector is worth evaluating when routing and transformations need to be maintained as one coherent pipeline and you can test the precise source/sink pair. Validate transform behavior with representative fixtures, including malformed JSON, unexpected field types, secrets, and events that must never be sampled out.
OpenTelemetry Collector: OTLP-oriented collection and export
The OpenTelemetry Collector is useful when applications already export OTLP logs or when a shared telemetry pipeline is an intentional standard. Its receivers, processors, and exporters are configurable components; the exact distribution and component version determine which inputs and outputs are available. A logs pipeline is separate from metrics and traces, even though the Collector framework can carry multiple signals.
For network exporters, the Collector resilience guide recommends sending queues and retry. In-memory queues can be lost on a crash and eventually fill; retry windows can expire. If restart survival matters, the file_storage extension can back a sending queue with persistent storage, but that still cannot survive every disk failure, full queue, or prolonged destination outage. Do not copy an old queue/retry default into a new deployment: verify the installed Collector distribution, exporter, and configuration schema, then monitor queue size, capacity, send failures, and freshness.
This is a strong candidate for an OTLP-native environment or an agent-to-gateway design. It is not automatically the best file tailer for every host, and an OTLP label on a receiver does not prove a destination accepts all three telemetry signals. Validate the logs endpoint and authentication separately.
Filebeat and Logstash: shipping plus processing
Filebeat is a log shipper with file inputs and an internal queue. Its current queue reference describes memory and disk queues; a full disk queue can pause or discard new events according to the input's behavior. The disk queue also costs disk I/O and must be bounded. Filebeat does not provide an arbitrary HTTP output just because another service has an HTTP ingest endpoint. Its supported output list is specific: Elasticsearch/Elastic Cloud, Logstash, Kafka, Redis, file, console, and discard in the current reference. Check the actual output protocol before drawing a destination arrow.
Logstash is a more substantial processing stage when inputs, codecs, enrichment, conditional routing, or an existing Elastic pipeline justify it. Its persistent queues are disabled by default. Enabling one can protect accepted events through a normal or abnormal Logstash restart, but the documentation warns that events not yet checkpointed may be lost and a disk or host failure is outside the queue's protection. A queue also cannot provide upstream acknowledgment for an input such as ordinary TCP or UDP that has no request/response acknowledgment. Pair Filebeat and Logstash when their output/input contract and operational ownership are understood; do not call the combination zero-loss by default.
Filebeat/Logstash may be a straightforward continuation for teams already operating them. For a new, simple source-to-receiver path, compare the extra runtime, queue, transformation configuration, and upgrade burden with a single agent. Compatibility and ownership matter more than an abstract “enterprise” label.
Decide by source, failure policy, and destination
Use a short decision path instead of a universal ranking:
- List the actual sources. Are logs in stdout/stderr, rotating files, the host journal, network appliances, or OTLP exporters? Test the exact collector input, cursor behavior, multiline assembly if needed, and permissions. A receiver cannot fix a missing source.
- Name the output contract. Confirm protocol, TLS validation, authentication, endpoint path, accepted payload fields, response semantics, and rate/size limits. A product name is not a wire protocol. Filebeat's Logstash output, for example, is not a generic JSON-over-HTTP sender.
- Choose the loss behavior. How long can the destination be unavailable? What happens when disk or memory is full? Can the source be paused and replayed, or is it a fire-and-forget feed? Disk queues plus source acknowledgment can support stronger at-least-once behavior on compatible paths, but they do not imply exactly-once, and a failed disk or an unacknowledged UDP source can still lose data.
- Normalize only what is useful. Agree on event time, severity, message, service, environment, and a safe correlation field. Parse multiline before treating each physical line as a record. Redact tokens and personal data before broad egress; preserve source semantics when mapping fields. Avoid dropping rare errors or required audit evidence through a blanket noise filter.
- Price the whole pipeline. Count source volume and bytes, network transfer, collector CPU and memory, queue disk, destination storage/retention, and operator time. Reducing useless debug output at the source is often clearer than shipping it everywhere and paying to discard it later. No product's advertised throughput number predicts your parser, burst pattern, or outage behavior.
If the source is a single application that can send authenticated HTTP JSON directly, an agent may be unnecessary. If the application should not block on the network, local stdout plus a collector is usually the cleaner boundary. If many services already emit OTLP logs, start by evaluating the Collector's logs pipeline. If Filebeat is already deployed, check whether its supported output path reaches your selected receiver before replacing it.
Prove the path during a controlled failure
Run this in a non-production environment before trusting the design in an incident:
- Emit a unique, non-sensitive marker from the real source, with a known event timestamp and service name. Verify the collector reads it and the destination stores a searchable row in the intended stream or index. Inspect the stored raw row and mapping; a receiver acknowledgment alone is insufficient. For an HTTP JSON sender, the data ingestion example shows a separate receiver-bound marker check.
- Pause or isolate the destination in the test environment, not the production ingest endpoint. Continue emitting a known count of markers. Watch the queue length, disk usage, retry/error counters, source pause, and drops. Record when the buffer reaches its limit and which component owns the failure.
- Restart the collector while records are queued. Restore the destination, then compare emitted, accepted, stored, missing, and duplicated marker IDs. An at-least-once path may legitimately replay duplicates; a low count may indicate loss or a query-time filter.
- Repeat with an invalid credential, malformed event, rotated source file, and full or unavailable queue storage where safe. Confirm that errors are visible to an operator and that the source has a defined fallback or loss policy.
Do not assume a single green pipeline metric proves completeness. Source counts, collector receive/drop/export counters, receiver responses, and stored-row checks observe different boundaries. Test real outage and restart behavior against the required loss budget, then keep a small synthetic marker check after routing or credential changes.
Choose and verify a Fluxtail log receiver
The backend should be selected separately from the collector. Fluxtail is a paid Starter/Pro, logs-focused destination, not a host agent or a general telemetry store. Its receiver matrix documents HTTP JSON, OTLP logs over HTTP or gRPC, and dedicated protocol receivers. For an application or general agent, HTTP JSON uses an exact HTTPS receiver URL and a receiver-bound Bearer token. An OpenTelemetry logs exporter uses the matching OTLP logs receiver; that receiver does not accept traces or metrics. The Docker collection guide describes one Fluent Bit path, but its source path, token, retry storage, and final field mapping must be validated in the deployment.
Once a known marker arrives, use the receiver's stream in Live Tail and check the documented search and filters against the fields actually stored. Fluxtail does not make a collector's missing event reappear, supply a source acknowledgment that the original protocol lacks, or turn a 202 queued response into proof of durable search storage. Fluxtail also documents Prometheus and StatsD receivers, but this log workflow is not a substitute for a metrics, tracing, APM, or SLO system. Choose the smallest pipeline whose failure behavior you can explain and test end to end.