[Filebeat] Filestream running as Log input under Elastic Agent or feature flag #46587

belimawr · 2025-09-12T16:33:01Z

Proposed commit message

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

~~## Disruptive User Impact~~

Author's Checklist

Convert configuration from Log input to Filestream
- Create PR
- Merge configuration conversion PR Convert Log input configuration to Filestream #47013
Add feature flag to protobuf
- Add feature flag to protobuf and import it for testing
- Finish/merge [PoC] Add LogAsFilestream feature flag elastic-agent-client#143
Add integration tests running Log input as Filestream and migrating file states
Ensure the current integration tests for the Log input pass when running Filestream
- Refactor the tests that only rely logs

How to test this PR locally

Manual test

Create a log file with at least 1kb

docker run -it --rm mingrammer/flog -n 20 > /tmp/flog.log

Start Filebeat with the following configuration

filebeat.inputs:
  - type: log
    id: log-as-filestream
    allow_deprecated_use: true
    paths:
      - /tmp/flog.log

output.file:
  path: "${path.home}"
  filename: output
  rotate_on_startup: false  

queue.mem:
  flush.timeout: 0

logging:
  to_stderr: true
  level: debug
  selectors:
    - "*"

#features:
#  log_input_run_as_filestream:
#    enabled: true

Look at the logs, you will see some logs from the Log input

{
  "log.level": "debug",
  "@timestamp": "2025-09-12T12:28:06.899-0400",
  "log.logger": "input.harvester",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Log).Read",
    "file.name": "log/log.go",
    "file.line": 111
  },
  "message": "End of file reached: /tmp/flog.log; Backoff now.",
  "service.name": "filebeat",
  "input_id": "94a20b13-6927-4ff4-8f99-4f750469ed96",
  "source_file": "/tmp/flog.log",
  "state_id": "native::26052-40",
  "finished": false,
  "os_id": "26052-40",
  "harvester_id": "69128be5-d1f4-4493-935a-889d0461c95d",
  "ecs.version": "1.6.0"
}

Stop Filebeat

Check the number of events published

% wc -l output-*.ndjson        
20 output-20250912.ndjson

Uncomment the features: section in the configuration

Start Filebeat again, you'll see some logs from the Filestream input

{
  "log.level": "debug",
  "@timestamp": "2025-09-12T12:31:07.586-0400",
  "log.logger": "input.filestream",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read",
    "file.name": "filestream/filestream.go",
    "file.line": 139
  },
  "message": "End of file reached: /tmp/flog.log; Backoff now.",
  "service.name": "filebeat",
  "id": "log-as-filestream",
  "source_file": "filestream::log-as-filestream::fingerprint::445d01af94a604742ab7bb9db8b5bceff4b780925c2f8c7729165076319fc016",
  "path": "/tmp/flog.log",
  "state-id": "fingerprint::445d01af94a604742ab7bb9db8b5bceff4b780925c2f8c7729165076319fc016",
  "ecs.version": "1.6.0"
}

Check the number of events published, it should still be 20

% wc -l output-*.ndjson        
20 output-20250912.ndjson

Elastic Agent

Create a log file with some lines
docker run -it --rm mingrammer/flog -n 20 > /tmp/flog.log

Run a standalone Elastic Agent with the following configuration (adjust the output settings as necessary)

elastic-agent.yml

outputs:
  default:
    type: elasticsearch
    hosts:
      - https://localhost:9200
    username: "elastic"
    password: "changeme"
    preset: latency
    ssl.verification_mode: none

inputs:
  - type: log
    id: your-input-id
    streams:
      - id: your-log-stream-id
        data_stream:
          dataset: generic
        # run_as_filestream: true
        paths:
          - /tmp/flog.log

agent.monitoring:
  enabled: false
  logs: false
  metrics: false

agent.logging:
  level: debug
  to_stderr: true

Ensure all events have been ingested
Look at the logs, you will see Log input logs as described in the manual test
Stop the Elastic Agent
Uncomment run_as_filestream: true from the configuration
Start the Elastic Agent again
Ensure no more data is added to the output, no data duplication.
Look at the logs, you will see Filestream input logs as described in the manual test

You can also collect the diagnostics and look at the registry

Collect the diagnostics and extract it
Go to components/log-default
Extract the registry: tar -xf registry.tar.gz
cat registry/filebeat/log.json|jq -Sc

You will see the entries from the Filestream input starting with the same offset as the ones from the Log input

{"id":3,"op":"set"}
{"k":"filebeat::logs::native::16-50","v":{"FileStateOS":{"device":50,"inode":16},"id":"native::16-50","identifier_name":"native","offset":2113,"prev_id":"","source":"/tmp/flog.log","timestamp":[280186759520503,1762292780],"ttl":
-1,"type":"log"}}                                                                                                
{"id":4,"op":"set"}
{"k":"filestream::your-log-stream-id::native::16-50","v":{"cursor":{"offset":2113},"meta":{"identifier_name":"native","source":"/tmp/flog.log"},"ttl":-1,"updated":[281470681743360,18446744011573954816]}}                    
{"id":5,"op":"remove"}
{"k":"filebeat::logs::native::16-50"}
{"id":6,"op":"set"}
{"k":"filestream::your-log-stream-id::native::16-50","v":{"cursor":{"offset":2113},"meta":{"identifier_name":"native","source":"/tmp/flog.log"},"ttl":-1,"updated":[281470681743360,18446744011573954816]}}

Run the tests

cd filebeat
mage clean
go test -v -count=1 -run=TestRunAsFilestream ./input/logv2
mage BuildSystemTestBinary 
go test -v -count=1 -tags=integration -run=TestLogAsFilestream ./tests/integration 

cd ../x-pack/filebeat
mage clean
mage BuildSystemTestBinary
go test -v -count=1 -tags=integration -run=TestLogAsFilestream ./tests/integration

Related issues

Convert Log input configuration to Filestream #47013

~~## Use cases~~
~~## Screenshots~~
~~## Logs~~

github-actions · 2025-09-12T16:33:10Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-09-12T16:33:45Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

…tream

elasticmachine · 2025-11-06T16:35:01Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

.buildkite/filebeat/filebeat-pipeline.yml

cmacknz · 2025-11-06T16:50:48Z

Have you tested this with the inputs when running as beats receivers? It is critically important that this change has no impact on that conversion right now.

Enabling this feature flag, and then running the log input as a beat receiver later just works.
Running the log input as a beat receiver first, and then enabling the flag just works
Running the log input as a beat receiver first, and then enabled and disabling the flag just works

Think of any cases where these changes could interact and make sure there is no impact.

belimawr · 2025-11-07T15:30:13Z

Have you tested this with the inputs when running as beats receivers? It is critically important that this change has no impact on that conversion right now.

Enabling this feature flag, and then running the log input as a beat receiver later just works.

Running the log input as a beat receiver first, and then enabling the flag just works

Running the log input as a beat receiver first, and then enabled and disabling the flag just works

Think of any cases where these changes could interact and make sure there is no impact.

It should have no effect because the changes are at the input instantiation. Anyways, I'll create tests for those scenarios as well.

filebeat/_meta/fields.common.yml

…o POC-log-as-filestream

AndersonQ

I left a few questions

AndersonQ · 2025-11-10T13:45:27Z

docs/reference/filebeat/exported-fields-log.md

+**`log.file.device_id`**
+:   The device ID used for the log file, this is used by the 'native' file identity.
+
+    type: keyword
+
+    required: False
+
+
+**`log.file.inode`**
+:   The inode of the log file, this is used by the 'native' file identity.
+
+    type: long
+
+    required: False
+
+


Out of curiosity, is this related to this PR?

Some Python tests started failing because those fields were not in fields.yml, so I added them there, which auto-generates this documentation.

I'm debating whether I should just modify the test to ignore those fields 🤔

AndersonQ · 2025-11-10T13:59:27Z

filebeat/input/logv2/input.go

+	// Only allow to run the Log input as Filestream if Filebeat
+	// is running under Elastic Agent.
+	if !management.UnderAgent() {
+		return false, nil
+	}


Shouldn't it be before the features.LogInputRunFilestream() or together, features.LogInputRunFilestream() && management.UnderAgent()?

Otherwise, it can run as filestream even if it isn't under agent.

Otherwise, it can run as filestream even if it isn't under agent.

That's the whole point: the feature flag overwrites everything. This is required to have all the Log input tests to run with Filestream. It also gives us flexibility to test or even use it without Elastic Agent.

Then, you you mind explaining it a bit better? Perhaps add on the feature flag if it's set, it runs. Then, if it isn't set, it needs to be running under agent.

Otherwise, it isn't clear if it was intentional or a mistake

AndersonQ · 2025-11-10T14:03:29Z

filebeat/input/logv2/input.go

+// PluginV2 returns a v2.Plugin with a manager that checks whether
+// the config is from a Log input that should run as Filestream.
+// If that is the case the Log input configuration is converted to
+// Filestream and the Filestream input returned.
+// Otherwise v2.ErrUnknownInput is returned.


It seems outdated, no error is returned

That's referring to the manager's behaviour. I'll clarify/re-write it.

AndersonQ · 2025-11-10T14:06:23Z

filebeat/input/logv2/input.go

+	// When inputs are created, inputs V2 are tried first, so if we
+	// are supposed to run as the Log input, return v2.ErrUnknownInput


I'm confused. Above you say the input v1 might return v2.ErrUnknownInput so filestream takes over. here you say v2 returns v2.ErrUnknownInput so it runs as the log input.

Is the doc outdated or did I miss something?

I see it can be confusing, I'll clarify the comments by not mentioning the expected behaviour of other parts of Filebeat.

Inputs V2 are tried first on a standalone Filebeat, but I believe I've seen the opposite happening. Regardless of that both, V1 and V2 inputs will always return v2.ErrUnknownInput if they should not start an input.

filebeat/tests/system/test_autodiscover.py

AndersonQ · 2025-11-10T14:31:27Z

filebeat/tests/system/test_modules.py

+        # if the test file contains '.journal', later it will try to remove
+        # the '--once' flag and the journald input will be used,
+        # so there is nothing to do here.


Sorry, what do you mean? I didn't understand what you're trying to clarify here

This whole test is very complex :/

It can work with log files and journal files, for journal files there is some special code. The variable test_file contains the full name of the file the test is going to ingest, if it contains .journal in the name, we know it is trying to ingest a journal file and therefore the journald input will be used.

When adding the changes for the Log input running as Filestream, some of the special handling conflicted, so I added ".journal" not in test_file to this condition to avoid having problems with the cmd.remove("--once") that is done here and down below on L207-L208.

…tream

belimawr · 2025-11-11T19:58:19Z

elastic/elastic-package#3012 should help testing this PR with Integrations.

…tream

[WIP] PoC to run Filestream as log input

78fbce1

belimawr self-assigned this Sep 12, 2025

belimawr added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Sep 12, 2025

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 12, 2025

mage update

5df3a38

belimawr changed the title ~~[WIP] PoC to run Filestream as log input~~ [PoC] Filestream running as log input Sep 12, 2025

belimawr added 21 commits September 22, 2025 17:17

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

d84f890

…tream

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

937cc57

…tream

Update configuration flag

1dbbe4a

[WIP] Add happy path integration tests

d5fd62a

improve tests

44ba92f

Only run under EA and update tests

2d4d74e

[WIP] Config translate

8aab9a7

[WIP] Test for parsing config

04225e3

Add feature flag to run log input as filestream

fb963d4

try fixing parsers

65f2dae

reformat code/fix code

bb3008c

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

1fbe05d

…tream

New approach for config translation

c9a3264

Add json parser and ensure Filestream config is valid

e74987d

Fix boolean types

5c85111

Merge configs and remove log input only fields

1c0b381

organise yaml

fc22095

Set type and take_over

6ff022a

refactor code

fd25b68

Rename test file

22581aa

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

5030e8b

…tream

Small refactoring

3ddd356

github-actions bot deployed to docs-preview November 4, 2025 22:25 View deployment

belimawr added 2 commits November 6, 2025 11:27

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

d979d76

…tream

make update/check

bbf93c7

belimawr marked this pull request as ready for review November 6, 2025 16:34

belimawr requested review from a team as code owners November 6, 2025 16:34

belimawr requested review from mauri870 and orestisfl November 6, 2025 16:35

github-actions bot deployed to docs-preview November 6, 2025 16:35 View deployment

belimawr requested a review from rdner November 6, 2025 16:35

v1v reviewed Nov 6, 2025

View reviewed changes

.buildkite/filebeat/filebeat-pipeline.yml Outdated Show resolved Hide resolved

Merge branch 'main' into POC-log-as-filestream

64385c0

github-actions bot deployed to docs-preview November 7, 2025 15:24 View deployment

colleenmcginnis reviewed Nov 7, 2025

View reviewed changes

filebeat/_meta/fields.common.yml Outdated Show resolved Hide resolved

filebeat/_meta/fields.common.yml Outdated Show resolved Hide resolved

belimawr added 2 commits November 7, 2025 17:38

Move mage PythonIntegTestLogAsFilestream to its own pipeline step

b819a97

Merge branch 'POC-log-as-filestream' of github.com:belimawr/beats int…

41824e4

…o POC-log-as-filestream

github-actions bot deployed to docs-preview November 7, 2025 22:39 View deployment

AndersonQ reviewed Nov 10, 2025

View reviewed changes

belimawr added 2 commits November 10, 2025 12:14

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

59ca6c4

…tream

Remove fields, update Python tests and docs

fe33cdd

belimawr requested a review from AndersonQ November 10, 2025 17:50

belimawr mentioned this pull request Nov 11, 2025

Allow users to specify a download source for beats packages elastic/elastic-package#1319

Open

belimawr added 2 commits November 11, 2025 15:12

Fix autodiscover tests

fcc47a8

Merge branch 'main' of github.com:elastic/beats into POC-log-as-files…

d2ceaf2

…tream

		// When inputs are created, inputs V2 are tried first, so if we
		// are supposed to run as the Log input, return v2.ErrUnknownInput

[Filebeat] Filestream running as Log input under Elastic Agent or feature flag #46587

Are you sure you want to change the base?

[Filebeat] Filestream running as Log input under Elastic Agent or feature flag #46587

Uh oh!

Conversation

belimawr commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Author's Checklist

How to test this PR locally

Manual test

Elastic Agent

Run the tests

Related issues

Uh oh!

github-actions bot commented Sep 12, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Sep 12, 2025

Uh oh!

elasticmachine commented Nov 6, 2025

Uh oh!

Uh oh!

cmacknz commented Nov 6, 2025

Uh oh!

belimawr commented Nov 7, 2025

Uh oh!

Uh oh!

Uh oh!

AndersonQ left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

belimawr commented Nov 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

belimawr commented Sep 12, 2025 •

edited

Loading