STORY · · 8 MIN
Updated
Contributing to EvidenceForge
A short write-up on contributing to EvidenceForge, testing security pipelines, and learning from a well-run open-source project.
A few weeks ago, David Bianco and Cisco Talos open-sourced EvidenceForge, a tool for generating realistic synthetic security logs.
I was immediately interested because this is a problem I have seen. Good labelled security data is hard to get. Real logs are difficult to share for obvious reasons. Public datasets are good, but often cleaned up to the point where the useful detail is gone. Building your own usually means creating a little lab, generating activity, running attacks, collecting telemetry, and then trying to label it all afterwards.
That is a lot of work before you get to the thing you actually wanted to test.
What I liked about EvidenceForge was the way it handles the story behind the logs. David describes one of the common problems with synthetic data as “the absence of a coherent causal story”. That rang true to me. A single Sysmon event can look realistic. A Zeek line can look realistic. A proxy log can look realistic. But if they do not agree with each other, they stop being useful pretty quickly.
EvidenceForge starts with the event and works outwards. The different outputs are supposed to describe the same thing from different angles, with the same users, hosts, timestamps, paths, hashes, and network details.
That is the kind of data I want for the tools I have been building: ThreatCaddy, MacCrab, and Forensicate.ai. Safe, labelled, repeatable data that I can use to test detections, parsers, controls, and investigation workflows.
So I left a comment asking whether PRs were welcome, to which David said “Depends on the PR, but sure! I already had some in mind after looking at the repo.”
Opening issues first
I tried to avoid arriving with a large PR out of nowhere, so I opened a few issues first: #283, #284, and #285.
That gave David a chance to say whether the ideas fit the project before I spent too much time on them. All three have since turned into merged features: fake credential spillage, hostile strings in log fields, and the prompt-injection demo, merged on 23 June 2026. A separate validation fix was accepted through an amended upstream commit; the superseded crash fix is described below.
Fake credential leaks
The first was PR #289, which added a spillage event type.
It creates fake credential leaks in realistic places: a key pasted into a shell, a token written by an application, an API key appearing in a request URL. Each one is also written into the ground truth as a labelled positive.
That gives people test data for the controls that are supposed to catch leaked secrets: log scrubbers, DLP checks, pre-ingest masking, SIEM rules, and detection content.
Most of the work was around making sure the fake credentials stayed fake. The generated values carry poison markers and use known non-production shapes, such as AWS documentation-style AKIA…EXAMPLE keys or sk_test_… tokens. Hosts are kept in reserved address ranges. The safety checks run at config load, validation, and generation time.
I also kept the generation deterministic. No model calls, no random drift, no “close enough”. If the same config is run twice, it should produce the same output.
That is one of the things I liked about working on EvidenceForge. It is built for repeatable testing, not just plausible demo data.
Hostile strings in log fields
The second merged feature was PR #296, which added adversarial_payload.
This came from a slightly different concern: logs are evidence, but they are also input. Often attacker-controlled input.
A User-Agent, a URL, a username, a filename, a process argument, an HTTP parameter. All of these can pass through agents, parsers, terminals, SIEMs, dashboards, notebooks, CSV exports, and spreadsheets.
adversarial_payload lets EvidenceForge place awkward payloads into log fields in a controlled way: ANSI escape sequences, CSV formula injection, CRLF log forging, Log4Shell-style strings, SQL injection-shaped values, oversized fields.
It is not trying to be dramatic. It is just putting ugly strings into the kinds of places where ugly strings can cause trouble.
From there you can test the pipeline. Does the parser handle it? Does the terminal render it safely? Does the CSV become dangerous when opened in a spreadsheet? Does the SIEM keep the field intact? Does the generated alert still match the ground truth?
For cleartext HTTP payloads, EvidenceForge can also emit a matching Snort or Suricata alert when an IDS sensor would have seen the payload on the path. That keeps the log view, alert view, and ground truth tied together.
A small validation bug
While working in that area, I noticed a validation bug in the spillage logic. That became PR #323.
The check was meant to stop Linux-only surfaces being used on the wrong operating system. It gated on os == "windows", but the classifier returns windows, linux, or unknown.
That meant a Linux-only surface on macOS or FreeBSD could pass validation, because those hosts were classified as unknown. Later, the emitter would drop the event because that surface was only implemented for Linux.
For a normal generator, that might be a small edge case. For a labelled dataset, it is more annoying. The ground truth says a credential exists, but the generated logs do not contain it.
The fix was small: gate on != "linux" and add regression tests for macOS, FreeBSD, and Windows. David accepted it through a rebased and amended upstream commit, expanded the tests to cover both affected surfaces, and recorded it in the v1.3.3 changelog. PR #323 was closed after that acceptance; it was not directly merged.
I also sent PR #297 for a KeyError: 'read' crash, where a baseline sshd process reading its own config could abort a run. David had already fixed it on dev, so my PR was closed as superseded.
That was fine. The issue was already fixed, and the test coverage was still useful.
Prompt injection through logs
The third merged feature was PR #333, a demo dataset for indirect prompt injection.
The genesis of this idea came from work I have been doing with Forensicate.ai.
LLM SOC copilots read logs. They read User-Agent strings, URLs, DNS queries, failed-logon usernames, process command lines, filenames, alert text, and investigation notes. A lot of that text can be controlled by an attacker.
If the copilot treats that text as instructions rather than evidence, the log field becomes part of the prompt.
I wanted a simple dataset that showed the issue without needing a callback server, an exploit chain, or a model-specific harness.
The approach is an echo-canary. Each injected field contains a unique token and tries to get the copilot to repeat that token in its own verdict. You run the copilot over the dataset and check whether the token appears in the output.
There is also a clean twin of the dataset, sharing the same baseline activity and almost all source-native rows. Differences include the injected fields and directly related transport rows, as David clarified in the merge review.
This does not solve prompt injection. It is just a small way to demonstrate one failure mode and test whether a tool keeps a basic boundary between untrusted log data and trusted instructions.
ArtifactForge
Working on EvidenceForge also gave me an idea for a companion project, which I floated in issue #332: ArtifactForge.
EvidenceForge generates what the sensors saw: logs, alerts, telemetry.
ArtifactForge would generate what a responder might find afterwards: filesystem residue, persistence entries, config changes, osquery output, Velociraptor-style collections, plists, SQLite artifacts, shell history, browser traces, and other parser-ready forensic material.
The useful part would be keeping it tied to the same scenario. If the logs say a user ran a file from a particular path at a particular time, the artifacts should agree. Same host, same user, same path, same hash, same timeline.
I am not thinking about full disk images or memory dumps. More like small, coherent bundles that existing forensic tools can parse.
It is only an idea for now, but I think there is room for something in that space. Detection engineering and incident response are usually tested separately, even though real investigations move between telemetry and artifacts constantly.
What I took from it
I enjoyed working on EvidenceForge because it is clear what the project cares about.
The output needs to be deterministic. The fake data needs to stay fake. The labels need to be true. The different log sources need to describe the same event. Those constraints are also what make the data useful for testing.
I also appreciated how David handled the contributions. He was clear in the issues, small fixes were taken seriously, and the project’s existing shape was protected. Some of his testing on my PRs signalled gaps in my own testing process, which I have since addressed. That is good to see in an open-source security project.
Thanks to David Bianco and Cisco Talos for open-sourcing EvidenceForge and for making it easy to contribute.