GitHub’s model for secret detection: Finding more secrets with less noise

Secret scanning works best when developers are confident in the findings. Catch too little, and real credentials can remain in a repository. Alert too often, and developers may spend time investigating strings that aren’t secrets.

Generic secrets make that balance especially difficult. The same string could be a real credential or a documentation placeholder, so detecting it requires understanding the surrounding code and text.

To address this challenge, we trained a secret classifier specifically for GitHub’s product needs: distinguishing real credentials from lookalikes while meeting the speed and efficiency requirements of scanning at scale.

In our companion post, Secret protection must scale with software, we explained why secret protection must keep pace with AI-accelerated development. Here, we go behind the scenes of the context-aware classifier powering that effort: how we designed, trained, and evaluated it to find more real secrets with less noise.

What we enabled

GitHub secret scanning operates at two key moments:

• Push protection helps prevent secrets from entering a repository by scanning changes before a push is accepted, giving developers an opportunity to address the finding.
• Post-push scanning works after code has been committed, continuously scanning repository content and creating alerts when potential secrets are detected so developers and security teams can investigate and remediate them.

We developed a fast classifier that reduced median post-push scan execution time by 86% compared with the previous LLM-based pipeline. That efficiency made it practical to extend generic secret detection to push protection, returning results before developers push changes, not only after code is committed.

Why secrets are hard to detect

Structured credentials often include a distinctive prefix, length, alphabet, or checksum. Generic secrets may not. Their syntax overlaps with build identifiers, checksums, asset hashes, sample values, and other non-sensitive strings. Context is the deciding signal.

These cases are hard negatives: strings that match a credential pattern but are not usable credentials. Adding more rules can suppress some of them, but each exception risks hiding a real secret. A learned classifier can instead use signals such as file path, secret type, variable names, and nearby code or text.

That is why we shipped a new scanner using a two-stage pipeline:

1. Candidate generation: Regex filters scan content for potential secrets with high recall. This stage is optimized to catch broad families of credential-like patterns quickly and deterministically.
2. Context-aware classification: A fine-tuned ModernBERT classifier evaluates each candidate and its surrounding context. It is trained to recognize the signals that separate real secrets from false positives, including placeholder values, examples, documentation snippets, generated content, and non-sensitive identifiers. Because the model is small and efficient to run on our inference infrastructure, it costs much less to serve and is simpler to scale than an LLM-based pipeline.

Impact:

• 11.5% user acceptance improvement, reducing false positives.
• 86% faster post-push secret detection execution time.
• More than 98% cost savings compared with the LLM-based pipeline.

How our pipeline works

As content is scanned, regex patterns cast a wide net and flag anything that looks like a credential, so real secrets aren’t missed at the start. Each candidate then goes to a fine-tuned ModernBERT classifier, which reads the value together with its file path, variable names, and surrounding code. The classifier estimates how likely it is to be a real secret. High-confidence findings become alerts, and lookalikes such as placeholders, test fixtures, and checksums are filtered out before they reach developers.

The important design choice is that the model does not replace deterministic scanning. Instead, it complements it. Regexes remain the right tool for finding candidate strings quickly. The classifier is used where machine learning provides the most leverage: distinguishing “looks like a secret” from “is likely a real secret.”

A hard-negative problem: secrets that almost look real

Most false positives are not random. They are hard negatives: strings that match a credential pattern but are not usable credentials. Examples include documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs, and generated fixtures. These cases are easy for a regex to match and hard for a purely rule-based system to suppress without accidentally losing real secrets.

ModernBERT helps because it can read the candidate together with nearby code and text providing contextual understanding comparable to an LLM without the latency and computational overhead of generating natural-language output. For example, a candidate surrounded by words like “example,” “dummy,” “fake,” or “replace-me” may be lower risk than a candidate assigned to an environment variable in a deployment file. The classifier learns these contextual differences from labeled examples rather than requiring engineers to encode every exception as another rule.

We chose ModernBERT because it provides a strong fit for this workload: it is compact, efficient, and well suited for classification over code-adjacent text. For secret scanning, the model needs to make a calibrated decision over short candidate windows with high throughput and low latency.

That makes a fine-tuned encoder classifier a better serving shape than a general-purpose LLM for this stage of the pipeline. The model can be trained directly on our decision boundary, and evaluated using the same recall and precision metrics that matter to secret scanning quality.

How we trained

The challenge was finding representative data for cases in which syntax alone is misleading. The training and evaluation strategy combines three sources: 

• Product-derived examples from GitHub repositories, using feedback from the secret detection experience and additional human validation.
• High-recall regex candidates from GitHub repository data, including human-annotated examples and weak labels used to expand coverage.
• Synthetic examples targeted at sparse and difficult cases, such as documentation secrets, test fixtures, public keys, checksums, structured lookalikes, and near-miss pairs.

Synthetic rows pass deterministic validation and model-based quality review before inclusion. Training emphasizes real secrets, realistic false positives, and hard negatives that frequently confuse regex-only approaches. Each model input includes the candidate value, file path, secret type, and a bounded amount of surrounding context. We select thresholds through offline evaluation and then validate behavior on production-like traffic before broader rollout.

Before splitting the data, we grouped rows that share a repository, candidate value, or example so near-duplicates could not appear in both training and evaluation.

Training focused on the cases that matter most for alert quality: real secrets, realistic false positives, and hard negatives that regexes frequently match. The input representation includes the candidate secret string, file path, type and a bounded amount of surrounding context. The output is a classification score used by the scanner to decide whether the candidate should proceed.

We optimized for the product tradeoff: preserve or improve recall while reducing false positives and serving cost. In practice, that means the classifier threshold is selected using offline evaluation, then validated against production-like traffic before rollout.

The most important training principle was to make the model see the confusing examples. We included candidates that are syntactically similar to real secrets but semantically different, so the model learns not only what a secret looks like, but also what a false positive looks like in context.

The trained model is exported to ONNX format with 8-bit quantization, enabling fast inference.

Evaluation results

We evaluated the new pipeline’s impact in production in two stages: first against the production baseline on the same public traffic and then through a customer A/B experiment. 

• Offline evaluation: On post-push scanning, the new pipeline achieved 13% higher precision and 19% higher recall than the previous LLM pipeline when evaluated on the same public traffic. 
• Online evaluation: We then validated whether these quality improvements translated to customer outcomes through an A/B experiment against the previous LLM pipeline. Repository owner-weighted acceptance increased by 11.5%. We weight acceptance by repository owner to reduce the influence of customers who bulk-resolve large volumes of alerts at once, which can otherwise distort a simple average. 

Take this with you

The best security tool is the one developers trust enough to use. Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught. Lower cost means we can keep improving coverage without making scanning prohibitively expensive.

For developers and security teams, this should translate into fewer false positives, and faster feedback in the places where secrets are most likely to be introduced: commits, pull requests, and everyday development workflows.

Hybrid systems are powerful. Regex gives us speed, transparency, and broad candidate coverage. ModernBERT gives us learned context understanding and false-positive reduction. The combination outperforms either approach used alone for this product constraint.

We also learned that model size is not the same as model usefulness. A smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical.

This is one step toward making secret scanning more accurate, scalable, and easy to use. Looking ahead, we are focused on two areas:

• Building a continuous improvement flywheel. We will use production feedback loops to identify new false negatives and false positives, turning real-world signals into higher-quality training and evaluation data.
• Exploring new model architectures. We are investigating approaches that can perform secret detection and extraction in a single step, reducing pipeline complexity while preserving accuracy and scale.

Acknowledgments

Huge thanks to the engineering, science and product teams across GitHub and Microsoft who helped build the training data, evaluation suite, serving stack, and rollout plan for this new secret scanner.

Written by

As a Senior Research Scientist in Microsoft CoreAI, I develop agentic model orchestration and routing techniques for GitHub Copilot. My research focuses on adaptive model selection, multi-model coordination, long-horizon agentic workflows, and developer security.

As a Partner Applied Science Manager in Microsoft CoreAI, I lead an applied research team that measurably boosts developer productivity with ML and AI. We evolve GitHub Copilot in Completion, Chat, Code Review, and agentic workflows by continuously refining models and applying in-context learning for deeper, context-aware assistance.