Mythos Preview's 89% Severity Match with Human Experts Drives New Calibration Pipeline for LLM Vulnerability Scanners
Mythos Preview, an automated vulnerability assessment tool, has demonstrated a significant 89% exact agreement rate with expert human triagers on severity classification, a key metric that is now driving the development of a formal calibration pipeline. This system aims to close the feedback loop for AI-powered security scanners, moving beyond static confidence scores to measurable, continuous improvement in accuracy. The core motivation is to institutionalize the data that gives teams confidence to deploy automated triage at scale, addressing a known gap in tools like Aura's `LLMVulnerabilityAnalyzer` which currently lacks a mechanism to track and enhance its performance over time based on human judgment.
The proposed pipeline centers on two critical components: a human review interface and a suite of calibration metrics. The interface will create a managed queue for verified findings, requiring human reviewers—both internal staff and external contractors—to assign a severity level, note their confidence, mark agreement with the LLM's initial call, and log the time spent. This structured data capture is designed to feed directly into the second component: a rigorous metrics dashboard. This will track not just the headline 89% exact agreement rate, but also the within-one-level agreement rate (reportedly 98% for Mythos Preview) and generate a detailed confusion matrix across all severity levels: Critical, High, Medium, Low, and Info.
The development signals a maturation in the operational deployment of LLMs for critical security workflows. By building a formal system to benchmark AI against human expertise, the project shifts vulnerability scanning from a black-box output to a calibrated, auditable process. The success of this pipeline could set a new standard for precision and accountability in automated security tooling, providing teams with the hard metrics needed to trust—and continually refine—AI-driven decisions in high-stakes environments.