The Problem: Zero CVEs on Every RPM Host
After wiring up security-exporter to the vuls-dictionary Helm chart, Debian-based hosts reported CVEs correctly. CentOS and RHEL hosts reported exactly zero. Every time.
No errors. No warnings. The vuls server returned HTTP 200 with a valid JSON response. The
scannedCves map was just empty.
This is the worst kind of bug: everything appears to succeed, but the result is silently wrong.
Diagnosing the Silent Drop
The security-exporter daemon collects the list of installed packages, serialises them into
a string, and POSTs them to the vuls server’s /vuls endpoint. The server passes that string
through ParsePackages to build its internal package map, then looks up CVEs for each package.
Here is ParsePackages from our pkgscanner package, which mirrors the contract the vuls
server expects:
// ParsePackages splits tab-separated "name\tstatus\tversion" lines into a Packages map.
// Only packages with status starting with "ii" (fully installed) are included.
func ParsePackages(raw string) Packages {
pkgs := make(Packages)
for _, line := range strings.Split(raw, "\n") {
line = strings.TrimSpace(line)
if line == "" {
continue
}
const expectedFields = 3
parts := strings.SplitN(line, "\t", expectedFields)
if len(parts) < expectedFields {
continue // <-- silently skipped, no log, no error
}
name := parts[0]
status := parts[1]
version := parts[2]
if name == "" || !strings.HasPrefix(status, "ii") {
continue // <-- also silently skipped
}
pkgs[name] = Package{Name: name, Version: version}
}
return pkgs
}
The contract is strict: three tab-separated fields, second field must start with "ii".
The original RPM collector was running a custom rpm -qa --queryformat that produced
space-separated output with no status prefix at all. Every single line failed the field-count
check and was silently dropped. The result was an empty Packages map, which caused vuls to
return an empty scannedCves — no error, no log line, just no results.
Fix Part 1: The 6-Field RPM Query Format
The vuls project documents the exact rpm -qa --queryformat string it expects. The collector
was updated to produce exactly that:
func (*rpmCollector) CollectPackages(ctx context.Context) (string, string, error) {
cmd := exec.CommandContext(ctx, "rpm", "-qa", "--queryformat",
"%{NAME} %{EPOCHNUM} %{VERSION} %{RELEASE} %{ARCH} %{SOURCERPM}\n")
out, err := cmd.Output()
if err != nil {
return "", "", err
}
pkgs, srcPkgs, err := parseRpmOutput(string(out))
if err != nil {
return "", "", err
}
return pkgs, srcPkgs, nil
}
parseRpmOutput then reformats each 6-field line into the two formats that ParsePackages
and ParseSrcPackages expect:
const rpmFieldCount = 6 // NAME EPOCHNUM VERSION RELEASE ARCH SOURCERPM
// pkgs line: name\tii\tepoch:version-release
pkgLines = append(pkgLines,
fmt.Sprintf("%s\tii\t%s:%s-%s", name, epoch, version, release))
// srcPkgs line: srcName\tsrcVersion\tbinaryName\tii
if srcName, srcVer, ok := parseSourceRPM(sourceRPM); ok {
srcLines = append(srcLines,
fmt.Sprintf("%s\t%s\t%s\tii", srcName, srcVer, name))
}
The status field is hardcoded to "ii" — RPM has no equivalent of dpkg’s per-package install
status in rpm -qa output; if the package appears, it is installed. The version string is
assembled as epoch:version-release, the standard RPM EVR format (e.g. 0:5.1.8-9.el9).
Fix Part 2: Source Package Extraction with the NVR Algorithm
Sending Packages alone was enough to get some CVE results back, but vuls2 uses source
packages (SrcPackages) for more accurate detection on RPM-based distros. The advisory
databases for RHEL and CentOS track many CVEs against source package names, not binary package
names. Without SrcPackages, vuls silently misses those CVEs.
The SOURCERPM field from rpm -qa looks like:
bash-5.1.8-9.el9.src.rpm
openssl-3.0.7-27.el9.src.rpm
perl-IO-Socket-SSL-2.074-2.el9.src.rpm
gpg-pubkey-(none)
The challenge: package names can contain hyphens. perl-IO-Socket-SSL-2.074-2.el9.src.rpm
has four hyphens before the version segment begins. A naive strings.Split(s, "-") and
taking the first element gives the wrong answer.
The correct approach is the last-two-hyphens algorithm. In NVR (Name-Version-Release)
format, release is always the last hyphen-delimited segment and version is always the
second-to-last. The name is everything before those two:
// parseSourceRPM extracts name and version from a SOURCERPM field like
// "bash-5.1.8-9.el9.src.rpm". Returns false for "(none)" or empty.
func parseSourceRPM(s string) (name, version string, ok bool) {
if s == "" || s == "(none)" {
return "", "", false
}
s = strings.TrimSuffix(s, ".src.rpm")
// NVR: name can contain hyphens, so find the last two hyphens.
releaseIdx := strings.LastIndex(s, "-")
if releaseIdx <= 0 {
return "", "", false
}
nameVersion := s[:releaseIdx]
release := s[releaseIdx+1:]
versionIdx := strings.LastIndex(nameVersion, "-")
if versionIdx <= 0 {
return "", "", false
}
name = nameVersion[:versionIdx]
ver := nameVersion[versionIdx+1:]
return name, ver + "-" + release, true
}
For bash-5.1.8-9.el9.src.rpm: name="bash", version="5.1.8-9.el9".
For perl-IO-Socket-SSL-2.074-2.el9.src.rpm: name="perl-IO-Socket-SSL",
version="2.074-2.el9". Correct, despite the four hyphens in the name.
Fix Part 3: Filtering Source Packages Before Sending
One more edge case. The vuls server validates that every binary name listed under a
SrcPackages entry actually exists in the Packages map. If any binary name is absent,
the server errors out for that source entry.
On RPM systems, SOURCERPM=(none) is already handled by parseSourceRPM returning false.
But there are other situations where a derived binary name does not appear in Packages — for
instance, debuginfo packages or sub-packages not present in the specific environment being
scanned.
A filterSrcPackages pass removes binary names absent from Packages, then drops any
source package entry that ends up with no valid binaries:
func filterSrcPackages(srcPkgs SrcPackages, packages Packages) SrcPackages {
for srcName, sp := range srcPkgs {
var filtered []string
for _, bn := range sp.BinaryNames {
if _, ok := packages[bn]; ok {
filtered = append(filtered, bn)
}
}
if len(filtered) == 0 {
delete(srcPkgs, srcName)
} else {
sp.BinaryNames = filtered
srcPkgs[srcName] = sp
}
}
return srcPkgs
}
The Helm Chart: Removing the Legacy CVE Pipeline
Separately, the vuls-dictionary Helm chart was carrying a multi-component CVE ingestion pipeline inherited from an earlier iteration of the setup:
- go-cve-dictionary fetched NVD and other CVE sources into PostgreSQL on a schedule
- CNPG (CloudNativePG) managed the PostgreSQL cluster
- dict-server served CVE data over a gRPC interface for the vuls server to query at scan time
This was a significant operational burden. PostgreSQL required persistent storage and CNPG cluster configuration. The fetch jobs had to run reliably on schedule; any failure produced stale or missing CVE data, often without any visible alert.
With vuls2, this entire stack is replaced by a single pre-built SQLite database: vuls-nightly-db. The database (~7 GB uncompressed) contains CVE data, advisory data, and detection logic for all supported distributions. It is published as an OCI image and updated nightly by the vuls project.
The chart now works like this:
Init containers (run once at pod start, skip if db >5 GB already on PVC):
1. fetch-vuls2-db -- pulls the OCI image via oras
2. decompress-vuls2-db -- decompresses with zstd into /vuls/vuls.db
Vuls server:
- Starts on port 5515
- Reads /vuls/vuls.db at startup
- No outbound network access needed at scan time
The resulting values.yaml is much simpler:
vuls2:
enabled: true
image:
repository: ghcr.io/vulsio/vuls-nightly-db
tag: "0"
vulsServer:
enabled: true
image:
repository: vuls/vuls
tag: "v0.38.6"
port: 5515
resultsStorage:
size: 15Gi
No PostgreSQL. No CNPG. No dict-server. No fetch jobs to babysit.
Lessons
Silent data drops are the worst class of bug. ParsePackages skips malformed lines
without logging anything. From the caller’s perspective the function succeeded — it returned a
valid, non-nil map. The map was just empty. There was no error to catch, no warning to grep
for. Diagnosing it required reading the vuls source code to understand the exact format
contract, which is not documented outside the code itself.
The actionable rule: when integrating with an upstream tool’s parsing function, verify your output format against the parser’s source, not just documentation. A one-field format mismatch can produce a completely silent failure.
Always use the upstream tool’s recommended query format. The vuls project documents the
rpm -qa --queryformat string it expects. Using a custom format that “looks similar” is not
enough — field count, separator character, and field ordering all matter exactly.
Source packages are load-bearing on RPM-based distros. On Debian, SrcPackages is an
enhancement that improves detection precision. On RHEL and CentOS, many CVEs in the advisory
databases are indexed against source package names only. Omitting SrcPackages means vuls
silently misses a meaningful fraction of its results.
Replacing a complex stateful pipeline with a static artefact is almost always worth it. PostgreSQL + go-cve-dictionary + dict-server was three components, each with its own failure modes and operational requirements. A nightly SQLite OCI image is one component pulled by an init container. The data freshness model is identical (both update on a schedule), but the failure surface is far smaller and operational overhead drops to nearly zero.