What It Was#
RPM Back-Port Publisher was an in-house pipeline built at Hawaiian Airlines to solve a specific, urgent problem: getting a patched version of Squid onto production RHEL 8 and 9 systems after Red Hat’s standard package update cycle couldn’t deliver one in a reasonable timeframe.
Both RHEL 8 and RHEL 9 were current, fully supported operating systems at the time. The issue was specific to Squid: Red Hat’s RHEL 8 and 9 repositories ship Squid 4.x/5.x, and Red Hat’s policy is not to perform major-version bumps within a stable release cycle — meaning their advisories addressed what they could within the 4.x/5.x branch, but couldn’t deliver Squid 6.8, where the actual fix lived.
The immediate trigger was CVE-2024-25111 (published March 2024) — a remotely exploitable Denial of Service vulnerability in Squid’s HTTP Chunked decoder, requiring no authentication, fixed only in Squid 6.8.
This was already a known pain point. CVE-2023-46847 (November 2023) — a heap buffer overflow allowing a remote attacker to write up to 2MB of arbitrary data via HTTP Digest Authentication — had put Squid on the radar months earlier. Red Hat issued advisories but the exposure on our systems was never fully closed through official channels. By the time CVE-2024-25111 landed in March 2024, we’d been waiting on a viable path to Squid 6.8 for over four months. That was the point at which building our own solution became the obvious path forward.
The pipeline took Fedora Rawhide RPM spec definitions and rebuilt them for RHEL 8 and 9 using Fedora’s own build toolchain, inside a custom Kubernetes pipeline framework. The resulting packages were signed with GPG keys retrieved from HashiCorp Vault and published to a private RPM repository on JFrog Artifactory. In-house RHEL instances picked them up at the next scheduled patching window.
What It Did#
- Spec back-porting — adapted Fedora Rawhide RPM specs to build cleanly against the target RHEL distribution using Fedora’s native build tools
- Kubernetes pipeline — the build, sign, and publish steps ran as Kubernetes jobs; schedulable, reproducible, auditable
- GPG signing — keys retrieved from HashiCorp Vault at build time; published packages were cryptographically verifiable by consuming systems
- Artifactory publishing — signed RPMs landed in a private repository scoped by OS version; RHEL instances consumed them through standard yum/dnf at the next patching window
- Patch lag reduction — delivered fixes on the team’s timeline rather than waiting on Red Hat’s back-port schedule
Why It Was Built#
Red Hat’s stable release policy means they don’t ship major package version bumps mid-lifecycle — a reasonable trade-off for enterprise stability, but one that left us without a path to Squid 6.8 on current, supported RHEL 8 and 9 systems. For an airline’s infrastructure on a controlled patching schedule, “wait for upstream” wasn’t an answer when production systems had been exposed for months. This pipeline gave the team full control over the build and distribution process while keeping signed-package integrity intact through Vault-managed GPG keys.
Outcome: Unified RHEL 8/9 RPM pipeline delivering signed packages and reducing patch lag by weeks.
Tech: Kubernetes, Python, Bash, JFrog Artifactory, HashiCorp Vault
Context: Hawaiian Airlines (2024–2025)
