Enhancing Deployment Safety at GitHub with eBPF: Breaking Circular Dependencies

Introduction: The Challenge of Self-Hosting GitHub

GitHub runs its own source code on github.com, making it its own biggest customer. While this approach allows internal testing before user-facing releases, it creates a critical circular dependency: if github.com goes down, teams lose access to the very code needed to fix it. To mitigate this, GitHub maintains mirrored repositories and built assets for rollbacks. However, even with these safeguards, deployment scripts can introduce new circular dependencies—such as pulling binaries from GitHub or calling internal services that themselves depend on GitHub. This article explores how GitHub leveraged eBPF to monitor and block such dependencies, ensuring safer deployments.

Enhancing Deployment Safety at GitHub with eBPF: Breaking Circular Dependencies — Source: github.blog

Types of Circular Dependencies in Deployments

Imagine a MySQL outage that prevents GitHub from serving release data. To resolve it, teams need to apply a configuration change via a deploy script on each affected node. During such an incident, three types of circular dependencies can emerge:

Direct Dependencies

The deploy script attempts to pull the latest release of an open-source tool from GitHub. Because GitHub is down, the script cannot complete, halting the entire deployment.

Hidden Dependencies

The deploy script uses a servicing tool already present on the machine. However, that tool checks GitHub for updates before executing; if it cannot reach GitHub, the script may fail or hang, depending on error handling.

Transient Dependencies

The deploy script calls an internal service (e.g., a migrations service) via API. That service then tries to fetch the latest open-source binary from GitHub, propagating the failure back to the deploy script.

Traditional Approach: Manual Review and Its Limitations

Historically, GitHub relied on individual teams to review deployment scripts and identify circular dependencies. This manual process was error-prone, as many dependencies—especially hidden or transient ones—are not obvious. Teams could not guarantee that every script would avoid creating a dependency on GitHub or internal services. A more automated, robust solution was needed.

How eBPF Provides a Systematic Solution

eBPF (extended Berkeley Packet Filter) is a kernel technology that allows running sandboxed programs in response to system events. GitHub used eBPF to selectively monitor and block certain network and file operations during deployment. By attaching eBPF programs to system calls (e.g., connect, open, read), they could intercept any attempt by a deploy script to access GitHub or other forbidden endpoints. This provided fine-grained control:

Monitor: Log all calls to potentially dangerous destinations (e.g., github.com, internal APIs) without blocking, for auditing.
Block: Prevent scripts from making such calls entirely, enforced at the kernel level.
Allowlist: Define permitted destinations (e.g., local mirrors, internal caching servers) to avoid false positives.

Implementation Details: Writing eBPF Programs for Deployment Safety

GitHub developed eBPF programs that attach to network-related syscalls. When a deploy script initiates a connection, the eBPF program checks the destination IP against a dynamically updated allowlist. If the destination is not allowed, the program returns an error to the caller, effectively blocking the operation. The same approach can be applied to file operations—blocking reads from specific paths that might trigger external dependencies.

Key implementation considerations:

Kernel Version Support: eBPF requires a modern Linux kernel (4.x+). GitHub ensured all production hosts ran compatible kernels.
Performance Overhead: eBPF programs are JIT-compiled and run in a sandbox, adding minimal latency. Benchmarks showed less than 1% overhead on deployment scripts.
Dynamic Updates: The allowlist can be updated in real-time via eBPF maps, avoiding the need to reload programs.
Fallback Mechanism: If eBPF fails to load, deployments fall back to the old manual review process.

Sample eBPF Program (Simplified)

Below is a pseudo-code snippet of an eBPF program that blocks connections to blacklisted IPs:

struct sockaddr_in *addr = (struct sockaddr_in *)ctx->args[1];
u32 dest_ip = addr->sin_addr.s_addr;

if (bpf_map_lookup_elem(&blacklist, &dest_ip)) {
    return -EPERM;
}
return 0;

Benefits and Results

By adopting eBPF, GitHub achieved:

Automatic enforcement: No need for manual script review; eBPF blocks dependencies at the kernel level.
Real-time protection: Even if a script adds a new call to GitHub mid-deployment, it is immediately blocked.
Auditability: All blocked attempts are logged, providing visibility into potential issues.
Reduced incident response time: During outages, teams can deploy fixes without worrying about secondary failures.

Getting Started with eBPF for Your Deployments

You can adapt GitHub’s approach to protect your own systems. Start by:

Identifying critical endpoints (e.g., production services, external APIs) that should not be called during deployments.
Writing eBPF programs to intercept relevant syscalls (connect, sendto, open). Tools like bpftrace and BCC simplify development.
Testing in a staging environment to ensure no legitimate calls are blocked.
Rolling out gradually, with monitoring and fallback mechanisms.

Conclusion

GitHub’s use of eBPF to enhance deployment safety demonstrates how kernel-level observability and control can break circular dependencies. By automating the enforcement of dependency boundaries, teams can reduce human error and improve reliability during incidents. eBPF offers a powerful, low-overhead way to secure deployments—a technique that can be applied across many organizations.

For more details, see the original blog post.

Tags: