Many reliability bugs do not occur during the main operation. They appear in the narrow gap between checking some state and acting on it: a signal arrives just before an event loop sleeps, a server closes a connection just before a client reuses it, or the garbage collector runs while Ruby is switching fibers.
I spent much of July working in those gaps. This monthly open source progress update covers 62 substantive pull requests across 20 repositories. The recurring goal was to preserve the information needed to make the right decision: when to deliver an interrupt, whether a request is safe to retry, which data must remain live, and how much work a server can accept.
Closing Gaps Inside CRuby
Three CRuby changes addressed information which could be lost across internal boundaries: a pending signal, the lifetime of a fiber during a context switch, and a line event carried by optimized bytecode. Each problem appeared only under narrow conditions, but could make higher-level behavior unreliable.
When Ctrl-C Arrives at the Wrong Moment
Consider a server shutting down after you press Ctrl-C. It stops accepting work, asks its workers to finish, closes their connections, and then exits. Ruby provides Thread.handle_interrupt so a program can defer an asynchronous exception while it is performing that cleanup.
Ruby Bug #22133 documented that the default SIGINT path bypassed Ruby's pending-interrupt queue and called the immediate interrupt path directly. Thread.handle_interrupt existed, but the exception produced by pressing Ctrl-C did not consistently obey it. The new path places that exception on the pending-interrupt queue, so the program receives it when the protected section permits delivery.
That change exposed a subtler race in native event loops. An event selector is the operating-system primitive which sleeps until a socket, timer, or other event becomes ready. Native extensions release Ruby's Global VM Lock while waiting so other Ruby threads can continue. In the problematic sequence:
- A signal exception becomes pending.
- The selector checks whether it is safe to sleep.
- The selector releases the lock and enters the operating-system wait.
- The pending exception has no new event with which to wake that wait.
The server is trying to shut down, but its event loop can enter a wait with nothing left to wake it. Without external intervention, that wait has no natural endpoint; in a supervised deployment, it instead lasts until the graceful-shutdown deadline expires and the worker receives SIGKILL. The existing no-GVL API could avoid blocking for an ordinary VM interrupt, but it could not detect an exception which was pending and temporarily masked by Thread.handle_interrupt. A new rb_nogvl flag covers that pending-interrupt state. IO::Event first adopted the final C API name, then centralized its native waits around the new flag with a compatibility path for older Rubies. A follow-up captures errno immediately after each selector call, preventing an unrelated thread-local error from being mistaken for the result of a skipped or interrupted wait. I exercised the complete Falcon shutdown case 2,000 times with two workers per run and saw no delayed worker dumps or shutdown failures.
These changes close a narrow shutdown race in which a termination signal arriving just before the event loop slept could cause a worker to miss the graceful-shutdown deadline and be forcibly terminated with SIGKILL.
Keeping Fibers Alive While Ruby Switches Between Them
Ruby Bug #22196 reported a heap-use-after-free in fiber_switch. During a nested sequence of Fiber#resume and Fiber.yield, Ruby could suspend a C function while it still held a raw pointer to the target fiber. If Ruby code discarded its last visible reference and another fiber ran the garbage collector, the target could be reclaimed before the suspended C function resumed.
The stale-pointer read became visible after Bug #21955 corrected stack cleanup for fibers terminated through Fiber#transfer. The fix keeps the Ruby fiber object alive until the context switch has finished using its native pointer.
Preserving TracePoint Events Through Optimization
Ruby Bug #22218 showed that an ordinary line TracePoint could miss a loop condition which really executed. CRuby already disabled the responsible jump-to-jump optimization when line coverage needed the skipped event, but ordinary line tracing did not receive the same protection. Jumps carrying line events are now preserved for both consumers. The interpreter retains an extra jump on the affected path, while YJIT and ZJIT can remove the overhead after compiling the control flow.
Entering Async from Non-Blocking Fibers
The runtime fixes support a simple promise at the library level: ordinary Ruby control flow should continue to work when Async is introduced underneath it. Previously, calling Sync or Async from an Enumerator, a job runner, or another resume-based non-blocking fiber could fail if no scheduler was already installed. The reactor now enters through Fiber.blocking, preserving the current fiber's identity and local storage while giving the scheduler the context it requires.
Improving Asynchronous Process Management
Process handling also became more direct. The io_uring backend already received completed waitid results from the kernel, but older Ruby APIs required it to perform a second reap operation to create a Process::Status. IO::Event now uses the new rb_process_status_for API to construct that result directly when it is available, while retaining the existing fallback for older Rubies.
Forking has a similar hidden boundary: the child receives a copy of process memory, but not a working copy of every thread and fiber represented by that memory. Async::Container previously forked from a short-lived native thread. That gave the child a clean stack, but moving Process.fork to a different native thread can conflict with libraries such as gRPC which maintain thread-affine native state. The new implementation forks from a blocking fiber on the caller's native thread, preserving the clean child stack without changing threads. Because scheduler counters are copied even though their blocked fibers are not, Async now resets inherited blocked-operation accounting in the child.
The surrounding lifecycle became easier to reason about too. Async::Container now installs graceful SIGINT and SIGTERM defaults without replacing handlers installed by an application. Controller#start already avoided starting a second container; it now returns whether it actually started one. Existing restart behavior was captured as a tested blue-green replacement, and a racy policy-stop test was rewritten around one coordinated child failure. The keyed reload code was not connected to the operational signal path and could fail while removing children, so it was removed while retaining the keyed lookup primitives Falcon uses.
Together, these changes let Async work through more kinds of Ruby control flow while worker processes start and stop with less inherited state.
Making HTTP Behavior More Predictable
HTTP software needs to preserve what happened at each stage of a request: whether a body was sent, whether a response began, why a stream closed, and whether active work has finished. Several changes made those states more explicit across retries, protocol errors, server shutdown, and method semantics.
Knowing When a Request Is Safe to Retry
Imagine sending a non-idempotent request over a pooled HTTPS connection. The server has already closed the connection, but the client has not noticed yet. If the client begins writing before it discovers the closure, it cannot safely assume the server did nothing. Retrying blindly could perform the operation twice.
This is why “retry the request” is not one operation. The client must distinguish several cases:
- The request was definitely not processed: rewind its body and send it again, regardless of the method.
- The outcome is ambiguous: retry only when the method is idempotent and the body can be rewound.
- A response has already begun: expose the error to the caller; a transparent retry would hide partial application behavior.
Async::HTTP already had bounded retry handling: refused requests were retried, and ambiguous connection failures were retried only for methods considered idempotent. What it did not do was rewind a body which the first attempt had already consumed. Protocol::HTTP therefore introduced Request#rewind! for failures known not to have been processed and Request#retry! for ambiguous failures. Async::HTTP now delegates to those operations before retrying.
Low-level HTTP/2 errors became easier to diagnose too. A received RST_STREAM previously produced a stream exception without a useful message; Protocol::HTTP2 now maps every known error code to a description. Its GOAWAY decoder also masks the reserved high bit from last_stream_id, as required by the wire format.
Those protocol-specific exceptions were still too narrow for a general HTTP client to act on. Protocol::HTTP::RemoteError now represents a failure at the remote endpoint while deliberately leaving the processing outcome unknown. Async::HTTP maps an HTTP/2 INTERNAL_ERROR received before response headers to that exception, preserves the original stream error as its cause, and applies the same conservative retry rule. Errors received after a response has begun are still exposed to the caller. Related readiness paths now use promises which carry either success or failure, allowing an HTTP/2 stream error before the initial headers to reject the wait directly.
The best ambiguous retry is one we can avoid. The HTTP/1 pool previously considered a connection reusable when it was idle and its stream reported itself readable. That non-consuming readiness check could not pass through a layered TLS stream to distinguish an open connection from an already-buffered EOF or close_notify. IO::Stream introduced peek_partial, which makes at most one non-blocking read while preserving any application byte in its buffer. Async::HTTP now uses it before assigning an idle HTTP/1 connection. If the peer's closure is already visible, the pool discards that connection and starts the request on a fresh one.
The result is fewer failures from stale pooled connections and fewer unsafe retries when a request's outcome is uncertain.
Waiting for HTTP/2 Requests to Finish
The HTTP/2 server already processed requests concurrently, but its enumeration loop could finish without owning and waiting for every active handler. The request queue now uses its explicit close operation instead of a nil sentinel, preserving requests which were already queued while rejecting late producers. A barrier owns active handlers until they finish, or stops the remaining handlers if request enumeration itself fails.
Adding QUERY and Tightening Accept Parsing
RFC 10008 defines the HTTP QUERY method for safe, idempotent queries which carry their input in the request body. It fills the gap between encoding potentially large or sensitive query input in a GET URI and using POST, whose safety and idempotency are not apparent to generic HTTP infrastructure. Protocol::HTTP added QUERY support, the automatically generated request helper received direct coverage and documentation, and QUERY was classified as idempotent even when it carries a body. The existing Accept parser was retained, but made stricter about malformed ranges, wildcards, quoted parameters, and quality values.
Saving Memory with a Perfect-Hash MIME Registry
A MIME registry contains thousands of names, but the data is mostly static: a known extension such as csv maps to a known media type. Loading the complete database as Ruby strings, arrays, hashes, and record objects spends heap memory on information which could remain in the native extension until it is requested.
The starting point was mime-types-mini, an older experiment which combined a native MIME index, compatibility packaging, and placeholder lookup interfaces. The new Protocol::Media::Registry made the native index the center of the design. At build time, it generates registry data from mime-types-data, uses gperf to produce perfect-hash tables for media types and extensions, and compiles those tables into the extension.
A perfect hash works here because the registered keys are known when the extension is built. Every known key maps to one unique table entry, so lookup does not require collision handling or a large Ruby Hash. The static database stays in compact compiled data, and a Ruby Record is created only for the result being requested. This greatly reduces the Ruby heap cost of MIME lookup. Environments which cannot compile the extension retain the same interface through a pure-Ruby fallback.
Not every media-type operation needs that registry. For example, parsing Accept: application/json, text/*;q=0.8 and selecting a compatible representation only requires small values for negotiation. The related Protocol::Media introduced Type for concrete media types, Range for wildcard matching, and Map for selecting exact or compatible registrations. Task-oriented guides cover parsing, matching, lookup, and content negotiation without requiring applications to load the compiled registry when they do not need it.
Applications which need MIME lookup get a compiled perfect-hash registry instead of a large Ruby object graph, while applications doing only content negotiation can use the smaller value types independently.
Introducing Falcon Clusters, Then Making Them Load-Aware
Falcon's existing server mode gives its workers a shared listener and lets the operating system distribute connections between them. That model is simple and efficient, but an external load balancer sees one shared endpoint rather than the capacity of each worker.
Falcon::Service::Cluster adds that complementary model: each worker binds and owns its endpoint inside the worker process, making it independently discoverable. Falcon::Listener records the endpoint's scheme, wire protocols, and concrete socket addresses as an explicit runtime value. Async::HTTP exposed stable HTTP/1 and HTTP/2 protocol names so that value does not need to refer to implementation constants. This became the foundation for endpoint discovery and, later, load-aware balancing.
Owning a listener is only useful to a cluster if clients can find it. Envoy is a proxy and load balancer which can learn about backends dynamically, but Falcon and its supervisor did not yet have a runtime path from a listener bound inside a worker to Envoy's control plane. Supporting work made worker registration carry explicit post-bind state, then added an adapter which converts each worker's listener into an Envoy Endpoint Discovery Service (EDS) endpoint.
A worker may report both an IP address and a Unix-domain socket, so the xDS control plane learned to group those addresses as alternatives for one logical backend and select HTTP/1 or HTTP/2 from the protocols Falcon reports. Shared listeners needed the opposite treatment: when several workers report the same socket, the Envoy monitor now deduplicates the endpoint and keeps it healthy while any reporting worker remains healthy. Its hook was also aligned with Falcon's worker lifecycle so the listener is passed explicitly.
The control-plane library supporting this integration gained documentation for its public API and focused builders for clusters and endpoints. These changes make the discovery path more direct: a worker binds a listener, the supervisor observes it, and Envoy learns where to send traffic.
At that point Envoy could discover the workers, but it still treated them as equally available. The subsequent load-balancing work added the feedback needed to distinguish an idle Ruby process from one handling a slow request, running garbage collection, or competing for CPU.
Process::Metrics already exposed a stateless CPU percentage. A new stateful Processor sampler compares cumulative CPU time across an interval, reports one fully occupied core as 1.0, and uses process start time to avoid carrying a baseline across PID reuse.
The supervisor previously exposed only service-level aggregation. It now provides immutable utilization snapshots keyed by worker. Async::gRPC::xDS then added Open Request Cost Aggregation (ORCA) and a typed weighted-round-robin policy. ORCA gives each backend a standard way to report its load; the Envoy integration uses an out-of-band service to report each worker's CPU utilization and request rate. A complete Docker Compose example connects the pieces and shows both endpoint discovery and load reports recovering when a worker restarts.
Load tells us that a worker is behaving differently, but not always why. New remote diagnostic tasks can capture memory dumps, scheduler and thread state, garbage-collector profiles, and bounded allocation traces from an individual worker. Discovery tells us where a process is; load reporting and diagnostics help explain how it is behaving.
A Falcon cluster can now direct traffic using the observed capacity of individual Ruby processes instead of assuming every healthy worker is equally available.
Tightening Developer Feedback Loops
Optional instrumentation should not become an unexpected runtime requirement. Async and Async::HTTP stopped installing traces and metrics as mandatory dependencies. Falcon and Async::Service::Supervisor now declare the observability providers their own tests exercise, rather than receiving them transitively from application-facing libraries.
A failure which appears only in CI is difficult to investigate if reproducing it also requires changing the test command. Sus now recognizes the debug environment variables used by GitHub Actions, GitLab CI, Buildkite, and Azure Pipelines, so re-running a job with the platform's debug switch also enables verbose test output. The logging beneath that output received its own reliability pass: Console added targeted tests for its compatibility layer, events, output wrappers, formatters, resolver, and terminal handling, bringing the library to complete local line coverage.
Focused regression coverage protects fixes which are easy to disturb indirectly. Async::Pool added a test for cancelling its gardener while it waits on a condition variable, preserving behavior fixed in supported Ruby releases.
Automated style feedback is only useful when it describes a real correction. The blank-line indentation cop in RuboCop::Socketry could calculate negative indentation for same-line nested arrays and hashes; it now keeps semantic nesting separate from visual indentation. It also mistook modifier conditionals such as a trailing if or unless for structural blocks; those modifiers no longer increase indentation depth.
Dependency bounds provide the same kind of early feedback. Decode's RBS generator relies on the TypeParam interface provided by RBS 4, but previously allowed an unconstrained future major version. Rather than discovering that incompatibility during generation, Decode now declares the compatible RBS 4.x series explicitly.
Documentation checks also made operational measurements easier to interpret. Process::Metrics now documents its Linux cgroup v1, cgroup v2, and /proc/meminfo host-memory readers, making the source of each value explicit.
Looking Forward
The most valuable reliability work often makes itself invisible. Application code should not need to understand a narrow signal race, a fiber's lifetime inside the VM, or whether a TLS shutdown is waiting in a socket buffer. Fixing those problems at the layer which owns the missing information keeps the public interface predictable.
The Falcon cluster work makes the same principle visible at a larger scale. A load balancer should not need to guess which processes exist or whether equal workers still have equal capacity. With listener ownership, dynamic discovery, per-worker load reporting, and diagnostics connected, the next step is to exercise the system under sustained production workloads and continue simplifying its operational model.