Being an account of the curious case of a persistent HTTPS connection which appeared ready for use after it had already received its final message, as recorded by Dr. Claude Watson.
Chapter I: The Failure After Success
The telegram arrived at Baker Street bearing a familiar exception:
Faraday::ConnectionFailed: EOFError
protocol-http1/.../connection.rb:in `read_response_line`
async-http/.../protocol/http1/client.rb:in `call`“A remote service closes a persistent connection,” I observed. “The client reuses it, waits for the next response, and discovers the end of the stream. Surely we need only retry?”
Holmes looked up sharply. “What kind of request, Watson?”
“A POST.”
That single word changed the case. If a connection fails while receiving a response, the client may not know whether the server processed the request. Retrying could repeat a payment, create a second record, or perform some other side effect twice.
The report, Persistent connection issues for non-idempotent requests, described around a thousand failures each day. The affected application could safely retry its particular operations, but a general-purpose HTTP client could not assume the same.
“Then our task,” Holmes said, “is not to invent confidence after an ambiguous failure. It is to avoid giving a new request to a connection we can already see is dead.”
Chapter II: The Persistent Suspect
An HTTP connection is expensive to establish, especially when TCP setup and a TLS handshake are involved. A connection pool therefore keeps completed HTTP/1 connections and lends them to later requests.
Before reusing one, Async::HTTP asked a seemingly straightforward question:
def viable?
self.idle? && @stream&.readable?
endThe connection had to be idle, its stream had to exist, and that stream had to appear capable of future reads. Yet the failure remained. A connection passed this check, received a new request, and then produced EOFError while the client waited for the response line.
“Our suspect presents valid papers,” I said.
“At one layer,” Holmes replied. “We have not yet asked who issued them.”
Chapter III: The False Trail of Retrying
The investigation had an earlier clue. In Overriding request.idempotent? for individual requests, the reporter had explored enabling Async::HTTP’s automatic retry path.
“If this application knows its POST is safe to repeat,” I said, “could it simply mark the request idempotent?”
“That may express a policy for this request,” Holmes replied. “Does the label itself make an operation safe to repeat?”
It did not. A general-purpose client could use the declaration when choosing whether to retry, but it could not infer safety merely from the HTTP method—or manufacture safety because a flag had been set.
“Then the client retries only when the caller explicitly permits it.”
“And what will it send?”
I looked again at the request body. It might already have been consumed by the first attempt. Retrying without rewinding could send an empty or incomplete body.
Async::HTTP later gained the ability to rewind suitable bodies, but that did not make every POST safe to replay. Retry policy could help after some failures, but it could not answer the question before us: why had the pool selected a connection whose peer had already shut it down?
Holmes drew a line through the word “retry” in our list of causes.
“A useful remedy in the right case,” he said, “but not the identity of our culprit.”
Chapter IV: The Socket That Told the Truth
We examined the connection from the transport upward. A TCP socket supplied encrypted records to TLS. When those records decoded into application data, HTTP/1 interpreted the resulting bytes as a response.
“The pool asked whether the stream was readable,” I said. “Surely that means data is waiting.”
“Data of what kind?” Holmes asked.
“HTTP response bytes.”
“You have crossed two protocol layers without examining either. What does the operating system actually promise when it marks a socket readable?”
I had mistaken a useful shorthand for a stronger guarantee. Descriptor readiness means that a read can make progress without blocking. It does not promise that the read will return application data; observing EOF is also progress.
Holmes arranged the layers on the board:
Ruby’s IO#wait_readable correctly reports this descriptor-level readiness. The discussion around Ruby feature #20215, IO#readable?, had already exposed the subtle difference between “data is available now,” “a read can observe EOF,” and “a future read might succeed.”
“Then the socket was not lying,” I said. “A read really could make progress.”
“Precisely. But we have not yet established what that progress means to TLS, much less HTTP.”
Chapter V: The Sealed TLS Message
“If a readable socket may reveal EOF,” I said, “can the pool simply peek at the TCP connection and reject it when the peer has closed?”
“Not when TLS stands between TCP and HTTP. A graceful TLS shutdown begins before the TCP connection necessarily reaches EOF.”
The peer first sends an encrypted alert record called close_notify. Its arrival makes the underlying socket readable just as an encrypted application-data record would.
But after OpenSSL reads and decrypts that record, it yields no HTTP bytes. It marks the TLS stream as cleanly closed.
“Then we should inspect the pending TCP bytes,” I suggested, “and look for the alert.”
“What does encryption permit us to learn from those bytes without asking TLS to process them?”
The raw socket could reveal that a TLS record was waiting, but not whether it contained HTTP application data or a shutdown alert. That distinction emerged only after OpenSSL authenticated and decoded the record.
From TCP’s point of view, the socket was readable. From HTTP’s point of view, the connection might already be finished. A viability check which stopped at the socket could not tell which.
“The connection is readable,” I concluded, “and dead.”
Holmes smiled. “At one layer, readable. At the next, closed. Now the title of our case begins to make sense.”
Chapter VI: A Peek Through the Layers
“Then let TLS read the pending record before the pool reuses the connection,” I said.
“And if the connection is healthy but idle?”
“The read would wait indefinitely for data which has no reason to arrive.”
“So it must be non-blocking. And if TLS decodes application data rather than close_notify?”
I saw the second constraint. The probe must not steal bytes from the HTTP reader which would eventually consume them.
We needed a layered peek: allow TLS to process at most one pending record without blocking, while retaining any decoded application bytes in the stream’s buffer.
The solution became IO::Stream::Readable#peek_partial, a layered, non-blocking peek:
def peek_partial(size = @minimum_read_size)
if @read_buffer.empty?
return nil if @finished
result = sysread_nonblock(size, @read_buffer)
case result
when :wait_readable, :wait_writable
return nil
when nil
@finished = true
return nil
end
end
@read_buffer.byteslice(0, size)
end“What does nil tell the caller?” I asked after studying the implementation.
“Only that the probe produced no application bytes. The stream itself records whether that was because the read would block or because TLS reached EOF.”
The method makes at most one non-blocking read attempt through the layered stream. If no record can be processed without waiting, it returns nil and leaves the stream open. If TLS decodes close_notify, the stream records EOF and also returns nil. If it decodes application data, it returns a view of those buffered bytes.
“But OpenSSL has consumed the encrypted record,” I said.
“Yes. The important boundary is the application-facing stream. peek_partial may consume encrypted bytes from the socket, but the decoded bytes remain in IO::Stream’s read buffer for the real HTTP consumer.”
The tests covered four distinct states: an idle open TLS connection, a clean TLS shutdown, an abrupt disconnect, and pending application data that had to survive the probe.
Chapter VII: The Protocol-Specific Verdict
“Now the pool can call peek_partial and keep any connection which returns no bytes,” I said.
“Not quite. What are the two reasons it may return no bytes?”
“The read would block, or the layered stream reached EOF. We must ask the stream whether it remains readable after the probe.”
With that distinction in place, Async::HTTP could probe idle HTTP/1 connections before reuse:
def viable?
return false unless self.idle?
return false unless @stream
# Process at most one pending read through the layered stream.
return false if @stream.peek_partial(1)
return @stream.readable?
rescue => error
Console.debug(self, "Connection viability probe failed!", exception: error)
return false
endHolmes read the outcomes as a sequence of deductions:
- If the read would block, no shutdown is presently observable and the connection remains a candidate for reuse.
- If the stream reaches EOF or decodes
close_notify, the connection is discarded. - If the probe raises because the transport was reset or otherwise failed, the connection is discarded.
- If application data appears while the HTTP/1 connection is meant to be idle, its state is unexpected and the connection is discarded; the peeked byte remains buffered.
“Then every pooled protocol should use this probe,” I said.
“Who owns transport reads on an idle HTTP/1 connection?”
“No one. Its previous response is complete.”
“And on HTTP/2?”
HTTP/2 maintains a background reader which owns the transport and dispatches frames for many concurrent streams. A pool-side probe would compete with that reader and could consume a frame it was responsible for processing. HTTP/1 made the new check safe precisely because an idle connection had no concurrent response reader. HTTP/2 therefore retained its existing viability logic.
“The primitive is layered,” I said, “but its safe use still depends on the protocol’s ownership model.”
“Exactly. A correct operation in one protocol state may be a race in another.”
The regression test recreated the complete sequence: make one request, return its TLS connection to the pool, close it from the server, then issue a non-idempotent POST. The stale connection was rejected and the request used a fresh one.
Epilogue: Readiness Is Relative
Async::HTTP v0.98.1 carried the fix. The reporter’s confirmation was pleasingly concise: “it does! w00t.”
“Have we proved the connection alive?” I asked. “Could the peer not close it immediately after the probe?”
“It could close immediately after any viability check. A network permits no permanent conclusion about the next moment.”
The fix therefore makes a narrower and more useful guarantee: when shutdown is already observable through the layered stream, the pool no longer assigns that connection to a new HTTP/1 request. It does not abolish the inherent close-after-check race.
The deeper lesson extends beyond TLS. Readiness belongs to a layer. A file descriptor can be ready while the protocol above it has no application data to offer. Whenever software wraps one stream in another—encryption, compression, framing, buffering—the question is not merely “can I read?” but “which layer can tell me what this read means?”
Holmes closed the casebook.
“The socket told us it had something to read, Watson. Only TLS could tell us what it meant.”
End of Account
Dr. Claude Watson
221B Baker Street
August 2026