Open Source Progress Report

The past two months have been busy. In New Zealand, we have just come out of lockdown, but the situation remains precarious.

Light Weight Per-Thread Fiber Scheduler

The proposal was recently accepted by Matz for experimentation and the CRuby implementation was merged. The interface provides a transparent bridge between native Ruby I/O and an underlying event loop, like that provided by Async. Between now and the release of Ruby 3, we need as much feedback on this interface as possible.

Async::Scheduler

Async provides a scheduler implementation which conforms to the requirements of the proposal. It allows you to use native Ruby I/O in an event driven fashion. You can see some examples in the specs.

Async do
	barrier = Async::Barrier.new
	
	3.times do
		barrier.async do
			# These requests will happen concurrently:
			Net::HTTP.get(URI "https://www.codeotaku.com/index")
		end
	end
	
	# Wait for all the requests to finish:
	barrier.wait
end

The current scheduler interface is not without limitations. We don't support all blocking operations including DNS resolution, thread synchronisation, process waiting, and libraries that use direct system calls for I/O. In addition, we are considering how to make Mutex aware of fibers and the event loop. That being said, this is a great first step and opens the door to the above improvements and more.

If you'd like to learn more about the scheduler proposal and it's implementation, you can checkout Improving Net::HTTP Concurrency, the Ruby Concurrency Progress Report and the Ruby Concurrency Final Report.

Socketry.io

Socketry.io is an initiative to build a sustainable open source business for the Socketry organsation and all the related gems. Initially, we are offering a premium subscription model for the Falcon application server. Looking forward, we plan on adding long-term support releases including exclusive access to backported bug fixes, advance notice of security issues, expanded documentation, and other business-specific requirements. If you have a business that depends on Falcon, now is a great time to get involved and support its future!

The current implementation of the Socketry.io website is closed-source but it's general purpose and can work with any kind of subscription-based product. It is built on top of Stripe, Utopia and Relaxo. If you would be interested to help work on an open source release of the web application, feel free to let me know!

Utopia Project

"How can I solve my problem with this library of code?" is the most important question that documentation needs to answer. But existing documentation tools often fail to provide the structure and organisation required for new users. Utopia Project provides task-centric documentation, cross-referenced with source code, and will be used as a starting point to rethink and improve the development experience.

I've already started rolling this out accross several projects:

Decode

In order to support the rich cross-referencing required by Utopia Project, I created Decode. It can quickly generate an index of Ruby source code using the parser gem, and can resolve symbols in source code comments and embedded in markdown easily.

As a general framework for source code analysis, its fairly abstract and in the future I hope to extend it to index C and C++ code, which is useful for handling native extensions.

Markly

Wrangling with Markdown documents is non-trivial, but the cmark implementation provides a full abstract syntax tree (AST) for reading, processing and generating documents. I made a fork of CommonMarker after wrangling with Kramdown. My primarily motivation was to improve the interface for document manipulations and rewriting. Utopia::Project uses this interface for processing source code comments, as well as rewriting the README, extracting the title of the document for the page title, etc.

# Read the existing markup from disk:
root = Markly.parse(File.read(readme_path))
node = root.first_child

# Skip first heading:
node = node.next if node.type == :heading

# Generates a new badge for the specified URL.
def badge_for(repository_url)
	"[![Development Status](#{repository_url}/workflows/Development/badge.svg)](#{repository_url}/actions?workflow=Development)"
end

# @returns [Boolean] If the node is a badge, a link which contains an image.
def badge?(node)
	return false unless node.type == :link
	return node.all?{|child| child.type == :image}
end

# @returns [Boolean] If the node contains at least one {badge?}.
def badges?(node)
	node.any?{|child| badge?(child)}
end

# Find the top level node which contains the badges:
while !badges?(node)
	node = node.next
end

# Replace the node with the updated badge:
replacement = Markly.parse(badge_for(repository_url))
node = node.replace(replacement.first_child)

# Write the updated markdown to disk:
node.write(readme_path, root.to_markdown)

Turbo Test

Turbo Test (working title) is a new collaboration between myself and Sam Saffron to improve the parallelism of the RSpec tests within Discourse. I've already created a small proof of concept task runner using Async::Container and preliminary results are promising.

We aim to have a working prototype within the next month. If you are a business that would benefit from such a tool, please help to accelerate its development by sponsoring my work.

Sponsorship

If you are a company and you depend on Ruby, I invite you to support my work. I also provide commercial support agreements and contracting for businesses that want to improve their infrastructure using Falcon and Async. Feel free to get in touch.