Embed a gem in a Rails project and enable autoreload

In the previous article, I created a dedicated gem for a JSON parser. In this article, I show how to embed it into a Ruby on Rails project and enable autoreload for the gem's classes in the development environment.

To use a gem in a Rails project, the default way is to publish the gem, either on a public or a private gem server. There's a nice guide on rubygems.org for this. I did not want to publish my gem on a public gem server, and setting up a private gem server that would be available to both my development and production environments seemed like a burden.

I chose instead to embed the gem in my Ruby on Rails project: the source of the gem is inside the Rails project, in the same git repository. It makes sense to me, because the gem is dedicated to this Rails application. And if later I want to share the gem, it can easily be moved to its own repository.

TL;DR

1. The source files of the gem are put into a directory embedded_gems inside the Rails project.
Project structure

2. The gem is referenced in the Gemfile of the Rails project with the path: option.


          gem 'format_gem', path: './embedded_gems/format_gem'

3. For the development environment, autoreload is enabled with a dedicated Zeitwerk::Loader and a file watcher, to reload the gem when a source file is modified, just like for the rest of the Rails app.

Benefits

The main project and the gem are in the same git repository: both can be deployed with a single git pull; Also related modifications of the app and the gem are in the same commit.
In the development environment, the classes of the gem are reloaded on each request without restarting the server, big time saver.
This solution does not require the installation/usage of a gem server.

Step-by-step walkthrough

To illustrate the solution, I'll create a gem FormatGem containing a service. I'll embed the gem in a Rails project called FormatApp, and have a simple controller to display the health status of the service.

This article is split into 5 sections:

Create the Rails project
Create the gem project
Embed the gem in the Rails project
Setup autoreload for the development environment
Summary

Prerequisite

I'm using Rails 6.1.3 in this article, and ruby and gem of course.


          $ rails --version; ruby --version; gem --version
          Rails 6.1.3
          ruby 2.7.1p83
          Gem 3.1.2

Create the Rails project

First I create the Rails project. It's a good opportunity to try the new option --minimal to get a lightweight application skeleton (that's perfect for this example, I just need a controller).



          $ rails new format_app --minimal
           create
           create README.md
           create Rakefile
           ... 
              run  bundle install
          Fetching gem metadata from https://rubygems.org/....
          Resolving dependencies...
           ... 
          Using sass-rails 6.0.0
          Using sqlite3 1.4.2
          Bundle complete! 7 Gemfile dependencies, 54 gems now installed.

Immediately check the Rails project is working properly by starting the server.


          $ cd format_app
          format_app$ rails s
          => Booting Puma
          => Rails 6.1.3 application starting in development
          => Run `bin/rails server --help` for more startup options
          Puma starting in single mode...
          * Listening on http://127.0.0.1:3000

I add a controller/action for the health status. For this, I create app/controllers/status_controller.rb with a health action, and declare the route to this action in config/routes.rb


          class StatusController < ApplicationController
            def health
              messages = []
              messages << 'Rails app is alive'
              messages << 'Service check to be implemented' # Here I'll add the service status
              render plain: messages.join("\n")
            end
          end


          Rails.application.routes.draw do
            get '/health', to: 'status#health'
          end

Really classic Rails stuff. I check http://127.0.0.1:3000/health, it returns the expected messages.

Create the gem project

In the previous article Tokenization with Flex and Ruby, I used the bundle command to create the gem. This command generates a lot of files, and most of them are only useful for gems that will be published and versioned. That's not my use case, so this time I'll create the gem from scratch. That's really simple, there's only one required file, the .gemspec file.

The gem is called format_gem. I create a directory embedded_gems/format_gem in the root directory of the Rails project, and put in it the following format_gem.gemspec file.


          Gem::Specification.new do |spec|
            spec.name          = 'format_gem'
            spec.authors       = ['me']
            spec.summary       = 'Gem containing FormatGem::FormatService'
            spec.version       = '0.1'
            spec.files         = Dir['lib/**/*']
          end

I chose to fill only the required Gemspec attributes. Note that the gem will include all files of the lib directory;

I immediately create the service FormatGem::FormatService. The source file of the class must be put in a format_gem directory, to match with the module name. That's important for the autoreload that will be set up in the last section. So here is lib/format_gem/format_service.rb:


          module FormatGem
            class FormatService
              def self.health
                "FormatService is ready"
              end
            end
          end

I check the gem before going on the next step, with the gem build command.


          format_app/embedded_gems/format_gem$ gem build
          WARNING:  licenses is empty, but is recommended.  Use a license identifier from http://spdx.org/licenses or 'Nonstandard' for a nonstandard license.
          WARNING:  no homepage specified
          WARNING:  See http://guides.rubygems.org/specification-reference/ for help
          Successfully built RubyGem
            Name: format_gem
            Version: 0.1
            File: format_gem-0.1.gem

The gem has been built with success. Ignore the warnings : that's just because I omitted some recommended Gemspec attributes. The gem build command have created a format_gem-0.1.gem file that won't be used. It can be safely deleted.


          format_app/embedded_gems/format_gem$ rm format_gem-0.1.gem

Embed the gem in the Rails project

The gem must be added to the Gemfile of the Rails project. As the gem is not published on any server, I use the path: option to indicate the source directory of the gem.


          # Embedded gems
          gem 'format_gem', path: './embedded_gems/format_gem'

And I update the controller to call the service.


          require 'format_gem/format_service'
           
          class StatusController < ApplicationController
            def health
              messages = []
              messages << 'Rails app is alive'
              messages << FormatGem::FormatService.health
              render plain: messages.join("\n")
            end
          end

I restart the Rails server because the Gemfile as been modified, and check that http://127.0.0.1:3000/health returns the expected messages.

Setup autoreload for the development environment

Thanks to its class loader, Rails automatically reloads classes on each request if the application files have changed in the development environment. Since Rails 6, the default class loader is Zeitwerk. Unfortunately, it works only for files in the app directory. In the current state of my application, the modifications in the gem, like in FormatService, are not applied until the Rails server is restarted.

Let's fix this :

I set up a dedicated Zeitwerk class loader for the gem. I use Zeitwerk::Registry.loader_for_gem(path) so I can specify the root path of the gem.
I instantiate an ActiveSupport::FileUpdateChecker to watch all the files of the gem, and trigger reload when a file is modified
I plug the watcher to check the files on each request with execute_if_updated

As autoreload is useful only the development environment, I implement all this operation in development.rb, in the Rails.application.configure block:


           ... 
           
          # Autoreload for each embedded gem
          Rails.root.join('embedded_gems').children.each do |gem_path|
           
            # Create a Zeitwerk class loader for each gem
            gem_lib_path = gem_path.join('lib').join(gem_path.basename) # i.e. '.../format_gem/lib/format_gem'
            gem_loader = Zeitwerk::Registry.loader_for_gem(gem_lib_path)
            gem_loader.enable_reloading
            gem_loader.setup
           
            # Create a file watcher that will reload the gem classes when a file changes
            file_watcher = ActiveSupport::FileUpdateChecker.new(gem_path.glob('**/*')) do
              gem_loader.reload
            end
           
            # Plug it to Rails to be executed on each request
            Rails.application.reloaders << Class.new do 
              def initialize(file_watcher)
                @file_watcher = file_watcher
              end
           
              def updated?
                @file_watcher.execute_if_updated
              end
            end.new(file_watcher)
          end

I restart the rails server a last time to integrate the changes of development.rb. Now any change in FormatService is immediately available in the Rails application without restarting the server. Note however it doesn't work for gems with C extensions (even when the .so is modified, classes are not reloaded).

Summary

Most of the time, the gems are meant to be shared and reused among projects, and so they're published. But for the gems that are dedicated to only one project, embedding the gems like this is a good alternative, and it simplifies the gems management. Format Express uses gems like this for each formatter, and it works like a charm.
The autoreloading trick in the development environment saves a lot of time, and it can be enabled for any local gem, as long as the gem is included in the Gemfile with the path: option.