How to debug a GenServer like Sherlock using IEx

A Sherlock Holmes approved guide πŸ•΅

Elixir, the path to functional programming βš—οΈ

With my Rust rediscovery a lot of functional programming concepts became less obscure. In order to go deeper into this paradigm I chose Elixir a powerful general purpose, functional programming language, compiled and executed inside the Erlang virtual machine (BEAM).

But why Elixir, if Erlang exists ? Because :

Elixir, is Erlang with underpants - Athoune

Unlike Erlang, the Elixir syntax is clear and elegant. The language comes with everything you need, included : interactive shell, deps tooling, building tooling and more…

Even if JosΓ© Valim, the Elixir creator was a core Ruby dev, Elixir is completely different since this is a functional programming language. No mutation, no inheritance, no classes, only pure functions. Pray your only true god, the pipe operator |> operator, used to compose functions.

Now, I know, the only thing you want is code and examples, so let’s take a quick look at some basic Elixir stuff using the interactive shell IEx.

A simple addition :

iex(1)> 2 + 2

A simple addition, inside a list

iex(1)> 2 + 2
iex(5)> list = [1,2,3,4][1, 2, 3, 4]
iex(6)>, fn e -> e + 1 end)
[2, 3, 4, 5]

List splitting (head and tail)

iex(7)> [head | tail] = list
[1, 2, 3, 4]
iex(8)> head
iex(9)> tail
[2, 3, 4]

Another important aspect of Elixir is pattern matching used to match values, data structures and much more, let’s try it :

iex(11)> x = {:this, :is, :a, :test}
{:this, :is, :a, :test}
iex(12)> {a, b, c, d} = x
{:this, :is, :a, :test}
iex(13)> a
iex(14)> b
iex(15)> c
iex(16)> d

Using pattern matching you can assign values but also destructure data to simplify interaction with it.

Feels the hype growing ? Nice !

As usual, I like a real project to experiment on something. In the Elixir case, I started a Discord bot project called o2m. The main idea of this bot is to send alert messages on a specific channel when a new episode from a selected podcast is available. Today, I was struggling with a bug on a production instance of o2m. The last episode was not fetched correctly and the action that write a “There is a new episode” message was not triggered correctly. To debug and inspect the current state of the application I used IEx in production.

Base ingredient of a good Elixir : Mix πŸ§™

Combined with IEx, there is Mix. Shipped with Elixir, this a build tool used for the following application related tasks :

  • creating
  • setting up needed deps
  • testing
  • compiling

The combo killer here is to start IEx inside the Mix project, in order to have all the dependencies imported inside the interactive shell

iex -S mix

After that, there is a lot of useful commands like recompile to refresh and recompile new code while your code is running. Yes, this is something that Elixir do by default, hot code reloading.

When your code is ready, you want to deploy it. With Mix the standard way is to used the release command

MIX_ENV=production mix release

This will create a precompile and packaged unit, with runtime included. With this, you do not have to install Erlang or Elixir on your production server. Boom.

If you look carefully, there is an interesting message and the end of the command output :

Release created at \_build/prod/rel/o2m!

    # To start your system
    _build/prod/rel/o2m/bin/o2m start

Once the release is running:

    # To connect to it remotely
    _build/prod/rel/o2m/bin/o2m remote

    # To stop it gracefully (you may also send SIGINT/SIGTERM)
    _build/prod/rel/o2m/bin/o2m stop

To list all commands:



To connect to it remotely

This means I can have access to an interactive shell on production while my code is running, hooray πŸŽ†

Sherlocking a GenServer πŸ”Ž

Did you read the title ? We are talking about GenServer here ! So what the fuck is this ?

Taken from the documentation :

A GenServer is a process like any other Elixir process and it can be used
to keep state, execute code asynchronously and so on. The advantage of using
a generic server process (GenServer) implemented using this module is that it
will have a standard set of interface functions and include functionality for
tracing and error reporting. It will also fit into a supervision tree.

In Elixir, this is the default and common module used to implement client-server behaviors.

Here is the example from the documentation

defmodule Stack do
use GenServer

    # Callbacks

    @impl true
    def init(stack) do
    {:ok, stack}

    @impl true
    def handle_call(:pop, \_from, [head | tail]) do
    {:reply, head, tail}

    @impl true
    def handle_cast({:push, element}, state) do
    {:noreply, [element | state]}


Basically, this is a data structure with a state responding to triggers using handlers that update or retrieve the current state

To launch and interact with the server, in IEx :

iex(41)> {:ok, pid} = GenServer.start_link(Stack, [:hello])

iex(42)>, :pop)

iex(43)> GenServer.cast(pid, {:push, :world})

iex(44)>, :pop)

What if, I want to access the current state, of the GenServer ?

In Erlang, there is the sys module with the get_state/2 function available

But, this is Erlang, and we use Elixir ! We’re doomed ? No ! Because everything available in Erlang is available inside Elixir using the : operator.

iex(18)> :io.format("Hello World~n")
Hello World

So, with our sys example :

iex(19)> :sys.get_state(pid)
[:!, :world]

Nice ! Now we can get state of a GenServer using the PID !

Now, let’s move from local machine to prod. In this case, processus are handled by a Supervisor but first things first, we want a IEx shell inside our running app. In order to illustrate this last part, I will be using one of my prod instances of o2m

o2m@16557a733b59:/opt/o2m\$ ./prod/rel/o2m/bin/o2m remote
Erlang/OTP 22 [erts-10.5.3][source] [64-bit][smp:2:2] [ds:2:2:10][async-threads:1] [hipe]

Interactive Elixir (1.9.2) - press Ctrl+C to exit (type h() ENTER for help)

Ok but, what is the pid of the GenServer I want to inspect ? I don’t know ! Here the first solution could be outputting pid to stdout but, imagine that our application kills and restart GenServer, on demand or our application is flooding stdout because of some errors, this is not a viable solution.

First, we could list all processus,

iex(o2m@16557a733b59)3> Supervisor.which_children(O2M.Supervisor)
{O2M, #PID<0.2368.0>, :worker, [O2M]},
{"jobs-", #PID<0.2365.0>, :worker, [Jobs]},
{"jobs-", #PID<0.2362.0>, :worker, [Jobs]},
{"jobs-", #PID<0.2358.0>, :worker,

Where O2M.Supervisor is the dedicated Supervisor of my application

Now I can identify the GenServer I want to inspect, let’s take the jobs- one, with associated pid 0.2358.0 (the last element of the list, here), here comes the fun

iex(o2m@16557a733b59)13> {_, pid, _, \_} = Supervisor.which_children(O2M.Supervisor) |> List.last
{"jobs-", #PID<0.2358.0>, :worker,
iex(o2m@16557a733b59)14> pid

Then, I can use the pid value to get GenServer state and inspect it to see if everything is ok :

iex(o2m@16557a733b59)15> :sys.get_state(pid)
date: "Mon, 04 Nov 2019 09:00:00 GMT",
show: "Harry Cover, le podcast des meilleures reprises",
title: "Yes we can work it out !",
url: ""

How ! Impressive ! It was a little bit Sherlock Holmes oriented debugging but that was fun. Keep in mind that this is a really simple operation here, from IEx everything is possible from starting new GenServers to modify state of a specific one and much more.

I said it at the beginning, Elixir is powerful.