Elixir 1.12.0 signal handlers are cool

2021-05-16 - Chris Grieger

I am deploying a few elixir services via Nomad. Nomad is a workload orchestrator (something like Kubernetes) built by HashiCorp. You can use it to deploy software. Nomad passes configuration, like secrets or service addresses, to its managed services via template files.

Each time a service is redeployed or rescheduled to another worker node, the configuration of dependent services has to be updated. When this happens, Nomad sends a configurable operating system signal to the dependent service.

By default, this causes the service to restart and creating a ripple effect, restarting all dependent services. In turn, this affects the templates of second-grade dependent services, causing even more service restarts. Sometimes this can lead to the system entering an unrecoverable state, never recovering from the constant restarts.

We can fix this by overriding the default exit signal behavior and allowing services to handle configuration changes without requiring a restart.

With version 1.12.0, Elixir now provides an API to watch for operating system exit signals via System.trap_signal/3. Trapping exit signals allows services to reload their configuration gracefully, only restarting processes when necessary.

defmodule TrapSignals.Application do
  # See https://hexdocs.pm/elixir/Application.html
  # for more information on OTP Applications
  @moduledoc false

  require Logger

  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Starts a worker by calling: TrapSignals.Worker.start_link(arg)
      # {TrapSignals.Worker, arg}
    ]

    # Trap SIGHUP signals, like nginx does it.
    System.trap_signal(:sighup, fn ->
      Logger.info("SIGHUP received.")
      Elixir.Code.eval_file(".config.exs")

      # Gracefully restart Ecto Repo's, GenServers, etc.
      # ...

      # Done
      Logger.info("Configuration reloaded.")

      :ok
    end)

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: TrapSignals.Supervisor]
    Supervisor.start_link(children, opts)
  end
end