Signal handling in Ruby and its internals

Mixing posix threads and signal handling usually is a bit of a nightmare.

Ceri Storey, 2013

I have been debugging signal handlers in Ruby and at some point I started to ask questions that no one could answer. The only way to find answers for them was to read the MRI internals. Just in case, I’ve decided to document my observations in a blog post.

I’m assuming that you already have a context about signal handling in Linux and the Ruby API for it.

In what context is the signal handler executed?

Ruby executes the signal handler in the same thread as the parent. It can be proven by

puts "parent: #{Thread.current.object_id}"
trap("TERM") { puts Thread.current.object_id }
sleep

The thread struct has interrupt_flag and interrupt_mask fields (dunno why they made it two fields).

When the signal is trapped, the current (main) thread is marked with TRAP_INTERRUPT_MASK ([1], [2]). The current executing thread is put on hold and the VM runs the signal handler.

What is safe to do from a signal handler?

I found only one place that explicitly forbids from being called inside a signal handler. This place is Mutex#lock. It prevents user from locking a mutex from the signal handler by the design. This is not a huge limitation, but it prevents you from using Logger which relies on using a mutex. However, puts still works.

Then how do you log from the signal handler?

I’ve questioned myself: why can’t you use Logger inside signal trap when Resque is doing it without any troubles? The answer is that Resque is using mono_logger, which is a mutex-free logger implementation. It works just well from the signal trap!

At Shopify we are logging to Kafka which doesn’t rely on a mutex, meaning that we are also free to log from a signal handler.


If you’re curious, here are the spots in MRI sources that define signal trap behaviour:

Further reading:

Comments

comments powered by Disqus