earlyoom - The Early OOM Daemon

The oom-killer generally has a bad reputation among Linux users. This may be part of the reason Linux invokes it only when it has absolutely no other choice. It will swap out the desktop environment, drop the whole page cache and empty every buffer before it will ultimately kill a process. At least that's what I think that it will do. I have yet to be patient enough to wait for it, sitting in front of an unresponsive system.

This made me and other people wonder if the oom-killer could be configured to step in earlier: [reddit r/linux][5], [superuser.com][2], [unix.stackexchange.com][3].

As it turns out, no, it can't. At least using the in-kernel oom-killer. In the user space, however, we can do whatever we want.

What does it do

earlyoom checks the amount of available memory and free swap up to 10 times a second (less often if there is a lot of free memory). By default if both are below 10%, it will kill the largest process (highest oom_score). The percentage value is configurable via command line arguments.

In the free -m output below, the available memory is 2170 MiB and the free swap is 231 MiB.

              total        used        free      shared  buff/cache   available
Mem:           7842        4523         137         841        3182        2170
Swap:          1023         792         231

Why is "available" memory checked as opposed to "free" memory? On a healthy Linux system, "free" memory is supposed to be close to zero, because Linux uses all available physical memory to cache disk access. These caches can be dropped any time the memory is needed for something else.

The "available" memory accounts for that. It sums up all memory that is unused or can be freed immediately.

Note that you need a recent version of free and Linux kernel 3.14+ to see the "available" column. If you have a recent kernel, but an old version of free, you can get the value from grep MemAvailable /proc/meminfo.

When both your available memory and free swap drop below 10% of the total, it will send the SIGTERM signal to the process that uses the most memory in the opinion of the kernel (/proc/*/oom_score). It can optionally (-i option) ignore any positive adjustments set in /proc/*/oom_score_adj to protect innocent victims (see below).

See also

• nohang, a similar project like earlyoom, written in Python and with additional features and configuration options.
• facebooks's pressure stall information (psi) kernel patches and the accompanying oomd userspace helper. The patches are merged in Linux 4.20.

Why not trigger the kernel oom killer?

Earlyoom does not use echo f > /proc/sysrq-trigger because the Chrome people made their browser (and all electron-based apps - vscode, skype, discord etc) always be the first (innocent!) victim by setting oom_score_adj very high. Instead, earlyoom finds out itself by reading through /proc/*/status (actually /proc/*/statm, which contains the same information but is easier to parse programmatically).

Additionally, in recent kernels (tested on 4.0.5), triggering the kernel oom killer manually may not work at all. That is, it may only free some graphics memory (that will be allocated immediately again) and not actually kill any process. Here you can see how this looks like on my machine (Intel integrated graphics).

How much memory does earlyoom use?

About 2 MiB (VmRSS), though only 220 kiB is private memory (RssAnon). The rest is the libc library (RssFile) that is shared with other processes. All memory is locked using mlockall() to make sure earlyoom does not slow down in low memory situations.

Use

Just start the executable you have just compiled:

./earlyoom

It will inform you how much memory and swap you have, what the minimum is, how much memory is available and how much swap is free.

./earlyoom
earlyoom v1.4-6-ga4021ae
mem total: 9823 MiB, swap total: 9823 MiB
sending SIGTERM when mem <= 10 % and swap <= 10 %,
        SIGKILL when mem <=  5 % and swap <=  5 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail:  5091 of  9823 MiB (51 %), swap free: 9823 of 9823 MiB (100 %)
mem avail:  5084 of  9823 MiB (51 %), swap free: 9823 of 9823 MiB (100 %)
mem avail:  5086 of  9823 MiB (51 %), swap free: 9823 of 9823 MiB (100 %)

[...]