357 lines
9.5 KiB
Plaintext
357 lines
9.5 KiB
Plaintext
This is nohang config file.
|
|
Lines starting with #, tabs and spaces are comments.
|
|
Lines starting with @ contain optional parameters.
|
|
All values are case sensitive.
|
|
Be careful: nohang doesn't forbid you to shoot yourself in the foot.
|
|
|
|
The configuration includes the following sections:
|
|
|
|
0. Common zram settings
|
|
1. Memory levels to respond to as an OOM threat
|
|
2. Response on PSI memory metrics
|
|
3. The frequency of checking the level of available memory
|
|
(and CPU usage)
|
|
4. The prevention of killing innocent victims
|
|
5. Impact on the badness of processes via matching their names, cgroups and
|
|
cmdlines with specified regular expressions
|
|
6. Customize corrective actions: the execution of a specific command
|
|
instead of sending the SIGTERM signal
|
|
7. GUI notifications:
|
|
- low memory warnings
|
|
- OOM prevention results
|
|
8. Output verbosity
|
|
9. Misc
|
|
|
|
Just read the description of the parameters and edit the values.
|
|
Please restart the program after editing the config.
|
|
|
|
###############################################################################
|
|
|
|
0. Common zram settings
|
|
|
|
See https://www.kernel.org/doc/Documentation/blockdev/zram.txt
|
|
You maybe need to set `ignore_zram = False` if you has a big zram disksize.
|
|
|
|
ignore_zram = False
|
|
|
|
###############################################################################
|
|
|
|
1. Thresholds below which a signal should be sent to the victim
|
|
|
|
Sets the available memory levels at or below which SIGTERM or SIGKILL
|
|
signals are sent. The signal will be sent if MemAvailable and
|
|
SwapFree (in /proc/meminfo) at the same time will drop below the
|
|
corresponding values. Can be specified in % (percent) and M (MiB).
|
|
Valid values are floating-point numbers from the range [0; 100] %.
|
|
|
|
MemAvailable levels.
|
|
|
|
mem_min_sigterm = 10 %
|
|
mem_min_sigkill = 5 %
|
|
|
|
SwapFree levels.
|
|
|
|
swap_min_sigterm = 15 %
|
|
swap_min_sigkill = 5 %
|
|
|
|
Specifying the total share of zram in memory, if exceeded the
|
|
corresponding signals are sent. As the share of zram in memory
|
|
increases, it may fall responsiveness of the system. 90 % is a
|
|
usual hang level, not recommended to set very high.
|
|
|
|
Can be specified in % and M. Valid values are floating-point
|
|
numbers from the range [0; 90] %.
|
|
|
|
zram_max_sigterm = 50 %
|
|
zram_max_sigkill = 60 %
|
|
|
|
###############################################################################
|
|
|
|
2. Response on PSI memory metrics (it needs Linux 4.20 and up)
|
|
|
|
About PSI:
|
|
https://facebookmicrosites.github.io/psi/
|
|
|
|
Disabled by default (ignore_psi = True).
|
|
|
|
ignore_psi = True
|
|
|
|
Choose a path to PSI file.
|
|
By default it monitors system-wide file: /proc/pressure/memory
|
|
You also can set file to monitor one cgroup slice.
|
|
For example:
|
|
psi_path = /sys/fs/cgroup/unified/user.slice/memory.pressure
|
|
psi_path = /sys/fs/cgroup/unified/system.slice/memory.pressure
|
|
psi_path = /sys/fs/cgroup/unified/system.slice/foo.service/memory.pressure
|
|
|
|
Execute the command
|
|
find /sys/fs/cgroup -name memory.pressure
|
|
to find available memory.pressue files (except /proc/pressure/memory).
|
|
(actual for cgroup2)
|
|
|
|
psi_path = /proc/pressure/memory
|
|
|
|
Valid psi_metrics are:
|
|
some_avg10
|
|
some_avg60
|
|
some_avg300
|
|
full_avg10
|
|
full_avg60
|
|
full_avg300
|
|
|
|
some_avg10 is most sensitive.
|
|
|
|
psi_metrics = some_avg10
|
|
|
|
sigterm_psi_threshold = 60
|
|
sigkill_psi_threshold = 90
|
|
|
|
>= 0, float
|
|
psi_excess_duration = 60
|
|
|
|
psi_post_action_delay = 90
|
|
|
|
|
|
###############################################################################
|
|
|
|
3. The frequency of checking the amount of available memory
|
|
(and CPU usage)
|
|
|
|
Coefficients that affect the intensity of monitoring. Reducing
|
|
the coefficients can reduce CPU usage and increase the periods
|
|
between memory checks.
|
|
|
|
Why three coefficients instead of one? Because the swap fill rate
|
|
is usually lower than the RAM fill rate.
|
|
|
|
It is possible to set a lower intensity of monitoring for swap
|
|
without compromising to prevent OOM and thus reduce the CPU load.
|
|
|
|
Default values are well for desktop. On servers without rapid
|
|
fluctuations in memory levels the values can be reduced.
|
|
|
|
Valid values are positive floating-point numbers.
|
|
|
|
rate_mem = 4000
|
|
rate_swap = 1500
|
|
rate_zram = 6000
|
|
|
|
See also https://github.com/rfjakob/earlyoom/issues/61
|
|
|
|
max_sleep = 3
|
|
min_sleep = 0.1
|
|
|
|
Sleep time if soft threshold exceeded.
|
|
|
|
over_sleep = 0.05
|
|
|
|
###############################################################################
|
|
|
|
4. The prevention of killing innocent victims
|
|
|
|
Valid values are integers from the range [0; 1000].
|
|
|
|
min_badness = 30
|
|
|
|
Valid values are non-negative floating-point numbers.
|
|
Min delay if a victim doesn't respond to SIGTERM in 10 ms.
|
|
|
|
min_delay_after_sigterm = 3
|
|
|
|
Valid values are True and False.
|
|
|
|
decrease_oom_score_adj = True
|
|
|
|
Valid values are integers from the range [0; 1000].
|
|
|
|
oom_score_adj_max = 0
|
|
|
|
###############################################################################
|
|
|
|
5. Impact on the badness of processes via matching their names,
|
|
cmdlines or UIDs with regular expressions using re.search().
|
|
|
|
See https://en.wikipedia.org/wiki/Regular_expression and
|
|
https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions
|
|
|
|
Enabling this options slows down the search for the victim
|
|
because the names, cmdlines or UIDs of all processes
|
|
(except init and kthreads) are compared with the
|
|
specified regex patterns (in fact slowing down is caused by
|
|
reading all /proc/*/cmdline and /proc/*/status files).
|
|
|
|
Use script `oom-sort` from nohang package to view
|
|
names, cmdlines and UIDs of processes.
|
|
|
|
|
|
5.1 Matching process names with RE patterns
|
|
|
|
Syntax:
|
|
|
|
@BADNESS_ADJ_RE_NAME badness_adj /// RE_pattern
|
|
|
|
New badness value will be += badness_adj
|
|
|
|
It is possible to compare multiple patterns
|
|
with different badness_adj values.
|
|
|
|
Example:
|
|
@BADNESS_ADJ_RE_NAME -500 /// ^sshd$
|
|
|
|
|
|
5.2 Matching CGroup-line (v1 and v2) with RE patterns
|
|
|
|
@BADNESS_ADJ_RE_CGROUP_V1 -50 /// ^/system.slice
|
|
|
|
@BADNESS_ADJ_RE_CGROUP_V1 50 /// foo.service
|
|
|
|
@BADNESS_ADJ_RE_CGROUP_V1 -50 /// ^/user.slice
|
|
|
|
@BADNESS_ADJ_RE_CGROUP_V2 100 /// ^/workload
|
|
|
|
|
|
5.3 Matching eUIDs with RE patterns
|
|
|
|
@BADNESS_ADJ_RE_UID -100 /// ^0$
|
|
|
|
|
|
5.4 Matching realpath with RE patterns
|
|
|
|
@BADNESS_ADJ_RE_REALPATH 20 /// ^/usr/bin/foo
|
|
|
|
|
|
5.5 Matching cmdlines with RE patterns
|
|
|
|
A good option that allows fine adjustment.
|
|
|
|
Prefer chromium tabs and electron-based apps
|
|
@BADNESS_ADJ_RE_CMDLINE 200 /// --type=renderer
|
|
|
|
Prefer firefox tabs (Web Content and WebExtensions)
|
|
@BADNESS_ADJ_RE_CMDLINE 100 /// -appomni
|
|
|
|
@BADNESS_ADJ_RE_CMDLINE -200 /// ^/usr/lib/virtualbox
|
|
|
|
|
|
5.6 Matching environ with RE patterns
|
|
|
|
@BADNESS_ADJ_RE_ENVIRON 100 /// USER=user
|
|
|
|
Note that you can control badness also via systemd units via
|
|
OOMScoreAdjust, see
|
|
www.freedesktop.org/software/systemd/man/systemd.exec.html#OOMScoreAdjust=
|
|
|
|
###############################################################################
|
|
|
|
6. Customize corrective actions.
|
|
|
|
TODO: docs
|
|
|
|
Syntax:
|
|
KEY REGEXP SEPARATOR COMMAND
|
|
|
|
@SOFT_ACTION_RE_NAME ^foo$ /// kill -SEGV $PID
|
|
@SOFT_ACTION_RE_NAME ^bash$ /// kill -9 $PID
|
|
|
|
@SOFT_ACTION_RE_CGROUP_V1 ^/system.slice/ /// systemctl restart $SERVICE
|
|
@SOFT_ACTION_RE_CGROUP_V1 foo.service$ /// systemctl restart $SERVICE
|
|
|
|
$PID will be replaced by process PID.
|
|
$NAME will be replaced by process name.
|
|
$SERVICE will be replaced by .service if it exists (overwise it will be
|
|
relpaced by empty line)
|
|
|
|
###############################################################################
|
|
|
|
7. GUI notifications:
|
|
- OOM prevention results and
|
|
- low memory warnings
|
|
|
|
gui_notifications = True
|
|
|
|
Enable GUI notifications about the low level of available memory.
|
|
Valid values are True and False.
|
|
|
|
gui_low_memory_warnings = True
|
|
|
|
Execute the command instead of sending GUI notifications if the value is
|
|
not empty line. For example:
|
|
warning_exe = cat /proc/meminfo &
|
|
|
|
warning_exe =
|
|
|
|
Can be specified in % (percent) and M (MiB).
|
|
Valid values are floating-point numbers from the range [0; 100] %.
|
|
|
|
mem_min_warnings = 25 %
|
|
|
|
swap_min_warnings = 35 %
|
|
|
|
zram_max_warnings = 40 %
|
|
|
|
psi_avg_warnings = 100
|
|
|
|
Valid values are floating-point numbers from the range [1; 300].
|
|
|
|
min_time_between_warnings = 20
|
|
|
|
Ampersands (&) will be replaced with asterisks (*) in process
|
|
names and in commands.
|
|
|
|
###############################################################################
|
|
|
|
8. Verbosity
|
|
|
|
Display the configuration when the program starts.
|
|
Valid values are True and False.
|
|
|
|
print_config = False
|
|
|
|
Print memory check results.
|
|
Valid values are True and False.
|
|
|
|
print_mem_check_results = True
|
|
|
|
min_mem_report_interval = 300
|
|
|
|
Print sleep periods between memory checks.
|
|
Valid values are True and False.
|
|
|
|
print_sleep_periods = False
|
|
|
|
print_total_stat = True
|
|
|
|
print_proc_table = True
|
|
|
|
Valid values:
|
|
None
|
|
cgroup_v1
|
|
cgroup_v2
|
|
cmdline
|
|
environ
|
|
realpath
|
|
All
|
|
|
|
extra_table_info = cgroup_v1
|
|
|
|
print_victim_info = True
|
|
|
|
print_victim_cmdline = True
|
|
|
|
max_ancestry_depth = 5
|
|
|
|
separate_log = True
|
|
|
|
psi_debug = False
|
|
|
|
###############################################################################
|
|
|
|
9. Misc
|
|
|
|
max_post_sigterm_victim_lifetime = 10
|
|
|
|
post_kill_exe =
|
|
|
|
forbid_negative_badness = True
|
|
|