Essential Crisis Tools for Linux 2024

Today, we’re going to talk about a topic that we all care about (at least I hope so 😅): crisis tools for Linux! Because when things go wrong and servers are struggling, there’s no time to waste. Often, when disaster strikes, you realize you don’t have the right tools installed. And that’s when the real drama begins… SSH is lagging, servers are unresponsive, packages won’t install, the firewall is acting up… A real nightmare!

So, what are these essential tools you need to have on hand when the situation goes south? I’d say the basics like ps, vmstat, top, and the like are a good start to quickly see what’s wrong. Just those can already save your bacon.

But if you want to play in the big leagues, you’ll need to bring out the heavy artillery. I’m talking about tracing tools like perf, bpftrace, opensnoop, execsnoop, and the whole crew. These will let you live-trace what’s happening on your system, hunt down the culprits, and end the carnage. Top-notch stuff!

Here’s how to proceed with a good old apt install…

Step 1: Install the Basic Packages

• procps: provides ps, vmstat, uptime, top for basic stats
• util-linux: dmesg, lsblk, lscpu for system logs and hardware info
• sysstat: iostat, mpstat, pidstat, sar for disk/CPU stats
• iproute2: ip, ss, nstat, tc, recommended network tools
• numactl: numastat for NUMA stats
• Command: sudo apt install procps util-linux sysstat iproute2 numactl

Step 2: Add Network Tools

• tcpdump: network sniffer
• nicstat: network interface stats
• ethtool: interface info
• Command: sudo apt install tcpdump nicstat ethtool

Step 3: Profiling and Tracing Tools

• linux-tools-common and linux-tools-$(uname -r): perf, turbostat
• bpfcc-tools (or bcc): a suite of powerful eBPF tools
• bpftrace: a dynamic eBPF scripting tool
• trace-cmd: command-line tool for ftrace
• Command: sudo apt install linux-tools-common linux-tools-$(uname -r) bpfcc-tools bpftrace trace-cmd

Step 4 (Optional): Hardware-Specific Tools

Depending on your servers, install the analysis tools for:

• Intel GPU: intel-gpu-tools
• NVIDIA GPU: nvidia-smi
• Other potential accelerators
  However, be careful, not everyone can tame these beasts. It requires some experience and good sense. But once you master them, you’ll be ready to tackle the worst issues. Unexpected crashes, hellish I/O, vicious memory leaks… nothing will stand in your way!

In short, believe me, it’s best to have all this stuff installed on your machines from the get-go. That way, no nasty surprises when disaster strikes. Sure, it takes up some disk space, but it’s nothing compared to the time you’ll waste trying to install it all during a crisis.

With that, I’ll leave you to ponder this. And remember, in the wonderful world of IT, it’s better to be safe than sorry!