How to Collect Metrics from a Remote Host over SSH

What this post is about. A monitoring collector has to read ordinary numbers off someone else’s Linux box — CPU usage, memory, disk, swap — and it is not allowed to install anything on that box. This is how we did it, over SSH, and what it cost.

What you need to know first. SSH is just a remote shell. If you can ssh user@host and run free -m, you have everything this post needs. No protocol internals, no agent, no daemon.

The problem

Three things make SSH collection harder than it sounds.

The customer will not install your software. A daemon on their machine means a version to deploy, a service to secure, and something to blame when it breaks. SSH needs nothing: the customer already has an SSH server, and the credentials are the ones they gave you for support.

Every operating system prints differently. Linux, Solaris, HP-UX, AIX and Windows all have a free-like command, and none of them agree on the layout.

Every user has a different language. A Chinese-locale df prints 文件系统 and 已用, not Filesystem and Used. A parser that hardcodes column positions breaks the first time it meets a machine it was not tested on.

The shape of the solution

For each metric we run one command on the remote host, read its output, and turn it into a number. The parts that need care are the three below: framing the output, finding the right column, and handling each platform’s layout.

Part 1 — how do you know where the command’s output ends?

Here is the problem in its raw form. When you run a command over SSH, what comes back is not just the command’s output. The shell also echoes back everything you typed, and there is no marker saying “the output starts here”.

The fix is to bracket the command with two sentinels built from a unique id, and to pad the command with empty echoes so the shell’s own echo of the input cannot corrupt them:

1
2
3
4
5
builder.append("echo '" + uid + "x';");   // start marker
builder.append("echo '';");                // keep the echo out of the way
builder.append(cmd + ";");                 // the actual command
builder.append("echo '';");
builder.append("echo 'x" + uid + "';");    // end marker

That is the whole trick, in five lines. The remote shell prints:

1
2
3
4
<uid>x
free total used free shared buff/cache available
...
x<uid>

and the reader stops at the closing marker. The two echo '' lines are the part people miss — without them, the shell’s echo of the typed command would land in the middle of the output and the markers would no longer match. The uid also means two collections running against the same host at the same time cannot mix up each other’s output.

Reading the result is a poll, not a blocking read, because a shell that never prints the closing marker would hang the reader forever:

1
2
3
4
protected String readInput(String uid, InputStream stream, long timeout, long waitInterval) {
    Thread.sleep(waitInterval);
    ...
    while (time <= timeout - waitInterval) {

The wait_interval (300 ms by default) is how often we check for new output; the timeout is enforced by the loop condition rather than by the SSH layer.

One more implementation detail: we keep a pool of open shells. Opening a fresh SSH session per metric would be absurd, so a command borrows a shell and returns it when it is done — including when it fails.

Part 2 — how do you find the right column when the language changes?

This is the part I would point at first, because it is the difference between a parser that works and one that works on the machine you developed on.

Instead of hardcoding “column 2 is the filesystem name”, we read the header line of the output, build a map from column name to column number, and then look up each column by name — trying every spelling it might have:

1
2
3
4
5
6
7
8
if string.match(line, "Filesystem") or string.match(line, "文件系统") then
    local cols = split_space(line);
    for index, title in ipairs(cols) do titles[title] = index; end
    has_cols = true;
end
...
    local used_index  = get_value(titles["used"], titles["Used"], titles["已用"]);
    local avail_index = get_value(titles["Available"], titles["avail"], titles["可用"]);

The first part finds the header line — in either language — and remembers which column number each title sits in. The second part looks up a column by name, trying the English spelling and the Chinese one. get_value returns the first spelling it finds, so the same code works on either.

The command is df -Pk, and both flags matter. -P forces the POSIX output format (one line per filesystem, no wrapping) and -k makes the unit 1024-byte blocks, so converting to bytes is a single multiplication instead of guessing whether the numbers are in 512-byte or 1K blocks.

Part 3 — the same metric, four different layouts

vmstat is the classic example of a command whose columns move between platforms. On Linux the row ends with us sy id wa st; on Solaris and HP-UX it ends with us sy id; AIX has yet another layout. So the parser has a branch per platform, and — this is the useful trick — it addresses the columns from the end of the line, because the leading columns differ wildly but the CPU columns are the ones we want:

1
2
3
4
5
6
7
8
-- Linux:   ... us sy id wa st
if os_type == "linux" then
    rs.idleCpuTime   = tonumber(values[#values - 2])
    rs.userCpuTime   = tonumber(values[#values - 4])
    rs.systemCpuTime = tonumber(values[#values - 3])
    rs.ioWaitCpuTime = tonumber(values[#values - 1])
-- Solaris & HP-UX:  ... us sy id
elseif os_type == "solaris" or os_type == "hpux" then

The command is vmstat 1 2. The first sample is the average since boot; the second is one second later. We want the second one, and we find it by locating the header line (the one containing buff on Linux) and taking the line after it. It is not elegant, and it is exactly the kind of thing that breaks on the next platform.

When we meet a platform we do not support, we say so instead of guessing:

1
2
3
4
else
    request:error(INVALID_NOT_SUPPORT, "can't support os[" .. os_type .. "] for task[host_cpu]");
    return;
end

Part 4 — the better pattern, in Perl

Some metrics were not worth parsing in Lua, so they moved into small Perl scripts. The pattern there is better than the shell-scraping one, and it is the direction the whole design should have gone.

The script prints a fixed, machine-readable header and then rows:

1
print "MEMORY_ALL,MEMORY_USED,MEMORY_FREE,LP_SIZE,LP_COUNT,LP_ADVISE\n";

and it branches per platform internally: on Linux it reads /proc/meminfo plus free -m plus ipcs -m (for shared memory); on AIX it uses svmon; on Solaris, prtconf and vmstat; on HP-UX, swapinfo; on Windows, WMIC. It also subtracts huge-page usage and adds Oracle’s shared segments to “used”, because a database server’s memory is not the same story as a web server’s.

The changelog at the top of that file is a good illustration of the cost of this approach. Over two months it records: “LP_COUNT is null in MSWin32”, “ipcs -m may be larger than memory”, “Set result file to encoding UTF8”, “Set fixed path [/usr/sbin]”. Every entry is a platform surprise.

The whole flow

1
2
3
4
5
6
7
8
9
10
11
12
13
14
collector
    │  ① borrow an SSH shell (kept alive, keep-alive configured)
    ▼
┌───────────────┐  ② write:                ┌──────────────────────────┐
│  shell pool   │ ──────────────────────► │  remote host             │
│               │    echo '<uid>x';        │    export ...;           │
│               │    echo '';              │    echo '<uid>x';        │
│               │    <the command>;        │    echo '';              │
│               │    echo '';              │    <the command's output>│
│               │    echo 'x<uid>';        │    echo '';              │
│               │ ──────────────────────► │    echo 'x<uid>';        │
│               │  ③ read until 'x<uid>'  └──────────────────────────┘
│  ④ parse the text into a metric row
└───────────────┘

The shape to notice: one SSH round trip per metric. A host reporting CPU, memory, swap, filesystem and IOPS is five round trips, and the collector holds one shell open per host while it does them.

That is the cost the whole approach is built on, and it is why the project later grew a second path: a small native agent running on the host that answers the same questions over the collector’s own binary protocol. The agent reads the numbers directly (no text parsing, no locale, no vmstat). The SSH path stayed as the fallback for hosts where nothing can be installed.

What it costs

A round trip per metric. Two hundred hosts reporting five numbers every thirty seconds is a thousand SSH command executions per interval. Pooling the shells makes the connection cheap; it does nothing for the latency.

Parsing human output is a permanent liability. The df header lookup survives a language change; it does not survive a df that renames a column. The vmstat offsets survive a version bump; they do not survive a platform whose header says something else.

The two collection paths can disagree. On the agent path, “total size” is the real filesystem total. On the SSH path it is free + used, which leaves out blocks the filesystem reserves — so the same disk reports a different size depending on how you asked. The metric name does not say which.

“Free” and “used” are only defensible if you define them once. The Perl script counts the page cache as free and adds Oracle’s shared segments to used; free -m counts the page cache as used. Both are reasonable. They are not the same number, and a dashboard that mixes them is lying.

Building the command by string concatenation is a shell-injection surface. The environment is quoted (export KEY='value';), but the command itself is assembled as text and handed to a shell.

There is a latency floor. We sleep briefly before reading, then poll. Every command costs a few hundred milliseconds of latency no matter how fast the remote command is.

What I would do differently

Return numbers, not text. The Perl scripts already do the right thing — a fixed header and machine-readable rows. The Lua tasks that scrape df and vmstat should have been written the same way. The rule: if you must run something on the remote host, make it print a format you define, and treat the output of df or vmstat as an implementation detail of that script rather than of your protocol.

Batch the commands. Nothing requires one round trip per metric. The marker approach generalises: run several commands in one shell invocation, each with its own pair of markers, and parse the sections. Five metrics become one round trip.

Make the column mapping table-driven. The per-platform values[#values - N] arithmetic is the part most likely to be wrong on a platform nobody tested. A table from platform to field positions, plus a test that runs the parser against a recorded sample of each platform’s output, turns a class of silent misreporting into a test failure.

Say which definition you are publishing. “Total size” should be either the filesystem’s real total or explicitly free + used — and the metric name should say which.

And keep pushing the agent. SSH is right for a customer who will not install anything, and it is fine to keep as a fallback. But it is the exception, not the default. The interesting question is not “how do we parse df“ but “how do we know when a host is worth installing an agent on”.

References