How to Design a Plugin System for Device Drivers

The collector ships as a product. The customers are database and host vendors, and the first thing each of them asks is “can we support our database without you shipping a new release for us?”

The answer has to be yes. But there are two very different requests hiding in that sentence:

  • “Our database is not on your list.” — needs a new driver: new connection code, new action vocabulary, new tasks.
  • “We want to monitor a metric you do not have.” — needs a new task: a JSON document, and nothing else.

The plugin system is designed so that the second request needs no code at all, and the first needs a self-contained jar.

What this post is about. It is about the plugin system of a monitoring collector: how a vendor’s driver is loaded at run time, what a plugin may do, and where the model stops being flexible. You will learn what a plugin is (a separate piece of code, loaded on demand, that talks to the host through a fixed interface), how pf4j loads one, and why “a new metric” and “a new database” cost very different amounts of work.

What you need to know first. A jar is a zip file of compiled Java classes; a classloader is the object that decides which class a name resolves to, so two jars can contain same-named classes without colliding; and an SPI (service provider interface) is the pattern where the host defines an interface and the plugin implements it.

The situation

The core cannot know about every database. A driver has to be able to:

  • create a live connection to a target and keep it alive across a fleet,
  • answer “can you do this kind of collection?” (the action vocabulary),
  • contribute a set of collection tasks,
  • and hook itself into the collection pipeline at the right phases.

None of that may require editing the core.

Why pf4j

The alternatives were a hand-rolled classloader, an annotation-processor-only design, or a plain “drop a jar in plugins/ and restart” with no isolation.

A hand-rolled classloader is about 150 lines, and then you have to re-implement dependency resolution between plugins, plugin lifecycle, and unloading. An annotation-only design cannot express “this plugin depends on that plugin”. pf4j gives all of it — lifecycle with start/stop/unload events, dependency ordering through Plugin-Dependencies, per-plugin classloaders, a generated extension index — and the price is a Java-shaped API, MANIFEST conventions, and some Gradle packaging work.

Packaging is a zip, not a jar: it contains classes/ (the plugin) and lib/ (the plugin’s own third-party jars). pf4j expands the zip at load time and gives each plugin a PluginClassLoader with classes/ and lib/*.jar on it. Because the exploded layout puts the manifest under classes/META-INF/, the descriptor finder has to be taught where to look: ScrapePluginManager.createPluginDescriptorFinder() overrides getManifestPath to return classes/META-INF/MANIFEST.MF for an exploded plugin and MANIFEST.MF for a plain one.

Living inside pf4j’s package

Look at the package name of those three files: org.pf4j.scrape. Not com.enmotech... — inside pf4j’s own namespace, on purpose.

pf4j’s PluginClassLoader is child-first, with two exceptions: java.* and org.pf4j.* are delegated to the parent (application) classloader. That single rule is the whole design:

  • The bridge classes (ScrapePlugin, ScrapeConnectionExtension) are loaded once, by the host.
  • A plugin’s driver object therefore is the core’s ScrapeDriver instance, not a look-alike from a different classloader.
  • Move the bridge to com.enmotech.bethunepro.scrape and every plugin would load its own copy, so the object handed to installDriver would not be the core’s ScrapeDriver — an immediate ClassCastException.

The cost is real: you own a namespace that is not yours, and the packaging rule that keeps core classes out of each plugin’s lib/ (exclude "*scrape*", "*pf4j*", "*kotlin*", "*logback*", "*slf4j*") becomes load-bearing rather than tidy. A bundled copy of the core inside a plugin would silently duplicate classes across the boundary.

The SPI: driver versus connection

The contract a plugin implements is two small interfaces. The first one is the driver:

1
2
3
4
5
6
interface ScrapeDriver {
    fun id(): String
    fun version(): String = "1.0.0"
    fun createConnection(instance: ScrapeInstance): ScrapeConnection?
    fun isSupportAction(action: String, instance: ScrapeInstance): Boolean
}

The four methods are: the driver’s id, its version, a factory that opens a connection to one target instance, and a yes/no test for whether this driver can perform a given action. The second interface, ScrapeConnection, describes a live session — isConnected(), checkLive(), attribute(), platform() — and is Closeable.

The split between the two interfaces is the first design decision worth naming. The driver is the vocabulary, the factory and the lifecycle: one long-lived, stateless object per vendor family, owning the action enum (OracleStatements, HostStatements, MysqlStatements…). The connection is a live session against one target: stateful, pooled, reconnectable, and closed when the instance goes away.

The granularity is per vendor family, never per metric. A driver is “Oracle”, not “Oracle session count”.

A plugin is a driver

1
2
3
4
5
6
7
abstract class ScrapePlugin : Plugin, ScrapeDriver, Dump {
    override val logger: Logger = Dump.getLogger(id())
    constructor() : super(EmptyWrapper)
    constructor(wrapper: PluginWrapper) : super(wrapper)
    var options: Container<Any?>? = null
    override fun id(): String = wrapper?.descriptor?.pluginId ?: "plugin"
}

ScrapePlugin inherits from pf4j’s Plugin (the base class every plugin extends) and from ScrapeDriver (the contract the core understands), so the plugin base class is the driver. wrapper?.descriptor?.pluginId reads the id out of the plugin’s manifest — with a fallback for a plugin that was constructed without one. The two constructors are the pf4j ones: one for a plugin that has a wrapper, one (EmptyWrapper) for one that does not yet.

The core never imports pf4j — the dependency arrow points the other way, from the pf4j bridge module to the core.

At startup, every loaded plugin that is a ScrapePlugin is installed as a driver. The manager subscribes to the register’s Started event, starts the plugins, filters the loaded plugins down to ScrapePlugins and calls scope.driverManager.installDriver(...) on each one.

That gives one deployable, one id, one version, one lifecycle. The limit: one plugin is exactly one driver. You cannot ship “mysql” and “mariadb” in one artifact, and the plugin id and the driver id are welded together — id() is literally the manifest’s Plugin-Id. The model conflates a unit of deployment with a unit of capability; the workaround is the dependency graph, and the Oracle plugin declares Plugin-Dependencies: host,database,dg.

What a plugin contributes

A driver’s install hook is where it adds both its tasks and its pipeline handlers:

1
2
3
4
5
6
7
8
override fun install(options: Container<Any?>?, scrapeRegister: ScrapeRegister) {
    super.install(options, scrapeRegister)
    scrapeRegister.taskManager.install { this.installOracleTask() }
    scrapeRegister.collector.install {
        oracleMonitorTaskHandlers()
        oracleInfoHandlers()
    }
}

The first install registers the plugin’s tasks (the things to collect); the second registers handlers — the interceptors that run when a task is collected. super.install is the base class’s own hook, called first.

So the plugin surface is not phase splicing. Pipeline.merge() exists in the core and would let a plugin insert a brand-new phase relative to an existing one — but it is called from nowhere except the ported pipeline test. What a plugin actually gets is the ability to register an interceptor for an existing phase, keyed by task id. The phases are fixed:

1
Before → Transform → Render → Valid → Collecting → Collected → After

Each arrow is one phase, in the order the work happens, and the list is fixed by the core. The hooks a plugin can fill are before, transform, render, valid, connectBefore, collected, success, failed, paramParser, last.

The interesting ones are the ones that change what actually runs:

  • before may short-circuit the entire task — return a ScrapeResponseData and the pipeline merges it and finishes. A plugin can therefore answer a task without touching the device.
  • render may return a different ScrapeTask — swapping the action and the script at run time. The host plugin uses this to substitute a Perl script on Solaris and HP-UX.
  • connectBefore may rewrite the instance before the connection is made — the Data Guard plugin injects ORACLE_VERSION into the environment there.
  • valid may veto execution.

What a plugin cannot do: add or reorder a phase, replace the driver of an existing instance, change another plugin’s action table.

Capability negotiation instead of a shared enum

The core holds no action table. An action is a string, and a driver claims it. The collector asks the driver, and falls back when the answer is no:

1
2
3
4
5
6
7
8
collectCall.instance =
    if (register.driverManager.take(instance.driver, instance.version)
        ?.isSupportAction(action, instance) == true) {
        // this driver can perform the action
    } else {
        // the instance's referenceId names another plugin, e.g. "host"
        val referenceId = instance.referenceId
    }

take(driver, version) looks up the driver by id and version; isSupportAction asks it whether it can do the work. When it cannot, the collector logs a line and takes the instance’s reference instance instead — a second instance, of a different plugin, that can do it.

This is the part that makes the plugin model work in practice. A database plugin implements only what JDBC can do; the shell-and-Perl part of the work is delegated to the host plugin through the instance’s referenceId. The database driver never needs to know that SSH exists.

The cost of string actions is that there is no compile-time checking, the action namespace is global and unversioned, and the capability check is a runtime string comparison on every dispatch.

The real extension point: a task is data

This is the part I would put in front of a customer, because it is the one that removes them from the code path entirely.

A task is a record:

1
2
3
4
5
6
7
data class ScrapeTask(
    val id: String = guid(),
    var action: String? = null,
    @JvmField var scripts: ScrapeScripts = ScrapeScripts(),
    @JvmField var inputParams: ScrapeParams = ScrapeParams(),
    @JvmField var outputParams: ScrapeParams = ScrapeParams()
)

A task has an id, the action to perform, the scripts to run, typed input and output parameters — and a depends map (task-id → task that can be either a reference or an inline definition), which is left out of the excerpt above. An action is a verb the driver understands (ORACLE_QUERY, LOG_FILE_READ, SHELL_TEXT_LINE, MYSQL_QUERY…), the scripts are keyed by a regex matched against the platform (type:os:arch:version:describe), and the parameters are typed.

A “custom task” (自定义采集项) is therefore a JSON document: pick an action a supported driver already understands, attach a script per platform, give it parameters and a cron. The app exposes it over HTTP, so a new metric on an already-supported database needs no code, no plugin and no release. A new database needs a plugin; a new metric does not. That is the line the whole plugin system is drawn to make possible.

Script selection is by platform regex, with a catch-all: the collector drops the entries whose version is the wildcard *, takes the first entry whose version regex matches the instance’s platform, and falls back to task.scripts["*"] if none matches. If nothing matches, subject.script stays null, the collector writes a debug line, and the caller gets an empty result — no data and no error. A monitoring product that silently stops reporting is worse than one that reports a failure.

Failure modes

Situation What happens
Plugin jar broken or manifest missing pf4j logs a PluginException per plugin and continues. The plugin is simply absent.
A plugin’s dependency is missing DependenciesNotFoundException is caught and logged; the plugin stays unresolved. But the install loop takes all plugins regardless of state, so an unresolved plugin is still registered as a driver.
Plugin throws during start() Caught per plugin; the others still start.
The driver’s install() throws The driver object is already in the container, so instances resolve to a driver whose tasks and handlers were never registered. The failure surfaces later as “the task@… can’t be found” at collect time.
No driver for an instance An exception from the Valid phase, caught by the engine and attached to that task’s response. One task fails, the request survives.
No script matches the platform Silent empty result.
Driver uninstalled The plugin is stopped and unloaded, and the instance manager closes the instances bound to it.

Trade-offs

“A plugin is a driver” buys simplicity and costs flexibility. One deployable with one id, one version and one lifecycle is a very clean model — and it means you cannot ship two drivers with different option sets in one artifact, and the identity of the plugin and of the capability can never diverge.

String actions buy extensibility and pay in typos. A typed enum in the core would require the core to know every action, defeating the plugin model; a generic registry erases to Any at runtime anyway. The trade is strings — which is what makes JSON-authored tasks possible — paid for with a global, unversioned namespace where two plugins can collide.

Task ids and collector handlers are keyed by a global string. Two third-party plugins that both define a task called host_info silently collide: last registration wins, and one plugin’s success handler fires for the other’s task. There is no plugin-scoped namespace.

The ordering between the plugin manager and the task manager is an undocumented assumption. Plugin-contributed tasks are only picked up during the RegisterEvents.Started pass, and that pass only sees them if the plugin manager’s handler runs first — which holds only because the plugin manager is installed first.

The extension index is dead weight. META-INF/extensions.idx is generated by kapt, packaged, and shipped — and then never consulted. Connections are created by driver.createConnection(instance), not by pf4j’s extension discovery. The machinery invites the reader to expect extension-point discovery that does not actually happen.

What I would do differently

Namespace everything by plugin. Task ids, handler keys and action names should be scoped by the plugin that owns them (oracle/oracle_info), and the collision problem disappears rather than being documented.

Make the platform regex required, and fail loudly. A task with no matching script should produce a failed metric, not an empty one. The “no data” and “could not run” cases must be distinguishable in the product, because a monitoring system that goes quiet is indistinguishable from one that is fine.

Make the unresolved-plugin case explicit. If a plugin’s dependencies are missing, it should not be installable as a driver. Right now the install loop takes every plugin regardless of state, and a half-initialised driver is worse than a missing one.

Delete the extension machinery, or wire it up. Either the connections are discovered through pf4j’s extension points — in which case the index has a purpose — or the index generation and the ScrapeConnectionExtension type should go, because shipping a mechanism that never runs is how a reader ends up believing the system works differently than it does.

And decide what a plugin version means. The driver key is id@version with an exact lookup, and the plugin version is the driver version. Bumping a plugin’s Plugin-Version therefore orphans every instance pinned to the old one, with no warning. A “latest” fallback, or an explicit compatibility range, would make upgrades a decision instead of a trap.

The part worth keeping exactly as it is: the SPI is two small interfaces, the plugin is a self-contained jar, and the task is data. A vendor can ship a driver without touching the core, an operator can add a metric without shipping anything, and the core never learns the name of a database it has never seen.

References

  • pf4j — the plugin framework this system uses
  • pf4j.org — the project’s home page
  • Developing plugins — the lifecycle, packaging and extension points
  • Kotlin coroutines — the model the collector’s pipelines run on
  • ktor — the HTTP framework the core’s pipeline copy is ported from