CLI said healthy. The dashboard said NOPE.

Somewhere between Monday and “hmm, why is it quiet,” a monitoring box I look after started lying to me in two voices.

The command line swore everything was golden. Service running. Version correct. Plugins listed. Health checks green, on schedule, chirping away like a nesting bird. Then I opened the dashboard and there was just… nothing. No data. And when I clicked the data source that used to feed every panel, the page was a blank obituary with no “configure” button to speak of.

CLI: “We’re fine.” Dashboard: “We are deceased.”

The fun part was that both of them were telling the truth.

The thing that actually happened

One of those boxes had auto-upgraded itself. I didn’t do it. Nobody I know did it. Some background process with too much confidence just… bumped the app to a newer minor against its will. And in that new version, a data source that used to be welded into the core became a standalone plugin.

Which is fine, in theory. The app tried to install it on boot. The install failed. Why? Because the upgrade had unpacked the plugin’s files as root, and the app runs as a different, unprivileged user.

Here’s the subtle bit: it could read those files fine. So the CLI said “plugin installed,” and the backend even logged “plugin registered.” But the app couldn’t replace them — deleting anything from a directory requires write permission on that directory, and it was root’s. So every update attempt died with:

unlinkat /…/plugins/prometheus/9593.js: permission denied

A plugin that registers in the backend but never renders in the UI. Visible to the CLI, invisible to everyone who actually uses the thing. The perfect little scam.

What most people would do wrong

Blame the encryption. There’s a very loud, very real security advisory right now about a default secret key being identical across installs — so the obvious next move is to swap the secret key and “re-encrypt” the database.

I’ll save you the weekend: that was a red herring for this. Changing the secret key on a box that already has data doesn’t rotate cleanly — the encryption uses a key-wrapping scheme, and the app’s own developers explicitly don’t support key→key rotation on an existing install. And it doesn’t even touch a data source that has no password stored anyway. So you’d be re-encrypting a ghost while the real culprit (some root-owned files) sits there laughing.

Check the logs before you touch the crypto. The logs always know.

The fix was one command

sudo chown -R <the-app-user> <the-plugin-directory>

That’s it. That’s the whole war. Give the files back to the user the app actually runs as, restart, and the background installer quietly finishes the job it’s been stuck on for a week. My data source had also been deleted in the confusion, so I recreated it — but with its original UID, which is the trick that saved me re-doing every dashboard and alert rule. Same identifier, snap, everything binds again.

The actual lesson

Unattended upgrades are a gift and a curse. A patch agent that bumps big pieces of software without a human is how you get “it was fine until I looked” incidents. Pin the thing, or scope the auto-updater away from it.
CLI health ≠ user-visible health. My checks all passed because they checked the process, not the product. The dashboard was the only thing looking at reality.
File ownership is a silent killer. It says nothing, logs nothing but a permission denial, and everything looks right until you read the untruncated line.
Recreate with the same identifiers. IDs/UIDs are referenced all over the place. Keep them and a replacement slots back in like it never left.

So: the box is purring again, the panels have their green back, and the person responsible for the “surprise upgrade” remains conveniently anonymous — which is to say, it was me-by-automation, and I’ve since locked the gate.

Moral of the story: when the CLI and the dashboard disagree, bet on the one that’s looking at the files.