Documentation — Architecture

TuneOps & Automation

An autonomous connection can run a health check every 30 minutes. Without care, one unresolved problem becomes dozens of alerts a day. TuneOps exists to turn a stream of findings into a small number of tickets that accurately reflect what's actually happening.

A finding is matched against tickets by a canonical key. Three outcomes: an open ticket's occurrence count is bumped, a resolved ticket is reopened, or a fresh ticket is created if nothing matches. A finding that stops appearing gets auto-resolved.

Every finding gets a canonical key

The mechanism this whole page depends on is a stable identity for "the same problem" across runs — a canonical key built from the connection, the check ID, and the specific object the finding is about. Two health check runs an hour apart both surfacing "tablespace USERS at 92%" produce the identical key, even though they're two separate runs.

This key is what Health Check Engine's finding_history table also uses — the two systems share the same identity concept, applied to two different purposes: history tracking, and ticket lifecycle.

Three outcomes, not one

Every finding from a completed health check is looked up by its canonical key, and exactly one of three things happens:

Match foundResult
An open ticket already has this keyThe existing ticket's occurrence_count is bumped and last_seen_at updated — no new ticket. Severity escalates if the finding got worse.
A resolved ticket has this keyThe ticket is reopened rather than duplicated — the same problem returning gets the same history, not a fresh start.
Nothing matchesA new ticket is created.

This is the exact mechanism behind a ticket that reads "confirmed 9 times" — nine separate health check runs, each independently re-detecting the identical canonical key, each bumping the same row rather than creating a new one.

Auto-resolve: closing tickets nobody has to close

Each health check run produces a set of active canonical keys. Any open ticket whose key is not in that active set — meaning the problem it was tracking no longer shows up — gets automatically resolved. Nobody has to remember to close a ticket once the underlying issue is fixed; the next clean run does it.

The lifecycle

StateMeaning
OPENNew finding, not yet triaged.
ACKNOWLEDGEDSomeone has seen it and is on it.
EXECUTINGA remediation is in progress.
RESOLVEDThe finding stopped recurring, or was manually closed.
DEFERREDParked deliberately — acknowledged as low priority for now.
REOPENEDA resolved or deferred ticket whose canonical key reappeared.

Notifications are rate-limited, not just deduplicated

Ticket creation and occurrence bumps can happen far more often than a human should be emailed about them — a connection running checks every 30 minutes could otherwise generate a notification every 30 minutes for the same unresolved issue. Two separate controls prevent that:

Dedup window: a 1-hour window per ticket — repeated occurrences inside that hour don't each trigger a fresh email.
Hard cap: a maximum of 10 notifications per ticket per hour, enforced by counting recent sends in tuneops_notification_log before allowing another.
This is deliberately layered on top of, not instead of, occurrence tracking. The ticket's occurrence count keeps rising with every recurrence even when no notification fires — you never lose the "how many times has this happened" signal, you just stop getting paged for it.
See Health Check Engine for how a finding is produced in the first place, or Architecture Overview for how this fits into the rest of the system.