Documentation — Architecture

# TuneOps & Automation

An autonomous connection can run a health check every 30 minutes. Without care, one unresolved problem becomes dozens of alerts a day. TuneOps exists to turn a stream of findings into a small number of tickets that accurately reflect what's actually happening.

## Every finding gets a canonical key

The mechanism this whole page depends on is a stable identity for "the same problem" across runs — a canonical key built from the connection, the check ID, and the specific object the finding is about. Two health check runs an hour apart both surfacing "tablespace USERS at 92%" produce the identical key, even though they're two separate runs.

This key is what Health Check Engine's `finding_history` table also uses — the two systems share the same identity concept, applied to two different purposes: history tracking, and ticket lifecycle.

## Three outcomes, not one

Every finding from a completed health check is looked up by its canonical key, and exactly one of three things happens:

Match found | Result
--- | ---
An open ticket already has this key | The existing ticket's `occurrence_count` is bumped and `last_seen_at` updated — no new ticket. Severity escalates if the finding got worse.
A resolved ticket has this key | The ticket is reopened rather than duplicated — the same problem returning gets the same history, not a fresh start.
Nothing matches | A new ticket is created.

This is the exact mechanism behind a ticket that reads "confirmed 9 times" — nine separate health check runs, each independently re-detecting the identical canonical key, each bumping the same row rather than creating a new one.

## Auto-resolve: closing tickets nobody has to close

Each health check run produces a set of active canonical keys. Any open ticket whose key is not in that active set — meaning the problem it was tracking no longer shows up — gets automatically resolved. Nobody has to remember to close a ticket once the underlying issue is fixed; the next clean run does it.

## The lifecycle

State | Meaning
--- | ---
OPEN | New finding, not yet triaged.
ACKNOWLEDGED | Someone has seen it and is on it.
EXECUTING | A remediation is in progress.
RESOLVED | The finding stopped recurring, or was manually closed.
DEFERRED | Parked deliberately — acknowledged as low priority for now.
REOPENED | A resolved or deferred ticket whose canonical key reappeared.

## Notifications are rate-limited, not just deduplicated

Ticket creation and occurrence bumps can happen far more often than a human should be emailed about them — a connection running checks every 30 minutes could otherwise generate a notification every 30 minutes for the same unresolved issue. Two separate controls prevent that:

→ Dedup window: a 1-hour window per ticket — repeated occurrences inside that hour don't each trigger a fresh email.
→ Hard cap: a maximum of 10 notifications per ticket per hour, enforced by counting recent sends in `tuneops_notification_log` before allowing another.
This is deliberately layered on top of, not instead of, occurrence tracking. The ticket's occurrence count keeps rising with every recurrence even when no notification fires — you never lose the "how many times has this happened" signal, you just stop getting paged for it.
See Health Check Engine for how a finding is produced in the first place, or Architecture Overview for how this fits into the rest of the system.