🌐 Connectivity

# TuneVault Agent Connection Failure — Diagnosis & Recovery Runbook

⏱ 10–30 min oracle-proxy.py + Agent Channel Updated 2026-05-17
🔴 Agent shows as disconnected in TuneVault
TuneVault polls your agent every 10 seconds via an outbound HTTPS long-poll from the Oracle server to `tunevault.app`. When the agent is disconnected, health checks queue but cannot run. No inbound ports are involved — the issue is always on the outbound side.

## 1. How the Agent Connection Works

TuneVault's architecture is outbound-only. The agent on your Oracle server initiates all connections — TuneVault's cloud never opens inbound connections to your network.

- `oracle-proxy.py` runs as a systemd service (`tunevault-proxy`) on the Oracle server.
- The proxy opens a long-poll HTTPS connection to `https://tunevault.app/api/agent/poll` every 10 seconds.
- TuneVault sends health check jobs through this channel. The proxy executes them against the local Oracle listener and returns results.
- No inbound ports required. No DNS records. No third-party tunnel services.

ℹ️ What "disconnected" means
If no poll from the agent has arrived in the last 30 seconds, TuneVault marks the connection as disconnected. Health checks fail with "Agent not connected." The agent reconnects automatically once it resumes polling — no manual intervention needed if the service recovers on its own.

## 2. Quick Diagnostic Checks

Run these from the Oracle server to pinpoint the failure mode fast.

    # 1. Check proxy service status
    sudo systemctl status tunevault-proxy

    # 2. Check last 50 log lines
    sudo journalctl -u tunevault-proxy -n 50 --no-pager

    # 3. Confirm outbound connectivity to TuneVault
    curl -sf https://tunevault.app/health
    # Expected: {"status":"ok",...}

    # 4. Check agent registration status (requires API key from installer)
    curl -sf -H "X-API-Key: YOUR_AGENT_API_KEY" \
      "https://tunevault.app/api/agent/status?connection_id=YOUR_CONNECTION_ID"
    # Expected: {"registered":true,"last_poll_at":"..."}

Service: inactive/failed
Go to Step 3 — restart the service.
curl tunevault.app fails
Go to Step 4 — outbound network block.
Service running, status shows disconnected
API key may be revoked. Go to Step 6.
registered:false in status response
Re-register the agent. Go to Step 6.

## 3. Verify Proxy Service on the Oracle Server

SSH to the Oracle server where the agent is installed. All commands run as `root` or the install user.

    # Check service state
    sudo systemctl status tunevault-proxy

    # If inactive or failed — restart
    sudo systemctl restart tunevault-proxy

    # Re-check after restart
    sudo systemctl status tunevault-proxy

    # Follow logs in real time to see poll loop activity
    sudo journalctl -u tunevault-proxy -f

A healthy agent logs a poll attempt every 10 seconds. If you see `poll loop error` repeating, the outbound connection is failing — go to Step 4.

### Common log errors and their causes

- `ConnectionError: Failed to establish a new connection` — outbound HTTPS to tunevault.app is blocked. Go to Step 4.
- `401 Unauthorized` on the poll endpoint — API key was revoked or is missing. Go to Step 6.
- `cx_Oracle.DatabaseError: ORA-12541: TNS:no listener` — Oracle listener is down. Go to Step 5.
- `cx_Oracle.DatabaseError: ORA-01017: invalid username/password` — reader credentials need updating. Regenerate via Settings → Connections.
- `SSL: CERTIFICATE_VERIFY_FAILED` — system time is wrong. Run `timedatectl` and sync NTP.

    # Check system time (clock skew breaks TLS)
    timedatectl status

    # Sync if needed (RHEL/OL)
    sudo chronyc makestep

    # Sync if needed (older systems)
    sudo ntpdate -u pool.ntp.org

## 4. Outbound Network Connectivity

The agent only needs outbound HTTPS (port 443) to `tunevault.app`. No inbound rules. No special DNS. No third-party services.

    # Confirm outbound 443 works from the Oracle server
    curl -sv https://tunevault.app/health 2>&1 | grep -E "Connected|HTTP|SSL"

    # If curl fails — test raw TCP connectivity
    nc -zv tunevault.app 443
    # Expected: Connection to tunevault.app 443 port [tcp/https] succeeded!

    # Check local iptables for outbound DROP/REJECT rules
    sudo iptables -L OUTPUT -n | grep -E "DROP|REJECT"

    # On RHEL/OL with firewalld
    sudo firewall-cmd --list-all

⚠️ Proxy server / corporate firewall
If the Oracle server routes outbound traffic through a corporate HTTP proxy, set the `HTTPS_PROXY` environment variable in the systemd service unit:

    sudo systemctl edit tunevault-proxy
    # Add under [Service]:
    Environment=HTTPS_PROXY=http://proxy.corp.example.com:3128

Then: `sudo systemctl daemon-reload && sudo systemctl restart tunevault-proxy`

## 5. Oracle Listener Checks

The agent connects to Oracle locally. If the listener is down, the agent polls fine but health check jobs fail.

    # Check listener status (run as oracle user)
    lsnrctl status

    # If listener is down
    lsnrctl start

    # Verify connection works with the reader account
    sqlplus tunevault_reader/PASSWORD@//YOUR_DB_HOST:YOUR_DB_PORT/YOURDBSID

    # Check listener log for ORA- errors
    tail -50 $ORACLE_BASE/diag/tnslsnr/$(hostname -s)/listener/alert/log.xml

ℹ️ EBS environments: APPS.DUAL probe
For Oracle E-Business Suite connections, the agent runs an additional `SELECT 1 FROM APPS.DUAL` probe to gate EBS-specific checks. If this fails with ORA-00942 (table not found), the APPS schema is not accessible via the reader account. Verify `tunevault_reader` has SELECT on APPS.DUAL.

## 6. Re-register or Reinstall the Agent

If the API key is invalid, the connection was deleted, or the agent config is corrupted — re-run the installer with a fresh token.

    # Get a fresh install token from TuneVault:
    # Settings → Connections → [your connection] → Reinstall Agent → Copy token

    # Re-run the installer — it detects an existing install and upgrades in place
    curl -sSL https://tunevault.app/install.sh | bash -s -- --token YOUR_NEW_TOKEN

    # The installer will:
    # - Update oracle-proxy.py to the latest version
    # - Write a new API key to /etc/tunevault/agent.conf
    # - Restart tunevault-proxy
    # - Run self-test and print PASS/FAIL for each check

⚠️ Old API keys are invalidated on reinstall
A fresh install token generates a new API key. The old key stops working immediately. If you have monitoring scripts that use the old key, update them.

### Manual re-registration (without full reinstall)

    # If you only need to re-register (service is running, key is stale):
    curl -sf -X POST https://tunevault.app/api/agent/confirm \
      -H "Content-Type: application/json" \
      -d '{"token":"YOUR_INSTALL_TOKEN"}'
    # Expected: {"ok":true,"connection_id":"..."}

    # Restart the proxy to pick up the new registration
    sudo systemctl restart tunevault-proxy

## 7. Fallback: Switch to Direct TCP Mode

If the agent cannot run on the Oracle server (strict egress policy, no Python, no root access), switch to Direct TCP. This bypasses the agent entirely — TuneVault connects directly to the Oracle listener over the network.

1. Go to Settings → Connections in TuneVault.
2. Edit the affected connection.
3. Change Connection Type to Direct TCP.
4. Enter the Oracle server's public IP (or VPN-accessible IP) and listener port (default 1521).
5. Save and run a new health check to verify connectivity.

⚠️ Direct TCP security note
Direct TCP requires the Oracle listener port to be reachable from TuneVault's server IPs. Restrict port 1521 to TuneVault's IP range using firewall rules — do not expose it to `0.0.0.0/0`. The agent (outbound-only) is preferred for production environments because it requires no inbound firewall rules.

## 8. Prevention

✅ Agent reliability checklist

- Ensure `tunevault-proxy` starts on boot: `sudo systemctl enable tunevault-proxy`
- Enable TuneVault's autonomous monitoring — it polls the agent every 15 minutes and alerts on disconnect
- Keep the Oracle listener in `auto-start` mode so it recovers after server reboots
- Keep NTP synchronised — clock skew breaks TLS certificate validation
- Do not revoke the agent API key from the TuneVault dashboard without immediately issuing a new install token
- Test outbound HTTPS before any corporate firewall change: `curl -sf https://tunevault.app/health`

    # Enable autostart
    sudo systemctl enable tunevault-proxy

    # Verify enabled
    sudo systemctl is-enabled tunevault-proxy
    # Expected: enabled

    # Quick connectivity self-test (run anytime)
    curl -sf https://tunevault.app/health | python3 -m json.tool

### Monitor your Oracle agent health automatically

TuneVault's autonomous monitoring detects agent disconnections within 15 minutes and alerts you before your team notices missing health check data.

Set Up Monitoring →