Monitoring

Scripts And Monitoring Recipes

Parse script output, observe unmanaged devices and build useful checks.

These examples distinguish capabilities already implemented from additional collectors or media components that must be configured. There is no universal way to read a physical device without some observable signal: API, SNMP, a stream, USB/serial, contact closure, an upstream managed device, or an external sensor.

Program reports 0 or 1

First decide whether 0/1 is the process exit code or text written to stdout, and which value means healthy. Printing 1 while exiting 0 does not automatically mean failure. Sysavia's script runner defaults successful process exits to up; a nonzero exit makes the check down. Explicit JSON status can report a business-level failure even when the wrapper itself completes successfully.

Create a script in Scripts, allow its ID in the target agent's remoteScriptIds, and add an Agent script check to the compute server. This Python wrapper treats stdout 1 as healthy and 0 as failed. Replace the command and arguments with the real executable; the command below is illustrative. Use absolute paths because scripts run from a temporary working directory.

import json
import subprocess

try:
    result = subprocess.run(
        ['/usr/local/bin/check-program'],
        capture_output=True, text=True, timeout=10
    )
    raw = result.stdout.strip()
    if result.returncode != 0:
        report = {'status': 'down', 'detail': f'Program exited {result.returncode}'}
    elif raw not in ('0', '1'):
        report = {'status': 'unknown', 'detail': 'Expected stdout 0 or 1'}
    else:
        code = int(raw)
        report = {'status': 'up' if code == 1 else 'down', 'value': code,
                  'metrics': {'programHealthy': code},
                  'detail': f'Program returned {code}'}
except (OSError, subprocess.TimeoutExpired) as error:
    report = {'status': 'unknown', 'detail': str(error)}
print(json.dumps(report))

For exit-code-driven programs, instead map result.returncode according to that program's documented convention. Use PowerShell's $LASTEXITCODE immediately after invoking an external executable on Windows. PowerShell cmdlet failures and $? are not interchangeable with external program exit codes.

Configure a rule: When Check / service is down → this script check → Send alert → Transfer files. Use matching cycles/cooldown to avoid transient alarms and repeated transfers. The transfer action selects an agent and its locally configured transfer job key. File jobs copy between absolute paths accessible to that agent, including mounted network shares; they are not an embedded SFTP service. Your organization needs live-action access for file, script, service, HTTP and API operations. Contact Sysavia support if these actions are unavailable. Scheduled script checks still require the agent's script allowlist. In-app Send alert writes to global Activity; external notifications can use a configured HTTP/API action.

Alternatively, an existing scheduled program can POST a named program.result event with data {"healthy":false} to /api/automation-events, using AUTOMATION_EVENT_TOKEN, and a custom event rule can filter that data. Event pushes describe an occurrence; periodic telemetry with TTL describes continuing health.

Parse script output

An Agent script check automatically parses a JSON object from stdout:

{
  "status": "down",
  "value": 0,
  "detail": "Program output is unavailable",
  "metrics": {
    "programHealthy": 0,
    "queueDepth": 14,
    "lastFrameAgeSeconds": 12
  }
}

Supported status values are up, down, and unknown. value is the check's reported value; flat metrics become system readings. Link those to Live values, canvas text, colors or rule conditions. Output must be one JSON document, without startup banners or other text on stdout. Put diagnostic logging on stderr. Without valid JSON, trimmed stdout becomes a string value, and exit code determines default health. For arbitrary text/CSV/XML or nested vendor JSON, parse it in the script and emit the normalized object above. Script output is bounded to 64 KiB and runtime to the configured timeout (maximum 300 seconds).

Use an Agent script check when the output must become a monitored reading. A Run script action reports command completion/failure; it does not automatically publish its report into a check. To react to an action's business result, use a follow-up monitored check with recovery verification, or have the script send an explicit custom event.

Detect an unplugged cable without an agent

The observation can come from a different device:

  • Ethernet: poll the connected managed switch's link state through SNMP or its API. Link-down identifies a failed link, not uniquely an unplugged cable: the far-end device could be powered off.
  • SDI/HDMI/ASI: read input-lock or signal-present state from the receiving router, encoder, capture card or multiviewer. A downstream receiver can detect loss even if the source is unmanaged.
  • Passive patching: use a downstream signal detector or sensor. Sysavia cannot directly sense a passive BNC cable with no telemetry at either end.

Create an API/SNMP/push check and associate it with the appropriate port under Edit ports. Use Port signal lost or the check-down trigger. Canvas Wire connected/removed triggers describe edits/disconnections in Sysavia's model; they are not automatic physical-cable detectors.

Equipment exposes only a web dashboard

If “cannot log in” means no SSH/RDP or agent installation, that does not prevent reading its HTTP status interface. Inspect the dashboard's requests and identify its read-only JSON endpoint, preferably a documented vendor API. In API integrations, configure base URL, GET operation, authentication and extraction path. For example, an illustrative response {"inputs":[{"locked":false}]} can use value path inputs.0.locked; configure an API check healthy when it equals true.

For outbound API checks, Sysavia supports unauthenticated requests, Bearer tokens, or a configured header backed by a server environment variable. It does not implement arbitrary vendor login/session renewal, browser-rendered dashboards, or a visual CSS-selector scraper. A dashboard returning HTTP 200 is not proof its equipment is healthy; it may be a login page or an application shell.

If the page has only HTML, an external collector can fetch/parse it; if JavaScript renders the status, a browser worker can inspect the rendered element. Normalize the result and push telemetry/custom events to Sysavia. Such a collector would need to be added or installed; an iframe alone cannot supply a dependable trigger. If the dashboard requires credentials you do not have, Sysavia cannot read protected status. Use a supported read-only account or a different observable signal.

Monitor display loss and cameras

Track three separate checks:

  1. Device reachability (ICMP, TCP, vendor API).
  2. Video availability (receiver lock, NVR channel status, or fresh decoded frames).
  3. Content quality (black/frozen image, silence, expected test pattern).

A camera answering ping can still have a failed stream. A working stream can still show a frozen picture. A monitor receiving valid input can still have a failed panel/backlight. Actual screen confirmation requires the monitor's own diagnostics or an external visual/light sensor. HDMI/SDI capture proves the sampled signal, not the pixels displayed by the physical monitor.

For a camera, use vendor/NVR API status now, or run a stream-probe script on a nearby agent and return JSON. A custom collector can use FFmpeg blackdetect, freezedetect, and silencedetect, then publish metrics such as lastFrameAgeSeconds, blackSeconds, and frozenSeconds. These filters are available in FFmpeg, but Sysavia does not yet bundle a continuous media-analysis service. Use duration thresholds and intended-content context; a static scene or intentional black is not automatically a fault. Never test only the open RTSP TCP port and call it video health.

UPS switches to battery

Use a UPS management card's SNMP/API, or a USB/serial-connected host running Network UPS Tools (NUT). An agent on that host can run a check using upsc rackups@localhost ups.status, with a subprocess timeout. Split the output into tokens: OL means on-line power, OB on battery, LB low battery. Publish upsOnBattery, upsLowBattery, batteryPercent, and runtimeSeconds where the device provides them. Parse NUT errors/missing data as unknown, not mains failure.

Examples: On battery for two matching cycles → alert; on battery AND remaining runtime < 300 seconds → alert + an approved workload-shutdown script; mains restored → recovery alert. These are configured checks/rules; Sysavia has no bundled NUT client or UPS-specific automatic discovery. Keep the monitoring host and network path powered so battery telemetry can continue.

Live feed windows

Live playback is available through + Playback. Windows are independent grid objects, separately movable and resizable from equipment. The same source can appear in several grids, including status-only dashboards. Optional equipment links can reference another grid. Monitor equipment remains a physical topology object.

  • RTSP/RTP cameras and physical capture use an on-demand FFmpeg relay on the panel or a capture sidecar beside an agent.
  • Browser-ready HLS/WHEP WebRTC and view-only VNC are supported, including a Proxmox QEMU console adapter. Hyper-V/vSphere currently need guest VNC or an existing console gateway.
  • Playback releases connections in hidden grids/tabs and Simulate, and resumes on return. Larger viewers use the same configured source resolution; resizing alone does not request a higher-quality stream.

See live playback setup for source configuration, capture hardware requirements, credentials, Nginx and known limits. Keep automated stream-health monitoring independent of whether someone has a preview open. Black/freeze/audio-loss analysis is not bundled.

Source references: go2rtc browser integration, Axis video streaming, FFmpeg filters, NUT configuration and status.