> For the complete documentation index, see [llms.txt](https://wiki.datadike.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://wiki.datadike.com/pam/deploy-operate/monitoring.md).

# Monitoring and health

How to tell whether the installation is healthy: the health endpoint an external monitor should poll, the console screens that show component and task state, and the alerts worth subscribing to.

## What this is and when to use it

This page is about the health of the installation — is the database reachable, are the components up, are the background tasks running. It is not about watching what a user is doing inside a session; that is live session monitoring, under **Audit › Sessions › Online**, and it answers a different question.

Health monitoring has three layers, and most installations only set up the second one:

1. **An external monitor** polling the health endpoint. This is the layer that wakes someone up at 03:00.
2. **Console screens** — **System Settings › Components** and the task dashboard reached from **System Settings › Operations**. These are accurate but you have to be looking at them.
3. **In-product alerting** — notification subscriptions for storage connectivity and component performance.

## The health endpoint

The solution exposes an unauthenticated health endpoint at `/api/health/`. It returns JSON:

```
{"status":true,"db_status":true,"db_time":0.0083,"redis_status":true,"redis_time":0.0022,"time":1788772352}
```

| Field          | Meaning                               |
| -------------- | ------------------------------------- |
| `status`       | Overall result                        |
| `db_status`    | Whether the database is reachable     |
| `db_time`      | Database response latency, in seconds |
| `redis_status` | Whether Redis is reachable            |
| `redis_time`   | Redis response latency, in seconds    |
| `time`         | Server timestamp                      |

{% hint style="info" %}
**Point your external monitor at this endpoint, not at the front page.** A monitor that fetches the login page proves only that the web server answered. This endpoint proves the database and Redis are reachable — the two dependencies whose loss makes the console useless while still returning a page.
{% endhint %}

The two timing fields are latency in seconds. Alerting only on `db_status: false` catches the outage; trending `db_time` and `redis_time` catches it coming.

**On a cluster**, the installation is fronted by a single virtual address, and each node also answers `/api/health/` directly. Poll both: the virtual address tells you whether the service is up, and the per-node responses tell you whether one node has quietly dropped out while the address still answers from the others. A monitor that watches only the virtual address cannot tell "the cluster is down" from "one node is down", and the second condition is the one that stays unnoticed for weeks.

## Component health in the console

**System Settings › Components › Components** lists every registered microservice with its live state.

| Column              | What it tells you                                                                                                            |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Name**            | Registered component, as `[Type]-ServerName`                                                                                 |
| **Online sessions** | Sessions the component is handling right now. Zero means idle; a persistently high number suggests it needs another instance |
| **Stat**            | CPU and memory use of the component process. Near 100% means the resource is saturated                                       |
| **Load status**     | **Normal**, **Offline**, or **High load**                                                                                    |
| **Remote address**  | Address of the server where the component runs                                                                               |

**Offline is the state that matters.** If the core API component is offline the whole platform is inaccessible. If a protocol connector is offline, sessions of that protocol stop while everything else looks fine — the console still loads, so the failure reads as "RDP is broken" rather than "a component is down".

The source recommends configuring external alerting for offline components through a monitoring system — Zabbix, Prometheus and Datadog are the examples given — rather than relying on someone opening this screen.

## The task engine: workers and queues

All automatic work runs in background tasks: log cleanup, periodic asset checks, account synchronisation, and the Ansible-based automations. The **Monitoring** button at the top of **System Settings › Operations** opens the task dashboard in a new browser tab.

**Workers tab.** Each row is a worker process named `queue@hostname`. Two workers are expected: the default maintenance worker, and the automation (Ansible) worker. Columns are **Worker**, **Status** (Online / Offline), **Active**, **Processed**, **Failed**, **Succeeded**, **Retried**, and **Load Average** (system load over 1, 5 and 15 minutes).

**Tasks tab.** Individual executions: **Name** (the internal task path, in the form `module.submodule.task_name`), **UUID** (click through for the full detail), **State** (`SUCCESS`, `FAILURE`, `PENDING`, `RETRY`), **args** and **kwargs**, **Result**, **Received**, and **Started**.

**Broker tab.** Queue state. Redis is the task broker, with two queues configured: `default` for maintenance and system checks, and `ansible` for the automations that talk to managed assets. Columns are **Queue**, **Messages** (pending), **Unacked** (delivered to a worker but not yet confirmed), **Ready**, **Consumers**, and **Idle since**.

{% hint style="warning" %}
**An offline worker stops every automatic task, silently.** Cleanup, credential rotation, connectivity checks, account synchronisation — all of it. Nothing in the console breaks, so the symptom shows up later as stale data or a rotation that never happened. If a worker is offline, the task service on that server is not running and infrastructure has to restart it.
{% endhint %}

Two numbers are worth watching over time:

* **Failed**, on the Workers tab, together with `FAILURE` states on the Tasks tab. Repeated failures point at asset connectivity, a configuration error, or database inconsistency.
* **Messages**, on the Broker tab. A queue that keeps growing means the workers are not consuming as fast as work arrives — server overload, or a stuck worker. The `ansible` queue is the one that backs up in environments with many managed assets, at automation peaks.

On the **Operations** list itself, two columns do the same job for scheduled work: a **Success/Total** count shown in red flags failing executions, and **Date last publish** should fall inside the task's own cycle. The source gives the concrete test: a task on a 3600-second cycle that has not run in the last two hours means you should go look at the workers.

## Alerts to subscribe to

**System Settings › Notifications › Subscription** binds event types to recipients. Two of the categories are about the infrastructure rather than about user behaviour:

| Message type                       | Fires when                                                                                                                  |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Storage — Connectivity**         | The configured storage backend cannot be reached. This is early warning: it arrives before recordings start failing to save |
| **Component — Server performance** | A component shows high CPU or memory use, or degraded performance                                                           |
| **Sessions — Command reject**      | A command in a live session was blocked by the command policy                                                               |
| **Job — Command reject**           | A scheduled job tried to run a blocked command                                                                              |

Each type has its own recipient list, set through **Edit recipient**. A subscription with no recipient is silent, which is indistinguishable from nothing having happened.

## Storage and retention: the growth to watch

Retention settings, under **System Settings › Retentions**, set how many days each class of record is kept. The nightly cleanup tasks then delete anything older.

{% hint style="danger" %}
**Retention deletes evidence, and nothing tells you it happened.** An investigation into an event older than the retention window returns an empty result, not an error — indistinguishable from "it never happened". Decide the retention values against your audit obligations before you need the evidence, not during the investigation.
{% endhint %}

The second trap is the split between metadata and files:

* The retention values in the console control **records in the database**.
* The lifecycle of the actual **recording files** is whatever the object storage bucket or container's own lifecycle policy says.

They are configured in two different places and do not coordinate. A bucket lifecycle shorter than your session-log retention leaves session records that list a replay which no longer exists; a longer one keeps files after the records that pointed to them are gone. Recording storage backends are managed under **System Settings › Storage › Object Storage**, where exactly one backend is marked **Default**; check which one that is before you reason about where anything is kept. Graphical (RDP) sessions consume far more space than terminal sessions, so storage growth tracks the mix of protocols you use, not the number of sessions.

## Network diagnostics from the server

**System Settings › Utils** runs Ping, Telnet, Nmap, Tcpdump and Traceroute **from the server**, with the result streamed into the Output area. That is the point of it: the result reflects connectivity between the solution and the asset, not between your workstation and the asset — which is the question you actually have when a session will not open.

* **Telnet** is the useful one for a failing asset: it tests a TCP port, where Ping only tests ICMP. Use it before registering a new asset to confirm the protocol port is reachable from the server.
* **Ping** shows resolved address, latency and packet loss, so it doubles as a DNS resolution check from the server.
* **Tcpdump** and **Nmap** are the heavy tools. Always apply source and destination filters to a Tcpdump in production, and treat the captured output as sensitive — it can contain the traffic of managed assets.

{% hint style="warning" %}
**Utils are reconnaissance tools running from inside your network.** The **Tools in the cockpit** switch on the **Basic** tab disables them entirely, for everyone, regardless of role permissions. Where they are enabled, restrict them by role to network and infrastructure profiles.
{% endhint %}

## How to verify your monitoring actually works

Monitoring is the one thing that fails without a symptom, because a monitor that never fires looks exactly like a system that never breaks. Check the monitor, not the system:

1. **Confirm your external check tests the endpoint's content, not just its status code.** Verify the monitor alerts when `db_status` or `redis_status` is `false` — an endpoint returning HTTP 200 with `db_status: false` is a real outage, and a monitor watching only the response code reports it as healthy.
2. **On a cluster, compare the virtual address response with each node's.** If your monitor only has one target, it cannot see a single node dropping out.
3. **Confirm an alert reaches a person.** Open each subscription you rely on and check that **Receiver** is not empty.
4. **Confirm the task engine is alive rather than merely present.** On the Workers tab both workers should be Online, and on the Operations list the **Date last publish** of a short-cycle task should be recent. A worker can be listed and idle.
5. **Confirm the storage path end to end** by opening a recent recording and playing it back. Storage connectivity alerts tell you the backend is unreachable; only a playback tells you writes are landing where reads look.

## Common errors

| Symptom                                                            | Cause                                                                                                        | Fix                                                                                              |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| Monitor reports healthy during a database outage                   | The check tests the login page or the HTTP status code only                                                  | Poll `/api/health/` and alert on `db_status` and `redis_status`                                  |
| The console loads but one protocol cannot open sessions            | That protocol's connector is Offline                                                                         | Check **System Settings › Components**; restart the component's service on the server it runs on |
| Whole platform inaccessible                                        | The core API component is offline                                                                            | Same screen; the core API is the single component whose loss takes everything with it            |
| Rotations, cleanups and connectivity checks all silently stopped   | A task worker is Offline, so no scheduled task runs                                                          | Check the Workers tab from **Operations › Monitoring**; restart the task service                 |
| Queue **Messages** keeps climbing                                  | Workers are not consuming at the rate work arrives, or one is stuck                                          | Check worker status and load; the `ansible` queue backs up first in large environments           |
| Session records exist but replays will not play                    | Recording files were removed by the bucket's own lifecycle policy, which is independent of console retention | Align the bucket lifecycle with your retention requirement                                       |
| An old event cannot be found in the audit trail at all             | Its retention window elapsed and the nightly cleanup deleted the record; no error is raised                  | Set retention values against your audit obligations in advance                                   |
| Alerts configured but never received                               | The message type has no recipient                                                                            | Set **Receiver** per message type under **Notifications › Subscription**                         |
| A diagnostic tool is missing from the console for an administrator | **Tools in the cockpit** is disabled, which overrides role permissions                                       | Re-enable it on the **Utils › Basic** tab, then restrict by role                                 |

## Related pages

* [Audit trail](https://wiki.datadike.com/pam/administration/audit/audit-trail.md)
* [Session recordings](https://wiki.datadike.com/pam/administration/audit/recordings.md)
* [Reports](https://wiki.datadike.com/pam/administration/audit/reports.md)
* [Ports](https://wiki.datadike.com/pam/reference/ports.md)
* [Back up and restore](https://wiki.datadike.com/pam/deploy-operate/backup-and-restore.md)
