mirror of
https://github.com/unshackle-dl/unshackle.git
synced 2026-11-14 21:15:06 +00:00
feat(serve): add operator dashboard endpoints and per-key rate limits
This commit is contained in:
imSp4rky
@@ -205,6 +205,31 @@ When the server rejects a request for authentication reasons, the response has a
|
||||
|
||||
In your client, treat any `401` as an authentication problem and parse *both* shapes, because the standard error body is not always present.
|
||||
|
||||
## Rate limit rejections
|
||||
|
||||
An API key that carries a rate limit gets `429 Too Many Requests` once it goes over, from the
|
||||
same middleware and in the same shape as the `401` bodies above, with a `Retry-After` header in
|
||||
seconds:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": 429,
|
||||
"message": "Rate limit exceeded."
|
||||
}
|
||||
```
|
||||
|
||||
A limit comes from `serve.tiers.<name>.rate_limit` or from the API key's own
|
||||
`serve.users.<key>.rate_limit`, in requests per hour. An API key with neither has no limit, and
|
||||
`GET /api/health` is never limited. See
|
||||
[dashboard rate limits](dashboard.md#rate-limits) for the config and the counters, and
|
||||
[services configuration](../../reference/configuration/services.md) for the keys themselves.
|
||||
|
||||
!!! warning "`429` is not the `RATE_LIMITED` error body"
|
||||
`RATE_LIMITED` in [Errors](errors.md) is a *service* rate limit that a handler raises, and it
|
||||
arrives in the standard envelope with an `error_code`. The `429` here comes from the
|
||||
middleware before any handler runs, and has the integer-`status` shape shown above. A client
|
||||
that reads `error_code` to classify a `429` must cope with it being absent.
|
||||
|
||||
## Access controls on top of the API key
|
||||
|
||||
After the server authenticates a request, several additional controls decide what that specific API key can do.
|
||||
@@ -283,9 +308,11 @@ With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located
|
||||
| Query-parameter alternative | `?secret_key=` on `GET /api/download/jobs/{job_id}/events` only |
|
||||
| Default bind address | `127.0.0.1:8786` |
|
||||
| Exempt endpoint | `GET /api/health` |
|
||||
| Failure status | `401 Unauthorized` |
|
||||
| Failure status | `401 Unauthorized`, or `429 Too Many Requests` for a key over its rate limit |
|
||||
| Missing-header body | `{"status": 401, "message": "Secret Key is Empty."}` |
|
||||
| Invalid-key body | `{"status": 401, "message": "Secret Key is Invalid."}` |
|
||||
| Over-limit body | `{"status": 429, "message": "Rate limit exceeded."}`, plus a `Retry-After` header |
|
||||
| Rate limit config | `serve.tiers.<name>.rate_limit`, overridden by `serve.users.<key>.rate_limit` |
|
||||
| Disable auth | `--no-key` (all requests allowed) |
|
||||
| Config location | `serve.api_secret` and `serve.users` in `unshackle.yaml` |
|
||||
|
||||
|
||||
@@ -43,10 +43,14 @@ All are `GET`.
|
||||
|
||||
| Path | Returns |
|
||||
|------|---------|
|
||||
| `/api/dashboard/status` | Version, `code_hash`, bind host and port, `mode` (`full`, `api_only`, `remote_only`), uptime, `requests_total`, `requests_rejected` (401s), `requests_by_key` (label → count), loaded service count, session count, `max_sessions`, `session_ttl`, job counts by status. |
|
||||
| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the first resolved title's display name), `auth_status`, `auth_error`, `server_account` (the server profile lent to the remote session, else `null`), `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`. `client` is whatever the remote client sent as `client` in its session create request (the current CLI sends nothing, so it is `{}`). `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. |
|
||||
| `/api/dashboard/status` | Version, `code_hash`, bind host and port, `mode` (`full`, `api_only`, `remote_only`), uptime, `requests_total`, `requests_rejected` (401s), `requests_by_key` (label → count), loaded service count, session count, `max_sessions` (`null` when the operator configured no limit), `session_ttl`, job counts by status. The service count includes services that failed to import - use `/api/dashboard/services` for the number that loaded. |
|
||||
| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the first resolved title's display name), `auth_status`, `auth_error`, `server_account` (the server profile lent to the remote session, else `null`), `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`, `log_seq`. `client` is whatever the remote client sent as `client` in its session create request: the CLI sends `version` and `platform`, an older client sends nothing and the field is `{}`. `log_seq` is the last sequence number in the remote session's service log - poll `/api/dashboard/sessions/{id}/logs` when it changes. `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. |
|
||||
| `/api/dashboard/jobs` | Every download job with full detail, regardless of owner. Empty in `--remote-only` mode. |
|
||||
| `/api/dashboard/logs` | `{"seq": N, "records": [...]}`: the last 1000 log records. `?since=<seq>` returns only newer records; `?level=WARNING` sets the minimum level; `?logger=serve` keeps one logger and its children (`aiohttp.access` is the noisiest). `ts` is a Unix epoch in seconds. |
|
||||
| `/api/dashboard/sessions/{id}/logs` | One remote session's service log: `{"session_id", "last_seq", "records": [...]}`. Each record has `seq`, `ts`, `level`, `message`. `?since=<seq>` returns only newer records. 404 when the remote session is unknown. |
|
||||
| `/api/dashboard/keys` | Every configured API key: what it may do and what it has done. One row per key in `serve.users`, plus `serve.api_secret` when it is set. |
|
||||
| `/api/dashboard/services` | Every discovered service and its load state, including the ones that failed to import. Not filtered by any allowlist. |
|
||||
| `/api/dashboard/health` | Preflight: whether this instance could finish a download. Cached 30 seconds. |
|
||||
| `/api/dashboard/events` | Server-Sent Events stream. With `?since=<seq>` it returns `{"seq", "stats", "events": [...]}` instead: every event after `seq` (the server keeps the last 20000, log records included, so a poll every few seconds is safe even under a busy access log) as one JSON burst, so a UI that polls every few seconds still sees a remote session that opened and closed in between. |
|
||||
|
||||
`requests_rejected` counts 401 responses; those never appear in `requests_by_key`. The
|
||||
@@ -64,6 +68,7 @@ those it sends:
|
||||
| `log` | One log record: `seq`, `ts`, `level`, `logger`, `msg`. |
|
||||
| `session` | `action` (`create`, `update` on every request the session makes and when titles or tracks load, or `delete`) plus the full remote session summary, so each event is a complete snapshot. A `delete` also carries `reason`: `closed` (the client ended it), `expired` (TTL sweep) or `evicted` (`max_sessions` reached). A session opened by `--skip-dl` can live only a few seconds, so a UI that wants history must keep its own record of these events. |
|
||||
| `job` | `event` (`queued`, `progress`, `status`, or the terminal status) plus the full job object. |
|
||||
| `service` | `action` (`staged`, `applied` or `failed`) and `tags`, the services it applies to. A `failed` action carries `errors` and an empty `tags`. Refetch `/api/dashboard/services` when one arrives, instead of polling for a reload. |
|
||||
|
||||
Every frame carries an `id:` (the event seq). After a dropped connection an `EventSource`
|
||||
reconnects with `Last-Event-ID` and the server replays the events it missed. The connection
|
||||
@@ -80,3 +85,160 @@ es.addEventListener("log", (e) => console.log(JSON.parse(e.data).msg));
|
||||
|
||||
CORS is open (`Access-Control-Allow-Origin: *`), so a dashboard page served from another
|
||||
origin works without a proxy.
|
||||
|
||||
## Remote session logs
|
||||
|
||||
A remote session mirrors the service's own `self.log` into a 500-record buffer at `INFO`,
|
||||
whatever level the server prints at. The remote session summary carries only a truncated
|
||||
`auth_error`; the buffer holds the real reason a login failed.
|
||||
|
||||
```
|
||||
GET /api/dashboard/sessions/{session_id}/logs?since=12
|
||||
{"session_id": "...", "last_seq": 14,
|
||||
"records": [{"seq": 13, "ts": 1756909254.1, "level": "ERROR", "message": "login rejected: MFA required"}]}
|
||||
```
|
||||
|
||||
The field name is `records`, the same one `/api/dashboard/logs` uses, so one dashboard reads
|
||||
both logs the same way. Each record has the shape the client route `/api/session/{id}/logs`
|
||||
returns. `last_seq` is the buffer's own last sequence number, so an empty page still reports
|
||||
the true value.
|
||||
|
||||
Reading is a cursor read, not a drain: the client draining the same buffer still gets every
|
||||
record. It also does not refresh the remote session's idle timer, so polling a remote session here
|
||||
never keeps it alive or makes it look active.
|
||||
|
||||
!!! note "`log_seq` is a poll trigger, not a push"
|
||||
A remote session summary reaches the dashboard only when something publishes a `session`
|
||||
event - a request the remote session makes, or an auth transition. A log line on its own
|
||||
publishes nothing. Poll this route when `log_seq` changes, and on a timer while a drawer
|
||||
is open.
|
||||
|
||||
## Keys
|
||||
|
||||
```
|
||||
GET /api/dashboard/keys
|
||||
[{"id": "3ed054be9371", "role": "user", "label": "tier1", "services": ["ATV", "DSNP"],
|
||||
"server_cdm": false, "server_accounts": false, "server_proxy": false,
|
||||
"tier": "bot", "rate_limit": 600, "window_used": 412,
|
||||
"requests": 41300, "rejected": 4, "bytes_out": 103079215104, "last_seen": 1756909188.2},
|
||||
{"id": "7c1d0f52a884", "role": "dashboard", "label": "dash…", "services": [],
|
||||
"server_cdm": false, "server_accounts": false, "server_proxy": false,
|
||||
"tier": null, "rate_limit": null, "window_used": 0,
|
||||
"requests": 8140, "rejected": 0, "bytes_out": 41200311, "last_seen": 1756909350.9}]
|
||||
```
|
||||
|
||||
`id` is a hash of the API key, so it stays stable across restarts and carries no key material.
|
||||
Two API keys with no `username` merge into one label in `requests_by_key` but stay separate
|
||||
here, so use `id` as the identity and `label` only for display.
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `role` | Which key this is: `user` (a `serve.users` entry), `admin` (`serve.api_secret`) or `dashboard` (`serve.dashboard.key`). Identity, not capability - read the grant fields for what the API key may do. |
|
||||
| `services` | The effective allowlist, the global list intersected with the API key's. `null` means unrestricted; `[]` means the API key reaches no service route at all. |
|
||||
| `server_cdm`, `server_accounts` | `false`, `true`, or the list of service tags the grant covers. |
|
||||
| `server_proxy` | Boolean. Only a literal `true` grants it, and an API key with no `users` entry does not get it. |
|
||||
| `rate_limit` | Requests per hour, from the API key's own value or its tier's. `null` means no limit. |
|
||||
| `window_used` | Requests counted in the current hourly window. |
|
||||
| `rejected` | Responses of 400 or worse that the server attributes to this API key, 429s included. The global `requests_rejected` stays 401-only. |
|
||||
| `bytes_out` | Response bodies only. An event stream reports no body length, so it counts as zero. |
|
||||
| `last_seen` | `null` until the API key makes its first request. |
|
||||
|
||||
Only a configured API key gets a row of counters. A request on an open route such as `/api/health` with an unknown key counts as `anonymous`, so an unauthenticated caller cannot grow the table.
|
||||
|
||||
Every API key the server counts has a row here, the dashboard key included, so a
|
||||
`requests_by_key` bucket other than `anonymous` always matches one. A "traffic by key" panel
|
||||
reads the rows here, which are unambiguous, and takes `anonymous` from `requests_by_key`. Read
|
||||
`role` before you draw the bars: the `dashboard` row is the panel measuring its own polling.
|
||||
|
||||
The dashboard key gets `services: []` and no grants unless the operator also gave it a
|
||||
`serve.users` entry, because it reaches nothing outside `/api/dashboard/`. An API key listed
|
||||
in more than one place still gets exactly one row.
|
||||
|
||||
### Rate limits
|
||||
|
||||
An API key over its limit gets `429` with a `Retry-After` header:
|
||||
|
||||
```yaml
|
||||
serve:
|
||||
tiers:
|
||||
bot: { rate_limit: 600 } # requests per hour
|
||||
users:
|
||||
"<api key>":
|
||||
username: bot-relay
|
||||
tier: bot
|
||||
# rate_limit: 900 # optional, wins over the tier
|
||||
```
|
||||
|
||||
The server keeps a fixed window, not a sliding one: it opens on the first request and resets
|
||||
an hour later. An API key with no tier and no override has no limit. A `tier` that names no
|
||||
entry under `serve.tiers`, or a `rate_limit` that is not a positive whole number, stops the
|
||||
server at startup, because the alternative is an API key that silently gets no limit. The rate
|
||||
limit never applies to the dashboard key, and never to `/api/health`.
|
||||
|
||||
## Services
|
||||
|
||||
```
|
||||
GET /api/dashboard/services
|
||||
[{"tag": "MAX", "state": "staged", "error": null,
|
||||
"commit": "9cc230c98f13b83f6c6fe831595b9702d3ba2786", "staged_commit": "5f0aa17...",
|
||||
"staged_since": 1756908900.0, "sessions": 1, "jobs": 1,
|
||||
"aliases": ["HBO"], "geofence": ["US"]}]
|
||||
```
|
||||
|
||||
`state` is `loaded`, `staged` or `failed`. A service that failed to import keeps its error
|
||||
here instead of disappearing, and `/status` still counts it in the service total - count the
|
||||
`state` values here for the number that loaded.
|
||||
|
||||
A `staged` service has an update on disk that a busy service blocks from the import: the repo
|
||||
pull already happened, so `commit` (what runs now) and `staged_commit` (what waits) differ. It swaps in when the last job for that tag finishes. Both commits are `null`
|
||||
for a service from a plain local directory rather than a git repo.
|
||||
|
||||
Watch the `service` event rather than polling for the swap.
|
||||
|
||||
## Health
|
||||
|
||||
```
|
||||
GET /api/dashboard/health
|
||||
{"generated_at": 1756909350.4, "status": "ok",
|
||||
"checks": [{"id": "ffmpeg", "label": "ffmpeg", "status": "ok",
|
||||
"detail": "7.1 · /usr/bin/ffmpeg", "ms": 4.1}]}
|
||||
```
|
||||
|
||||
An instance can answer every request and still fail every download, because `mp4decrypt` is
|
||||
missing or a vault's credentials expired. This finds that before a job does.
|
||||
|
||||
`status` is `ok`, `degraded` (an optional dependency is missing) or `failing` (a required one
|
||||
is). Player binaries are not checked: a headless server never has one and it plays nothing.
|
||||
Each check is `ok`, `warn` or `fail`. Check ids are the binary names, `cdm`,
|
||||
`vault:<name>` (the vault's configured `name`, so two vaults of the same type stay apart), and
|
||||
`proxies`.
|
||||
|
||||
The server caches the result for 30 seconds and keeps every probe shallow: a vault answers one
|
||||
lookup that should miss, and no probe allocates anything. A failing probe's `detail` has the
|
||||
vault's password or token masked. This is a panel, not a liveness probe - do not poll it faster
|
||||
than the cache: every refresh that the cache does not answer costs one real lookup against each
|
||||
configured vault, which a vault that meters or rate limits its callers will count.
|
||||
|
||||
!!! warning "An unreachable vault costs a slow, partial answer"
|
||||
The probes run one after another, so a cold read takes as long as the probes that finish
|
||||
inside the deadline. The endpoint gives up after 15 seconds and answers with the checks that
|
||||
did complete, plus one `probe` check with the status `fail` naming how many finished. A
|
||||
database vault that accepts the connection and then stalls is what this deadline is for: its
|
||||
driver applies no timeout of its own, so without one the request would never answer.
|
||||
|
||||
The probe that timed out keeps running in its thread until its driver returns, and the
|
||||
server caches the partial answer for the full 30 seconds, so callers cannot stack up more
|
||||
of them. A second caller waits behind the first, because the server shares one refresh
|
||||
rather than running it twice.
|
||||
|
||||
!!! warning "Proxy providers are only checked at construction"
|
||||
The proxy check builds each proxy provider but never asks it for a proxy, because for some
|
||||
proxy providers that allocates a connection. A proxy provider that only fails on its first
|
||||
real use still reads `ok`.
|
||||
|
||||
## Counters reset on restart
|
||||
|
||||
Stats, the log ring and the event history are all in memory. A restart drops every number to
|
||||
zero. `started_at` in the `stats` frame (with `version` and `code_hash`) is the signal: it
|
||||
arrives in the first frame after a reconnect, so a dashboard can tell a restart from a bug
|
||||
without any extra field.
|
||||
|
||||
@@ -19,7 +19,7 @@ Start the server with [`unshackle serve`](../../guide/cli-reference.md). By defa
|
||||
- **Service allowlist.** The effective allowlist for your API key filters the service-facing endpoints (the intersection of the global `serve.services` list and your per-key list). The server treats services you cannot use as unknown.
|
||||
|
||||
!!! warning "Error responses have two shapes"
|
||||
Structured application errors (raised inside a handler) return `{"status": "error", "error_code": "...", "message": "...", "timestamp": "..."}`. The authentication middleware, however, returns a **different** shape: `{"status": 401, "message": "..."}` with an integer `status` and no `error_code`. Clients must accept both shapes. The [Error responses](#error-responses) section at the bottom of this page gives the full error model.
|
||||
Structured application errors (raised inside a handler) return `{"status": "error", "error_code": "...", "message": "...", "timestamp": "..."}`. The authentication middleware, however, returns a **different** shape: `{"status": 401, "message": "..."}` with an integer `status` and no `error_code`. The same middleware answers `429` in that shape when an API key goes over its rate limit. Clients must accept both shapes. The [Error responses](#error-responses) section at the bottom of this page gives the full error model.
|
||||
|
||||
---
|
||||
|
||||
@@ -1141,7 +1141,7 @@ The server serialises every structured error raised inside a handler to the same
|
||||
- `debug_info` appears only when you start the server with `--debug-api`.
|
||||
|
||||
!!! warning "Authentication errors differ"
|
||||
Errors raised by the API-key middleware use a distinct shape (`{"status": 401, "message": "Secret Key is Empty."}` or `{"status": 401, "message": "Secret Key is Invalid."}`) with an integer `status` and no `error_code`, `timestamp`, or `details`. A malformed JSON body on any POST, by contrast, returns the standard envelope with `error_code: "INVALID_INPUT"`.
|
||||
Errors raised by the API-key middleware use a distinct shape (`{"status": 401, "message": "Secret Key is Empty."}`, `{"status": 401, "message": "Secret Key is Invalid."}`, or `{"status": 429, "message": "Rate limit exceeded."}` with a `Retry-After` header) with an integer `status` and no `error_code`, `timestamp`, or `details`. A `429` from the middleware therefore carries no `error_code`, unlike the `RATE_LIMITED` code below, which a handler raises for a rate limit the *service* imposed. A malformed JSON body on any POST, by contrast, returns the standard envelope with `error_code: "INVALID_INPUT"`.
|
||||
|
||||
### Error codes and default status
|
||||
|
||||
|
||||
@@ -109,6 +109,15 @@ but the defaults hold for the overwhelming majority of responses.
|
||||
| `CONFLICT` | 409 | The target resource is in a state that disallows the action (retrying a running job, prioritising a non-queued job, clearing cache during an active download). |
|
||||
| `RATE_LIMITED` | 429 | The streaming service is rate-limiting requests. Retryable. |
|
||||
|
||||
!!! warning "Not every `429` is `RATE_LIMITED`"
|
||||
The server has a rate limit of its own. An API key over `serve.tiers.<name>.rate_limit`
|
||||
or `serve.users.<key>.rate_limit` gets a `429` from the authentication middleware, before
|
||||
any handler runs, so it carries no `error_code` and is not the `RATE_LIMITED` code above.
|
||||
It is the shape `{"status": 429, "message": "Rate limit exceeded."}` with a `Retry-After`
|
||||
header. Read `Retry-After` and wait it out rather than retrying on a backoff of your own:
|
||||
the window is fixed, so an early retry only spends the next window. See
|
||||
[Authentication](authentication.md#rate-limit-rejections).
|
||||
|
||||
### Server errors (5xx)
|
||||
|
||||
| Code | Status | Meaning |
|
||||
|
||||
Reference in New Issue
Block a user