feat(api): add job retry/priority/clear, history, config and maintenance endpoints

Persistent job history (capped by serve.history_limit, stored under cache), profiles/config reads, maintenance + env-check endpoints, and 409 conflict code.
2026-11-13 12:35:05 +00:00 · 2026-07-01 22:42:48 -06:00
parent f69ebb4fba
commit 7fdd514e18
8 changed files with 1345 additions and 61 deletions
@@ -7,6 +7,10 @@ queued -> cancelled
 downloading -> cancelled
 ```

+## Progress reporting
+
+A multi-episode request (`wanted: ["S01E01", ...]`) is a single job that processes episodes sequentially. During a run the job status updates live: `current_title` names the episode downloading now, `output_files` grows as each episode finishes muxing, and `track_progress` lists per-track progress when `downloads` > 1 runs tracks concurrently. `progress` and track counts reset as each episode starts; use `output_files` against `parameters.wanted` to derive overall completion.
+
 Jobs are retained for 24 hours after completion (override via top-level `download_job_retention_hours` in `unshackle.yaml`). The server runs up to 2 concurrent download jobs by default; override via top-level `max_concurrent_downloads`. This is independent of `serve.downloads`, which controls parallel tracks **within** a single job.

 Remote sessions are managed by `SessionStore` (`unshackle/core/api/session_store.py`); idle sessions and their `InputBridge` instances are cleaned up by a background loop started/stopped with the app lifecycle.
@@ -242,6 +242,7 @@ List all download jobs with optional filtering and sorting. Disabled in `--remot
 | `service` | string | all | Filter by service tag |
 | `sort_by` | string | `created_time` | Sort field: `created_time`, `started_time`, `completed_time`, `progress`, `status`, `service` |
 | `sort_order` | string | `desc` | Sort order: `asc`, `desc` |
+| `full` | string | `false` | `true` to include full job details per job (parameters, timestamps, `output_files`, error fields) |

 ```bash
 curl -H "X-Secret-Key: $KEY" "http://localhost:8786/api/download/jobs?status=completed"
@@ -256,22 +257,263 @@ Get detailed information about a specific download job including progress, param
 ```json
 {
  "job_id": "504db959-80b0-446c-a764-7924b761d613",
-  "status": "completed",
+  "status": "downloading",
  "created_time": "2026-02-27T18:00:00.000000",
  "service": "EXAMPLE1",
  "title_id": "abc123def456",
-  "progress": 100.0,
+  "title": "Example Show",
+  "progress": 74.9,
+  "phase": "downloading video 1080p SDR",
+  "current_title": "S01E04",
+  "completed_tracks": 4,
+  "total_tracks": 13,
+  "active_tracks": ["video 1080p SDR"],
+  "track_progress": [
+    { "label": "video 1080p SDR", "progress": 74.9, "speed": "DASH 265.5 MB/s" }
+  ],
+  "segments_done": 75.0,
+  "segments_total": 100.0,
+  "speed": "DASH 265.5 MB/s",
  "parameters": { },
  "started_time": "2026-02-27T18:00:01.000000",
-  "completed_time": "2026-02-27T18:00:15.000000",
-  "output_files": [],
+  "completed_time": null,
+  "output_files": [
+    "/downloads/Example.Show.S01E01.mkv",
+    "/downloads/Example.Show.S01E02.mkv",
+    "/downloads/Example.Show.S01E03.mkv"
+  ],
  "error_message": null,
  "error_details": null
 }
 ```

+**Progress fields:**
+
+| Field | Type | Description |
+| --- | --- | --- |
+| `progress` | number | Overall progress for the title currently downloading, 0–100. Resets per episode on multi-episode jobs |
+| `phase` | string | Human-readable current step, e.g. `downloading video 1080p SDR`, `muxing` |
+| `title` | string | Human-readable show/movie name, available once the download starts |
+| `current_title` | string | `SxxEyy` code of the episode downloading now (episodic jobs only) |
+| `completed_tracks` / `total_tracks` | int | Track counts for the current title |
+| `active_tracks` | array[string] | Labels of tracks downloading now |
+| `track_progress` | array | Per-track `{label, progress, speed}` for each concurrently active track |
+| `segments_done` / `segments_total` | number | Segment counts of the track downloading now |
+| `speed` | string | Transfer speed of the track downloading now |
+| `output_files` | array[string] | Final file paths, appended as each title finishes muxing |
+
+The list endpoint (`GET /api/download/jobs`) returns the same progress fields per job but omits `parameters`, timestamps, `output_files`, and error details.
+
 ---

 ## DELETE /api/download/jobs/{job_id}

-Cancel a queued or running download job. Returns 400 if the job has already terminated.
+Cancel a queued or running download job, or remove a finished one.
+
+- `queued` / `downloading` job: cancelled, returns `200` with `{"status": "success", "message": "Job cancelled"}`.
+- `completed` / `failed` / `cancelled` job: removed from the manager entirely, returns `204` (no body).
+- Unknown job: `404`.
+
+---
+
+## POST /api/download/jobs/clear-finished
+
+Remove all `completed`, `failed`, and `cancelled` jobs from the manager. Disabled in `--remote-only` mode.
+
+```json
+{ "removed": 3 }
+```
+
+---
+
+## POST /api/download/jobs/{job_id}/retry
+
+Enqueue a new job reusing a finished job's service, title, and original (unredacted) parameters. Only allowed on `completed`, `failed`, or `cancelled` jobs — returns `409` otherwise, `404` if unknown. The original job is left in place.
+
+Creation-time authorization is re-applied at retry time: the job's service must pass the caller's service allowlist (otherwise `400` invalid service), and jobs whose original parameters carried a `cdm` override or per-job credentials return `403` if `serve.cdm_overrides` / `serve.allow_job_credentials` no longer permit them.
+
+Returns `202` with the same shape as `POST /api/download`:
+
+```json
+{
+  "job_id": "b7e9c1f2-...",
+  "status": "queued",
+  "created_time": "2026-02-27T18:00:00.000000"
+}
+```
+
+---
+
+## POST /api/download/jobs/{job_id}/priority
+
+Move a `queued` job to the front of the download queue so a worker picks it up next. Returns `409` if the job is not queued, `404` if unknown.
+
+```json
+{ "job_id": "504db959-...", "position": "front" }
+```
+
+---
+
+## GET /api/profiles
+
+List named credential profiles per service, usable as the `profile` parameter on search/list/download. Only services whose credentials are configured as a `profile-name: credential` mapping appear (a `default` key is included if present); a service with a single plain (unnamed) credential is omitted entirely. Filtered by the caller's service allowlist. Disabled in `--remote-only` mode.
+
+```bash
+curl -H "X-Secret-Key: $KEY" http://localhost:8786/api/profiles
+```
+
+```json
+{ "profiles": { "EXAMPLE1": ["default", "uk", "us"], "EXAMPLE2": ["alice"] } }
+```
+
+---
+
+## GET /api/config
+
+Read-only, redacted view of the effective server configuration for a UI settings page. Secrets (`api_secret`, `users`, credentials, tokens) are never included; secret-looking keys inside `dl` are masked with `***`. Disabled in `--remote-only` mode.
+
+```bash
+curl -H "X-Secret-Key: $KEY" http://localhost:8786/api/config
+```
+
+```json
+{
+  "config": {
+    "dl": { "lang": "orig", "sub_format": "srt", "downloads": 12 },
+    "serve": {
+      "max_concurrent_downloads": 2,
+      "job_retention_hours": 24,
+      "history_limit": 100,
+      "services": null,
+      "remote_only": false,
+      "cdm_overrides": null,
+      "allow_job_credentials": false
+    },
+    "directories": {
+      "downloads": "/srv/downloads",
+      "temp": "/srv/temp",
+      "cache": "/srv/cache"
+    },
+    "services": ["EXAMPLE1", "EXAMPLE2"]
+  }
+}
+```
+
+| Field | Description |
+| --- | --- |
+| `dl` | Default `dl` parameters from the config file (redacted) |
+| `serve.services` | Global service allowlist, or `null` when all services are allowed |
+| `serve.cdm_overrides` | Permitted per-request CDM device names (list), `true`, or `null` |
+| `services` | Available service tags, filtered by the caller's allowlist |
+
+---
+
+## GET /api/history
+
+Persistent download history. Every job that reaches a terminal state (`completed`, `failed`, `cancelled`) is appended as one JSON line to `api_history.jsonl` in the cache directory, surviving restarts and job cleanup. Only the newest `serve.history_limit` entries are retained (default `100`, `0` for unlimited; see Advanced Config). Newest first. Corrupt lines are skipped; a missing file yields an empty list. Entries are filtered by the caller's service allowlist. Disabled in `--remote-only` mode.
+
+**Query parameters:**
+| Parameter | Type | Default | Description |
+| --- | --- | --- | --- |
+| `limit` | int | `100` | Maximum entries returned (must be >= 1) |
+| `service` | string | all | Filter by service tag (case-insensitive) |
+
+```bash
+curl -H "X-Secret-Key: $KEY" "http://localhost:8786/api/history?limit=50&service=EXAMPLE1"
+```
+
+```json
+{
+  "history": [
+    {
+      "job_id": "504db959-80b0-446c-a764-7924b761d613",
+      "service": "EXAMPLE1",
+      "title_id": "abc123def456",
+      "title": "Example Show",
+      "status": "completed",
+      "created_time": "2026-02-27T18:00:00.000000",
+      "completed_time": "2026-02-27T18:12:34.000000",
+      "output_files": ["/downloads/Example.Show.S01E01.mkv"],
+      "parameters": { "wanted": ["S01E01"], "quality": [1080], "skip_dl": false },
+      "error_message": null
+    }
+  ],
+  "count": 1
+}
+```
+
+`error_message` is redacted with the same secret-scrubbing applied to job responses. `parameters` is the redacted download selection the job ran with (quality/vcodec/range/wanted/skip_dl, etc.). `completed_time` is `null` for jobs cancelled while still queued.
+
+---
+
+## DELETE /api/history/{job_id}
+
+Remove a single history entry by `job_id` (the file is rewritten without it). Returns `204` on success, `404` if no matching entry exists or it falls outside the caller's service allowlist. Corrupt lines in the history file are preserved.
+
+```bash
+curl -X DELETE -H "X-Secret-Key: $KEY" "http://localhost:8786/api/history/504db959-80b0-446c-a764-7924b761d613"
+```
+
+---
+
+## POST /api/maintenance/clear-cache
+
+Delete the contents of the cache directory (the directory is recreated empty). `freed_bytes` is best effort. This also removes `api_history.jsonl`, so persistent download history is cleared. Returns `409` with the active job ids if any job is currently downloading, since in-flight jobs may be reading the cache. Disabled in `--remote-only` mode.
+
+```bash
+curl -X POST -H "X-Secret-Key: $KEY" http://localhost:8786/api/maintenance/clear-cache
+```
+
+```json
+{ "cleared": true, "freed_bytes": 1234567 }
+```
+
+---
+
+## POST /api/maintenance/clear-temp
+
+Same as clear-cache, for the temp directory. Also refuses with `409` while any job is downloading — active workers write segment files under temp.
+
+```json
+{ "cleared": true, "freed_bytes": 0 }
+```
+
+---
+
+## POST /api/maintenance/refresh-services
+
+Force-sync (git pull, hard reset to upstream) every service repo configured in `directories.services`. `refreshed` is `true` when all repos synced (or none are configured). Disabled in `--remote-only` mode.
+
+```bash
+curl -X POST -H "X-Secret-Key: $KEY" http://localhost:8786/api/maintenance/refresh-services
+```
+
+```json
+{
+  "refreshed": true,
+  "repos": [
+    { "spec": "https://github.com/example/services", "updated": true, "changes": ["~ EXAMPLE1 (modified)"] }
+  ]
+}
+```
+
+An empty `changes` array means the repo was already up to date. A repo that failed to sync has `updated: false` (and flips `refreshed` to `false`).
+
+---
+
+## GET /api/env/check
+
+Install status of the binaries `unshackle env check` inspects (FFmpeg, MKVToolNix, shaka-packager, etc.). `version` is a best-effort probe and may be `null` even when installed. Disabled in `--remote-only` mode.
+
+```bash
+curl -H "X-Secret-Key: $KEY" http://localhost:8786/api/env/check
+```
+
+```json
+{
+  "checks": [
+    { "name": "FFmpeg", "installed": true, "version": "7.1.0", "required": true },
+    { "name": "mp4decrypt", "installed": false, "version": null, "required": false }
+  ]
+}
+```