feat(vaults): tolerate vault failures during key get/add

Wrap vault get_key/add_key/add_keys calls in broad exception handlers so a single failing vault (network, auth, driver error) no longer aborts the operation - other vaults are still consulted/written. Failure cause is logged at WARNING so issues remain debuggable.

Inspired by unshackle-dl/unshackle#104 by @CodeName393.

Co-authored-by: CodeName393 <62503817+CodeName393@users.noreply.github.com>

docs/ADVANCED_CONFIG.md

@@ -46,6 +46,22 @@ unshackle serve --remote-only            # Only expose remote service session en
   - `username` - Internal logging name for the user (not visible to users)
   - `services` - Optional per-user service allowlist. Effective access is the intersection of global and per-user allowlists.
 
+#### Server-side `dl` defaults
+
+Any key accepted by `/api/download` (see `docs/API.md`) can also be declared directly under `serve:` and the REST API will treat it as a default. Per-request bodies still win. Use this to raise concurrency, force `best_available`, etc. without each client repeating the values:
+
+```yaml
+serve:
+  api_secret: "..."
+  users: { ... }
+  downloads: 4              # parallel tracks per job
+  workers: 16               # threads per track
+  best_available: true
+  no_proxy_download: false
+```
+
+Layering: built-in defaults < `serve.*` overrides < service-specific defaults < request body.
+
 For example,
 
 ```yaml
@@ -88,6 +104,26 @@ See [API.md](API.md) for full REST API documentation with endpoints, parameters,
 
 ---
 
+## max_concurrent_downloads (int)
+
+Maximum number of `/api/download` jobs the serve queue manager will execute in parallel. Each job runs the full `dl` pipeline (authenticate, fetch tracks, decrypt, mux) in its own worker subprocess. This is independent of `serve.downloads`, which controls parallel tracks **inside** a single job. Default: `2`.
+
+```yaml
+max_concurrent_downloads: 4
+```
+
+---
+
+## download_job_retention_hours (int)
+
+How long completed, failed, or cancelled download jobs remain queryable via `/api/download/jobs/{job_id}` before the periodic cleanup loop drops them. Default: `24`.
+
+```yaml
+download_job_retention_hours: 48
+```
+
+---
+
 ## debug (bool)
 
 Enables comprehensive debug logging. Default: `false`

docs/API.md

@@ -77,6 +77,22 @@ There is no separate "tier" flag. Whether the server can return KID:KEY for a se
 
 Per-service CDM type can be pinned via `config.cdm` (`widevine`/`playready`) or per-service `cdm_type`; otherwise the server picks the type the user has devices for.
 
+### Server-side `dl` defaults
+
+Any flag accepted by `/api/download` (see the table below) can be declared under `serve:` in `unshackle.yaml` and the API will apply it as a default. Request-body values still win. Useful for raising concurrency without changing every client call:
+
+```yaml
+serve:
+  api_secret: "..."
+  users: { ... }
+  downloads: 4        # parallel tracks per download job
+  workers: 16         # threads per track segment fetch
+  best_available: true
+  no_proxy_download: false
+```
+
+Layering order: built-in defaults < `serve.*` overrides < service-specific click defaults < request body.
+
 ---
 
 ## Endpoint Map
@@ -300,10 +316,12 @@ Start a download job. Returns immediately with a job ID (HTTP 202). Disabled in
 | `profile` | string | `null` | Profile for credentials/cookies |
 | `proxy` | string | `null` | Proxy URI or country code |
 | `no_proxy` | boolean | `false` | Disable all proxy use |
+| `no_proxy_download` | boolean | `false` | Bypass proxy for segment downloads only. Manifest, license, and auth still use proxy |
 | `workers` | int | `null` | Max threads per track download |
 | `downloads` | int | `1` | Concurrent track downloads |
-| `slow` | boolean | `false` | Add 60-120s delay between titles |
+| `slow` | boolean or string | `null` | Add randomized delay between titles. `true` = 60-120s, or `"MIN-MAX"` string (e.g., `"20-40"`). Min must be >= 20 |
 | `best_available` | boolean | `false` | Continue if requested quality unavailable |
+| `worst` | boolean | `false` | Select the lowest bitrate track within the specified quality. Requires `quality` |
 | `skip_dl` | boolean | `false` | Skip download, only get decryption keys |
 | `export` | boolean | `false` | Export manifest, track URLs, keys, and subtitles to JSON in the exports directory |
 | `cdm_only` | boolean | `null` | Only use CDM (`true`) or only vaults (`false`) |
@@ -758,6 +776,6 @@ queued -> cancelled
 downloading -> cancelled
 ```
 
-Jobs are retained for 24 hours after completion. The server supports up to 2 concurrent downloads by default.
+Jobs are retained for 24 hours after completion (override via top-level `download_job_retention_hours` in `unshackle.yaml`). The server runs up to 2 concurrent download jobs by default; override via top-level `max_concurrent_downloads`. This is independent of `serve.downloads`, which controls parallel tracks **within** a single job.
 
 Remote sessions are managed by `SessionStore` (`unshackle/core/api/session_store.py`); idle sessions and their `InputBridge` instances are cleaned up by a background loop started/stopped with the app lifecycle.

docs/DOWNLOAD_CONFIG.md

@@ -102,6 +102,7 @@ to a CLI option on the `dl` command. CLI arguments always take priority over con
 | `require_subs` | list | `[]` | Required subtitle languages (skip title if missing) |
 | `forced_subs` | bool | `false` | Include forced subtitle tracks |
 | `exact_lang` | bool | `false` | Exact language matching (no regional variants) |
+| `latest_episode` | bool | `false` | Download only the single most recent episode of a series |
 
 **Track selection:**
 
@@ -147,6 +148,7 @@ to a CLI option on the `dl` command. CLI arguments always take priority over con
 | `downloads` | int | `1` | Concurrent track downloads |
 | `workers` | int | `min(16, cpu_count + 4)` | Max threads per track download (segments / ranged parts) |
 | `slow` | bool or `MIN-MAX` | `false` | Randomized delay between titles. `true` uses 60-120s; pass `MIN-MAX` (e.g., `20-40`) for a custom range |
+| `no_proxy_download` | bool | `false` | Bypass proxy for segment downloads only. Manifest, license, and auth still use proxy |
 | `skip_dl` | bool | `false` | Skip download, only get decryption keys |
 | `cdm_only` | bool | `null` | Only use CDM (`true`) or only vaults (`false`) |
 

unshackle/core/api/download_manager.py

@@ -150,6 +150,21 @@ def _perform_download(
     if params.get("export"):
         params["export"] = bool(params["export"])
 
+    # Normalize slow: accept string "MIN-MAX", list/tuple, or True (default 60-120)
+    slow_raw = params.get("slow")
+    if slow_raw is not None and not isinstance(slow_raw, tuple):
+        if isinstance(slow_raw, bool):
+            params["slow"] = (60, 120) if slow_raw else None
+        elif isinstance(slow_raw, list) and len(slow_raw) == 2:
+            params["slow"] = (int(slow_raw[0]), int(slow_raw[1]))
+        elif isinstance(slow_raw, str):
+            from unshackle.core.utils.click_types import SLOW_DELAY_RANGE
+
+            try:
+                params["slow"] = SLOW_DELAY_RANGE.convert(slow_raw, None, None)
+            except click.BadParameter as exc:
+                raise Exception(f"Invalid slow parameter: {exc}")
+
     # Convert wanted episode strings to internal "SxE" format
     # Accepts: "S01E01", "S01-S03", "s1e1", "1x1", or already-parsed format
     wanted_raw = params.get("wanted")
@@ -180,6 +195,7 @@ def _perform_download(
     ctx.params = {
         "proxy": params.get("proxy"),
         "no_proxy": params.get("no_proxy", False),
+        "no_proxy_download": params.get("no_proxy_download", False),
         "profile": params.get("profile"),
         "repack": params.get("repack", False),
         "tag": params.get("tag"),
@@ -318,6 +334,7 @@ def _perform_download(
                 export=params.get("export"),
                 cdm_only=params.get("cdm_only"),
                 no_proxy=params.get("no_proxy", False),
+                no_proxy_download=params.get("no_proxy_download", False),
                 no_folder=params.get("no_folder", False),
                 no_source=params.get("no_source", False),
                 no_mux=params.get("no_mux", False),
@@ -653,8 +670,11 @@ class DownloadQueueManager:
             if stdout.strip():
                 log.debug(f"Worker stdout for job {job.job_id}: {stdout.strip()}")
             if stderr.strip():
-                log.warning(f"Worker stderr for job {job.job_id}: {stderr.strip()}")
                 job.worker_stderr = stderr.strip()
+                if returncode != 0:
+                    log.warning(f"Worker stderr for job {job.job_id}: {stderr.strip()}")
+                else:
+                    log.debug(f"Worker stderr for job {job.job_id}: {stderr.strip()}")
 
             result_data: Optional[Dict[str, Any]] = None
             try:

unshackle/core/api/handlers.py

@@ -58,7 +58,9 @@ DEFAULT_DOWNLOAD_PARAMS = {
     "skip_dl": False,
     "export": False,
     "cdm_only": None,
+    "proxy": None,
     "no_proxy": False,
+    "no_proxy_download": False,
     "no_folder": False,
     "no_source": False,
     "no_mux": False,
@@ -67,7 +69,11 @@ DEFAULT_DOWNLOAD_PARAMS = {
     "worst": False,
     "best_available": False,
     "repack": False,
+    "tag": None,
+    "tmdb_id": None,
     "imdb_id": None,
+    "animeapi_id": None,
+    "enrich": False,
     "output_dir": None,
     "no_cache": False,
     "reset_cache": False,
@@ -1026,7 +1032,7 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
         return "Cannot use both s_lang and require_subs"
 
     if "range" in data and data["range"]:
-        valid_ranges = ["SDR", "HDR10", "HDR10+", "DV", "HLG"]
+        valid_ranges = ["SDR", "HDR10", "HDR10+", "DV", "HLG", "HYBRID"]
         if isinstance(data["range"], list):
             for r in data["range"]:
                 if r.upper() not in valid_ranges:
@@ -1097,8 +1103,17 @@ async def download_handler(data: Dict[str, Any], request: Optional[web.Request]
 
         # Create download job with filtered parameters (exclude service and title_id as they're already passed)
         filtered_params = {k: v for k, v in data.items() if k not in ["service", "title_id"]}
-        # Merge defaults with provided parameters (user params override service defaults, which override global defaults)
+        # Overlay any dl-relevant keys from `serve:` config (e.g. downloads, workers) so the API
-        params_with_defaults = {**DEFAULT_DOWNLOAD_PARAMS, **service_specific_defaults, **filtered_params}
+        # respects server-side defaults without each client having to send them.
+        serve_overrides = {
+            k: v for k, v in (config.serve or {}).items() if k in DEFAULT_DOWNLOAD_PARAMS and v is not None
+        }
+        params_with_defaults = {
+            **DEFAULT_DOWNLOAD_PARAMS,
+            **serve_overrides,
+            **service_specific_defaults,
+            **filtered_params,
+        }
         job = manager.create_job(normalized_service, title_id, **params_with_defaults)
 
         return web.json_response(

unshackle/core/api/routes.py

@@ -630,8 +630,10 @@ async def download(request: web.Request) -> web.Response:
                 type: boolean
                 description: Download audio description tracks (default - false)
               slow:
-                type: boolean
+                oneOf:
-                description: Add 60-120s delay between downloads (default - false)
+                  - type: boolean
+                  - type: string
+                description: Add randomized delay between downloads. `true` for default 60-120s, or `"MIN-MAX"` string (e.g., `"20-40"`). Min must be >= 20 (default - null)
               split_audio:
                 type: boolean
                 description: Create separate output files per audio codec instead of merging all audio (default - null)
@@ -650,6 +652,9 @@ async def download(request: web.Request) -> web.Response:
               no_proxy:
                 type: boolean
                 description: Force disable all proxy use (default - false)
+              no_proxy_download:
+                type: boolean
+                description: Bypass proxy for segment downloads only. Manifest, license, and auth still use proxy (default - false)
               tag:
                 type: string
                 description: Set the group tag to be used (default - None)
@@ -680,6 +685,9 @@ async def download(request: web.Request) -> web.Response:
               best_available:
                 type: boolean
                 description: Continue with best available if requested quality unavailable (default - false)
+              worst:
+                type: boolean
+                description: Select the lowest bitrate track within the specified quality. Requires `quality` (default - false)
               repack:
                 type: boolean
                 description: Add REPACK tag to the output filename (default - false)

unshackle/core/titles/episode.py

@@ -160,7 +160,7 @@ class Series(SortedKeyList, ABC):
         sum(seasons.values())
         season_breakdown = ", ".join(f"S{season}({count})" for season, count in sorted(seasons.items()))
         tree = Tree(
-            f"{num_seasons} seasons, {season_breakdown}",
+            f"{num_seasons} season{'s'[:num_seasons^1]}, {season_breakdown}",
             guide_style="bright_black",
         )
         if verbose:

unshackle/core/vaults.py

@@ -1,3 +1,4 @@
+import logging
 from typing import Any, Iterator, Optional, Union
 from uuid import UUID
 
@@ -5,6 +6,8 @@ from unshackle.core.config import config
 from unshackle.core.utilities import import_module_by_path
 from unshackle.core.vault import Vault
 
+log = logging.getLogger(__name__)
+
 _VAULTS = sorted(
     (path for path in config.directories.vaults.glob("*.py") if path.stem.lower() != "__init__"), key=lambda x: x.stem
 )
@@ -48,7 +51,13 @@ class Vaults:
     def get_key(self, kid: Union[UUID, str]) -> tuple[Optional[str], Optional[Vault]]:
         """Get Key from the first Vault it can by KID (Key ID) and Service."""
         for vault in self.vaults:
+            try:
                 key = vault.get_key(kid, self.service)
+            except (PermissionError, NotImplementedError):
+                continue
+            except Exception as e:
+                log.warning(f"Failed to get key from Vault '{vault.name}': {e}")
+                continue
             if key and key.count("0") != len(key):
                 return key, vault
         return None, None
@@ -62,6 +71,8 @@ class Vaults:
                     success += vault.add_key(self.service, kid, key)
                 except (PermissionError, NotImplementedError):
                     pass
+                except Exception as e:
+                    log.warning(f"Failed to add key to Vault '{vault.name}': {e}")
         return success
 
     def add_keys(self, kid_keys: dict[Union[UUID, str], str]) -> int:
@@ -79,6 +90,8 @@ class Vaults:
                     success += 1
                 except (PermissionError, NotImplementedError):
                     pass
+                except Exception as e:
+                    log.warning(f"Failed to add keys to Vault '{vault.name}': {e}")
         return success
 
 

unshackle/unshackle-example.yaml

@@ -522,6 +522,14 @@ serve:
   # playready_devices:  # PlayReady device paths (auto-populated from directories.prds)
   #   - '/path/to/device.prd'
 
+  # Optional: any /api/download flag can be set here as a server-side default.
+  # Per-request body values still win. Useful for raising concurrency without
+  # changing every client call. Full list of accepted keys: see docs/API.md.
+  # downloads: 4              # parallel tracks per download job
+  # workers: 16               # threads per track segment fetch
+  # best_available: true
+  # no_proxy_download: false
+
 # Remote Services Configuration
 # Connect to a remote unshackle server (unshackle serve) to use its services
 # without needing the service code locally. Use with: unshackle dl --remote