Compare commits

..
16 Commits
Author SHA1 Message Date
imSp4rky ce69e8a028 docs(config): make docs/ canonical, slim unshackle-example.yaml to a starter 2026-06-21 21:23:03 -06:00
imSp4rky 605c0e65cb feat(docs): add per-vault network timeout configuration for remote vaults 2026-06-21 20:57:17 -06:00
imSp4rky ecf0bf41ab feat(vaults): add configurable network timeout for remote vaults
Default 10s on API/HTTP vault requests, overridable per-vault (`timeout:`) or globally (`vault_timeout`); previously they could hang indefinitely on an unreachable host.
2026-06-21 20:39:37 -06:00
imSp4rky 3f5b5e0fe1 refactor(api): drop dead error codes, lock error contract harness
Remove 4 unused APIErrorCode members; strengthen test_errors.py to pin the full (status, code, retryable) contract before the PRD 05 collapse.
2026-06-21 14:13:44 -06:00
imSp4rky 2f86d782bf fix(api): annotate routes.py to pass strict mypy
Type cors_middleware and the service_data dict; annotations only, no behaviour change.
2026-06-21 13:58:23 -06:00
imSp4rky d4b0c81c63 refactor(api): collapse route registration into single ROUTES table
Derive both setup_routes and setup_swagger from one ROUTES table, and move the repeated trailing APIError->response block into an @api_handler decorator.
2026-06-21 13:47:23 -06:00
imSp4rky 9cb20693e0 refactor(api): use Event.wait and a shared to_enum helper
Replace InputBridge's poll loop with Event.wait(timeout) and collapse the three download_manager enum maps into one to_enum() helper. No behaviour change.
2026-06-21 13:31:08 -06:00
imSp4rky fb90cd87b0 refactor(api): dedup handler serializers, validation, and service setup
Collapse repeated serializer/validation/CDM/list-setup logic in handlers.py behind small helpers; output behaviour unchanged.
2026-06-21 13:23:22 -06:00
imSp4rky 8a9a1e1bdc refactor(api): remove dead code paths in serve/remote layer
Delete code paths that never executed in the serve/API/remote stack:

- BridgeCancelledError: raised but never caught; replace raises with RuntimeError and drop the class.
- InputBridge.error setter and _error field: written at the handler layer, never read (session.auth_error is the surfaced channel).
- cancel_event param and _check_cancel inner fn on _perform_download: the only caller always passed None (the live job.cancel_event is untouched).
- result_with_progress monkeypatch: its status pings were redundant with the file-polled progress_sink, which is now the sole path.
- session param on _resolve_manifest_data: unused in the body.

Reconcile SessionStore's hardcoded 600s auth-state TTL into a shared AUTH_INPUT_TIMEOUT constant referenced by both the InputBridge input timeout and the store, removing the magic number without changing the longer grace window.

Update input_bridge/session_store tests for the new channels and add an assertion that auth-state entries expire at the single rule.
2026-06-21 13:07:45 -06:00
AviDev 40eafce17b feat(api): expose active-track segment counts and transfer speed in job progress (#124)
The job progress already reports a weighted percentage, phase and track counts.
The per-track tee in build_job_progress_callables already tracks each track's
segment counts and sees the downloader's speed string, so also surface the
currently-downloading track's segments_done / segments_total and speed in the
emitted progress (and on the job + its to_dict). Lets a client show
"197/663 segments, 3.2 MB/s" instead of only a percentage.
2026-06-21 12:40:16 -06:00
AviDev 733cb02391 feat(api): expose title description, date and cover_url in serialize_title (#119)
Surface optional display metadata a service can provide: the synopsis
(title.description) and a release/air date or poster image URL stashed in
title.data ("date" / "cover_url"). Purely additive, all fields default to
null when the service does not set them, so existing clients are unaffected.
2026-06-21 12:38:27 -06:00
sk8ordi3 5b724b48f8 feat: add automatic Firefox cookie and localstorage extraction (#112)
* feat: add automatic Firefox cookie and localstorage extraction

* address security and consistency findings from PR review
2026-06-21 12:26:17 -06:00
imSp4rky a2f670b140 feat: support Python 3.14 and widen range to 3.11-3.14
Move the supported interpreter range to >=3.11,<3.15, dropping the EOL 3.10 floor and unblocking installs on 3.13/3.14.

- pyproject: requires-python ">=3.10,<3.13" -> ">=3.11,<3.15"
- update_checker: replace deprecated asyncio.get_event_loop() with asyncio.to_thread() (warns on 3.12+, slated for removal)
- test_compression: simplify _run() to plain asyncio.run()
- README: bump documented Python range to 3.11 - 3.14
- uv.lock: regenerate for the new interpreter floor
2026-06-20 15:25:42 -06:00
a9c677c349 feat: Native ExpressVPN HTTPS Proxy Provider (#126)
* feat: add native ExpressVPN HTTPS proxy provider

Introduces ExpressVPN as a first-class native proxy provider for unshackle.

This provider implements the full browser extension proxy authentication flow — from OAuth PKCE bootstrap through subscription validation to authenticated HTTPS proxy endpoints — without requiring any external tools or manual token management.

Supports smart location selection, city-level targeting with intelligent abbreviation matching, and pinned server selection. All tokens are automatically cached and refreshed.

No new dependencies required.

* fix: address PR #126 review feedback

- __repr__ no longer triggers network/OAuth calls; reads cached locations only
- _match_city uses tiered priority buckets (abbrev > exact > prefix > substring)
- Token cache writes race-free: os.open + os.fchmod(0o600) on Unix, fallback on Windows
- Cookie/cache paths resolve under config.directories, not CWD
- _get_connection_token parses response JSON once
- Drop unused PKCE state parameter (never validated)
- Add inline doc for _is_jwt_expired no-exp behavior
- Fix 3 lines exceeding 120-col project style

---------

Co-authored-by: MrMovies-Dev <MrMovies-Dev@users.noreply.github.com>
Co-authored-by: sp4rk.y <george+slight@slight.me>
2026-06-17 16:51:33 -06:00
imSp4rky ba69bc7d61 feat(dl): add --merge-video to merge video language variants
Group selected videos by (resolution, range, codec) and mux each group into one MKV; only language is collapsed, so ranges/codecs/resolutions stay in separate files. Adds --merge-video flag + muxing.merge_video config (global and per-service), docs, and tests.
2026-06-17 13:31:21 -06:00
imSp4rky 2e1ffebec0 docs: rename to match other docs. 2026-06-15 20:55:27 -06:00
40 changed files with 2873 additions and 1642 deletions
+1 -1
View File
@@ -39,7 +39,7 @@ unshackle --help
External tools on your `PATH` (recommended versions):
- [Python](https://www.python.org/) - 3.10 - 3.12
- [Python](https://www.python.org/) - 3.11 - 3.14
- [uv](https://docs.astral.sh/uv/) - ≥ 0.5
- [FFmpeg](https://ffmpeg.org/) - ≥ 6.0
- [MKVToolNix](https://mkvtoolnix.download/) - ≥ 80
+16 -2
View File
@@ -159,11 +159,25 @@ debug_keys: true
---
## debug_requests (bool)
Controls whether HTTP-library logs are shown during `--debug` sessions. Default: `false`
When `--debug` is active and this is `false`, unshackle silences the noisy per-request logs emitted by `urllib3`, `requests`, `rnet`, `httpx`, and related libraries (e.g. `GET ...seg-001.m4s 200`) so that unshackle's own structured DEBUG output stays readable. Set to `true` to see every individual HTTP request, including per-segment downloads.
Has no effect when `--debug` is not active.
```yaml
debug_requests: true
```
---
## set_terminal_bg (bool)
Controls whether unshackle should set the terminal background color. Default: `false`
Controls whether unshackle applies its theme background color to the terminal. Default: `false`
For example,
When enabled, the Rich console theme's background color is appended to the foreground styles for both the general color scheme and the ASCII art banner, so the full color palette renders correctly on terminals whose default background differs from the theme. Set to `true` if the banner or colored output looks off on your terminal.
```yaml
set_terminal_bg: true
+92
View File
@@ -93,6 +93,31 @@ serve:
Layering order: built-in defaults < `serve.*` overrides < service-specific click defaults < request body.
### Per-request override gates
Two security gates under `serve:` guard capabilities that `/api/download` clients can request. Both default OFF; an unset or `false` value causes the server to reject the request with `403 FORBIDDEN`.
**`cdm_overrides`** — a `/api/download` job body may include a `cdm` field to select a specific server-side device. When this gate is off, every such request is rejected. Set to `true` to allow any device the server has, or to a list of device names to restrict to an explicit subset:
```yaml
serve:
# Allow any server-side device:
cdm_overrides: true
# Or restrict to specific devices:
cdm_overrides:
- generic_nexus_4464_l3
```
**`allow_job_credentials`** — a `/api/download` job body may include a `credential` (single) or `credentials` (map) field to authenticate the job with client-supplied secrets instead of the server's configured credentials. When this gate is off, any job containing either field is rejected. Each distinct credential gets its own isolated token cache on the server. Set to `true` to allow:
```yaml
serve:
allow_job_credentials: true
```
Both gates are independent and can be combined. A typical locked-down deployment leaves both unset; a trusted single-client deployment might enable both.
---
## Endpoint Map
@@ -779,3 +804,70 @@ downloading -> cancelled
Jobs are retained for 24 hours after completion (override via top-level `download_job_retention_hours` in `unshackle.yaml`). The server runs up to 2 concurrent download jobs by default; override via top-level `max_concurrent_downloads`. This is independent of `serve.downloads`, which controls parallel tracks **within** a single job.
Remote sessions are managed by `SessionStore` (`unshackle/core/api/session_store.py`); idle sessions and their `InputBridge` instances are cleaned up by a background loop started/stopped with the app lifecycle.
---
## Client Configuration: `remote_services`
`remote_services` is a top-level block in `unshackle.yaml` on the **client** machine. It tells `dl` how to reach a remote `unshackle serve` instance instead of running service code locally. The client handles track selection, download, decrypt, and mux; the server handles auth, manifest parsing, and (optionally) DRM licensing.
### CLI flags
| Flag | Description |
| --- | --- |
| `--remote` | Use a remote server instead of local service code. Reads `remote_services` from config. |
| `--server <name>` | Select a named server when multiple are configured. If only one entry exists it is picked automatically. |
```bash
unshackle dl --remote EXAMPLE1 "https://example.com/show/abc123"
unshackle dl --remote --server us-server EXAMPLE1 "https://example.com/show/abc123"
```
### Config structure
```yaml
remote_services:
my-server:
url: "http://192.168.1.100:8786" # base URL of the remote serve instance
api_key: "your-secret-key-here" # sent as X-Secret-Key; omit or set "" for --no-key servers
# server_cdm: the server runs its own CDM and returns KID:KEY pairs (mode: server_cdm).
# false (default): client runs its own CDM; license challenges are proxied through the server.
server_cdm: false
# Per-service overrides applied on the client side after the remote session is set up.
# Supported keys: decryption, downloader, cdm (and any free-form service config keys).
services:
EXAMPLE1:
decryption: mp4decrypt # override decryption tool for this service (shaka or mp4decrypt)
```
**`server_cdm`** — when `false` (default) the client builds CDM challenges locally and calls `POST /api/session/{id}/license` with `mode: "proxy"`; the server forwards the challenge to the service's license endpoint and returns raw license bytes. When `true` the client skips local CDM setup and calls with `mode: "server_cdm"`; the server performs the full CDM flow and returns `{ "keys": { "<KID>": "<KEY>" } }`. The server-side user key must have a matching device configured (`devices` / `playready_devices`) for `server_cdm` to work.
### Multiple servers
When more than one server is configured, `--server <name>` is required:
```yaml
remote_services:
us-server:
url: "https://us.example.com:8786"
api_key: "us-key"
server_cdm: true
services:
EXAMPLE1:
decryption: mp4decrypt
eu-server:
url: "https://eu.example.com:8786"
api_key: "eu-key"
server_cdm: false
```
```bash
unshackle dl --remote --server us-server EXAMPLE1 "https://example.com/show/abc123"
```
### Cache forwarding
On session open the client sends any local `*.json` cache files for the service (tokens, etc.) to the server so it can skip interactive re-auth. On session close the server returns updated cache files; the client writes them back locally so the next remote session can forward them again.
+112 -37
View File
@@ -178,78 +178,141 @@ remote_cdm:
use_vaults: true # Integrate with vault system
```
**Advanced Example with Field Mapping:**
**Full field reference:**
```yaml
remote_cdm:
- name: advanced_custom_api
type: custom_api
host: https://api.example.com
device:
name: L1
type: ANDROID
security_level: 1
timeout: 30 # Default request timeout (seconds); overridden per-endpoint
# Authentication configuration
device:
name: L1 # Passed as 'scheme' in base request params
type: ANDROID # CHROME, ANDROID, or PLAYREADY
system_id: 26830 # Widevine system ID (informational; not sent)
security_level: 1 # 1–3 for Widevine; 2000/3000 for PlayReady
# Authentication — applied to every request via session headers or Basic auth
auth:
type: header
header_name: X-API-Key
key: YOUR_SECRET_KEY
custom_headers:
type: header # bearer | header | basic | body | query
header_name: X-API-Key # used when type=header (default: Authorization)
key: YOUR_SECRET_KEY # API key / token / password
# bearer_token: TOKEN # explicit token for type=bearer (falls back to 'key')
# username: user # required for type=basic
# password: pass # required for type=basic
custom_headers: # merged into every request
User-Agent: Unshackle/3.1.0
X-Client-Version: "1.0"
# Endpoint configuration
# Endpoints — both keys are required
endpoints:
get_request:
get_request: # first call: obtain challenge or cached keys
path: /v2/challenge
method: POST
timeout: 30
decrypt_response:
timeout: 30 # overrides top-level timeout for this endpoint
decrypt_response: # second call: exchange license response for keys
path: /v2/decrypt
method: POST
timeout: 30
# Request parameter mapping
# request_mapping — transform outgoing parameters before each request
# Base params sent to get_request: scheme, init_data, [service], [service_certificate]
# Base params sent to decrypt_response: scheme, session_id, init_data, license_request, license_response
request_mapping:
get_request:
param_names:
init_data: pssh # Rename 'init_data' to 'pssh'
scheme: device_type # Rename 'scheme' to 'device_type'
static_params:
api_version: "2.0" # Add static parameter
param_names: # rename base param keys
init_data: pssh
scheme: device_type
static_params: # always-added fixed values
api_version: "2.0"
conditional_params: # added only when condition is true
- condition: "is_playready == True"
params:
drm_type: playready
transforms: # applied after renaming; type values listed below
- param: pssh
type: base64_encode
nested_params: # group named params under a parent key
payload:
- pssh
- device_type
exclude_params: # remove unwanted params before sending
- service_certificate
decrypt_response:
param_names:
license_request: challenge
license_response: license
# Response field mapping
# response_mapping — extract and validate fields from API responses
response_mapping:
get_request:
fields:
challenge: data.challenge # Deep field access
fields: # map standard names to dot-path locations in response JSON
challenge: data.challenge
session_id: session.id
success_conditions:
- status == 'ok' # Validate response
message: status.message
response_types: # classify response so CDM can choose the right path
- condition: "message_type == 'license-request'"
type: license_request # CDM expects challenge + session_id
- condition: "message_type == 'cached-keys'"
type: cached_keys # CDM skips license round-trip
success_conditions: # all must be true; failure raises RequestException
- "message == 'success'"
error_fields: # inspected when success_conditions fail (default: error, message, details)
- error
- details
transforms: # applied to extracted fields
- field: challenge
type: base64_decode
decrypt_response:
fields:
keys: data.keys
key_fields:
kid: key_id # Map 'kid' field
key: content_key # Map 'key' field
message: status.message
key_fields: # map key-object property names to standard names
kid: key_id # default: kid
key: content_key # default: key
type: key_type # default: type (value used as-is; CONTENT treated as content key)
success_conditions:
- "message == 'success'"
# caching — vault and API-level key caching
caching:
enabled: true
use_vaults: true
check_cached_first: true # Check cache before API calls
enabled: true # master switch (default: true)
use_vaults: true # check/write unshackle key vaults (default: true)
check_cached_first: true # check vaults before making any API call (default: true)
```
**Supported Authentication Types:**
- `bearer` - Bearer token authentication
- `header` - Custom header authentication
- `basic` - HTTP Basic authentication
- `body` - Credentials in request body
- `query` - Authentication added to query string parameters
**Supported `transforms` types** (for both `request_mapping` and `response_mapping`):
`base64_encode`, `base64_decode`, `hex_encode`, `hex_decode`, `json_stringify`, `json_parse`, `parse_key_string`
**Supported authentication types:**
- `bearer` — `Authorization: Bearer <key>` (or `bearer_token` field)
- `header` — arbitrary header; set `header_name` (default `Authorization`)
- `basic` — HTTP Basic auth via `username` / `password`
- `body` — credentials injected into request body (handled by request_mapping)
- `query` — credentials appended to query string (handled by request_mapping)
### PyPlayReady Remote CDM
Connects to a running `unshackle serve` instance for PlayReady licensing. Uses
`pyplayready.remote.remotecdm.RemoteCdm` under the hood. No `type` field is needed —
the entry is routed to PlayReady when `device_type: PLAYREADY` is set (same branching
logic as the legacy Widevine path).
```yaml
remote_cdm:
- name: playready_remote
device_name: my_prd_device # Device name registered on the serve instance
device_type: PLAYREADY # Required to select the PlayReady code path
system_id: 0 # Not used by pyplayready RemoteCdm; set to 0
security_level: 3000 # 2000 = SL2000, 3000 = SL3000
host: http://127.0.0.1:8786/playready # Must include the /playready path suffix
secret: your-api-secret-key # X-Secret-Key sent to the serve instance
```
**Note:** The `/playready` path suffix in `host` is required — `pyplayready.RemoteCdm`
appends its own endpoint paths on top of this base, so omitting the suffix will result
in 404 errors.
### Legacy PyWidevine Serve Format
@@ -393,6 +456,16 @@ Additional behavior:
- `no_push` (bool): Optional per-vault flag. When `true`, the vault will not receive pushed keys (writes) but
will still be queried and can provide keys for lookups. Useful for read-only/backup vaults.
- `timeout` (float): Optional per-vault network timeout in seconds for the remote `API` and `HTTP` vaults.
Defaults to the global `vault_timeout` (10 seconds) so an unreachable vault host fails fast instead of
hanging the download. Set it on a vault to override the global, e.g. `timeout: 5.0`. Has no effect on
SQLite (local) or MySQL (uses its driver's own connect timeout).
Set a global default for all remote vaults at the top level of the config:
```yaml
vault_timeout: 10.0 # seconds; per-vault `timeout:` overrides this
```
### Using an API Vault
@@ -407,6 +480,7 @@ not work in unshackle. The API format can be seen in the [API Vault Code](unshac
# uri: "https://api.example.com/key-vault"
token: "random secret key" # authorization token
# no_push: true # optional; make this API vault read-only (lookups only)
# timeout: 5.0 # optional; per-vault network timeout (seconds), overrides vault_timeout
```
### Using a MySQL Vault
@@ -470,6 +544,7 @@ is useful for integrating with various third-party key vault APIs.
api_mode: "json" # query, json, or decrypt_labs
# username: "user" # required for query mode only
# no_push: false # optional; defaults to false
# timeout: 5.0 # optional; per-vault network timeout (seconds), overrides vault_timeout
```
**Supported API Modes:**
+78
View File
@@ -241,3 +241,81 @@ Common issues:
- [Gluetun Wiki](https://github.com/qdm12/gluetun-wiki) - Official provider documentation
- [Gluetun GitHub](https://github.com/qdm12/gluetun)
---
# ExpressVPN HTTPS Proxy
ExpressVPN is a standalone HTTPS proxy provider — it does **not** use Docker or the gluetun sub-provider system. It authenticates via a browser-extension OAuth flow (Keycloak/PKCE) and returns an authenticated `https://cat:<token>@<host>:443` proxy URL.
## Prerequisites
An active ExpressVPN subscription is required. The provider resolves proxy-capable locations through ExpressVPN's own API, so no local VPN client or Docker container is needed.
## Authentication
The provider uses a token cascade (tried in order):
1. **Cached tokens** — `cache/global/expressvpn_tokens.json` (auto-written after first auth; access and refresh tokens are refreshed automatically when they expire).
2. **Exported browser cookies** — `cookies/vpn/expressvpn.txt` (also accepted at `cookies/vpns/expressvpn.txt`). Must contain the `KEYCLOAK_IDENTITY` and `KEYCLOAK_SESSION` cookies from `auth.expressvpn.com`. Used for a one-time PKCE bootstrap that populates the token cache.
3. **Desktop `account.json`** — configured via the `account_json` key (see below).
Export cookies from a browser that is logged in to `auth.expressvpn.com`. Any Netscape-format, JSON array, or `name=value` file is accepted.
## Configuration
Add to `~/.config/unshackle/unshackle.yaml` under `proxy_providers`:
```yaml
proxy_providers:
expressvpn:
region_map:
us: "ny" # --proxy expressvpn:us → New York (default city)
gb: "london" # --proxy expressvpn:gb → London
de: "frankfurt" # --proxy expressvpn:de → Frankfurt
es: "madrid" # --proxy expressvpn:es → Madrid
mx: # --proxy expressvpn:mx → smart/random MX location
```
`region_map` maps a country code to a default city preset. An empty (null) value enables smart connection (a random location in that country). The preset is only applied when the CLI query contains no city; an explicit city in the query always takes precedence.
### Optional keys
| Key | Default | Description |
|-----|---------|-------------|
| `region_map` | `{}` | Country → default city presets (see above) |
| `server_map` | `{}` | Alias → ExpressVPN location slug or direct hostname |
| `cookie_path` | `cookies/vpn/expressvpn.txt` | Path to exported browser cookies |
| `cache_path` | `cache/global/expressvpn_tokens.json` | Token cache location |
| `account_json` | *(none)* | Path to ExpressVPN desktop `account.json` |
| `refresh_token` | *(none)* | OAuth refresh token (manual override) |
| `timeout` | `10.0` | HTTP request timeout in seconds |
## Usage
```bash
unshackle dl SERVICE CONTENT --proxy expressvpn:us # random/smart US location
unshackle dl SERVICE CONTENT --proxy expressvpn:us-ny # specific city (New York)
unshackle dl SERVICE CONTENT --proxy expressvpn:us-ny-2 # pinned server #2 in New York
unshackle dl SERVICE CONTENT --proxy expressvpn:mx # random/smart Mexico location
unshackle dl SERVICE CONTENT --proxy expressvpn:es-madrid # specific city (Madrid)
```
Query format: `expressvpn:<country>[-<city>[-<N>]]`
- `country` — 2-letter country code (e.g. `us`, `gb`, `de`).
- `city` — optional city abbreviation or slug matched against location names using first-letter abbreviation (`ny` → "New York"), exact slug, prefix, then substring strategies.
- `N` — optional 1-based server index (e.g. `-2` selects the second server in the matched city). If `N` exceeds the available server count, a random server is selected.
## City matching
The city part of the query is matched against ExpressVPN location names using these strategies (in priority order):
1. First-letter abbreviation — `ny` matches "New York"
2. Exact slug — `miami` matches "Miami"
3. Prefix match — `mia` matches "Miami"
4. Substring match — `york` matches "New York"
## Token cache
After the first successful authentication the token cache is written to `cache/global/expressvpn_tokens.json` (mode 0600). Subsequent runs refresh expired access/SRT/connection tokens automatically without re-reading cookies.
+77 -7
View File
@@ -32,16 +32,66 @@ filenames:
## output_template (dict)
Configure custom output filename templates for movies, series, and songs.
Configure custom output filename templates for movies, series, songs, and albums.
This is **required** in your `unshackle.yaml` — a warning is shown if not configured.
Available variables: `{title}`, `{year}`, `{season}`, `{episode}`, `{season_episode}`, `{episode_name}`,
`{quality}`, `{resolution}`, `{source}`, `{audio}`, `{audio_channels}`, `{audio_full}`,
`{video}`, `{hdr}`, `{hfr}`, `{atmos}`, `{dual}`, `{multi}`, `{tag}`, `{edition}`, `{repack}`,
`{lang_tag}`, `{track_number}`, `{artist}`, `{album}`, `{disc}`
Add `?` suffix to make a variable conditional (omitted when empty): `{year?}`, `{hdr?}`, `{repack?}`
### General variables
| Variable | Meaning |
|---|---|
| `title` | Title of the movie, episode, or track |
| `year` | Release year |
| `source` | Service source tag (e.g. `AMZN`, `NF`) |
| `tag` | Group/release tag configured via `tag:` |
| `edition` | Edition label (e.g. `Extended`, `Theatrical`) |
| `repack` | `REPACK` when the release is a repack, otherwise empty |
| `lang_tag` | Language tag produced by `language_tags` rules (e.g. `NORDiC`) |
### Series / episode variables
| Variable | Meaning |
|---|---|
| `season` | Season number (e.g. `1`) |
| `episode` | Episode number (e.g. `1`) |
| `season_episode` | Combined season/episode (e.g. `S01E01`) |
| `episode_name` | Episode title |
### Video / audio variables
| Variable | Meaning |
|---|---|
| `quality` | Resolution label (e.g. `1080p`, `2160p`) |
| `resolution` | Raw resolution string |
| `video` | Video codec label (e.g. `H.264`, `H.265`) |
| `hdr` | HDR format label (e.g. `HDR`, `DV`) |
| `hfr` | HFR label when frame rate exceeds normal threshold |
| `audio` | Audio codec label (e.g. `AAC`, `EAC3`) |
| `audio_channels` | Channel count (e.g. `2.0`, `5.1`) |
| `audio_full` | Combined codec + channels (e.g. `DDP5.1`) |
| `atmos` | `Atmos` when Dolby Atmos is present, otherwise empty |
| `dual` | `DUAL` when two audio languages are present, otherwise empty |
| `multi` | `MULTi` when three or more audio languages are present, otherwise empty |
### Music variables
| Variable | Meaning |
|---|---|
| `track_number` | Track number within album |
| `disc` | Disc number |
| `track_total` | Total tracks on disc |
| `disc_total` | Total discs in release |
| `artist` | Track artist |
| `album_artist` | Album-level artist (may differ from track artist) |
| `album` | Album title |
| `release_type` | Release type (e.g. `Album`, `Single`, `EP`) |
| `genre` | Genre string |
| `explicit` | `Explicit` when track has explicit content flag, otherwise empty |
| `isrc` | International Standard Recording Code |
| `upc` | Universal Product Code (album barcode) |
| `label` | Record label name |
```yaml
output_template:
# Scene-style (dot-separated)
@@ -71,7 +121,9 @@ If not configured, the default folder naming is used:
- Songs: `Artist - Album (Year)`
`folder` accepts either a single string (applies to all title kinds) or a mapping with per-kind
templates keyed by `movies`, `series`, and/or `songs`. Unknown keys are warned about and ignored.
templates keyed by `movies`, `series`, `songs`, and/or `albums`. Unknown keys are warned about and ignored.
Useful music variables for album folders: `album_artist`, `album`, `artist`, `year`, `genre`, `label`, `release_type`, `track_total`, `disc_total`.
```yaml
output_template:
@@ -90,6 +142,7 @@ output_template:
# movies: '{title} ({year})'
# series: '{title} ({year?})'
# songs: '{artist} - {album} ({year?})'
# albums: '{album_artist} - {album} ({year?})'
```
Example outputs:
@@ -204,6 +257,23 @@ Enable/disable tagging downloaded files with IMDB/TMDB/TVDB identifiers (when av
Note: The `--split-audio` CLI flag overrides this setting. When `--split-audio` is passed,
`merge_audio` is effectively set to `false` for that run.
- `merge_video`
Merge video **language variants** into one file. Default: `false`
- `false`: One MKV per video track (the default behaviour).
- `true`: Group the selected video tracks by `(resolution, range, codec)` and merge
each group into one MKV, so only language varies within a file. The player switches
between the language tracks. No re-encode, no concatenation.
Only the language dimension is collapsed. Different **resolutions**, **ranges**
(SDR/HDR10/HDR10+/DV/HYBRID) and **codecs** (H264/H265) always stay in separate files.
For example, `-r HYBRID,DV,HDR10,SDR --merge-video` produces one file per range (never a
single combined file), while a title offering English + French video of the same
resolution/range/codec produces one file containing both video tracks.
Note: The `--merge-video` CLI flag overrides this setting. Can be set per service under
`services.<TAG>.muxing.merge_video`. Change `group_videos_by_variant` in
`unshackle/commands/dl.py` to adjust the grouping.
- `default_language` (dict)
Override which track is flagged as the default in the muxed MKV, regardless
of the title's original language. Useful when you always want your player to
+18
View File
@@ -0,0 +1,18 @@
# docs/
This folder is the **canonical configuration reference** for unshackle. `unshackle/unshackle-example.yaml` is a minimal starter config only — it intentionally omits most options. Come here for the full picture.
## Contents
| Document | Description |
|---|---|
| [Advanced & System Configuration](ADVANCED_CONFIG.md) | `serve`, debugging, and system-level options not covered elsewhere |
| [REST API Documentation](API.md) | HTTP API exposed by `unshackle serve`: endpoints, auth, remote downloads, CDM hosting |
| [Structured Debug Logging](DEBUG_LOGGING.md) | `-d`/`--debug` JSONL output: schema, redaction rules, `log_event`/`timed_operation` conventions |
| [Download & Processing Configuration](DOWNLOAD_CONFIG.md) | Downloader settings, concurrency, retries, and post-processing behaviour |
| [DRM & CDM Configuration](DRM_CONFIG.md) | Widevine, PlayReady, and ClearKey CDM setup; local and remote CDMs; key vaults |
| [Gluetun VPN Proxy](GLUETUN.md) | Docker-managed VPN proxies via Gluetun; setup, WireGuard ready-detection |
| [Network & Proxy Configuration](NETWORK_CONFIG.md) | Proxy providers, geofence bypass, per-service proxy overrides |
| [Output & Naming Configuration](OUTPUT_CONFIG.md) | Output directory layout, filename templates, folder naming |
| [Service Integration & Authentication](SERVICE_CONFIG.md) | Per-service config blocks, cookies, credentials, and metadata sources |
| [Subtitle Processing Configuration](SUBTITLE_CONFIG.md) | `subtitle:` key options: format conversion, styling, forced-sub rules |
+41
View File
@@ -150,6 +150,47 @@ Please be aware that this information is sensitive and to keep it safe. Do not s
---
## firefox_cookies (dict)
Automatically extract cookies from your running Firefox installation instead of maintaining a
`cookies/<SERVICE>.txt` file. Configured as a top-level dict keyed by service tag; each value is
a settings dict for that service.
```yaml
firefox_cookies:
EXAMPLE:
hosts:
- example.com
- api.example.com
local_storage: false # optional, default false
```
**Fields:**
- `hosts` — list of host patterns to match against Firefox's cookie store. Each pattern must be at
least 3 characters. A pattern matches the exact host **and** any dot-suffix subdomain
(e.g. `example.com` also matches `sub.example.com` and `.example.com`).
- `local_storage` — when `true`, also harvest `localStorage` entries for the listed hosts from
Firefox's `webappsstore.sqlite`. Default: `false`.
At download time unshackle picks the **most recently modified** valid Firefox profile (the one
whose `cookies.sqlite` has the latest mtime), copies the database to a restricted temporary
directory (`0700`), and reads cookies from it without touching the live profile. If extraction
returns no cookies or fails for any reason, it falls back silently to the normal file-based cookie
(`cookies/<SERVICE>.txt` or `cookies/<SERVICE>/<profile>.txt`).
**Notes:**
- Firefox must not have an exclusive write lock on the database at the moment of extraction (i.e.
Firefox may be open — the extractor copies both `cookies.sqlite` and its WAL file to capture any
unflushed writes).
- `local_storage: true` is only needed for services that store auth tokens in the browser's
localStorage rather than in HTTP cookies.
- The feature is implemented in `core/utils/firefox_cookie_extractor.py`; the fallback logic lives
in `dl.get_cookie_jar` (`commands/dl.py`).
---
## tmdb_api_key (str)
API key for The Movie Database (TMDB). This is used for tagging downloaded files with TMDB,
+1 -1
View File
@@ -2,7 +2,7 @@
This document covers subtitle processing and formatting options under the top-level `subtitle:` key in `unshackle.yaml`.
For the canonical example, see `unshackle/unshackle-example.yaml`.
This document is the canonical reference for subtitle configuration. `unshackle/unshackle-example.yaml` is a minimal starter only.
## subtitle (dict)
+1 -1
View File
@@ -7,7 +7,7 @@ name = "unshackle"
version = "5.2.0"
description = "Modular Movie, TV, and Music Archival Software."
authors = [{ name = "unshackle team" }]
requires-python = ">=3.10,<3.13"
requires-python = ">=3.11,<3.15"
readme = "README.md"
license = "GPL-3.0-only"
keywords = [
+129
View File
@@ -0,0 +1,129 @@
"""Tests for ``--merge-video`` track grouping.
``group_videos_by_variant`` (``unshackle/commands/dl.py``) decides which selected video
tracks share one MKV when merge mode is on. The rule: group by ``(resolution, range,
codec)`` so only language varies within a file; resolutions, ranges and codecs stay
separate. With ``merge=False`` every track is its own group (one file per track).
These lock the pure grouping unit; the surrounding mux loop is Click-command orchestration.
"""
from __future__ import annotations
from unshackle.commands.dl import group_videos_by_variant
from unshackle.core.tracks import Video
def make_video(
track_id: str,
*,
range_: Video.Range,
height: int,
codec: Video.Codec,
language: str = "en",
) -> Video:
return Video(
id_=track_id,
url=f"https://example.test/{track_id}.m3u8",
language=language,
codec=codec,
range_=range_,
width=int(height * 16 / 9),
height=height,
bitrate=1_000_000,
)
HEVC = Video.Codec.HEVC
AVC = Video.Codec.AVC
SDR = Video.Range.SDR
HDR10 = Video.Range.HDR10
DV = Video.Range.DV
def test_merge_collapses_language_only() -> None:
"""Same (height, range, codec), different language → one group."""
videos = [
make_video("en", range_=SDR, height=1080, codec=HEVC, language="en"),
make_video("fr", range_=SDR, height=1080, codec=HEVC, language="fr"),
]
groups = group_videos_by_variant(videos, merge=True)
assert len(groups) == 1
assert [v.id for v in groups[0]] == ["en", "fr"]
def test_merge_splits_on_codec() -> None:
"""H264 vs H265 of the same resolution+range → separate groups."""
videos = [
make_video("hevc", range_=SDR, height=1080, codec=HEVC),
make_video("avc", range_=SDR, height=1080, codec=AVC),
]
groups = group_videos_by_variant(videos, merge=True)
assert len(groups) == 2
assert all(len(g) == 1 for g in groups)
def test_merge_splits_on_range() -> None:
"""SDR vs HDR10 of the same resolution+codec → separate groups."""
videos = [
make_video("sdr", range_=SDR, height=1080, codec=HEVC),
make_video("hdr10", range_=HDR10, height=1080, codec=HEVC),
]
groups = group_videos_by_variant(videos, merge=True)
assert len(groups) == 2
def test_merge_splits_on_resolution() -> None:
"""1080p vs 2160p of the same range+codec → separate groups."""
videos = [
make_video("1080", range_=SDR, height=1080, codec=HEVC),
make_video("2160", range_=SDR, height=2160, codec=HEVC),
]
groups = group_videos_by_variant(videos, merge=True)
assert len(groups) == 2
def test_merge_multi_range_yields_one_group_per_range() -> None:
"""Regression guard: -r HYBRID,DV,HDR10,SDR must never collapse into one file.
HYBRID is resolved upstream into a DV deliverable plus the requested standalone
ranges; here the four selected single-range tracks must stay in four groups.
"""
videos = [
make_video("sdr", range_=SDR, height=2160, codec=HEVC),
make_video("hdr10", range_=HDR10, height=2160, codec=HEVC),
make_video("dv", range_=DV, height=2160, codec=HEVC),
make_video("dv-hybrid", range_=DV, height=1080, codec=HEVC), # different height
]
groups = group_videos_by_variant(videos, merge=True)
assert len(groups) == 4
def test_no_merge_yields_one_group_per_track() -> None:
"""merge=False reproduces today's per-track behaviour exactly."""
videos = [
make_video("en", range_=SDR, height=1080, codec=HEVC, language="en"),
make_video("fr", range_=SDR, height=1080, codec=HEVC, language="fr"),
make_video("avc", range_=SDR, height=1080, codec=AVC),
]
groups = group_videos_by_variant(videos, merge=False)
assert len(groups) == 3
assert all(len(g) == 1 for g in groups)
def test_merge_preserves_first_seen_order() -> None:
"""Group order follows first-seen track order, for stable output filenames."""
videos = [
make_video("hevc-en", range_=SDR, height=1080, codec=HEVC, language="en"),
make_video("avc-en", range_=SDR, height=1080, codec=AVC, language="en"),
make_video("hevc-fr", range_=SDR, height=1080, codec=HEVC, language="fr"),
]
groups = group_videos_by_variant(videos, merge=True)
# HEVC group seen first (and gathers both languages), AVC group second.
assert [v.id for v in groups[0]] == ["hevc-en", "hevc-fr"]
assert [v.id for v in groups[1]] == ["avc-en"]
def test_empty_input_returns_empty() -> None:
assert group_videos_by_variant([], merge=True) == []
assert group_videos_by_variant([], merge=False) == []
+1 -1
View File
@@ -20,7 +20,7 @@ class _FakeReq:
def _run(coro):
return asyncio.get_event_loop().run_until_complete(coro) if False else asyncio.run(coro)
return asyncio.run(coro)
def test_skips_when_client_does_not_accept_gzip() -> None:
+52 -1
View File
@@ -17,23 +17,32 @@ def _body(resp) -> dict:
def test_api_error_default_http_status_per_code() -> None:
# Every live APIErrorCode and the status a client observes for it. This is the
# status half of the error contract PRD 05 must preserve through the collapse.
cases = {
APIErrorCode.INVALID_INPUT: 400,
APIErrorCode.INVALID_SERVICE: 400,
APIErrorCode.AUTH_REQUIRED: 401,
APIErrorCode.INVALID_PROXY: 400,
APIErrorCode.INVALID_PARAMETERS: 400,
APIErrorCode.AUTH_FAILED: 401,
APIErrorCode.FORBIDDEN: 403,
APIErrorCode.GEOFENCE: 403,
APIErrorCode.NOT_FOUND: 404,
APIErrorCode.NO_CONTENT: 404,
APIErrorCode.JOB_NOT_FOUND: 404,
APIErrorCode.SESSION_NOT_FOUND: 404,
APIErrorCode.TRACK_NOT_FOUND: 404,
APIErrorCode.RATE_LIMITED: 429,
APIErrorCode.INTERNAL_ERROR: 500,
APIErrorCode.DOWNLOAD_ERROR: 500,
APIErrorCode.WORKER_ERROR: 500,
APIErrorCode.SERVICE_ERROR: 502,
APIErrorCode.DRM_ERROR: 502,
APIErrorCode.NETWORK_ERROR: 503,
APIErrorCode.SERVICE_UNAVAILABLE: 503,
}
# Lock that the cases above are the *complete* live set — a new/removed code trips this.
assert {c.name for c in cases} == {c.name for c in APIErrorCode}
for code, expected in cases.items():
assert APIError(code, "x").http_status == expected, code
@@ -125,3 +134,45 @@ def test_handle_api_exception_categorizes_generic() -> None:
resp = handle_api_exception(ConnectionError("oops"))
body = _body(resp)
assert body["error_code"] == "NETWORK_ERROR"
# --- Error contract harness (PRD 05 gate) ---------------------------------
# Locks the FULL observable response — (http status, error_code, retryable) — that
# a client sees for a generic (non-APIError) exception flowing through the
# handle_api_exception funnel that every route's `except Exception` path calls.
# This is the equivalence reference: after the categorize_exception collapse,
# any tuple that changes here is a client-visible behaviour change, not a refactor.
@pytest.mark.parametrize(
"exc, status, code, retryable",
[
(Exception("Invalid credentials provided"), 401, "AUTH_FAILED", False),
(Exception("Connection refused"), 503, "NETWORK_ERROR", True),
(TimeoutError("read timeout"), 503, "NETWORK_ERROR", True),
(Exception("Not available in your region"), 403, "GEOFENCE", False),
(Exception("Title not found"), 404, "NOT_FOUND", False),
(Exception("HTTP 429 too many requests"), 429, "RATE_LIMITED", True),
(Exception("DRM license fetch failed"), 502, "DRM_ERROR", False),
(Exception("503 service unavailable"), 503, "SERVICE_UNAVAILABLE", True),
(ValueError("malformed body"), 400, "INVALID_INPUT", False),
(RuntimeError("totally novel failure xyz"), 500, "INTERNAL_ERROR", False),
],
)
def test_error_contract_generic_exception(exc: Exception, status: int, code: str, retryable: bool) -> None:
resp = handle_api_exception(exc)
assert resp.status == status
body = _body(resp)
assert body["error_code"] == code
# retryable is only serialized when True (build_error_response omits it otherwise).
assert body.get("retryable", False) is retryable
def test_error_contract_apierror_passthrough_full_shape() -> None:
# APIError carriers keep status + code + message + details + retryable verbatim.
err = APIError(APIErrorCode.RATE_LIMITED, "slow down", details={"retry_after": 5}, retryable=True)
resp = handle_api_exception(err)
assert resp.status == 429
body = _body(resp)
assert body["error_code"] == "RATE_LIMITED"
assert body["message"] == "slow down"
assert body["details"] == {"retry_after": 5}
assert body["retryable"] is True
@@ -199,6 +199,22 @@ def test_serialize_drm_widevine_minimal() -> None:
assert info["license_url"] == "https://lic.example.com/wv"
def test_serialize_drm_playready_pssh_without_dumps_is_omitted() -> None:
# pyplayready's PSSH exposes no dumps()/to_base64()/__bytes__; serialization
# must omit the pssh field rather than emit an object repr.
class _PSSH:
pass
class _PlayReady:
def __init__(self) -> None:
self._pssh = _PSSH()
self.kids = ["00112233445566778899aabbccddeeff"]
info = serialize_drm(_PlayReady())[0]
assert "pssh" not in info
assert info["kids"] == ["00112233445566778899aabbccddeeff"]
# ---------- validate_service ----------
+6 -10
View File
@@ -7,7 +7,7 @@ import time
import pytest
from unshackle.core.api.input_bridge import AuthStatus, BridgeCancelledError, InputBridge
from unshackle.core.api.input_bridge import AuthStatus, InputBridge
pytestmark = pytest.mark.unit
@@ -16,7 +16,6 @@ def test_initial_status_is_authenticating() -> None:
bridge = InputBridge()
assert bridge.status is AuthStatus.AUTHENTICATING
assert bridge.get_pending_prompt() is None
assert bridge.error is None
def test_submit_response_returns_false_when_no_prompt_pending() -> None:
@@ -49,16 +48,15 @@ def test_request_input_blocks_until_submit() -> None:
def test_request_input_times_out() -> None:
bridge = InputBridge()
with pytest.raises(TimeoutError):
with pytest.raises(TimeoutError, match="No client response"):
bridge.request_input("hello", timeout=0.6)
assert bridge.status is AuthStatus.FAILED
assert "timed out" in (bridge.error or "")
def test_cancel_before_request_raises_immediately() -> None:
bridge = InputBridge()
bridge.cancel()
with pytest.raises(BridgeCancelledError):
with pytest.raises(RuntimeError, match="cancelled"):
bridge.request_input("hello", timeout=5)
assert bridge.status is AuthStatus.FAILED
@@ -70,7 +68,7 @@ def test_cancel_unblocks_pending_request() -> None:
def worker() -> None:
try:
bridge.request_input("OTP?", timeout=5)
except BridgeCancelledError as e:
except RuntimeError as e:
exc.append(e)
t = threading.Thread(target=worker)
@@ -82,7 +80,7 @@ def test_cancel_unblocks_pending_request() -> None:
bridge.cancel()
t.join(timeout=2)
assert exc and isinstance(exc[0], BridgeCancelledError)
assert exc and isinstance(exc[0], RuntimeError)
assert bridge.status is AuthStatus.FAILED
@@ -92,9 +90,7 @@ def test_get_pending_prompt_returns_none_outside_pending_state() -> None:
assert bridge.get_pending_prompt() is None
def test_status_and_error_setters() -> None:
def test_status_setter() -> None:
bridge = InputBridge()
bridge.status = AuthStatus.AUTHENTICATED
bridge.error = "boom"
assert bridge.status is AuthStatus.AUTHENTICATED
assert bridge.error == "boom"
+40 -5
View File
@@ -11,7 +11,8 @@ import pytest
from aiohttp import web
from unshackle.core.api.compression import compression_middleware
from unshackle.core.api.routes import cors_middleware, setup_routes
from unshackle.core.api.errors import APIError, APIErrorCode
from unshackle.core.api.routes import api_handler, cors_middleware, setup_routes
pytestmark = pytest.mark.unit
@@ -88,10 +89,44 @@ def test_setup_routes_remote_only_excludes_list_and_download(make_app) -> None:
assert ("POST", "/api/list-tracks") not in paths
assert ("POST", "/api/download") not in paths
assert ("GET", "/api/download/jobs") not in paths
# session endpoints still present
assert ("POST", "/api/session/create") in paths
assert ("GET", "/api/session/{session_id}/titles") in paths
assert ("POST", "/api/session/{session_id}/license") in paths
# The full remote subset is still registered after the single-table collapse.
remote_subset = {
("GET", "/api/health"),
("GET", "/api/services"),
("POST", "/api/search"),
("POST", "/api/session/create"),
("GET", "/api/session/{session_id}/titles"),
("POST", "/api/session/{session_id}/tracks"),
("POST", "/api/session/{session_id}/segments"),
("POST", "/api/session/{session_id}/license"),
("GET", "/api/session/{session_id}/prompt"),
("POST", "/api/session/{session_id}/prompt"),
("GET", "/api/session/{session_id}"),
("DELETE", "/api/session/{session_id}"),
}
assert remote_subset.issubset(paths)
async def test_api_handler_maps_apierror(aiohttp_client) -> None:
"""The error wrapper turns a raised APIError into the same structured response
the per-handler except blocks produced (status + code from APIError)."""
@api_handler
async def boom(request: web.Request) -> web.Response:
raise APIError(APIErrorCode.NOT_FOUND, "nope", details={"x": 1})
app = web.Application()
app["debug_api"] = False
app.router.add_get("/boom", boom)
client = await aiohttp_client(app)
resp = await client.get("/boom")
assert resp.status == 404 # NOT_FOUND -> 404 per errors.APIError._default_http_status
body = await resp.json()
assert body["status"] == "error"
assert body["error_code"] == "NOT_FOUND"
assert body["message"] == "nope"
assert body["details"] == {"x": 1}
async def test_cors_preflight_returns_headers(make_app, aiohttp_client) -> None:
+15
View File
@@ -85,6 +85,21 @@ async def test_cleanup_expired_keeps_pending_input_under_grace(store: SessionSto
assert store.session_count == 1
async def test_cleanup_expired_drops_pending_input_past_grace(store: SessionStore) -> None:
"""The auth-state grace is the single AUTH_INPUT_TIMEOUT rule, not a separate magic number."""
from datetime import datetime, timedelta, timezone
from unshackle.core.api.input_bridge import AUTH_INPUT_TIMEOUT
entry = await store.create("ATV", _FakeService())
entry.input_bridge = InputBridge()
entry.auth_status = AuthStatus.PENDING_INPUT
entry.last_accessed = datetime.now(timezone.utc) - timedelta(seconds=AUTH_INPUT_TIMEOUT + 1)
removed = await store.cleanup_expired()
assert removed == 1
assert store.session_count == 0
async def test_cancel_all_bridges(store: SessionStore) -> None:
a = await store.create("ATV", _FakeService())
b = await store.create("NF", _FakeService())
+82 -22
View File
@@ -58,7 +58,7 @@ from unshackle.core.music import (
write_music_manifest,
write_music_metadata,
)
from unshackle.core.proxies import Basic, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
from unshackle.core.proxies import Basic, ExpressVPN, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
from unshackle.core.service import Service
from unshackle.core.services import Services
from unshackle.core.title_cacher import get_account_hash
@@ -100,6 +100,22 @@ def normalize_dl_config(dl_config: dict[str, Any]) -> dict[str, Any]:
return {DL_OPTION_ALIASES.get(key, key): value for key, value in dl_config.items()}
def group_videos_by_variant(videos: list[Video], *, merge: bool) -> list[list[Video]]:
"""Group video tracks for muxing.
When ``merge`` is True, tracks sharing ``(height, range, codec)`` are grouped into one
file so only language varies within a group; different resolutions, ranges and codecs
stay in separate groups (separate files). When False, each track is its own group
(one file per track, the default behaviour). Group order follows first-seen track order.
"""
if not merge:
return [[video] for video in videos]
groups: dict[tuple[Any, ...], list[Video]] = {}
for video in videos:
groups.setdefault((video.height, video.range, video.codec), []).append(video)
return list(groups.values())
def apply_service_dl_overrides(ctx: click.Context, service_dl_config: dict[str, Any], log: logging.Logger) -> None:
"""Apply ``services.<TAG>.dl`` config onto ``ctx.params``. Explicit CLI/env values win;
defaults and global ``dl:`` default_map values are replaced."""
@@ -459,6 +475,13 @@ class dl:
default=None,
help="Create separate output files per audio codec instead of merging all audio.",
)
@click.option(
"--merge-video",
"merge_video",
is_flag=True,
default=None,
help="Mux all selected video tracks into a single file instead of one file per track.",
)
@click.option(
"--select-titles",
is_flag=True,
@@ -1027,6 +1050,8 @@ class dl:
with console.status("Loading Proxy Providers...", spinner="dots"):
if config.proxy_providers.get("basic"):
self.proxy_providers.append(Basic(**config.proxy_providers["basic"]))
if config.proxy_providers.get("expressvpn"):
self.proxy_providers.append(ExpressVPN(**config.proxy_providers["expressvpn"]))
if config.proxy_providers.get("nordvpn"):
self.proxy_providers.append(NordVPN(**config.proxy_providers["nordvpn"]))
if config.proxy_providers.get("surfsharkvpn"):
@@ -1046,8 +1071,8 @@ class dl:
# requesting proxy from a specific proxy provider
requested_provider, proxy = proxy.split(":", maxsplit=1)
# Match simple region codes (us, ca, uk1) or provider:region format (nordvpn:ca, windscribe:us)
if re.match(r"^[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
r"^[a-z]+:[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE
if re.match(r"^[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
r"^[a-z]+:[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE
):
proxy = proxy.lower()
# Preserve the original user query (region code) for service-specific proxy_map overrides.
@@ -1082,7 +1107,13 @@ class dl:
else:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
else:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
display = None
if hasattr(proxy_provider, "last_connection_display"):
display = proxy_provider.last_connection_display()
if display:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
else:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
else:
for proxy_provider in self.proxy_providers:
proxy_uri = proxy_provider.get_proxy(proxy)
@@ -1098,7 +1129,13 @@ class dl:
else:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
else:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
display = None
if hasattr(proxy_provider, "last_connection_display"):
display = proxy_provider.last_connection_display()
if display:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
else:
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
break
# Store proxy query info for service-specific overrides
ctx.params["proxy_query"] = proxy_query
@@ -1173,6 +1210,7 @@ class dl:
worst: bool,
best_available: bool,
split_audio: Optional[bool] = None,
merge_video: Optional[bool] = None,
real_video_bitrate: bool = False,
real_audio_bitrate: bool = False,
progress_sink: Optional[Callable[[dict[str, Any]], None]] = None,
@@ -3062,6 +3100,9 @@ class dl:
# When we split audio (merge_audio=False), multiple outputs may exist per title, so suffix codec.
append_audio_codec_suffix = not merge_audio
# Mux all selected video tracks into one file instead of one file per track.
merge_video = merge_video if merge_video is not None else config.muxing.get("merge_video", False)
multiplex_tasks: list[tuple[TaskID, Tracks, Optional[Audio.Codec]]] = []
# Track hybrid-processing outputs explicitly so we can always clean them up,
# even if muxing fails early (e.g. SystemExit) before the normal delete loop.
@@ -3097,22 +3138,25 @@ class dl:
task_tracks = clone_tracks_for_audio(base_tracks, codec_audio_tracks)
multiplex_tasks.append((task_id, task_tracks, audio_codec))
def mux_video_standalone(video_track: Optional[Video]) -> None:
if video_track and video_track.dv_compatible_bitstream:
apply_dv_fixup(video_track)
def mux_video_group(video_tracks: list[Optional[Video]]) -> None:
for video_track in video_tracks:
if video_track and video_track.dv_compatible_bitstream:
apply_dv_fixup(video_track)
task_description = "Multiplexing"
if video_track:
# All tracks in a merged group share height/range/codec, so describe from the first.
head = next((v for v in video_tracks if v), None)
if head:
if len(quality) > 1:
task_description += f" {video_track.height}p"
task_description += f" {head.height}p"
if len(range_) > 1:
task_description += f" {video_track.range.name}"
task_description += f" {head.range.name}"
if len(vcodec) > 1:
task_description += f" {video_track.codec.name}"
task_description += f" {head.codec.name}"
task_tracks = Tracks(title.tracks) + title.tracks.chapters + title.tracks.attachments
if video_track:
task_tracks.videos = [video_track]
if head:
task_tracks.videos = [v for v in video_tracks if v]
enqueue_mux_tasks(task_description, task_tracks)
@@ -3172,16 +3216,18 @@ class dl:
enqueue_mux_tasks(task_description, task_tracks)
# Mux every requested range standalone, skipping the ingredient-only DV.
for video_track in original_videos:
if video_track.hybrid_base_only:
continue
mux_video_standalone(video_track)
# merge_video collapses only language variants (same height/range/codec).
standalone_videos = [v for v in original_videos if not v.hybrid_base_only]
for group in group_videos_by_variant(standalone_videos, merge=merge_video):
mux_video_group(group)
console.print()
else:
# Normal mode: process each video track separately
for video_track in title.tracks.videos or [None]:
mux_video_standalone(video_track)
# Normal mode: one file per video track, unless merge_video groups
# same-(height, range, codec) language variants into one file.
groups = group_videos_by_variant(title.tracks.videos, merge=merge_video)
for group in groups or [[None]]:
mux_video_group(group)
if progress_sink:
progress_sink(
@@ -4111,8 +4157,20 @@ class dl:
return default_cookie_file
@staticmethod
def get_cookie_jar(service: str, profile: Optional[str]) -> Optional[MozillaCookieJar]:
def get_cookie_jar(service: str, profile: Optional[str]) -> Optional[CookieJar]:
"""Get Service Cookies for Profile."""
# Check if automatic Firefox cookie extraction is configured for this service
ff_settings = getattr(config, "firefox_cookies", {}).get(service)
if ff_settings:
try:
from unshackle.core.utils.firefox_cookie_extractor import get_firefox_cookies
extracted_jar = get_firefox_cookies(ff_settings)
if extracted_jar:
return extracted_jar
except Exception:
# Fallback to file-based if Firefox extraction fails
pass
cookie_file = dl.get_cookie_path(service, profile)
if cookie_file:
cookie_jar = MozillaCookieJar(cookie_file)
@@ -4129,6 +4187,8 @@ class dl:
cookie_file.write_text(cookie_data, "utf8")
cookie_jar.load(ignore_discard=True, ignore_expires=True)
return cookie_jar
return None
@staticmethod
def save_cookies(path: Path, cookies: CookieJar):
+19 -5
View File
@@ -16,7 +16,7 @@ from unshackle.core import binaries
from unshackle.core.config import config
from unshackle.core.console import console
from unshackle.core.constants import context_settings
from unshackle.core.proxies import Basic, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
from unshackle.core.proxies import Basic, ExpressVPN, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
from unshackle.core.service import Service
from unshackle.core.services import Services
from unshackle.core.utils.click_types import ContextData
@@ -67,6 +67,8 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
with console.status("Loading Proxy Providers...", spinner="dots"):
if config.proxy_providers.get("basic"):
proxy_providers.append(Basic(**config.proxy_providers["basic"]))
if config.proxy_providers.get("expressvpn"):
proxy_providers.append(ExpressVPN(**config.proxy_providers["expressvpn"]))
if config.proxy_providers.get("nordvpn"):
proxy_providers.append(NordVPN(**config.proxy_providers["nordvpn"]))
if config.proxy_providers.get("surfsharkvpn"):
@@ -86,8 +88,8 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
# requesting proxy from a specific proxy provider
requested_provider, proxy = proxy.split(":", maxsplit=1)
# Match simple region codes (us, ca, uk1) or provider:region format (nordvpn:ca, windscribe:us)
if re.match(r"^[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
r"^[a-z]+:[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE
if re.match(r"^[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
r"^[a-z]+:[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE
):
proxy = proxy.lower()
with console.status(f"Getting a Proxy to {proxy}...", spinner="dots"):
@@ -103,13 +105,25 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
log.error(f"The proxy provider {requested_provider} had no proxy for {proxy}")
sys.exit(1)
proxy = ctx.params["proxy"] = proxy_uri
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
display = None
if hasattr(proxy_provider, "last_connection_display"):
display = proxy_provider.last_connection_display()
if display:
log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
else:
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
else:
for proxy_provider in proxy_providers:
proxy_uri = proxy_provider.get_proxy(proxy)
if proxy_uri:
proxy = ctx.params["proxy"] = proxy_uri
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
display = None
if hasattr(proxy_provider, "last_connection_display"):
display = proxy_provider.last_connection_display()
if display:
log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
else:
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
break
else:
log.info(f"Using explicit Proxy: {proxy}")
+28 -38
View File
@@ -104,6 +104,10 @@ class DownloadJob:
completed_tracks: int = 0
total_tracks: int = 0
active_tracks: List[str] = field(default_factory=list)
# Segment counts and transfer speed of the track downloading now (granular display)
segments_done: float = 0.0
segments_total: float = 0.0
speed: Optional[str] = None
# Subtitles skipped under skip_subtitle_errors (non-fatal). Each entry is a dl.SkippedSubtitle
# dict (id / language / title) so a client can report which weren't available.
@@ -125,6 +129,9 @@ class DownloadJob:
"completed_tracks": self.completed_tracks,
"total_tracks": self.total_tracks,
"active_tracks": self.active_tracks,
"segments_done": self.segments_done,
"segments_total": self.segments_total,
"speed": self.speed,
"skipped_subtitles": self.skipped_subtitles,
}
@@ -148,25 +155,25 @@ class DownloadJob:
return result
def to_enum(values: List[str], enum_cls: type[Enum]) -> List[Enum]:
"""Map case-insensitive strings (by member name or value) to enum members, dropping unknowns."""
lookup = {m.name.upper(): m for m in enum_cls}
lookup.update({str(m.value).upper(): m for m in enum_cls})
return [lookup[v.upper()] for v in values if v.upper() in lookup]
def _perform_download(
job_id: str,
service: str,
title_id: str,
params: Dict[str, Any],
cancel_event: Optional[threading.Event] = None,
progress_callback: Optional[Callable[[Dict[str, Any]], None]] = None,
) -> List[str]:
"""Execute the synchronous download logic for a job."""
def _check_cancel(stage: str):
if cancel_event and cancel_event.is_set():
raise Exception(f"Job was cancelled {stage}")
from contextlib import redirect_stderr, redirect_stdout
from io import StringIO
_check_cancel("before execution started")
# Import dl.py components lazily to avoid circular deps during module import
import click
import yaml
@@ -197,9 +204,7 @@ def _perform_download(
if isinstance(vcodec_raw, str):
vcodec_raw = [vcodec_raw]
if isinstance(vcodec_raw, list) and vcodec_raw and not isinstance(vcodec_raw[0], Video.Codec):
codec_map = {c.name.upper(): c for c in Video.Codec}
codec_map.update({c.value.upper(): c for c in Video.Codec})
params["vcodec"] = [codec_map[v.upper()] for v in vcodec_raw if v.upper() in codec_map]
params["vcodec"] = to_enum(vcodec_raw, Video.Codec)
else:
params["vcodec"] = []
@@ -208,17 +213,14 @@ def _perform_download(
if isinstance(range_raw, str):
range_raw = [range_raw]
if isinstance(range_raw, list) and range_raw and not isinstance(range_raw[0], Video.Range):
range_map = {r.name.upper(): r for r in Video.Range}
range_map.update({r.value.upper(): r for r in Video.Range})
params["range"] = [range_map[r.upper()] for r in range_raw if r.upper() in range_map]
params["range"] = to_enum(range_raw, Video.Range)
else:
params["range"] = [Video.Range.SDR]
sub_format_raw = params.get("sub_format")
if sub_format_raw and isinstance(sub_format_raw, str):
sub_map = {c.name.upper(): c for c in Subtitle.Codec}
sub_map.update({c.value.upper(): c for c in Subtitle.Codec})
params["sub_format"] = sub_map.get(sub_format_raw.upper())
mapped = to_enum([sub_format_raw], Subtitle.Codec)
params["sub_format"] = mapped[0] if mapped else None
if params.get("export"):
params["export"] = bool(params["export"])
@@ -321,8 +323,6 @@ def _perform_download(
service_module = Services.load(service)
_check_cancel("before service instantiation")
try:
import inspect
@@ -360,29 +360,11 @@ def _perform_download(
original_download_dir = config.directories.downloads
_check_cancel("before download execution")
stdout_capture = StringIO()
stderr_capture = StringIO()
# The progress_sink (dl.build_job_progress_callables) owns the percentage; status changes
# are emitted here.
if progress_callback:
progress_callback({"progress": 0.0, "status": "starting"})
original_result = dl_instance.result
def result_with_progress(*args, **kwargs):
try:
progress_callback({"status": "downloading"})
result = original_result(*args, **kwargs)
progress_callback({"progress": 100.0, "status": "completed"})
return result
except Exception as e:
progress_callback({"progress": 0.0, "status": "failed", "error": str(e)})
raise
dl_instance.result = result_with_progress
# progress_sink (dl.build_job_progress_callables) owns percentage; terminal status
# (success/failure) is reported by the worker's result file.
try:
with redirect_stdout(stdout_capture), redirect_stderr(stderr_capture):
dl_instance.result(
@@ -756,6 +738,14 @@ class DownloadQueueManager:
job.completed_tracks = int(progress_data["completed_tracks"])
if "active_tracks" in progress_data:
job.active_tracks = list(progress_data["active_tracks"])
for _sk in ("segments_done", "segments_total"):
if _sk in progress_data:
try:
setattr(job, _sk, float(progress_data[_sk]))
except (TypeError, ValueError):
pass
if progress_data.get("speed"):
job.speed = str(progress_data["speed"])
if progress_data.get("skipped_subtitles"):
job.skipped_subtitles = progress_data["skipped_subtitles"]
if "progress" in progress_data:
+1 -1
View File
@@ -60,7 +60,7 @@ def main(argv: list[str]) -> int:
log.error(f"Failed to write progress update: {e}")
output_files = _perform_download(
job_id, service, title_id, params, cancel_event=None, progress_callback=progress_callback
job_id, service, title_id, params, progress_callback=progress_callback
)
result = {"status": "success", "output_files": output_files}
-8
View File
@@ -21,14 +21,10 @@ class APIErrorCode(str, Enum):
# Client errors (4xx)
INVALID_INPUT = "INVALID_INPUT" # Missing or malformed request data
INVALID_SERVICE = "INVALID_SERVICE" # Unknown service name
INVALID_TITLE_ID = "INVALID_TITLE_ID" # Invalid or malformed title ID
INVALID_PROFILE = "INVALID_PROFILE" # Profile doesn't exist
INVALID_PROXY = "INVALID_PROXY" # Invalid proxy specification
INVALID_LANGUAGE = "INVALID_LANGUAGE" # Invalid language code
INVALID_PARAMETERS = "INVALID_PARAMETERS" # Invalid download parameters
AUTH_FAILED = "AUTH_FAILED" # Authentication failure (invalid credentials/cookies)
AUTH_REQUIRED = "AUTH_REQUIRED" # Missing authentication
FORBIDDEN = "FORBIDDEN" # Action not allowed
GEOFENCE = "GEOFENCE" # Content not available in region
@@ -84,13 +80,9 @@ class APIError(Exception):
# 400 Bad Request
APIErrorCode.INVALID_INPUT: 400,
APIErrorCode.INVALID_SERVICE: 400,
APIErrorCode.INVALID_TITLE_ID: 400,
APIErrorCode.INVALID_PROFILE: 400,
APIErrorCode.INVALID_PROXY: 400,
APIErrorCode.INVALID_LANGUAGE: 400,
APIErrorCode.INVALID_PARAMETERS: 400,
# 401 Unauthorized
APIErrorCode.AUTH_REQUIRED: 401,
APIErrorCode.AUTH_FAILED: 401,
# 403 Forbidden
APIErrorCode.FORBIDDEN: 403,
+143 -257
View File
@@ -1,6 +1,7 @@
import asyncio
import enum
import logging
from types import SimpleNamespace
from typing import Any, Dict, List, Optional
from aiohttp import web
@@ -200,6 +201,44 @@ def instantiate_service(
return parent_ctx.invoke(service_module.cli, title=title, **extras)
def setup_list_service(data: Dict[str, Any], normalized_service: str, profile: Optional[str], title_id: str) -> Any:
"""Build and authenticate a service instance for list_titles / list_tracks.
Runs the shared preamble: load yaml → resolve proxy → load CDM → build ctx →
instantiate → authenticate. Raises APIError on proxy failure.
"""
from unshackle.commands.dl import dl
service_config = load_service_yaml(normalized_service)
proxy_param = data.get("proxy")
no_proxy = data.get("no_proxy", False)
proxy_providers = []
if not no_proxy:
proxy_providers = initialize_proxy_providers()
if proxy_param and not no_proxy:
try:
proxy_param = resolve_proxy(proxy_param, proxy_providers)
except ValueError as e:
raise APIError(
APIErrorCode.INVALID_PROXY,
f"Proxy error: {e}",
details={"proxy": proxy_param, "service": normalized_service},
)
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
service_module = Services.load(normalized_service)
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
cookies = dl.get_cookie_jar(normalized_service, profile)
credential = dl.get_credentials(normalized_service, profile)
service_instance.authenticate(cookies, credential)
return service_instance
def get_allowed_services(request: Optional[web.Request] = None) -> Optional[List[str]]:
"""Get effective service allowlist considering global + per-key config.
@@ -247,37 +286,50 @@ def validate_service(service_tag: str, request: Optional[web.Request] = None) ->
return None
def require_fields(data: Dict[str, Any], *names: str) -> None:
"""Raise INVALID_INPUT for the first missing/falsy required field."""
for name in names:
if not data.get(name):
raise APIError(
APIErrorCode.INVALID_INPUT,
f"Missing required parameter: {name}",
details={"missing_parameter": name},
)
def serialize_title(title: Title_T) -> Dict[str, Any]:
"""Convert a title object to JSON-serializable dict."""
title_language = str(title.language) if hasattr(title, "language") and title.language else None
# Optional display metadata a service may provide: a synopsis (title.description) and a
# release/air date or poster image URL stashed in title.data. Surfaced so a client can show
# a richer listing without re-fetching the page.
description = getattr(title, "description", None) or None
_data = getattr(title, "data", None)
date = _data.get("date") if isinstance(_data, dict) else None
cover_url = _data.get("cover_url") if isinstance(_data, dict) else None
if isinstance(title, Episode):
episode_name = title.name if title.name else f"Episode {title.number:02d}"
result = {
"type": "episode",
"name": episode_name,
"series_title": str(title.title),
"season": title.season,
"number": title.number,
"year": title.year,
"id": str(title.id) if hasattr(title, "id") else None,
"language": title_language,
}
elif isinstance(title, Movie):
result = {
"type": "movie",
"name": str(title.name) if hasattr(title, "name") else str(title),
"year": title.year,
"id": str(title.id) if hasattr(title, "id") else None,
"language": title_language,
}
is_episode = isinstance(title, Episode)
if is_episode:
name = title.name if title.name else f"Episode {title.number:02d}"
else:
result = {
"type": "other",
"name": str(title.name) if hasattr(title, "name") else str(title),
"id": str(title.id) if hasattr(title, "id") else None,
"language": title_language,
}
name = str(title.name) if hasattr(title, "name") else str(title)
result = {
"type": "episode" if is_episode else "movie" if isinstance(title, Movie) else "other",
"name": name,
"id": str(title.id) if hasattr(title, "id") else None,
"language": title_language,
"description": description,
"date": date,
"cover_url": cover_url,
}
# "other" titles carry no year; only Episode/Movie do.
if isinstance(title, (Episode, Movie)):
result["year"] = title.year
if is_episode:
result["series_title"] = str(title.title)
result["season"] = title.season
result["number"] = title.number
return result
@@ -346,46 +398,19 @@ def serialize_drm(drm_list) -> Optional[List[Dict[str, Any]]]:
drm_class = drm.__class__.__name__
drm_info["type"] = drm_class.lower()
# Get PSSH - handle both Widevine and PlayReady
if hasattr(drm, "_pssh") and drm._pssh:
pssh_obj = None
# PSSH: pywidevine exposes dumps(); pyplayready's PSSH has no base64 method
# here, so PlayReady omits the field (unchanged from prior behaviour).
pssh_obj = getattr(drm, "_pssh", None)
if pssh_obj is not None and hasattr(pssh_obj, "dumps"):
try:
pssh_obj = drm._pssh
# Try to get base64 representation
if hasattr(pssh_obj, "dumps"):
# pywidevine PSSH has dumps() method
drm_info["pssh"] = pssh_obj.dumps()
elif hasattr(pssh_obj, "__bytes__"):
# Convert to base64
import base64
drm_info["pssh"] = base64.b64encode(bytes(pssh_obj)).decode()
elif hasattr(pssh_obj, "to_base64"):
drm_info["pssh"] = pssh_obj.to_base64()
else:
# Fallback - str() works for pywidevine PSSH
pssh_str = str(pssh_obj)
# Check if it's already base64-like or an object repr
if not pssh_str.startswith("<"):
drm_info["pssh"] = pssh_str
drm_info["pssh"] = pssh_obj.dumps()
except (ValueError, TypeError, KeyError):
# Some PSSH implementations can fail to parse/serialize; log and continue.
pssh_type = type(pssh_obj).__name__ if pssh_obj is not None else None
log.warning(
"Failed to extract/serialize PSSH for DRM type=%s pssh_type=%s",
"Failed to serialize PSSH for DRM type=%s pssh_type=%s",
drm_class,
pssh_type,
type(pssh_obj).__name__,
exc_info=True,
)
except Exception:
# Don't silently swallow unexpected failures; make them visible and propagate.
pssh_type = type(pssh_obj).__name__ if pssh_obj is not None else None
log.exception(
"Unexpected error while extracting/serializing PSSH for DRM type=%s pssh_type=%s",
drm_class,
pssh_type,
)
raise
# Get KIDs
if hasattr(drm, "kids") and drm.kids:
@@ -406,15 +431,21 @@ def serialize_drm(drm_list) -> Optional[List[Dict[str, Any]]]:
return result if result else None
def enum_name(value: Any) -> str:
"""Return an enum-like value's .name, falling back to str()."""
return value.name if hasattr(value, "name") else str(value)
def descriptor_name(track: Any) -> Optional[str]:
"""Manifest descriptor (HLS/DASH/URL) name for a track, or None."""
descriptor = getattr(track, "descriptor", None)
return enum_name(descriptor) if descriptor else None
def serialize_video_track(track: Video, include_url: bool = False) -> Dict[str, Any]:
"""Convert video track to JSON-serializable dict."""
codec_name = track.codec.name if hasattr(track.codec, "name") else str(track.codec)
range_name = track.range.name if hasattr(track.range, "name") else str(track.range)
# Serialize the manifest descriptor (HLS, DASH, URL, etc.)
descriptor_name = None
if hasattr(track, "descriptor") and track.descriptor:
descriptor_name = track.descriptor.name if hasattr(track.descriptor, "name") else str(track.descriptor)
codec_name = enum_name(track.codec)
range_name = enum_name(track.range)
result = {
"id": str(track.id),
@@ -429,7 +460,7 @@ def serialize_video_track(track: Video, include_url: bool = False) -> Dict[str,
"range_display": DYNAMIC_RANGE_MAP.get(range_name, range_name),
"language": str(track.language) if track.language else None,
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
"descriptor": descriptor_name,
"descriptor": descriptor_name(track),
}
if include_url and hasattr(track, "url") and track.url:
result["url"] = str(track.url)
@@ -438,12 +469,7 @@ def serialize_video_track(track: Video, include_url: bool = False) -> Dict[str,
def serialize_audio_track(track: Audio, include_url: bool = False) -> Dict[str, Any]:
"""Convert audio track to JSON-serializable dict."""
codec_name = track.codec.name if hasattr(track.codec, "name") else str(track.codec)
# Serialize the manifest descriptor (HLS, DASH, URL, etc.)
descriptor_name = None
if hasattr(track, "descriptor") and track.descriptor:
descriptor_name = track.descriptor.name if hasattr(track.descriptor, "name") else str(track.descriptor)
codec_name = enum_name(track.codec)
result = {
"id": str(track.id),
@@ -455,7 +481,7 @@ def serialize_audio_track(track: Audio, include_url: bool = False) -> Dict[str,
"atmos": track.atmos if hasattr(track, "atmos") else False,
"descriptive": track.descriptive if hasattr(track, "descriptive") else False,
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
"descriptor": descriptor_name,
"descriptor": descriptor_name(track),
}
if include_url and hasattr(track, "url") and track.url:
result["url"] = str(track.url)
@@ -464,19 +490,14 @@ def serialize_audio_track(track: Audio, include_url: bool = False) -> Dict[str,
def serialize_subtitle_track(track: Subtitle, include_url: bool = False) -> Dict[str, Any]:
"""Convert subtitle track to JSON-serializable dict."""
# Get descriptor for compatibility
descriptor_name = None
if hasattr(track, "descriptor") and track.descriptor:
descriptor_name = track.descriptor.name if hasattr(track.descriptor, "name") else str(track.descriptor)
result = {
"id": str(track.id),
"codec": track.codec.name if hasattr(track.codec, "name") else str(track.codec),
"codec": enum_name(track.codec),
"language": str(track.language) if track.language else None,
"forced": track.forced if hasattr(track, "forced") else False,
"sdh": track.sdh if hasattr(track, "sdh") else False,
"cc": track.cc if hasattr(track, "cc") else False,
"descriptor": descriptor_name,
"descriptor": descriptor_name(track),
}
if include_url and hasattr(track, "url") and track.url:
result["url"] = str(track.url)
@@ -575,24 +596,11 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] =
async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response:
"""Handle list-titles request."""
require_fields(data, "service", "title_id")
service_tag = data.get("service")
title_id = data.get("title_id")
profile = data.get("profile")
if not service_tag:
raise APIError(
APIErrorCode.INVALID_INPUT,
"Missing required parameter: service",
details={"missing_parameter": "service"},
)
if not title_id:
raise APIError(
APIErrorCode.INVALID_INPUT,
"Missing required parameter: title_id",
details={"missing_parameter": "title_id"},
)
normalized_service = validate_service(service_tag, request)
if not normalized_service:
raise APIError(
@@ -602,37 +610,7 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
)
try:
from unshackle.commands.dl import dl
service_config = load_service_yaml(normalized_service)
proxy_param = data.get("proxy")
no_proxy = data.get("no_proxy", False)
proxy_providers = []
if not no_proxy:
proxy_providers = initialize_proxy_providers()
if proxy_param and not no_proxy:
try:
resolved_proxy = resolve_proxy(proxy_param, proxy_providers)
proxy_param = resolved_proxy
except ValueError as e:
raise APIError(
APIErrorCode.INVALID_PROXY,
f"Proxy error: {e}",
details={"proxy": proxy_param, "service": normalized_service},
)
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
service_module = Services.load(normalized_service)
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
cookies = dl.get_cookie_jar(normalized_service, profile)
credential = dl.get_credentials(normalized_service, profile)
service_instance.authenticate(cookies, credential)
service_instance = setup_list_service(data, normalized_service, profile, title_id)
titles = service_instance.get_titles()
if hasattr(titles, "__iter__") and not isinstance(titles, str):
@@ -656,24 +634,11 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response:
"""Handle list-tracks request."""
require_fields(data, "service", "title_id")
service_tag = data.get("service")
title_id = data.get("title_id")
profile = data.get("profile")
if not service_tag:
raise APIError(
APIErrorCode.INVALID_INPUT,
"Missing required parameter: service",
details={"missing_parameter": "service"},
)
if not title_id:
raise APIError(
APIErrorCode.INVALID_INPUT,
"Missing required parameter: title_id",
details={"missing_parameter": "title_id"},
)
normalized_service = validate_service(service_tag, request)
if not normalized_service:
raise APIError(
@@ -683,37 +648,7 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
)
try:
from unshackle.commands.dl import dl
service_config = load_service_yaml(normalized_service)
proxy_param = data.get("proxy")
no_proxy = data.get("no_proxy", False)
proxy_providers = []
if not no_proxy:
proxy_providers = initialize_proxy_providers()
if proxy_param and not no_proxy:
try:
resolved_proxy = resolve_proxy(proxy_param, proxy_providers)
proxy_param = resolved_proxy
except ValueError as e:
raise APIError(
APIErrorCode.INVALID_PROXY,
f"Proxy error: {e}",
details={"proxy": proxy_param, "service": normalized_service},
)
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
service_module = Services.load(normalized_service)
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
cookies = dl.get_cookie_jar(normalized_service, profile)
credential = dl.get_credentials(normalized_service, profile)
service_instance.authenticate(cookies, credential)
service_instance = setup_list_service(data, normalized_service, profile, title_id)
titles = service_instance.get_titles()
wanted_param = data.get("wanted")
@@ -855,6 +790,25 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
)
VALID_VCODECS = ["H264", "H265", "H.264", "H.265", "AVC", "HEVC", "VC1", "VC-1", "VP8", "VP9", "AV1"]
VALID_ACODECS = ["AAC", "AC3", "EC3", "EAC3", "DD", "DD+", "AC4", "OPUS", "FLAC", "ALAC", "VORBIS", "OGG", "DTS"]
def check_codec(value: Any, allowed: List[str], name: str) -> Optional[str]:
"""Validate a comma-string or list of codec tokens against `allowed` (case-insensitive)."""
if isinstance(value, str):
tokens = [v.strip() for v in value.split(",") if v.strip()]
elif isinstance(value, list):
tokens = [str(v).strip() for v in value if str(v).strip()]
else:
return f"{name} must be a string or list"
invalid = [token for token in tokens if token.upper() not in allowed]
if invalid:
return f"Invalid {name}: {', '.join(invalid)}. Must be one of: {', '.join(allowed)}"
return None
def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
"""
Validate download parameters and return error message if invalid.
@@ -863,44 +817,14 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
None if valid, error message string if invalid
"""
if "vcodec" in data and data["vcodec"]:
valid_vcodecs = ["H264", "H265", "H.264", "H.265", "AVC", "HEVC", "VC1", "VC-1", "VP8", "VP9", "AV1"]
if isinstance(data["vcodec"], str):
vcodec_values = [v.strip() for v in data["vcodec"].split(",") if v.strip()]
elif isinstance(data["vcodec"], list):
vcodec_values = [str(v).strip() for v in data["vcodec"] if str(v).strip()]
else:
return "vcodec must be a string or list"
invalid = [value for value in vcodec_values if value.upper() not in valid_vcodecs]
if invalid:
return f"Invalid vcodec: {', '.join(invalid)}. Must be one of: {', '.join(valid_vcodecs)}"
err = check_codec(data["vcodec"], VALID_VCODECS, "vcodec")
if err:
return err
if "acodec" in data and data["acodec"]:
valid_acodecs = [
"AAC",
"AC3",
"EC3",
"EAC3",
"DD",
"DD+",
"AC4",
"OPUS",
"FLAC",
"ALAC",
"VORBIS",
"OGG",
"DTS",
]
if isinstance(data["acodec"], str):
acodec_values = [v.strip() for v in data["acodec"].split(",") if v.strip()]
elif isinstance(data["acodec"], list):
acodec_values = [str(v).strip() for v in data["acodec"] if str(v).strip()]
else:
return "acodec must be a string or list"
invalid = [value for value in acodec_values if value.upper() not in valid_acodecs]
if invalid:
return f"Invalid acodec: {', '.join(invalid)}. Must be one of: {', '.join(valid_acodecs)}"
err = check_codec(data["acodec"], VALID_ACODECS, "acodec")
if err:
return err
if "sub_format" in data and data["sub_format"]:
valid_sub_formats = ["SRT", "VTT", "ASS", "SSA", "TTML", "STPP", "WVTT", "SMI", "SUB", "MPL2", "TMP"]
@@ -972,23 +896,10 @@ async def download_handler(data: Dict[str, Any], request: Optional[web.Request]
"""Handle download request - create and queue a download job."""
from unshackle.core.api.download_manager import get_download_manager
require_fields(data, "service", "title_id")
service_tag = data.get("service")
title_id = data.get("title_id")
if not service_tag:
raise APIError(
APIErrorCode.INVALID_INPUT,
"Missing required parameter: service",
details={"missing_parameter": "service"},
)
if not title_id:
raise APIError(
APIErrorCode.INVALID_INPUT,
"Missing required parameter: title_id",
details={"missing_parameter": "title_id"},
)
normalized_service = validate_service(service_tag, request)
if not normalized_service:
raise APIError(
@@ -1426,7 +1337,6 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
session.auth_status = AuthStatus.FAILED
session.auth_error = str(e)
bridge.status = AuthStatus.FAILED
bridge.error = str(e)
asyncio.create_task(_run_auth())
@@ -1671,31 +1581,12 @@ async def session_segments_handler(
else:
track_info["cookies"] = {}
# Include manifest-specific data for segment resolution
# Include manifest-specific data for segment resolution. Round-trip through
# JSON so any non-serializable value becomes its str() (default=str).
if hasattr(track, "data") and track.data:
track_data = {}
for key, val in track.data.items():
if isinstance(val, dict):
# Convert non-serializable values
serializable = {}
for k, v in val.items():
try:
import json
import json
json.dumps(v)
serializable[k] = v
except (TypeError, ValueError):
serializable[k] = str(v)
track_data[key] = serializable
else:
try:
import json
json.dumps(val)
track_data[key] = val
except (TypeError, ValueError):
track_data[key] = str(val)
track_info["data"] = track_data
track_info["data"] = json.loads(json.dumps(track.data, default=str))
else:
track_info["data"] = {}
@@ -1715,15 +1606,10 @@ async def session_segments_handler(
)
class _CdmTypeStub:
"""Lightweight CDM stub so ``is_playready_cdm()`` can detect CDM type.
Used on the server when the client sends ``cdm_type`` but the server
does not need a full CDM (e.g. for cache key / device selection only).
"""
def __init__(self, cdm_type: str) -> None:
self.is_playready = cdm_type == "playready"
def _cdm_type_stub(cdm_type: str) -> SimpleNamespace:
"""Lightweight CDM stand-in so ``is_playready_cdm()`` (reads ``.is_playready``)
can detect the type without loading a full CDM."""
return SimpleNamespace(is_playready=cdm_type == "playready")
def _resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]) -> Optional[Any]:
@@ -1751,10 +1637,10 @@ def _resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional
if cdm_name and isinstance(cdm_name, str):
detected_type = _detect_cdm_type(cdm_name, app_config)
if detected_type:
return _CdmTypeStub(detected_type)
return _cdm_type_stub(detected_type)
if cdm_type:
return _CdmTypeStub(cdm_type)
return _cdm_type_stub(cdm_type)
return None
+10 -28
View File
@@ -10,7 +10,6 @@ response value.
from __future__ import annotations
import threading
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional
@@ -25,8 +24,9 @@ class AuthStatus(Enum):
FAILED = "failed"
class BridgeCancelledError(Exception):
"""Raised when the bridge is cancelled (client disconnected, session deleted)."""
# Seconds the server waits for a remote client to answer an input prompt before
# giving up; also the TTL SessionStore grants AUTHENTICATING/PENDING_INPUT sessions.
AUTH_INPUT_TIMEOUT = 600.0
@dataclass
@@ -40,12 +40,11 @@ class InputBridge:
_prompt: Optional[str] = field(default=None, init=False, repr=False)
_response: Optional[str] = field(default=None, init=False, repr=False)
_status: AuthStatus = field(default=AuthStatus.AUTHENTICATING, init=False)
_error: Optional[str] = field(default=None, init=False, repr=False)
_cancelled: bool = field(default=False, init=False, repr=False)
_response_ready: threading.Event = field(default_factory=threading.Event, init=False, repr=False)
_lock: threading.Lock = field(default_factory=threading.Lock, init=False, repr=False)
def request_input(self, prompt: str, timeout: float = 600.0) -> str:
def request_input(self, prompt: str, timeout: float = AUTH_INPUT_TIMEOUT) -> str:
"""Block until the remote client submits a response for *prompt*.
Args:
@@ -57,32 +56,26 @@ class InputBridge:
Raises:
TimeoutError: If no response is received within *timeout*.
BridgeCancelledError: If the bridge was cancelled.
RuntimeError: If the bridge was cancelled.
"""
with self._lock:
if self._cancelled:
raise BridgeCancelledError("Session was cancelled")
raise RuntimeError("Session was cancelled")
self._prompt = prompt
self._response = None
self._status = AuthStatus.PENDING_INPUT
self._response_ready.clear()
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if self._response_ready.wait(timeout=0.5):
break
with self._lock:
if self._cancelled:
raise BridgeCancelledError("Session was cancelled")
else:
# cancel() and submit_response() both .set() the event, so a single wait
# returns on signal-or-cancel; no poll loop needed.
if not self._response_ready.wait(timeout=timeout):
with self._lock:
self._status = AuthStatus.FAILED
self._error = "Input request timed out waiting for client response"
raise TimeoutError(f"No client response for prompt within {timeout}s")
with self._lock:
if self._cancelled:
raise BridgeCancelledError("Session was cancelled")
raise RuntimeError("Session was cancelled")
response = self._response or ""
self._prompt = None
self._response = None
@@ -115,7 +108,6 @@ class InputBridge:
with self._lock:
self._cancelled = True
self._status = AuthStatus.FAILED
self._error = "Session cancelled"
self._response_ready.set()
@property
@@ -127,13 +119,3 @@ class InputBridge:
def status(self, value: AuthStatus) -> None:
with self._lock:
self._status = value
@property
def error(self) -> Optional[str]:
with self._lock:
return self._error
@error.setter
def error(self, value: Optional[str]) -> None:
with self._lock:
self._error = value
+14 -1
View File
@@ -6,7 +6,7 @@ a bitrate-weighted percentage, track counts, and the labels of the tracks downlo
from __future__ import annotations
from typing import Any, Callable
from typing import Any, Callable, Optional
from unshackle.core.constants import AnyTrack
@@ -70,6 +70,9 @@ def build_job_progress_callables(
fractions = [0.0] * total
done = [False] * total
started = [False] * total
seg_done = [0.0] * total
seg_total = [0.0] * total
speeds: list[Optional[str]] = [None] * total
def emit() -> None:
completed = sum(done)
@@ -82,6 +85,8 @@ def build_job_progress_callables(
phase += f" (+{len(active) - 3} more)"
else:
phase = f"downloading {completed}/{total} tracks"
# segment counts and transfer speed of the track downloading now, for a granular display
active_i = next((i for i in range(total) if started[i] and not done[i]), None)
sink(
{
"progress": progress,
@@ -89,6 +94,9 @@ def build_job_progress_callables(
"completed_tracks": completed,
"total_tracks": total,
"active_tracks": active,
"segments_done": seg_done[active_i] if active_i is not None else 0.0,
"segments_total": seg_total[active_i] if active_i is not None else 0.0,
"speed": speeds[active_i] if active_i is not None else None,
"status": "downloading",
}
)
@@ -104,6 +112,11 @@ def build_job_progress_callables(
counts["completed"] = kwargs["completed"]
if "advance" in kwargs:
counts["completed"] += kwargs["advance"]
seg_done[index] = counts["completed"]
seg_total[index] = counts["total"]
_dl = kwargs.get("downloaded")
if isinstance(_dl, str) and _dl.endswith("/s"):
speeds[index] = _dl
if kwargs.get("downloaded") in JOB_PROGRESS_TERMINAL_STATES:
done[index] = True
fractions[index] = 1.0
+91 -118
View File
@@ -1,5 +1,8 @@
import functools
import logging
import re
from collections.abc import Awaitable, Callable
from typing import Any
import click
from aiohttp import web
@@ -18,9 +21,12 @@ from unshackle.core.update_checker import UpdateChecker
@web.middleware
async def cors_middleware(request: web.Request, handler):
async def cors_middleware(
request: web.Request, handler: Callable[[web.Request], Awaitable[web.StreamResponse]]
) -> web.StreamResponse:
"""Add CORS headers to all responses."""
# Handle preflight requests
response: web.StreamResponse
if request.method == "OPTIONS":
response = web.Response()
else:
@@ -37,7 +43,24 @@ async def cors_middleware(request: web.Request, handler):
log = logging.getLogger("api")
# Route handler signature: takes the request, returns a response.
Handler = Callable[[web.Request], Awaitable[web.Response]]
def api_handler(handler: Handler) -> Handler:
"""Wrap a route handler so any raised APIError becomes a structured error response."""
@functools.wraps(handler)
async def wrapper(request: web.Request) -> web.Response:
try:
return await handler(request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
return wrapper
@api_handler
async def health(request: web.Request) -> web.Response:
"""
Health check endpoint.
@@ -84,6 +107,7 @@ async def health(request: web.Request) -> web.Response:
return web.json_response({"status": "ok", "version": __version__, "update_check": update_info})
@api_handler
async def services(request: web.Request) -> web.Response:
"""
List available services.
@@ -161,7 +185,14 @@ async def services(request: web.Request) -> web.Response:
services_info = []
for tag in service_tags:
service_data = {"tag": tag, "aliases": [], "geofence": [], "title_regex": None, "url": None, "help": None}
service_data: dict[str, Any] = {
"tag": tag,
"aliases": [],
"geofence": [],
"title_regex": None,
"url": None,
"help": None,
}
try:
service_module = Services.load(tag)
@@ -268,6 +299,7 @@ async def services(request: web.Request) -> web.Response:
return handle_api_exception(e, context={"operation": "list_services"}, debug_mode=debug_mode)
@api_handler
async def search(request: web.Request) -> web.Response:
"""
Search for titles from a service.
@@ -347,14 +379,13 @@ async def search(request: web.Request) -> web.Response:
try:
return await search_handler(data, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in search")
debug_mode = request.app.get("debug_api", False)
return handle_api_exception(e, context={"operation": "search"}, debug_mode=debug_mode)
@api_handler
async def list_titles(request: web.Request) -> web.Response:
"""
List titles for a service and title ID.
@@ -474,13 +505,10 @@ async def list_titles(request: web.Request) -> web.Response:
request.app.get("debug_api", False),
)
try:
return await list_titles_handler(data, request)
except APIError as e:
debug_mode = request.app.get("debug_api", False)
return build_error_response(e, debug_mode)
return await list_titles_handler(data, request)
@api_handler
async def list_tracks(request: web.Request) -> web.Response:
"""
List tracks for a title, separated by type.
@@ -527,13 +555,10 @@ async def list_tracks(request: web.Request) -> web.Response:
request.app.get("debug_api", False),
)
try:
return await list_tracks_handler(data, request)
except APIError as e:
debug_mode = request.app.get("debug_api", False)
return build_error_response(e, debug_mode)
return await list_tracks_handler(data, request)
@api_handler
async def download(request: web.Request) -> web.Response:
"""
Download content based on provided parameters.
@@ -767,13 +792,10 @@ async def download(request: web.Request) -> web.Response:
request.app.get("debug_api", False),
)
try:
return await download_handler(data, request)
except APIError as e:
debug_mode = request.app.get("debug_api", False)
return build_error_response(e, debug_mode)
return await download_handler(data, request)
@api_handler
async def download_jobs(request: web.Request) -> web.Response:
"""
List all download jobs with optional filtering and sorting.
@@ -847,13 +869,10 @@ async def download_jobs(request: web.Request) -> web.Response:
"sort_by": request.query.get("sort_by", "created_time"),
"sort_order": request.query.get("sort_order", "desc"),
}
try:
return await list_download_jobs_handler(query_params, request)
except APIError as e:
debug_mode = request.app.get("debug_api", False)
return build_error_response(e, debug_mode)
return await list_download_jobs_handler(query_params, request)
@api_handler
async def download_job_detail(request: web.Request) -> web.Response:
"""
Get download job details.
@@ -875,13 +894,10 @@ async def download_job_detail(request: web.Request) -> web.Response:
description: Server error
"""
job_id = request.match_info["job_id"]
try:
return await get_download_job_handler(job_id, request)
except APIError as e:
debug_mode = request.app.get("debug_api", False)
return build_error_response(e, debug_mode)
return await get_download_job_handler(job_id, request)
@api_handler
async def cancel_download_job(request: web.Request) -> web.Response:
"""
Cancel download job.
@@ -905,13 +921,10 @@ async def cancel_download_job(request: web.Request) -> web.Response:
description: Server error
"""
job_id = request.match_info["job_id"]
try:
return await cancel_download_job_handler(job_id, request)
except APIError as e:
debug_mode = request.app.get("debug_api", False)
return build_error_response(e, debug_mode)
return await cancel_download_job_handler(job_id, request)
@api_handler
async def session_create(request: web.Request) -> web.Response:
"""
Create a remote-dl session.
@@ -964,8 +977,6 @@ async def session_create(request: web.Request) -> web.Response:
)
try:
return await session_create_handler(data, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session create")
return handle_api_exception(
@@ -973,6 +984,7 @@ async def session_create(request: web.Request) -> web.Response:
)
@api_handler
async def session_titles(request: web.Request) -> web.Response:
"""
Get titles for an authenticated session.
@@ -994,8 +1006,6 @@ async def session_titles(request: web.Request) -> web.Response:
session_id = request.match_info["session_id"]
try:
return await session_titles_handler(session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session titles")
return handle_api_exception(
@@ -1003,6 +1013,7 @@ async def session_titles(request: web.Request) -> web.Response:
)
@api_handler
async def session_tracks(request: web.Request) -> web.Response:
"""
Get tracks and chapters for a specific title.
@@ -1043,8 +1054,6 @@ async def session_tracks(request: web.Request) -> web.Response:
)
try:
return await session_tracks_handler(data, session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session tracks")
return handle_api_exception(
@@ -1052,6 +1061,7 @@ async def session_tracks(request: web.Request) -> web.Response:
)
@api_handler
async def session_segments(request: web.Request) -> web.Response:
"""
Resolve segment URLs for selected tracks.
@@ -1094,8 +1104,6 @@ async def session_segments(request: web.Request) -> web.Response:
)
try:
return await session_segments_handler(data, session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session segments")
return handle_api_exception(
@@ -1103,6 +1111,7 @@ async def session_segments(request: web.Request) -> web.Response:
)
@api_handler
async def session_license(request: web.Request) -> web.Response:
"""
Proxy DRM license through authenticated service.
@@ -1151,8 +1160,6 @@ async def session_license(request: web.Request) -> web.Response:
)
try:
return await session_license_handler(data, session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session license")
return handle_api_exception(
@@ -1160,6 +1167,7 @@ async def session_license(request: web.Request) -> web.Response:
)
@api_handler
async def session_info(request: web.Request) -> web.Response:
"""
Get session info.
@@ -1179,12 +1187,10 @@ async def session_info(request: web.Request) -> web.Response:
description: Session not found
"""
session_id = request.match_info["session_id"]
try:
return await session_info_handler(session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
return await session_info_handler(session_id, request)
@api_handler
async def session_delete(request: web.Request) -> web.Response:
"""
Delete a session.
@@ -1204,12 +1210,10 @@ async def session_delete(request: web.Request) -> web.Response:
description: Session not found
"""
session_id = request.match_info["session_id"]
try:
return await session_delete_handler(session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
return await session_delete_handler(session_id, request)
@api_handler
async def session_prompt_get(request: web.Request) -> web.Response:
"""
Poll for pending interactive prompts during authentication.
@@ -1245,8 +1249,6 @@ async def session_prompt_get(request: web.Request) -> web.Response:
session_id = request.match_info["session_id"]
try:
return await session_prompt_get_handler(session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session prompt get")
return handle_api_exception(
@@ -1254,6 +1256,7 @@ async def session_prompt_get(request: web.Request) -> web.Response:
)
@api_handler
async def session_prompt_submit(request: web.Request) -> web.Response:
"""
Submit a response to a pending interactive prompt.
@@ -1304,8 +1307,6 @@ async def session_prompt_submit(request: web.Request) -> web.Response:
)
try:
return await session_prompt_post_handler(data, session_id, request)
except APIError as e:
return build_error_response(e, request.app.get("debug_api", False))
except Exception as e:
log.exception("Error in session prompt submit")
return handle_api_exception(
@@ -1313,48 +1314,42 @@ async def session_prompt_submit(request: web.Request) -> web.Response:
)
def _setup_remote_session_routes(app: web.Application) -> None:
"""Setup remote-DL session endpoints only."""
app.router.add_get("/api/health", health)
app.router.add_get("/api/services", services)
app.router.add_post("/api/search", search)
app.router.add_post("/api/session/create", session_create)
app.router.add_get("/api/session/{session_id}/titles", session_titles)
app.router.add_post("/api/session/{session_id}/tracks", session_tracks)
app.router.add_post("/api/session/{session_id}/segments", session_segments)
app.router.add_post("/api/session/{session_id}/license", session_license)
app.router.add_get("/api/session/{session_id}/prompt", session_prompt_get)
app.router.add_post("/api/session/{session_id}/prompt", session_prompt_submit)
app.router.add_get("/api/session/{session_id}", session_info)
app.router.add_delete("/api/session/{session_id}", session_delete)
# Single source of truth for all API routes. `remote` marks endpoints exposed in
# --remote-only server mode; full mode registers every route. Both setup_routes and
# setup_swagger derive from this table so the live routes and the docs cannot drift.
ROUTES: list[tuple[str, str, Handler, bool]] = [
("GET", "/api/health", health, True),
("GET", "/api/services", services, True),
("POST", "/api/search", search, True),
("POST", "/api/list-titles", list_titles, False),
("POST", "/api/list-tracks", list_tracks, False),
("POST", "/api/download", download, False),
("GET", "/api/download/jobs", download_jobs, False),
("GET", "/api/download/jobs/{job_id}", download_job_detail, False),
("DELETE", "/api/download/jobs/{job_id}", cancel_download_job, False),
("POST", "/api/session/create", session_create, True),
("GET", "/api/session/{session_id}/titles", session_titles, True),
("POST", "/api/session/{session_id}/tracks", session_tracks, True),
("POST", "/api/session/{session_id}/segments", session_segments, True),
("POST", "/api/session/{session_id}/license", session_license, True),
("GET", "/api/session/{session_id}/prompt", session_prompt_get, True),
("POST", "/api/session/{session_id}/prompt", session_prompt_submit, True),
("GET", "/api/session/{session_id}", session_info, True),
("DELETE", "/api/session/{session_id}", session_delete, True),
]
def setup_routes(app: web.Application, remote_only: bool = False) -> None:
"""Setup API routes. When remote_only=True, only expose remote session endpoints."""
if remote_only:
_setup_remote_session_routes(app)
return
app.router.add_get("/api/health", health)
app.router.add_get("/api/services", services)
app.router.add_post("/api/search", search)
app.router.add_post("/api/list-titles", list_titles)
app.router.add_post("/api/list-tracks", list_tracks)
app.router.add_post("/api/download", download)
app.router.add_get("/api/download/jobs", download_jobs)
app.router.add_get("/api/download/jobs/{job_id}", download_job_detail)
app.router.add_delete("/api/download/jobs/{job_id}", cancel_download_job)
# Remote-DL session endpoints
app.router.add_post("/api/session/create", session_create)
app.router.add_get("/api/session/{session_id}/titles", session_titles)
app.router.add_post("/api/session/{session_id}/tracks", session_tracks)
app.router.add_post("/api/session/{session_id}/segments", session_segments)
app.router.add_post("/api/session/{session_id}/license", session_license)
app.router.add_get("/api/session/{session_id}/prompt", session_prompt_get)
app.router.add_post("/api/session/{session_id}/prompt", session_prompt_submit)
app.router.add_get("/api/session/{session_id}", session_info)
app.router.add_delete("/api/session/{session_id}", session_delete)
add: dict[str, Callable[..., Any]] = {
"GET": app.router.add_get,
"POST": app.router.add_post,
"DELETE": app.router.add_delete,
}
for method, path, handler, remote in ROUTES:
if remote_only and not remote:
continue
add[method](path, handler)
def setup_swagger(app: web.Application) -> None:
@@ -1369,27 +1364,5 @@ def setup_swagger(app: web.Application) -> None:
),
)
# Add routes with OpenAPI documentation
swagger.add_routes(
[
web.get("/api/health", health),
web.get("/api/services", services),
web.post("/api/search", search),
web.post("/api/list-titles", list_titles),
web.post("/api/list-tracks", list_tracks),
web.post("/api/download", download),
web.get("/api/download/jobs", download_jobs),
web.get("/api/download/jobs/{job_id}", download_job_detail),
web.delete("/api/download/jobs/{job_id}", cancel_download_job),
# Remote-DL session endpoints
web.post("/api/session/create", session_create),
web.get("/api/session/{session_id}/titles", session_titles),
web.post("/api/session/{session_id}/tracks", session_tracks),
web.post("/api/session/{session_id}/segments", session_segments),
web.post("/api/session/{session_id}/license", session_license),
web.get("/api/session/{session_id}/prompt", session_prompt_get),
web.post("/api/session/{session_id}/prompt", session_prompt_submit),
web.get("/api/session/{session_id}", session_info),
web.delete("/api/session/{session_id}", session_delete),
]
)
route: dict[str, Callable[..., Any]] = {"GET": web.get, "POST": web.post, "DELETE": web.delete}
swagger.add_routes([route[method](path, handler) for method, path, handler, _ in ROUTES])
+2 -2
View File
@@ -14,7 +14,7 @@ from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from unshackle.core.api.input_bridge import AuthStatus, InputBridge
from unshackle.core.api.input_bridge import AUTH_INPUT_TIMEOUT, AuthStatus, InputBridge
from unshackle.core.api.sanitize import sanitize_log
from unshackle.core.config import config
from unshackle.core.tracks import Track
@@ -127,7 +127,7 @@ class SessionStore:
for sid, entry in self._sessions.items():
elapsed = (now - entry.last_accessed).total_seconds()
if entry.auth_status in (AuthStatus.AUTHENTICATING, AuthStatus.PENDING_INPUT):
if elapsed > 600:
if elapsed > AUTH_INPUT_TIMEOUT:
expired.append(sid)
elif elapsed > self._ttl:
expired.append(sid)
+2
View File
@@ -47,6 +47,7 @@ class Config:
self.curl_impersonate: dict = kwargs.get("curl_impersonate") or {}
self.remote_cdm: list[dict] = kwargs.get("remote_cdm") or []
self.credentials: dict = kwargs.get("credentials") or {}
self.firefox_cookies: dict = kwargs.get("firefox_cookies") or {}
self.subtitle: dict = kwargs.get("subtitle") or {}
self.directories = self._Directories()
@@ -74,6 +75,7 @@ class Config:
self.audio: dict = kwargs.get("audio") or {}
self.headers: dict = kwargs.get("headers") or {}
self.key_vaults: list[dict[str, Any]] = kwargs.get("key_vaults", [])
self.vault_timeout: float = kwargs.get("vault_timeout", 10.0)
self.muxing: dict = kwargs.get("muxing") or {}
self.proxy_providers: dict = kwargs.get("proxy_providers") or {}
self.remote_services: dict = kwargs.get("remote_services") or {}
+2 -1
View File
@@ -1,8 +1,9 @@
from .basic import Basic
from .expressvpn import ExpressVPN
from .gluetun import Gluetun
from .hola import Hola
from .nordvpn import NordVPN
from .surfsharkvpn import SurfsharkVPN
from .windscribevpn import WindscribeVPN
__all__ = ("Basic", "Gluetun", "Hola", "NordVPN", "SurfsharkVPN", "WindscribeVPN")
__all__ = ("Basic", "ExpressVPN", "Gluetun", "Hola", "NordVPN", "SurfsharkVPN", "WindscribeVPN")
+845
View File
@@ -0,0 +1,845 @@
from __future__ import annotations
import base64
import hashlib
import json
import logging
import os
import random
import re
import secrets
import time
from pathlib import Path
from typing import Optional
from urllib.parse import parse_qs, urlparse
import requests
from unshackle.core.config import config
from unshackle.core.proxies.proxy import Proxy
log = logging.getLogger("proxies.expressvpn")
class ExpressVPN(Proxy):
"""
ExpressVPN HTTPS proxy provider.
This provider follows the browser extension proxy flow:
browser cookies or a cached refresh token are exchanged for API tokens,
those tokens are used to resolve proxy-capable locations, and get_proxy()
returns an authenticated HTTPS proxy URL.
Query format:
country -- smart connection (random location)
country-city -- specific city
country-city-N / cityN -- pinned server by position
full-slug -- ExpressVPN location slug
hostname.expressprovider.com -- direct hostname
"""
CLIENT_ID = "f457fed092a54b9e9f1e2113782d74a2"
AUTH_BASE = "https://auth.expressvpn.com/realms/xvpn/protocol/openid-connect"
API_BASE = "https://cp.expressapisv2.net"
EXTENSION_REDIRECT_URI = "chrome-extension://fgddmllnllkalaagkghckoinaemmogpe/src/html/auth-callback.html"
USER_AGENT = (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
)
def __init__(
self,
region_map: Optional[dict[str, str]] = None,
server_map: Optional[dict[str, str]] = None,
refresh_token: Optional[str] = None,
access_token: Optional[str] = None,
connection_token: Optional[str] = None,
account_json: Optional[str] = None,
cookie_path: Optional[str] = None,
cache_path: Optional[str] = None,
timeout: float = 10.0,
):
"""
Proxy Service using ExpressVPN browser-extension proxy credentials.
Args:
region_map: Optional country-to-preset mapping. Keys are country
codes (e.g. ``"us"``), values are optional city/server presets
(e.g. ``"ny-02"``). When a key is used without a city in the
CLI query, the preset is applied. Empty/null values enable
smart connection (random location in that country).
server_map: Optional aliases using the same convention as other
proxy providers. Values may be ExpressVPN location slugs or
concrete .expressprovider.com hosts.
refresh_token: Optional OAuth refresh token. If omitted, cached
tokens or browser cookies are used.
access_token: Optional OAuth access token for advanced/manual use.
connection_token: Optional cached connection authorization token.
account_json: Optional path to ExpressVPN desktop account.json.
cookie_path: Optional path to exported ExpressVPN browser cookies.
cache_path: Optional path for cached ExpressVPN tokens.
timeout: Request timeout in seconds.
"""
if region_map is not None and not isinstance(region_map, dict):
raise TypeError(f"Expected region_map to be a dict mapping aliases to locations, not '{region_map!r}'.")
if server_map is not None and not isinstance(server_map, dict):
raise TypeError(f"Expected server_map to be a dict mapping aliases to locations, not '{server_map!r}'.")
# region_map: country code → optional preset (city[-server])
self.region_map = {
str(k).lower().strip(): (str(v).lower().strip() if v else None)
for k, v in (region_map or {}).items()
}
# server_map: legacy alias → location slug or direct hostname
self.server_map = {
str(k).lower().strip(): str(v).lower().strip()
for k, v in (server_map or {}).items()
}
self.refresh_token = refresh_token or None
self.access_token = access_token or None
self.connection_token = connection_token or None
self.account_json = Path(account_json).expanduser() if account_json else None
self.timeout = timeout
self._tokens: Optional[dict] = None
self._srt: Optional[str] = None
self._locations: Optional[list[dict]] = None
# Display info set during _resolve_endpoint for log messages
self._last_location_name: Optional[str] = None
self._last_server_index: Optional[int] = None
self._last_server_total: Optional[int] = None
self._last_endpoint_host: Optional[str] = None
self.cookie_path = Path(cookie_path).expanduser() if cookie_path else self._default_cookie_path()
self.cache_path = Path(cache_path).expanduser() if cache_path else self._default_cache_path()
def __repr__(self) -> str:
if self._locations is not None:
locations = self._locations
countries = len({
str(loc.get("country_code") or "").upper()
for loc in locations if loc.get("country_code")
})
servers = len(locations)
return (
f"{countries} Countr{'ies' if countries != 1 else 'y'} "
f"({servers} Server{'s' if servers != 1 else ''})"
)
alias_count = len(self.region_map) + len(self.server_map)
if alias_count:
return f"{alias_count} Region Alias{'es' if alias_count != 1 else ''} (ExpressVPN HTTPS Proxy)"
return "ExpressVPN HTTPS Proxy"
def get_proxy(self, query: str) -> Optional[str]:
query = query.strip().lower()
endpoint = self._resolve_endpoint(query)
if not endpoint:
return None
connection_token = self._get_connection_token()
if not connection_token:
log.error("ExpressVPN: connection token was not available")
return None
display = self.last_connection_display()
log.debug("ExpressVPN proxy ready: %s", display or f"https://cat:***@{endpoint}:443")
return f"https://cat:{connection_token}@{endpoint}:443"
def last_connection_display(self) -> Optional[str]:
"""Return a human-readable string describing the last resolved connection.
Used by the download/search commands to display a friendly log message
instead of a raw sanitized proxy URL.
Returns:
A string like ``"(USA - New York, #3 of 5): .214"`` or None if
no connection has been resolved yet.
"""
if not self._last_location_name or not self._last_endpoint_host:
return None
# Strip the .expressprovider.com suffix for readability
short_host = self._last_endpoint_host.replace(".expressprovider.com", "")
parts = [self._last_location_name]
if self._last_server_index is not None and self._last_server_total is not None:
parts.append(f"#{self._last_server_index} of {self._last_server_total}")
return f"({', '.join(parts)}): {short_host}"
# ------------------------------------------------------------------
# Endpoint Resolution
# ------------------------------------------------------------------
def _resolve_endpoint(self, query: str) -> Optional[str]:
# Reset display info
self._last_location_name = None
self._last_server_index = None
self._last_server_total = None
self._last_endpoint_host = None
# 1) Check legacy server_map first (direct hostname / slug aliases)
mapped = self.server_map.get(query)
if mapped:
query = mapped
# 2) Direct hostname pass-through
if "expressprovider" in query:
host = query if query.endswith(".expressprovider.com") else f"{query}.expressprovider.com"
self._last_endpoint_host = host
return host
# 3) Parse the query into (country, city, server_num)
country, city, server_num = self._parse_query(query)
# 4) If the query matched a country in region_map and no city was
# specified in the CLI, apply the preset from region_map
if country and not city:
preset = self.region_map.get(country)
if preset:
# Preset is like "ny-02" or "ny" or "ny2"
_, preset_city, preset_server = self._parse_query(f"{country}-{preset}")
city = preset_city
server_num = preset_server
# 5) Resolve the location
if country:
location = self._resolve_location_by_country(country, city)
else:
# Fallback: try the full query as a location slug
location = self._resolve_location_by_slug(query)
if not location:
log.warning("ExpressVPN: no location matched query '%s'", query)
return None
self._last_location_name = location.get("name")
# 6) Get endpoints and pick server
endpoints = self._get_endpoints_for_location(location)
if not endpoints:
log.error("ExpressVPN: no proxy endpoints returned for %s", location.get("name") or query)
return None
host = self._pick_endpoint(endpoints, server_num)
return host
def _parse_query(self, query: str) -> tuple[Optional[str], Optional[str], Optional[int]]:
"""Parse a query string into (country_code, city_query, server_number).
Supported formats::
us -> ("us", None, None)
us-ny -> ("us", "ny", None)
us-ny-2 -> ("us", "ny", 2)
us-ny2 -> ("us", "ny", 2)
us-ny-02 -> ("us", "ny", 2)
usa-new-york -> (None, None, None) -- treated as slug
mx -> ("mx", None, None)
When the query doesn't look like a country[-city] pattern, all
fields are returned as None and the caller falls back to slug
resolution.
"""
# Try to extract a trailing server number: "us-ny-2" or "us-ny2"
server_num = None
base_query = query
# Match trailing -N or trailing digits glued to alpha
num_match = re.match(r"^(.+?)[-]?(\d+)$", query)
if num_match:
candidate_base = num_match.group(1).rstrip("-")
candidate_num = int(num_match.group(2))
# Only treat as server number if the base contains alpha characters
# and the number is reasonable (1-99)
if re.search(r"[a-z]", candidate_base) and 1 <= candidate_num <= 99:
base_query = candidate_base
server_num = candidate_num
# Try to split as country-city
# Country codes are 2 letters; city is the rest
country_match = re.match(r"^([a-z]{2})(?:-(.+))?$", base_query)
if country_match:
country = country_match.group(1)
city = country_match.group(2) # may be None
# Verify this is a valid country code by checking locations
locations = self._get_locations()
valid_countries = {str(loc.get("country_code") or "").lower() for loc in locations}
if country in valid_countries:
return country, city or None, server_num
# Not a country-city pattern; return None to trigger slug fallback
return None, None, None
def _resolve_location_by_country(
self, country_code: str, city_query: Optional[str] = None
) -> Optional[dict]:
"""Find a location by country code and optional city query.
If *city_query* is None, a random location in the country is selected
(smart connection). Otherwise, the city is matched against location
names using abbreviation, prefix, and substring strategies.
"""
locations = self._get_locations()
if not locations:
return None
# Filter locations for this country
country_locations = [
loc for loc in locations
if str(loc.get("country_code") or "").lower() == country_code
]
if not country_locations:
log.warning("ExpressVPN: no locations found for country '%s'", country_code.upper())
return None
if not city_query:
# Smart connection: random location in the country
return random.choice(country_locations)
# Match city query against location names
return self._match_city(country_locations, city_query)
def _match_city(self, locations: list[dict], city_query: str) -> Optional[dict]:
"""Match a short city query against a list of locations.
Matching strategies (in priority order):
1. First-letter abbreviation: ``"ny"`` matches ``"New York"``
2. Exact slug match: ``"miami"`` matches ``"miami"``
3. Prefix match on slug: ``"mia"`` matches ``"miami"``
4. Substring match on slug: ``"york"`` matches ``"new-york"``
Returns the first match from the highest-priority non-empty bucket.
"""
city_query = city_query.strip().lower()
abbreviation_hits: list[dict] = []
exact_hits: list[dict] = []
prefix_hits: list[dict] = []
substring_hits: list[dict] = []
for loc in locations:
full_name = str(loc.get("name") or "")
# Extract city part: strip country prefix like "USA - "
city_part = re.sub(r"^[A-Z]{2,}(?:\s*-\s*)", "", full_name, count=1).strip()
if not city_part:
city_part = full_name
city_slug = _slugify(city_part)
# Strategy 1: first-letter abbreviation
words = re.findall(r"[a-zA-Z]+", city_part)
abbreviation = "".join(w[0] for w in words).lower() if words else ""
if city_query == abbreviation:
abbreviation_hits.append(loc)
continue
# Strategy 2: exact slug match
if city_query == city_slug:
exact_hits.append(loc)
continue
# Strategy 3: prefix match on slug
if city_slug.startswith(city_query):
prefix_hits.append(loc)
continue
# Strategy 4: substring match on slug
if city_query in city_slug:
substring_hits.append(loc)
# Return the first match from the highest-priority non-empty bucket
candidates = abbreviation_hits or exact_hits or prefix_hits or substring_hits
if not candidates:
log.warning("ExpressVPN: no city matched '%s' in available locations", city_query)
return None
if len(candidates) > 1:
names = [c.get("name") for c in candidates]
log.debug("ExpressVPN: city '%s' matched %d locations: %s", city_query, len(candidates), names)
return candidates[0]
def _resolve_location_by_slug(self, query: str) -> Optional[dict]:
"""Resolve a location by matching the full query against slugified names, IDs, or country codes."""
target = query.strip().lower()
locations = self._get_locations()
if not locations:
return None
for location in locations:
location_id = str(location.get("id") or "").lower()
location_name = str(location.get("name") or "").lower()
country_code = str(location.get("country_code") or "").lower()
slug = _slugify(location_name)
if target in (location_id, location_name, country_code, slug):
return location
return None
def _pick_endpoint(self, endpoints: list[dict], server_num: Optional[int] = None) -> Optional[str]:
"""Select an endpoint host from the list, optionally by position index.
Args:
endpoints: List of endpoint dicts with ``"host"`` keys.
server_num: 1-based server position index, or None for random.
Returns:
The selected hostname, or None if no endpoints are available.
"""
hosts = [str(ep.get("host") or "").lower() for ep in endpoints if ep.get("host")]
if not hosts:
return None
total = len(hosts)
self._last_server_total = total
if server_num is not None:
if server_num < 1 or server_num > total:
log.warning(
"ExpressVPN: server #%d is not available (only %d server%s), selecting random",
server_num, total, "s" if total != 1 else "",
)
chosen = random.choice(hosts)
self._last_server_index = hosts.index(chosen) + 1
else:
chosen = hosts[server_num - 1]
self._last_server_index = server_num
else:
chosen = random.choice(hosts)
self._last_server_index = hosts.index(chosen) + 1
self._last_endpoint_host = chosen
return chosen
# ------------------------------------------------------------------
# Locations & Endpoints API
# ------------------------------------------------------------------
def _get_locations(self) -> list[dict]:
if self._locations is not None:
return self._locations
srt = self._get_srt()
if not srt:
return []
response = self._request(
"GET",
f"{self.API_BASE}/ids2/locations",
headers=self._api_headers(srt),
params={"protocols": "proxy"},
)
self._locations = response.json().get("locations", [])
return self._locations
def _get_endpoints_for_location(self, location: dict) -> list[dict]:
srt = self._get_srt()
if not srt:
return []
location_id = location.get("id")
response = self._request(
"POST",
f"{self.API_BASE}/ids2/locations/{location_id}/instances",
headers=self._api_headers(srt),
params={"protocols": "proxy"},
json={},
)
return response.json().get("endpoints", [])
# ------------------------------------------------------------------
# Token Management
# ------------------------------------------------------------------
def _get_tokens(self) -> dict:
# Return in-memory cache if access_token is still valid
if self._tokens:
access_token = self._tokens.get("access_token")
if access_token and not self._is_jwt_expired(access_token):
return self._tokens
cached = self._load_cached_tokens()
tokens = {
**cached,
**{k: v for k, v in {
"access_token": self.access_token,
"refresh_token": self.refresh_token,
"connection_token": self.connection_token,
}.items() if v},
}
access_token = tokens.get("access_token")
if access_token and not self._is_jwt_expired(access_token):
self._tokens = tokens
return tokens
refresh_token = tokens.get("refresh_token")
if refresh_token:
refreshed = self._refresh_access_token(refresh_token)
if refreshed:
tokens.update(
{
"access_token": refreshed.get("access_token"),
"refresh_token": refreshed.get("refresh_token") or refresh_token,
}
)
self._save_cached_tokens(tokens)
self._tokens = tokens
return tokens
log.warning("ExpressVPN: refresh token failed or session expired")
if self.account_json and self.account_json.is_file():
try:
data = json.loads(self.account_json.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as error:
log.error("ExpressVPN: failed to read account_json %s: %s", self.account_json, error)
else:
tokens.update(
{
"access_token": data.get("accessToken"),
"connection_token": data.get("connectionToken"),
"subscription_id": data.get("subscriptionId"),
}
)
tokens = {k: v for k, v in tokens.items() if v}
self._tokens = tokens
return tokens
if self.cookie_path.is_file():
bootstrapped = self._run_pkce_bootstrap()
if bootstrapped:
tokens.update(
{
"access_token": bootstrapped.get("access_token"),
"refresh_token": bootstrapped.get("refresh_token"),
}
)
self._save_cached_tokens(tokens)
self._tokens = tokens
return tokens
tokens = {k: v for k, v in tokens.items() if v}
self._tokens = tokens
return tokens
def _refresh_access_token(self, refresh_token: str) -> Optional[dict]:
response = self._request(
"POST",
f"{self.AUTH_BASE}/token",
data={
"grant_type": "refresh_token",
"client_id": self.CLIENT_ID,
"refresh_token": refresh_token,
},
headers=self._form_headers(),
allow_error=True,
)
if not response.ok:
log.warning("ExpressVPN: access token refresh failed with HTTP %s", response.status_code)
return None
return response.json()
def _run_pkce_bootstrap(self) -> Optional[dict]:
cookies = self._load_cookies()
keycloak_identity = cookies.get("KEYCLOAK_IDENTITY")
keycloak_session = cookies.get("KEYCLOAK_SESSION")
auth_session_id = cookies.get("AUTH_SESSION_ID")
if not keycloak_identity or not keycloak_session:
log.error("ExpressVPN: KEYCLOAK_IDENTITY or KEYCLOAK_SESSION is missing from %s", self.cookie_path)
return None
verifier = base64.urlsafe_b64encode(
secrets.token_bytes(32)
).decode("utf-8").rstrip("=")
challenge = base64.urlsafe_b64encode(
hashlib.sha256(verifier.encode("utf-8")).digest()
).decode("utf-8").rstrip("=")
auth_cookies = requests.cookies.RequestsCookieJar()
auth_cookies.set("KEYCLOAK_IDENTITY", keycloak_identity, domain="auth.expressvpn.com")
auth_cookies.set("KEYCLOAK_SESSION", keycloak_session, domain="auth.expressvpn.com")
if auth_session_id:
auth_cookies.set("AUTH_SESSION_ID", auth_session_id, domain="auth.expressvpn.com")
response = self._request(
"GET",
f"{self.AUTH_BASE}/auth",
params={
"client_id": self.CLIENT_ID,
"response_type": "code",
"redirect_uri": self.EXTENSION_REDIRECT_URI,
"scope": "profile offline_access",
"code_challenge": challenge,
"code_challenge_method": "S256",
"ui_locales": "en",
},
headers=self._browser_headers(),
cookies=auth_cookies,
allow_redirects=False,
allow_error=True,
)
redirect_url = response.headers.get("Location", "")
code = parse_qs(urlparse(redirect_url).query).get("code", [None])[0]
if not code:
log.error("ExpressVPN: OAuth bootstrap did not return an authorization code")
return None
token_response = self._request(
"POST",
f"{self.AUTH_BASE}/token",
data={
"grant_type": "authorization_code",
"client_id": self.CLIENT_ID,
"code": code,
"redirect_uri": self.EXTENSION_REDIRECT_URI,
"code_verifier": verifier,
},
headers=self._form_headers(),
allow_error=True,
)
if not token_response.ok:
log.error("ExpressVPN: OAuth token exchange failed with HTTP %s", token_response.status_code)
return None
return token_response.json()
def _get_srt(self) -> Optional[str]:
if self._srt and not self._is_jwt_expired(self._srt):
return self._srt
tokens = self._get_tokens()
cached_srt = tokens.get("srt")
if cached_srt and not self._is_jwt_expired(cached_srt):
self._srt = cached_srt
return self._srt
access_token = tokens.get("access_token")
if not access_token:
log.error("ExpressVPN: access token was not available")
return None
response = self._request(
"POST",
f"{self.API_BASE}/srs2/subscription_receipts",
headers=self._api_headers(access_token),
json={},
)
receipts = response.json().get("srts", [])
active_subscription_id = tokens.get("subscription_id")
srt = self._select_srt(receipts, active_subscription_id)
if not srt:
log.error("ExpressVPN: no active subscription receipt was found")
return None
self._srt = srt
tokens["srt"] = srt
self._save_cached_tokens(tokens)
return srt
def _get_connection_token(self) -> Optional[str]:
tokens = self._get_tokens()
connection_token = tokens.get("connection_token")
if connection_token and not self._is_jwt_expired(connection_token):
return connection_token
auth_token = self._get_srt() or tokens.get("access_token")
if not auth_token:
return connection_token
response = self._request(
"POST",
f"{self.API_BASE}/srs2/connection_token",
headers=self._api_headers(auth_token),
json={},
)
body = response.json()
connection_token = body.get("connection_token") or body.get("token")
if connection_token:
tokens["connection_token"] = connection_token
self._save_cached_tokens(tokens)
return connection_token
# ------------------------------------------------------------------
# Cookies & Cache
# ------------------------------------------------------------------
def _load_cookies(self) -> dict[str, str]:
if not self.cookie_path.is_file():
return {}
try:
content = self.cookie_path.read_text(encoding="utf-8").strip()
except OSError as error:
log.error("ExpressVPN: failed to read cookies file %s: %s", self.cookie_path, error)
return {}
cookies: dict[str, str] = {}
try:
data = json.loads(content)
except json.JSONDecodeError:
for line in content.splitlines():
if not line.strip() or line.startswith("#"):
continue
parts = line.rstrip("\n").split("\t")
if len(parts) >= 7 and "expressvpn.com" in parts[0]:
cookies[parts[5]] = parts[6]
elif "=" in line:
name, value = line.split("=", maxsplit=1)
cookies[name.strip()] = value.strip()
else:
if isinstance(data, list):
for cookie in data:
if isinstance(cookie, dict) and "expressvpn.com" in str(cookie.get("domain", "")):
name = cookie.get("name")
value = cookie.get("value")
if name and value is not None:
cookies[str(name)] = str(value)
elif isinstance(data, dict):
cookies.update({str(k): str(v) for k, v in data.items()})
return cookies
def _load_cached_tokens(self) -> dict:
if not self.cache_path.is_file():
return {}
try:
data = json.loads(self.cache_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as error:
log.warning("ExpressVPN: failed to read token cache %s: %s", self.cache_path, error)
return {}
return data if isinstance(data, dict) else {}
def _save_cached_tokens(self, tokens: dict) -> None:
# Invalidate in-memory cache so next _get_tokens() re-reads
self._tokens = tokens.copy()
try:
self.cache_path.parent.mkdir(parents=True, exist_ok=True)
payload = json.dumps({k: v for k, v in tokens.items() if v}, indent=2)
_write_private(self.cache_path, payload)
except OSError as error:
log.error("ExpressVPN: failed to save token cache %s: %s", self.cache_path, error)
# ------------------------------------------------------------------
# HTTP & Helpers
# ------------------------------------------------------------------
def _request(self, method: str, url: str, allow_error: bool = False, **kwargs) -> requests.Response:
kwargs.setdefault("timeout", self.timeout)
response = requests.request(method, url, **kwargs)
if response.ok or allow_error:
return response
raise ValueError(f"ExpressVPN request failed with HTTP {response.status_code}: {url}")
def _api_headers(self, token: str) -> dict[str, str]:
return {
"Authorization": f"Bearer {token}",
"Accept": "application/json",
"Content-Type": "application/json",
"User-Agent": self.USER_AGENT,
"X-Client-App-Version": "12.0.0",
"X-Client-OS": "Windows",
"X-Client-Device-Model": "Browser Extension",
}
def _browser_headers(self) -> dict[str, str]:
return {
"User-Agent": self.USER_AGENT,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
}
def _form_headers(self) -> dict[str, str]:
return {
"Content-Type": "application/x-www-form-urlencoded",
"Accept": "application/json",
"User-Agent": self.USER_AGENT,
}
def _select_srt(self, receipts: list[dict], active_subscription_id: Optional[str]) -> Optional[str]:
for receipt in receipts:
srt = receipt.get("srt")
if "xv.vpn" in self._decode_jwt_payload(srt).get("entitlements", {}):
return srt
if active_subscription_id:
for receipt in receipts:
if receipt.get("subscription_id") == active_subscription_id:
return receipt.get("srt")
if receipts:
return receipts[0].get("srt")
return None
def _decode_jwt_payload(self, token: Optional[str]) -> dict:
if not token:
return {}
try:
parts = token.split(".")
if len(parts) < 2:
return {}
payload = parts[1] + "=" * ((4 - len(parts[1]) % 4) % 4)
return json.loads(base64.urlsafe_b64decode(payload).decode("utf-8"))
except (ValueError, json.JSONDecodeError):
return {}
def _is_jwt_expired(self, token: str) -> bool:
expires_at = self._decode_jwt_payload(token).get("exp")
if not expires_at:
# Tokens without ``exp`` are treated as non-expiring. Keycloak
# access/refresh tokens always carry ``exp``; proprietary SRT/
# connection tokens may omit it — they are refreshed via the
# normal SRT → connection_token flow anyway.
return False
return time.time() >= (int(expires_at) - 300)
def _default_cookie_path(self) -> Path:
cookies_dir = Path(config.directories.cookies)
for folder in ("vpn", "vpns"):
candidate = cookies_dir / folder / "expressvpn.txt"
if candidate.is_file():
return candidate
return cookies_dir / "vpn" / "expressvpn.txt"
def _default_cache_path(self) -> Path:
cache_dir = Path(config.directories.cache)
for folder in ("vpn", "vpns"):
candidate = cache_dir / "global" / f"{folder}_expressvpn_tokens.json"
if candidate.is_file():
return candidate
candidate = cache_dir / "global" / "expressvpn_tokens.json"
if candidate.is_file():
return candidate
return cache_dir / "global" / "expressvpn_tokens.json"
def close(self) -> None:
pass
def __enter__(self) -> ExpressVPN:
return self
def __exit__(self, *_: object) -> None:
self.close()
def _slugify(value: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
def _write_private(path: Path, content: str) -> None:
"""Write *content* to *path* with owner-only permissions (0600).
Opens with ``os.open`` so the file is **created** at 0600 — no window
where it is world-readable. On Windows ``os.fchmod`` is unavailable,
so we fall back to ``path.chmod`` after close.
"""
if hasattr(os, "fchmod"):
fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
try:
os.fchmod(fd, 0o600)
os.write(fd, content.encode("utf-8"))
finally:
os.close(fd)
else:
path.write_text(content, encoding="utf-8")
try:
path.chmod(0o600)
except OSError:
pass
+3
View File
@@ -19,6 +19,7 @@ def initialize_proxy_providers() -> List[Any]:
from unshackle.core import binaries
from unshackle.core.config import config as main_config
from unshackle.core.proxies.basic import Basic
from unshackle.core.proxies.expressvpn import ExpressVPN
from unshackle.core.proxies.hola import Hola
from unshackle.core.proxies.nordvpn import NordVPN
from unshackle.core.proxies.surfsharkvpn import SurfsharkVPN
@@ -27,6 +28,8 @@ def initialize_proxy_providers() -> List[Any]:
if proxy_config.get("basic"):
proxy_providers.append(Basic(**proxy_config["basic"]))
if proxy_config.get("expressvpn"):
proxy_providers.append(ExpressVPN(**proxy_config["expressvpn"]))
if proxy_config.get("nordvpn"):
proxy_providers.append(NordVPN(**proxy_config["nordvpn"]))
if proxy_config.get("surfsharkvpn"):
+2 -2
View File
@@ -198,7 +198,7 @@ def _build_tracks(data: Dict[str, Any]) -> Tracks:
return tracks
def _resolve_manifest_data(tracks: Tracks, manifests: list, session: Any) -> None:
def _resolve_manifest_data(tracks: Tracks, manifests: list) -> None:
"""Re-parse serialized manifests and populate track.data for downloading.
The server serializes DASH and ISM manifest XML as zlib-compressed base64.
@@ -604,7 +604,7 @@ class RemoteService:
for k, v in result.get("session_cookies", {}).items():
self._session.cookies.set(k, v)
_resolve_manifest_data(tracks, result.get("manifests", []), self._session)
_resolve_manifest_data(tracks, result.get("manifests", []))
self._server_cdm_type = result.get("server_cdm_type", "widevine")
+1 -2
View File
@@ -209,8 +209,7 @@ class UpdateChecker:
return None
try:
loop = asyncio.get_event_loop()
latest_version = await loop.run_in_executor(None, cls._fetch_latest_version)
latest_version = await asyncio.to_thread(cls._fetch_latest_version)
if latest_version and cls._compare_versions(current_version, latest_version):
return latest_version
@@ -0,0 +1,161 @@
"""
Firefox Cookie and LocalStorage Extractor Utility for unshackle.
Provides a secure, read-only mechanism to extract authentication tokens from Firefox.
"""
import sqlite3
import platform
import os
import shutil
import tempfile
from pathlib import Path
from http.cookiejar import CookieJar, Cookie
from typing import Dict, Optional, List
def get_firefox_root() -> Path:
"""Locates the operating system specific root directory for Firefox profiles."""
system = platform.system()
home = Path.home()
if system == 'Windows':
return Path(os.getenv('APPDATA', '')) / 'Mozilla' / 'Firefox'
elif system == 'Darwin':
return home / 'Library' / 'Application Support' / 'Firefox'
elif system == 'Linux':
paths = [
home / '.mozilla' / 'firefox',
home / 'snap' / 'firefox' / 'common' / '.mozilla' / 'firefox',
home / '.var' / 'app' / 'org.mozilla.firefox' / '.mozilla' / 'firefox'
]
for p in paths:
if p.exists():
return p
return paths[0]
raise RuntimeError(f"Unsupported operating system: {system}")
def get_latest_profile_path(ff_root: Path) -> Path:
"""Finds the most recently modified valid Firefox profile subdirectory."""
search_dirs = [ff_root / 'Profiles', ff_root]
valid_profiles = []
for base in search_dirs:
if base.exists():
valid_profiles.extend([p for p in base.iterdir() if p.is_dir() and (p / 'cookies.sqlite').exists()])
if not valid_profiles:
raise FileNotFoundError("No active Firefox profiles detected.")
return max(valid_profiles, key=lambda p: (p / 'cookies.sqlite').stat().st_mtime)
def get_local_storage_data(profile_path: Path, hosts: List[str], tmp_dir_path: Path) -> Dict[str, str]:
"""
Extracts LocalStorage tokens from webappsstore.sqlite with strict origin matching.
Operates within a secure temporary directory.
"""
ls_db = profile_path / 'webappsstore.sqlite'
if not ls_db.exists():
return {}
temp_ls = tmp_dir_path / "webappsstore.sqlite"
extracted_storage = {}
try:
shutil.copy2(ls_db, temp_ls)
conn = sqlite3.connect(temp_ls)
cursor = conn.cursor()
for host in hosts:
# Firefox stores LocalStorage as reversed host (e.g., moc.viki.:https:443)
# Use strict prefix matching to prevent over-broad harvesting
reversed_host = host[::-1]
query = "SELECT key, value FROM webappsstore2 WHERE originKey LIKE ?"
cursor.execute(query, (f"{reversed_host}.%",))
for key, value in cursor.fetchall():
extracted_storage[key] = value
conn.close()
except Exception:
pass
return extracted_storage
def get_firefox_cookies(service_settings: dict) -> Optional[CookieJar]:
"""
Extracts cookies and optionally localStorage from Firefox based on provided host patterns.
Implements security hardening and data consistency (WAL support) as per PR review.
"""
raw_hosts = service_settings.get('hosts', [])
# Filter empty or dangerously short hosts to prevent full store dumps (Finding #3)
priority_hosts = [h.strip().lower() for h in raw_hosts if h and len(h.strip()) >= 3]
use_local_storage = service_settings.get('local_storage', False)
if not priority_hosts:
return None
try:
profile_path = get_latest_profile_path(get_firefox_root())
except Exception:
return None
cookie_jar = CookieJar()
# Use a secure temporary directory with restricted permissions (Finding #5)
with tempfile.TemporaryDirectory(prefix="unshackle_ff_") as tmp_dir:
tmp_dir_path = Path(tmp_dir)
os.chmod(tmp_dir, 0o700)
temp_db = tmp_dir_path / "cookies.sqlite"
temp_wal = tmp_dir_path / "cookies.sqlite-wal"
try:
# Copy main DB and its WAL log to ensure fresh data capture (Finding #5)
shutil.copy2(profile_path / 'cookies.sqlite', temp_db)
wal_path = profile_path / 'cookies.sqlite-wal'
if wal_path.exists():
shutil.copy2(wal_path, temp_wal)
conn = sqlite3.connect(temp_db)
cursor = conn.cursor()
for host in priority_hosts:
# Precise matching: exact host or dot-prefixed domain suffix (Finding #2)
query = """
SELECT host, name, value, path, expiry, isSecure, isHttpOnly
FROM moz_cookies
WHERE host = ? OR host LIKE ?
"""
cursor.execute(query, (host, f"%.{host}"))
rows = cursor.fetchall()
for h, n, v, p, e, secure, httponly in rows:
# Final validation to ensure proper domain suffix
if h == host or h.endswith(f".{host}"):
c = Cookie(
version=0, name=n, value=v, port=None, port_specified=False,
domain=h, domain_specified=True, domain_initial_dot=h.startswith('.'),
path=p, path_specified=True, secure=bool(secure), expires=e,
discard=False, comment=None, comment_url=None,
rest={'HttpOnly': str(bool(httponly))}, rfc2109=False
)
cookie_jar.set_cookie(c)
conn.close()
# Optional LocalStorage harvesting (Finding #4)
if use_local_storage:
storage_data = get_local_storage_data(profile_path, priority_hosts, tmp_dir_path)
for key, val in storage_data.items():
c = Cookie(
version=0, name=key, value=val, port=None, port_specified=False,
domain=".localstorage", domain_specified=True, domain_initial_dot=False,
path='/', path_specified=True, secure=False, expires=None,
discard=True, comment=None, comment_url=None, rest={}, rfc2109=False
)
cookie_jar.set_cookie(c)
except Exception:
return None
return cookie_jar if len(list(cookie_jar)) > 0 else None
+4
View File
@@ -1,3 +1,4 @@
import inspect
import logging
import time
from typing import Any, Iterator, Optional, Union
@@ -34,6 +35,9 @@ class Vaults:
module = _MODULES.get(type_)
if not module:
raise ValueError(f"Unable to find vault command by the name '{type_}'.")
# only pass the global default to vaults that declare a timeout param (per-vault config still wins)
if "timeout" not in kwargs and "timeout" in inspect.signature(module).parameters:
kwargs["timeout"] = config.vault_timeout
try:
vault = module(**kwargs)
self.vaults.append(vault)
+35 -793
View File
@@ -1,815 +1,57 @@
# API key for The Movie Database (TMDB)
tmdb_api_key: ""
# Client ID for SIMKL API (optional, improves metadata matching)
# Get your free client ID at: https://simkl.com/settings/developer/
simkl_client_id: ""
# Optional ipinfo.io API token. When set, unshackle uses the free Lite endpoint
# which has higher rate limits and richer IP info (ASN, org, continent).
# Get a free token at: https://ipinfo.io/signup
ipinfo_api_key: ""
# Group or Username to postfix to the end of all download filenames following a dash
tag: user_tag
# Enable/disable tagging with group name (default: true)
tag_group_name: true
# Enable/disable tagging with IMDB/TMDB/TVDB details (default: true)
tag_imdb_tmdb: true
# Set terminal background color (custom option not in CONFIG.md)
set_terminal_bg: false
# Custom output templates for filenames
# Configure output_template in your unshackle.yaml to control filename format.
# If not configured, default scene-style templates are used and a warning is shown.
# Available variables: {title}, {year}, {season}, {episode}, {season_episode}, {episode_name},
# {quality}, {resolution}, {source}, {audio}, {audio_channels}, {audio_full},
# {video}, {hdr}, {hfr}, {atmos}, {dual}, {multi}, {tag}, {edition}, {repack},
# {lang_tag}
# Conditional variables (included only if present): Add ? suffix like {year?}, {episode_name?}, {hdr?}
# Customize the templates below:
# unshackle configuration (copy to unshackle.yaml)
#
# Example outputs:
# Scene movies: 'The.Matrix.1999.1080p.SERVICE.WEB-DL.DDP5.1.H.264-EXAMPLE'
# Scene movies (HDR): 'Dune.2021.2160p.SERVICE.WEB-DL.DDP5.1.HDR10.H.265-EXAMPLE'
# Scene movies (REPACK): 'Dune.2021.REPACK.2160p.SERVICE.WEB-DL.DDP5.1.H.265-EXAMPLE'
# Scene series: 'Breaking.Bad.2008.S01E01.Pilot.1080p.SERVICE.WEB-DL.DDP5.1.H.264-EXAMPLE'
# Plex movies: 'The Matrix (1999) 1080p'
# Plex series: 'Breaking Bad S01E01 Pilot'
# This is a MINIMAL starter. Every option has a sane default, so unshackle runs
# even with most of this file empty or removed.
#
# The full, canonical configuration reference lives in docs/ — start at
# docs/README.md. Each section below points to the doc that documents it.
# Group or username appended to download filenames after a dash.
tag: "NOGRP"
# --- Output filename & folder templates --- docs/OUTPUT_CONFIG.md
# Full variable list (35 vars incl. music) and all naming options are
# documented in docs/OUTPUT_CONFIG.md. If omitted, scene-style defaults are
# used (with a warning). Conditional variables take a ? suffix, e.g. {year?}.
output_template:
# Scene-style naming (dot-separated)
movies: '{title}.{year}.{repack?}.{edition?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}'
series: '{title}.{year?}.{season_episode}.{episode_name?}.{repack?}.{edition?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}'
songs: '{track_number}.{title}.{repack?}.{edition?}.{source?}.WEB-DL.{audio_full}.{atmos?}-{tag}'
movies: "{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
series: "{title}.{year?}.{season_episode}.{episode_name?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
songs: "{track_number}.{title}.{repack?}.{edition?}.{lang_tag?}.{source?}.WEB-DL.{audio_full}.{atmos?}-{tag}"
folder:
movies: "{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
series: "{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
albums: "{album_artist} - {album} ({year?})"
#
# Plex-friendly naming (space-separated, clean format)
# Plex-friendly alternatives:
# movies: '{title} ({year}) {quality}'
# series: '{title} {season_episode} {episode_name?}'
# songs: '{track_number}. {title}'
#
# Minimal naming (basic info only)
# movies: '{title}.{year}.{quality}'
# series: '{title}.{season_episode}.{episode_name?}'
#
# Custom scene-style with specific elements
# movies: '{title}.{year}.{quality}.{hdr?}.{source}.WEB-DL.{audio_full}.{video}-{tag}'
# series: '{title}.{year?}.{season_episode}.{episode_name?}.{quality}.{hdr?}.{source}.WEB-DL.{audio_full}.{atmos?}.{video}-{tag}'
#
# Folder naming (optional). Controls the folder name for downloaded content.
# If not configured, series folders are derived from the series template (minus episode info),
# movie folders use "{title} ({year})", and song folders use "{artist} - {album} ({year})".
# Uses the same template variables as the file templates above.
#
# Scene-style folder:
# folder: '{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}'
#
# Plex-friendly folder:
# Folder naming (optional). One template for all, or a per-kind map.
# Valid kinds: movies, series, songs, albums (album vars: see docs).
# folder: '{title} ({year?})'
#
# Per-title-type folder templates (optional). Override folder naming separately for
# movies, series, and albums. Useful when music libraries need artist/album-style folders
# while movies/series follow a different scheme. Any kind omitted falls back to the
# default for that title type.
#
# For music: "songs" (above) names each track file, while "albums" (below) names the
# album folder the tracks are saved into. Album folder vars: {album_artist}, {album},
# {artist}, {year}, {genre}, {label}, {release_type}, {track_total}, {disc_total}.
#
# folder:
# movies: '{title} ({year})'
# series: '{title} ({year?})'
# albums: '{album_artist} - {album} ({year?})'
# Language-based tagging for output filenames
# Automatically adds language identifiers (e.g., DANiSH, NORDiC, DKsubs) based on
# audio and subtitle track languages. Rules are evaluated in order; first match wins.
# Use {lang_tag?} in your output_template to place the tag in the filename.
#
# Conditions (all conditions in a rule must match):
# audio: <lang> - any audio track matches this language
# subs_contain: <lang> - any subtitle matches this language
# subs_contain_all: [lang, ...] - subtitles include ALL listed languages
#
# language_tags:
# rules:
# - audio: da
# tag: DANiSH
# - audio: sv
# tag: SWEDiSH
# - audio: nb
# tag: NORWEGiAN
# - audio: en
# subs_contain_all: [da, sv, nb]
# tag: NORDiC
# - audio: en
# subs_contain: da
# tag: DKsubs
# Check for updates from GitHub repository on startup (default: true)
update_checks: true
# How often to check for updates, in hours (default: 24)
update_check_interval: 24
# Title caching configuration
# Cache title metadata to reduce redundant API calls
title_cache_enabled: true # Enable/disable title caching globally (default: true)
title_cache_time: 1800 # Cache duration in seconds (default: 1800 = 30 minutes)
title_cache_max_retention: 86400 # Maximum cache retention for fallback when API fails (default: 86400 = 24 hours)
# Filename Configuration
unicode_filenames: false # optionally replace non-ASCII characters with ASCII equivalents
# Debug logging configuration
# Comprehensive JSON-based debug logging for troubleshooting and service development
debug:
false # Enable structured JSON debug logging (default: false)
# When enabled with --debug flag or set to true:
# - Creates JSON Lines (.jsonl) log files with complete debugging context
# - Logs: session info, CLI params, service config, CDM details, authentication,
# titles, tracks metadata, DRM operations, vault queries, errors with stack traces
# - File location: logs/unshackle_debug_{service}_{timestamp}.jsonl
# - Also creates text log: logs/unshackle_root_{timestamp}.log
debug_keys:
false # Log decryption keys in debug logs (default: false)
# Set to true to include actual decryption keys in logs
# Useful for debugging key retrieval and decryption issues
# SECURITY NOTE: Passwords, tokens, cookies, and session tokens
# are ALWAYS redacted regardless of this setting
debug_requests:
false # Show per-request HTTP library logs under --debug (default: false)
# When false, --debug mutes the noisy per-segment connection logs from the
# HTTP libraries (urllib3/requests/rnet, e.g. 'GET ...m4s 200'), keeping
# unshackle's own DEBUG output readable.
# Set to true to see every individual request/segment download.
# Only affects: content_key, key fields (the actual CEKs)
# Never affects: kid, keys_count, key_id (metadata is always logged)
# Muxing configuration
muxing:
set_title: false
# merge_audio: Merge all audio tracks into each output file
# true (default): All selected audio in one MKV per quality
# false: Separate MKV per (quality, audio_codec) combination
# Example: Title.1080p.AAC.mkv, Title.1080p.EC3.mkv
merge_audio: true
# default_language: Override which track is flagged as the default in the muxed MKV.
# audio: BCP-47 tag of the preferred default audio track (e.g. pl, en, pt-BR).
# Wins over the title's original_language. Falls back to is_original_lang
# if no matching track is present.
# video: BCP-47 tag of the preferred default video track. Falls back to the
# original-language / first-track rule if no match is found.
# subtitle: BCP-47 tag of the preferred default subtitle track. Falls back to
# the existing rule (forced sub matching the audio language) if no
# matching subtitle is present.
# default_language:
# audio: pl
# video: pl
# subtitle: pl
# Login credentials for each Service
credentials:
# Direct credentials (no profile support)
EXAMPLE: email@example.com:password
# Per-profile credentials with default fallback
SERVICE_NAME:
default: default@email.com:password # Used when no -p/--profile is specified
profile1: user1@email.com:password1
profile2: user2@email.com:password2
# Per-profile credentials without default (requires -p/--profile)
SERVICE_NAME2:
john: john@example.com:johnspassword
jane: jane@example.com:janespassword
# You can also use list format for passwords with special characters
SERVICE_NAME3:
default: ["user@email.com", ":PasswordWith:Colons"]
# Override default directories used across unshackle
directories:
cache: Cache
cookies: Cookies
dcsl: DCSL # Device Certificate Status List
downloads: Downloads
logs: Logs
temp: Temp
wvds: WVDs
prds: PRDs
exports: Exports # JSON export output from --export flag
# Additional directories that can be configured:
# commands: Commands
services:
- /path/to/services
- /other/path/to/services
# vaults: Vaults
# fonts: Fonts
# Pre-define which Widevine or PlayReady device to use for each Service
# --- DRM / CDM --- docs/DRM_CONFIG.md
# Which Widevine (.wvd) or PlayReady (.prd) device to use. 'default' is the
# global fallback. Per-service, per-profile, quality-based selection and remote
# CDMs are all documented in docs/DRM_CONFIG.md.
cdm:
# Global default CDM device (fallback for all services/profiles)
default: WVD_1
# Direct service-specific CDM
DIFFERENT_EXAMPLE: PRD_1
# Per-profile CDM configuration
EXAMPLE:
john_sd: chromecdm_903_l3 # Profile 'john_sd' uses Chrome CDM L3
jane_uhd: nexus_5_l1 # Profile 'jane_uhd' uses Nexus 5 L1
default: generic_android_l3 # Default CDM for this service
# NEW: Quality-based CDM selection
# Use different CDMs based on video resolution
# Supports operators: >=, >, <=, <, or exact match
EXAMPLE_QUALITY:
"<=1080": generic_android_l3 # Use L3 for 1080p and below
">1080": nexus_5_l1 # Use L1 for above 1080p (1440p, 2160p)
default: generic_android_l3 # Optional: fallback if no quality match
# You can mix profiles and quality thresholds in the same service
NETFLIX:
# Profile-based selection (existing functionality)
john: netflix_l3_profile
jane: netflix_l1_profile
# Quality-based selection (new functionality)
"<=720": netflix_mobile_l3
"1080": netflix_standard_l3
">=1440": netflix_premium_l1
# Fallback
default: netflix_standard_l3
# Use pywidevine Serve-compliant Remote CDMs
# Example: Custom CDM API Configuration
# This demonstrates the highly configurable custom_api type that can adapt to any CDM API format
# - name: "chrome"
# type: "custom_api"
# host: "http://remotecdm.test/"
# timeout: 30
# device:
# name: "ChromeCDM"
# type: "CHROME"
# system_id: 34312
# security_level: 3
# auth:
# type: "header"
# header_name: "x-api-key"
# key: "YOUR_API_KEY_HERE"
# custom_headers:
# User-Agent: "Unshackle/2.0.0"
# endpoints:
# get_request:
# path: "/get-challenge"
# method: "POST"
# timeout: 30
# decrypt_response:
# path: "/get-keys"
# method: "POST"
# timeout: 30
# request_mapping:
# get_request:
# param_names:
# scheme: "device"
# init_data: "init_data"
# static_params:
# scheme: "Widevine"
# decrypt_response:
# param_names:
# scheme: "device"
# license_request: "license_request"
# license_response: "license_response"
# static_params:
# scheme: "Widevine"
# response_mapping:
# get_request:
# fields:
# challenge: "challenge"
# session_id: "session_id"
# message: "message"
# message_type: "message_type"
# response_types:
# - condition: "message_type == 'license-request'"
# type: "license_request"
# success_conditions:
# - "message == 'success'"
# decrypt_response:
# fields:
# keys: "keys"
# message: "message"
# key_fields:
# kid: "kid"
# key: "key"
# type: "type"
# success_conditions:
# - "message == 'success'"
# caching:
# enabled: true
# use_vaults: true
# check_cached_first: true
remote_cdm:
- name: "chrome"
device_name: chrome
device_type: CHROME
system_id: 27175
security_level: 3
host: https://domain.com/api
secret: secret_key
- name: "chrome-2"
device_name: chrome
device_type: CHROME
system_id: 26830
security_level: 3
host: https://domain-2.com/api
secret: secret_key
- name: "decrypt_labs_chrome"
type: "decrypt_labs" # Required to identify as DecryptLabs CDM
device_name: "ChromeCDM" # Scheme identifier - must match exactly
device_type: CHROME
system_id: 4464 # Doesn't matter
security_level: 3
host: "https://keyxtractor.decryptlabs.com"
secret: "your_decrypt_labs_api_key_here" # Replace with your API key
- name: "decrypt_labs_l1"
type: "decrypt_labs"
device_name: "L1" # Scheme identifier - must match exactly
device_type: ANDROID
system_id: 4464
security_level: 1
host: "https://keyxtractor.decryptlabs.com"
secret: "your_decrypt_labs_api_key_here"
- name: "decrypt_labs_l2"
type: "decrypt_labs"
device_name: "L2" # Scheme identifier - must match exactly
device_type: ANDROID
system_id: 4464
security_level: 2
host: "https://keyxtractor.decryptlabs.com"
secret: "your_decrypt_labs_api_key_here"
- name: "decrypt_labs_playready_sl2"
type: "decrypt_labs"
device_name: "SL2" # Scheme identifier - must match exactly
device_type: PLAYREADY
system_id: 0
security_level: 2000
host: "https://keyxtractor.decryptlabs.com"
secret: "your_decrypt_labs_api_key_here"
- name: "decrypt_labs_playready_sl3"
type: "decrypt_labs"
device_name: "SL3" # Scheme identifier - must match exactly
device_type: PLAYREADY
system_id: 0
security_level: 3000
host: "https://keyxtractor.decryptlabs.com"
secret: "your_decrypt_labs_api_key_here"
# PyPlayReady RemoteCdm - connects to an unshackle serve instance
- name: "playready_remote"
device_name: "my_prd_device" # Device name on the serve instance
device_type: PLAYREADY
system_id: 0
security_level: 3000 # 2000 for SL2000, 3000 for SL3000
host: "http://127.0.0.1:8786/playready" # Include /playready path
secret: "your-api-secret-key"
# Key Vaults store your obtained Content Encryption Keys (CEKs)
# Use 'no_push: true' to prevent a vault from receiving pushed keys
# while still allowing it to provide keys when requested
# --- Key vaults --- docs/DRM_CONFIG.md
# Where obtained Content Encryption Keys (CEKs) are stored and retrieved.
key_vaults:
- type: SQLite
name: Local
path: key_store.db
# Additional vault types:
# - type: API
# name: "Remote Vault"
# uri: "https://key-vault.example.com"
# token: "secret_token"
# no_push: true # This vault will only provide keys, not receive them
# - type: MySQL
# name: "MySQL Vault"
# host: "127.0.0.1"
# port: 3306
# database: vault
# username: user
# password: pass
# no_push: false # Default behavior - vault both provides and receives keys
# Choose what software to use to download data
downloader: requests
# Options: requests
# Downloading now uses the unified in-process requests/rnet downloader; the legacy
# aria2c, curl_impersonate, and n_m3u8dl_re backends have been removed.
# rnet TLS impersonation preset (not a downloader). Selects the browser
# fingerprint the HTTP session impersonates.
curl_impersonate:
browser: chrome120
# Pre-define default options and switches of the dl command
# Audio track selection preferences
audio:
# Codec priority order used as a tiebreaker when multiple audio tracks share the same
# bitrate and language. Listed codecs are ranked in the order given; codecs not in the
# list keep their bitrate-based ordering and are placed after all listed codecs.
# Atmos still trumps codec priority. Valid names: AAC, AC3, EC3, AC4, OPUS, OGG, DTS, ALAC, FLAC.
# codec_priority: [FLAC, ALAC, AC4, EC3, DTS, AC3, OPUS, AAC, OGG]
# Global defaults for the dl command. Any dl option can be set here.
# Use the option name with dashes as underscores (--v-lang -> v_lang).
# range and list work as-is (range_/list_ also accepted).
#
# Any of these can also be overridden per service under services.<TAG>.dl
# (see the services section below). Precedence, highest first:
# CLI arguments > services.<TAG>.dl > this dl section > built-in defaults
dl:
sub_format: srt
downloads: 4
workers: 16
lang:
- en
- fr
v_lang: [en] # --v-lang: video language(s), if different from audio
s_lang: [en, de] # --s-lang: subtitle language(s)
range: SDR # --range: SDR, HDR10, HDR10+, DV
quality: [1080] # --quality: resolution(s), int or list
slow: false # --slow: true = random 60-120s delay between titles, or "MIN-MAX" string
# Nesting a SERVICE TAG here provides defaults for that service's OWN command
# options (e.g. a service-defined --bitrate flag) -- NOT for the dl options
# above. To override dl options per service, use services.<TAG>.dl instead.
EXAMPLE:
bitrate: CBR
# Chapter Name to use when exporting a Chapter without a Name
chapter_fallback_name: "Chapter {j:02}"
# Case-Insensitive dictionary of headers for all Services
headers:
Accept-Language: "en-US,en;q=0.8"
User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
# Override default filenames used across unshackle
filenames:
debug_log: "unshackle_debug_{service}_{time}.jsonl" # JSON Lines debug log file
config: "config.yaml"
root_config: "unshackle.yaml"
chapters: "Chapters_{title}_{random}.txt"
subtitle: "Subtitle_{id}_{language}.srt"
# conversion_method:
# - auto (default): Smart routing - subby for WebVTT/SAMI, pycaption for others
# - subby: Always use subby with advanced processing
# - pycaption: Use only pycaption library (no SubtitleEdit, no subby)
# - subtitleedit: Prefer SubtitleEdit when available, fall back to pycaption
# - pysubs2: Use pysubs2 library (supports SRT/SSA/ASS/WebVTT/TTML/SAMI/MicroDVD/MPL2/TMP)
subtitle:
conversion_method: auto
# Which backend converts subtitles (data-driven registry, pin-then-fallback)
# - auto (default): best available by rank (SubtitleEdit > subby/pysubs2 > pycaption)
# - subby | pysubs2 | subtitleedit | pycaption: pin that backend first, still falls back
# Styled ASS/SSA are never auto-downconverted to SRT (kept as-is); --sub-format srt overrides.
# SubtitleEdit on Linux/macOS = install the SE5 "SeConv" (seconv) CLI on PATH or unshackle/binaries/.
sdh_method: auto
# Method to use for SDH (hearing impaired) stripping
# - auto (default): Try subby (SRT only), then SubtitleEdit (if available), then subtitle-filter
# - subby: Use subby library (SRT only)
# - subtitleedit: Use SubtitleEdit / seconv (SE5 CLI, cross-platform), falls back to subtitle-filter
# - filter-subs: Use subtitle-filter library directly
strip_sdh: true
# Automatically create stripped (non-SDH) versions of SDH subtitles
# Set to false to disable automatic SDH stripping entirely (default: true)
convert_before_strip: true
# Auto-convert VTT/other formats to SRT before using subtitle-filter
# This ensures compatibility when subtitle-filter is used as fallback (default: true)
preserve_formatting: true
# Preserve original subtitle formatting (tags, positioning, styling)
# When true, skips pycaption processing for WebVTT files to keep tags like <i>, <b>, positioning intact
# Combined with no sub_format setting, ensures subtitles remain in their original format (default: true)
output_mode: mux
# Output mode for subtitles
# - mux: Embed subtitles in MKV container only (default)
# - sidecar: Save subtitles as separate files only
# - both: Embed in MKV AND save as sidecar files
sidecar_format: srt
# Format for sidecar subtitle files
# Options: srt, vtt, ass, original (keep current format)
# Configuration for pywidevine and pyplayready's serve functionality
# Also used for remote services (unshackle serve)
serve:
api_secret: "your-secret-key-here"
# Compression level for API payloads (manifests, cache, cookies)
# 0=off, 1=fast, 6=balanced, 9=max compression (default: 1)
compression_level: 1
# Session inactivity timeout in seconds (default: 300 = 5 minutes)
# Sessions are automatically deleted after this many seconds of inactivity
# Each API request resets the timer
session_ttl: 300
# Maximum concurrent sessions before oldest is evicted (default: 100)
max_sessions: 100
# Global service allowlist (optional)
# Only these services will be exposed on the API. If omitted, all services are available.
# services:
# - SERVICE_TAG_1
# - SERVICE_TAG_2
users:
secret_key_for_user:
devices: # Widevine devices (WVDs) this user can access
- generic_nexus_4464_l3
playready_devices: # PlayReady devices (PRDs) this user can access
- playready_device_sl3000
username: user
# Per-user service allowlist (optional)
# Restricts this user to only the listed services. If omitted, user can access
# all globally-allowed services. Effective access is the intersection of global
# and per-user allowlists.
# services:
# - SERVICE_TAG_1
# devices: # Widevine device paths (auto-populated from directories.wvds)
# - '/path/to/device.wvd'
# playready_devices: # PlayReady device paths (auto-populated from directories.prds)
# - '/path/to/device.prd'
# Per-request override gates (both default OFF = locked to server-side config).
# A /api/download job may ask to use a specific server-side CDM device or to
# authenticate with client-supplied credentials; both are rejected with 403 unless
# explicitly opted in here, so a default deployment can't be driven to use arbitrary
# devices or inject credentials.
# cdm_overrides: true # true = allow any device; or a list of permitted device names:
# cdm_overrides:
# - generic_nexus_4464_l3
# allow_job_credentials: true # allow a per-request `credential` / `credentials` in a job
# # (each distinct credential gets an isolated token cache)
# Optional: any /api/download flag can be set here as a server-side default.
# Per-request body values still win. Useful for raising concurrency without
# changing every client call. Full list of accepted keys: see docs/API.md.
# downloads: 4 # parallel tracks per download job
# workers: 16 # threads per track segment fetch
# best_available: true
# no_proxy_download: false
# Remote Services Configuration
# Connect to a remote unshackle server (unshackle serve) to use its services
# without needing the service code locally. Use with: unshackle dl --remote
# If multiple servers are configured, specify which with: --server <name>
remote_services:
# Server name (used with --server flag if multiple configured)
my-server:
url: "http://192.168.1.100:8786"
api_key: "your-secret-key-here"
# Server-CDM mode: server handles all DRM licensing using its own CDM devices
# When false (default), client uses its own CDM and proxies license requests through the server
server_cdm: false
# Per-service overrides for remote services
# Override downloader, decryption tool, or CDM settings per service on the client
services:
# Example: Override the decryption tool for specific services
# EXAMPLE_SERVICE:
# decryption: mp4decrypt # Override decryption tool (shaka, mp4decrypt)
# Example: Multiple servers
# us-server:
# url: "https://us.example.com:8786"
# api_key: "us-api-key"
# server_cdm: true
# services:
# EXAMPLE:
# decryption: mp4decrypt
# eu-server:
# url: "https://eu.example.com:8786"
# api_key: "eu-api-key"
# server_cdm: false
# Configuration data for each Service
services:
# Service-specific configuration goes here
# Profile-specific configurations can be nested under service names
# You can override ANY global configuration option on a per-service basis
# This allows fine-tuned control for services with special requirements
# Supported overrides: dl, curl_impersonate, subtitle, muxing, headers, etc.
# Example: Comprehensive service configuration showing all features
EXAMPLE:
# Standard service config
api_key: "service_api_key"
# Service certificate for Widevine L1/L2 (base64 encoded)
# This certificate is automatically used when L1/L2 schemes are selected
# Services obtain this from their DRM provider or license server
certificate: |
CAUSwwUKvQIIAxIQ5US6QAvBDzfTtjb4tU/7QxiH8c+TBSKOAjCCAQoCggEBAObzvlu2hZRsapAPx4Aa4GUZj4/GjxgXUtBH4THSkM40x63wQeyVxlEEo
# ... (full base64 certificate here)
# Profile-specific device configurations
profiles:
john_sd:
device:
app_name: "AIV"
device_model: "SHIELD Android TV"
jane_uhd:
device:
app_name: "AIV"
device_model: "Fire TV Stick 4K"
# Service-specific proxy mappings
# Override global proxy selection with specific servers for this service
# When --proxy matches a key in proxy_map, the mapped server will be used
# instead of the default/random server selection
proxy_map:
nordvpn:ca: ca1577 # Use ca1577 when --proxy nordvpn:ca is specified
nordvpn:us: us9842 # Use us9842 when --proxy nordvpn:us is specified
us: 123 # Use server 123 (from any provider) when --proxy us is specified
gb: 456 # Use server 456 (from any provider) when --proxy gb is specified
# Without this service, --proxy nordvpn:ca picks a random CA server
# With this config, --proxy nordvpn:ca EXAMPLE uses ca1577 specifically
# Other services or no service specified will still use random selection
# NEW: Configuration overrides (can be combined with profiles and certificates)
# Override dl command defaults for this service. ANY dl option works here,
# same key naming as the global dl section (v_lang, range_, list_, ...).
# These beat the global dl section; explicit CLI arguments beat both.
dl:
downloads: 4 # Limit concurrent track downloads (global default: 6)
workers: 8 # Reduce workers per track (global default: 16)
lang: ["en", "es-419"] # Different language priority for this service
v_lang: [en] # Video language differs from audio on this service
s_lang: [all] # Grab every subtitle language here
range: HDR10 # Prefer HDR10 on this service (CLI -r still wins)
quality: 720 # This service rate-limits 1080p; int or list both work
sub_format: srt # Force SRT subtitle format
slow: true # Pace title downloads (60-120s); or "20-40" for a custom range
repack: false # Even if the global dl section sets repack: true
# Override subtitle processing for this service
subtitle:
conversion_method: pycaption # Use specific subtitle converter
sdh_method: auto
# Service-specific headers
headers:
User-Agent: "Service-specific user agent string"
Accept-Language: "en-US,en;q=0.9"
# Override muxing options
muxing:
set_title: true
# Remap service-provided titles before naming/output
# Keyed by the exact title the service returns -> desired output title.
title_map:
Service Title: Desired Title
# Example: Service with different regions per profile
SERVICE_NAME:
profiles:
us_account:
region: "US"
api_endpoint: "https://api.us.service.com"
uk_account:
region: "GB"
api_endpoint: "https://api.uk.service.com"
# Example: Rate-limited service
RATE_LIMITED_SERVICE:
dl:
downloads: 2 # Limit concurrent downloads
workers: 4 # Reduce workers to avoid rate limits
# Notes on service-specific overrides:
# - Overrides are merged with global config, not replaced
# - Only specified keys are overridden, others use global defaults
# - Reserved keys (profiles, api_key, certificate, etc.) are NOT treated as overrides
# - Any dict-type config option can be overridden (dl, subtitle, muxing, headers, etc.)
# - Any dl option can be overridden; dashes become underscores (--v-lang -> v_lang)
# - Precedence: CLI arguments > service dl override > global dl section > built-in defaults
# External proxy provider services
proxy_providers:
nordvpn:
username: username_from_service_credentials
password: password_from_service_credentials
# server_map: global mapping that applies to ALL services
# Difference from service-specific proxy_map:
# - server_map: applies to ALL services when --proxy nordvpn:us is used
# - proxy_map: only applies to the specific service configured (see services: EXAMPLE: proxy_map above)
# - proxy_map takes precedence over server_map for that service
server_map:
us: 12 # force US server #12 for US proxies
ca:calgary: 2534 # force CA server #2534 for Calgary proxies
us:seattle: 7890 # force US server #7890 for Seattle proxies
surfsharkvpn:
username: your_surfshark_service_username # Service credentials from https://my.surfshark.com/vpn/manual-setup/main/openvpn
password: your_surfshark_service_password # Service credentials (not your login password)
server_map:
us: 3844 # force US server #3844 for US proxies
gb: 2697 # force GB server #2697 for GB proxies
au: 4621 # force AU server #4621 for AU proxies
us:seattle: 5678 # force US server #5678 for Seattle proxies
ca:toronto: 1234 # force CA server #1234 for Toronto proxies
windscribevpn:
username: your_windscribe_username # Service credentials from https://windscribe.com/getconfig/openvpn
password: your_windscribe_password # Service credentials (not your login password)
server_map:
us: "us-central-096.totallyacdn.com" # force US server
gb: "uk-london-055.totallyacdn.com" # force GB server
us:seattle: "us-west-011.totallyacdn.com" # force US Seattle server
ca:toronto: "ca-toronto-012.totallyacdn.com" # force CA Toronto server
# Gluetun: Dynamic Docker-based VPN proxy (supports 50+ VPN providers)
# Creates Docker containers running Gluetun to bridge VPN connections to HTTP proxies
# Requires Docker to be installed and running
# Usage: --proxy gluetun:windscribe:us or --proxy gluetun:nordvpn:de
gluetun:
# Global settings
base_port: 8888 # Starting port for HTTP proxies (increments for each container)
auto_cleanup: true # Automatically remove containers when done
container_prefix: "unshackle-gluetun" # Docker container name prefix
verify_ip: true # Verify VPN IP matches expected region
# Optional HTTP proxy authentication (for the proxy itself, not VPN)
# auth_user: proxy_user
# auth_password: proxy_password
# VPN provider configurations
providers:
# Windscribe (WireGuard) - Get credentials from https://windscribe.com/getconfig/wireguard
windscribe:
vpn_type: wireguard
credentials:
private_key: "YOUR_WIREGUARD_PRIVATE_KEY"
addresses: "YOUR_WIREGUARD_ADDRESS" # e.g., "10.x.x.x/32"
# Map friendly names to country codes
server_countries:
us: US
uk: GB
ca: CA
de: DE
# NordVPN (OpenVPN) - Get service credentials from https://my.nordaccount.com/dashboard/nordvpn/manual-configuration/
# Note: Service credentials are NOT your email+password - generate them from the link above
# nordvpn:
# vpn_type: openvpn
# credentials:
# username: "YOUR_NORDVPN_SERVICE_USERNAME"
# password: "YOUR_NORDVPN_SERVICE_PASSWORD"
# server_countries:
# us: US
# uk: GB
# ExpressVPN (OpenVPN) - Get credentials from ExpressVPN setup page
# expressvpn:
# vpn_type: openvpn
# credentials:
# username: "YOUR_EXPRESSVPN_USERNAME"
# password: "YOUR_EXPRESSVPN_PASSWORD"
# server_countries:
# us: US
# uk: GB
# Surfshark (WireGuard) - Get credentials from https://my.surfshark.com/vpn/manual-setup/main/wireguard
# surfshark:
# vpn_type: wireguard
# credentials:
# private_key: "YOUR_SURFSHARK_PRIVATE_KEY"
# addresses: "YOUR_SURFSHARK_ADDRESS"
# server_countries:
# us: US
# uk: GB
# Specific server selection: Use format like "us1239" to select specific servers
# Example: --proxy gluetun:nordvpn:us1239 connects to us1239.nordvpn.com
# Supported providers: nordvpn, surfshark, expressvpn, cyberghost
basic:
GB:
- "socks5://username:password@bhx.socks.ipvanish.com:1080" # 1 (Birmingham)
- "socks5://username:password@gla.socks.ipvanish.com:1080" # 2 (Glasgow)
AU:
- "socks5://username:password@syd.socks.ipvanish.com:1080" # 1 (Sydney)
- "https://username:password@au-syd.prod.surfshark.com" # 2 (Sydney)
- "https://username:password@au-bne.prod.surfshark.com" # 3 (Brisbane)
BG: "https://username:password@bg-sof.prod.surfshark.com"
# --- Everything else lives in docs/ ---
# credentials, directories, proxy providers, subtitles, muxing, serve/REST API,
# remote services, per-service overrides, debug logging, Firefox cookie import,
# and more. See docs/README.md for the full index.
+10 -4
View File
@@ -10,9 +10,10 @@ from unshackle.core.vault import Vault
class API(Vault):
"""Key Vault using a simple RESTful HTTP API call."""
def __init__(self, name: str, uri: str, token: str, no_push: bool = False):
def __init__(self, name: str, uri: str, token: str, no_push: bool = False, timeout: float = 10.0):
super().__init__(name, no_push)
self.uri = uri.rstrip("/")
self.timeout = timeout
self.session = Session()
self.session.headers.update({"User-Agent": f"unshackle v{__version__}"})
self.session.headers.update({"Authorization": f"Bearer {token}"})
@@ -22,7 +23,7 @@ class API(Vault):
kid = kid.hex
data = self.session.get(
url=f"{self.uri}/{service.lower()}/{kid}", headers={"Accept": "application/json"}
url=f"{self.uri}/{service.lower()}/{kid}", headers={"Accept": "application/json"}, timeout=self.timeout
).json()
code = int(data.get("code", 0))
@@ -55,6 +56,7 @@ class API(Vault):
url=f"{self.uri}/{service.lower()}",
params={"page": page, "total": 10},
headers={"Accept": "application/json"},
timeout=self.timeout,
).json()
code = int(data.get("code", 0))
@@ -89,7 +91,10 @@ class API(Vault):
kid = kid.hex
data = self.session.post(
url=f"{self.uri}/{service.lower()}/{kid}", json={"content_key": key}, headers={"Accept": "application/json"}
url=f"{self.uri}/{service.lower()}/{kid}",
json={"content_key": key},
headers={"Accept": "application/json"},
timeout=self.timeout,
).json()
code = int(data.get("code", 0))
@@ -135,6 +140,7 @@ class API(Vault):
url=f"{self.uri}/{service.lower()}",
json={"content_keys": batch_keys},
headers={"Accept": "application/json"},
timeout=self.timeout,
)
# Check for HTTP errors that suggest batch is too large
@@ -181,7 +187,7 @@ class API(Vault):
return total_added
def get_services(self) -> Iterator[str]:
data = self.session.post(url=self.uri, headers={"Accept": "application/json"}).json()
data = self.session.post(url=self.uri, headers={"Accept": "application/json"}, timeout=self.timeout).json()
code = int(data.get("code", 0))
message = data.get("message")
+14 -4
View File
@@ -34,6 +34,7 @@ class HTTP(Vault):
username: Optional[str] = None,
api_mode: str = "query",
no_push: bool = False,
timeout: float = 10.0,
):
"""
Initialize HTTP Vault.
@@ -49,6 +50,7 @@ class HTTP(Vault):
"""
super().__init__(name, no_push)
self.url = host
self.timeout = timeout
self.password = api_key or password
if not self.password:
raise ValueError("Either password or api_key is required")
@@ -84,7 +86,7 @@ class HTTP(Vault):
"token": self.password,
}
r = self.session.post(self.url, json=request_payload)
r = self.session.post(self.url, json=request_payload, timeout=self.timeout)
if r.status_code == 404:
return {"status": "not_found"}
@@ -115,7 +117,7 @@ class HTTP(Vault):
try:
request_payload = {"service": service.lower(), "kid": kid}
response = self.session.post(self.url, json=request_payload)
response = self.session.post(self.url, json=request_payload, timeout=self.timeout)
if not response.ok:
return None
@@ -182,6 +184,7 @@ class HTTP(Vault):
response = self.session.get(
self.url,
params={"service": service.lower(), "username": self.username, "password": self.password, "kid": kid},
timeout=self.timeout,
)
data = response.json()
@@ -200,7 +203,9 @@ class HTTP(Vault):
return iter([])
else: # query mode
response = self.session.get(
self.url, params={"service": service.lower(), "username": self.username, "password": self.password}
self.url,
params={"service": service.lower(), "username": self.username, "password": self.password},
timeout=self.timeout,
)
data = response.json()
@@ -250,6 +255,7 @@ class HTTP(Vault):
"key": key,
"title": title,
},
timeout=self.timeout,
)
data = response.json()
@@ -301,6 +307,7 @@ class HTTP(Vault):
"key": key,
"title": title,
},
timeout=self.timeout,
)
data = response.json()
@@ -323,7 +330,9 @@ class HTTP(Vault):
return iter([])
else: # query mode
response = self.session.get(
self.url, params={"username": self.username, "password": self.password, "list_services": True}
self.url,
params={"username": self.username, "password": self.password, "list_services": True},
timeout=self.timeout,
)
data = response.json()
@@ -395,6 +404,7 @@ class HTTP(Vault):
"key": key,
"title": title,
},
timeout=self.timeout,
)
try:
Generated
+708 -290
View File
File diff suppressed because it is too large Load Diff