mirror of
https://github.com/unshackle-dl/unshackle.git
synced 2026-11-22 08:55:07 +00:00
Compare commits
16
Commits
5.2.0
..
ce69e8a028
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ce69e8a028 | ||
|
|
605c0e65cb | ||
|
|
ecf0bf41ab | ||
|
|
3f5b5e0fe1 | ||
|
|
2f86d782bf | ||
|
|
d4b0c81c63 | ||
|
|
9cb20693e0 | ||
|
|
fb90cd87b0 | ||
|
|
8a9a1e1bdc | ||
|
|
40eafce17b | ||
|
|
733cb02391 | ||
|
|
5b724b48f8 | ||
|
|
a2f670b140 | ||
|
|
a9c677c349 | ||
|
|
ba69bc7d61 | ||
|
|
2e1ffebec0 |
@@ -39,7 +39,7 @@ unshackle --help
|
||||
|
||||
External tools on your `PATH` (recommended versions):
|
||||
|
||||
- [Python](https://www.python.org/) - 3.10 - 3.12
|
||||
- [Python](https://www.python.org/) - 3.11 - 3.14
|
||||
- [uv](https://docs.astral.sh/uv/) - ≥ 0.5
|
||||
- [FFmpeg](https://ffmpeg.org/) - ≥ 6.0
|
||||
- [MKVToolNix](https://mkvtoolnix.download/) - ≥ 80
|
||||
|
||||
+16
-2
@@ -159,11 +159,25 @@ debug_keys: true
|
||||
|
||||
---
|
||||
|
||||
## debug_requests (bool)
|
||||
|
||||
Controls whether HTTP-library logs are shown during `--debug` sessions. Default: `false`
|
||||
|
||||
When `--debug` is active and this is `false`, unshackle silences the noisy per-request logs emitted by `urllib3`, `requests`, `rnet`, `httpx`, and related libraries (e.g. `GET ...seg-001.m4s 200`) so that unshackle's own structured DEBUG output stays readable. Set to `true` to see every individual HTTP request, including per-segment downloads.
|
||||
|
||||
Has no effect when `--debug` is not active.
|
||||
|
||||
```yaml
|
||||
debug_requests: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## set_terminal_bg (bool)
|
||||
|
||||
Controls whether unshackle should set the terminal background color. Default: `false`
|
||||
Controls whether unshackle applies its theme background color to the terminal. Default: `false`
|
||||
|
||||
For example,
|
||||
When enabled, the Rich console theme's background color is appended to the foreground styles for both the general color scheme and the ASCII art banner, so the full color palette renders correctly on terminals whose default background differs from the theme. Set to `true` if the banner or colored output looks off on your terminal.
|
||||
|
||||
```yaml
|
||||
set_terminal_bg: true
|
||||
|
||||
+92
@@ -93,6 +93,31 @@ serve:
|
||||
|
||||
Layering order: built-in defaults < `serve.*` overrides < service-specific click defaults < request body.
|
||||
|
||||
### Per-request override gates
|
||||
|
||||
Two security gates under `serve:` guard capabilities that `/api/download` clients can request. Both default OFF; an unset or `false` value causes the server to reject the request with `403 FORBIDDEN`.
|
||||
|
||||
**`cdm_overrides`** — a `/api/download` job body may include a `cdm` field to select a specific server-side device. When this gate is off, every such request is rejected. Set to `true` to allow any device the server has, or to a list of device names to restrict to an explicit subset:
|
||||
|
||||
```yaml
|
||||
serve:
|
||||
# Allow any server-side device:
|
||||
cdm_overrides: true
|
||||
|
||||
# Or restrict to specific devices:
|
||||
cdm_overrides:
|
||||
- generic_nexus_4464_l3
|
||||
```
|
||||
|
||||
**`allow_job_credentials`** — a `/api/download` job body may include a `credential` (single) or `credentials` (map) field to authenticate the job with client-supplied secrets instead of the server's configured credentials. When this gate is off, any job containing either field is rejected. Each distinct credential gets its own isolated token cache on the server. Set to `true` to allow:
|
||||
|
||||
```yaml
|
||||
serve:
|
||||
allow_job_credentials: true
|
||||
```
|
||||
|
||||
Both gates are independent and can be combined. A typical locked-down deployment leaves both unset; a trusted single-client deployment might enable both.
|
||||
|
||||
---
|
||||
|
||||
## Endpoint Map
|
||||
@@ -779,3 +804,70 @@ downloading -> cancelled
|
||||
Jobs are retained for 24 hours after completion (override via top-level `download_job_retention_hours` in `unshackle.yaml`). The server runs up to 2 concurrent download jobs by default; override via top-level `max_concurrent_downloads`. This is independent of `serve.downloads`, which controls parallel tracks **within** a single job.
|
||||
|
||||
Remote sessions are managed by `SessionStore` (`unshackle/core/api/session_store.py`); idle sessions and their `InputBridge` instances are cleaned up by a background loop started/stopped with the app lifecycle.
|
||||
|
||||
---
|
||||
|
||||
## Client Configuration: `remote_services`
|
||||
|
||||
`remote_services` is a top-level block in `unshackle.yaml` on the **client** machine. It tells `dl` how to reach a remote `unshackle serve` instance instead of running service code locally. The client handles track selection, download, decrypt, and mux; the server handles auth, manifest parsing, and (optionally) DRM licensing.
|
||||
|
||||
### CLI flags
|
||||
|
||||
| Flag | Description |
|
||||
| --- | --- |
|
||||
| `--remote` | Use a remote server instead of local service code. Reads `remote_services` from config. |
|
||||
| `--server <name>` | Select a named server when multiple are configured. If only one entry exists it is picked automatically. |
|
||||
|
||||
```bash
|
||||
unshackle dl --remote EXAMPLE1 "https://example.com/show/abc123"
|
||||
unshackle dl --remote --server us-server EXAMPLE1 "https://example.com/show/abc123"
|
||||
```
|
||||
|
||||
### Config structure
|
||||
|
||||
```yaml
|
||||
remote_services:
|
||||
my-server:
|
||||
url: "http://192.168.1.100:8786" # base URL of the remote serve instance
|
||||
api_key: "your-secret-key-here" # sent as X-Secret-Key; omit or set "" for --no-key servers
|
||||
|
||||
# server_cdm: the server runs its own CDM and returns KID:KEY pairs (mode: server_cdm).
|
||||
# false (default): client runs its own CDM; license challenges are proxied through the server.
|
||||
server_cdm: false
|
||||
|
||||
# Per-service overrides applied on the client side after the remote session is set up.
|
||||
# Supported keys: decryption, downloader, cdm (and any free-form service config keys).
|
||||
services:
|
||||
EXAMPLE1:
|
||||
decryption: mp4decrypt # override decryption tool for this service (shaka or mp4decrypt)
|
||||
```
|
||||
|
||||
**`server_cdm`** — when `false` (default) the client builds CDM challenges locally and calls `POST /api/session/{id}/license` with `mode: "proxy"`; the server forwards the challenge to the service's license endpoint and returns raw license bytes. When `true` the client skips local CDM setup and calls with `mode: "server_cdm"`; the server performs the full CDM flow and returns `{ "keys": { "<KID>": "<KEY>" } }`. The server-side user key must have a matching device configured (`devices` / `playready_devices`) for `server_cdm` to work.
|
||||
|
||||
### Multiple servers
|
||||
|
||||
When more than one server is configured, `--server <name>` is required:
|
||||
|
||||
```yaml
|
||||
remote_services:
|
||||
us-server:
|
||||
url: "https://us.example.com:8786"
|
||||
api_key: "us-key"
|
||||
server_cdm: true
|
||||
services:
|
||||
EXAMPLE1:
|
||||
decryption: mp4decrypt
|
||||
|
||||
eu-server:
|
||||
url: "https://eu.example.com:8786"
|
||||
api_key: "eu-key"
|
||||
server_cdm: false
|
||||
```
|
||||
|
||||
```bash
|
||||
unshackle dl --remote --server us-server EXAMPLE1 "https://example.com/show/abc123"
|
||||
```
|
||||
|
||||
### Cache forwarding
|
||||
|
||||
On session open the client sends any local `*.json` cache files for the service (tokens, etc.) to the server so it can skip interactive re-auth. On session close the server returns updated cache files; the client writes them back locally so the next remote session can forward them again.
|
||||
|
||||
+112
-37
@@ -178,78 +178,141 @@ remote_cdm:
|
||||
use_vaults: true # Integrate with vault system
|
||||
```
|
||||
|
||||
**Advanced Example with Field Mapping:**
|
||||
**Full field reference:**
|
||||
|
||||
```yaml
|
||||
remote_cdm:
|
||||
- name: advanced_custom_api
|
||||
type: custom_api
|
||||
host: https://api.example.com
|
||||
device:
|
||||
name: L1
|
||||
type: ANDROID
|
||||
security_level: 1
|
||||
timeout: 30 # Default request timeout (seconds); overridden per-endpoint
|
||||
|
||||
# Authentication configuration
|
||||
device:
|
||||
name: L1 # Passed as 'scheme' in base request params
|
||||
type: ANDROID # CHROME, ANDROID, or PLAYREADY
|
||||
system_id: 26830 # Widevine system ID (informational; not sent)
|
||||
security_level: 1 # 1–3 for Widevine; 2000/3000 for PlayReady
|
||||
|
||||
# Authentication — applied to every request via session headers or Basic auth
|
||||
auth:
|
||||
type: header
|
||||
header_name: X-API-Key
|
||||
key: YOUR_SECRET_KEY
|
||||
custom_headers:
|
||||
type: header # bearer | header | basic | body | query
|
||||
header_name: X-API-Key # used when type=header (default: Authorization)
|
||||
key: YOUR_SECRET_KEY # API key / token / password
|
||||
# bearer_token: TOKEN # explicit token for type=bearer (falls back to 'key')
|
||||
# username: user # required for type=basic
|
||||
# password: pass # required for type=basic
|
||||
custom_headers: # merged into every request
|
||||
User-Agent: Unshackle/3.1.0
|
||||
X-Client-Version: "1.0"
|
||||
|
||||
# Endpoint configuration
|
||||
# Endpoints — both keys are required
|
||||
endpoints:
|
||||
get_request:
|
||||
get_request: # first call: obtain challenge or cached keys
|
||||
path: /v2/challenge
|
||||
method: POST
|
||||
timeout: 30
|
||||
decrypt_response:
|
||||
timeout: 30 # overrides top-level timeout for this endpoint
|
||||
decrypt_response: # second call: exchange license response for keys
|
||||
path: /v2/decrypt
|
||||
method: POST
|
||||
timeout: 30
|
||||
|
||||
# Request parameter mapping
|
||||
# request_mapping — transform outgoing parameters before each request
|
||||
# Base params sent to get_request: scheme, init_data, [service], [service_certificate]
|
||||
# Base params sent to decrypt_response: scheme, session_id, init_data, license_request, license_response
|
||||
request_mapping:
|
||||
get_request:
|
||||
param_names:
|
||||
init_data: pssh # Rename 'init_data' to 'pssh'
|
||||
scheme: device_type # Rename 'scheme' to 'device_type'
|
||||
static_params:
|
||||
api_version: "2.0" # Add static parameter
|
||||
param_names: # rename base param keys
|
||||
init_data: pssh
|
||||
scheme: device_type
|
||||
static_params: # always-added fixed values
|
||||
api_version: "2.0"
|
||||
conditional_params: # added only when condition is true
|
||||
- condition: "is_playready == True"
|
||||
params:
|
||||
drm_type: playready
|
||||
transforms: # applied after renaming; type values listed below
|
||||
- param: pssh
|
||||
type: base64_encode
|
||||
nested_params: # group named params under a parent key
|
||||
payload:
|
||||
- pssh
|
||||
- device_type
|
||||
exclude_params: # remove unwanted params before sending
|
||||
- service_certificate
|
||||
decrypt_response:
|
||||
param_names:
|
||||
license_request: challenge
|
||||
license_response: license
|
||||
|
||||
# Response field mapping
|
||||
# response_mapping — extract and validate fields from API responses
|
||||
response_mapping:
|
||||
get_request:
|
||||
fields:
|
||||
challenge: data.challenge # Deep field access
|
||||
fields: # map standard names to dot-path locations in response JSON
|
||||
challenge: data.challenge
|
||||
session_id: session.id
|
||||
success_conditions:
|
||||
- status == 'ok' # Validate response
|
||||
message: status.message
|
||||
response_types: # classify response so CDM can choose the right path
|
||||
- condition: "message_type == 'license-request'"
|
||||
type: license_request # CDM expects challenge + session_id
|
||||
- condition: "message_type == 'cached-keys'"
|
||||
type: cached_keys # CDM skips license round-trip
|
||||
success_conditions: # all must be true; failure raises RequestException
|
||||
- "message == 'success'"
|
||||
error_fields: # inspected when success_conditions fail (default: error, message, details)
|
||||
- error
|
||||
- details
|
||||
transforms: # applied to extracted fields
|
||||
- field: challenge
|
||||
type: base64_decode
|
||||
decrypt_response:
|
||||
fields:
|
||||
keys: data.keys
|
||||
key_fields:
|
||||
kid: key_id # Map 'kid' field
|
||||
key: content_key # Map 'key' field
|
||||
message: status.message
|
||||
key_fields: # map key-object property names to standard names
|
||||
kid: key_id # default: kid
|
||||
key: content_key # default: key
|
||||
type: key_type # default: type (value used as-is; CONTENT treated as content key)
|
||||
success_conditions:
|
||||
- "message == 'success'"
|
||||
|
||||
# caching — vault and API-level key caching
|
||||
caching:
|
||||
enabled: true
|
||||
use_vaults: true
|
||||
check_cached_first: true # Check cache before API calls
|
||||
enabled: true # master switch (default: true)
|
||||
use_vaults: true # check/write unshackle key vaults (default: true)
|
||||
check_cached_first: true # check vaults before making any API call (default: true)
|
||||
```
|
||||
|
||||
**Supported Authentication Types:**
|
||||
- `bearer` - Bearer token authentication
|
||||
- `header` - Custom header authentication
|
||||
- `basic` - HTTP Basic authentication
|
||||
- `body` - Credentials in request body
|
||||
- `query` - Authentication added to query string parameters
|
||||
**Supported `transforms` types** (for both `request_mapping` and `response_mapping`):
|
||||
`base64_encode`, `base64_decode`, `hex_encode`, `hex_decode`, `json_stringify`, `json_parse`, `parse_key_string`
|
||||
|
||||
**Supported authentication types:**
|
||||
- `bearer` — `Authorization: Bearer <key>` (or `bearer_token` field)
|
||||
- `header` — arbitrary header; set `header_name` (default `Authorization`)
|
||||
- `basic` — HTTP Basic auth via `username` / `password`
|
||||
- `body` — credentials injected into request body (handled by request_mapping)
|
||||
- `query` — credentials appended to query string (handled by request_mapping)
|
||||
|
||||
### PyPlayReady Remote CDM
|
||||
|
||||
Connects to a running `unshackle serve` instance for PlayReady licensing. Uses
|
||||
`pyplayready.remote.remotecdm.RemoteCdm` under the hood. No `type` field is needed —
|
||||
the entry is routed to PlayReady when `device_type: PLAYREADY` is set (same branching
|
||||
logic as the legacy Widevine path).
|
||||
|
||||
```yaml
|
||||
remote_cdm:
|
||||
- name: playready_remote
|
||||
device_name: my_prd_device # Device name registered on the serve instance
|
||||
device_type: PLAYREADY # Required to select the PlayReady code path
|
||||
system_id: 0 # Not used by pyplayready RemoteCdm; set to 0
|
||||
security_level: 3000 # 2000 = SL2000, 3000 = SL3000
|
||||
host: http://127.0.0.1:8786/playready # Must include the /playready path suffix
|
||||
secret: your-api-secret-key # X-Secret-Key sent to the serve instance
|
||||
```
|
||||
|
||||
**Note:** The `/playready` path suffix in `host` is required — `pyplayready.RemoteCdm`
|
||||
appends its own endpoint paths on top of this base, so omitting the suffix will result
|
||||
in 404 errors.
|
||||
|
||||
### Legacy PyWidevine Serve Format
|
||||
|
||||
@@ -393,6 +456,16 @@ Additional behavior:
|
||||
|
||||
- `no_push` (bool): Optional per-vault flag. When `true`, the vault will not receive pushed keys (writes) but
|
||||
will still be queried and can provide keys for lookups. Useful for read-only/backup vaults.
|
||||
- `timeout` (float): Optional per-vault network timeout in seconds for the remote `API` and `HTTP` vaults.
|
||||
Defaults to the global `vault_timeout` (10 seconds) so an unreachable vault host fails fast instead of
|
||||
hanging the download. Set it on a vault to override the global, e.g. `timeout: 5.0`. Has no effect on
|
||||
SQLite (local) or MySQL (uses its driver's own connect timeout).
|
||||
|
||||
Set a global default for all remote vaults at the top level of the config:
|
||||
|
||||
```yaml
|
||||
vault_timeout: 10.0 # seconds; per-vault `timeout:` overrides this
|
||||
```
|
||||
|
||||
### Using an API Vault
|
||||
|
||||
@@ -407,6 +480,7 @@ not work in unshackle. The API format can be seen in the [API Vault Code](unshac
|
||||
# uri: "https://api.example.com/key-vault"
|
||||
token: "random secret key" # authorization token
|
||||
# no_push: true # optional; make this API vault read-only (lookups only)
|
||||
# timeout: 5.0 # optional; per-vault network timeout (seconds), overrides vault_timeout
|
||||
```
|
||||
|
||||
### Using a MySQL Vault
|
||||
@@ -470,6 +544,7 @@ is useful for integrating with various third-party key vault APIs.
|
||||
api_mode: "json" # query, json, or decrypt_labs
|
||||
# username: "user" # required for query mode only
|
||||
# no_push: false # optional; defaults to false
|
||||
# timeout: 5.0 # optional; per-vault network timeout (seconds), overrides vault_timeout
|
||||
```
|
||||
|
||||
**Supported API Modes:**
|
||||
|
||||
@@ -241,3 +241,81 @@ Common issues:
|
||||
|
||||
- [Gluetun Wiki](https://github.com/qdm12/gluetun-wiki) - Official provider documentation
|
||||
- [Gluetun GitHub](https://github.com/qdm12/gluetun)
|
||||
|
||||
---
|
||||
|
||||
# ExpressVPN HTTPS Proxy
|
||||
|
||||
ExpressVPN is a standalone HTTPS proxy provider — it does **not** use Docker or the gluetun sub-provider system. It authenticates via a browser-extension OAuth flow (Keycloak/PKCE) and returns an authenticated `https://cat:<token>@<host>:443` proxy URL.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
An active ExpressVPN subscription is required. The provider resolves proxy-capable locations through ExpressVPN's own API, so no local VPN client or Docker container is needed.
|
||||
|
||||
## Authentication
|
||||
|
||||
The provider uses a token cascade (tried in order):
|
||||
|
||||
1. **Cached tokens** — `cache/global/expressvpn_tokens.json` (auto-written after first auth; access and refresh tokens are refreshed automatically when they expire).
|
||||
2. **Exported browser cookies** — `cookies/vpn/expressvpn.txt` (also accepted at `cookies/vpns/expressvpn.txt`). Must contain the `KEYCLOAK_IDENTITY` and `KEYCLOAK_SESSION` cookies from `auth.expressvpn.com`. Used for a one-time PKCE bootstrap that populates the token cache.
|
||||
3. **Desktop `account.json`** — configured via the `account_json` key (see below).
|
||||
|
||||
Export cookies from a browser that is logged in to `auth.expressvpn.com`. Any Netscape-format, JSON array, or `name=value` file is accepted.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add to `~/.config/unshackle/unshackle.yaml` under `proxy_providers`:
|
||||
|
||||
```yaml
|
||||
proxy_providers:
|
||||
expressvpn:
|
||||
region_map:
|
||||
us: "ny" # --proxy expressvpn:us → New York (default city)
|
||||
gb: "london" # --proxy expressvpn:gb → London
|
||||
de: "frankfurt" # --proxy expressvpn:de → Frankfurt
|
||||
es: "madrid" # --proxy expressvpn:es → Madrid
|
||||
mx: # --proxy expressvpn:mx → smart/random MX location
|
||||
```
|
||||
|
||||
`region_map` maps a country code to a default city preset. An empty (null) value enables smart connection (a random location in that country). The preset is only applied when the CLI query contains no city; an explicit city in the query always takes precedence.
|
||||
|
||||
### Optional keys
|
||||
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `region_map` | `{}` | Country → default city presets (see above) |
|
||||
| `server_map` | `{}` | Alias → ExpressVPN location slug or direct hostname |
|
||||
| `cookie_path` | `cookies/vpn/expressvpn.txt` | Path to exported browser cookies |
|
||||
| `cache_path` | `cache/global/expressvpn_tokens.json` | Token cache location |
|
||||
| `account_json` | *(none)* | Path to ExpressVPN desktop `account.json` |
|
||||
| `refresh_token` | *(none)* | OAuth refresh token (manual override) |
|
||||
| `timeout` | `10.0` | HTTP request timeout in seconds |
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
unshackle dl SERVICE CONTENT --proxy expressvpn:us # random/smart US location
|
||||
unshackle dl SERVICE CONTENT --proxy expressvpn:us-ny # specific city (New York)
|
||||
unshackle dl SERVICE CONTENT --proxy expressvpn:us-ny-2 # pinned server #2 in New York
|
||||
unshackle dl SERVICE CONTENT --proxy expressvpn:mx # random/smart Mexico location
|
||||
unshackle dl SERVICE CONTENT --proxy expressvpn:es-madrid # specific city (Madrid)
|
||||
```
|
||||
|
||||
Query format: `expressvpn:<country>[-<city>[-<N>]]`
|
||||
|
||||
- `country` — 2-letter country code (e.g. `us`, `gb`, `de`).
|
||||
- `city` — optional city abbreviation or slug matched against location names using first-letter abbreviation (`ny` → "New York"), exact slug, prefix, then substring strategies.
|
||||
- `N` — optional 1-based server index (e.g. `-2` selects the second server in the matched city). If `N` exceeds the available server count, a random server is selected.
|
||||
|
||||
## City matching
|
||||
|
||||
The city part of the query is matched against ExpressVPN location names using these strategies (in priority order):
|
||||
|
||||
1. First-letter abbreviation — `ny` matches "New York"
|
||||
2. Exact slug — `miami` matches "Miami"
|
||||
3. Prefix match — `mia` matches "Miami"
|
||||
4. Substring match — `york` matches "New York"
|
||||
|
||||
## Token cache
|
||||
|
||||
After the first successful authentication the token cache is written to `cache/global/expressvpn_tokens.json` (mode 0600). Subsequent runs refresh expired access/SRT/connection tokens automatically without re-reading cookies.
|
||||
|
||||
+77
-7
@@ -32,16 +32,66 @@ filenames:
|
||||
|
||||
## output_template (dict)
|
||||
|
||||
Configure custom output filename templates for movies, series, and songs.
|
||||
Configure custom output filename templates for movies, series, songs, and albums.
|
||||
This is **required** in your `unshackle.yaml` — a warning is shown if not configured.
|
||||
|
||||
Available variables: `{title}`, `{year}`, `{season}`, `{episode}`, `{season_episode}`, `{episode_name}`,
|
||||
`{quality}`, `{resolution}`, `{source}`, `{audio}`, `{audio_channels}`, `{audio_full}`,
|
||||
`{video}`, `{hdr}`, `{hfr}`, `{atmos}`, `{dual}`, `{multi}`, `{tag}`, `{edition}`, `{repack}`,
|
||||
`{lang_tag}`, `{track_number}`, `{artist}`, `{album}`, `{disc}`
|
||||
|
||||
Add `?` suffix to make a variable conditional (omitted when empty): `{year?}`, `{hdr?}`, `{repack?}`
|
||||
|
||||
### General variables
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `title` | Title of the movie, episode, or track |
|
||||
| `year` | Release year |
|
||||
| `source` | Service source tag (e.g. `AMZN`, `NF`) |
|
||||
| `tag` | Group/release tag configured via `tag:` |
|
||||
| `edition` | Edition label (e.g. `Extended`, `Theatrical`) |
|
||||
| `repack` | `REPACK` when the release is a repack, otherwise empty |
|
||||
| `lang_tag` | Language tag produced by `language_tags` rules (e.g. `NORDiC`) |
|
||||
|
||||
### Series / episode variables
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `season` | Season number (e.g. `1`) |
|
||||
| `episode` | Episode number (e.g. `1`) |
|
||||
| `season_episode` | Combined season/episode (e.g. `S01E01`) |
|
||||
| `episode_name` | Episode title |
|
||||
|
||||
### Video / audio variables
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `quality` | Resolution label (e.g. `1080p`, `2160p`) |
|
||||
| `resolution` | Raw resolution string |
|
||||
| `video` | Video codec label (e.g. `H.264`, `H.265`) |
|
||||
| `hdr` | HDR format label (e.g. `HDR`, `DV`) |
|
||||
| `hfr` | HFR label when frame rate exceeds normal threshold |
|
||||
| `audio` | Audio codec label (e.g. `AAC`, `EAC3`) |
|
||||
| `audio_channels` | Channel count (e.g. `2.0`, `5.1`) |
|
||||
| `audio_full` | Combined codec + channels (e.g. `DDP5.1`) |
|
||||
| `atmos` | `Atmos` when Dolby Atmos is present, otherwise empty |
|
||||
| `dual` | `DUAL` when two audio languages are present, otherwise empty |
|
||||
| `multi` | `MULTi` when three or more audio languages are present, otherwise empty |
|
||||
|
||||
### Music variables
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `track_number` | Track number within album |
|
||||
| `disc` | Disc number |
|
||||
| `track_total` | Total tracks on disc |
|
||||
| `disc_total` | Total discs in release |
|
||||
| `artist` | Track artist |
|
||||
| `album_artist` | Album-level artist (may differ from track artist) |
|
||||
| `album` | Album title |
|
||||
| `release_type` | Release type (e.g. `Album`, `Single`, `EP`) |
|
||||
| `genre` | Genre string |
|
||||
| `explicit` | `Explicit` when track has explicit content flag, otherwise empty |
|
||||
| `isrc` | International Standard Recording Code |
|
||||
| `upc` | Universal Product Code (album barcode) |
|
||||
| `label` | Record label name |
|
||||
|
||||
```yaml
|
||||
output_template:
|
||||
# Scene-style (dot-separated)
|
||||
@@ -71,7 +121,9 @@ If not configured, the default folder naming is used:
|
||||
- Songs: `Artist - Album (Year)`
|
||||
|
||||
`folder` accepts either a single string (applies to all title kinds) or a mapping with per-kind
|
||||
templates keyed by `movies`, `series`, and/or `songs`. Unknown keys are warned about and ignored.
|
||||
templates keyed by `movies`, `series`, `songs`, and/or `albums`. Unknown keys are warned about and ignored.
|
||||
|
||||
Useful music variables for album folders: `album_artist`, `album`, `artist`, `year`, `genre`, `label`, `release_type`, `track_total`, `disc_total`.
|
||||
|
||||
```yaml
|
||||
output_template:
|
||||
@@ -90,6 +142,7 @@ output_template:
|
||||
# movies: '{title} ({year})'
|
||||
# series: '{title} ({year?})'
|
||||
# songs: '{artist} - {album} ({year?})'
|
||||
# albums: '{album_artist} - {album} ({year?})'
|
||||
```
|
||||
|
||||
Example outputs:
|
||||
@@ -204,6 +257,23 @@ Enable/disable tagging downloaded files with IMDB/TMDB/TVDB identifiers (when av
|
||||
Note: The `--split-audio` CLI flag overrides this setting. When `--split-audio` is passed,
|
||||
`merge_audio` is effectively set to `false` for that run.
|
||||
|
||||
- `merge_video`
|
||||
Merge video **language variants** into one file. Default: `false`
|
||||
- `false`: One MKV per video track (the default behaviour).
|
||||
- `true`: Group the selected video tracks by `(resolution, range, codec)` and merge
|
||||
each group into one MKV, so only language varies within a file. The player switches
|
||||
between the language tracks. No re-encode, no concatenation.
|
||||
|
||||
Only the language dimension is collapsed. Different **resolutions**, **ranges**
|
||||
(SDR/HDR10/HDR10+/DV/HYBRID) and **codecs** (H264/H265) always stay in separate files.
|
||||
For example, `-r HYBRID,DV,HDR10,SDR --merge-video` produces one file per range (never a
|
||||
single combined file), while a title offering English + French video of the same
|
||||
resolution/range/codec produces one file containing both video tracks.
|
||||
|
||||
Note: The `--merge-video` CLI flag overrides this setting. Can be set per service under
|
||||
`services.<TAG>.muxing.merge_video`. Change `group_videos_by_variant` in
|
||||
`unshackle/commands/dl.py` to adjust the grouping.
|
||||
|
||||
- `default_language` (dict)
|
||||
Override which track is flagged as the default in the muxed MKV, regardless
|
||||
of the title's original language. Useful when you always want your player to
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# docs/
|
||||
|
||||
This folder is the **canonical configuration reference** for unshackle. `unshackle/unshackle-example.yaml` is a minimal starter config only — it intentionally omits most options. Come here for the full picture.
|
||||
|
||||
## Contents
|
||||
|
||||
| Document | Description |
|
||||
|---|---|
|
||||
| [Advanced & System Configuration](ADVANCED_CONFIG.md) | `serve`, debugging, and system-level options not covered elsewhere |
|
||||
| [REST API Documentation](API.md) | HTTP API exposed by `unshackle serve`: endpoints, auth, remote downloads, CDM hosting |
|
||||
| [Structured Debug Logging](DEBUG_LOGGING.md) | `-d`/`--debug` JSONL output: schema, redaction rules, `log_event`/`timed_operation` conventions |
|
||||
| [Download & Processing Configuration](DOWNLOAD_CONFIG.md) | Downloader settings, concurrency, retries, and post-processing behaviour |
|
||||
| [DRM & CDM Configuration](DRM_CONFIG.md) | Widevine, PlayReady, and ClearKey CDM setup; local and remote CDMs; key vaults |
|
||||
| [Gluetun VPN Proxy](GLUETUN.md) | Docker-managed VPN proxies via Gluetun; setup, WireGuard ready-detection |
|
||||
| [Network & Proxy Configuration](NETWORK_CONFIG.md) | Proxy providers, geofence bypass, per-service proxy overrides |
|
||||
| [Output & Naming Configuration](OUTPUT_CONFIG.md) | Output directory layout, filename templates, folder naming |
|
||||
| [Service Integration & Authentication](SERVICE_CONFIG.md) | Per-service config blocks, cookies, credentials, and metadata sources |
|
||||
| [Subtitle Processing Configuration](SUBTITLE_CONFIG.md) | `subtitle:` key options: format conversion, styling, forced-sub rules |
|
||||
@@ -150,6 +150,47 @@ Please be aware that this information is sensitive and to keep it safe. Do not s
|
||||
|
||||
---
|
||||
|
||||
## firefox_cookies (dict)
|
||||
|
||||
Automatically extract cookies from your running Firefox installation instead of maintaining a
|
||||
`cookies/<SERVICE>.txt` file. Configured as a top-level dict keyed by service tag; each value is
|
||||
a settings dict for that service.
|
||||
|
||||
```yaml
|
||||
firefox_cookies:
|
||||
EXAMPLE:
|
||||
hosts:
|
||||
- example.com
|
||||
- api.example.com
|
||||
local_storage: false # optional, default false
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
|
||||
- `hosts` — list of host patterns to match against Firefox's cookie store. Each pattern must be at
|
||||
least 3 characters. A pattern matches the exact host **and** any dot-suffix subdomain
|
||||
(e.g. `example.com` also matches `sub.example.com` and `.example.com`).
|
||||
- `local_storage` — when `true`, also harvest `localStorage` entries for the listed hosts from
|
||||
Firefox's `webappsstore.sqlite`. Default: `false`.
|
||||
|
||||
At download time unshackle picks the **most recently modified** valid Firefox profile (the one
|
||||
whose `cookies.sqlite` has the latest mtime), copies the database to a restricted temporary
|
||||
directory (`0700`), and reads cookies from it without touching the live profile. If extraction
|
||||
returns no cookies or fails for any reason, it falls back silently to the normal file-based cookie
|
||||
(`cookies/<SERVICE>.txt` or `cookies/<SERVICE>/<profile>.txt`).
|
||||
|
||||
**Notes:**
|
||||
|
||||
- Firefox must not have an exclusive write lock on the database at the moment of extraction (i.e.
|
||||
Firefox may be open — the extractor copies both `cookies.sqlite` and its WAL file to capture any
|
||||
unflushed writes).
|
||||
- `local_storage: true` is only needed for services that store auth tokens in the browser's
|
||||
localStorage rather than in HTTP cookies.
|
||||
- The feature is implemented in `core/utils/firefox_cookie_extractor.py`; the fallback logic lives
|
||||
in `dl.get_cookie_jar` (`commands/dl.py`).
|
||||
|
||||
---
|
||||
|
||||
## tmdb_api_key (str)
|
||||
|
||||
API key for The Movie Database (TMDB). This is used for tagging downloaded files with TMDB,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
This document covers subtitle processing and formatting options under the top-level `subtitle:` key in `unshackle.yaml`.
|
||||
|
||||
For the canonical example, see `unshackle/unshackle-example.yaml`.
|
||||
This document is the canonical reference for subtitle configuration. `unshackle/unshackle-example.yaml` is a minimal starter only.
|
||||
|
||||
## subtitle (dict)
|
||||
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ name = "unshackle"
|
||||
version = "5.2.0"
|
||||
description = "Modular Movie, TV, and Music Archival Software."
|
||||
authors = [{ name = "unshackle team" }]
|
||||
requires-python = ">=3.10,<3.13"
|
||||
requires-python = ">=3.11,<3.15"
|
||||
readme = "README.md"
|
||||
license = "GPL-3.0-only"
|
||||
keywords = [
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Tests for ``--merge-video`` track grouping.
|
||||
|
||||
``group_videos_by_variant`` (``unshackle/commands/dl.py``) decides which selected video
|
||||
tracks share one MKV when merge mode is on. The rule: group by ``(resolution, range,
|
||||
codec)`` so only language varies within a file; resolutions, ranges and codecs stay
|
||||
separate. With ``merge=False`` every track is its own group (one file per track).
|
||||
|
||||
These lock the pure grouping unit; the surrounding mux loop is Click-command orchestration.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unshackle.commands.dl import group_videos_by_variant
|
||||
from unshackle.core.tracks import Video
|
||||
|
||||
|
||||
def make_video(
|
||||
track_id: str,
|
||||
*,
|
||||
range_: Video.Range,
|
||||
height: int,
|
||||
codec: Video.Codec,
|
||||
language: str = "en",
|
||||
) -> Video:
|
||||
return Video(
|
||||
id_=track_id,
|
||||
url=f"https://example.test/{track_id}.m3u8",
|
||||
language=language,
|
||||
codec=codec,
|
||||
range_=range_,
|
||||
width=int(height * 16 / 9),
|
||||
height=height,
|
||||
bitrate=1_000_000,
|
||||
)
|
||||
|
||||
|
||||
HEVC = Video.Codec.HEVC
|
||||
AVC = Video.Codec.AVC
|
||||
SDR = Video.Range.SDR
|
||||
HDR10 = Video.Range.HDR10
|
||||
DV = Video.Range.DV
|
||||
|
||||
|
||||
def test_merge_collapses_language_only() -> None:
|
||||
"""Same (height, range, codec), different language → one group."""
|
||||
videos = [
|
||||
make_video("en", range_=SDR, height=1080, codec=HEVC, language="en"),
|
||||
make_video("fr", range_=SDR, height=1080, codec=HEVC, language="fr"),
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=True)
|
||||
assert len(groups) == 1
|
||||
assert [v.id for v in groups[0]] == ["en", "fr"]
|
||||
|
||||
|
||||
def test_merge_splits_on_codec() -> None:
|
||||
"""H264 vs H265 of the same resolution+range → separate groups."""
|
||||
videos = [
|
||||
make_video("hevc", range_=SDR, height=1080, codec=HEVC),
|
||||
make_video("avc", range_=SDR, height=1080, codec=AVC),
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=True)
|
||||
assert len(groups) == 2
|
||||
assert all(len(g) == 1 for g in groups)
|
||||
|
||||
|
||||
def test_merge_splits_on_range() -> None:
|
||||
"""SDR vs HDR10 of the same resolution+codec → separate groups."""
|
||||
videos = [
|
||||
make_video("sdr", range_=SDR, height=1080, codec=HEVC),
|
||||
make_video("hdr10", range_=HDR10, height=1080, codec=HEVC),
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=True)
|
||||
assert len(groups) == 2
|
||||
|
||||
|
||||
def test_merge_splits_on_resolution() -> None:
|
||||
"""1080p vs 2160p of the same range+codec → separate groups."""
|
||||
videos = [
|
||||
make_video("1080", range_=SDR, height=1080, codec=HEVC),
|
||||
make_video("2160", range_=SDR, height=2160, codec=HEVC),
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=True)
|
||||
assert len(groups) == 2
|
||||
|
||||
|
||||
def test_merge_multi_range_yields_one_group_per_range() -> None:
|
||||
"""Regression guard: -r HYBRID,DV,HDR10,SDR must never collapse into one file.
|
||||
|
||||
HYBRID is resolved upstream into a DV deliverable plus the requested standalone
|
||||
ranges; here the four selected single-range tracks must stay in four groups.
|
||||
"""
|
||||
videos = [
|
||||
make_video("sdr", range_=SDR, height=2160, codec=HEVC),
|
||||
make_video("hdr10", range_=HDR10, height=2160, codec=HEVC),
|
||||
make_video("dv", range_=DV, height=2160, codec=HEVC),
|
||||
make_video("dv-hybrid", range_=DV, height=1080, codec=HEVC), # different height
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=True)
|
||||
assert len(groups) == 4
|
||||
|
||||
|
||||
def test_no_merge_yields_one_group_per_track() -> None:
|
||||
"""merge=False reproduces today's per-track behaviour exactly."""
|
||||
videos = [
|
||||
make_video("en", range_=SDR, height=1080, codec=HEVC, language="en"),
|
||||
make_video("fr", range_=SDR, height=1080, codec=HEVC, language="fr"),
|
||||
make_video("avc", range_=SDR, height=1080, codec=AVC),
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=False)
|
||||
assert len(groups) == 3
|
||||
assert all(len(g) == 1 for g in groups)
|
||||
|
||||
|
||||
def test_merge_preserves_first_seen_order() -> None:
|
||||
"""Group order follows first-seen track order, for stable output filenames."""
|
||||
videos = [
|
||||
make_video("hevc-en", range_=SDR, height=1080, codec=HEVC, language="en"),
|
||||
make_video("avc-en", range_=SDR, height=1080, codec=AVC, language="en"),
|
||||
make_video("hevc-fr", range_=SDR, height=1080, codec=HEVC, language="fr"),
|
||||
]
|
||||
groups = group_videos_by_variant(videos, merge=True)
|
||||
# HEVC group seen first (and gathers both languages), AVC group second.
|
||||
assert [v.id for v in groups[0]] == ["hevc-en", "hevc-fr"]
|
||||
assert [v.id for v in groups[1]] == ["avc-en"]
|
||||
|
||||
|
||||
def test_empty_input_returns_empty() -> None:
|
||||
assert group_videos_by_variant([], merge=True) == []
|
||||
assert group_videos_by_variant([], merge=False) == []
|
||||
@@ -20,7 +20,7 @@ class _FakeReq:
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.get_event_loop().run_until_complete(coro) if False else asyncio.run(coro)
|
||||
return asyncio.run(coro)
|
||||
|
||||
|
||||
def test_skips_when_client_does_not_accept_gzip() -> None:
|
||||
|
||||
@@ -17,23 +17,32 @@ def _body(resp) -> dict:
|
||||
|
||||
|
||||
def test_api_error_default_http_status_per_code() -> None:
|
||||
# Every live APIErrorCode and the status a client observes for it. This is the
|
||||
# status half of the error contract PRD 05 must preserve through the collapse.
|
||||
cases = {
|
||||
APIErrorCode.INVALID_INPUT: 400,
|
||||
APIErrorCode.INVALID_SERVICE: 400,
|
||||
APIErrorCode.AUTH_REQUIRED: 401,
|
||||
APIErrorCode.INVALID_PROXY: 400,
|
||||
APIErrorCode.INVALID_PARAMETERS: 400,
|
||||
APIErrorCode.AUTH_FAILED: 401,
|
||||
APIErrorCode.FORBIDDEN: 403,
|
||||
APIErrorCode.GEOFENCE: 403,
|
||||
APIErrorCode.NOT_FOUND: 404,
|
||||
APIErrorCode.NO_CONTENT: 404,
|
||||
APIErrorCode.JOB_NOT_FOUND: 404,
|
||||
APIErrorCode.SESSION_NOT_FOUND: 404,
|
||||
APIErrorCode.TRACK_NOT_FOUND: 404,
|
||||
APIErrorCode.RATE_LIMITED: 429,
|
||||
APIErrorCode.INTERNAL_ERROR: 500,
|
||||
APIErrorCode.DOWNLOAD_ERROR: 500,
|
||||
APIErrorCode.WORKER_ERROR: 500,
|
||||
APIErrorCode.SERVICE_ERROR: 502,
|
||||
APIErrorCode.DRM_ERROR: 502,
|
||||
APIErrorCode.NETWORK_ERROR: 503,
|
||||
APIErrorCode.SERVICE_UNAVAILABLE: 503,
|
||||
}
|
||||
# Lock that the cases above are the *complete* live set — a new/removed code trips this.
|
||||
assert {c.name for c in cases} == {c.name for c in APIErrorCode}
|
||||
for code, expected in cases.items():
|
||||
assert APIError(code, "x").http_status == expected, code
|
||||
|
||||
@@ -125,3 +134,45 @@ def test_handle_api_exception_categorizes_generic() -> None:
|
||||
resp = handle_api_exception(ConnectionError("oops"))
|
||||
body = _body(resp)
|
||||
assert body["error_code"] == "NETWORK_ERROR"
|
||||
|
||||
|
||||
# --- Error contract harness (PRD 05 gate) ---------------------------------
|
||||
# Locks the FULL observable response — (http status, error_code, retryable) — that
|
||||
# a client sees for a generic (non-APIError) exception flowing through the
|
||||
# handle_api_exception funnel that every route's `except Exception` path calls.
|
||||
# This is the equivalence reference: after the categorize_exception collapse,
|
||||
# any tuple that changes here is a client-visible behaviour change, not a refactor.
|
||||
@pytest.mark.parametrize(
|
||||
"exc, status, code, retryable",
|
||||
[
|
||||
(Exception("Invalid credentials provided"), 401, "AUTH_FAILED", False),
|
||||
(Exception("Connection refused"), 503, "NETWORK_ERROR", True),
|
||||
(TimeoutError("read timeout"), 503, "NETWORK_ERROR", True),
|
||||
(Exception("Not available in your region"), 403, "GEOFENCE", False),
|
||||
(Exception("Title not found"), 404, "NOT_FOUND", False),
|
||||
(Exception("HTTP 429 too many requests"), 429, "RATE_LIMITED", True),
|
||||
(Exception("DRM license fetch failed"), 502, "DRM_ERROR", False),
|
||||
(Exception("503 service unavailable"), 503, "SERVICE_UNAVAILABLE", True),
|
||||
(ValueError("malformed body"), 400, "INVALID_INPUT", False),
|
||||
(RuntimeError("totally novel failure xyz"), 500, "INTERNAL_ERROR", False),
|
||||
],
|
||||
)
|
||||
def test_error_contract_generic_exception(exc: Exception, status: int, code: str, retryable: bool) -> None:
|
||||
resp = handle_api_exception(exc)
|
||||
assert resp.status == status
|
||||
body = _body(resp)
|
||||
assert body["error_code"] == code
|
||||
# retryable is only serialized when True (build_error_response omits it otherwise).
|
||||
assert body.get("retryable", False) is retryable
|
||||
|
||||
|
||||
def test_error_contract_apierror_passthrough_full_shape() -> None:
|
||||
# APIError carriers keep status + code + message + details + retryable verbatim.
|
||||
err = APIError(APIErrorCode.RATE_LIMITED, "slow down", details={"retry_after": 5}, retryable=True)
|
||||
resp = handle_api_exception(err)
|
||||
assert resp.status == 429
|
||||
body = _body(resp)
|
||||
assert body["error_code"] == "RATE_LIMITED"
|
||||
assert body["message"] == "slow down"
|
||||
assert body["details"] == {"retry_after": 5}
|
||||
assert body["retryable"] is True
|
||||
|
||||
@@ -199,6 +199,22 @@ def test_serialize_drm_widevine_minimal() -> None:
|
||||
assert info["license_url"] == "https://lic.example.com/wv"
|
||||
|
||||
|
||||
def test_serialize_drm_playready_pssh_without_dumps_is_omitted() -> None:
|
||||
# pyplayready's PSSH exposes no dumps()/to_base64()/__bytes__; serialization
|
||||
# must omit the pssh field rather than emit an object repr.
|
||||
class _PSSH:
|
||||
pass
|
||||
|
||||
class _PlayReady:
|
||||
def __init__(self) -> None:
|
||||
self._pssh = _PSSH()
|
||||
self.kids = ["00112233445566778899aabbccddeeff"]
|
||||
|
||||
info = serialize_drm(_PlayReady())[0]
|
||||
assert "pssh" not in info
|
||||
assert info["kids"] == ["00112233445566778899aabbccddeeff"]
|
||||
|
||||
|
||||
# ---------- validate_service ----------
|
||||
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ import time
|
||||
|
||||
import pytest
|
||||
|
||||
from unshackle.core.api.input_bridge import AuthStatus, BridgeCancelledError, InputBridge
|
||||
from unshackle.core.api.input_bridge import AuthStatus, InputBridge
|
||||
|
||||
pytestmark = pytest.mark.unit
|
||||
|
||||
@@ -16,7 +16,6 @@ def test_initial_status_is_authenticating() -> None:
|
||||
bridge = InputBridge()
|
||||
assert bridge.status is AuthStatus.AUTHENTICATING
|
||||
assert bridge.get_pending_prompt() is None
|
||||
assert bridge.error is None
|
||||
|
||||
|
||||
def test_submit_response_returns_false_when_no_prompt_pending() -> None:
|
||||
@@ -49,16 +48,15 @@ def test_request_input_blocks_until_submit() -> None:
|
||||
|
||||
def test_request_input_times_out() -> None:
|
||||
bridge = InputBridge()
|
||||
with pytest.raises(TimeoutError):
|
||||
with pytest.raises(TimeoutError, match="No client response"):
|
||||
bridge.request_input("hello", timeout=0.6)
|
||||
assert bridge.status is AuthStatus.FAILED
|
||||
assert "timed out" in (bridge.error or "")
|
||||
|
||||
|
||||
def test_cancel_before_request_raises_immediately() -> None:
|
||||
bridge = InputBridge()
|
||||
bridge.cancel()
|
||||
with pytest.raises(BridgeCancelledError):
|
||||
with pytest.raises(RuntimeError, match="cancelled"):
|
||||
bridge.request_input("hello", timeout=5)
|
||||
assert bridge.status is AuthStatus.FAILED
|
||||
|
||||
@@ -70,7 +68,7 @@ def test_cancel_unblocks_pending_request() -> None:
|
||||
def worker() -> None:
|
||||
try:
|
||||
bridge.request_input("OTP?", timeout=5)
|
||||
except BridgeCancelledError as e:
|
||||
except RuntimeError as e:
|
||||
exc.append(e)
|
||||
|
||||
t = threading.Thread(target=worker)
|
||||
@@ -82,7 +80,7 @@ def test_cancel_unblocks_pending_request() -> None:
|
||||
|
||||
bridge.cancel()
|
||||
t.join(timeout=2)
|
||||
assert exc and isinstance(exc[0], BridgeCancelledError)
|
||||
assert exc and isinstance(exc[0], RuntimeError)
|
||||
assert bridge.status is AuthStatus.FAILED
|
||||
|
||||
|
||||
@@ -92,9 +90,7 @@ def test_get_pending_prompt_returns_none_outside_pending_state() -> None:
|
||||
assert bridge.get_pending_prompt() is None
|
||||
|
||||
|
||||
def test_status_and_error_setters() -> None:
|
||||
def test_status_setter() -> None:
|
||||
bridge = InputBridge()
|
||||
bridge.status = AuthStatus.AUTHENTICATED
|
||||
bridge.error = "boom"
|
||||
assert bridge.status is AuthStatus.AUTHENTICATED
|
||||
assert bridge.error == "boom"
|
||||
|
||||
@@ -11,7 +11,8 @@ import pytest
|
||||
from aiohttp import web
|
||||
|
||||
from unshackle.core.api.compression import compression_middleware
|
||||
from unshackle.core.api.routes import cors_middleware, setup_routes
|
||||
from unshackle.core.api.errors import APIError, APIErrorCode
|
||||
from unshackle.core.api.routes import api_handler, cors_middleware, setup_routes
|
||||
|
||||
pytestmark = pytest.mark.unit
|
||||
|
||||
@@ -88,10 +89,44 @@ def test_setup_routes_remote_only_excludes_list_and_download(make_app) -> None:
|
||||
assert ("POST", "/api/list-tracks") not in paths
|
||||
assert ("POST", "/api/download") not in paths
|
||||
assert ("GET", "/api/download/jobs") not in paths
|
||||
# session endpoints still present
|
||||
assert ("POST", "/api/session/create") in paths
|
||||
assert ("GET", "/api/session/{session_id}/titles") in paths
|
||||
assert ("POST", "/api/session/{session_id}/license") in paths
|
||||
# The full remote subset is still registered after the single-table collapse.
|
||||
remote_subset = {
|
||||
("GET", "/api/health"),
|
||||
("GET", "/api/services"),
|
||||
("POST", "/api/search"),
|
||||
("POST", "/api/session/create"),
|
||||
("GET", "/api/session/{session_id}/titles"),
|
||||
("POST", "/api/session/{session_id}/tracks"),
|
||||
("POST", "/api/session/{session_id}/segments"),
|
||||
("POST", "/api/session/{session_id}/license"),
|
||||
("GET", "/api/session/{session_id}/prompt"),
|
||||
("POST", "/api/session/{session_id}/prompt"),
|
||||
("GET", "/api/session/{session_id}"),
|
||||
("DELETE", "/api/session/{session_id}"),
|
||||
}
|
||||
assert remote_subset.issubset(paths)
|
||||
|
||||
|
||||
async def test_api_handler_maps_apierror(aiohttp_client) -> None:
|
||||
"""The error wrapper turns a raised APIError into the same structured response
|
||||
the per-handler except blocks produced (status + code from APIError)."""
|
||||
|
||||
@api_handler
|
||||
async def boom(request: web.Request) -> web.Response:
|
||||
raise APIError(APIErrorCode.NOT_FOUND, "nope", details={"x": 1})
|
||||
|
||||
app = web.Application()
|
||||
app["debug_api"] = False
|
||||
app.router.add_get("/boom", boom)
|
||||
client = await aiohttp_client(app)
|
||||
|
||||
resp = await client.get("/boom")
|
||||
assert resp.status == 404 # NOT_FOUND -> 404 per errors.APIError._default_http_status
|
||||
body = await resp.json()
|
||||
assert body["status"] == "error"
|
||||
assert body["error_code"] == "NOT_FOUND"
|
||||
assert body["message"] == "nope"
|
||||
assert body["details"] == {"x": 1}
|
||||
|
||||
|
||||
async def test_cors_preflight_returns_headers(make_app, aiohttp_client) -> None:
|
||||
|
||||
@@ -85,6 +85,21 @@ async def test_cleanup_expired_keeps_pending_input_under_grace(store: SessionSto
|
||||
assert store.session_count == 1
|
||||
|
||||
|
||||
async def test_cleanup_expired_drops_pending_input_past_grace(store: SessionStore) -> None:
|
||||
"""The auth-state grace is the single AUTH_INPUT_TIMEOUT rule, not a separate magic number."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from unshackle.core.api.input_bridge import AUTH_INPUT_TIMEOUT
|
||||
|
||||
entry = await store.create("ATV", _FakeService())
|
||||
entry.input_bridge = InputBridge()
|
||||
entry.auth_status = AuthStatus.PENDING_INPUT
|
||||
entry.last_accessed = datetime.now(timezone.utc) - timedelta(seconds=AUTH_INPUT_TIMEOUT + 1)
|
||||
removed = await store.cleanup_expired()
|
||||
assert removed == 1
|
||||
assert store.session_count == 0
|
||||
|
||||
|
||||
async def test_cancel_all_bridges(store: SessionStore) -> None:
|
||||
a = await store.create("ATV", _FakeService())
|
||||
b = await store.create("NF", _FakeService())
|
||||
|
||||
+82
-22
@@ -58,7 +58,7 @@ from unshackle.core.music import (
|
||||
write_music_manifest,
|
||||
write_music_metadata,
|
||||
)
|
||||
from unshackle.core.proxies import Basic, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
|
||||
from unshackle.core.proxies import Basic, ExpressVPN, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
|
||||
from unshackle.core.service import Service
|
||||
from unshackle.core.services import Services
|
||||
from unshackle.core.title_cacher import get_account_hash
|
||||
@@ -100,6 +100,22 @@ def normalize_dl_config(dl_config: dict[str, Any]) -> dict[str, Any]:
|
||||
return {DL_OPTION_ALIASES.get(key, key): value for key, value in dl_config.items()}
|
||||
|
||||
|
||||
def group_videos_by_variant(videos: list[Video], *, merge: bool) -> list[list[Video]]:
|
||||
"""Group video tracks for muxing.
|
||||
|
||||
When ``merge`` is True, tracks sharing ``(height, range, codec)`` are grouped into one
|
||||
file so only language varies within a group; different resolutions, ranges and codecs
|
||||
stay in separate groups (separate files). When False, each track is its own group
|
||||
(one file per track, the default behaviour). Group order follows first-seen track order.
|
||||
"""
|
||||
if not merge:
|
||||
return [[video] for video in videos]
|
||||
groups: dict[tuple[Any, ...], list[Video]] = {}
|
||||
for video in videos:
|
||||
groups.setdefault((video.height, video.range, video.codec), []).append(video)
|
||||
return list(groups.values())
|
||||
|
||||
|
||||
def apply_service_dl_overrides(ctx: click.Context, service_dl_config: dict[str, Any], log: logging.Logger) -> None:
|
||||
"""Apply ``services.<TAG>.dl`` config onto ``ctx.params``. Explicit CLI/env values win;
|
||||
defaults and global ``dl:`` default_map values are replaced."""
|
||||
@@ -459,6 +475,13 @@ class dl:
|
||||
default=None,
|
||||
help="Create separate output files per audio codec instead of merging all audio.",
|
||||
)
|
||||
@click.option(
|
||||
"--merge-video",
|
||||
"merge_video",
|
||||
is_flag=True,
|
||||
default=None,
|
||||
help="Mux all selected video tracks into a single file instead of one file per track.",
|
||||
)
|
||||
@click.option(
|
||||
"--select-titles",
|
||||
is_flag=True,
|
||||
@@ -1027,6 +1050,8 @@ class dl:
|
||||
with console.status("Loading Proxy Providers...", spinner="dots"):
|
||||
if config.proxy_providers.get("basic"):
|
||||
self.proxy_providers.append(Basic(**config.proxy_providers["basic"]))
|
||||
if config.proxy_providers.get("expressvpn"):
|
||||
self.proxy_providers.append(ExpressVPN(**config.proxy_providers["expressvpn"]))
|
||||
if config.proxy_providers.get("nordvpn"):
|
||||
self.proxy_providers.append(NordVPN(**config.proxy_providers["nordvpn"]))
|
||||
if config.proxy_providers.get("surfsharkvpn"):
|
||||
@@ -1046,8 +1071,8 @@ class dl:
|
||||
# requesting proxy from a specific proxy provider
|
||||
requested_provider, proxy = proxy.split(":", maxsplit=1)
|
||||
# Match simple region codes (us, ca, uk1) or provider:region format (nordvpn:ca, windscribe:us)
|
||||
if re.match(r"^[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
|
||||
r"^[a-z]+:[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE
|
||||
if re.match(r"^[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
|
||||
r"^[a-z]+:[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE
|
||||
):
|
||||
proxy = proxy.lower()
|
||||
# Preserve the original user query (region code) for service-specific proxy_map overrides.
|
||||
@@ -1082,7 +1107,13 @@ class dl:
|
||||
else:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
else:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
display = None
|
||||
if hasattr(proxy_provider, "last_connection_display"):
|
||||
display = proxy_provider.last_connection_display()
|
||||
if display:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
|
||||
else:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
else:
|
||||
for proxy_provider in self.proxy_providers:
|
||||
proxy_uri = proxy_provider.get_proxy(proxy)
|
||||
@@ -1098,7 +1129,13 @@ class dl:
|
||||
else:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
else:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
display = None
|
||||
if hasattr(proxy_provider, "last_connection_display"):
|
||||
display = proxy_provider.last_connection_display()
|
||||
if display:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
|
||||
else:
|
||||
self.log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
break
|
||||
# Store proxy query info for service-specific overrides
|
||||
ctx.params["proxy_query"] = proxy_query
|
||||
@@ -1173,6 +1210,7 @@ class dl:
|
||||
worst: bool,
|
||||
best_available: bool,
|
||||
split_audio: Optional[bool] = None,
|
||||
merge_video: Optional[bool] = None,
|
||||
real_video_bitrate: bool = False,
|
||||
real_audio_bitrate: bool = False,
|
||||
progress_sink: Optional[Callable[[dict[str, Any]], None]] = None,
|
||||
@@ -3062,6 +3100,9 @@ class dl:
|
||||
# When we split audio (merge_audio=False), multiple outputs may exist per title, so suffix codec.
|
||||
append_audio_codec_suffix = not merge_audio
|
||||
|
||||
# Mux all selected video tracks into one file instead of one file per track.
|
||||
merge_video = merge_video if merge_video is not None else config.muxing.get("merge_video", False)
|
||||
|
||||
multiplex_tasks: list[tuple[TaskID, Tracks, Optional[Audio.Codec]]] = []
|
||||
# Track hybrid-processing outputs explicitly so we can always clean them up,
|
||||
# even if muxing fails early (e.g. SystemExit) before the normal delete loop.
|
||||
@@ -3097,22 +3138,25 @@ class dl:
|
||||
task_tracks = clone_tracks_for_audio(base_tracks, codec_audio_tracks)
|
||||
multiplex_tasks.append((task_id, task_tracks, audio_codec))
|
||||
|
||||
def mux_video_standalone(video_track: Optional[Video]) -> None:
|
||||
if video_track and video_track.dv_compatible_bitstream:
|
||||
apply_dv_fixup(video_track)
|
||||
def mux_video_group(video_tracks: list[Optional[Video]]) -> None:
|
||||
for video_track in video_tracks:
|
||||
if video_track and video_track.dv_compatible_bitstream:
|
||||
apply_dv_fixup(video_track)
|
||||
|
||||
task_description = "Multiplexing"
|
||||
if video_track:
|
||||
# All tracks in a merged group share height/range/codec, so describe from the first.
|
||||
head = next((v for v in video_tracks if v), None)
|
||||
if head:
|
||||
if len(quality) > 1:
|
||||
task_description += f" {video_track.height}p"
|
||||
task_description += f" {head.height}p"
|
||||
if len(range_) > 1:
|
||||
task_description += f" {video_track.range.name}"
|
||||
task_description += f" {head.range.name}"
|
||||
if len(vcodec) > 1:
|
||||
task_description += f" {video_track.codec.name}"
|
||||
task_description += f" {head.codec.name}"
|
||||
|
||||
task_tracks = Tracks(title.tracks) + title.tracks.chapters + title.tracks.attachments
|
||||
if video_track:
|
||||
task_tracks.videos = [video_track]
|
||||
if head:
|
||||
task_tracks.videos = [v for v in video_tracks if v]
|
||||
|
||||
enqueue_mux_tasks(task_description, task_tracks)
|
||||
|
||||
@@ -3172,16 +3216,18 @@ class dl:
|
||||
enqueue_mux_tasks(task_description, task_tracks)
|
||||
|
||||
# Mux every requested range standalone, skipping the ingredient-only DV.
|
||||
for video_track in original_videos:
|
||||
if video_track.hybrid_base_only:
|
||||
continue
|
||||
mux_video_standalone(video_track)
|
||||
# merge_video collapses only language variants (same height/range/codec).
|
||||
standalone_videos = [v for v in original_videos if not v.hybrid_base_only]
|
||||
for group in group_videos_by_variant(standalone_videos, merge=merge_video):
|
||||
mux_video_group(group)
|
||||
|
||||
console.print()
|
||||
else:
|
||||
# Normal mode: process each video track separately
|
||||
for video_track in title.tracks.videos or [None]:
|
||||
mux_video_standalone(video_track)
|
||||
# Normal mode: one file per video track, unless merge_video groups
|
||||
# same-(height, range, codec) language variants into one file.
|
||||
groups = group_videos_by_variant(title.tracks.videos, merge=merge_video)
|
||||
for group in groups or [[None]]:
|
||||
mux_video_group(group)
|
||||
|
||||
if progress_sink:
|
||||
progress_sink(
|
||||
@@ -4111,8 +4157,20 @@ class dl:
|
||||
return default_cookie_file
|
||||
|
||||
@staticmethod
|
||||
def get_cookie_jar(service: str, profile: Optional[str]) -> Optional[MozillaCookieJar]:
|
||||
def get_cookie_jar(service: str, profile: Optional[str]) -> Optional[CookieJar]:
|
||||
"""Get Service Cookies for Profile."""
|
||||
# Check if automatic Firefox cookie extraction is configured for this service
|
||||
ff_settings = getattr(config, "firefox_cookies", {}).get(service)
|
||||
if ff_settings:
|
||||
try:
|
||||
from unshackle.core.utils.firefox_cookie_extractor import get_firefox_cookies
|
||||
extracted_jar = get_firefox_cookies(ff_settings)
|
||||
if extracted_jar:
|
||||
return extracted_jar
|
||||
except Exception:
|
||||
# Fallback to file-based if Firefox extraction fails
|
||||
pass
|
||||
|
||||
cookie_file = dl.get_cookie_path(service, profile)
|
||||
if cookie_file:
|
||||
cookie_jar = MozillaCookieJar(cookie_file)
|
||||
@@ -4129,6 +4187,8 @@ class dl:
|
||||
cookie_file.write_text(cookie_data, "utf8")
|
||||
cookie_jar.load(ignore_discard=True, ignore_expires=True)
|
||||
return cookie_jar
|
||||
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def save_cookies(path: Path, cookies: CookieJar):
|
||||
|
||||
@@ -16,7 +16,7 @@ from unshackle.core import binaries
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.console import console
|
||||
from unshackle.core.constants import context_settings
|
||||
from unshackle.core.proxies import Basic, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
|
||||
from unshackle.core.proxies import Basic, ExpressVPN, Gluetun, Hola, NordVPN, SurfsharkVPN, WindscribeVPN
|
||||
from unshackle.core.service import Service
|
||||
from unshackle.core.services import Services
|
||||
from unshackle.core.utils.click_types import ContextData
|
||||
@@ -67,6 +67,8 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
|
||||
with console.status("Loading Proxy Providers...", spinner="dots"):
|
||||
if config.proxy_providers.get("basic"):
|
||||
proxy_providers.append(Basic(**config.proxy_providers["basic"]))
|
||||
if config.proxy_providers.get("expressvpn"):
|
||||
proxy_providers.append(ExpressVPN(**config.proxy_providers["expressvpn"]))
|
||||
if config.proxy_providers.get("nordvpn"):
|
||||
proxy_providers.append(NordVPN(**config.proxy_providers["nordvpn"]))
|
||||
if config.proxy_providers.get("surfsharkvpn"):
|
||||
@@ -86,8 +88,8 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
|
||||
# requesting proxy from a specific proxy provider
|
||||
requested_provider, proxy = proxy.split(":", maxsplit=1)
|
||||
# Match simple region codes (us, ca, uk1) or provider:region format (nordvpn:ca, windscribe:us)
|
||||
if re.match(r"^[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
|
||||
r"^[a-z]+:[a-z]{2}(?:\d+)?$", proxy, re.IGNORECASE
|
||||
if re.match(r"^[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE) or re.match(
|
||||
r"^[a-z]+:[a-z]{2}(?:[-][a-z0-9]+)*(?:\d+)?$", proxy, re.IGNORECASE
|
||||
):
|
||||
proxy = proxy.lower()
|
||||
with console.status(f"Getting a Proxy to {proxy}...", spinner="dots"):
|
||||
@@ -103,13 +105,25 @@ def search(ctx: click.Context, no_proxy: bool, profile: Optional[str] = None, pr
|
||||
log.error(f"The proxy provider {requested_provider} had no proxy for {proxy}")
|
||||
sys.exit(1)
|
||||
proxy = ctx.params["proxy"] = proxy_uri
|
||||
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
display = None
|
||||
if hasattr(proxy_provider, "last_connection_display"):
|
||||
display = proxy_provider.last_connection_display()
|
||||
if display:
|
||||
log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
|
||||
else:
|
||||
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
else:
|
||||
for proxy_provider in proxy_providers:
|
||||
proxy_uri = proxy_provider.get_proxy(proxy)
|
||||
if proxy_uri:
|
||||
proxy = ctx.params["proxy"] = proxy_uri
|
||||
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
display = None
|
||||
if hasattr(proxy_provider, "last_connection_display"):
|
||||
display = proxy_provider.last_connection_display()
|
||||
if display:
|
||||
log.info(f"Using {proxy_provider.__class__.__name__} Proxy {display}")
|
||||
else:
|
||||
log.info(f"Using {proxy_provider.__class__.__name__} Proxy: {proxy}")
|
||||
break
|
||||
else:
|
||||
log.info(f"Using explicit Proxy: {proxy}")
|
||||
|
||||
@@ -104,6 +104,10 @@ class DownloadJob:
|
||||
completed_tracks: int = 0
|
||||
total_tracks: int = 0
|
||||
active_tracks: List[str] = field(default_factory=list)
|
||||
# Segment counts and transfer speed of the track downloading now (granular display)
|
||||
segments_done: float = 0.0
|
||||
segments_total: float = 0.0
|
||||
speed: Optional[str] = None
|
||||
|
||||
# Subtitles skipped under skip_subtitle_errors (non-fatal). Each entry is a dl.SkippedSubtitle
|
||||
# dict (id / language / title) so a client can report which weren't available.
|
||||
@@ -125,6 +129,9 @@ class DownloadJob:
|
||||
"completed_tracks": self.completed_tracks,
|
||||
"total_tracks": self.total_tracks,
|
||||
"active_tracks": self.active_tracks,
|
||||
"segments_done": self.segments_done,
|
||||
"segments_total": self.segments_total,
|
||||
"speed": self.speed,
|
||||
"skipped_subtitles": self.skipped_subtitles,
|
||||
}
|
||||
|
||||
@@ -148,25 +155,25 @@ class DownloadJob:
|
||||
return result
|
||||
|
||||
|
||||
def to_enum(values: List[str], enum_cls: type[Enum]) -> List[Enum]:
|
||||
"""Map case-insensitive strings (by member name or value) to enum members, dropping unknowns."""
|
||||
lookup = {m.name.upper(): m for m in enum_cls}
|
||||
lookup.update({str(m.value).upper(): m for m in enum_cls})
|
||||
return [lookup[v.upper()] for v in values if v.upper() in lookup]
|
||||
|
||||
|
||||
def _perform_download(
|
||||
job_id: str,
|
||||
service: str,
|
||||
title_id: str,
|
||||
params: Dict[str, Any],
|
||||
cancel_event: Optional[threading.Event] = None,
|
||||
progress_callback: Optional[Callable[[Dict[str, Any]], None]] = None,
|
||||
) -> List[str]:
|
||||
"""Execute the synchronous download logic for a job."""
|
||||
|
||||
def _check_cancel(stage: str):
|
||||
if cancel_event and cancel_event.is_set():
|
||||
raise Exception(f"Job was cancelled {stage}")
|
||||
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
from io import StringIO
|
||||
|
||||
_check_cancel("before execution started")
|
||||
|
||||
# Import dl.py components lazily to avoid circular deps during module import
|
||||
import click
|
||||
import yaml
|
||||
@@ -197,9 +204,7 @@ def _perform_download(
|
||||
if isinstance(vcodec_raw, str):
|
||||
vcodec_raw = [vcodec_raw]
|
||||
if isinstance(vcodec_raw, list) and vcodec_raw and not isinstance(vcodec_raw[0], Video.Codec):
|
||||
codec_map = {c.name.upper(): c for c in Video.Codec}
|
||||
codec_map.update({c.value.upper(): c for c in Video.Codec})
|
||||
params["vcodec"] = [codec_map[v.upper()] for v in vcodec_raw if v.upper() in codec_map]
|
||||
params["vcodec"] = to_enum(vcodec_raw, Video.Codec)
|
||||
else:
|
||||
params["vcodec"] = []
|
||||
|
||||
@@ -208,17 +213,14 @@ def _perform_download(
|
||||
if isinstance(range_raw, str):
|
||||
range_raw = [range_raw]
|
||||
if isinstance(range_raw, list) and range_raw and not isinstance(range_raw[0], Video.Range):
|
||||
range_map = {r.name.upper(): r for r in Video.Range}
|
||||
range_map.update({r.value.upper(): r for r in Video.Range})
|
||||
params["range"] = [range_map[r.upper()] for r in range_raw if r.upper() in range_map]
|
||||
params["range"] = to_enum(range_raw, Video.Range)
|
||||
else:
|
||||
params["range"] = [Video.Range.SDR]
|
||||
|
||||
sub_format_raw = params.get("sub_format")
|
||||
if sub_format_raw and isinstance(sub_format_raw, str):
|
||||
sub_map = {c.name.upper(): c for c in Subtitle.Codec}
|
||||
sub_map.update({c.value.upper(): c for c in Subtitle.Codec})
|
||||
params["sub_format"] = sub_map.get(sub_format_raw.upper())
|
||||
mapped = to_enum([sub_format_raw], Subtitle.Codec)
|
||||
params["sub_format"] = mapped[0] if mapped else None
|
||||
|
||||
if params.get("export"):
|
||||
params["export"] = bool(params["export"])
|
||||
@@ -321,8 +323,6 @@ def _perform_download(
|
||||
|
||||
service_module = Services.load(service)
|
||||
|
||||
_check_cancel("before service instantiation")
|
||||
|
||||
try:
|
||||
import inspect
|
||||
|
||||
@@ -360,29 +360,11 @@ def _perform_download(
|
||||
|
||||
original_download_dir = config.directories.downloads
|
||||
|
||||
_check_cancel("before download execution")
|
||||
|
||||
stdout_capture = StringIO()
|
||||
stderr_capture = StringIO()
|
||||
|
||||
# The progress_sink (dl.build_job_progress_callables) owns the percentage; status changes
|
||||
# are emitted here.
|
||||
if progress_callback:
|
||||
progress_callback({"progress": 0.0, "status": "starting"})
|
||||
original_result = dl_instance.result
|
||||
|
||||
def result_with_progress(*args, **kwargs):
|
||||
try:
|
||||
progress_callback({"status": "downloading"})
|
||||
result = original_result(*args, **kwargs)
|
||||
progress_callback({"progress": 100.0, "status": "completed"})
|
||||
return result
|
||||
except Exception as e:
|
||||
progress_callback({"progress": 0.0, "status": "failed", "error": str(e)})
|
||||
raise
|
||||
|
||||
dl_instance.result = result_with_progress
|
||||
|
||||
# progress_sink (dl.build_job_progress_callables) owns percentage; terminal status
|
||||
# (success/failure) is reported by the worker's result file.
|
||||
try:
|
||||
with redirect_stdout(stdout_capture), redirect_stderr(stderr_capture):
|
||||
dl_instance.result(
|
||||
@@ -756,6 +738,14 @@ class DownloadQueueManager:
|
||||
job.completed_tracks = int(progress_data["completed_tracks"])
|
||||
if "active_tracks" in progress_data:
|
||||
job.active_tracks = list(progress_data["active_tracks"])
|
||||
for _sk in ("segments_done", "segments_total"):
|
||||
if _sk in progress_data:
|
||||
try:
|
||||
setattr(job, _sk, float(progress_data[_sk]))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
if progress_data.get("speed"):
|
||||
job.speed = str(progress_data["speed"])
|
||||
if progress_data.get("skipped_subtitles"):
|
||||
job.skipped_subtitles = progress_data["skipped_subtitles"]
|
||||
if "progress" in progress_data:
|
||||
|
||||
@@ -60,7 +60,7 @@ def main(argv: list[str]) -> int:
|
||||
log.error(f"Failed to write progress update: {e}")
|
||||
|
||||
output_files = _perform_download(
|
||||
job_id, service, title_id, params, cancel_event=None, progress_callback=progress_callback
|
||||
job_id, service, title_id, params, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
result = {"status": "success", "output_files": output_files}
|
||||
|
||||
@@ -21,14 +21,10 @@ class APIErrorCode(str, Enum):
|
||||
# Client errors (4xx)
|
||||
INVALID_INPUT = "INVALID_INPUT" # Missing or malformed request data
|
||||
INVALID_SERVICE = "INVALID_SERVICE" # Unknown service name
|
||||
INVALID_TITLE_ID = "INVALID_TITLE_ID" # Invalid or malformed title ID
|
||||
INVALID_PROFILE = "INVALID_PROFILE" # Profile doesn't exist
|
||||
INVALID_PROXY = "INVALID_PROXY" # Invalid proxy specification
|
||||
INVALID_LANGUAGE = "INVALID_LANGUAGE" # Invalid language code
|
||||
INVALID_PARAMETERS = "INVALID_PARAMETERS" # Invalid download parameters
|
||||
|
||||
AUTH_FAILED = "AUTH_FAILED" # Authentication failure (invalid credentials/cookies)
|
||||
AUTH_REQUIRED = "AUTH_REQUIRED" # Missing authentication
|
||||
FORBIDDEN = "FORBIDDEN" # Action not allowed
|
||||
GEOFENCE = "GEOFENCE" # Content not available in region
|
||||
|
||||
@@ -84,13 +80,9 @@ class APIError(Exception):
|
||||
# 400 Bad Request
|
||||
APIErrorCode.INVALID_INPUT: 400,
|
||||
APIErrorCode.INVALID_SERVICE: 400,
|
||||
APIErrorCode.INVALID_TITLE_ID: 400,
|
||||
APIErrorCode.INVALID_PROFILE: 400,
|
||||
APIErrorCode.INVALID_PROXY: 400,
|
||||
APIErrorCode.INVALID_LANGUAGE: 400,
|
||||
APIErrorCode.INVALID_PARAMETERS: 400,
|
||||
# 401 Unauthorized
|
||||
APIErrorCode.AUTH_REQUIRED: 401,
|
||||
APIErrorCode.AUTH_FAILED: 401,
|
||||
# 403 Forbidden
|
||||
APIErrorCode.FORBIDDEN: 403,
|
||||
|
||||
+143
-257
@@ -1,6 +1,7 @@
|
||||
import asyncio
|
||||
import enum
|
||||
import logging
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from aiohttp import web
|
||||
@@ -200,6 +201,44 @@ def instantiate_service(
|
||||
return parent_ctx.invoke(service_module.cli, title=title, **extras)
|
||||
|
||||
|
||||
def setup_list_service(data: Dict[str, Any], normalized_service: str, profile: Optional[str], title_id: str) -> Any:
|
||||
"""Build and authenticate a service instance for list_titles / list_tracks.
|
||||
|
||||
Runs the shared preamble: load yaml → resolve proxy → load CDM → build ctx →
|
||||
instantiate → authenticate. Raises APIError on proxy failure.
|
||||
"""
|
||||
from unshackle.commands.dl import dl
|
||||
|
||||
service_config = load_service_yaml(normalized_service)
|
||||
|
||||
proxy_param = data.get("proxy")
|
||||
no_proxy = data.get("no_proxy", False)
|
||||
proxy_providers = []
|
||||
|
||||
if not no_proxy:
|
||||
proxy_providers = initialize_proxy_providers()
|
||||
|
||||
if proxy_param and not no_proxy:
|
||||
try:
|
||||
proxy_param = resolve_proxy(proxy_param, proxy_providers)
|
||||
except ValueError as e:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_PROXY,
|
||||
f"Proxy error: {e}",
|
||||
details={"proxy": proxy_param, "service": normalized_service},
|
||||
)
|
||||
|
||||
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
|
||||
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
|
||||
service_module = Services.load(normalized_service)
|
||||
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
|
||||
|
||||
cookies = dl.get_cookie_jar(normalized_service, profile)
|
||||
credential = dl.get_credentials(normalized_service, profile)
|
||||
service_instance.authenticate(cookies, credential)
|
||||
return service_instance
|
||||
|
||||
|
||||
def get_allowed_services(request: Optional[web.Request] = None) -> Optional[List[str]]:
|
||||
"""Get effective service allowlist considering global + per-key config.
|
||||
|
||||
@@ -247,37 +286,50 @@ def validate_service(service_tag: str, request: Optional[web.Request] = None) ->
|
||||
return None
|
||||
|
||||
|
||||
def require_fields(data: Dict[str, Any], *names: str) -> None:
|
||||
"""Raise INVALID_INPUT for the first missing/falsy required field."""
|
||||
for name in names:
|
||||
if not data.get(name):
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
f"Missing required parameter: {name}",
|
||||
details={"missing_parameter": name},
|
||||
)
|
||||
|
||||
|
||||
def serialize_title(title: Title_T) -> Dict[str, Any]:
|
||||
"""Convert a title object to JSON-serializable dict."""
|
||||
title_language = str(title.language) if hasattr(title, "language") and title.language else None
|
||||
# Optional display metadata a service may provide: a synopsis (title.description) and a
|
||||
# release/air date or poster image URL stashed in title.data. Surfaced so a client can show
|
||||
# a richer listing without re-fetching the page.
|
||||
description = getattr(title, "description", None) or None
|
||||
_data = getattr(title, "data", None)
|
||||
date = _data.get("date") if isinstance(_data, dict) else None
|
||||
cover_url = _data.get("cover_url") if isinstance(_data, dict) else None
|
||||
|
||||
if isinstance(title, Episode):
|
||||
episode_name = title.name if title.name else f"Episode {title.number:02d}"
|
||||
result = {
|
||||
"type": "episode",
|
||||
"name": episode_name,
|
||||
"series_title": str(title.title),
|
||||
"season": title.season,
|
||||
"number": title.number,
|
||||
"year": title.year,
|
||||
"id": str(title.id) if hasattr(title, "id") else None,
|
||||
"language": title_language,
|
||||
}
|
||||
elif isinstance(title, Movie):
|
||||
result = {
|
||||
"type": "movie",
|
||||
"name": str(title.name) if hasattr(title, "name") else str(title),
|
||||
"year": title.year,
|
||||
"id": str(title.id) if hasattr(title, "id") else None,
|
||||
"language": title_language,
|
||||
}
|
||||
is_episode = isinstance(title, Episode)
|
||||
if is_episode:
|
||||
name = title.name if title.name else f"Episode {title.number:02d}"
|
||||
else:
|
||||
result = {
|
||||
"type": "other",
|
||||
"name": str(title.name) if hasattr(title, "name") else str(title),
|
||||
"id": str(title.id) if hasattr(title, "id") else None,
|
||||
"language": title_language,
|
||||
}
|
||||
name = str(title.name) if hasattr(title, "name") else str(title)
|
||||
|
||||
result = {
|
||||
"type": "episode" if is_episode else "movie" if isinstance(title, Movie) else "other",
|
||||
"name": name,
|
||||
"id": str(title.id) if hasattr(title, "id") else None,
|
||||
"language": title_language,
|
||||
"description": description,
|
||||
"date": date,
|
||||
"cover_url": cover_url,
|
||||
}
|
||||
# "other" titles carry no year; only Episode/Movie do.
|
||||
if isinstance(title, (Episode, Movie)):
|
||||
result["year"] = title.year
|
||||
if is_episode:
|
||||
result["series_title"] = str(title.title)
|
||||
result["season"] = title.season
|
||||
result["number"] = title.number
|
||||
|
||||
return result
|
||||
|
||||
@@ -346,46 +398,19 @@ def serialize_drm(drm_list) -> Optional[List[Dict[str, Any]]]:
|
||||
drm_class = drm.__class__.__name__
|
||||
drm_info["type"] = drm_class.lower()
|
||||
|
||||
# Get PSSH - handle both Widevine and PlayReady
|
||||
if hasattr(drm, "_pssh") and drm._pssh:
|
||||
pssh_obj = None
|
||||
# PSSH: pywidevine exposes dumps(); pyplayready's PSSH has no base64 method
|
||||
# here, so PlayReady omits the field (unchanged from prior behaviour).
|
||||
pssh_obj = getattr(drm, "_pssh", None)
|
||||
if pssh_obj is not None and hasattr(pssh_obj, "dumps"):
|
||||
try:
|
||||
pssh_obj = drm._pssh
|
||||
# Try to get base64 representation
|
||||
if hasattr(pssh_obj, "dumps"):
|
||||
# pywidevine PSSH has dumps() method
|
||||
drm_info["pssh"] = pssh_obj.dumps()
|
||||
elif hasattr(pssh_obj, "__bytes__"):
|
||||
# Convert to base64
|
||||
import base64
|
||||
|
||||
drm_info["pssh"] = base64.b64encode(bytes(pssh_obj)).decode()
|
||||
elif hasattr(pssh_obj, "to_base64"):
|
||||
drm_info["pssh"] = pssh_obj.to_base64()
|
||||
else:
|
||||
# Fallback - str() works for pywidevine PSSH
|
||||
pssh_str = str(pssh_obj)
|
||||
# Check if it's already base64-like or an object repr
|
||||
if not pssh_str.startswith("<"):
|
||||
drm_info["pssh"] = pssh_str
|
||||
drm_info["pssh"] = pssh_obj.dumps()
|
||||
except (ValueError, TypeError, KeyError):
|
||||
# Some PSSH implementations can fail to parse/serialize; log and continue.
|
||||
pssh_type = type(pssh_obj).__name__ if pssh_obj is not None else None
|
||||
log.warning(
|
||||
"Failed to extract/serialize PSSH for DRM type=%s pssh_type=%s",
|
||||
"Failed to serialize PSSH for DRM type=%s pssh_type=%s",
|
||||
drm_class,
|
||||
pssh_type,
|
||||
type(pssh_obj).__name__,
|
||||
exc_info=True,
|
||||
)
|
||||
except Exception:
|
||||
# Don't silently swallow unexpected failures; make them visible and propagate.
|
||||
pssh_type = type(pssh_obj).__name__ if pssh_obj is not None else None
|
||||
log.exception(
|
||||
"Unexpected error while extracting/serializing PSSH for DRM type=%s pssh_type=%s",
|
||||
drm_class,
|
||||
pssh_type,
|
||||
)
|
||||
raise
|
||||
|
||||
# Get KIDs
|
||||
if hasattr(drm, "kids") and drm.kids:
|
||||
@@ -406,15 +431,21 @@ def serialize_drm(drm_list) -> Optional[List[Dict[str, Any]]]:
|
||||
return result if result else None
|
||||
|
||||
|
||||
def enum_name(value: Any) -> str:
|
||||
"""Return an enum-like value's .name, falling back to str()."""
|
||||
return value.name if hasattr(value, "name") else str(value)
|
||||
|
||||
|
||||
def descriptor_name(track: Any) -> Optional[str]:
|
||||
"""Manifest descriptor (HLS/DASH/URL) name for a track, or None."""
|
||||
descriptor = getattr(track, "descriptor", None)
|
||||
return enum_name(descriptor) if descriptor else None
|
||||
|
||||
|
||||
def serialize_video_track(track: Video, include_url: bool = False) -> Dict[str, Any]:
|
||||
"""Convert video track to JSON-serializable dict."""
|
||||
codec_name = track.codec.name if hasattr(track.codec, "name") else str(track.codec)
|
||||
range_name = track.range.name if hasattr(track.range, "name") else str(track.range)
|
||||
|
||||
# Serialize the manifest descriptor (HLS, DASH, URL, etc.)
|
||||
descriptor_name = None
|
||||
if hasattr(track, "descriptor") and track.descriptor:
|
||||
descriptor_name = track.descriptor.name if hasattr(track.descriptor, "name") else str(track.descriptor)
|
||||
codec_name = enum_name(track.codec)
|
||||
range_name = enum_name(track.range)
|
||||
|
||||
result = {
|
||||
"id": str(track.id),
|
||||
@@ -429,7 +460,7 @@ def serialize_video_track(track: Video, include_url: bool = False) -> Dict[str,
|
||||
"range_display": DYNAMIC_RANGE_MAP.get(range_name, range_name),
|
||||
"language": str(track.language) if track.language else None,
|
||||
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
|
||||
"descriptor": descriptor_name,
|
||||
"descriptor": descriptor_name(track),
|
||||
}
|
||||
if include_url and hasattr(track, "url") and track.url:
|
||||
result["url"] = str(track.url)
|
||||
@@ -438,12 +469,7 @@ def serialize_video_track(track: Video, include_url: bool = False) -> Dict[str,
|
||||
|
||||
def serialize_audio_track(track: Audio, include_url: bool = False) -> Dict[str, Any]:
|
||||
"""Convert audio track to JSON-serializable dict."""
|
||||
codec_name = track.codec.name if hasattr(track.codec, "name") else str(track.codec)
|
||||
|
||||
# Serialize the manifest descriptor (HLS, DASH, URL, etc.)
|
||||
descriptor_name = None
|
||||
if hasattr(track, "descriptor") and track.descriptor:
|
||||
descriptor_name = track.descriptor.name if hasattr(track.descriptor, "name") else str(track.descriptor)
|
||||
codec_name = enum_name(track.codec)
|
||||
|
||||
result = {
|
||||
"id": str(track.id),
|
||||
@@ -455,7 +481,7 @@ def serialize_audio_track(track: Audio, include_url: bool = False) -> Dict[str,
|
||||
"atmos": track.atmos if hasattr(track, "atmos") else False,
|
||||
"descriptive": track.descriptive if hasattr(track, "descriptive") else False,
|
||||
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
|
||||
"descriptor": descriptor_name,
|
||||
"descriptor": descriptor_name(track),
|
||||
}
|
||||
if include_url and hasattr(track, "url") and track.url:
|
||||
result["url"] = str(track.url)
|
||||
@@ -464,19 +490,14 @@ def serialize_audio_track(track: Audio, include_url: bool = False) -> Dict[str,
|
||||
|
||||
def serialize_subtitle_track(track: Subtitle, include_url: bool = False) -> Dict[str, Any]:
|
||||
"""Convert subtitle track to JSON-serializable dict."""
|
||||
# Get descriptor for compatibility
|
||||
descriptor_name = None
|
||||
if hasattr(track, "descriptor") and track.descriptor:
|
||||
descriptor_name = track.descriptor.name if hasattr(track.descriptor, "name") else str(track.descriptor)
|
||||
|
||||
result = {
|
||||
"id": str(track.id),
|
||||
"codec": track.codec.name if hasattr(track.codec, "name") else str(track.codec),
|
||||
"codec": enum_name(track.codec),
|
||||
"language": str(track.language) if track.language else None,
|
||||
"forced": track.forced if hasattr(track, "forced") else False,
|
||||
"sdh": track.sdh if hasattr(track, "sdh") else False,
|
||||
"cc": track.cc if hasattr(track, "cc") else False,
|
||||
"descriptor": descriptor_name,
|
||||
"descriptor": descriptor_name(track),
|
||||
}
|
||||
if include_url and hasattr(track, "url") and track.url:
|
||||
result["url"] = str(track.url)
|
||||
@@ -575,24 +596,11 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] =
|
||||
|
||||
async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response:
|
||||
"""Handle list-titles request."""
|
||||
require_fields(data, "service", "title_id")
|
||||
service_tag = data.get("service")
|
||||
title_id = data.get("title_id")
|
||||
profile = data.get("profile")
|
||||
|
||||
if not service_tag:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
"Missing required parameter: service",
|
||||
details={"missing_parameter": "service"},
|
||||
)
|
||||
|
||||
if not title_id:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
"Missing required parameter: title_id",
|
||||
details={"missing_parameter": "title_id"},
|
||||
)
|
||||
|
||||
normalized_service = validate_service(service_tag, request)
|
||||
if not normalized_service:
|
||||
raise APIError(
|
||||
@@ -602,37 +610,7 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
|
||||
)
|
||||
|
||||
try:
|
||||
from unshackle.commands.dl import dl
|
||||
|
||||
service_config = load_service_yaml(normalized_service)
|
||||
|
||||
proxy_param = data.get("proxy")
|
||||
no_proxy = data.get("no_proxy", False)
|
||||
proxy_providers = []
|
||||
|
||||
if not no_proxy:
|
||||
proxy_providers = initialize_proxy_providers()
|
||||
|
||||
if proxy_param and not no_proxy:
|
||||
try:
|
||||
resolved_proxy = resolve_proxy(proxy_param, proxy_providers)
|
||||
proxy_param = resolved_proxy
|
||||
except ValueError as e:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_PROXY,
|
||||
f"Proxy error: {e}",
|
||||
details={"proxy": proxy_param, "service": normalized_service},
|
||||
)
|
||||
|
||||
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
|
||||
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
|
||||
service_module = Services.load(normalized_service)
|
||||
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
|
||||
|
||||
cookies = dl.get_cookie_jar(normalized_service, profile)
|
||||
credential = dl.get_credentials(normalized_service, profile)
|
||||
service_instance.authenticate(cookies, credential)
|
||||
|
||||
service_instance = setup_list_service(data, normalized_service, profile, title_id)
|
||||
titles = service_instance.get_titles()
|
||||
|
||||
if hasattr(titles, "__iter__") and not isinstance(titles, str):
|
||||
@@ -656,24 +634,11 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
|
||||
|
||||
async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response:
|
||||
"""Handle list-tracks request."""
|
||||
require_fields(data, "service", "title_id")
|
||||
service_tag = data.get("service")
|
||||
title_id = data.get("title_id")
|
||||
profile = data.get("profile")
|
||||
|
||||
if not service_tag:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
"Missing required parameter: service",
|
||||
details={"missing_parameter": "service"},
|
||||
)
|
||||
|
||||
if not title_id:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
"Missing required parameter: title_id",
|
||||
details={"missing_parameter": "title_id"},
|
||||
)
|
||||
|
||||
normalized_service = validate_service(service_tag, request)
|
||||
if not normalized_service:
|
||||
raise APIError(
|
||||
@@ -683,37 +648,7 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
|
||||
)
|
||||
|
||||
try:
|
||||
from unshackle.commands.dl import dl
|
||||
|
||||
service_config = load_service_yaml(normalized_service)
|
||||
|
||||
proxy_param = data.get("proxy")
|
||||
no_proxy = data.get("no_proxy", False)
|
||||
proxy_providers = []
|
||||
|
||||
if not no_proxy:
|
||||
proxy_providers = initialize_proxy_providers()
|
||||
|
||||
if proxy_param and not no_proxy:
|
||||
try:
|
||||
resolved_proxy = resolve_proxy(proxy_param, proxy_providers)
|
||||
proxy_param = resolved_proxy
|
||||
except ValueError as e:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_PROXY,
|
||||
f"Proxy error: {e}",
|
||||
details={"proxy": proxy_param, "service": normalized_service},
|
||||
)
|
||||
|
||||
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
|
||||
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
|
||||
service_module = Services.load(normalized_service)
|
||||
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
|
||||
|
||||
cookies = dl.get_cookie_jar(normalized_service, profile)
|
||||
credential = dl.get_credentials(normalized_service, profile)
|
||||
service_instance.authenticate(cookies, credential)
|
||||
|
||||
service_instance = setup_list_service(data, normalized_service, profile, title_id)
|
||||
titles = service_instance.get_titles()
|
||||
|
||||
wanted_param = data.get("wanted")
|
||||
@@ -855,6 +790,25 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
|
||||
)
|
||||
|
||||
|
||||
VALID_VCODECS = ["H264", "H265", "H.264", "H.265", "AVC", "HEVC", "VC1", "VC-1", "VP8", "VP9", "AV1"]
|
||||
VALID_ACODECS = ["AAC", "AC3", "EC3", "EAC3", "DD", "DD+", "AC4", "OPUS", "FLAC", "ALAC", "VORBIS", "OGG", "DTS"]
|
||||
|
||||
|
||||
def check_codec(value: Any, allowed: List[str], name: str) -> Optional[str]:
|
||||
"""Validate a comma-string or list of codec tokens against `allowed` (case-insensitive)."""
|
||||
if isinstance(value, str):
|
||||
tokens = [v.strip() for v in value.split(",") if v.strip()]
|
||||
elif isinstance(value, list):
|
||||
tokens = [str(v).strip() for v in value if str(v).strip()]
|
||||
else:
|
||||
return f"{name} must be a string or list"
|
||||
|
||||
invalid = [token for token in tokens if token.upper() not in allowed]
|
||||
if invalid:
|
||||
return f"Invalid {name}: {', '.join(invalid)}. Must be one of: {', '.join(allowed)}"
|
||||
return None
|
||||
|
||||
|
||||
def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
|
||||
"""
|
||||
Validate download parameters and return error message if invalid.
|
||||
@@ -863,44 +817,14 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
|
||||
None if valid, error message string if invalid
|
||||
"""
|
||||
if "vcodec" in data and data["vcodec"]:
|
||||
valid_vcodecs = ["H264", "H265", "H.264", "H.265", "AVC", "HEVC", "VC1", "VC-1", "VP8", "VP9", "AV1"]
|
||||
if isinstance(data["vcodec"], str):
|
||||
vcodec_values = [v.strip() for v in data["vcodec"].split(",") if v.strip()]
|
||||
elif isinstance(data["vcodec"], list):
|
||||
vcodec_values = [str(v).strip() for v in data["vcodec"] if str(v).strip()]
|
||||
else:
|
||||
return "vcodec must be a string or list"
|
||||
|
||||
invalid = [value for value in vcodec_values if value.upper() not in valid_vcodecs]
|
||||
if invalid:
|
||||
return f"Invalid vcodec: {', '.join(invalid)}. Must be one of: {', '.join(valid_vcodecs)}"
|
||||
err = check_codec(data["vcodec"], VALID_VCODECS, "vcodec")
|
||||
if err:
|
||||
return err
|
||||
|
||||
if "acodec" in data and data["acodec"]:
|
||||
valid_acodecs = [
|
||||
"AAC",
|
||||
"AC3",
|
||||
"EC3",
|
||||
"EAC3",
|
||||
"DD",
|
||||
"DD+",
|
||||
"AC4",
|
||||
"OPUS",
|
||||
"FLAC",
|
||||
"ALAC",
|
||||
"VORBIS",
|
||||
"OGG",
|
||||
"DTS",
|
||||
]
|
||||
if isinstance(data["acodec"], str):
|
||||
acodec_values = [v.strip() for v in data["acodec"].split(",") if v.strip()]
|
||||
elif isinstance(data["acodec"], list):
|
||||
acodec_values = [str(v).strip() for v in data["acodec"] if str(v).strip()]
|
||||
else:
|
||||
return "acodec must be a string or list"
|
||||
|
||||
invalid = [value for value in acodec_values if value.upper() not in valid_acodecs]
|
||||
if invalid:
|
||||
return f"Invalid acodec: {', '.join(invalid)}. Must be one of: {', '.join(valid_acodecs)}"
|
||||
err = check_codec(data["acodec"], VALID_ACODECS, "acodec")
|
||||
if err:
|
||||
return err
|
||||
|
||||
if "sub_format" in data and data["sub_format"]:
|
||||
valid_sub_formats = ["SRT", "VTT", "ASS", "SSA", "TTML", "STPP", "WVTT", "SMI", "SUB", "MPL2", "TMP"]
|
||||
@@ -972,23 +896,10 @@ async def download_handler(data: Dict[str, Any], request: Optional[web.Request]
|
||||
"""Handle download request - create and queue a download job."""
|
||||
from unshackle.core.api.download_manager import get_download_manager
|
||||
|
||||
require_fields(data, "service", "title_id")
|
||||
service_tag = data.get("service")
|
||||
title_id = data.get("title_id")
|
||||
|
||||
if not service_tag:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
"Missing required parameter: service",
|
||||
details={"missing_parameter": "service"},
|
||||
)
|
||||
|
||||
if not title_id:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
"Missing required parameter: title_id",
|
||||
details={"missing_parameter": "title_id"},
|
||||
)
|
||||
|
||||
normalized_service = validate_service(service_tag, request)
|
||||
if not normalized_service:
|
||||
raise APIError(
|
||||
@@ -1426,7 +1337,6 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
|
||||
session.auth_status = AuthStatus.FAILED
|
||||
session.auth_error = str(e)
|
||||
bridge.status = AuthStatus.FAILED
|
||||
bridge.error = str(e)
|
||||
|
||||
asyncio.create_task(_run_auth())
|
||||
|
||||
@@ -1671,31 +1581,12 @@ async def session_segments_handler(
|
||||
else:
|
||||
track_info["cookies"] = {}
|
||||
|
||||
# Include manifest-specific data for segment resolution
|
||||
# Include manifest-specific data for segment resolution. Round-trip through
|
||||
# JSON so any non-serializable value becomes its str() (default=str).
|
||||
if hasattr(track, "data") and track.data:
|
||||
track_data = {}
|
||||
for key, val in track.data.items():
|
||||
if isinstance(val, dict):
|
||||
# Convert non-serializable values
|
||||
serializable = {}
|
||||
for k, v in val.items():
|
||||
try:
|
||||
import json
|
||||
import json
|
||||
|
||||
json.dumps(v)
|
||||
serializable[k] = v
|
||||
except (TypeError, ValueError):
|
||||
serializable[k] = str(v)
|
||||
track_data[key] = serializable
|
||||
else:
|
||||
try:
|
||||
import json
|
||||
|
||||
json.dumps(val)
|
||||
track_data[key] = val
|
||||
except (TypeError, ValueError):
|
||||
track_data[key] = str(val)
|
||||
track_info["data"] = track_data
|
||||
track_info["data"] = json.loads(json.dumps(track.data, default=str))
|
||||
else:
|
||||
track_info["data"] = {}
|
||||
|
||||
@@ -1715,15 +1606,10 @@ async def session_segments_handler(
|
||||
)
|
||||
|
||||
|
||||
class _CdmTypeStub:
|
||||
"""Lightweight CDM stub so ``is_playready_cdm()`` can detect CDM type.
|
||||
|
||||
Used on the server when the client sends ``cdm_type`` but the server
|
||||
does not need a full CDM (e.g. for cache key / device selection only).
|
||||
"""
|
||||
|
||||
def __init__(self, cdm_type: str) -> None:
|
||||
self.is_playready = cdm_type == "playready"
|
||||
def _cdm_type_stub(cdm_type: str) -> SimpleNamespace:
|
||||
"""Lightweight CDM stand-in so ``is_playready_cdm()`` (reads ``.is_playready``)
|
||||
can detect the type without loading a full CDM."""
|
||||
return SimpleNamespace(is_playready=cdm_type == "playready")
|
||||
|
||||
|
||||
def _resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]) -> Optional[Any]:
|
||||
@@ -1751,10 +1637,10 @@ def _resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional
|
||||
if cdm_name and isinstance(cdm_name, str):
|
||||
detected_type = _detect_cdm_type(cdm_name, app_config)
|
||||
if detected_type:
|
||||
return _CdmTypeStub(detected_type)
|
||||
return _cdm_type_stub(detected_type)
|
||||
|
||||
if cdm_type:
|
||||
return _CdmTypeStub(cdm_type)
|
||||
return _cdm_type_stub(cdm_type)
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -10,7 +10,6 @@ response value.
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
@@ -25,8 +24,9 @@ class AuthStatus(Enum):
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
class BridgeCancelledError(Exception):
|
||||
"""Raised when the bridge is cancelled (client disconnected, session deleted)."""
|
||||
# Seconds the server waits for a remote client to answer an input prompt before
|
||||
# giving up; also the TTL SessionStore grants AUTHENTICATING/PENDING_INPUT sessions.
|
||||
AUTH_INPUT_TIMEOUT = 600.0
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -40,12 +40,11 @@ class InputBridge:
|
||||
_prompt: Optional[str] = field(default=None, init=False, repr=False)
|
||||
_response: Optional[str] = field(default=None, init=False, repr=False)
|
||||
_status: AuthStatus = field(default=AuthStatus.AUTHENTICATING, init=False)
|
||||
_error: Optional[str] = field(default=None, init=False, repr=False)
|
||||
_cancelled: bool = field(default=False, init=False, repr=False)
|
||||
_response_ready: threading.Event = field(default_factory=threading.Event, init=False, repr=False)
|
||||
_lock: threading.Lock = field(default_factory=threading.Lock, init=False, repr=False)
|
||||
|
||||
def request_input(self, prompt: str, timeout: float = 600.0) -> str:
|
||||
def request_input(self, prompt: str, timeout: float = AUTH_INPUT_TIMEOUT) -> str:
|
||||
"""Block until the remote client submits a response for *prompt*.
|
||||
|
||||
Args:
|
||||
@@ -57,32 +56,26 @@ class InputBridge:
|
||||
|
||||
Raises:
|
||||
TimeoutError: If no response is received within *timeout*.
|
||||
BridgeCancelledError: If the bridge was cancelled.
|
||||
RuntimeError: If the bridge was cancelled.
|
||||
"""
|
||||
with self._lock:
|
||||
if self._cancelled:
|
||||
raise BridgeCancelledError("Session was cancelled")
|
||||
raise RuntimeError("Session was cancelled")
|
||||
self._prompt = prompt
|
||||
self._response = None
|
||||
self._status = AuthStatus.PENDING_INPUT
|
||||
self._response_ready.clear()
|
||||
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if self._response_ready.wait(timeout=0.5):
|
||||
break
|
||||
with self._lock:
|
||||
if self._cancelled:
|
||||
raise BridgeCancelledError("Session was cancelled")
|
||||
else:
|
||||
# cancel() and submit_response() both .set() the event, so a single wait
|
||||
# returns on signal-or-cancel; no poll loop needed.
|
||||
if not self._response_ready.wait(timeout=timeout):
|
||||
with self._lock:
|
||||
self._status = AuthStatus.FAILED
|
||||
self._error = "Input request timed out waiting for client response"
|
||||
raise TimeoutError(f"No client response for prompt within {timeout}s")
|
||||
|
||||
with self._lock:
|
||||
if self._cancelled:
|
||||
raise BridgeCancelledError("Session was cancelled")
|
||||
raise RuntimeError("Session was cancelled")
|
||||
response = self._response or ""
|
||||
self._prompt = None
|
||||
self._response = None
|
||||
@@ -115,7 +108,6 @@ class InputBridge:
|
||||
with self._lock:
|
||||
self._cancelled = True
|
||||
self._status = AuthStatus.FAILED
|
||||
self._error = "Session cancelled"
|
||||
self._response_ready.set()
|
||||
|
||||
@property
|
||||
@@ -127,13 +119,3 @@ class InputBridge:
|
||||
def status(self, value: AuthStatus) -> None:
|
||||
with self._lock:
|
||||
self._status = value
|
||||
|
||||
@property
|
||||
def error(self) -> Optional[str]:
|
||||
with self._lock:
|
||||
return self._error
|
||||
|
||||
@error.setter
|
||||
def error(self, value: Optional[str]) -> None:
|
||||
with self._lock:
|
||||
self._error = value
|
||||
|
||||
@@ -6,7 +6,7 @@ a bitrate-weighted percentage, track counts, and the labels of the tracks downlo
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from unshackle.core.constants import AnyTrack
|
||||
|
||||
@@ -70,6 +70,9 @@ def build_job_progress_callables(
|
||||
fractions = [0.0] * total
|
||||
done = [False] * total
|
||||
started = [False] * total
|
||||
seg_done = [0.0] * total
|
||||
seg_total = [0.0] * total
|
||||
speeds: list[Optional[str]] = [None] * total
|
||||
|
||||
def emit() -> None:
|
||||
completed = sum(done)
|
||||
@@ -82,6 +85,8 @@ def build_job_progress_callables(
|
||||
phase += f" (+{len(active) - 3} more)"
|
||||
else:
|
||||
phase = f"downloading {completed}/{total} tracks"
|
||||
# segment counts and transfer speed of the track downloading now, for a granular display
|
||||
active_i = next((i for i in range(total) if started[i] and not done[i]), None)
|
||||
sink(
|
||||
{
|
||||
"progress": progress,
|
||||
@@ -89,6 +94,9 @@ def build_job_progress_callables(
|
||||
"completed_tracks": completed,
|
||||
"total_tracks": total,
|
||||
"active_tracks": active,
|
||||
"segments_done": seg_done[active_i] if active_i is not None else 0.0,
|
||||
"segments_total": seg_total[active_i] if active_i is not None else 0.0,
|
||||
"speed": speeds[active_i] if active_i is not None else None,
|
||||
"status": "downloading",
|
||||
}
|
||||
)
|
||||
@@ -104,6 +112,11 @@ def build_job_progress_callables(
|
||||
counts["completed"] = kwargs["completed"]
|
||||
if "advance" in kwargs:
|
||||
counts["completed"] += kwargs["advance"]
|
||||
seg_done[index] = counts["completed"]
|
||||
seg_total[index] = counts["total"]
|
||||
_dl = kwargs.get("downloaded")
|
||||
if isinstance(_dl, str) and _dl.endswith("/s"):
|
||||
speeds[index] = _dl
|
||||
if kwargs.get("downloaded") in JOB_PROGRESS_TERMINAL_STATES:
|
||||
done[index] = True
|
||||
fractions[index] = 1.0
|
||||
|
||||
+91
-118
@@ -1,5 +1,8 @@
|
||||
import functools
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any
|
||||
|
||||
import click
|
||||
from aiohttp import web
|
||||
@@ -18,9 +21,12 @@ from unshackle.core.update_checker import UpdateChecker
|
||||
|
||||
|
||||
@web.middleware
|
||||
async def cors_middleware(request: web.Request, handler):
|
||||
async def cors_middleware(
|
||||
request: web.Request, handler: Callable[[web.Request], Awaitable[web.StreamResponse]]
|
||||
) -> web.StreamResponse:
|
||||
"""Add CORS headers to all responses."""
|
||||
# Handle preflight requests
|
||||
response: web.StreamResponse
|
||||
if request.method == "OPTIONS":
|
||||
response = web.Response()
|
||||
else:
|
||||
@@ -37,7 +43,24 @@ async def cors_middleware(request: web.Request, handler):
|
||||
|
||||
log = logging.getLogger("api")
|
||||
|
||||
# Route handler signature: takes the request, returns a response.
|
||||
Handler = Callable[[web.Request], Awaitable[web.Response]]
|
||||
|
||||
|
||||
def api_handler(handler: Handler) -> Handler:
|
||||
"""Wrap a route handler so any raised APIError becomes a structured error response."""
|
||||
|
||||
@functools.wraps(handler)
|
||||
async def wrapper(request: web.Request) -> web.Response:
|
||||
try:
|
||||
return await handler(request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
@api_handler
|
||||
async def health(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Health check endpoint.
|
||||
@@ -84,6 +107,7 @@ async def health(request: web.Request) -> web.Response:
|
||||
return web.json_response({"status": "ok", "version": __version__, "update_check": update_info})
|
||||
|
||||
|
||||
@api_handler
|
||||
async def services(request: web.Request) -> web.Response:
|
||||
"""
|
||||
List available services.
|
||||
@@ -161,7 +185,14 @@ async def services(request: web.Request) -> web.Response:
|
||||
services_info = []
|
||||
|
||||
for tag in service_tags:
|
||||
service_data = {"tag": tag, "aliases": [], "geofence": [], "title_regex": None, "url": None, "help": None}
|
||||
service_data: dict[str, Any] = {
|
||||
"tag": tag,
|
||||
"aliases": [],
|
||||
"geofence": [],
|
||||
"title_regex": None,
|
||||
"url": None,
|
||||
"help": None,
|
||||
}
|
||||
|
||||
try:
|
||||
service_module = Services.load(tag)
|
||||
@@ -268,6 +299,7 @@ async def services(request: web.Request) -> web.Response:
|
||||
return handle_api_exception(e, context={"operation": "list_services"}, debug_mode=debug_mode)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def search(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Search for titles from a service.
|
||||
@@ -347,14 +379,13 @@ async def search(request: web.Request) -> web.Response:
|
||||
|
||||
try:
|
||||
return await search_handler(data, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in search")
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return handle_api_exception(e, context={"operation": "search"}, debug_mode=debug_mode)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def list_titles(request: web.Request) -> web.Response:
|
||||
"""
|
||||
List titles for a service and title ID.
|
||||
@@ -474,13 +505,10 @@ async def list_titles(request: web.Request) -> web.Response:
|
||||
request.app.get("debug_api", False),
|
||||
)
|
||||
|
||||
try:
|
||||
return await list_titles_handler(data, request)
|
||||
except APIError as e:
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return build_error_response(e, debug_mode)
|
||||
return await list_titles_handler(data, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def list_tracks(request: web.Request) -> web.Response:
|
||||
"""
|
||||
List tracks for a title, separated by type.
|
||||
@@ -527,13 +555,10 @@ async def list_tracks(request: web.Request) -> web.Response:
|
||||
request.app.get("debug_api", False),
|
||||
)
|
||||
|
||||
try:
|
||||
return await list_tracks_handler(data, request)
|
||||
except APIError as e:
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return build_error_response(e, debug_mode)
|
||||
return await list_tracks_handler(data, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def download(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Download content based on provided parameters.
|
||||
@@ -767,13 +792,10 @@ async def download(request: web.Request) -> web.Response:
|
||||
request.app.get("debug_api", False),
|
||||
)
|
||||
|
||||
try:
|
||||
return await download_handler(data, request)
|
||||
except APIError as e:
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return build_error_response(e, debug_mode)
|
||||
return await download_handler(data, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def download_jobs(request: web.Request) -> web.Response:
|
||||
"""
|
||||
List all download jobs with optional filtering and sorting.
|
||||
@@ -847,13 +869,10 @@ async def download_jobs(request: web.Request) -> web.Response:
|
||||
"sort_by": request.query.get("sort_by", "created_time"),
|
||||
"sort_order": request.query.get("sort_order", "desc"),
|
||||
}
|
||||
try:
|
||||
return await list_download_jobs_handler(query_params, request)
|
||||
except APIError as e:
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return build_error_response(e, debug_mode)
|
||||
return await list_download_jobs_handler(query_params, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def download_job_detail(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Get download job details.
|
||||
@@ -875,13 +894,10 @@ async def download_job_detail(request: web.Request) -> web.Response:
|
||||
description: Server error
|
||||
"""
|
||||
job_id = request.match_info["job_id"]
|
||||
try:
|
||||
return await get_download_job_handler(job_id, request)
|
||||
except APIError as e:
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return build_error_response(e, debug_mode)
|
||||
return await get_download_job_handler(job_id, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def cancel_download_job(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Cancel download job.
|
||||
@@ -905,13 +921,10 @@ async def cancel_download_job(request: web.Request) -> web.Response:
|
||||
description: Server error
|
||||
"""
|
||||
job_id = request.match_info["job_id"]
|
||||
try:
|
||||
return await cancel_download_job_handler(job_id, request)
|
||||
except APIError as e:
|
||||
debug_mode = request.app.get("debug_api", False)
|
||||
return build_error_response(e, debug_mode)
|
||||
return await cancel_download_job_handler(job_id, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_create(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Create a remote-dl session.
|
||||
@@ -964,8 +977,6 @@ async def session_create(request: web.Request) -> web.Response:
|
||||
)
|
||||
try:
|
||||
return await session_create_handler(data, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session create")
|
||||
return handle_api_exception(
|
||||
@@ -973,6 +984,7 @@ async def session_create(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_titles(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Get titles for an authenticated session.
|
||||
@@ -994,8 +1006,6 @@ async def session_titles(request: web.Request) -> web.Response:
|
||||
session_id = request.match_info["session_id"]
|
||||
try:
|
||||
return await session_titles_handler(session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session titles")
|
||||
return handle_api_exception(
|
||||
@@ -1003,6 +1013,7 @@ async def session_titles(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_tracks(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Get tracks and chapters for a specific title.
|
||||
@@ -1043,8 +1054,6 @@ async def session_tracks(request: web.Request) -> web.Response:
|
||||
)
|
||||
try:
|
||||
return await session_tracks_handler(data, session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session tracks")
|
||||
return handle_api_exception(
|
||||
@@ -1052,6 +1061,7 @@ async def session_tracks(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_segments(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Resolve segment URLs for selected tracks.
|
||||
@@ -1094,8 +1104,6 @@ async def session_segments(request: web.Request) -> web.Response:
|
||||
)
|
||||
try:
|
||||
return await session_segments_handler(data, session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session segments")
|
||||
return handle_api_exception(
|
||||
@@ -1103,6 +1111,7 @@ async def session_segments(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_license(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Proxy DRM license through authenticated service.
|
||||
@@ -1151,8 +1160,6 @@ async def session_license(request: web.Request) -> web.Response:
|
||||
)
|
||||
try:
|
||||
return await session_license_handler(data, session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session license")
|
||||
return handle_api_exception(
|
||||
@@ -1160,6 +1167,7 @@ async def session_license(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_info(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Get session info.
|
||||
@@ -1179,12 +1187,10 @@ async def session_info(request: web.Request) -> web.Response:
|
||||
description: Session not found
|
||||
"""
|
||||
session_id = request.match_info["session_id"]
|
||||
try:
|
||||
return await session_info_handler(session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
return await session_info_handler(session_id, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_delete(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Delete a session.
|
||||
@@ -1204,12 +1210,10 @@ async def session_delete(request: web.Request) -> web.Response:
|
||||
description: Session not found
|
||||
"""
|
||||
session_id = request.match_info["session_id"]
|
||||
try:
|
||||
return await session_delete_handler(session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
return await session_delete_handler(session_id, request)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_prompt_get(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Poll for pending interactive prompts during authentication.
|
||||
@@ -1245,8 +1249,6 @@ async def session_prompt_get(request: web.Request) -> web.Response:
|
||||
session_id = request.match_info["session_id"]
|
||||
try:
|
||||
return await session_prompt_get_handler(session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session prompt get")
|
||||
return handle_api_exception(
|
||||
@@ -1254,6 +1256,7 @@ async def session_prompt_get(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
@api_handler
|
||||
async def session_prompt_submit(request: web.Request) -> web.Response:
|
||||
"""
|
||||
Submit a response to a pending interactive prompt.
|
||||
@@ -1304,8 +1307,6 @@ async def session_prompt_submit(request: web.Request) -> web.Response:
|
||||
)
|
||||
try:
|
||||
return await session_prompt_post_handler(data, session_id, request)
|
||||
except APIError as e:
|
||||
return build_error_response(e, request.app.get("debug_api", False))
|
||||
except Exception as e:
|
||||
log.exception("Error in session prompt submit")
|
||||
return handle_api_exception(
|
||||
@@ -1313,48 +1314,42 @@ async def session_prompt_submit(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
def _setup_remote_session_routes(app: web.Application) -> None:
|
||||
"""Setup remote-DL session endpoints only."""
|
||||
app.router.add_get("/api/health", health)
|
||||
app.router.add_get("/api/services", services)
|
||||
app.router.add_post("/api/search", search)
|
||||
app.router.add_post("/api/session/create", session_create)
|
||||
app.router.add_get("/api/session/{session_id}/titles", session_titles)
|
||||
app.router.add_post("/api/session/{session_id}/tracks", session_tracks)
|
||||
app.router.add_post("/api/session/{session_id}/segments", session_segments)
|
||||
app.router.add_post("/api/session/{session_id}/license", session_license)
|
||||
app.router.add_get("/api/session/{session_id}/prompt", session_prompt_get)
|
||||
app.router.add_post("/api/session/{session_id}/prompt", session_prompt_submit)
|
||||
app.router.add_get("/api/session/{session_id}", session_info)
|
||||
app.router.add_delete("/api/session/{session_id}", session_delete)
|
||||
# Single source of truth for all API routes. `remote` marks endpoints exposed in
|
||||
# --remote-only server mode; full mode registers every route. Both setup_routes and
|
||||
# setup_swagger derive from this table so the live routes and the docs cannot drift.
|
||||
ROUTES: list[tuple[str, str, Handler, bool]] = [
|
||||
("GET", "/api/health", health, True),
|
||||
("GET", "/api/services", services, True),
|
||||
("POST", "/api/search", search, True),
|
||||
("POST", "/api/list-titles", list_titles, False),
|
||||
("POST", "/api/list-tracks", list_tracks, False),
|
||||
("POST", "/api/download", download, False),
|
||||
("GET", "/api/download/jobs", download_jobs, False),
|
||||
("GET", "/api/download/jobs/{job_id}", download_job_detail, False),
|
||||
("DELETE", "/api/download/jobs/{job_id}", cancel_download_job, False),
|
||||
("POST", "/api/session/create", session_create, True),
|
||||
("GET", "/api/session/{session_id}/titles", session_titles, True),
|
||||
("POST", "/api/session/{session_id}/tracks", session_tracks, True),
|
||||
("POST", "/api/session/{session_id}/segments", session_segments, True),
|
||||
("POST", "/api/session/{session_id}/license", session_license, True),
|
||||
("GET", "/api/session/{session_id}/prompt", session_prompt_get, True),
|
||||
("POST", "/api/session/{session_id}/prompt", session_prompt_submit, True),
|
||||
("GET", "/api/session/{session_id}", session_info, True),
|
||||
("DELETE", "/api/session/{session_id}", session_delete, True),
|
||||
]
|
||||
|
||||
|
||||
def setup_routes(app: web.Application, remote_only: bool = False) -> None:
|
||||
"""Setup API routes. When remote_only=True, only expose remote session endpoints."""
|
||||
if remote_only:
|
||||
_setup_remote_session_routes(app)
|
||||
return
|
||||
|
||||
app.router.add_get("/api/health", health)
|
||||
app.router.add_get("/api/services", services)
|
||||
app.router.add_post("/api/search", search)
|
||||
app.router.add_post("/api/list-titles", list_titles)
|
||||
app.router.add_post("/api/list-tracks", list_tracks)
|
||||
app.router.add_post("/api/download", download)
|
||||
app.router.add_get("/api/download/jobs", download_jobs)
|
||||
app.router.add_get("/api/download/jobs/{job_id}", download_job_detail)
|
||||
app.router.add_delete("/api/download/jobs/{job_id}", cancel_download_job)
|
||||
|
||||
# Remote-DL session endpoints
|
||||
app.router.add_post("/api/session/create", session_create)
|
||||
app.router.add_get("/api/session/{session_id}/titles", session_titles)
|
||||
app.router.add_post("/api/session/{session_id}/tracks", session_tracks)
|
||||
app.router.add_post("/api/session/{session_id}/segments", session_segments)
|
||||
app.router.add_post("/api/session/{session_id}/license", session_license)
|
||||
app.router.add_get("/api/session/{session_id}/prompt", session_prompt_get)
|
||||
app.router.add_post("/api/session/{session_id}/prompt", session_prompt_submit)
|
||||
app.router.add_get("/api/session/{session_id}", session_info)
|
||||
app.router.add_delete("/api/session/{session_id}", session_delete)
|
||||
add: dict[str, Callable[..., Any]] = {
|
||||
"GET": app.router.add_get,
|
||||
"POST": app.router.add_post,
|
||||
"DELETE": app.router.add_delete,
|
||||
}
|
||||
for method, path, handler, remote in ROUTES:
|
||||
if remote_only and not remote:
|
||||
continue
|
||||
add[method](path, handler)
|
||||
|
||||
|
||||
def setup_swagger(app: web.Application) -> None:
|
||||
@@ -1369,27 +1364,5 @@ def setup_swagger(app: web.Application) -> None:
|
||||
),
|
||||
)
|
||||
|
||||
# Add routes with OpenAPI documentation
|
||||
swagger.add_routes(
|
||||
[
|
||||
web.get("/api/health", health),
|
||||
web.get("/api/services", services),
|
||||
web.post("/api/search", search),
|
||||
web.post("/api/list-titles", list_titles),
|
||||
web.post("/api/list-tracks", list_tracks),
|
||||
web.post("/api/download", download),
|
||||
web.get("/api/download/jobs", download_jobs),
|
||||
web.get("/api/download/jobs/{job_id}", download_job_detail),
|
||||
web.delete("/api/download/jobs/{job_id}", cancel_download_job),
|
||||
# Remote-DL session endpoints
|
||||
web.post("/api/session/create", session_create),
|
||||
web.get("/api/session/{session_id}/titles", session_titles),
|
||||
web.post("/api/session/{session_id}/tracks", session_tracks),
|
||||
web.post("/api/session/{session_id}/segments", session_segments),
|
||||
web.post("/api/session/{session_id}/license", session_license),
|
||||
web.get("/api/session/{session_id}/prompt", session_prompt_get),
|
||||
web.post("/api/session/{session_id}/prompt", session_prompt_submit),
|
||||
web.get("/api/session/{session_id}", session_info),
|
||||
web.delete("/api/session/{session_id}", session_delete),
|
||||
]
|
||||
)
|
||||
route: dict[str, Callable[..., Any]] = {"GET": web.get, "POST": web.post, "DELETE": web.delete}
|
||||
swagger.add_routes([route[method](path, handler) for method, path, handler, _ in ROUTES])
|
||||
|
||||
@@ -14,7 +14,7 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from unshackle.core.api.input_bridge import AuthStatus, InputBridge
|
||||
from unshackle.core.api.input_bridge import AUTH_INPUT_TIMEOUT, AuthStatus, InputBridge
|
||||
from unshackle.core.api.sanitize import sanitize_log
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.tracks import Track
|
||||
@@ -127,7 +127,7 @@ class SessionStore:
|
||||
for sid, entry in self._sessions.items():
|
||||
elapsed = (now - entry.last_accessed).total_seconds()
|
||||
if entry.auth_status in (AuthStatus.AUTHENTICATING, AuthStatus.PENDING_INPUT):
|
||||
if elapsed > 600:
|
||||
if elapsed > AUTH_INPUT_TIMEOUT:
|
||||
expired.append(sid)
|
||||
elif elapsed > self._ttl:
|
||||
expired.append(sid)
|
||||
|
||||
@@ -47,6 +47,7 @@ class Config:
|
||||
self.curl_impersonate: dict = kwargs.get("curl_impersonate") or {}
|
||||
self.remote_cdm: list[dict] = kwargs.get("remote_cdm") or []
|
||||
self.credentials: dict = kwargs.get("credentials") or {}
|
||||
self.firefox_cookies: dict = kwargs.get("firefox_cookies") or {}
|
||||
self.subtitle: dict = kwargs.get("subtitle") or {}
|
||||
|
||||
self.directories = self._Directories()
|
||||
@@ -74,6 +75,7 @@ class Config:
|
||||
self.audio: dict = kwargs.get("audio") or {}
|
||||
self.headers: dict = kwargs.get("headers") or {}
|
||||
self.key_vaults: list[dict[str, Any]] = kwargs.get("key_vaults", [])
|
||||
self.vault_timeout: float = kwargs.get("vault_timeout", 10.0)
|
||||
self.muxing: dict = kwargs.get("muxing") or {}
|
||||
self.proxy_providers: dict = kwargs.get("proxy_providers") or {}
|
||||
self.remote_services: dict = kwargs.get("remote_services") or {}
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
from .basic import Basic
|
||||
from .expressvpn import ExpressVPN
|
||||
from .gluetun import Gluetun
|
||||
from .hola import Hola
|
||||
from .nordvpn import NordVPN
|
||||
from .surfsharkvpn import SurfsharkVPN
|
||||
from .windscribevpn import WindscribeVPN
|
||||
|
||||
__all__ = ("Basic", "Gluetun", "Hola", "NordVPN", "SurfsharkVPN", "WindscribeVPN")
|
||||
__all__ = ("Basic", "ExpressVPN", "Gluetun", "Hola", "NordVPN", "SurfsharkVPN", "WindscribeVPN")
|
||||
|
||||
@@ -0,0 +1,845 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import secrets
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
import requests
|
||||
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.proxies.proxy import Proxy
|
||||
|
||||
log = logging.getLogger("proxies.expressvpn")
|
||||
|
||||
|
||||
class ExpressVPN(Proxy):
|
||||
"""
|
||||
ExpressVPN HTTPS proxy provider.
|
||||
|
||||
This provider follows the browser extension proxy flow:
|
||||
browser cookies or a cached refresh token are exchanged for API tokens,
|
||||
those tokens are used to resolve proxy-capable locations, and get_proxy()
|
||||
returns an authenticated HTTPS proxy URL.
|
||||
|
||||
Query format:
|
||||
country -- smart connection (random location)
|
||||
country-city -- specific city
|
||||
country-city-N / cityN -- pinned server by position
|
||||
full-slug -- ExpressVPN location slug
|
||||
hostname.expressprovider.com -- direct hostname
|
||||
"""
|
||||
|
||||
CLIENT_ID = "f457fed092a54b9e9f1e2113782d74a2"
|
||||
AUTH_BASE = "https://auth.expressvpn.com/realms/xvpn/protocol/openid-connect"
|
||||
API_BASE = "https://cp.expressapisv2.net"
|
||||
EXTENSION_REDIRECT_URI = "chrome-extension://fgddmllnllkalaagkghckoinaemmogpe/src/html/auth-callback.html"
|
||||
USER_AGENT = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
region_map: Optional[dict[str, str]] = None,
|
||||
server_map: Optional[dict[str, str]] = None,
|
||||
refresh_token: Optional[str] = None,
|
||||
access_token: Optional[str] = None,
|
||||
connection_token: Optional[str] = None,
|
||||
account_json: Optional[str] = None,
|
||||
cookie_path: Optional[str] = None,
|
||||
cache_path: Optional[str] = None,
|
||||
timeout: float = 10.0,
|
||||
):
|
||||
"""
|
||||
Proxy Service using ExpressVPN browser-extension proxy credentials.
|
||||
|
||||
Args:
|
||||
region_map: Optional country-to-preset mapping. Keys are country
|
||||
codes (e.g. ``"us"``), values are optional city/server presets
|
||||
(e.g. ``"ny-02"``). When a key is used without a city in the
|
||||
CLI query, the preset is applied. Empty/null values enable
|
||||
smart connection (random location in that country).
|
||||
server_map: Optional aliases using the same convention as other
|
||||
proxy providers. Values may be ExpressVPN location slugs or
|
||||
concrete .expressprovider.com hosts.
|
||||
refresh_token: Optional OAuth refresh token. If omitted, cached
|
||||
tokens or browser cookies are used.
|
||||
access_token: Optional OAuth access token for advanced/manual use.
|
||||
connection_token: Optional cached connection authorization token.
|
||||
account_json: Optional path to ExpressVPN desktop account.json.
|
||||
cookie_path: Optional path to exported ExpressVPN browser cookies.
|
||||
cache_path: Optional path for cached ExpressVPN tokens.
|
||||
timeout: Request timeout in seconds.
|
||||
"""
|
||||
if region_map is not None and not isinstance(region_map, dict):
|
||||
raise TypeError(f"Expected region_map to be a dict mapping aliases to locations, not '{region_map!r}'.")
|
||||
if server_map is not None and not isinstance(server_map, dict):
|
||||
raise TypeError(f"Expected server_map to be a dict mapping aliases to locations, not '{server_map!r}'.")
|
||||
|
||||
# region_map: country code → optional preset (city[-server])
|
||||
self.region_map = {
|
||||
str(k).lower().strip(): (str(v).lower().strip() if v else None)
|
||||
for k, v in (region_map or {}).items()
|
||||
}
|
||||
# server_map: legacy alias → location slug or direct hostname
|
||||
self.server_map = {
|
||||
str(k).lower().strip(): str(v).lower().strip()
|
||||
for k, v in (server_map or {}).items()
|
||||
}
|
||||
self.refresh_token = refresh_token or None
|
||||
self.access_token = access_token or None
|
||||
self.connection_token = connection_token or None
|
||||
self.account_json = Path(account_json).expanduser() if account_json else None
|
||||
self.timeout = timeout
|
||||
self._tokens: Optional[dict] = None
|
||||
self._srt: Optional[str] = None
|
||||
self._locations: Optional[list[dict]] = None
|
||||
|
||||
# Display info set during _resolve_endpoint for log messages
|
||||
self._last_location_name: Optional[str] = None
|
||||
self._last_server_index: Optional[int] = None
|
||||
self._last_server_total: Optional[int] = None
|
||||
self._last_endpoint_host: Optional[str] = None
|
||||
|
||||
self.cookie_path = Path(cookie_path).expanduser() if cookie_path else self._default_cookie_path()
|
||||
self.cache_path = Path(cache_path).expanduser() if cache_path else self._default_cache_path()
|
||||
|
||||
def __repr__(self) -> str:
|
||||
if self._locations is not None:
|
||||
locations = self._locations
|
||||
countries = len({
|
||||
str(loc.get("country_code") or "").upper()
|
||||
for loc in locations if loc.get("country_code")
|
||||
})
|
||||
servers = len(locations)
|
||||
return (
|
||||
f"{countries} Countr{'ies' if countries != 1 else 'y'} "
|
||||
f"({servers} Server{'s' if servers != 1 else ''})"
|
||||
)
|
||||
alias_count = len(self.region_map) + len(self.server_map)
|
||||
if alias_count:
|
||||
return f"{alias_count} Region Alias{'es' if alias_count != 1 else ''} (ExpressVPN HTTPS Proxy)"
|
||||
return "ExpressVPN HTTPS Proxy"
|
||||
|
||||
def get_proxy(self, query: str) -> Optional[str]:
|
||||
query = query.strip().lower()
|
||||
endpoint = self._resolve_endpoint(query)
|
||||
if not endpoint:
|
||||
return None
|
||||
|
||||
connection_token = self._get_connection_token()
|
||||
if not connection_token:
|
||||
log.error("ExpressVPN: connection token was not available")
|
||||
return None
|
||||
|
||||
display = self.last_connection_display()
|
||||
log.debug("ExpressVPN proxy ready: %s", display or f"https://cat:***@{endpoint}:443")
|
||||
return f"https://cat:{connection_token}@{endpoint}:443"
|
||||
|
||||
def last_connection_display(self) -> Optional[str]:
|
||||
"""Return a human-readable string describing the last resolved connection.
|
||||
|
||||
Used by the download/search commands to display a friendly log message
|
||||
instead of a raw sanitized proxy URL.
|
||||
|
||||
Returns:
|
||||
A string like ``"(USA - New York, #3 of 5): .214"`` or None if
|
||||
no connection has been resolved yet.
|
||||
"""
|
||||
if not self._last_location_name or not self._last_endpoint_host:
|
||||
return None
|
||||
|
||||
# Strip the .expressprovider.com suffix for readability
|
||||
short_host = self._last_endpoint_host.replace(".expressprovider.com", "")
|
||||
|
||||
parts = [self._last_location_name]
|
||||
if self._last_server_index is not None and self._last_server_total is not None:
|
||||
parts.append(f"#{self._last_server_index} of {self._last_server_total}")
|
||||
|
||||
return f"({', '.join(parts)}): {short_host}"
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Endpoint Resolution
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _resolve_endpoint(self, query: str) -> Optional[str]:
|
||||
# Reset display info
|
||||
self._last_location_name = None
|
||||
self._last_server_index = None
|
||||
self._last_server_total = None
|
||||
self._last_endpoint_host = None
|
||||
|
||||
# 1) Check legacy server_map first (direct hostname / slug aliases)
|
||||
mapped = self.server_map.get(query)
|
||||
if mapped:
|
||||
query = mapped
|
||||
|
||||
# 2) Direct hostname pass-through
|
||||
if "expressprovider" in query:
|
||||
host = query if query.endswith(".expressprovider.com") else f"{query}.expressprovider.com"
|
||||
self._last_endpoint_host = host
|
||||
return host
|
||||
|
||||
# 3) Parse the query into (country, city, server_num)
|
||||
country, city, server_num = self._parse_query(query)
|
||||
|
||||
# 4) If the query matched a country in region_map and no city was
|
||||
# specified in the CLI, apply the preset from region_map
|
||||
if country and not city:
|
||||
preset = self.region_map.get(country)
|
||||
if preset:
|
||||
# Preset is like "ny-02" or "ny" or "ny2"
|
||||
_, preset_city, preset_server = self._parse_query(f"{country}-{preset}")
|
||||
city = preset_city
|
||||
server_num = preset_server
|
||||
|
||||
# 5) Resolve the location
|
||||
if country:
|
||||
location = self._resolve_location_by_country(country, city)
|
||||
else:
|
||||
# Fallback: try the full query as a location slug
|
||||
location = self._resolve_location_by_slug(query)
|
||||
|
||||
if not location:
|
||||
log.warning("ExpressVPN: no location matched query '%s'", query)
|
||||
return None
|
||||
|
||||
self._last_location_name = location.get("name")
|
||||
|
||||
# 6) Get endpoints and pick server
|
||||
endpoints = self._get_endpoints_for_location(location)
|
||||
if not endpoints:
|
||||
log.error("ExpressVPN: no proxy endpoints returned for %s", location.get("name") or query)
|
||||
return None
|
||||
|
||||
host = self._pick_endpoint(endpoints, server_num)
|
||||
return host
|
||||
|
||||
def _parse_query(self, query: str) -> tuple[Optional[str], Optional[str], Optional[int]]:
|
||||
"""Parse a query string into (country_code, city_query, server_number).
|
||||
|
||||
Supported formats::
|
||||
|
||||
us -> ("us", None, None)
|
||||
us-ny -> ("us", "ny", None)
|
||||
us-ny-2 -> ("us", "ny", 2)
|
||||
us-ny2 -> ("us", "ny", 2)
|
||||
us-ny-02 -> ("us", "ny", 2)
|
||||
usa-new-york -> (None, None, None) -- treated as slug
|
||||
mx -> ("mx", None, None)
|
||||
|
||||
When the query doesn't look like a country[-city] pattern, all
|
||||
fields are returned as None and the caller falls back to slug
|
||||
resolution.
|
||||
"""
|
||||
# Try to extract a trailing server number: "us-ny-2" or "us-ny2"
|
||||
server_num = None
|
||||
base_query = query
|
||||
|
||||
# Match trailing -N or trailing digits glued to alpha
|
||||
num_match = re.match(r"^(.+?)[-]?(\d+)$", query)
|
||||
if num_match:
|
||||
candidate_base = num_match.group(1).rstrip("-")
|
||||
candidate_num = int(num_match.group(2))
|
||||
# Only treat as server number if the base contains alpha characters
|
||||
# and the number is reasonable (1-99)
|
||||
if re.search(r"[a-z]", candidate_base) and 1 <= candidate_num <= 99:
|
||||
base_query = candidate_base
|
||||
server_num = candidate_num
|
||||
|
||||
# Try to split as country-city
|
||||
# Country codes are 2 letters; city is the rest
|
||||
country_match = re.match(r"^([a-z]{2})(?:-(.+))?$", base_query)
|
||||
if country_match:
|
||||
country = country_match.group(1)
|
||||
city = country_match.group(2) # may be None
|
||||
|
||||
# Verify this is a valid country code by checking locations
|
||||
locations = self._get_locations()
|
||||
valid_countries = {str(loc.get("country_code") or "").lower() for loc in locations}
|
||||
if country in valid_countries:
|
||||
return country, city or None, server_num
|
||||
|
||||
# Not a country-city pattern; return None to trigger slug fallback
|
||||
return None, None, None
|
||||
|
||||
def _resolve_location_by_country(
|
||||
self, country_code: str, city_query: Optional[str] = None
|
||||
) -> Optional[dict]:
|
||||
"""Find a location by country code and optional city query.
|
||||
|
||||
If *city_query* is None, a random location in the country is selected
|
||||
(smart connection). Otherwise, the city is matched against location
|
||||
names using abbreviation, prefix, and substring strategies.
|
||||
"""
|
||||
locations = self._get_locations()
|
||||
if not locations:
|
||||
return None
|
||||
|
||||
# Filter locations for this country
|
||||
country_locations = [
|
||||
loc for loc in locations
|
||||
if str(loc.get("country_code") or "").lower() == country_code
|
||||
]
|
||||
if not country_locations:
|
||||
log.warning("ExpressVPN: no locations found for country '%s'", country_code.upper())
|
||||
return None
|
||||
|
||||
if not city_query:
|
||||
# Smart connection: random location in the country
|
||||
return random.choice(country_locations)
|
||||
|
||||
# Match city query against location names
|
||||
return self._match_city(country_locations, city_query)
|
||||
|
||||
def _match_city(self, locations: list[dict], city_query: str) -> Optional[dict]:
|
||||
"""Match a short city query against a list of locations.
|
||||
|
||||
Matching strategies (in priority order):
|
||||
1. First-letter abbreviation: ``"ny"`` matches ``"New York"``
|
||||
2. Exact slug match: ``"miami"`` matches ``"miami"``
|
||||
3. Prefix match on slug: ``"mia"`` matches ``"miami"``
|
||||
4. Substring match on slug: ``"york"`` matches ``"new-york"``
|
||||
|
||||
Returns the first match from the highest-priority non-empty bucket.
|
||||
"""
|
||||
city_query = city_query.strip().lower()
|
||||
abbreviation_hits: list[dict] = []
|
||||
exact_hits: list[dict] = []
|
||||
prefix_hits: list[dict] = []
|
||||
substring_hits: list[dict] = []
|
||||
|
||||
for loc in locations:
|
||||
full_name = str(loc.get("name") or "")
|
||||
# Extract city part: strip country prefix like "USA - "
|
||||
city_part = re.sub(r"^[A-Z]{2,}(?:\s*-\s*)", "", full_name, count=1).strip()
|
||||
if not city_part:
|
||||
city_part = full_name
|
||||
|
||||
city_slug = _slugify(city_part)
|
||||
|
||||
# Strategy 1: first-letter abbreviation
|
||||
words = re.findall(r"[a-zA-Z]+", city_part)
|
||||
abbreviation = "".join(w[0] for w in words).lower() if words else ""
|
||||
if city_query == abbreviation:
|
||||
abbreviation_hits.append(loc)
|
||||
continue
|
||||
|
||||
# Strategy 2: exact slug match
|
||||
if city_query == city_slug:
|
||||
exact_hits.append(loc)
|
||||
continue
|
||||
|
||||
# Strategy 3: prefix match on slug
|
||||
if city_slug.startswith(city_query):
|
||||
prefix_hits.append(loc)
|
||||
continue
|
||||
|
||||
# Strategy 4: substring match on slug
|
||||
if city_query in city_slug:
|
||||
substring_hits.append(loc)
|
||||
|
||||
# Return the first match from the highest-priority non-empty bucket
|
||||
candidates = abbreviation_hits or exact_hits or prefix_hits or substring_hits
|
||||
if not candidates:
|
||||
log.warning("ExpressVPN: no city matched '%s' in available locations", city_query)
|
||||
return None
|
||||
|
||||
if len(candidates) > 1:
|
||||
names = [c.get("name") for c in candidates]
|
||||
log.debug("ExpressVPN: city '%s' matched %d locations: %s", city_query, len(candidates), names)
|
||||
|
||||
return candidates[0]
|
||||
|
||||
def _resolve_location_by_slug(self, query: str) -> Optional[dict]:
|
||||
"""Resolve a location by matching the full query against slugified names, IDs, or country codes."""
|
||||
target = query.strip().lower()
|
||||
locations = self._get_locations()
|
||||
if not locations:
|
||||
return None
|
||||
|
||||
for location in locations:
|
||||
location_id = str(location.get("id") or "").lower()
|
||||
location_name = str(location.get("name") or "").lower()
|
||||
country_code = str(location.get("country_code") or "").lower()
|
||||
slug = _slugify(location_name)
|
||||
if target in (location_id, location_name, country_code, slug):
|
||||
return location
|
||||
return None
|
||||
|
||||
def _pick_endpoint(self, endpoints: list[dict], server_num: Optional[int] = None) -> Optional[str]:
|
||||
"""Select an endpoint host from the list, optionally by position index.
|
||||
|
||||
Args:
|
||||
endpoints: List of endpoint dicts with ``"host"`` keys.
|
||||
server_num: 1-based server position index, or None for random.
|
||||
|
||||
Returns:
|
||||
The selected hostname, or None if no endpoints are available.
|
||||
"""
|
||||
hosts = [str(ep.get("host") or "").lower() for ep in endpoints if ep.get("host")]
|
||||
if not hosts:
|
||||
return None
|
||||
|
||||
total = len(hosts)
|
||||
self._last_server_total = total
|
||||
|
||||
if server_num is not None:
|
||||
if server_num < 1 or server_num > total:
|
||||
log.warning(
|
||||
"ExpressVPN: server #%d is not available (only %d server%s), selecting random",
|
||||
server_num, total, "s" if total != 1 else "",
|
||||
)
|
||||
chosen = random.choice(hosts)
|
||||
self._last_server_index = hosts.index(chosen) + 1
|
||||
else:
|
||||
chosen = hosts[server_num - 1]
|
||||
self._last_server_index = server_num
|
||||
else:
|
||||
chosen = random.choice(hosts)
|
||||
self._last_server_index = hosts.index(chosen) + 1
|
||||
|
||||
self._last_endpoint_host = chosen
|
||||
return chosen
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Locations & Endpoints API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _get_locations(self) -> list[dict]:
|
||||
if self._locations is not None:
|
||||
return self._locations
|
||||
|
||||
srt = self._get_srt()
|
||||
if not srt:
|
||||
return []
|
||||
|
||||
response = self._request(
|
||||
"GET",
|
||||
f"{self.API_BASE}/ids2/locations",
|
||||
headers=self._api_headers(srt),
|
||||
params={"protocols": "proxy"},
|
||||
)
|
||||
self._locations = response.json().get("locations", [])
|
||||
return self._locations
|
||||
|
||||
def _get_endpoints_for_location(self, location: dict) -> list[dict]:
|
||||
srt = self._get_srt()
|
||||
if not srt:
|
||||
return []
|
||||
|
||||
location_id = location.get("id")
|
||||
response = self._request(
|
||||
"POST",
|
||||
f"{self.API_BASE}/ids2/locations/{location_id}/instances",
|
||||
headers=self._api_headers(srt),
|
||||
params={"protocols": "proxy"},
|
||||
json={},
|
||||
)
|
||||
return response.json().get("endpoints", [])
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Token Management
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _get_tokens(self) -> dict:
|
||||
# Return in-memory cache if access_token is still valid
|
||||
if self._tokens:
|
||||
access_token = self._tokens.get("access_token")
|
||||
if access_token and not self._is_jwt_expired(access_token):
|
||||
return self._tokens
|
||||
|
||||
cached = self._load_cached_tokens()
|
||||
tokens = {
|
||||
**cached,
|
||||
**{k: v for k, v in {
|
||||
"access_token": self.access_token,
|
||||
"refresh_token": self.refresh_token,
|
||||
"connection_token": self.connection_token,
|
||||
}.items() if v},
|
||||
}
|
||||
|
||||
access_token = tokens.get("access_token")
|
||||
if access_token and not self._is_jwt_expired(access_token):
|
||||
self._tokens = tokens
|
||||
return tokens
|
||||
|
||||
refresh_token = tokens.get("refresh_token")
|
||||
if refresh_token:
|
||||
refreshed = self._refresh_access_token(refresh_token)
|
||||
if refreshed:
|
||||
tokens.update(
|
||||
{
|
||||
"access_token": refreshed.get("access_token"),
|
||||
"refresh_token": refreshed.get("refresh_token") or refresh_token,
|
||||
}
|
||||
)
|
||||
self._save_cached_tokens(tokens)
|
||||
self._tokens = tokens
|
||||
return tokens
|
||||
log.warning("ExpressVPN: refresh token failed or session expired")
|
||||
|
||||
if self.account_json and self.account_json.is_file():
|
||||
try:
|
||||
data = json.loads(self.account_json.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
log.error("ExpressVPN: failed to read account_json %s: %s", self.account_json, error)
|
||||
else:
|
||||
tokens.update(
|
||||
{
|
||||
"access_token": data.get("accessToken"),
|
||||
"connection_token": data.get("connectionToken"),
|
||||
"subscription_id": data.get("subscriptionId"),
|
||||
}
|
||||
)
|
||||
tokens = {k: v for k, v in tokens.items() if v}
|
||||
self._tokens = tokens
|
||||
return tokens
|
||||
|
||||
if self.cookie_path.is_file():
|
||||
bootstrapped = self._run_pkce_bootstrap()
|
||||
if bootstrapped:
|
||||
tokens.update(
|
||||
{
|
||||
"access_token": bootstrapped.get("access_token"),
|
||||
"refresh_token": bootstrapped.get("refresh_token"),
|
||||
}
|
||||
)
|
||||
self._save_cached_tokens(tokens)
|
||||
self._tokens = tokens
|
||||
return tokens
|
||||
|
||||
tokens = {k: v for k, v in tokens.items() if v}
|
||||
self._tokens = tokens
|
||||
return tokens
|
||||
|
||||
def _refresh_access_token(self, refresh_token: str) -> Optional[dict]:
|
||||
response = self._request(
|
||||
"POST",
|
||||
f"{self.AUTH_BASE}/token",
|
||||
data={
|
||||
"grant_type": "refresh_token",
|
||||
"client_id": self.CLIENT_ID,
|
||||
"refresh_token": refresh_token,
|
||||
},
|
||||
headers=self._form_headers(),
|
||||
allow_error=True,
|
||||
)
|
||||
if not response.ok:
|
||||
log.warning("ExpressVPN: access token refresh failed with HTTP %s", response.status_code)
|
||||
return None
|
||||
return response.json()
|
||||
|
||||
def _run_pkce_bootstrap(self) -> Optional[dict]:
|
||||
cookies = self._load_cookies()
|
||||
keycloak_identity = cookies.get("KEYCLOAK_IDENTITY")
|
||||
keycloak_session = cookies.get("KEYCLOAK_SESSION")
|
||||
auth_session_id = cookies.get("AUTH_SESSION_ID")
|
||||
if not keycloak_identity or not keycloak_session:
|
||||
log.error("ExpressVPN: KEYCLOAK_IDENTITY or KEYCLOAK_SESSION is missing from %s", self.cookie_path)
|
||||
return None
|
||||
|
||||
verifier = base64.urlsafe_b64encode(
|
||||
secrets.token_bytes(32)
|
||||
).decode("utf-8").rstrip("=")
|
||||
challenge = base64.urlsafe_b64encode(
|
||||
hashlib.sha256(verifier.encode("utf-8")).digest()
|
||||
).decode("utf-8").rstrip("=")
|
||||
auth_cookies = requests.cookies.RequestsCookieJar()
|
||||
auth_cookies.set("KEYCLOAK_IDENTITY", keycloak_identity, domain="auth.expressvpn.com")
|
||||
auth_cookies.set("KEYCLOAK_SESSION", keycloak_session, domain="auth.expressvpn.com")
|
||||
if auth_session_id:
|
||||
auth_cookies.set("AUTH_SESSION_ID", auth_session_id, domain="auth.expressvpn.com")
|
||||
|
||||
response = self._request(
|
||||
"GET",
|
||||
f"{self.AUTH_BASE}/auth",
|
||||
params={
|
||||
"client_id": self.CLIENT_ID,
|
||||
"response_type": "code",
|
||||
"redirect_uri": self.EXTENSION_REDIRECT_URI,
|
||||
"scope": "profile offline_access",
|
||||
"code_challenge": challenge,
|
||||
"code_challenge_method": "S256",
|
||||
"ui_locales": "en",
|
||||
},
|
||||
headers=self._browser_headers(),
|
||||
cookies=auth_cookies,
|
||||
allow_redirects=False,
|
||||
allow_error=True,
|
||||
)
|
||||
|
||||
redirect_url = response.headers.get("Location", "")
|
||||
code = parse_qs(urlparse(redirect_url).query).get("code", [None])[0]
|
||||
if not code:
|
||||
log.error("ExpressVPN: OAuth bootstrap did not return an authorization code")
|
||||
return None
|
||||
|
||||
token_response = self._request(
|
||||
"POST",
|
||||
f"{self.AUTH_BASE}/token",
|
||||
data={
|
||||
"grant_type": "authorization_code",
|
||||
"client_id": self.CLIENT_ID,
|
||||
"code": code,
|
||||
"redirect_uri": self.EXTENSION_REDIRECT_URI,
|
||||
"code_verifier": verifier,
|
||||
},
|
||||
headers=self._form_headers(),
|
||||
allow_error=True,
|
||||
)
|
||||
if not token_response.ok:
|
||||
log.error("ExpressVPN: OAuth token exchange failed with HTTP %s", token_response.status_code)
|
||||
return None
|
||||
return token_response.json()
|
||||
|
||||
def _get_srt(self) -> Optional[str]:
|
||||
if self._srt and not self._is_jwt_expired(self._srt):
|
||||
return self._srt
|
||||
|
||||
tokens = self._get_tokens()
|
||||
cached_srt = tokens.get("srt")
|
||||
if cached_srt and not self._is_jwt_expired(cached_srt):
|
||||
self._srt = cached_srt
|
||||
return self._srt
|
||||
|
||||
access_token = tokens.get("access_token")
|
||||
if not access_token:
|
||||
log.error("ExpressVPN: access token was not available")
|
||||
return None
|
||||
|
||||
response = self._request(
|
||||
"POST",
|
||||
f"{self.API_BASE}/srs2/subscription_receipts",
|
||||
headers=self._api_headers(access_token),
|
||||
json={},
|
||||
)
|
||||
receipts = response.json().get("srts", [])
|
||||
active_subscription_id = tokens.get("subscription_id")
|
||||
srt = self._select_srt(receipts, active_subscription_id)
|
||||
if not srt:
|
||||
log.error("ExpressVPN: no active subscription receipt was found")
|
||||
return None
|
||||
|
||||
self._srt = srt
|
||||
tokens["srt"] = srt
|
||||
self._save_cached_tokens(tokens)
|
||||
return srt
|
||||
|
||||
def _get_connection_token(self) -> Optional[str]:
|
||||
tokens = self._get_tokens()
|
||||
connection_token = tokens.get("connection_token")
|
||||
if connection_token and not self._is_jwt_expired(connection_token):
|
||||
return connection_token
|
||||
|
||||
auth_token = self._get_srt() or tokens.get("access_token")
|
||||
if not auth_token:
|
||||
return connection_token
|
||||
|
||||
response = self._request(
|
||||
"POST",
|
||||
f"{self.API_BASE}/srs2/connection_token",
|
||||
headers=self._api_headers(auth_token),
|
||||
json={},
|
||||
)
|
||||
body = response.json()
|
||||
connection_token = body.get("connection_token") or body.get("token")
|
||||
if connection_token:
|
||||
tokens["connection_token"] = connection_token
|
||||
self._save_cached_tokens(tokens)
|
||||
return connection_token
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Cookies & Cache
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _load_cookies(self) -> dict[str, str]:
|
||||
if not self.cookie_path.is_file():
|
||||
return {}
|
||||
|
||||
try:
|
||||
content = self.cookie_path.read_text(encoding="utf-8").strip()
|
||||
except OSError as error:
|
||||
log.error("ExpressVPN: failed to read cookies file %s: %s", self.cookie_path, error)
|
||||
return {}
|
||||
|
||||
cookies: dict[str, str] = {}
|
||||
try:
|
||||
data = json.loads(content)
|
||||
except json.JSONDecodeError:
|
||||
for line in content.splitlines():
|
||||
if not line.strip() or line.startswith("#"):
|
||||
continue
|
||||
parts = line.rstrip("\n").split("\t")
|
||||
if len(parts) >= 7 and "expressvpn.com" in parts[0]:
|
||||
cookies[parts[5]] = parts[6]
|
||||
elif "=" in line:
|
||||
name, value = line.split("=", maxsplit=1)
|
||||
cookies[name.strip()] = value.strip()
|
||||
else:
|
||||
if isinstance(data, list):
|
||||
for cookie in data:
|
||||
if isinstance(cookie, dict) and "expressvpn.com" in str(cookie.get("domain", "")):
|
||||
name = cookie.get("name")
|
||||
value = cookie.get("value")
|
||||
if name and value is not None:
|
||||
cookies[str(name)] = str(value)
|
||||
elif isinstance(data, dict):
|
||||
cookies.update({str(k): str(v) for k, v in data.items()})
|
||||
return cookies
|
||||
|
||||
def _load_cached_tokens(self) -> dict:
|
||||
if not self.cache_path.is_file():
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(self.cache_path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
log.warning("ExpressVPN: failed to read token cache %s: %s", self.cache_path, error)
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
def _save_cached_tokens(self, tokens: dict) -> None:
|
||||
# Invalidate in-memory cache so next _get_tokens() re-reads
|
||||
self._tokens = tokens.copy()
|
||||
try:
|
||||
self.cache_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = json.dumps({k: v for k, v in tokens.items() if v}, indent=2)
|
||||
_write_private(self.cache_path, payload)
|
||||
except OSError as error:
|
||||
log.error("ExpressVPN: failed to save token cache %s: %s", self.cache_path, error)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# HTTP & Helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _request(self, method: str, url: str, allow_error: bool = False, **kwargs) -> requests.Response:
|
||||
kwargs.setdefault("timeout", self.timeout)
|
||||
response = requests.request(method, url, **kwargs)
|
||||
if response.ok or allow_error:
|
||||
return response
|
||||
raise ValueError(f"ExpressVPN request failed with HTTP {response.status_code}: {url}")
|
||||
|
||||
def _api_headers(self, token: str) -> dict[str, str]:
|
||||
return {
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": self.USER_AGENT,
|
||||
"X-Client-App-Version": "12.0.0",
|
||||
"X-Client-OS": "Windows",
|
||||
"X-Client-Device-Model": "Browser Extension",
|
||||
}
|
||||
|
||||
def _browser_headers(self) -> dict[str, str]:
|
||||
return {
|
||||
"User-Agent": self.USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
|
||||
"Accept-Language": "en-US,en;q=0.5",
|
||||
}
|
||||
|
||||
def _form_headers(self) -> dict[str, str]:
|
||||
return {
|
||||
"Content-Type": "application/x-www-form-urlencoded",
|
||||
"Accept": "application/json",
|
||||
"User-Agent": self.USER_AGENT,
|
||||
}
|
||||
|
||||
def _select_srt(self, receipts: list[dict], active_subscription_id: Optional[str]) -> Optional[str]:
|
||||
for receipt in receipts:
|
||||
srt = receipt.get("srt")
|
||||
if "xv.vpn" in self._decode_jwt_payload(srt).get("entitlements", {}):
|
||||
return srt
|
||||
|
||||
if active_subscription_id:
|
||||
for receipt in receipts:
|
||||
if receipt.get("subscription_id") == active_subscription_id:
|
||||
return receipt.get("srt")
|
||||
|
||||
if receipts:
|
||||
return receipts[0].get("srt")
|
||||
return None
|
||||
|
||||
def _decode_jwt_payload(self, token: Optional[str]) -> dict:
|
||||
if not token:
|
||||
return {}
|
||||
try:
|
||||
parts = token.split(".")
|
||||
if len(parts) < 2:
|
||||
return {}
|
||||
payload = parts[1] + "=" * ((4 - len(parts[1]) % 4) % 4)
|
||||
return json.loads(base64.urlsafe_b64decode(payload).decode("utf-8"))
|
||||
except (ValueError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
def _is_jwt_expired(self, token: str) -> bool:
|
||||
expires_at = self._decode_jwt_payload(token).get("exp")
|
||||
if not expires_at:
|
||||
# Tokens without ``exp`` are treated as non-expiring. Keycloak
|
||||
# access/refresh tokens always carry ``exp``; proprietary SRT/
|
||||
# connection tokens may omit it — they are refreshed via the
|
||||
# normal SRT → connection_token flow anyway.
|
||||
return False
|
||||
return time.time() >= (int(expires_at) - 300)
|
||||
|
||||
def _default_cookie_path(self) -> Path:
|
||||
cookies_dir = Path(config.directories.cookies)
|
||||
for folder in ("vpn", "vpns"):
|
||||
candidate = cookies_dir / folder / "expressvpn.txt"
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return cookies_dir / "vpn" / "expressvpn.txt"
|
||||
|
||||
def _default_cache_path(self) -> Path:
|
||||
cache_dir = Path(config.directories.cache)
|
||||
for folder in ("vpn", "vpns"):
|
||||
candidate = cache_dir / "global" / f"{folder}_expressvpn_tokens.json"
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
candidate = cache_dir / "global" / "expressvpn_tokens.json"
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return cache_dir / "global" / "expressvpn_tokens.json"
|
||||
|
||||
def close(self) -> None:
|
||||
pass
|
||||
|
||||
def __enter__(self) -> ExpressVPN:
|
||||
return self
|
||||
|
||||
def __exit__(self, *_: object) -> None:
|
||||
self.close()
|
||||
|
||||
|
||||
def _slugify(value: str) -> str:
|
||||
return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
|
||||
|
||||
|
||||
def _write_private(path: Path, content: str) -> None:
|
||||
"""Write *content* to *path* with owner-only permissions (0600).
|
||||
|
||||
Opens with ``os.open`` so the file is **created** at 0600 — no window
|
||||
where it is world-readable. On Windows ``os.fchmod`` is unavailable,
|
||||
so we fall back to ``path.chmod`` after close.
|
||||
"""
|
||||
if hasattr(os, "fchmod"):
|
||||
fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
os.write(fd, content.encode("utf-8"))
|
||||
finally:
|
||||
os.close(fd)
|
||||
else:
|
||||
path.write_text(content, encoding="utf-8")
|
||||
try:
|
||||
path.chmod(0o600)
|
||||
except OSError:
|
||||
pass
|
||||
@@ -19,6 +19,7 @@ def initialize_proxy_providers() -> List[Any]:
|
||||
from unshackle.core import binaries
|
||||
from unshackle.core.config import config as main_config
|
||||
from unshackle.core.proxies.basic import Basic
|
||||
from unshackle.core.proxies.expressvpn import ExpressVPN
|
||||
from unshackle.core.proxies.hola import Hola
|
||||
from unshackle.core.proxies.nordvpn import NordVPN
|
||||
from unshackle.core.proxies.surfsharkvpn import SurfsharkVPN
|
||||
@@ -27,6 +28,8 @@ def initialize_proxy_providers() -> List[Any]:
|
||||
|
||||
if proxy_config.get("basic"):
|
||||
proxy_providers.append(Basic(**proxy_config["basic"]))
|
||||
if proxy_config.get("expressvpn"):
|
||||
proxy_providers.append(ExpressVPN(**proxy_config["expressvpn"]))
|
||||
if proxy_config.get("nordvpn"):
|
||||
proxy_providers.append(NordVPN(**proxy_config["nordvpn"]))
|
||||
if proxy_config.get("surfsharkvpn"):
|
||||
|
||||
@@ -198,7 +198,7 @@ def _build_tracks(data: Dict[str, Any]) -> Tracks:
|
||||
return tracks
|
||||
|
||||
|
||||
def _resolve_manifest_data(tracks: Tracks, manifests: list, session: Any) -> None:
|
||||
def _resolve_manifest_data(tracks: Tracks, manifests: list) -> None:
|
||||
"""Re-parse serialized manifests and populate track.data for downloading.
|
||||
|
||||
The server serializes DASH and ISM manifest XML as zlib-compressed base64.
|
||||
@@ -604,7 +604,7 @@ class RemoteService:
|
||||
for k, v in result.get("session_cookies", {}).items():
|
||||
self._session.cookies.set(k, v)
|
||||
|
||||
_resolve_manifest_data(tracks, result.get("manifests", []), self._session)
|
||||
_resolve_manifest_data(tracks, result.get("manifests", []))
|
||||
|
||||
self._server_cdm_type = result.get("server_cdm_type", "widevine")
|
||||
|
||||
|
||||
@@ -209,8 +209,7 @@ class UpdateChecker:
|
||||
return None
|
||||
|
||||
try:
|
||||
loop = asyncio.get_event_loop()
|
||||
latest_version = await loop.run_in_executor(None, cls._fetch_latest_version)
|
||||
latest_version = await asyncio.to_thread(cls._fetch_latest_version)
|
||||
|
||||
if latest_version and cls._compare_versions(current_version, latest_version):
|
||||
return latest_version
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
Firefox Cookie and LocalStorage Extractor Utility for unshackle.
|
||||
Provides a secure, read-only mechanism to extract authentication tokens from Firefox.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import platform
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from http.cookiejar import CookieJar, Cookie
|
||||
from typing import Dict, Optional, List
|
||||
|
||||
|
||||
def get_firefox_root() -> Path:
|
||||
"""Locates the operating system specific root directory for Firefox profiles."""
|
||||
system = platform.system()
|
||||
home = Path.home()
|
||||
if system == 'Windows':
|
||||
return Path(os.getenv('APPDATA', '')) / 'Mozilla' / 'Firefox'
|
||||
elif system == 'Darwin':
|
||||
return home / 'Library' / 'Application Support' / 'Firefox'
|
||||
elif system == 'Linux':
|
||||
paths = [
|
||||
home / '.mozilla' / 'firefox',
|
||||
home / 'snap' / 'firefox' / 'common' / '.mozilla' / 'firefox',
|
||||
home / '.var' / 'app' / 'org.mozilla.firefox' / '.mozilla' / 'firefox'
|
||||
]
|
||||
for p in paths:
|
||||
if p.exists():
|
||||
return p
|
||||
return paths[0]
|
||||
raise RuntimeError(f"Unsupported operating system: {system}")
|
||||
|
||||
|
||||
def get_latest_profile_path(ff_root: Path) -> Path:
|
||||
"""Finds the most recently modified valid Firefox profile subdirectory."""
|
||||
search_dirs = [ff_root / 'Profiles', ff_root]
|
||||
valid_profiles = []
|
||||
for base in search_dirs:
|
||||
if base.exists():
|
||||
valid_profiles.extend([p for p in base.iterdir() if p.is_dir() and (p / 'cookies.sqlite').exists()])
|
||||
|
||||
if not valid_profiles:
|
||||
raise FileNotFoundError("No active Firefox profiles detected.")
|
||||
|
||||
return max(valid_profiles, key=lambda p: (p / 'cookies.sqlite').stat().st_mtime)
|
||||
|
||||
|
||||
def get_local_storage_data(profile_path: Path, hosts: List[str], tmp_dir_path: Path) -> Dict[str, str]:
|
||||
"""
|
||||
Extracts LocalStorage tokens from webappsstore.sqlite with strict origin matching.
|
||||
Operates within a secure temporary directory.
|
||||
"""
|
||||
ls_db = profile_path / 'webappsstore.sqlite'
|
||||
if not ls_db.exists():
|
||||
return {}
|
||||
|
||||
temp_ls = tmp_dir_path / "webappsstore.sqlite"
|
||||
extracted_storage = {}
|
||||
|
||||
try:
|
||||
shutil.copy2(ls_db, temp_ls)
|
||||
conn = sqlite3.connect(temp_ls)
|
||||
cursor = conn.cursor()
|
||||
|
||||
for host in hosts:
|
||||
# Firefox stores LocalStorage as reversed host (e.g., moc.viki.:https:443)
|
||||
# Use strict prefix matching to prevent over-broad harvesting
|
||||
reversed_host = host[::-1]
|
||||
query = "SELECT key, value FROM webappsstore2 WHERE originKey LIKE ?"
|
||||
cursor.execute(query, (f"{reversed_host}.%",))
|
||||
|
||||
for key, value in cursor.fetchall():
|
||||
extracted_storage[key] = value
|
||||
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return extracted_storage
|
||||
|
||||
|
||||
def get_firefox_cookies(service_settings: dict) -> Optional[CookieJar]:
|
||||
"""
|
||||
Extracts cookies and optionally localStorage from Firefox based on provided host patterns.
|
||||
Implements security hardening and data consistency (WAL support) as per PR review.
|
||||
"""
|
||||
raw_hosts = service_settings.get('hosts', [])
|
||||
# Filter empty or dangerously short hosts to prevent full store dumps (Finding #3)
|
||||
priority_hosts = [h.strip().lower() for h in raw_hosts if h and len(h.strip()) >= 3]
|
||||
use_local_storage = service_settings.get('local_storage', False)
|
||||
|
||||
if not priority_hosts:
|
||||
return None
|
||||
|
||||
try:
|
||||
profile_path = get_latest_profile_path(get_firefox_root())
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
cookie_jar = CookieJar()
|
||||
|
||||
# Use a secure temporary directory with restricted permissions (Finding #5)
|
||||
with tempfile.TemporaryDirectory(prefix="unshackle_ff_") as tmp_dir:
|
||||
tmp_dir_path = Path(tmp_dir)
|
||||
os.chmod(tmp_dir, 0o700)
|
||||
|
||||
temp_db = tmp_dir_path / "cookies.sqlite"
|
||||
temp_wal = tmp_dir_path / "cookies.sqlite-wal"
|
||||
|
||||
try:
|
||||
# Copy main DB and its WAL log to ensure fresh data capture (Finding #5)
|
||||
shutil.copy2(profile_path / 'cookies.sqlite', temp_db)
|
||||
wal_path = profile_path / 'cookies.sqlite-wal'
|
||||
if wal_path.exists():
|
||||
shutil.copy2(wal_path, temp_wal)
|
||||
|
||||
conn = sqlite3.connect(temp_db)
|
||||
cursor = conn.cursor()
|
||||
|
||||
for host in priority_hosts:
|
||||
# Precise matching: exact host or dot-prefixed domain suffix (Finding #2)
|
||||
query = """
|
||||
SELECT host, name, value, path, expiry, isSecure, isHttpOnly
|
||||
FROM moz_cookies
|
||||
WHERE host = ? OR host LIKE ?
|
||||
"""
|
||||
cursor.execute(query, (host, f"%.{host}"))
|
||||
rows = cursor.fetchall()
|
||||
|
||||
for h, n, v, p, e, secure, httponly in rows:
|
||||
# Final validation to ensure proper domain suffix
|
||||
if h == host or h.endswith(f".{host}"):
|
||||
c = Cookie(
|
||||
version=0, name=n, value=v, port=None, port_specified=False,
|
||||
domain=h, domain_specified=True, domain_initial_dot=h.startswith('.'),
|
||||
path=p, path_specified=True, secure=bool(secure), expires=e,
|
||||
discard=False, comment=None, comment_url=None,
|
||||
rest={'HttpOnly': str(bool(httponly))}, rfc2109=False
|
||||
)
|
||||
cookie_jar.set_cookie(c)
|
||||
conn.close()
|
||||
|
||||
# Optional LocalStorage harvesting (Finding #4)
|
||||
if use_local_storage:
|
||||
storage_data = get_local_storage_data(profile_path, priority_hosts, tmp_dir_path)
|
||||
for key, val in storage_data.items():
|
||||
c = Cookie(
|
||||
version=0, name=key, value=val, port=None, port_specified=False,
|
||||
domain=".localstorage", domain_specified=True, domain_initial_dot=False,
|
||||
path='/', path_specified=True, secure=False, expires=None,
|
||||
discard=True, comment=None, comment_url=None, rest={}, rfc2109=False
|
||||
)
|
||||
cookie_jar.set_cookie(c)
|
||||
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
return cookie_jar if len(list(cookie_jar)) > 0 else None
|
||||
@@ -1,3 +1,4 @@
|
||||
import inspect
|
||||
import logging
|
||||
import time
|
||||
from typing import Any, Iterator, Optional, Union
|
||||
@@ -34,6 +35,9 @@ class Vaults:
|
||||
module = _MODULES.get(type_)
|
||||
if not module:
|
||||
raise ValueError(f"Unable to find vault command by the name '{type_}'.")
|
||||
# only pass the global default to vaults that declare a timeout param (per-vault config still wins)
|
||||
if "timeout" not in kwargs and "timeout" in inspect.signature(module).parameters:
|
||||
kwargs["timeout"] = config.vault_timeout
|
||||
try:
|
||||
vault = module(**kwargs)
|
||||
self.vaults.append(vault)
|
||||
|
||||
@@ -1,815 +1,57 @@
|
||||
# API key for The Movie Database (TMDB)
|
||||
tmdb_api_key: ""
|
||||
|
||||
# Client ID for SIMKL API (optional, improves metadata matching)
|
||||
# Get your free client ID at: https://simkl.com/settings/developer/
|
||||
simkl_client_id: ""
|
||||
|
||||
# Optional ipinfo.io API token. When set, unshackle uses the free Lite endpoint
|
||||
# which has higher rate limits and richer IP info (ASN, org, continent).
|
||||
# Get a free token at: https://ipinfo.io/signup
|
||||
ipinfo_api_key: ""
|
||||
|
||||
# Group or Username to postfix to the end of all download filenames following a dash
|
||||
tag: user_tag
|
||||
|
||||
# Enable/disable tagging with group name (default: true)
|
||||
tag_group_name: true
|
||||
|
||||
# Enable/disable tagging with IMDB/TMDB/TVDB details (default: true)
|
||||
tag_imdb_tmdb: true
|
||||
|
||||
# Set terminal background color (custom option not in CONFIG.md)
|
||||
set_terminal_bg: false
|
||||
|
||||
# Custom output templates for filenames
|
||||
# Configure output_template in your unshackle.yaml to control filename format.
|
||||
# If not configured, default scene-style templates are used and a warning is shown.
|
||||
# Available variables: {title}, {year}, {season}, {episode}, {season_episode}, {episode_name},
|
||||
# {quality}, {resolution}, {source}, {audio}, {audio_channels}, {audio_full},
|
||||
# {video}, {hdr}, {hfr}, {atmos}, {dual}, {multi}, {tag}, {edition}, {repack},
|
||||
# {lang_tag}
|
||||
# Conditional variables (included only if present): Add ? suffix like {year?}, {episode_name?}, {hdr?}
|
||||
# Customize the templates below:
|
||||
# unshackle configuration (copy to unshackle.yaml)
|
||||
#
|
||||
# Example outputs:
|
||||
# Scene movies: 'The.Matrix.1999.1080p.SERVICE.WEB-DL.DDP5.1.H.264-EXAMPLE'
|
||||
# Scene movies (HDR): 'Dune.2021.2160p.SERVICE.WEB-DL.DDP5.1.HDR10.H.265-EXAMPLE'
|
||||
# Scene movies (REPACK): 'Dune.2021.REPACK.2160p.SERVICE.WEB-DL.DDP5.1.H.265-EXAMPLE'
|
||||
# Scene series: 'Breaking.Bad.2008.S01E01.Pilot.1080p.SERVICE.WEB-DL.DDP5.1.H.264-EXAMPLE'
|
||||
# Plex movies: 'The Matrix (1999) 1080p'
|
||||
# Plex series: 'Breaking Bad S01E01 Pilot'
|
||||
# This is a MINIMAL starter. Every option has a sane default, so unshackle runs
|
||||
# even with most of this file empty or removed.
|
||||
#
|
||||
# The full, canonical configuration reference lives in docs/ — start at
|
||||
# docs/README.md. Each section below points to the doc that documents it.
|
||||
|
||||
# Group or username appended to download filenames after a dash.
|
||||
tag: "NOGRP"
|
||||
|
||||
# --- Output filename & folder templates --- docs/OUTPUT_CONFIG.md
|
||||
# Full variable list (35 vars incl. music) and all naming options are
|
||||
# documented in docs/OUTPUT_CONFIG.md. If omitted, scene-style defaults are
|
||||
# used (with a warning). Conditional variables take a ? suffix, e.g. {year?}.
|
||||
output_template:
|
||||
# Scene-style naming (dot-separated)
|
||||
movies: '{title}.{year}.{repack?}.{edition?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}'
|
||||
series: '{title}.{year?}.{season_episode}.{episode_name?}.{repack?}.{edition?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}'
|
||||
songs: '{track_number}.{title}.{repack?}.{edition?}.{source?}.WEB-DL.{audio_full}.{atmos?}-{tag}'
|
||||
movies: "{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
|
||||
series: "{title}.{year?}.{season_episode}.{episode_name?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
|
||||
songs: "{track_number}.{title}.{repack?}.{edition?}.{lang_tag?}.{source?}.WEB-DL.{audio_full}.{atmos?}-{tag}"
|
||||
folder:
|
||||
movies: "{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
|
||||
series: "{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}"
|
||||
albums: "{album_artist} - {album} ({year?})"
|
||||
|
||||
#
|
||||
# Plex-friendly naming (space-separated, clean format)
|
||||
# Plex-friendly alternatives:
|
||||
# movies: '{title} ({year}) {quality}'
|
||||
# series: '{title} {season_episode} {episode_name?}'
|
||||
# songs: '{track_number}. {title}'
|
||||
#
|
||||
# Minimal naming (basic info only)
|
||||
# movies: '{title}.{year}.{quality}'
|
||||
# series: '{title}.{season_episode}.{episode_name?}'
|
||||
#
|
||||
# Custom scene-style with specific elements
|
||||
# movies: '{title}.{year}.{quality}.{hdr?}.{source}.WEB-DL.{audio_full}.{video}-{tag}'
|
||||
# series: '{title}.{year?}.{season_episode}.{episode_name?}.{quality}.{hdr?}.{source}.WEB-DL.{audio_full}.{atmos?}.{video}-{tag}'
|
||||
#
|
||||
# Folder naming (optional). Controls the folder name for downloaded content.
|
||||
# If not configured, series folders are derived from the series template (minus episode info),
|
||||
# movie folders use "{title} ({year})", and song folders use "{artist} - {album} ({year})".
|
||||
# Uses the same template variables as the file templates above.
|
||||
#
|
||||
# Scene-style folder:
|
||||
# folder: '{title}.{year?}.{repack?}.{edition?}.{lang_tag?}.{quality}.{source}.WEB-DL.{dual?}.{multi?}.{audio_full}.{atmos?}.{hdr?}.{hfr?}.{video}-{tag}'
|
||||
#
|
||||
# Plex-friendly folder:
|
||||
# Folder naming (optional). One template for all, or a per-kind map.
|
||||
# Valid kinds: movies, series, songs, albums (album vars: see docs).
|
||||
# folder: '{title} ({year?})'
|
||||
#
|
||||
# Per-title-type folder templates (optional). Override folder naming separately for
|
||||
# movies, series, and albums. Useful when music libraries need artist/album-style folders
|
||||
# while movies/series follow a different scheme. Any kind omitted falls back to the
|
||||
# default for that title type.
|
||||
#
|
||||
# For music: "songs" (above) names each track file, while "albums" (below) names the
|
||||
# album folder the tracks are saved into. Album folder vars: {album_artist}, {album},
|
||||
# {artist}, {year}, {genre}, {label}, {release_type}, {track_total}, {disc_total}.
|
||||
#
|
||||
# folder:
|
||||
# movies: '{title} ({year})'
|
||||
# series: '{title} ({year?})'
|
||||
# albums: '{album_artist} - {album} ({year?})'
|
||||
|
||||
# Language-based tagging for output filenames
|
||||
# Automatically adds language identifiers (e.g., DANiSH, NORDiC, DKsubs) based on
|
||||
# audio and subtitle track languages. Rules are evaluated in order; first match wins.
|
||||
# Use {lang_tag?} in your output_template to place the tag in the filename.
|
||||
#
|
||||
# Conditions (all conditions in a rule must match):
|
||||
# audio: <lang> - any audio track matches this language
|
||||
# subs_contain: <lang> - any subtitle matches this language
|
||||
# subs_contain_all: [lang, ...] - subtitles include ALL listed languages
|
||||
#
|
||||
# language_tags:
|
||||
# rules:
|
||||
# - audio: da
|
||||
# tag: DANiSH
|
||||
# - audio: sv
|
||||
# tag: SWEDiSH
|
||||
# - audio: nb
|
||||
# tag: NORWEGiAN
|
||||
# - audio: en
|
||||
# subs_contain_all: [da, sv, nb]
|
||||
# tag: NORDiC
|
||||
# - audio: en
|
||||
# subs_contain: da
|
||||
# tag: DKsubs
|
||||
|
||||
# Check for updates from GitHub repository on startup (default: true)
|
||||
update_checks: true
|
||||
|
||||
# How often to check for updates, in hours (default: 24)
|
||||
update_check_interval: 24
|
||||
|
||||
# Title caching configuration
|
||||
# Cache title metadata to reduce redundant API calls
|
||||
title_cache_enabled: true # Enable/disable title caching globally (default: true)
|
||||
title_cache_time: 1800 # Cache duration in seconds (default: 1800 = 30 minutes)
|
||||
title_cache_max_retention: 86400 # Maximum cache retention for fallback when API fails (default: 86400 = 24 hours)
|
||||
|
||||
# Filename Configuration
|
||||
unicode_filenames: false # optionally replace non-ASCII characters with ASCII equivalents
|
||||
|
||||
# Debug logging configuration
|
||||
# Comprehensive JSON-based debug logging for troubleshooting and service development
|
||||
debug:
|
||||
false # Enable structured JSON debug logging (default: false)
|
||||
# When enabled with --debug flag or set to true:
|
||||
# - Creates JSON Lines (.jsonl) log files with complete debugging context
|
||||
# - Logs: session info, CLI params, service config, CDM details, authentication,
|
||||
# titles, tracks metadata, DRM operations, vault queries, errors with stack traces
|
||||
# - File location: logs/unshackle_debug_{service}_{timestamp}.jsonl
|
||||
# - Also creates text log: logs/unshackle_root_{timestamp}.log
|
||||
|
||||
debug_keys:
|
||||
false # Log decryption keys in debug logs (default: false)
|
||||
# Set to true to include actual decryption keys in logs
|
||||
# Useful for debugging key retrieval and decryption issues
|
||||
# SECURITY NOTE: Passwords, tokens, cookies, and session tokens
|
||||
# are ALWAYS redacted regardless of this setting
|
||||
|
||||
debug_requests:
|
||||
false # Show per-request HTTP library logs under --debug (default: false)
|
||||
# When false, --debug mutes the noisy per-segment connection logs from the
|
||||
# HTTP libraries (urllib3/requests/rnet, e.g. 'GET ...m4s 200'), keeping
|
||||
# unshackle's own DEBUG output readable.
|
||||
# Set to true to see every individual request/segment download.
|
||||
# Only affects: content_key, key fields (the actual CEKs)
|
||||
# Never affects: kid, keys_count, key_id (metadata is always logged)
|
||||
|
||||
# Muxing configuration
|
||||
muxing:
|
||||
set_title: false
|
||||
# merge_audio: Merge all audio tracks into each output file
|
||||
# true (default): All selected audio in one MKV per quality
|
||||
# false: Separate MKV per (quality, audio_codec) combination
|
||||
# Example: Title.1080p.AAC.mkv, Title.1080p.EC3.mkv
|
||||
merge_audio: true
|
||||
# default_language: Override which track is flagged as the default in the muxed MKV.
|
||||
# audio: BCP-47 tag of the preferred default audio track (e.g. pl, en, pt-BR).
|
||||
# Wins over the title's original_language. Falls back to is_original_lang
|
||||
# if no matching track is present.
|
||||
# video: BCP-47 tag of the preferred default video track. Falls back to the
|
||||
# original-language / first-track rule if no match is found.
|
||||
# subtitle: BCP-47 tag of the preferred default subtitle track. Falls back to
|
||||
# the existing rule (forced sub matching the audio language) if no
|
||||
# matching subtitle is present.
|
||||
# default_language:
|
||||
# audio: pl
|
||||
# video: pl
|
||||
# subtitle: pl
|
||||
|
||||
# Login credentials for each Service
|
||||
credentials:
|
||||
# Direct credentials (no profile support)
|
||||
EXAMPLE: email@example.com:password
|
||||
|
||||
# Per-profile credentials with default fallback
|
||||
SERVICE_NAME:
|
||||
default: default@email.com:password # Used when no -p/--profile is specified
|
||||
profile1: user1@email.com:password1
|
||||
profile2: user2@email.com:password2
|
||||
|
||||
# Per-profile credentials without default (requires -p/--profile)
|
||||
SERVICE_NAME2:
|
||||
john: john@example.com:johnspassword
|
||||
jane: jane@example.com:janespassword
|
||||
|
||||
# You can also use list format for passwords with special characters
|
||||
SERVICE_NAME3:
|
||||
default: ["user@email.com", ":PasswordWith:Colons"]
|
||||
|
||||
# Override default directories used across unshackle
|
||||
directories:
|
||||
cache: Cache
|
||||
cookies: Cookies
|
||||
dcsl: DCSL # Device Certificate Status List
|
||||
downloads: Downloads
|
||||
logs: Logs
|
||||
temp: Temp
|
||||
wvds: WVDs
|
||||
prds: PRDs
|
||||
exports: Exports # JSON export output from --export flag
|
||||
# Additional directories that can be configured:
|
||||
# commands: Commands
|
||||
services:
|
||||
- /path/to/services
|
||||
- /other/path/to/services
|
||||
# vaults: Vaults
|
||||
# fonts: Fonts
|
||||
|
||||
# Pre-define which Widevine or PlayReady device to use for each Service
|
||||
# --- DRM / CDM --- docs/DRM_CONFIG.md
|
||||
# Which Widevine (.wvd) or PlayReady (.prd) device to use. 'default' is the
|
||||
# global fallback. Per-service, per-profile, quality-based selection and remote
|
||||
# CDMs are all documented in docs/DRM_CONFIG.md.
|
||||
cdm:
|
||||
# Global default CDM device (fallback for all services/profiles)
|
||||
default: WVD_1
|
||||
|
||||
# Direct service-specific CDM
|
||||
DIFFERENT_EXAMPLE: PRD_1
|
||||
|
||||
# Per-profile CDM configuration
|
||||
EXAMPLE:
|
||||
john_sd: chromecdm_903_l3 # Profile 'john_sd' uses Chrome CDM L3
|
||||
jane_uhd: nexus_5_l1 # Profile 'jane_uhd' uses Nexus 5 L1
|
||||
default: generic_android_l3 # Default CDM for this service
|
||||
|
||||
# NEW: Quality-based CDM selection
|
||||
# Use different CDMs based on video resolution
|
||||
# Supports operators: >=, >, <=, <, or exact match
|
||||
EXAMPLE_QUALITY:
|
||||
"<=1080": generic_android_l3 # Use L3 for 1080p and below
|
||||
">1080": nexus_5_l1 # Use L1 for above 1080p (1440p, 2160p)
|
||||
default: generic_android_l3 # Optional: fallback if no quality match
|
||||
|
||||
# You can mix profiles and quality thresholds in the same service
|
||||
NETFLIX:
|
||||
# Profile-based selection (existing functionality)
|
||||
john: netflix_l3_profile
|
||||
jane: netflix_l1_profile
|
||||
# Quality-based selection (new functionality)
|
||||
"<=720": netflix_mobile_l3
|
||||
"1080": netflix_standard_l3
|
||||
">=1440": netflix_premium_l1
|
||||
# Fallback
|
||||
default: netflix_standard_l3
|
||||
|
||||
# Use pywidevine Serve-compliant Remote CDMs
|
||||
|
||||
# Example: Custom CDM API Configuration
|
||||
# This demonstrates the highly configurable custom_api type that can adapt to any CDM API format
|
||||
# - name: "chrome"
|
||||
# type: "custom_api"
|
||||
# host: "http://remotecdm.test/"
|
||||
# timeout: 30
|
||||
# device:
|
||||
# name: "ChromeCDM"
|
||||
# type: "CHROME"
|
||||
# system_id: 34312
|
||||
# security_level: 3
|
||||
# auth:
|
||||
# type: "header"
|
||||
# header_name: "x-api-key"
|
||||
# key: "YOUR_API_KEY_HERE"
|
||||
# custom_headers:
|
||||
# User-Agent: "Unshackle/2.0.0"
|
||||
# endpoints:
|
||||
# get_request:
|
||||
# path: "/get-challenge"
|
||||
# method: "POST"
|
||||
# timeout: 30
|
||||
# decrypt_response:
|
||||
# path: "/get-keys"
|
||||
# method: "POST"
|
||||
# timeout: 30
|
||||
# request_mapping:
|
||||
# get_request:
|
||||
# param_names:
|
||||
# scheme: "device"
|
||||
# init_data: "init_data"
|
||||
# static_params:
|
||||
# scheme: "Widevine"
|
||||
# decrypt_response:
|
||||
# param_names:
|
||||
# scheme: "device"
|
||||
# license_request: "license_request"
|
||||
# license_response: "license_response"
|
||||
# static_params:
|
||||
# scheme: "Widevine"
|
||||
# response_mapping:
|
||||
# get_request:
|
||||
# fields:
|
||||
# challenge: "challenge"
|
||||
# session_id: "session_id"
|
||||
# message: "message"
|
||||
# message_type: "message_type"
|
||||
# response_types:
|
||||
# - condition: "message_type == 'license-request'"
|
||||
# type: "license_request"
|
||||
# success_conditions:
|
||||
# - "message == 'success'"
|
||||
# decrypt_response:
|
||||
# fields:
|
||||
# keys: "keys"
|
||||
# message: "message"
|
||||
# key_fields:
|
||||
# kid: "kid"
|
||||
# key: "key"
|
||||
# type: "type"
|
||||
# success_conditions:
|
||||
# - "message == 'success'"
|
||||
# caching:
|
||||
# enabled: true
|
||||
# use_vaults: true
|
||||
# check_cached_first: true
|
||||
|
||||
remote_cdm:
|
||||
- name: "chrome"
|
||||
device_name: chrome
|
||||
device_type: CHROME
|
||||
system_id: 27175
|
||||
security_level: 3
|
||||
host: https://domain.com/api
|
||||
secret: secret_key
|
||||
- name: "chrome-2"
|
||||
device_name: chrome
|
||||
device_type: CHROME
|
||||
system_id: 26830
|
||||
security_level: 3
|
||||
host: https://domain-2.com/api
|
||||
secret: secret_key
|
||||
|
||||
- name: "decrypt_labs_chrome"
|
||||
type: "decrypt_labs" # Required to identify as DecryptLabs CDM
|
||||
device_name: "ChromeCDM" # Scheme identifier - must match exactly
|
||||
device_type: CHROME
|
||||
system_id: 4464 # Doesn't matter
|
||||
security_level: 3
|
||||
host: "https://keyxtractor.decryptlabs.com"
|
||||
secret: "your_decrypt_labs_api_key_here" # Replace with your API key
|
||||
- name: "decrypt_labs_l1"
|
||||
type: "decrypt_labs"
|
||||
device_name: "L1" # Scheme identifier - must match exactly
|
||||
device_type: ANDROID
|
||||
system_id: 4464
|
||||
security_level: 1
|
||||
host: "https://keyxtractor.decryptlabs.com"
|
||||
secret: "your_decrypt_labs_api_key_here"
|
||||
|
||||
- name: "decrypt_labs_l2"
|
||||
type: "decrypt_labs"
|
||||
device_name: "L2" # Scheme identifier - must match exactly
|
||||
device_type: ANDROID
|
||||
system_id: 4464
|
||||
security_level: 2
|
||||
host: "https://keyxtractor.decryptlabs.com"
|
||||
secret: "your_decrypt_labs_api_key_here"
|
||||
|
||||
- name: "decrypt_labs_playready_sl2"
|
||||
type: "decrypt_labs"
|
||||
device_name: "SL2" # Scheme identifier - must match exactly
|
||||
device_type: PLAYREADY
|
||||
system_id: 0
|
||||
security_level: 2000
|
||||
host: "https://keyxtractor.decryptlabs.com"
|
||||
secret: "your_decrypt_labs_api_key_here"
|
||||
|
||||
- name: "decrypt_labs_playready_sl3"
|
||||
type: "decrypt_labs"
|
||||
device_name: "SL3" # Scheme identifier - must match exactly
|
||||
device_type: PLAYREADY
|
||||
system_id: 0
|
||||
security_level: 3000
|
||||
host: "https://keyxtractor.decryptlabs.com"
|
||||
secret: "your_decrypt_labs_api_key_here"
|
||||
|
||||
# PyPlayReady RemoteCdm - connects to an unshackle serve instance
|
||||
- name: "playready_remote"
|
||||
device_name: "my_prd_device" # Device name on the serve instance
|
||||
device_type: PLAYREADY
|
||||
system_id: 0
|
||||
security_level: 3000 # 2000 for SL2000, 3000 for SL3000
|
||||
host: "http://127.0.0.1:8786/playready" # Include /playready path
|
||||
secret: "your-api-secret-key"
|
||||
|
||||
# Key Vaults store your obtained Content Encryption Keys (CEKs)
|
||||
# Use 'no_push: true' to prevent a vault from receiving pushed keys
|
||||
# while still allowing it to provide keys when requested
|
||||
# --- Key vaults --- docs/DRM_CONFIG.md
|
||||
# Where obtained Content Encryption Keys (CEKs) are stored and retrieved.
|
||||
key_vaults:
|
||||
- type: SQLite
|
||||
name: Local
|
||||
path: key_store.db
|
||||
# Additional vault types:
|
||||
# - type: API
|
||||
# name: "Remote Vault"
|
||||
# uri: "https://key-vault.example.com"
|
||||
# token: "secret_token"
|
||||
# no_push: true # This vault will only provide keys, not receive them
|
||||
# - type: MySQL
|
||||
# name: "MySQL Vault"
|
||||
# host: "127.0.0.1"
|
||||
# port: 3306
|
||||
# database: vault
|
||||
# username: user
|
||||
# password: pass
|
||||
# no_push: false # Default behavior - vault both provides and receives keys
|
||||
|
||||
# Choose what software to use to download data
|
||||
downloader: requests
|
||||
# Options: requests
|
||||
# Downloading now uses the unified in-process requests/rnet downloader; the legacy
|
||||
# aria2c, curl_impersonate, and n_m3u8dl_re backends have been removed.
|
||||
|
||||
# rnet TLS impersonation preset (not a downloader). Selects the browser
|
||||
# fingerprint the HTTP session impersonates.
|
||||
curl_impersonate:
|
||||
browser: chrome120
|
||||
|
||||
# Pre-define default options and switches of the dl command
|
||||
# Audio track selection preferences
|
||||
audio:
|
||||
# Codec priority order used as a tiebreaker when multiple audio tracks share the same
|
||||
# bitrate and language. Listed codecs are ranked in the order given; codecs not in the
|
||||
# list keep their bitrate-based ordering and are placed after all listed codecs.
|
||||
# Atmos still trumps codec priority. Valid names: AAC, AC3, EC3, AC4, OPUS, OGG, DTS, ALAC, FLAC.
|
||||
# codec_priority: [FLAC, ALAC, AC4, EC3, DTS, AC3, OPUS, AAC, OGG]
|
||||
|
||||
# Global defaults for the dl command. Any dl option can be set here.
|
||||
# Use the option name with dashes as underscores (--v-lang -> v_lang).
|
||||
# range and list work as-is (range_/list_ also accepted).
|
||||
#
|
||||
# Any of these can also be overridden per service under services.<TAG>.dl
|
||||
# (see the services section below). Precedence, highest first:
|
||||
# CLI arguments > services.<TAG>.dl > this dl section > built-in defaults
|
||||
dl:
|
||||
sub_format: srt
|
||||
downloads: 4
|
||||
workers: 16
|
||||
lang:
|
||||
- en
|
||||
- fr
|
||||
v_lang: [en] # --v-lang: video language(s), if different from audio
|
||||
s_lang: [en, de] # --s-lang: subtitle language(s)
|
||||
range: SDR # --range: SDR, HDR10, HDR10+, DV
|
||||
quality: [1080] # --quality: resolution(s), int or list
|
||||
slow: false # --slow: true = random 60-120s delay between titles, or "MIN-MAX" string
|
||||
# Nesting a SERVICE TAG here provides defaults for that service's OWN command
|
||||
# options (e.g. a service-defined --bitrate flag) -- NOT for the dl options
|
||||
# above. To override dl options per service, use services.<TAG>.dl instead.
|
||||
EXAMPLE:
|
||||
bitrate: CBR
|
||||
|
||||
# Chapter Name to use when exporting a Chapter without a Name
|
||||
chapter_fallback_name: "Chapter {j:02}"
|
||||
|
||||
# Case-Insensitive dictionary of headers for all Services
|
||||
headers:
|
||||
Accept-Language: "en-US,en;q=0.8"
|
||||
User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
|
||||
|
||||
# Override default filenames used across unshackle
|
||||
filenames:
|
||||
debug_log: "unshackle_debug_{service}_{time}.jsonl" # JSON Lines debug log file
|
||||
config: "config.yaml"
|
||||
root_config: "unshackle.yaml"
|
||||
chapters: "Chapters_{title}_{random}.txt"
|
||||
subtitle: "Subtitle_{id}_{language}.srt"
|
||||
|
||||
# conversion_method:
|
||||
# - auto (default): Smart routing - subby for WebVTT/SAMI, pycaption for others
|
||||
# - subby: Always use subby with advanced processing
|
||||
# - pycaption: Use only pycaption library (no SubtitleEdit, no subby)
|
||||
# - subtitleedit: Prefer SubtitleEdit when available, fall back to pycaption
|
||||
# - pysubs2: Use pysubs2 library (supports SRT/SSA/ASS/WebVTT/TTML/SAMI/MicroDVD/MPL2/TMP)
|
||||
subtitle:
|
||||
conversion_method: auto
|
||||
# Which backend converts subtitles (data-driven registry, pin-then-fallback)
|
||||
# - auto (default): best available by rank (SubtitleEdit > subby/pysubs2 > pycaption)
|
||||
# - subby | pysubs2 | subtitleedit | pycaption: pin that backend first, still falls back
|
||||
# Styled ASS/SSA are never auto-downconverted to SRT (kept as-is); --sub-format srt overrides.
|
||||
# SubtitleEdit on Linux/macOS = install the SE5 "SeConv" (seconv) CLI on PATH or unshackle/binaries/.
|
||||
|
||||
sdh_method: auto
|
||||
# Method to use for SDH (hearing impaired) stripping
|
||||
# - auto (default): Try subby (SRT only), then SubtitleEdit (if available), then subtitle-filter
|
||||
# - subby: Use subby library (SRT only)
|
||||
# - subtitleedit: Use SubtitleEdit / seconv (SE5 CLI, cross-platform), falls back to subtitle-filter
|
||||
# - filter-subs: Use subtitle-filter library directly
|
||||
|
||||
strip_sdh: true
|
||||
# Automatically create stripped (non-SDH) versions of SDH subtitles
|
||||
# Set to false to disable automatic SDH stripping entirely (default: true)
|
||||
|
||||
convert_before_strip: true
|
||||
# Auto-convert VTT/other formats to SRT before using subtitle-filter
|
||||
# This ensures compatibility when subtitle-filter is used as fallback (default: true)
|
||||
|
||||
preserve_formatting: true
|
||||
# Preserve original subtitle formatting (tags, positioning, styling)
|
||||
# When true, skips pycaption processing for WebVTT files to keep tags like <i>, <b>, positioning intact
|
||||
# Combined with no sub_format setting, ensures subtitles remain in their original format (default: true)
|
||||
|
||||
output_mode: mux
|
||||
# Output mode for subtitles
|
||||
# - mux: Embed subtitles in MKV container only (default)
|
||||
# - sidecar: Save subtitles as separate files only
|
||||
# - both: Embed in MKV AND save as sidecar files
|
||||
|
||||
sidecar_format: srt
|
||||
# Format for sidecar subtitle files
|
||||
# Options: srt, vtt, ass, original (keep current format)
|
||||
|
||||
# Configuration for pywidevine and pyplayready's serve functionality
|
||||
# Also used for remote services (unshackle serve)
|
||||
serve:
|
||||
api_secret: "your-secret-key-here"
|
||||
|
||||
# Compression level for API payloads (manifests, cache, cookies)
|
||||
# 0=off, 1=fast, 6=balanced, 9=max compression (default: 1)
|
||||
compression_level: 1
|
||||
|
||||
# Session inactivity timeout in seconds (default: 300 = 5 minutes)
|
||||
# Sessions are automatically deleted after this many seconds of inactivity
|
||||
# Each API request resets the timer
|
||||
session_ttl: 300
|
||||
|
||||
# Maximum concurrent sessions before oldest is evicted (default: 100)
|
||||
max_sessions: 100
|
||||
|
||||
# Global service allowlist (optional)
|
||||
# Only these services will be exposed on the API. If omitted, all services are available.
|
||||
# services:
|
||||
# - SERVICE_TAG_1
|
||||
# - SERVICE_TAG_2
|
||||
|
||||
users:
|
||||
secret_key_for_user:
|
||||
devices: # Widevine devices (WVDs) this user can access
|
||||
- generic_nexus_4464_l3
|
||||
playready_devices: # PlayReady devices (PRDs) this user can access
|
||||
- playready_device_sl3000
|
||||
username: user
|
||||
# Per-user service allowlist (optional)
|
||||
# Restricts this user to only the listed services. If omitted, user can access
|
||||
# all globally-allowed services. Effective access is the intersection of global
|
||||
# and per-user allowlists.
|
||||
# services:
|
||||
# - SERVICE_TAG_1
|
||||
# devices: # Widevine device paths (auto-populated from directories.wvds)
|
||||
# - '/path/to/device.wvd'
|
||||
# playready_devices: # PlayReady device paths (auto-populated from directories.prds)
|
||||
# - '/path/to/device.prd'
|
||||
|
||||
# Per-request override gates (both default OFF = locked to server-side config).
|
||||
# A /api/download job may ask to use a specific server-side CDM device or to
|
||||
# authenticate with client-supplied credentials; both are rejected with 403 unless
|
||||
# explicitly opted in here, so a default deployment can't be driven to use arbitrary
|
||||
# devices or inject credentials.
|
||||
# cdm_overrides: true # true = allow any device; or a list of permitted device names:
|
||||
# cdm_overrides:
|
||||
# - generic_nexus_4464_l3
|
||||
# allow_job_credentials: true # allow a per-request `credential` / `credentials` in a job
|
||||
# # (each distinct credential gets an isolated token cache)
|
||||
|
||||
# Optional: any /api/download flag can be set here as a server-side default.
|
||||
# Per-request body values still win. Useful for raising concurrency without
|
||||
# changing every client call. Full list of accepted keys: see docs/API.md.
|
||||
# downloads: 4 # parallel tracks per download job
|
||||
# workers: 16 # threads per track segment fetch
|
||||
# best_available: true
|
||||
# no_proxy_download: false
|
||||
|
||||
# Remote Services Configuration
|
||||
# Connect to a remote unshackle server (unshackle serve) to use its services
|
||||
# without needing the service code locally. Use with: unshackle dl --remote
|
||||
# If multiple servers are configured, specify which with: --server <name>
|
||||
remote_services:
|
||||
# Server name (used with --server flag if multiple configured)
|
||||
my-server:
|
||||
url: "http://192.168.1.100:8786"
|
||||
api_key: "your-secret-key-here"
|
||||
|
||||
# Server-CDM mode: server handles all DRM licensing using its own CDM devices
|
||||
# When false (default), client uses its own CDM and proxies license requests through the server
|
||||
server_cdm: false
|
||||
|
||||
# Per-service overrides for remote services
|
||||
# Override downloader, decryption tool, or CDM settings per service on the client
|
||||
services:
|
||||
# Example: Override the decryption tool for specific services
|
||||
# EXAMPLE_SERVICE:
|
||||
# decryption: mp4decrypt # Override decryption tool (shaka, mp4decrypt)
|
||||
|
||||
# Example: Multiple servers
|
||||
# us-server:
|
||||
# url: "https://us.example.com:8786"
|
||||
# api_key: "us-api-key"
|
||||
# server_cdm: true
|
||||
# services:
|
||||
# EXAMPLE:
|
||||
# decryption: mp4decrypt
|
||||
# eu-server:
|
||||
# url: "https://eu.example.com:8786"
|
||||
# api_key: "eu-api-key"
|
||||
# server_cdm: false
|
||||
|
||||
# Configuration data for each Service
|
||||
services:
|
||||
# Service-specific configuration goes here
|
||||
# Profile-specific configurations can be nested under service names
|
||||
|
||||
# You can override ANY global configuration option on a per-service basis
|
||||
# This allows fine-tuned control for services with special requirements
|
||||
# Supported overrides: dl, curl_impersonate, subtitle, muxing, headers, etc.
|
||||
|
||||
# Example: Comprehensive service configuration showing all features
|
||||
EXAMPLE:
|
||||
# Standard service config
|
||||
api_key: "service_api_key"
|
||||
|
||||
# Service certificate for Widevine L1/L2 (base64 encoded)
|
||||
# This certificate is automatically used when L1/L2 schemes are selected
|
||||
# Services obtain this from their DRM provider or license server
|
||||
certificate: |
|
||||
CAUSwwUKvQIIAxIQ5US6QAvBDzfTtjb4tU/7QxiH8c+TBSKOAjCCAQoCggEBAObzvlu2hZRsapAPx4Aa4GUZj4/GjxgXUtBH4THSkM40x63wQeyVxlEEo
|
||||
# ... (full base64 certificate here)
|
||||
|
||||
# Profile-specific device configurations
|
||||
profiles:
|
||||
john_sd:
|
||||
device:
|
||||
app_name: "AIV"
|
||||
device_model: "SHIELD Android TV"
|
||||
jane_uhd:
|
||||
device:
|
||||
app_name: "AIV"
|
||||
device_model: "Fire TV Stick 4K"
|
||||
|
||||
# Service-specific proxy mappings
|
||||
# Override global proxy selection with specific servers for this service
|
||||
# When --proxy matches a key in proxy_map, the mapped server will be used
|
||||
# instead of the default/random server selection
|
||||
proxy_map:
|
||||
nordvpn:ca: ca1577 # Use ca1577 when --proxy nordvpn:ca is specified
|
||||
nordvpn:us: us9842 # Use us9842 when --proxy nordvpn:us is specified
|
||||
us: 123 # Use server 123 (from any provider) when --proxy us is specified
|
||||
gb: 456 # Use server 456 (from any provider) when --proxy gb is specified
|
||||
# Without this service, --proxy nordvpn:ca picks a random CA server
|
||||
# With this config, --proxy nordvpn:ca EXAMPLE uses ca1577 specifically
|
||||
# Other services or no service specified will still use random selection
|
||||
|
||||
# NEW: Configuration overrides (can be combined with profiles and certificates)
|
||||
# Override dl command defaults for this service. ANY dl option works here,
|
||||
# same key naming as the global dl section (v_lang, range_, list_, ...).
|
||||
# These beat the global dl section; explicit CLI arguments beat both.
|
||||
dl:
|
||||
downloads: 4 # Limit concurrent track downloads (global default: 6)
|
||||
workers: 8 # Reduce workers per track (global default: 16)
|
||||
lang: ["en", "es-419"] # Different language priority for this service
|
||||
v_lang: [en] # Video language differs from audio on this service
|
||||
s_lang: [all] # Grab every subtitle language here
|
||||
range: HDR10 # Prefer HDR10 on this service (CLI -r still wins)
|
||||
quality: 720 # This service rate-limits 1080p; int or list both work
|
||||
sub_format: srt # Force SRT subtitle format
|
||||
slow: true # Pace title downloads (60-120s); or "20-40" for a custom range
|
||||
repack: false # Even if the global dl section sets repack: true
|
||||
|
||||
# Override subtitle processing for this service
|
||||
subtitle:
|
||||
conversion_method: pycaption # Use specific subtitle converter
|
||||
sdh_method: auto
|
||||
|
||||
# Service-specific headers
|
||||
headers:
|
||||
User-Agent: "Service-specific user agent string"
|
||||
Accept-Language: "en-US,en;q=0.9"
|
||||
|
||||
# Override muxing options
|
||||
muxing:
|
||||
set_title: true
|
||||
|
||||
# Remap service-provided titles before naming/output
|
||||
# Keyed by the exact title the service returns -> desired output title.
|
||||
title_map:
|
||||
Service Title: Desired Title
|
||||
|
||||
# Example: Service with different regions per profile
|
||||
SERVICE_NAME:
|
||||
profiles:
|
||||
us_account:
|
||||
region: "US"
|
||||
api_endpoint: "https://api.us.service.com"
|
||||
uk_account:
|
||||
region: "GB"
|
||||
api_endpoint: "https://api.uk.service.com"
|
||||
|
||||
# Example: Rate-limited service
|
||||
RATE_LIMITED_SERVICE:
|
||||
dl:
|
||||
downloads: 2 # Limit concurrent downloads
|
||||
workers: 4 # Reduce workers to avoid rate limits
|
||||
|
||||
# Notes on service-specific overrides:
|
||||
# - Overrides are merged with global config, not replaced
|
||||
# - Only specified keys are overridden, others use global defaults
|
||||
# - Reserved keys (profiles, api_key, certificate, etc.) are NOT treated as overrides
|
||||
# - Any dict-type config option can be overridden (dl, subtitle, muxing, headers, etc.)
|
||||
# - Any dl option can be overridden; dashes become underscores (--v-lang -> v_lang)
|
||||
# - Precedence: CLI arguments > service dl override > global dl section > built-in defaults
|
||||
|
||||
# External proxy provider services
|
||||
proxy_providers:
|
||||
nordvpn:
|
||||
username: username_from_service_credentials
|
||||
password: password_from_service_credentials
|
||||
# server_map: global mapping that applies to ALL services
|
||||
# Difference from service-specific proxy_map:
|
||||
# - server_map: applies to ALL services when --proxy nordvpn:us is used
|
||||
# - proxy_map: only applies to the specific service configured (see services: EXAMPLE: proxy_map above)
|
||||
# - proxy_map takes precedence over server_map for that service
|
||||
server_map:
|
||||
us: 12 # force US server #12 for US proxies
|
||||
ca:calgary: 2534 # force CA server #2534 for Calgary proxies
|
||||
us:seattle: 7890 # force US server #7890 for Seattle proxies
|
||||
surfsharkvpn:
|
||||
username: your_surfshark_service_username # Service credentials from https://my.surfshark.com/vpn/manual-setup/main/openvpn
|
||||
password: your_surfshark_service_password # Service credentials (not your login password)
|
||||
server_map:
|
||||
us: 3844 # force US server #3844 for US proxies
|
||||
gb: 2697 # force GB server #2697 for GB proxies
|
||||
au: 4621 # force AU server #4621 for AU proxies
|
||||
us:seattle: 5678 # force US server #5678 for Seattle proxies
|
||||
ca:toronto: 1234 # force CA server #1234 for Toronto proxies
|
||||
windscribevpn:
|
||||
username: your_windscribe_username # Service credentials from https://windscribe.com/getconfig/openvpn
|
||||
password: your_windscribe_password # Service credentials (not your login password)
|
||||
server_map:
|
||||
us: "us-central-096.totallyacdn.com" # force US server
|
||||
gb: "uk-london-055.totallyacdn.com" # force GB server
|
||||
us:seattle: "us-west-011.totallyacdn.com" # force US Seattle server
|
||||
ca:toronto: "ca-toronto-012.totallyacdn.com" # force CA Toronto server
|
||||
|
||||
# Gluetun: Dynamic Docker-based VPN proxy (supports 50+ VPN providers)
|
||||
# Creates Docker containers running Gluetun to bridge VPN connections to HTTP proxies
|
||||
# Requires Docker to be installed and running
|
||||
# Usage: --proxy gluetun:windscribe:us or --proxy gluetun:nordvpn:de
|
||||
gluetun:
|
||||
# Global settings
|
||||
base_port: 8888 # Starting port for HTTP proxies (increments for each container)
|
||||
auto_cleanup: true # Automatically remove containers when done
|
||||
container_prefix: "unshackle-gluetun" # Docker container name prefix
|
||||
verify_ip: true # Verify VPN IP matches expected region
|
||||
# Optional HTTP proxy authentication (for the proxy itself, not VPN)
|
||||
# auth_user: proxy_user
|
||||
# auth_password: proxy_password
|
||||
|
||||
# VPN provider configurations
|
||||
providers:
|
||||
# Windscribe (WireGuard) - Get credentials from https://windscribe.com/getconfig/wireguard
|
||||
windscribe:
|
||||
vpn_type: wireguard
|
||||
credentials:
|
||||
private_key: "YOUR_WIREGUARD_PRIVATE_KEY"
|
||||
addresses: "YOUR_WIREGUARD_ADDRESS" # e.g., "10.x.x.x/32"
|
||||
# Map friendly names to country codes
|
||||
server_countries:
|
||||
us: US
|
||||
uk: GB
|
||||
ca: CA
|
||||
de: DE
|
||||
|
||||
# NordVPN (OpenVPN) - Get service credentials from https://my.nordaccount.com/dashboard/nordvpn/manual-configuration/
|
||||
# Note: Service credentials are NOT your email+password - generate them from the link above
|
||||
# nordvpn:
|
||||
# vpn_type: openvpn
|
||||
# credentials:
|
||||
# username: "YOUR_NORDVPN_SERVICE_USERNAME"
|
||||
# password: "YOUR_NORDVPN_SERVICE_PASSWORD"
|
||||
# server_countries:
|
||||
# us: US
|
||||
# uk: GB
|
||||
|
||||
# ExpressVPN (OpenVPN) - Get credentials from ExpressVPN setup page
|
||||
# expressvpn:
|
||||
# vpn_type: openvpn
|
||||
# credentials:
|
||||
# username: "YOUR_EXPRESSVPN_USERNAME"
|
||||
# password: "YOUR_EXPRESSVPN_PASSWORD"
|
||||
# server_countries:
|
||||
# us: US
|
||||
# uk: GB
|
||||
|
||||
# Surfshark (WireGuard) - Get credentials from https://my.surfshark.com/vpn/manual-setup/main/wireguard
|
||||
# surfshark:
|
||||
# vpn_type: wireguard
|
||||
# credentials:
|
||||
# private_key: "YOUR_SURFSHARK_PRIVATE_KEY"
|
||||
# addresses: "YOUR_SURFSHARK_ADDRESS"
|
||||
# server_countries:
|
||||
# us: US
|
||||
# uk: GB
|
||||
|
||||
# Specific server selection: Use format like "us1239" to select specific servers
|
||||
# Example: --proxy gluetun:nordvpn:us1239 connects to us1239.nordvpn.com
|
||||
# Supported providers: nordvpn, surfshark, expressvpn, cyberghost
|
||||
|
||||
basic:
|
||||
GB:
|
||||
- "socks5://username:password@bhx.socks.ipvanish.com:1080" # 1 (Birmingham)
|
||||
- "socks5://username:password@gla.socks.ipvanish.com:1080" # 2 (Glasgow)
|
||||
AU:
|
||||
- "socks5://username:password@syd.socks.ipvanish.com:1080" # 1 (Sydney)
|
||||
- "https://username:password@au-syd.prod.surfshark.com" # 2 (Sydney)
|
||||
- "https://username:password@au-bne.prod.surfshark.com" # 3 (Brisbane)
|
||||
BG: "https://username:password@bg-sof.prod.surfshark.com"
|
||||
# --- Everything else lives in docs/ ---
|
||||
# credentials, directories, proxy providers, subtitles, muxing, serve/REST API,
|
||||
# remote services, per-service overrides, debug logging, Firefox cookie import,
|
||||
# and more. See docs/README.md for the full index.
|
||||
|
||||
+10
-4
@@ -10,9 +10,10 @@ from unshackle.core.vault import Vault
|
||||
class API(Vault):
|
||||
"""Key Vault using a simple RESTful HTTP API call."""
|
||||
|
||||
def __init__(self, name: str, uri: str, token: str, no_push: bool = False):
|
||||
def __init__(self, name: str, uri: str, token: str, no_push: bool = False, timeout: float = 10.0):
|
||||
super().__init__(name, no_push)
|
||||
self.uri = uri.rstrip("/")
|
||||
self.timeout = timeout
|
||||
self.session = Session()
|
||||
self.session.headers.update({"User-Agent": f"unshackle v{__version__}"})
|
||||
self.session.headers.update({"Authorization": f"Bearer {token}"})
|
||||
@@ -22,7 +23,7 @@ class API(Vault):
|
||||
kid = kid.hex
|
||||
|
||||
data = self.session.get(
|
||||
url=f"{self.uri}/{service.lower()}/{kid}", headers={"Accept": "application/json"}
|
||||
url=f"{self.uri}/{service.lower()}/{kid}", headers={"Accept": "application/json"}, timeout=self.timeout
|
||||
).json()
|
||||
|
||||
code = int(data.get("code", 0))
|
||||
@@ -55,6 +56,7 @@ class API(Vault):
|
||||
url=f"{self.uri}/{service.lower()}",
|
||||
params={"page": page, "total": 10},
|
||||
headers={"Accept": "application/json"},
|
||||
timeout=self.timeout,
|
||||
).json()
|
||||
|
||||
code = int(data.get("code", 0))
|
||||
@@ -89,7 +91,10 @@ class API(Vault):
|
||||
kid = kid.hex
|
||||
|
||||
data = self.session.post(
|
||||
url=f"{self.uri}/{service.lower()}/{kid}", json={"content_key": key}, headers={"Accept": "application/json"}
|
||||
url=f"{self.uri}/{service.lower()}/{kid}",
|
||||
json={"content_key": key},
|
||||
headers={"Accept": "application/json"},
|
||||
timeout=self.timeout,
|
||||
).json()
|
||||
|
||||
code = int(data.get("code", 0))
|
||||
@@ -135,6 +140,7 @@ class API(Vault):
|
||||
url=f"{self.uri}/{service.lower()}",
|
||||
json={"content_keys": batch_keys},
|
||||
headers={"Accept": "application/json"},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
# Check for HTTP errors that suggest batch is too large
|
||||
@@ -181,7 +187,7 @@ class API(Vault):
|
||||
return total_added
|
||||
|
||||
def get_services(self) -> Iterator[str]:
|
||||
data = self.session.post(url=self.uri, headers={"Accept": "application/json"}).json()
|
||||
data = self.session.post(url=self.uri, headers={"Accept": "application/json"}, timeout=self.timeout).json()
|
||||
|
||||
code = int(data.get("code", 0))
|
||||
message = data.get("message")
|
||||
|
||||
@@ -34,6 +34,7 @@ class HTTP(Vault):
|
||||
username: Optional[str] = None,
|
||||
api_mode: str = "query",
|
||||
no_push: bool = False,
|
||||
timeout: float = 10.0,
|
||||
):
|
||||
"""
|
||||
Initialize HTTP Vault.
|
||||
@@ -49,6 +50,7 @@ class HTTP(Vault):
|
||||
"""
|
||||
super().__init__(name, no_push)
|
||||
self.url = host
|
||||
self.timeout = timeout
|
||||
self.password = api_key or password
|
||||
if not self.password:
|
||||
raise ValueError("Either password or api_key is required")
|
||||
@@ -84,7 +86,7 @@ class HTTP(Vault):
|
||||
"token": self.password,
|
||||
}
|
||||
|
||||
r = self.session.post(self.url, json=request_payload)
|
||||
r = self.session.post(self.url, json=request_payload, timeout=self.timeout)
|
||||
|
||||
if r.status_code == 404:
|
||||
return {"status": "not_found"}
|
||||
@@ -115,7 +117,7 @@ class HTTP(Vault):
|
||||
try:
|
||||
request_payload = {"service": service.lower(), "kid": kid}
|
||||
|
||||
response = self.session.post(self.url, json=request_payload)
|
||||
response = self.session.post(self.url, json=request_payload, timeout=self.timeout)
|
||||
|
||||
if not response.ok:
|
||||
return None
|
||||
@@ -182,6 +184,7 @@ class HTTP(Vault):
|
||||
response = self.session.get(
|
||||
self.url,
|
||||
params={"service": service.lower(), "username": self.username, "password": self.password, "kid": kid},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
@@ -200,7 +203,9 @@ class HTTP(Vault):
|
||||
return iter([])
|
||||
else: # query mode
|
||||
response = self.session.get(
|
||||
self.url, params={"service": service.lower(), "username": self.username, "password": self.password}
|
||||
self.url,
|
||||
params={"service": service.lower(), "username": self.username, "password": self.password},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
@@ -250,6 +255,7 @@ class HTTP(Vault):
|
||||
"key": key,
|
||||
"title": title,
|
||||
},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
@@ -301,6 +307,7 @@ class HTTP(Vault):
|
||||
"key": key,
|
||||
"title": title,
|
||||
},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
@@ -323,7 +330,9 @@ class HTTP(Vault):
|
||||
return iter([])
|
||||
else: # query mode
|
||||
response = self.session.get(
|
||||
self.url, params={"username": self.username, "password": self.password, "list_services": True}
|
||||
self.url,
|
||||
params={"username": self.username, "password": self.password, "list_services": True},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
@@ -395,6 +404,7 @@ class HTTP(Vault):
|
||||
"key": key,
|
||||
"title": title,
|
||||
},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user