diff --git a/docs/API.md b/docs/API.md index 34b8cdd..f652fed 100644 --- a/docs/API.md +++ b/docs/API.md @@ -72,7 +72,7 @@ Search for titles from a streaming service. **Required parameters:** | Parameter | Type | Description | | --- | --- | --- | -| `service` | string | Service tag (e.g., `NF`, `AMZN`, `ATV`) | +| `service` | string | Service tag | | `query` | string | Search query | **Optional parameters:** @@ -85,18 +85,18 @@ Search for titles from a streaming service. ```bash curl -X POST http://localhost:8786/api/search \ -H "Content-Type: application/json" \ - -d '{"service": "ATV", "query": "hijack"}' + -d '{"service": "EXAMPLE", "query": "example show"}' ``` ```json { "results": [ { - "id": "umc.cmc.1dg08zn0g3zx52hs8npoj5qe3", - "title": "Hijack", + "id": "abc123def456", + "title": "Example Show", "description": null, "label": "TV Show", - "url": "https://tv.apple.com/us/show/hijack/umc.cmc.1dg08zn0g3zx52hs8npoj5qe3" + "url": "https://example.com/show/abc123def456" } ], "count": 1 @@ -118,7 +118,7 @@ Get available titles (seasons/episodes/movies) for a service and title ID. ```bash curl -X POST http://localhost:8786/api/list-titles \ -H "Content-Type: application/json" \ - -d '{"service": "ATV", "title_id": "umc.cmc.1dg08zn0g3zx52hs8npoj5qe3"}' + -d '{"service": "EXAMPLE", "title_id": "abc123def456"}' ``` ```json @@ -126,12 +126,12 @@ curl -X POST http://localhost:8786/api/list-titles \ "titles": [ { "type": "episode", - "name": "Final Call", - "series_title": "Hijack", + "name": "Pilot", + "series_title": "Example Show", "season": 1, "number": 1, - "year": 2023, - "id": "umc.cmc.4levibvvz01hl4zsm0jdk5v2p" + "year": 2024, + "id": "abc123def789" } ] } @@ -161,8 +161,8 @@ Get video, audio, and subtitle tracks for a title. curl -X POST http://localhost:8786/api/list-tracks \ -H "Content-Type: application/json" \ -d '{ - "service": "ATV", - "title_id": "umc.cmc.1dg08zn0g3zx52hs8npoj5qe3", + "service": "EXAMPLE", + "title_id": "abc123def456", "wanted": ["S01E01"] }' ``` @@ -261,8 +261,8 @@ Start a download job. Returns immediately with a job ID (HTTP 202). curl -X POST http://localhost:8786/api/download \ -H "Content-Type: application/json" \ -d '{ - "service": "ATV", - "title_id": "umc.cmc.1dg08zn0g3zx52hs8npoj5qe3", + "service": "EXAMPLE", + "title_id": "abc123def456", "wanted": ["S01E01"], "quality": [1080, 2160], "vcodec": ["H265"], @@ -303,7 +303,7 @@ curl http://localhost:8786/api/download/jobs curl "http://localhost:8786/api/download/jobs?status=completed" # Filter by service -curl "http://localhost:8786/api/download/jobs?service=ATV" +curl "http://localhost:8786/api/download/jobs?service=EXAMPLE" ``` --- @@ -321,8 +321,8 @@ curl http://localhost:8786/api/download/jobs/504db959-80b0-446c-a764-7924b761d61 "job_id": "504db959-80b0-446c-a764-7924b761d613", "status": "completed", "created_time": "2026-02-27T18:00:00.000000", - "service": "ATV", - "title_id": "umc.cmc.1dg08zn0g3zx52hs8npoj5qe3", + "service": "EXAMPLE", + "title_id": "abc123def456", "progress": 100.0, "parameters": { ... }, "started_time": "2026-02-27T18:00:01.000000", diff --git a/docs/DOWNLOAD_CONFIG.md b/docs/DOWNLOAD_CONFIG.md index c75d6d9..3c6cd07 100644 --- a/docs/DOWNLOAD_CONFIG.md +++ b/docs/DOWNLOAD_CONFIG.md @@ -62,9 +62,9 @@ Example mapping: ```yaml downloader: - NF: requests - AMZN: n_m3u8dl_re - DSNP: n_m3u8dl_re + EXAMPLE: requests + EXAMPLE2: n_m3u8dl_re + EXAMPLE3: n_m3u8dl_re default: requests ``` @@ -131,11 +131,11 @@ downloads: 4 workers: 16 ``` -to set `--bitrate=CVBR` for the AMZN service, +to set `--bitrate=CVBR` for a specific service, ```yaml lang: de -AMZN: +EXAMPLE: bitrate: CVBR ``` @@ -229,9 +229,9 @@ dl: lang: en downloads: 4 workers: 16 - AMZN: + EXAMPLE: bitrate: CVBR - NF: + EXAMPLE2: worst: true quality: 1080 ``` @@ -307,8 +307,8 @@ Example mapping: ```yaml decryption: - ATVP: mp4decrypt - AMZN: shaka + EXAMPLE: mp4decrypt + EXAMPLE2: shaka default: shaka ``` diff --git a/docs/DRM_CONFIG.md b/docs/DRM_CONFIG.md index 4b2b9f6..44ca027 100644 --- a/docs/DRM_CONFIG.md +++ b/docs/DRM_CONFIG.md @@ -12,8 +12,8 @@ for a matching file. For example, ```yaml -AMZN: chromecdm_903_l3 -NF: nexus_6_l1 +EXAMPLE: chromecdm_903_l3 +EXAMPLE2: nexus_6_l1 ``` You may also specify this device based on the profile used. @@ -21,9 +21,9 @@ You may also specify this device based on the profile used. For example, ```yaml -AMZN: chromecdm_903_l3 -NF: nexus_6_l1 -DSNP: +EXAMPLE: chromecdm_903_l3 +EXAMPLE2: nexus_6_l1 +EXAMPLE3: john_sd: chromecdm_903_l3 jane_uhd: nexus_5_l1 ``` @@ -35,8 +35,8 @@ For example, the following has the same result as the previous example, as well services and profiles being pre-defined to use `chromecdm_903_l3`. ```yaml -NF: nexus_6_l1 -DSNP: +EXAMPLE2: nexus_6_l1 +EXAMPLE3: jane_uhd: nexus_5_l1 default: chromecdm_903_l3 ``` @@ -56,12 +56,12 @@ EXAMPLE: You can mix profiles and quality thresholds in the same service: ```yaml -NETFLIX: - john: netflix_l3_profile # Profile-based selection - "<=720": netflix_mobile_l3 # Quality-based selection - "1080": netflix_standard_l3 # Exact match for 1080p - ">=1440": netflix_premium_l1 # Quality-based selection - default: netflix_standard_l3 # Fallback +EXAMPLE: + john: example_l3_profile # Profile-based selection + "<=720": example_mobile_l3 # Quality-based selection + "1080": example_standard_l3 # Exact match for 1080p + ">=1440": example_premium_l1 # Quality-based selection + default: example_standard_l3 # Fallback ``` --- @@ -311,8 +311,8 @@ Or configure per-service with a `DEFAULT` fallback: ```yaml decryption: DEFAULT: shaka - AMZN: mp4decrypt - NF: shaka + EXAMPLE: mp4decrypt + EXAMPLE2: shaka ``` Service keys are case-insensitive (normalized to uppercase internally). diff --git a/docs/OUTPUT_CONFIG.md b/docs/OUTPUT_CONFIG.md index b366816..c53af37 100644 --- a/docs/OUTPUT_CONFIG.md +++ b/docs/OUTPUT_CONFIG.md @@ -56,10 +56,10 @@ output_template: ``` Example outputs: -- Scene movies: `The.Matrix.1999.1080p.NF.WEB-DL.DDP5.1.H.264-EXAMPLE` -- Scene movies (REPACK): `Dune.2021.REPACK.2160p.HBO.WEB-DL.DDP5.1.H.265-EXAMPLE` -- Scene series: `Breaking.Bad.2008.S01E01.Pilot.1080p.NF.WEB-DL.DDP5.1.H.264-EXAMPLE` -- Plex movies: `The Matrix (1999) 1080p` +- Scene movies: `Example.Movie.2024.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG` +- Scene movies (REPACK): `Example.Movie.2024.REPACK.2160p.EXAMPLE.WEB-DL.DDP5.1.H.265-TAG` +- Scene series: `Example.Show.2024.S01E01.Pilot.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG` +- Plex movies: `Example Movie (2024) 1080p` --- @@ -106,10 +106,10 @@ output_template: ``` Example outputs: -- Danish audio: `Show.S01E01.DANiSH.1080p.NF.WEB-DL.DDP5.1.H.264-TAG` -- English audio + multiple Nordic subs: `Show.S01E01.NORDiC.1080p.NF.WEB-DL.DDP5.1.H.264-TAG` -- English audio + Danish subs only: `Show.S01E01.DKsubs.1080p.NF.WEB-DL.DDP5.1.H.264-TAG` -- No matching languages: `Show.S01E01.1080p.NF.WEB-DL.DDP5.1.H.264-TAG` +- Danish audio: `Example.Show.S01E01.DANiSH.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG` +- English audio + multiple Nordic subs: `Example.Show.S01E01.NORDiC.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG` +- English audio + Danish subs only: `Example.Show.S01E01.DKsubs.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG` +- No matching languages: `Example.Show.S01E01.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG` ### Example: Other regional tags diff --git a/docs/REMOTE-SERVICES-FLOW.md b/docs/REMOTE-SERVICES-FLOW.md deleted file mode 100644 index c839af0..0000000 --- a/docs/REMOTE-SERVICES-FLOW.md +++ /dev/null @@ -1,283 +0,0 @@ -# Remote Services — Client ↔ Server Architecture - -The `--remote` flag on the `dl` command connects to a remote unshackle server -(`unshackle serve`) that holds service plugins. The client never has service -code — it sends credentials/cookies, the server authenticates and fetches -titles/tracks, and the client handles downloading, decryption, and muxing locally. - -## How It Works - -The `RemoteService` adapter in `remote_service.py` implements the same interface -as a local `Service`. From dl.py's perspective, it's just another service — the -entire download pipeline runs unchanged. - -``` -unshackle dl --remote [-s server_name] SERVICE_TAG TITLE_ID [options] -``` - -## Session Lifecycle - -```mermaid -sequenceDiagram - participant Client as Client (dl --remote) - participant Server as Server (serve API) - participant Service as Service Plugin - - Note over Client: Load credentials/cookies locally - Note over Client: Detect client region via IP check - - Client->>Server: POST /api/session/create - Note right of Client: {service, title_id, credentials,
cookies, client_region, profile} - - Note over Server: Check if server region matches client
Skip proxy if same region - Server->>Service: authenticate(cookies, credential) - Service-->>Server: Authenticated session - Server-->>Client: {session_id, service} - - Client->>Server: GET /api/session/{id}/titles - Server->>Service: get_titles(title_id) - Server-->>Client: Serialized titles (episodes/movies) - - Note over Client: dl.py filters by --wanted - - Client->>Server: POST /api/session/{id}/tracks - Server->>Service: get_tracks(title) - Server->>Service: get_chapters(title) - Note over Server: Extract manifest XML as base64
Extract service session cookies/headers
Detect server CDM type from track DRM + config - Server-->>Client: Tracks + manifests + chapters +
session cookies + server_cdm_type - - Note over Client: Re-parse manifest locally (DASH/ISM)
HLS tracks download from URL directly - Note over Client: Match tracks by ID to populate track.data - Note over Client: dl.py runs full track selection pipeline - - rect rgb(40, 40, 60) - Note over Client: Licensing (depends on mode) - end - - rect rgb(40, 60, 40) - Note over Client: Download + Post-Processing (all local) - Note over Client: Concurrent downloads via ThreadPoolExecutor - Note over Client: Decrypt (mp4decrypt / Shaka Packager) - Note over Client: FFMPEG repack - Note over Client: Subtitle conversion - Note over Client: Hybrid HDR10+DV injection (dovi_tool) - Note over Client: Per-resolution muxing (mkvmerge) - Note over Client: Output naming from template - Note over Client: Tagging (TMDB/IMDB/TVDB) - end - - Client->>Server: DELETE /api/session/{id} - Server-->>Client: Session cleaned up -``` - ---- - -## Licensing Modes - -### Proxy Mode (`server_cdm: false` — default) - -The client has its own CDM (WVD/PRD file). The server only proxies the license -request through the authenticated service session. - -```mermaid -sequenceDiagram - participant Client as Client - participant Server as Server - participant Service as Service Plugin - participant CDM as Client CDM (local WVD/PRD) - - Note over Client: track.download() discovers DRM
(PSSH from manifest/init data) - - Client->>CDM: Generate challenge from PSSH - CDM-->>Client: Challenge bytes - - Client->>Server: POST /api/session/{id}/license - Note right of Client: {track_id, challenge (base64),
drm_type, pssh} - - Server->>Service: get_widevine_license(challenge, title, track)
or get_playready_license(challenge, title, track) - Service-->>Server: Raw license response bytes - - Server-->>Client: {license: base64_encoded_bytes} - - Client->>CDM: Parse license response - CDM-->>Client: Content keys (KID:KEY) - - Note over Client: Keys stored in local vaults - Note over Client: Decrypt track with keys -``` - -**Key points:** - -- Client must have a local CDM device file (`.wvd` or `.prd`) -- Server never sees decryption keys — only forwards encrypted license blob -- Client parses the license locally to extract keys -- Keys are cached in local vaults for future use - -### Server-CDM Mode (`server_cdm: true`) - -The server handles all CDM operations using its own devices. The client does -not need a local CDM. There are two paths depending on when DRM is discovered: - -#### Path A: Pre-fetch (DASH services with manifest DRM) - -For services where DRM info is in the manifest (ContentProtection elements), -the server resolves keys before downloads start. - -```mermaid -sequenceDiagram - participant Client as Client - participant Server as Server - participant Service as Service Plugin - participant CDM as Server CDM (WVD/PRD) - - Note over Client: dl.py calls service.resolve_server_keys()
before downloads start - - Client->>Server: POST /api/session/{id}/license - Note right of Client: {track_ids: [...], mode: "server_cdm",
drm_type from server_cdm_type} - - loop Per track with DRM - Note over Server: Extract PSSH from track manifest
ContentProtection elements - Server->>CDM: Load device from serve.users config - Server->>CDM: Generate challenge from PSSH - Server->>Service: get_license(challenge, title, track) - Service-->>Server: License response - Server->>CDM: Parse license → extract keys - end - - Server-->>Client: {keys: {track_id: {kid: key, ...}}} - - Note over Client: Inject keys into track.drm.content_keys - Note over Client: prepare_drm finds keys → skips CDM call - Note over Client: Keys stored in local vaults -``` - -#### Path B: On-demand (HLS services / late DRM discovery) - -For services like ATV where DRM is only discovered during download (from init -segments or EXT-X-KEY tags), keys are fetched per-track during download. - -```mermaid -sequenceDiagram - participant Client as Client - participant Server as Server - participant Service as Service Plugin - participant CDM as Server CDM (WVD/PRD) - - Note over Client: track.download() discovers DRM
(PSSH from init data / EXT-X-KEY) - Note over Client: prepare_drm calls licence callback - - Client->>Server: POST /api/session/{id}/license - Note right of Client: {track_id, pssh, drm_type,
mode: "server_cdm"} - - Server->>CDM: Load device (Widevine or PlayReady) - Server->>CDM: Generate challenge from PSSH - Server->>Service: get_license(challenge, title, track) - Service-->>Server: License response - Server->>CDM: Parse license → extract keys - CDM-->>Server: Content keys - - Server-->>Client: {keys: {kid: key, ...}} - - Note over Client: Inject keys into track.drm.content_keys - Note over Client: prepare_drm sees keys → skips re-raise - Note over Client: Keys stored in local vaults -``` - -**Key points:** - -- Client does NOT need a local CDM device file -- Server uses devices from `serve.users.{api_key}.devices` (Widevine) and - `serve.users.{api_key}.playready_devices` (PlayReady) -- Server detects DRM type from actual track DRM objects and available devices -- Keys are returned as `{kid_hex: key_hex}` pairs -- Keys are still cached in client's local vaults (unless `--cdm-only` is used) -- `prepare_drm` skips local CDM if all KIDs already have keys - ---- - -## Configuration - -### Client (`unshackle.yaml`) - -```yaml -remote_services: - my_server: - url: "https://server:8786" - api_key: "your-secret-key" - server_cdm: true # server handles licensing (optional, default: false) - services: # per-service overrides (optional) - EXAMPLE: - downloader: n_m3u8dl_re - decryption: mp4decrypt - EXAMPLE2: - downloader: n_m3u8dl_re -``` - -### Server (`unshackle.yaml`) - -```yaml -serve: - api_secret: "your-secret-key" - users: - "your-secret-key": - username: api_user - devices: # Widevine CDMs - - xiaomi_mi_a1_15.0.0_l3 - playready_devices: # PlayReady CDMs - - qingdao_haier_tv_sl3000 -``` - ---- - -## Manifest Data Transfer - -Track manifests (DASH XML, ISM XML) cannot be JSON-serialized directly (they -contain lxml Element objects). The server serializes them as base64 strings in -the `/tracks` response. The client decodes and re-parses them locally. - -| Manifest | Serialization | Client Re-parse | Notes | -| -------- | --------------------------- | ---------------------------------------------- | --------------------------------- | -| **DASH** | `etree.tostring()` → base64 | `DASH(etree.fromstring(xml), url).to_tracks()` | Match by track ID (crc32 hash) | -| **HLS** | Not needed | Downloads playlist from `track.url` directly | `HLS.download_track()` re-fetches | -| **ISM** | `etree.tostring()` → base64 | `ISM(etree.fromstring(xml), url).to_tracks()` | Match by track ID | - -The `/tracks` response also includes: - -- `session_headers` / `session_cookies` — for CDN authentication -- `server_cdm_type` — "widevine" or "playready" (detected from track DRM + config) - ---- - -## Region & Proxy Handling - -1. Client detects its own country via `get_cached_ip_info()` -2. Sends `client_region` in session create (no IP sent, just country code) -3. Server checks its own region — if it matches, no proxy needed -4. If regions differ, server resolves a proxy from its own providers -5. Client can also send explicit `--proxy` which takes precedence - ---- - -## Comparison: Proxy vs Server-CDM - -| | Proxy Mode (default) | Server-CDM Mode | -| ----------------------- | ----------------------------- | -------------------------------- | -| **Client needs CDM** | Yes (WVD/PRD file) | No | -| **License request** | Client sends challenge bytes | Client sends PSSH (or track IDs) | -| **License response** | Raw license bytes | KID:KEY pairs | -| **Key extraction** | Client CDM parses license | Server CDM parses license | -| **What leaves server** | Encrypted license blob | Decryption keys | -| **Vault caching** | Client caches keys | Client caches keys | -| **`--cdm-only` effect** | Skip vaults, CDM only | Skip vaults, server CDM only | -| **Config** | `server_cdm: false` (default) | `server_cdm: true` | - -## API Endpoints - -| Endpoint | Method | Purpose | -| --------------------------- | ------ | --------------------------------- | -| `/api/session/create` | POST | Authenticate, create session | -| `/api/session/{id}/titles` | GET | Get titles for session | -| `/api/session/{id}/tracks` | POST | Get tracks + manifests + chapters | -| `/api/session/{id}/license` | POST | License proxy or server-CDM keys | -| `/api/session/{id}` | GET | Check session validity | -| `/api/session/{id}` | DELETE | Cleanup session | diff --git a/docs/SERVICE_CONFIG.md b/docs/SERVICE_CONFIG.md index 5711b3d..bf2634f 100644 --- a/docs/SERVICE_CONFIG.md +++ b/docs/SERVICE_CONFIG.md @@ -17,7 +17,7 @@ a dictionary. For example, ```yaml -NOW: +EXAMPLE: client: auth_scheme: MESSO # ... more sensitive data @@ -55,20 +55,20 @@ Specify login credentials to use for each Service, and optionally per-profile. For example, ```yaml -ALL4: jane@gmail.com:LoremIpsum100 # directly -AMZN: # or per-profile, optionally with a default +EXAMPLE: jane@example.tld:LoremIpsum100 # directly +EXAMPLE2: # or per-profile, optionally with a default default: jane@example.tld:LoremIpsum99 # <-- used by default if -p/--profile is not used - james: james@gmail.com:TheFriend97 + james: james@example.tld:TheFriend97 john: john@example.tld:LoremIpsum98 -NF: # the `default` key is not necessary, but no credential will be used by default - john: john@gmail.com:TheGuyWhoPaysForTheNetflix69420 +EXAMPLE3: # the `default` key is not necessary, but no credential will be used by default + john: john@example.tld:SecretPassword123 ``` -The value should be in string form, i.e. `john@gmail.com:password123` or `john:password123`. +The value should be in string form, i.e. `john@example.tld:password123` or `john:password123`. Any arbitrary values can be used on the left (username/password/phone) and right (password/secret). -You can also specify these in list form, i.e., `["john@gmail.com", ":PasswordWithAColon"]`. +You can also specify these in list form, i.e., `["john@example.tld", ":PasswordWithAColon"]`. -If you specify multiple credentials with keys like the `AMZN` and `NF` example above, then you should +If you specify multiple credentials with keys like the `EXAMPLE2` and `EXAMPLE3` example above, then you should use a `default` key or no credential will be loaded automatically unless you use `-p/--profile`. You do not have to use a `default` key at all. diff --git a/unshackle/core/api/routes.py b/unshackle/core/api/routes.py index 430beb3..6dd8301 100644 --- a/unshackle/core/api/routes.py +++ b/unshackle/core/api/routes.py @@ -247,7 +247,7 @@ async def search(request: web.Request) -> web.Response: properties: service: type: string - description: Service tag (e.g., NF, AMZN, ATV) + description: Service tag query: type: string description: Search query string