kenzuya/unshackle
1
0
Fork 2
mirror of https://github.com/unshackle-dl/unshackle.git synced 2026-11-14 21:15:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: restructure documentation into MkDocs Material site

Browse Source
This commit is contained in:
imSp4rky
2026-07-03 19:24:11 -06:00
parent b892e3ce70
commit 1d8cec6d2b
51 changed files with 9398 additions and 4356 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide/cli-reference.md
+550
View File
@@ -0,0 +1,550 @@
# CLI Reference
A complete reference for every `unshackle` command, subcommand, and flag. Commands are grouped by purpose:
- **[`dl`](#dl)**: the download pipeline (the command you use most).
- **[`search`](#search)**: find titles on a service.
- **[`import`](#import)**: rebuild a download from an export file.
- **[`cfg`](#cfg)**, **[`env`](#env)**: manage configuration and the environment.
- **[`kv`](#kv)**: Key Vault operations.
- **[`wvd`](#wvd)**, **[`prd`](#prd)**: Widevine / PlayReady device management.
- **[`serve`](#serve)**: run the local CDM + REST API server.
- **[`util`](#util)**: helper media utilities.
!!! tip "Getting help on any command"
Every command accepts `-h`, `--help`, or `-?`. For example `unshackle dl --help` or `unshackle wvd new --help`. Service-specific arguments for `dl` and `search` (such as a title ID or search query) are defined by each service, so check `unshackle dl SERVICE --help` for those.
---
## Root command
```
unshackle [OPTIONS] COMMAND [ARGS]...
```
> unshackle: Modular Movie, TV, and Music Archival Software.
Every invocation prints a banner and, when `update_checks` is enabled in your config, checks for a newer release.
| Option | Description |
|---|---|
| `-v`, `--version` | Print version information and exit. |
| `-d`, `--debug` | Enable `DEBUG`-level logs and JSON debug logging. Silences noisy HTTP libraries unless `debug_requests` is set in config. |
| `-h`, `--help`, `-?` | Show help and exit. |
Commands are auto-discovered, so third-party command modules dropped into the commands directory appear automatically.
---
## `dl`
The download command. It is itself a group whose **subcommands are the installed streaming services**, so an invocation always has three layers:
```
unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...]
```
- `OPTIONS`: every flag below, parsed at the `dl` level.
- `SERVICE`: a service tag (e.g. `NF`, `DSNP`). Tags are case-insensitive and honour each service's aliases (`disney+`, `DSNP`, etc. all resolve to the same tag).
- `SERVICE ARGS`: the title, URL, or ID, plus any service-specific options. These belong to the service, not to `dl`.
!!! example "Typical downloads"
```shell
# Best available 1080p SDR of a Netflix title
unshackle dl -q 1080 NF 81234567
# 4K HDR10, English audio + subs, from a URL
unshackle dl -q 2160 -r HDR10 -l en DSNP "https://www.disneyplus.com/…"
# A season range, HEVC video, EC3 audio
unshackle dl -w S01-S03 -v H.265 -a EC3 SERVICE TITLE-ID
```
!!! note "Config-driven defaults"
Any `dl` flag can be given a default under the `dl:` section of `unshackle.yaml`, and per-service under `services.<TAG>.dl`. Explicit command-line values (and environment values) always win over both; service defaults only fill in options you did not set. See the [configuration file](../getting-started/configuration-file.md) guide.
### Quality, codec, bitrate & range
| Flag | Default | Description |
|---|---|---|
| `-p`, `--profile` | — | Profile for credentials and cookies. |
| `-q`, `--quality` | best | Resolution(s), comma-separated, e.g. `1080,720`. `-q 1080` also matches anamorphic tracks by 16:9 canvas. |
| `-v`, `--vcodec` | any | Video codec(s). Accepts names or values: `AVC`/`H.264`, `HEVC`/`H.265`, `VC1`, `VP8`, `VP9`, `AV1`. |
| `-a`, `--acodec` | any | Audio codec(s), comma-separated. Accepts `AAC`, `AC3`/`DD`, `EC3`/`DD+`/`eac3`/`ddp`, `AC4`, `OPUS`, `OGG`/`vorbis`, `DTS`, `ALAC`, `FLAC`. |
| `-vb`, `--vbitrate` | highest | Exact video bitrate in kbps. |
| `-ab`, `--abitrate` | highest | Exact audio bitrate in kbps. |
| `-vb-range`, `--vbitrate-range` | — | Video bitrate range in kbps, e.g. `6000-7000`; picks highest within. Mutually exclusive with `--vbitrate`. |
| `-ab-range`, `--abitrate-range` | — | Audio bitrate range in kbps, e.g. `128-256`. Mutually exclusive with `--abitrate`. |
| `-r`, `--range` | `SDR` | Colour range(s): `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `DV`, `HYBRID`. |
| `-c`, `--channels` | — | Audio channels; matches sub-layouts (5.1 ≈ 6.0). |
| `-naa`, `--noatmos` | off | Exclude Dolby Atmos audio tracks. |
| `--worst` | off | Pick the lowest bitrate within the requested quality. **Requires `-q`.** |
| `--best-available` | off | Continue with the best available if a requested resolution/language is absent, instead of failing. |
| `-rvb`, `--real-video-bitrate` | off | Probe real media size for true video bitrates, overriding the manifest. |
| `-rab`, `--real-audio-bitrate` | off | Same for audio (slower). |
!!! note "`HYBRID` range"
`-r HYBRID` fetches both HDR10/HDR10+ and Dolby Vision and merges them with `dovi_tool`. It requires the `dovi_tool` binary on your `PATH`.
### Language & subtitles
| Flag | Default | Description |
|---|---|---|
| `-l`, `--lang` | `orig` | Language(s) for **both** video and audio. `orig` = the title's original language; e.g. `orig,en`. |
| `-vl`, `--v-lang` | — | Video-only language (overrides `-l` for video). |
| `-al`, `--a-lang` | — | Audio-only language (overrides `-l` for audio). |
| `-sl`, `--s-lang` | `all` | Subtitle language(s). |
| `--require-subs` | — | Required subtitle langs; keeps **all** subs only if these exist. **Cannot combine with `--s-lang`.** |
| `-fs`, `--forced-subs` | off | Include forced subtitle tracks. |
| `--exact-lang` | off | Exact matching only: `-l es-419` matches `es-419`, not `es-ES`. |
| `--sub-format` | — | Output subtitle format (`SRT`/`srt`, `VTT`/`webvtt`, `ASS`/`ssa`, `TTML`, `SMI`, …), or `original` to keep the source format. |
The special language tokens `orig`, `all`, and `best` are honoured everywhere a language is expected.
### Title & episode selection
| Flag | Description |
|---|---|
| `-w`, `--wanted` | Wanted episodes, e.g. `S01-S05,S07`, `S01E01-S02E03`. Supports exclusions with a leading `-` (e.g. `-S03`). |
| `--select-titles` | Interactively select titles (series only). **Cannot combine with `-w`.** |
| `--latest-episode` | Download only the single most recent episode. |
| `--list-titles` | List titles only; do not download. |
### Track-type inclusion / exclusion
Keep only certain track types, or skip certain track types. Attachments are always kept.
| Include-only | Skip |
|---|---|
| `-V`, `--video-only` | `-nv`, `--no-video` |
| `-A`, `--audio-only` | `-na`, `--no-audio` |
| `-S`, `--subs-only` | `-ns`, `--no-subs` |
| `-C`, `--chapters-only` | `-nc`, `--no-chapters` |
| Flag | Description |
|---|---|
| `-ad`, `--audio-description` | Include descriptive (audio-description) tracks. |
| `--skip-subtitle-errors` | Skip a failed subtitle instead of aborting the title. Video/audio failures remain fatal. |
### Output, muxing & files
| Flag | Description |
|---|---|
| `--split-audio` | Write a separate output file per audio codec instead of merging. Defaults to config `muxing.merge_audio`. |
| `--merge-video` | Mux all selected video tracks into one file. Defaults to config `muxing.merge_video`. |
| `-o`, `--output` | Override the output directory for this run. |
| `--no-folder` | Disable folder creation for TV shows. |
| `--no-source` | Remove the source tag from the filename/path. |
| `--no-mux` | Do not mux; keep individual track files. |
| `--tag` | Group tag override. |
| `--repack` | Add a `REPACK` tag to the filename. |
### Metadata & tagging
| Flag | Description |
|---|---|
| `--tmdb` | TMDB ID (integer). |
| `--imdb` | IMDb ID, e.g. `tt1375666`. |
| `--animeapi` | AnimeAPI ID, e.g. `mal:12345` or `anilist:98765` (defaults to MAL). Back-fills TMDB/IMDb. |
| `--enrich` | Override show title/year from the external source. **Requires** one of `--tmdb`, `--imdb`, or `--animeapi`. |
### DRM, keys & decryption
| Flag | Description |
|---|---|
| `--cdm-only` / `--vaults-only` | Use only the CDM, or only Key Vaults, for key acquisition. |
| `--skip-dl` | Skip the download but still retrieve keys. |
| `--export` | Export track info and keys to a JSON file in the exports directory. |
### Network & proxy
| Flag | Description |
|---|---|
| `--proxy` | Proxy URI, a 2-letter country code resolved from configured providers, or `provider:region` (e.g. `nordvpn:ca`, `gluetun:us`, `protonvpn:de:berlin`). |
| `--no-proxy` | Force-disable all proxy use. |
| `--no-proxy-download` | Bypass the proxy for **segment downloads only** (manifest, licence, and auth stay proxied). |
| `--remote` | Use a remote unshackle server. |
| `--server` | Name a remote server from the `remote_services` config. |
### Concurrency, caching & listing
| Flag | Default | Description |
|---|---|---|
| `--workers` | downloader default | Per-track download threads. |
| `--downloads` | `1` | Number of tracks downloaded concurrently. |
| `--no-cache` | off | Bypass the title cache. |
| `--reset-cache` | off | Clear the title cache. |
| `--list` | off | List available/would-be-downloaded tracks; do not download. |
| `--slow` | — | Inter-title delay. Bare `--slow` = 60-120s; `--slow 20-40` = custom range (minimum 20s). |
!!! warning "Some flags cannot be combined"
`--require-subs` and `--s-lang`; `--select-titles` and `--wanted`; `--worst` requires `--quality`; `--vbitrate` and `--vbitrate-range` (and the audio equivalents) are mutually exclusive.
---
## `search`
Search a service for titles. Like `dl`, `search` is a group whose subcommands are the installed services, and it reuses `dl`'s authentication, cookie, and proxy machinery.
```
unshackle search [OPTIONS] SERVICE [QUERY...]
```
The query syntax is defined per service. Results are printed as a tree of titles with their service IDs. Feed an ID straight into `dl`.
| Option | Description |
|---|---|
| `-p`, `--profile` | Profile for credentials and cookies. |
| `--proxy` | Proxy URI, 2-letter country code, or `provider:region`. |
| `--no-proxy` | Force-disable all proxy use. |
!!! example
```shell
unshackle search NF "The Office"
unshackle search -p myprofile NF "breaking bad"
unshackle search --proxy nordvpn:ca SERVICE "query"
```
---
## `import`
Reconstruct a download (download → decrypt → mux) from an `--export` JSON file without re-contacting the service. It re-fetches the manifest, injects the stored keys, and then runs the normal `dl` pipeline. The service tag is read from the export file, not passed by you.
```
unshackle import EXPORT_FILE [DL_ARGS...]
```
Any `dl` options placed after the file are forwarded verbatim, so you can override quality, range, proxy, and so on.
!!! example
```shell
unshackle import export.json
unshackle import export.json -r HDR10 --proxy US
```
The export file must be a valid v2 export (created by a current build of unshackle via `dl --export`) and must contain a `service` tag.
---
## `cfg`
Read, set, delete, or list configuration values in `unshackle.yaml` without hand-editing YAML.
```
unshackle cfg [KEY] [VALUE] [OPTIONS]
```
| Argument / Option | Description |
|---|---|
| `KEY` | Dotted path into the config, e.g. `tag`, `serve.api_secret`, `directories.downloads`. |
| `VALUE` | Value to set. Parsed as a Python literal when possible, so `true`, `123`, `['a','b']`, and `{'k':1}` become real types; bare words stay strings. |
| `--unset` | Remove the configuration value. |
| `--list` | List all set configuration values. |
!!! warning "Comments are stripped"
Writing a value through `cfg` rewrites the YAML file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error.
!!! example
```shell
unshackle cfg --list # print entire config
unshackle cfg serve.api_secret # read a nested key
unshackle cfg tag MYGROUP # set a string
unshackle cfg update_checks false # literal → boolean False
unshackle cfg directories.downloads /mnt/dl # set a nested path
unshackle cfg tag --unset # remove a key
```
---
## `env`
Inspect and manage the project environment.
```
unshackle env COMMAND [ARGS]...
```
### `env check`
```
unshackle env check
```
Prints a dependency table (Category / Tool / Status / Required / Purpose) showing which external tools resolve on your `PATH`, and a summary of how many required tools are installed.
| Category | Tools |
|---|---|
| Core | FFmpeg*, FFprobe*, MKVToolNix*, mkvpropedit* |
| DRM | shaka-packager*, mp4decrypt, ML-Worker |
| HDR | dovi_tool, HDR10Plus_tool |
| Subtitle | SubtitleEdit, CCExtractor |
| Player | FFplay, MPV |
| Network | HolaProxy, Caddy, Docker, git |
<small>* required; all others are optional.</small>
### `env info`
```
unshackle env info
```
Shows where the config was loaded from (or lists the candidate config locations if none was found) and prints a table of every configured directory (downloads, temp, cache, cookies, logs, exports, WVDs, PRDs, services, and more).
### `env clear`
Clear an environment directory. The directory is emptied and recreated, and the number of files and bytes freed is reported.
| Command | Description |
|---|---|
| `unshackle env clear cache [SERVICE]` | Clear the cache directory, or just one service's cache subdirectory. |
| `unshackle env clear temp` | Clear the temp directory. |
!!! example
```shell
unshackle env check
unshackle env info
unshackle env clear cache
unshackle env clear cache NF # one service only
unshackle env clear temp
```
---
## `kv`
Manage Key Vaults. Vaults are configured under `key_vaults` in `unshackle.yaml`, each with a `name`, a `type` (`sqlite`, `mysql`, `http`, `api`), and type-specific options. Service tags are normalised automatically.
```
unshackle kv COMMAND [ARGS]...
```
### `kv copy TO_VAULT FROM_VAULT...`
Copy keys from one or more source vaults into a destination vault. Rows with matching KIDs are skipped unless the existing row has no key; existing data is never altered or deleted.
| Option | Description |
|---|---|
| `-s`, `--service` | Only copy data for a specific service. |
| `-l`, `--local-only` | Only copy data for services installed locally. Mutually exclusive with `--service`. |
### `kv sync VAULT...`
Ensure two or more vaults hold the same set of keys, essentially a chained bidirectional copy. Requires more than one vault. Accepts the same `--service` / `--local-only` options as `copy`.
### `kv add FILE SERVICE VAULT...`
Add content keys to one or more vaults for a service. `FILE` contains one `KID:KEY` pair per line (32 hex : 32 hex, UTF-8). Lines that don't match are skipped.
### `kv search KID`
Search configured vaults for a KID (32 hex characters, no dashes) and report any matching key.
| Option | Description |
|---|---|
| `-s`, `--service` | Limit the search to a specific service tag. |
| `-v`, `--vault` | Limit the search to a specific configured vault by name. |
!!! note
Remote vaults cannot be enumerated without a service, so pass `--service` when searching them.
### `kv prepare VAULT...`
Create service tables on vaults that use tables, for every installed service, where they don't already exist.
!!! example
```shell
unshackle kv copy main backup1 backup2
unshackle kv copy main backup -s NF
unshackle kv sync main mysql_vault
unshackle kv add keys.txt NF main backup
unshackle kv search 0123456789abcdef0123456789abcdef -s NF -v main
unshackle kv prepare main mysql_vault
```
---
## `wvd`
Manage Widevine Device (`.wvd`) files. Devices live in the configured WVDs directory.
```
unshackle wvd COMMAND [ARGS]...
```
| Command | Description |
|---|---|
| `wvd add PATHS...` | Validate and move one or more `.wvd` files into the WVDs directory. |
| `wvd delete NAMES...` | Delete `.wvd` files by name (without extension). Prompts for confirmation. |
| `wvd parse PATH` | Parse a `.wvd` and print its System ID, security level, type, flags, and client info. Relative paths resolve against the WVDs directory. |
| `wvd dump WVD_PATHS... OUT_DIR` | Extract a device's contents (metadata, private key, client ID, VMP) into `OUT_DIR/<name>/`. With no paths, dumps every WVD in the WVDs directory. |
| `wvd new NAME PRIVATE_KEY CLIENT_ID [FILE_HASHES]` | Create a new `.wvd` from a PEM private key and a `ClientIdentification` blob, optionally with a VMP (`FileHashes`) blob. |
`wvd new` options:
| Option | Default | Description |
|---|---|---|
| `-t`, `--type` | `Android` | Device type. |
| `-l`, `--level` | `1` | Security level (1-3). |
| `-o`, `--output` | WVDs dir | Output directory. |
!!! example
```shell
unshackle wvd add device.wvd
unshackle wvd parse nexus_6p_1234_l1
unshackle wvd dump device.wvd ./out_dir
unshackle wvd new "Nexus 6P" private_key.pem client_id.bin vmp.bin -t ANDROID -l 3
```
---
## `prd`
Manage PlayReady Device (`.prd`) files. Devices live in the configured PRDs directory. Built on `pyplayready`.
```
unshackle prd COMMAND [ARGS]...
```
### `prd new PATHS...`
Create a new `.prd`. Provide either a single folder containing `zgpriv.dat` (group key) and `bgroupcert.dat` (group certificate), or two file paths (group key, then group certificate).
| Option | Description |
|---|---|
| `-e`, `--encryption_key` | Optional device ECC private encryption key (generated if omitted). |
| `-s`, `--signing_key` | Optional device ECC private signing key (generated if omitted). |
| `-o`, `--output` | Output directory or `.prd` file path. |
`prd new` will not overwrite an existing file.
### `prd reprovision PRD_PATH`
Reprovision an existing device by replacing its leaf certificate and keys. The device must support reprovisioning (version 3 or higher). Accepts the same `-e` / `-s` / `-o` options; defaults to overwriting the device in place.
### `prd test DEVICE`
Test a device against the Microsoft PlayReady demo server and print the returned keys.
| Option | Default | Description |
|---|---|---|
| `-c`, `--ckt` | `aesctr` | Content key encryption type: `aesctr` or `aescbc`. |
| `-sl`, `--security-level` | `2000` | Minimum security level: `150`, `2000`, or `3000`. |
!!! example
```shell
unshackle prd new ./device_folder/
unshackle prd new zgpriv.dat bgroupcert.dat -o /out/dir
unshackle prd reprovision mydevice.prd
unshackle prd test mydevice.prd -sl 3000 -c aescbc
```
---
## `serve`
Serve your local Widevine/PlayReady devices and the REST API for remote access. Built on `aiohttp`.
```
unshackle serve [OPTIONS]
```
| Option | Default | Description |
|---|---|---|
| `-h`, `--host` | `127.0.0.1` | Host to serve from. |
| `-p`, `--port` | `8786` | Port to serve from. |
| `--caddy` | off | Also serve through Caddy (requires the `caddy` binary and a `Caddyfile`). |
| `--api-only` | off | Serve only the REST API, not the CDM endpoints. Incompatible with `--no-widevine`/`--no-playready`. |
| `--no-widevine` | off | Disable the Widevine CDM endpoints. |
| `--no-playready` | off | Disable the PlayReady CDM endpoints. |
| `--no-key` | off | Disable API-key authentication (allows all requests). |
| `--debug-api` | off | Include tracebacks/stderr in API error responses. |
| `--debug` | off | Enable debug logging for API operations. |
| `--remote-only` | off | Expose only the remote service session endpoints (health, services, search, session). Implies `--api-only`. |
WVD files in the WVDs directory and PRD files in the PRDs directory are auto-loaded. The REST API lives under `http://<host>:<port>/api/`, with Swagger UI at `/api/docs/` and a health check at `/api/health` (exempt from auth).
!!! warning "Configure `api_secret` first"
Unless you pass `--no-key`, `serve.api_secret` must be set in your config. Requests authenticate with the `X-Secret-Key` header. Running with `--no-key` disables authentication entirely and should only be used on a trusted, private network.
```yaml title="unshackle.yaml"
serve:
api_secret: "your-api-secret"
users:
your-secret-key:
devices: ["device_name"] # Widevine devices
playready_devices: ["device_name"] # PlayReady devices
username: user
```
!!! example
```shell
unshackle serve
unshackle serve -h 0.0.0.0 -p 8080
unshackle serve --api-only
unshackle serve --remote-only
unshackle serve --caddy
```
---
## `util`
Various helper media utilities. When a command is given a directory, it processes every `.mkv`/`.mp4` inside it in natural (S01E01 before S01E10) order.
```
unshackle util COMMAND [ARGS]...
```
### `util refresh-services`
```
unshackle util refresh-services
```
Force a refresh (git pull / hard reset) of all service repos configured under `directories.services`.
### `util crop PATH ASPECT`
Losslessly crop H.264/H.265 video at the bitstream level. `ASPECT` is a `W:H` target such as `2.39:1`.
| Option | Default | Description |
|---|---|---|
| `--letter` / `--pillar` | `--letter` | Crop top/bottom (`--letter`) or the sides (`--pillar`). |
| `-o`, `--offset` | `0` | Fine-tune the computed crop area if not perfectly centred. |
| `-p`, `--preview` | off | Preview the crop in MPV (or FFplay) instead of writing a file. |
### `util range PATH`
Losslessly set the video range flag to full or limited at the bitstream level.
| Option | Default | Description |
|---|---|---|
| `--full` / `--limited` | — | Full (0-255) or limited (16-235) range. |
| `-p`, `--preview` | off | Preview instead of writing a file. |
### `util test PATH`
Decode an entire video with FFmpeg and report any corruptions or errors. All streams are tested by default; subtitles cannot be tested.
| Option | Default | Description |
|---|---|---|
| `-m`, `--map` | `0` | Test specific streams via FFmpeg's `-map`, e.g. `0:v:0` or `0:a`. |
!!! example
```shell
unshackle util refresh-services
unshackle util crop video.mkv 2.39:1
unshackle util crop ./season/ 16:9 --pillar -o 10
unshackle util range video.mkv --full
unshackle util test video.mkv -m 0:v:0
```
---
## See also
- [Installation](../getting-started/installation.md): install unshackle and its dependencies.
- [Quickstart](../getting-started/quickstart.md): your first download.
- [Configuration file](../getting-started/configuration-file.md): set defaults for any `dl` flag and configure vaults, proxies, and directories.
docs/guide/downloading.md
+563
View File
@@ -0,0 +1,563 @@
# Downloading
`unshackle dl` is the main download command: it fetches a title's manifest from a
streaming service, selects exactly the tracks you asked for, acquires and applies
decryption keys, and muxes everything into a finished Matroska file. The flags below
cover quality and codec selection, language and subtitle handling, episode ranges, and
the download flow itself, with examples throughout.
!!! note "Where the full flag list lives"
This page covers the flags you will use most and explains how they interact. It is
not an exhaustive enumeration. For the complete, always-current list of every option
and its exact wording, run `unshackle dl --help`. Every flag shown here can also be
given a default in your configuration file. See
[Configuration file](../getting-started/configuration-file.md).
## Command shape
A download is always three layers: the root CLI, the `dl` command (which is where all
the flags below are parsed), and a **service tag** subcommand that carries the title
argument.
```shell
unshackle dl [OPTIONS] SERVICE [SERVICE ARGUMENTS]
```
- `unshackle`: the root command.
- `dl`: the download command. **All of the options on this page are parsed here**, so
they go *before* the service tag.
- `SERVICE`: a service tag such as `NF`, `DSNP`, `AMZN`. Service tags are
case-insensitive and honour each service's aliases (for example `disney+` resolves to
its real tag).
- `SERVICE ARGUMENTS`: usually a title ID or URL. This positional argument belongs to
the **service**, not to `dl`, so it comes *after* the tag.
```shell title="A minimal download"
unshackle dl NF 81234567
```
```shell title="Flags belong to dl, the title belongs to the service"
unshackle dl -q 1080 -v H.265 -r HDR10 --lang en DSNP 'https://www.disneyplus.com/…'
```
!!! tip "See what a service accepts"
Because the positional argument is defined by each service, run
`unshackle dl SERVICE --help` (e.g. `unshackle dl NF --help`) to see that service's
own argument and any service-specific options.
## The download flow
When you run a download, `dl` moves through the pipeline below. Understanding the order
helps explain why, for example, `--list` stops early and why keys are fetched even with
`--skip-dl`.
1. **Setup**: load the DRM CDM, key vaults, proxy providers, cookies, and credentials.
2. **Authenticate**: sign in to the service with your profile's cookies/credentials.
3. **Fetch titles**: retrieve the movie, episode list, or album (cached unless
`--no-cache`/`--reset-cache`).
4. **Filter titles**: apply `--wanted`, `--latest-episode`, or `--select-titles`.
5. **Get tracks**: parse the manifest into video, audio, subtitle, and chapter tracks.
6. **Select tracks**: narrow each type down using your quality, codec, range, language,
and bitrate flags.
7. **Download**: pull segments for the selected tracks (concurrently per `--downloads`).
8. **License & decrypt**: acquire content keys (vault → CDM), decrypt each track.
9. **Post-process**: extract closed captions, convert subtitles, repack, mux with
`mkvmerge`, and move the finished file to your downloads directory.
`--list` and `--list-titles` stop after selection/filtering and print what *would* be
downloaded. `--skip-dl` runs the license step but skips the actual segment download.
## Choosing quality
### Resolution
`-q` / `--quality` takes one or more target resolutions (heights) as a comma-separated
list. Without it, unshackle picks the **best available** resolution.
```shell title="Single resolution"
unshackle dl -q 1080 NF 81234567
```
```shell title="Multiple resolutions in one run"
unshackle dl -q 2160,1080,720 NF 81234567
```
!!! note "16:9 canvas matching"
Resolution matching is by track **height** first. If no track has an exact matching
height, unshackle falls back to a 16:9-canvas match, so `-q 1080` will also match an
anamorphic `1920×804` track (computed as `int(width × 9 / 16)`).
### Best-available and worst
By default a missing requested resolution is an error. Two flags change that:
| Flag | Behaviour |
| --- | --- |
| `--best-available` | If the requested resolution(s) aren't present, continue with the best that *is* available instead of failing. Also softens missing audio/subtitle languages and hybrid fallbacks. |
| `--worst` | Within the specified quality, pick the **lowest** bitrate rendition. **Requires `-q/--quality`.** |
```shell title="Never fail on a missing resolution"
unshackle dl -q 2160 --best-available AMZN 0ABC123
```
## Video codec and color range
### Codec
`-v` / `--vcodec` selects one or more video codecs; the default is any codec. It accepts
either enum **names** or their **values**, comma-separated.
| Name | Value |
| --- | --- |
| `AVC` | `H.264` |
| `HEVC` | `H.265` |
| `VC1` | `VC-1` |
| `VP8` | `VP8` |
| `VP9` | `VP9` |
| `AV1` | `AV1` |
```shell title="Either spelling works"
unshackle dl -v HEVC NF 81234567
unshackle dl -v H.265 NF 81234567
unshackle dl -v hevc,avc NF 81234567
```
### Color range
`-r` / `--range` selects one or more color ranges; the default is `SDR`.
| Range | Meaning |
| --- | --- |
| `SDR` | Standard dynamic range (default) |
| `HLG` | Hybrid Log-Gamma |
| `HDR10` | HDR10 |
| `HDR10P` | HDR10+ |
| `DV` | Dolby Vision |
| `HYBRID` | Fetch both an HDR10/HDR10+ base and a DV track, then merge them |
```shell title="Grab HDR10 and Dolby Vision in one run"
unshackle dl -q 2160 -r HDR10,DV DSNP '…'
```
!!! warning "HYBRID requires dovi_tool"
`-r HYBRID` produces a single hybrid stream by injecting the Dolby Vision RPU
metadata onto an HDR10/HDR10+ base layer using **dovi_tool**. It requires the
`dovi_tool` binary on your PATH and needs both an HDR10(+) *and* a DV track to be
available, otherwise the title fails (unless `--best-available` lets it fall back).
When HDR10+ is present it is preferred over HDR10 as the base layer.
To keep an HDR10+ deliverable *alongside* the hybrid, request both ranges:
`-r HYBRID,HDR10P`. `-r HYBRID` on its own muxes only the merged hybrid.
## Bitrate selection
By default unshackle keeps the highest-bitrate rendition for each selected
resolution/codec/range/language combination. You can constrain this.
| Flag | Purpose |
| --- | --- |
| `-vb` / `--vbitrate` | Exact video bitrate in kbps. |
| `-ab` / `--abitrate` | Exact audio bitrate in kbps. |
| `-vb-range` / `--vbitrate-range` | Video bitrate range in kbps, e.g. `6000-7000`; picks the highest within it. |
| `-ab-range` / `--abitrate-range` | Audio bitrate range in kbps, e.g. `128-256`. |
!!! warning "Exact vs range are mutually exclusive"
`--vbitrate` cannot be combined with `--vbitrate-range`, and likewise for the audio
pair. Pick one form.
### Real bitrate probing
Manifest-declared bitrates are sometimes rounded or wrong. Services often advertise a
**peak** or **nominal** bandwidth that is far from the stream's real average. This matters
because a track's bitrate drives the track listing, the sort order, *and* the
`--vbitrate`/`--vbitrate-range` selection above. A bogus declared value therefore makes
unshackle pick the wrong track. These flags remove that risk by probing the actual media
size to compute a true bitrate for the top renditions, overriding the manifest value:
- `-rvb` / `--real-video-bitrate`: probe real video bitrates (per codec/range).
- `-rab` / `--real-audio-bitrate`: probe real audio bitrates (per codec/channels/
language). Slower than the video variant because there are more renditions to probe.
!!! note "Reading the probed numbers"
A single-file track (DASH `SegmentBase`/`BaseURL`) is measured **exactly**. A
multi-segment track (most HLS) is a **sampled estimate**, normally within a few percent
of the true value. For MPEG-TS HLS the probed figure also reads a few percent *above*
the demuxed elementary stream, because the segment bytes include container overhead.
That is the real *delivered* size, not an over-report or a bug.
!!! tip "Why not every rendition is probed"
Probing does not touch every rendition. Only the top rendition of each quality tier is
probed in parallel (video grouped by codec+range, audio by codec+channels+language),
and the pass then reaches downward only as far as it must to keep the ranking correct.
This keeps probing fast even when a service exposes dozens of renditions. Tracks whose
duration can't be determined fall back to `ffprobe`, and a probe failure is non-fatal:
the manifest value is simply left in place.
## Audio codec, channels, and Atmos
### Codec
`-a` / `--acodec` selects one or more audio codecs (comma-separated); the default is any.
| Name | Value | Codec |
| --- | --- | --- |
| `AAC` | `AAC` | Advanced Audio Coding |
| `AC3` | `DD` | Dolby Digital |
| `EC3` | `DD+` | Dolby Digital Plus |
| `AC4` | `AC-4` | Dolby AC-4 |
| `OPUS` | `OPUS` | Opus |
| `OGG` | `VORB` | Vorbis |
| `DTS` | `DTS` | DTS |
| `ALAC` | `ALAC` | Apple Lossless |
| `FLAC` | `FLAC` | FLAC |
Names, values, and a few aliases are accepted: `eac3` and `ddp` both resolve to `EC3`,
and `vorbis` resolves to `OGG`.
```shell title="Prefer Dolby Digital Plus, fall back to AAC"
unshackle dl -a EC3,AAC NF 81234567
```
### Channels and Atmos
- `-c` / `--channels`: desired channel layout, e.g. `5.1` or `2.0`. Matching is by
ceiling, so `5.1` implicitly matches a `6.0`-reported layout.
- `-naa` / `--noatmos`: exclude Dolby Atmos audio tracks from selection.
```shell title="5.1 audio, no Atmos"
unshackle dl -c 5.1 --noatmos AMZN 0ABC123
```
## Languages
### Video and audio language
`-l` / `--lang` sets the wanted language(s) for **both video and audio**. The default is
`orig`, the title's original language.
```shell title="Original language plus English"
unshackle dl -l orig,en NF 81234567
```
The special token `orig` is resolved to the title's actual original language everywhere
it is used. You can override each stream type independently:
- `-vl` / `--v-lang`: language for **video only** (overrides `-l` for video). Useful
when the burned-in video language differs from the audio you want.
- `-al` / `--a-lang`: language for **audio only** (overrides `-l` for audio).
```shell title="English audio over the original video"
unshackle dl -al en NF 81234567
```
### Exact vs fuzzy matching
By default language matching is fuzzy: `-l en` also accepts `en-US`, `en-GB`, etc. (up
to a small distance). Pass `--exact-lang` to require an exact match: with it, `-l es-419`
matches only `es-419`, not `es-ES`.
!!! tip "The `all` and `best` tokens"
For audio, the tokens `all` and `best` bypass the usual one-track-per-language pick
and instead select the best track for **each** language present. For subtitles, `all`
keeps every subtitle language.
## Subtitles
### Selecting subtitle languages
`-sl` / `--s-lang` sets the wanted subtitle language(s). The default is `all`, meaning
every available subtitle language is downloaded.
```shell title="Only English and Spanish subtitles"
unshackle dl -sl en,es NF 81234567
```
### Requiring subtitles
`--require-subs` takes a list of languages that **must** exist. If they all exist, *all*
subtitles are downloaded; if any is missing, the title fails. This is useful for gating a
download on the presence of a specific subtitle track.
!!! warning "Cannot combine with `--s-lang`"
`--require-subs` and `--s-lang` are mutually exclusive. Use one or the other.
### Forced subtitles and output format
- `-fs` / `--forced-subs`: include forced subtitle tracks (signs/foreign dialogue).
Without this flag, forced subs are dropped from selection.
- `--sub-format`: set the output subtitle format, converting only when necessary.
Accepts codec names/values and common aliases (`srt`, `vtt`, `ass`, `ssa`, `ttml`,
etc.), or the literal `original` to keep the source format.
| Value | Format |
| --- | --- |
| `SRT` / `srt` | SubRip |
| `VTT` / `vtt` | WebVTT |
| `ASS` / `ass` | Advanced SubStation Alpha |
| `SSA` / `ssa` | SubStation Alpha |
| `TTML` / `ttml` | Timed Text Markup |
| `original` | Keep the source format, no conversion |
```shell title="Convert subtitles to SRT"
unshackle dl --sub-format srt NF 81234567
```
!!! note "SDH stripping happens by default"
When a subtitle track is marked SDH (for the deaf/hard-of-hearing) and there is no
plain same-language subtitle, unshackle produces a stripped, non-SDH variant
automatically. This behaviour is governed by the `subtitle.strip_sdh` config option
(default on).
## Selecting episodes
For series, several flags control which episodes are downloaded. By default, **all**
episodes are downloaded.
### Wanted ranges
`-w` / `--wanted` accepts flexible season/episode ranges, comma-separated. Prefix a token
with `-` to exclude it.
=== "Whole seasons"
```shell
unshackle dl -w S01 NF 81234567
unshackle dl -w S01-S03 NF 81234567
unshackle dl -w S01,S03,S05 NF 81234567
```
=== "Specific episodes"
```shell
unshackle dl -w S01E01 NF 81234567
unshackle dl -w S01E01-S01E05 NF 81234567
unshackle dl -w S01E01-S02E03 NF 81234567
```
=== "Exclusions"
```shell
# Seasons 1 through 5, but not season 3
unshackle dl -w S01-S05,-S03 NF 81234567
```
### Other selection flags
| Flag | Behaviour |
| --- | --- |
| `--select-titles` | Interactively pick episodes from a list (Series only). **Cannot combine with `-w`.** |
| `--latest-episode` | Download only the single most recent episode. |
| `--list-titles` | List the titles that would be downloaded, then stop. |
```shell title="Grab just the newest episode"
unshackle dl --latest-episode NF 81234567
```
## Including and excluding track types
You can restrict a download to certain track categories, either positively (only these)
or negatively (everything but these).
**Only these types** (`*-only` flags):
- `-V` / `--video-only`
- `-A` / `--audio-only`
- `-S` / `--subs-only`
- `-C` / `--chapters-only`
**Skip these types** (`no-*` flags):
- `-nv` / `--no-video`
- `-na` / `--no-audio`
- `-ns` / `--no-subs`
- `-nc` / `--no-chapters`
Additional track-type flags:
- `-ad` / `--audio-description`: include descriptive (audio-description) tracks, which
are dropped by default.
- `--skip-subtitle-errors`: if a subtitle fails to download, skip it and continue rather
than aborting the whole title. Video and audio failures remain fatal.
```shell title="Subtitles only"
unshackle dl -S -sl en NF 81234567
```
```shell title="Everything except chapters"
unshackle dl -nc NF 81234567
```
!!! note
You can combine an `*-only` flag with `no-*` flags to fine-tune: `*-only` chooses the
starting set of categories, then `no-*` subtracts from it. Attachments (e.g. fonts)
are always kept.
## Listing and dry runs
Before committing to a long download, inspect what unshackle *would* do:
| Flag | Effect |
| --- | --- |
| `--list` | List the available tracks and which would be selected, then stop. No download. |
| `--list-titles` | List the titles (episodes) that would be downloaded, then stop. |
| `--skip-dl` | Skip downloading but still acquire the decryption keys. |
```shell title="See the track selection without downloading"
unshackle dl -q 1080 -v H.265 -r HDR10 --list NF 81234567
```
## Output and muxing
### Output location and folders
- `-o` / `--output`: override the output directory for this run (otherwise the
configured downloads directory is used).
- `--no-folder`: do not create a per-show folder for TV downloads.
- `--no-source`: remove the service source tag from the filename and path.
```shell title="Send this download somewhere specific"
unshackle dl -o ~/Videos/incoming NF 81234567
```
### Muxing behaviour
By default, tracks are muxed into a single Matroska (`.mkv`) file with `mkvmerge`. These
flags change how the output is assembled:
| Flag | Behaviour | Default source |
| --- | --- | --- |
| `--no-mux` | Do not mux; keep the individual track files. | — |
| `--split-audio` | Write a separate output file per audio codec instead of merging all audio. | config `muxing.merge_audio` (on) |
| `--merge-video` | Mux all selected video tracks into one file. | config `muxing.merge_video` (off) |
### Naming tags
- `--tag`: set the release group tag (overrides the configured tag).
- `--repack`: add a `REPACK` tag to the output filename.
```shell title="Custom group tag and a REPACK label"
unshackle dl --tag MYGRP --repack NF 81234567
```
## Proxies
`--proxy` accepts a full proxy URI, a 2-letter country code (resolved through your
configured proxy providers), or a `provider:region` form.
```shell title="Proxy forms"
unshackle dl --proxy us NF 81234567
unshackle dl --proxy nordvpn:ca NF 81234567
unshackle dl --proxy 'http://user:pass@host:8080' NF 81234567
```
Two related flags:
- `--no-proxy`: force-disable all proxy use for this run.
- `--no-proxy-download`: bypass the proxy for **segment downloads only**. The manifest,
license, and authentication requests still go through the proxy. This is useful when the
proxy is only needed to satisfy geo-checks, not to move the bulk of the data.
## Performance and caching
| Flag | Purpose |
| --- | --- |
| `--workers N` | Threads used per track for segment downloads. Default depends on the downloader. |
| `--downloads N` | Number of tracks downloaded concurrently. Default `1`. |
| `--slow [MIN-MAX]` | Add a delay between titles to look more like a real device. `--slow` alone means 60-120s; `--slow 20-40` sets a custom range. Minimum 20s. |
| `--no-cache` | Bypass the title cache for this download. |
| `--reset-cache` | Clear the title cache before fetching. |
```shell title="Two tracks at a time, eight threads each"
unshackle dl --downloads 2 --workers 8 NF 81234567
```
```shell title="Space out a season download"
unshackle dl -w S01 --slow 30-60 NF 81234567
```
## Keys, vaults, and export
By default, unshackle checks your **key vaults** first and only asks a **CDM** to license
a key when the vault misses. You can force one side or the other:
- `--cdm-only`: only use the CDM (skip vaults).
- `--vaults-only`: only use key vaults (never license via the CDM); a missing key fails.
```shell title="Fetch keys only, no download"
unshackle dl --skip-dl NF 81234567
```
`--export` writes a JSON file, into the configured exports directory, containing track
info and the acquired content keys for each title. This is the format consumed by
`unshackle import` to reconstruct a download later.
```shell title="Export track info and keys"
unshackle dl --skip-dl --export NF 81234567
```
!!! note "Region is recorded only with a proxy"
When `--proxy` is used, the export records the region so an import can reproduce the
correct geofence. Without a proxy, no region is stored.
## Metadata and tagging
unshackle looks up metadata automatically, but you can override the identifiers used for
tagging and naming:
| Flag | Example | Purpose |
| --- | --- | --- |
| `--tmdb` | `--tmdb 27205` | Use this TMDB ID instead of automatic lookup. |
| `--imdb` | `--imdb tt1375666` | Use this IMDb ID. |
| `--animeapi` | `--animeapi mal:12345` | Resolve via AnimeAPI (`mal:`/`anilist:` prefix; defaults to MAL). |
| `--enrich` | — | Override the show title and year from an external source. **Requires** one of `--tmdb`, `--imdb`, or `--animeapi`. |
```shell title="Force the right IMDb match and enrich the title"
unshackle dl --imdb tt1375666 --enrich NF 81234567
```
## Configuration defaults
Every flag on this page can be set as a default in your configuration file so you don't
have to type it each time, and defaults can be scoped per service. See
[Configuration file](../getting-started/configuration-file.md) for the `dl:` block and
per-service overrides.
!!! tip "Explicit flags always win"
A value you pass on the command line (or via an environment variable) always takes
precedence over both the global `dl:` defaults and any per-service `dl:` overrides.
Config only fills in options you didn't set explicitly.
## Quick reference
A condensed lookup of the most-used options. Run `unshackle dl --help` for the complete,
authoritative list.
| Flag | Short | Meaning |
| --- | --- | --- |
| `--quality` | `-q` | Target resolution(s), e.g. `1080,720`. |
| `--vcodec` | `-v` | Video codec(s). |
| `--acodec` | `-a` | Audio codec(s). |
| `--range` | `-r` | Color range(s); default `SDR`. |
| `--channels` | `-c` | Audio channel layout. |
| `--noatmos` | `-naa` | Exclude Atmos audio. |
| `--lang` | `-l` | Video + audio language(s); default `orig`. |
| `--a-lang` | `-al` | Audio-only language override. |
| `--v-lang` | `-vl` | Video-only language override. |
| `--s-lang` | `-sl` | Subtitle language(s); default `all`. |
| `--forced-subs` | `-fs` | Include forced subtitles. |
| `--sub-format` | | Output subtitle format. |
| `--wanted` | `-w` | Episode/season range. |
| `--select-titles` | | Interactively pick episodes. |
| `--latest-episode` | | Only the newest episode. |
| `--video-only` / `--audio-only` / `--subs-only` | `-V` / `-A` / `-S` | Restrict track types. |
| `--no-video` / `--no-audio` / `--no-subs` / `--no-chapters` | `-nv` / `-na` / `-ns` / `-nc` | Skip track types. |
| `--worst` | | Lowest bitrate within `-q`. |
| `--best-available` | | Degrade gracefully instead of failing. |
| `--output` | `-o` | Output directory for this run. |
| `--split-audio` / `--merge-video` / `--no-mux` | | Muxing behaviour. |
| `--proxy` / `--no-proxy` / `--no-proxy-download` | | Proxy control. |
| `--workers` / `--downloads` / `--slow` | | Concurrency and pacing. |
| `--list` / `--list-titles` / `--skip-dl` | | Dry runs. |
| `--cdm-only` / `--vaults-only` | | Key source control. |
| `--export` | | Export track info and keys to JSON. |
| `--tmdb` / `--imdb` / `--animeapi` / `--enrich` | | Metadata overrides. |
docs/guide/drm-and-cdm.md
+524
View File
@@ -0,0 +1,524 @@
# DRM & CDM Setup
Most commercial streaming content is protected by a Digital Rights Management (DRM)
system. To download and play it back, unshackle needs to obtain the **content keys**
for the encrypted tracks and then decrypt them. This page explains the DRM systems
unshackle supports, how to provide a device so it can talk to a license server, the
difference between local and remote CDMs, and how to choose a decryption backend.
!!! note "Who this page is for"
This is a practical configuration guide for end users of the unshackle CLI.
A few sections are marked **Developer**. They describe internal behaviour that is
useful when writing a service, but you do not need them to download protected
content.
## Key concepts
Before configuring anything, it helps to know the vocabulary. unshackle draws a firm
line between two things:
- A **CDM** (Content Decryption Module) is the trusted "black box" that turns a
*license challenge* into *content keys*. It is backed by a **device**, a file
containing the cryptographic identity of a real client (a phone, a browser, etc.).
- A **DRM system** is the wrapper around a CDM that parses the protection metadata,
drives the challenge → license → keys handshake, and finally invokes an external
tool to physically decrypt the downloaded file.
A handful of other terms show up throughout:
| Term | Meaning |
|------|---------|
| **PSSH** | Protection System Specific Header. The per-system init data extracted from the file's init segment or the manifest. |
| **KID** | Key ID: a 16-byte UUID identifying *which* key a track needs. |
| **CEK** | Content Encryption Key: the actual AES key, mapped from its KID. |
| **Challenge** | The license request the CDM produces, sent to the service's license server. |
| **License** | The server's response, which the CDM parses to recover the content keys. |
For a Widevine or PlayReady title, the flow is: unshackle reads the PSSH and KIDs
from the track, asks the CDM for a license challenge, sends that challenge to the
service's license server, hands the response back to the CDM to extract the keys, and
then decrypts the file with those keys.
## Supported DRM systems
unshackle implements five DRM systems. Only the first two need a device file and a
license-server handshake; the rest are special cases.
| System | Device file | License flow | Decryption |
|--------|-------------|--------------|------------|
| **Widevine** (Google) | `.wvd` | CDM challenge → license server → CDM parses → keys | shaka-packager / mp4decrypt |
| **PlayReady** (Microsoft) | `.prd` | CDM challenge (WRM header) → license server → CDM parses → keys | shaka-packager / mp4decrypt |
| **ClearKey (CENC)**, W3C EME `org.w3.clearkey` | none | JSON `{"kids":[…],"type":"temporary"}` → license server returns a JWK Set. No CDM. | shaka-packager / mp4decrypt |
| **ClearKey (HLS)**, AES-128 `EXT-X-KEY` | none | Key fetched directly from a URI or `data:` URL. No CDM, no challenge. | in-process AES-CBC |
| **MonaLisa** (proprietary, WASM) | `.mld` | Ticket from the service API; keys extracted locally via WASM. No license server. | ML-Worker binary + AES-ECB, per segment |
!!! warning "Two different ClearKeys"
"ClearKey (HLS)" and "ClearKey (CENC)" are entirely different systems that happen
to share a name. The HLS variant is the AES-128 key referenced by an `EXT-X-KEY`
tag and is decrypted in-process. The CENC variant is the W3C EME
`org.w3.clearkey` scheme, delivers keys as a JWK Set, and is decrypted through
shaka-packager or mp4decrypt just like Widevine. Neither involves a device file.
### Widevine vs PlayReady
These are the two "real" DRMs, and in unshackle they behave almost identically: the
same PSSH/KID handling, the same decryption backends, and the same key-caching flow.
The differences that matter to you are:
- **Device file**: Widevine uses a `.wvd`; PlayReady uses a `.prd`.
- **Availability**: some services offer only one system, some offer both. When a
manifest carries both, unshackle picks the appropriate CDM for each track (see
[Widevine and PlayReady on the same title](#widevine-and-playready-on-the-same-title)).
To download a Widevine-protected title you need a `.wvd` device; for PlayReady you
need a `.prd` device. You can hold both and let unshackle choose, or force one with
the `--drm` flag.
!!! info "Obtaining device files"
A `.wvd` or `.prd` encodes the private keys of a real client device. unshackle
does **not** ship with one and cannot generate the underlying secret material for
you. You must supply your own. The `wvd` and `prd` commands below manage and
provision device files once you have the raw key material.
## Providing a device with the `wvd` and `prd` commands
Device files live in dedicated directories, by default:
| Kind | Default directory |
|------|-------------------|
| Widevine `.wvd` | `data/WVDs` |
| PlayReady `.prd` | `data/PRDs` |
You can move these with the `directories` section of your `unshackle.yaml`:
```yaml title="unshackle.yaml"
directories:
wvds: /home/george/unshackle-data/WVDs
prds: /home/george/unshackle-data/PRDs
```
### Widevine devices: the `wvd` command
The `wvd` command group manages `.wvd` files.
=== "Add an existing device"
```console
$ unshackle wvd add ./nexus_6p.wvd
```
The file is validated and moved into the WVDs directory. You can add several at
once by passing multiple paths.
=== "Inspect a device"
```console
$ unshackle wvd parse nexus_6p
```
Prints the System ID, Security Level, Type, Flags, and whether a Private Key,
Client ID, and VMP are present. A bare name (no extension) is resolved inside the
WVDs directory; a path with an extension is used as-is.
=== "Create from raw material"
```console
$ unshackle wvd new "Nexus 6P" ./private_key.pem ./client_id.bin ./vmp.bin -t ANDROID -l 3
```
Builds a `.wvd` from a PEM private key, a Client ID blob, and an optional
FileHashes (VMP) blob. `-t/--type` is the device type (default `Android`) and
`-l/--level` is the security level 1 to 3 (default `1`). The VMP argument is
optional.
=== "Delete / dump"
```console
$ unshackle wvd delete nexus_6p
$ unshackle wvd dump nexus_6p ./out
```
`delete` removes a device (with a confirmation prompt). `dump` extracts a
device's contents (metadata, private key, Client ID, VMP) into a folder.
!!! warning "v1 WVD files must be migrated"
If loading a device fails with a message about a "v1 WVD file", it was made with
an older format. Migrate it with pywidevine before use:
```console
$ pywidevine migrate --help
```
### PlayReady devices: the `prd` command
The `prd` command group creates and maintains `.prd` files.
=== "Create a new device"
```console
$ unshackle prd new ./zgpriv.dat ./bgroupcert.dat
```
Provision a new device from a group key and group certificate. You can instead
point at a single folder that contains `zgpriv.dat` and `bgroupcert.dat`:
```console
$ unshackle prd new ./device_folder/
```
A fresh leaf certificate is generated and the device is written to the PRDs
directory (or to `-o/--output`). Use `-e/--encryption_key` and `-s/--signing_key`
to supply your own ECC keys instead of generating them.
=== "Reprovision"
```console
$ unshackle prd reprovision ./data/PRDs/mydevice.prd
```
Regenerates the leaf certificate and encryption/signing keys of an existing
device in place. The device must support reprovisioning (a group key must be
present, i.e. a version 3 or higher device).
=== "Test against Microsoft's demo server"
```console
$ unshackle prd test ./data/PRDs/mydevice.prd -sl 2000 -c aesctr
```
Runs a full challenge/license/keys cycle against Microsoft's public PlayReady
test server and prints the returned keys, confirming the device works.
`-sl/--security-level` accepts `150`, `2000`, or `3000` (default `2000`);
`-c/--ckt` accepts `aesctr` or `aescbc`.
## Selecting which device to use: the `cdm` config
Having a device on disk is not enough; you must tell unshackle which device to use
for which service. That is the job of the `cdm` section, which maps a service tag to
a device name (the file's name without extension). A `default` entry acts as the
fallback:
```yaml title="unshackle.yaml"
cdm:
default: nexus_6p_26830_l3
AMZN: chromecdm_l3
NF: sm_g935f_l1
```
Lookups are case-insensitive. On the command line you can override the configured
device for a single run:
```console
$ unshackle dl -w chromecdm_l3 AMZN B0ABCDEF
```
!!! tip "The device name is the file name"
A device named `chromecdm_l3` in the config resolves to `chromecdm_l3.wvd` in the
WVDs directory (or `chromecdm_l3.prd` in the PRDs directory). It may also be the
`name` of a remote CDM defined in `remote_cdm`. See
[Remote CDMs](#remote-cdms) below.
### Advanced selection: by quality, DRM, or profile
Instead of a plain name, a `cdm` entry can be a nested mapping, letting unshackle pick
a different device based on the track quality, the DRM system, or the active service
profile.
=== "By quality"
Keys are matched against the video height. You can use exact heights or
comparisons (`>=1080`, `>720`, `<=576`, `<480`). More specific matches win.
```yaml
cdm:
SOMESERVICE:
">=1080": sm_g935f_l1
default: chromecdm_l3
```
=== "By DRM system"
Keys `widevine` and `playready` select a device per system. This pairs with the
`--drm` flag, which accepts `wv`/`widevine` or `pr`/`playready`.
```yaml
cdm:
SOMESERVICE:
widevine: sm_g935f_l1
playready: mydevice_sl3
```
=== "By profile"
Keys are matched against the active profile, with a `default` fallback.
```yaml
cdm:
SOMESERVICE:
premium: sm_g935f_l1
default: chromecdm_l3
```
## Local vs remote CDMs
A CDM can run in two places:
- A **local CDM** loads a `.wvd` or `.prd` device from your machine and performs the
challenge/license/keys handshake locally. This is what the `cdm` mapping above
resolves to by default.
- A **remote CDM** delegates the CDM operations to an HTTP API. Your device (and its
secrets) live on the remote server; unshackle sends it PSSH/challenge data and
receives keys back. This is useful for sharing a high-security-level device, using
a hosted key service, or keeping device secrets off your download machine.
The device name in your `cdm` mapping resolves to a remote CDM whenever it matches the
`name` of an entry in the `remote_cdm` list. Otherwise it falls back to a local
`.prd`/`.wvd` file of that name.
## Remote CDMs
Remote CDMs are declared as a list under `remote_cdm`. Each entry has a `name` (which
you then reference from the `cdm` mapping) and a shape that depends on its `type`.
There are four kinds.
### Widevine remote CDM (pywidevine serve)
With no `type` field and a non-PlayReady device type, the entry is treated as a
standard pywidevine "serve" endpoint:
```yaml title="unshackle.yaml"
remote_cdm:
- name: my_wv_remote
device_type: ANDROID
system_id: 26830
security_level: 3
host: https://cdm.example.com
secret: your-api-secret
device_name: sm_g935f_l1
cdm:
default: my_wv_remote
```
`security_level` defaults to `3000` if omitted. Field names are read
case-insensitively, so `Device Type`/`device_type`, `System ID`/`system_id`, etc. are
all accepted.
### PlayReady remote CDM
If the device type is `PLAYREADY`, the entry is routed to the pyplayready remote CDM:
```yaml title="unshackle.yaml"
remote_cdm:
- name: my_pr_remote
device_type: PLAYREADY
security_level: 3000
host: https://cdm.example.com
secret: your-api-secret
device_name: mydevice_sl3
```
!!! warning "The `host` must end in `/playready`"
For a PlayReady remote CDM the `host` URL has to include the trailing
`/playready` path segment (e.g. `https://cdm.example.com/playready`). pyplayready's
`RemoteCdm` treats `host` as a base and appends its own endpoint paths on top of it,
so a host without the suffix points the requests at the wrong location. The symptom
is `404` responses from the server rather than an obvious configuration error.
### Decrypt Labs (hosted KeyXtractor API)
Set `type: decrypt_labs` to use Decrypt Labs' hosted KeyXtractor service, which
includes an intelligent key-caching layer:
```yaml title="unshackle.yaml"
remote_cdm:
- name: decryptlabs
type: decrypt_labs
device_name: L1 # ChromeCDM, L1, L2 (Widevine) or SL2, SL3 (PlayReady)
# host: https://keyxtractor.decryptlabs.com # optional, this is the default
# secret: ... # optional if decrypt_labs_api_key is set globally
decrypt_labs_api_key: your-decrypt-labs-key
```
The API key is sent as the `decrypt-labs-api-key` header. If you omit `secret` on the
entry, unshackle falls back to the global `decrypt_labs_api_key` value; if neither is
present it raises an error. The `host` defaults to
`https://keyxtractor.decryptlabs.com`. PlayReady devices (`SL2`/`SL3`) default to
security level `2000`/`3000` respectively; Widevine devices default to
`system_id: 26830`, `security_level: 3`.
### Custom API (`custom_api`)
For an arbitrary CDM HTTP API, `type: custom_api` provides a fully configurable client
that adapts to almost any request/response shape through YAML alone. A minimal example:
```yaml title="unshackle.yaml"
remote_cdm:
- name: my_custom
type: custom_api
device:
name: ChromeCDM
type: CHROME # CHROME, ANDROID, or PLAYREADY
system_id: 26830
security_level: 3
auth:
type: bearer # header, bearer, or basic
key: your-token
endpoints:
get_request:
path: /get-challenge
method: POST
decrypt_response:
path: /get-keys
method: POST
timeout: 30
```
The `custom_api` type supports much more: request/response field remapping,
transforms (base64/hex/JSON encoding, `kid:key` parsing), conditional parameters,
success/error detection, and vault-backed caching. Its full schema is aimed at
developers integrating a bespoke key server; see the developer note below.
!!! note "Developer: full `custom_api` schema"
A `custom_api` entry accepts these sub-objects: `device` (`name`, `type`,
`system_id`, `security_level`); `auth` (`type` of `header`/`bearer`/`basic` with
the matching credential fields, plus `custom_headers`); `endpoints`
(`get_request` and `decrypt_response`, each `{path, method, timeout}`);
per-endpoint `request_mapping` (`param_names`, `static_params`,
`conditional_params`, `transforms`, `nested_params`, `exclude_params`) and
`response_mapping` (`fields`, `transforms`, `response_types`,
`success_conditions`, `error_fields`, `key_fields`); `caching` (`enabled`,
`use_vaults`, `check_cached_first`); and `legacy`/`timeout`. PlayReady mode is
inferred when `device.type == PLAYREADY` or the device name is `SL2`/`SL3`.
## Decryption backends
Once the content keys are known, the encrypted file (for Widevine, PlayReady, and
ClearKey-CENC) is decrypted by one of two external tools:
- **shaka-packager** is the default.
- **mp4decrypt** is part of the Bento4 tools.
Both must be discoverable in unshackle's bundled `binaries/` directory or on your
`PATH`. If the selected tool's binary is missing at decrypt time, unshackle raises an
"executable not found but is required" error. See [dependencies](../getting-started/installation.md) for
installation.
You choose the backend with the `decryption` key. It can be a single value applied
globally, or a per-service mapping:
=== "Global default"
```yaml title="unshackle.yaml"
decryption: shaka
```
=== "Per service"
```yaml title="unshackle.yaml"
decryption:
default: shaka
SOMESERVICE: mp4decrypt
```
!!! info "How the value is interpreted"
The default is `shaka`. Only the exact value `mp4decrypt` selects mp4decrypt;
**anything else** (including `shaka`) selects shaka-packager. Per-service lookups
are case-insensitive, with the `default`/`DEFAULT` entry used as the fallback.
The two backends produce equivalent output for standard CENC content. shaka-packager
is generally preferred; switch a specific service to `mp4decrypt` if you hit a track
that shaka-packager cannot handle.
!!! note "ClearKey (HLS) does not use these tools"
HLS AES-128 ClearKey content is decrypted in-process using AES-CBC and ignores the
`decryption` setting entirely. There is nothing to configure.
## ClearKey
### ClearKey over HLS (AES-128)
When an HLS playlist references a key with an `EXT-X-KEY` tag, unshackle fetches the
key directly (from the given URI or an inline `data:` URL) and decrypts each segment
in-process with AES-CBC. There is no device, no CDM, and no license challenge, so
there is nothing to set up. Such tracks are handled automatically during download.
### ClearKey over CENC (W3C EME `org.w3.clearkey`)
The CENC ClearKey scheme is a proper license flow, but the "license server" simply
returns the keys as a JWK Set rather than a DRM-wrapped blob. unshackle builds a W3C
EME request of the form:
```json
{"kids": ["<base64url-kid>", …], "type": "temporary"}
```
It sends this to the manifest's license URL (or to whatever a service's ClearKey
license hook returns) and reads the JSON Web Key set back. No device file is needed.
The decrypted content still goes through shaka-packager or mp4decrypt like any other
CENC track, so the `decryption` setting applies.
## Widevine and PlayReady on the same title
A single manifest can advertise both Widevine and PlayReady. unshackle picks the CDM
per track: if a track needs a system that your loaded CDM does not provide, it
transparently switches to the appropriate device for that track using your `cdm`
mapping. If it cannot find a matching device, it fails with a clear message such as:
```
Title needs a Widevine CDM but SOMESERVICE is configured with PlayReady.
```
Use the `cdm` mapping's [DRM-based selection](#advanced-selection-by-quality-drm-or-profile)
to configure a device for each system, or force one system with `--drm` on the
command line.
## Keys, vaults, and caching
Fetching a license is comparatively slow and, for some services, rate-limited.
unshackle therefore resolves each key ID in this order before ever contacting a
license server:
1. Keys already known for the current DRM object.
2. An in-process license-key cache (for the running session).
3. Your configured [key vaults](vaults.md).
4. Only if still missing, a live license request.
After a successful license request, the recovered keys are written back to all
configured vaults so future downloads can skip the license server. Two flags let you
control this trade-off on the `dl` command:
| Flag | Effect |
|------|--------|
| `--vaults-only` | Never send a license request; use only cached/vault keys. Fails if a key is missing. |
| `--cdm-only` | Ignore vaults for sourcing keys and always perform a fresh license request. |
See [key vaults](vaults.md) for how to configure vault backends.
## Debugging DRM issues
A few configuration toggles help when a title will not decrypt:
| Config key | Default | Effect |
|------------|---------|--------|
| `debug` | `false` | Global debug output. |
| `debug_keys` | `false` | Log decryption keys. |
| `debug_requests` | `false` | Log HTTP requests. |
```yaml title="unshackle.yaml"
debug_keys: true
debug_requests: true
```
!!! warning "Keys and paths in logs"
When key logging is enabled, content keys (KID:key pairs) are written to the logs.
Treat those logs as sensitive. The unrelated `redact_paths` option (default
`true`) masks local directory paths in log output.
## See also
- [Downloading](downloading.md): the `dl` command and its DRM-related flags.
- [Key vaults](vaults.md): caching content keys across downloads.
- [Configuration](../reference/configuration.md): the full `unshackle.yaml` reference.
- [Dependencies](../getting-started/installation.md): installing shaka-packager and mp4decrypt.
docs/guide/output-and-naming.md
+384
View File
@@ -0,0 +1,384 @@
# Output & Naming
This page explains where unshackle writes finished downloads and how it names them. You control naming with **output templates** (the filename) and **folder templates** (the directory the file lands in), using a set of `{variable}` placeholders that unshackle fills in from the media it just downloaded. It also covers muxing options, release-group tagging, Unicode filenames, and how to relocate unshackle's working directories.
If you have not configured `unshackle.yaml` yet, read [Configuration File](../getting-started/configuration-file.md) first. Everything below lives in that file.
## How an output path is assembled
When a download finishes muxing, unshackle builds the final path from three pieces:
```text
<output directory> / <folder template> / <filename template><extension>
```
- **Output directory**: `directories.downloads` from your config, or whatever you pass to `-o/--output` on the command line for a single run.
- **Folder template**: an optional per-title subfolder. TV episodes and music tracks are **always** placed in a folder; movies only get one if you define a `movies` folder template (see [Folder templates](#folder-templates)).
- **Filename template**: the `output_template` for the title kind (movie, series, or song).
- **Extension**: chosen by the muxer: `.mkv` for video, `.mka` for audio-only, `.mks` for subtitle-only.
!!! example "A typical episode path"
```text
/home/user/Downloads/
└─ Show Name S01 1080p AMZN WEB-DL DDP5.1 H.264/
└─ Show.Name.S01E01.Pilot.1080p.AMZN.WEB-DL.DDP5.1.H.264-TAG.mkv
```
The folder name comes from the `series` folder template and the file from the `series` output template.
Two command-line flags change this layout for a single run:
| Flag | Effect |
|---|---|
| `-o`, `--output` | Use this directory instead of `directories.downloads`. |
| `--no-folder` | Skip the per-title subfolder; write the file directly into the output directory. |
| `--no-source` | Omit the service source tag from both the filename and folder (the `{source}` variable resolves to empty). |
## Output templates
Filename templates live under the `output_template` key, keyed by title kind. Each value is a format string built from `{variable}` placeholders.
```yaml title="unshackle.yaml"
output_template:
movies: "{title}.{year}.{quality}.{source}.WEB-DL.{audio_full}.{video}-{tag}"
series: "{title}.{season_episode}.{episode_name?}.{quality}.{source}.WEB-DL.{audio_full}.{video}-{tag}"
songs: "{track_number}. {title}"
```
The recognized kinds are:
| Kind | Applies to | Default when unset |
|---|---|---|
| `movies` | Movies | *(required for movie downloads to be named, no built-in default)* |
| `series` | TV episodes | *(required for episode downloads to be named)* |
| `songs` | Music tracks | `"{track_number}. {title}"` |
| `albums` | Album **folders** only (see below) | — |
!!! note
`movies` and `series` have no hardcoded filename fallback. If you download those title types, define their templates. `songs` falls back to `"{track_number}. {title}"` when omitted.
### Required vs. optional variables
- `{variable}`: **required**. If it is missing from the download's context, formatting raises an error. (Note that most variables are always present in context but resolve to an *empty string* when not applicable, so being "present but empty" is different from "missing".)
- `{variable?}`: **optional**. A trailing `?` marks the variable conditional. When it resolves to empty, unshackle removes the placeholder *and* one adjacent separator (`.`, `-`, or space), so the surrounding text collapses cleanly.
!!! tip "Use `?` for anything that isn't always there"
Episode names, editions, tags, HDR labels and audio extras aren't present on every title. Mark them optional so you never get doubled dots or dangling separators:
```yaml
series: "{title}.{season_episode}.{episode_name?}.{quality}.{hdr?}.{source}-{tag?}"
```
For an episode with no on-screen name, `{episode_name?}` and its adjacent dot simply disappear.
### Separator style (dots vs. spaces)
unshackle looks at the characters you place *between* variables to decide the filename's separator style. If spaces outnumber dots in your template, it sanitizes with spaces; otherwise it uses dots (the scene-style default). This is also applied to the auto-generated date separators for daily/dated content.
=== "Scene style (dots)"
```yaml
output_template:
movies: "{title}.{year}.{quality}.{source}.WEB-DL.{audio_full}.{video}-{tag}"
```
```text
The.Movie.2024.1080p.NF.WEB-DL.DDP5.1.H.264-TAG.mkv
```
=== "Readable style (spaces)"
```yaml
output_template:
movies: "{title} ({year}) {quality} {source} WEB-DL {audio_full} {video}"
```
```text
The Movie (2024) 1080p NF WEB-DL DDP5.1 H.264.mkv
```
## Template variables
Every variable below is valid in both output and folder templates. Values are derived from the downloaded media (via MediaInfo), from the title metadata, or from your config. Variables that don't apply to a given title resolve to an empty string.
### Common / video
| Variable | Meaning | Example |
|---|---|---|
| `title` | Title name (movie/show/song name; `$` is rendered as `S`) | `The Show` |
| `year` | Release year | `2024` |
| `source` | Service tag / class name (empty with `--no-source`) | `NF` |
| `quality` | Resolution with scan suffix | `1080p`, `2160p`, `576i` |
| `resolution` | Resolution number only | `1080` |
| `video` | Video codec | `H.264`, `H.265` |
| `hdr` | Dynamic-range label | `HDR`, `HDR10P`, `DV`, `DV.HDR10`, `HLG` |
| `hfr` | High frame rate marker (frame rate > 30) | `HFR` |
| `edition` | Edition label from the track, if any | `Directors Cut` |
| `tag` | Release-group tag (from `config.tag` or `--tag`) | `TAG` |
| `repack` | `REPACK` when the `--repack` flag is used | `REPACK` |
| `lang_tag` | Result of your `language_tags` rules, if configured | `MULTi` |
!!! note "How `language_tags` rules match"
Rules are tried in order, but language comparison is **fuzzy**: a rule condition of `en` matches `en-US`, `en-GB`, and any other `en-*` variant. Within a single rule, **all** conditions must match (AND logic) for it to apply. The first rule that fully matches wins; if none match, `{lang_tag?}` is removed from the filename cleanly.
### Episode-specific
| Variable | Meaning | Example |
|---|---|---|
| `season` | Zero-padded season (`S%02d`) | `S01` |
| `episode` | Zero-padded episode (`E%02d`) | `E05` |
| `season_episode` | Combined | `S01E05` |
| `episode_name` | Episode title | `Pilot` |
| `date` | ISO air date for daily/dated content | `2024-06-01` |
!!! note "Daily & sports content"
When an episode has an air date, unshackle switches to date-based naming automatically: `season` and `season_episode` become the formatted air date, `episode` and `year` are cleared, and `{date}` holds the ISO date. The date's internal separator (dots or spaces) follows your `series` template style.
### Audio
| Variable | Meaning | Example |
|---|---|---|
| `audio` | Audio codec | `DDP`, `DD`, `AAC` |
| `audio_channels` | Channel layout | `5.1`, `2.0` |
| `audio_full` | Codec + channels combined | `DDP5.1` |
| `atmos` | `Atmos` when any track carries JOC | `Atmos` |
| `dual` | `DUAL` for exactly two distinct audio languages | `DUAL` |
| `multi` | `MULTi` for more than two audio languages | `MULTi` |
### Music-specific
| Variable | Meaning |
|---|---|
| `track_number` | Zero-padded track number |
| `artist` / `album_artist` | Artist / album artist (falls back to artist) |
| `album` | Album name |
| `disc` | Disc number (only shown when greater than 1) |
| `track_total` / `disc_total` | Totals |
| `release_type` | `album`, `single`, `ep`, etc. |
| `genre` | Genre |
| `explicit` | `Explicit` when flagged |
| `isrc` / `upc` / `label` | ISRC, UPC and record label |
!!! warning "Unknown variables are ignored, not errors"
If you use a variable that isn't in the list above, unshackle emits a warning at startup but continues. Double-check spelling. A typo'd `{quailty}` will simply be left in the filename literally or dropped, not filled in.
## Folder templates
Folder templates are nested under a special `folder` key inside `output_template`. This key is handled separately from the filename kinds. You can give it a single string (used for all title kinds) or a per-kind map.
=== "Single folder template"
```yaml title="unshackle.yaml"
output_template:
movies: "{title}.{year}.{quality}.{source}-{tag}"
series: "{title}.{season_episode}.{quality}.{source}-{tag}"
folder: "{title}.{year}"
```
=== "Per-kind folder templates"
```yaml title="unshackle.yaml"
output_template:
movies: "{title}.{year}.{quality}.{source}-{tag}"
series: "{title}.{season_episode}.{quality}.{source}-{tag}"
folder:
movies: "{title} ({year})"
series: "{title} ({year})/Season {season}"
albums: "{artist} - {album} ({year})"
```
The per-kind folder keys are `movies`, `series`, `songs`, and `albums`. Any other key produces a startup warning. Path separators (`/` or `\`) are allowed in folder templates. Each segment is formatted independently, so you can build nested directory structures like `Show/Season 01`.
**Fallback behavior:**
- A per-kind folder template wins if present; otherwise the single `folder` string is used; otherwise unshackle falls back to a built-in default.
- Built-in defaults when no folder template is set: movies get no folder at all unless a `movies` folder template exists; series fall back to a folder *derived* from the `series` output template (stripping `{episode}`, `{episode_name}`, and collapsing `{season_episode}` down to `{season}`); music albums fall back to `{artist} - {album} ({year})`.
!!! note "Movies are flat by default"
Episodes and songs are always foldered. Movies are written directly into the output directory *unless* you define a `movies` folder template. Set one if you want each movie in its own directory.
## Muxing options
Muxing (combining video, audio, subtitle, chapter and attachment tracks into a single Matroska file with `mkvmerge`) is configured under the `muxing` key.
```yaml title="unshackle.yaml"
muxing:
set_title: true
merge_video: true
merge_audio: true
default_language:
audio: en
subtitle: en
```
| Key | Type | Default | Effect |
|---|---|---|---|
| `set_title` | bool | `true` | Write the title name into the MKV container title with `--title`. Set to `false` to omit it. |
| `merge_video` | bool | `true` | Group video tracks that share the same resolution, range, and codec into one file so only language varies inside it. |
| `merge_audio` | bool | `true` | Merge audio tracks of the same kind so multiple languages sit in one file. |
| `default_language` | map | *(unset)* | Preferred language per track type (`video` / `audio` / `subtitle`). A track in the preferred language is flagged as the default track. |
When no preferred language is matched, unshackle applies sensible defaults: the video default falls back to the title language / original-language track / first track; the audio default is the original-language track; and the subtitle default is a forced track matching the first audio's language.
!!! note "`default_language` only sets the default-track flag"
`default_language` controls **which track carries the MKV `--default-track` flag**. Nothing else. It does not change track *selection* (that stays with `-l`/`--alang` and friends), and it does not touch which track is marked as original: `--original-flag` still tags the *true* original-audio track. That is the point: you can make your player default to, say, Polish audio on an English-original title without altering the original marker. When the configured language isn't present in the manifest, each track type falls back to its normal default rule described above.
!!! warning "`merge_video` collapses only the language dimension"
`merge_video` groups video tracks by `(resolution, range, codec)` and merges them so that only **language** differs within a single file, with no re-encode and no concatenation. Different resolutions, ranges (SDR / HDR10 / HDR10+ / DV / HYBRID), and codecs (H.264 / H.265) **always** land in separate files. So `-r HYBRID,DV,HDR10,SDR --merge-video` produces one file *per range* (never a single fused file), whereas English + French video of identical resolution, range, and codec yields one file containing both video tracks. `merge_audio` works the same way for audio languages.
A few muxing behaviors are automatic and not configurable:
- `--no-date` is always passed for privacy (no timestamps embedded).
- Descriptive audio gets the visually-impaired flag; SDH subtitles get the hearing-impaired flag; forced subtitles get the forced flag.
- Chapters are written from the title's chapter list (see [Chapters](#chapters-naming) below).
The `--no-mux` flag skips muxing entirely and writes the individual track files, each with a track-type suffix, into the same output/folder structure.
## Tags & group naming
The release-group tag is the `-TAG` portion at the end of scene-style names, produced by the `{tag}` variable.
```yaml title="unshackle.yaml"
tag: "MYGROUP"
tag_group_name: true
tag_imdb_tmdb: true
```
| Key | Type | Default | Effect |
|---|---|---|---|
| `tag` | str | `""` | The release-group tag used for `{tag}`. |
| `tag_group_name` | bool | `true` | Write the group name (`config.tag`) into the MKV `Group` metadata tag. |
| `tag_imdb_tmdb` | bool | `true` | Look up and embed IMDb / TMDB / TVDB external-ID tags in the MKV metadata (uses `tmdb_api_key` / `simkl_client_id` when available). |
You can override the tag per run without editing config:
```bash
unshackle dl --tag OTHERGROUP AMZN "..."
```
The `--repack` flag adds the `REPACK` marker (surfaced through the `{repack}` variable) to the filename for that run:
```bash
unshackle dl --repack NF "..."
```
!!! note "Empty tags collapse cleanly"
If `config.tag` is empty and your template ends with `-{tag?}`, the trailing `-` and the placeholder are both removed. Use the optional form `{tag?}` if you sometimes download without a tag set.
## Unicode filenames
By default unshackle transliterates non-ASCII characters (Korean, Japanese, Chinese, accented Latin, etc.) into ASCII equivalents, which is safest for older tools, DDL, and P2P sharing. Set `unicode_filenames` to keep the native characters.
```yaml title="unshackle.yaml"
unicode_filenames: true
```
| Value | Result |
|---|---|
| `false` (default) | `기생충` → transliterated ASCII; accents stripped. |
| `true` | Native characters preserved in filenames and folders. |
Regardless of this setting, a set of filesystem-unsafe and structural characters is always removed or replaced during sanitization (for example `/` and `;` become ` & `, and characters like `\ * ! ? , ' " < > | $ # ~` are stripped). This is why you should never put path-unsafe characters directly in a template. unshackle warns about templates containing `< > : " / \ | ? *` at startup (folder templates are checked per path segment, so `/` is allowed there as a directory separator).
## Directory & filename configuration
The `directories` key relocates unshackle's working folders. Only the folders listed here can be moved; core package paths are protected and silently ignored if you try to override them.
```yaml title="unshackle.yaml"
directories:
downloads: "~/Media/unshackle"
temp: "/mnt/fast/unshackle-temp"
cache: "~/.cache/unshackle"
cookies: "~/.config/unshackle/cookies"
logs: "~/.config/unshackle/logs"
```
| Name | Purpose |
|---|---|
| `downloads` | Default output directory for finished files. |
| `temp` | Working directory for in-progress downloads, muxing, and intermediate files. |
| `cache` | Title/HTTP cache and update-check state. |
| `cookies` | Per-service cookie files. |
| `logs` | Log files. |
| `exports` | Export JSONs. |
| `services`, `vaults`, `fonts`, `commands` | Search paths for services, key-vault backends, bundled fonts, and CLI commands. |
| `wvds`, `prds`, `dcsl` | Widevine devices, PlayReady devices, and DCSL data. |
Paths support `~` expansion. The names `app_dirs`, `core_dir`, `namespace_dir`, `user_configs`, and `data` are protected and cannot be changed.
### Filename patterns
The `filenames` key overrides the naming patterns for a handful of internal files. These are **not** your media output names. They are logs, temp files, and per-service configs.
```yaml title="unshackle.yaml"
filenames:
log: "unshackle_{name}_{time}.log"
chapters: "Chapters_{title}_{random}.txt"
subtitle: "Subtitle_{id}_{language}.srt"
```
| Name | Default | Where |
|---|---|---|
| `log` | `unshackle_{name}_{time}.log` | `directories.logs` |
| `debug_log` | `unshackle_debug_{service}_{time}.jsonl` | `directories.logs` |
| `config` | `config.yaml` | per-service directory |
| `root_config` | `unshackle.yaml` | main config filename |
| `chapters` | `Chapters_{title}_{random}.txt` | `directories.temp` |
| `subtitle` | `Subtitle_{id}_{language}.srt` | `directories.temp` |
### Chapters naming {#chapters-naming}
When a title has chapters, they are written to an OGM chapters file (`filenames.chapters`) and passed to the muxer. Unnamed chapters use the `chapter_fallback_name` template:
```yaml title="unshackle.yaml"
chapter_fallback_name: "Chapter {i:02}"
```
The fallback supports two placeholders: `{i}` (chapter number, starting at 1) and `{j}` (which increments only for unnamed chapters). Standard format directives like `{i:02}` are allowed.
## Removed & migrated options
!!! warning "`scene_naming` was removed"
The old `scene_naming` option no longer exists. If it is present in your config, unshackle exits immediately with an error asking you to configure `output_template` instead. Migrate any scene-naming setup to the templates described on this page.
## Worked examples
=== "Movies"
```yaml
output_template:
movies: "{title}.{year}.{edition?}.{quality}.{source}.WEB-DL.{audio_full}.{atmos?}.{video}.{hdr?}-{tag?}"
folder:
movies: "{title} ({year})"
```
```text
The Movie (2024)/
└─ The.Movie.2024.1080p.NF.WEB-DL.DDP5.1.Atmos.H.265.HDR-TAG.mkv
```
=== "TV series"
```yaml
output_template:
series: "{title}.{season_episode}.{episode_name?}.{quality}.{source}.WEB-DL.{audio_full}.{video}-{tag}"
folder:
series: "{title} ({year})/Season {season}"
```
```text
The Show (2023)/Season S02/
└─ The.Show.S02E04.The.Reckoning.2160p.AMZN.WEB-DL.DDP5.1.H.265-TAG.mkv
```
=== "Music"
```yaml
output_template:
songs: "{track_number}. {title}"
folder:
albums: "{album_artist} - {album} ({year})"
```
```text
The Artist - The Album (2024)/
└─ 01. Opening Track.mka
```
## See also
- [Configuration File](../getting-started/configuration-file.md): the full `unshackle.yaml` structure.
- [Downloading](downloading.md): the `dl` command and per-run flags like `-o`, `--no-folder`, `--tag`, and `--repack`.
docs/guide/proxies-and-vpn.md
+639
View File
@@ -0,0 +1,639 @@
# Proxies & VPN
Most streaming services are geo-restricted, region-priced, or simply refuse to hand
over a manifest from the "wrong" country. unshackle routes its traffic through a proxy
so the service sees the location you choose instead of your own. You can point it at a
single proxy URL, or configure one of several **proxy providers** (commercial VPNs and
self-hosted tunnels), then ask for a country by its two-letter code and let
unshackle pick a working server for you.
The `--proxy` flag controls proxying, and a query is resolved to a real proxy URL by the
providers unshackle supports: [basic static proxies](#basic-static-proxies),
[Gluetun](#gluetun) (documented in the most detail, since it is the most flexible),
[ExpressVPN](#expressvpn), [NordVPN](#nordvpn), [Proton VPN](#proton-vpn),
[Hola](#hola), [Surfshark](#surfshark), and [Windscribe](#windscribe).
!!! note "Where proxies are configured"
All provider configuration lives under the top-level `proxy_providers:` key in your
`unshackle.yaml`. Each provider has its own sub-key, and the keys inside it map
directly onto that provider's settings. See the
[Configuration file](../getting-started/configuration-file.md) page for where the
file lives and how it is loaded.
## The `--proxy` flag
Proxying is controlled at download time by three options on the `dl` command:
| Option | Effect |
|---|---|
| `--proxy` | The proxy to use. Either an explicit URI, or a query that unshackle resolves against your configured providers. |
| `--no-proxy` | Force **all** proxy use off. No providers are initialised and no proxy query is resolved. |
| `--no-proxy-download` | Use the proxy for the manifest, licence, and authentication, but bypass it for the **segment downloads** themselves. |
```shell title="Explicit proxy URI"
unshackle dl --proxy http://user:pass@1.2.3.4:8080 NF 81234567
```
```shell title="Resolve a US server from your configured providers"
unshackle dl --proxy us NF 81234567
```
```shell title="Target a specific provider"
unshackle dl --proxy nordvpn:us NF 81234567
```
!!! tip "Set a default proxy in config"
Like every `dl` flag, `--proxy` can be given a default in your configuration so you
do not have to pass it on every command. See
[Downloading](downloading.md) for how `dl` options and config defaults interact.
### `--no-proxy-download`
Segment downloads are usually the largest share of traffic, and a residential-grade VPN
proxy is often the slowest link in the chain. `--no-proxy-download` lets you keep the
proxy where it matters (passing the geo-check for the manifest and the licence) while
pulling the actual audio and video segments over your normal connection for full speed.
```shell title="Proxy the manifest and licence, download segments direct"
unshackle dl --proxy gb --no-proxy-download ITV 10a1234
```
!!! warning "When this is unsafe"
Some services tie segment delivery (CDN tokens, per-segment geo-checks) to the same
region as the manifest. On those services, bypassing the proxy for downloads will
cause segment fetches to fail or return the wrong region. If downloads break with
`--no-proxy-download`, remove it.
## How proxy resolution works
The value you pass to `--proxy` is interpreted in one of three ways. unshackle checks
them in order.
```mermaid
flowchart TD
A["--proxy value"] --> B{"Looks like a URI?<br/>http:// https:// socks"}
B -- yes --> C["Use it verbatim"]
B -- no --> D{"Has a provider prefix?<br/>provider:query"}
D -- yes --> E["Ask that one provider<br/>for the query"]
D -- no --> F["Ask every provider in order,<br/>first match wins"]
```
1. **An explicit URI.** Anything shaped like `http://…`, `https://…`, or a `socks…`
URI is used exactly as given. unshackle logs `Using explicit Proxy: …` and does no
lookup.
2. **A provider-prefixed query**: `provider:query`, for example `nordvpn:us` or
`gluetun:windscribe:us`. unshackle finds the provider whose name matches the prefix
(case-insensitive) and asks *only* that provider to resolve the remainder. If no such
provider is loaded, or it returns no proxy, the download errors.
3. **A bare query**: a region like `us`, `gb`, `us:seattle`, or `us1234`. unshackle
asks each loaded provider **in order** and uses the first proxy any of them returns.
The query is matched against a region grammar (a two-letter country code, optionally
followed by a server number and/or a `:city` or `-city` part), lowercased, and then
handed to the provider. The exact forms each provider accepts are documented in its
section below.
### Provider load order
When you run a download without `--no-proxy`, unshackle instantiates every configured
provider once, in this fixed order, and that same order is used for bare-query
resolution:
**Basic → ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola**
So if you have both Basic and NordVPN configured and run `--proxy us`, a `us` entry in
your Basic config wins because Basic is tried first. To skip ahead to a specific
provider, prefix the query (`--proxy nordvpn:us`).
!!! note "Auto-loading providers"
Most providers only load when you configure them under `proxy_providers:`. Three are
special: **ExpressVPN** and **Proton VPN** also load automatically if their cookie
file already exists on disk (even with no YAML), and **Hola** loads automatically
whenever the `hola-proxy` binary is found on your `PATH`.
### Connection feedback
After a provider resolves a query, unshackle prints a line so you can confirm
where you came out:
- **Gluetun** reports the verified exit IP, country, and city of the container.
- **ExpressVPN** and **Proton VPN** print a location summary such as
`(USA - New York, #3 of 5): .214`.
- Other providers log the resolved proxy URL.
## Basic (static proxies)
The `Basic` provider is pure static configuration, with no accounts and no network calls. You
list proxy URLs per country code, and unshackle serves them back when you query that
country. This is the right choice when you already have proxy endpoints (from a proxy
seller, your own servers, or a corporate gateway).
Under each two-letter country code you can put either a single proxy string or a list of
them:
```yaml title="unshackle.yaml"
proxy_providers:
basic:
us: http://user:pass@1.2.3.4:8080
de:
- http://a.example:8080
- socks5://b.example:1080
```
**Query forms:**
| Query | Meaning |
|---|---|
| `us` | A proxy for the US. If several are listed, one is chosen at random. |
| `us2` | The **2nd** entry in the US list (1-based). |
- If a country has a single proxy string, that proxy is always returned.
- If it has a list and you give a plain country code, one is picked at random; append an
index (`de1`, `de2`) to pin a specific entry.
- Each URL is normalised (a missing scheme defaults to `http`) and validated, so a
malformed URI is rejected up front rather than failing mid-download.
```shell
unshackle dl --proxy de2 NF 81234567
```
## Gluetun
Gluetun is unshackle's most flexible proxy backend, and the one to reach for if you want
one configuration to cover many VPN providers. Instead of returning a remote proxy URL,
Gluetun **launches a local Docker container** running the
[`qmcgaw/gluetun`](https://github.com/qdm12/gluetun) image. That container establishes a
WireGuard or OpenVPN tunnel to your chosen VPN provider and exposes a plain HTTP proxy on
`localhost`, which unshackle then routes traffic through. When the download finishes, the
container is torn down automatically.
This means you get access to the **50+ VPN providers** that Gluetun supports, using your own
VPN subscription, through a single uniform interface, with no per-provider integration
in unshackle itself.
!!! warning "Docker is required"
Gluetun needs a working [Docker](https://docs.docker.com/get-docker/) installation on
your `PATH`. If Docker is not found, the provider fails to load with an error linking
to the install docs. The container also needs the `NET_ADMIN` capability and access
to `/dev/net/tun` to bring up the tunnel. unshackle requests both automatically.
### Query format
Gluetun uses a nested selector: the provider prefix picks Gluetun, and the rest names the
**VPN provider** and the **region**.
```
gluetun:<vpn-provider>:<region>
```
```shell title="A Windscribe US exit via Gluetun"
unshackle dl --proxy gluetun:windscribe:us NF 81234567
```
```shell title="A NordVPN Germany exit via Gluetun"
unshackle dl --proxy gluetun:nordvpn:de NF 81234567
```
After unshackle strips the leading `gluetun:`, the provider receives exactly
`provider:region` (two colon-separated parts; anything else is an error). The
`<vpn-provider>` must be one you have configured under `gluetun.providers`, and friendly
names are normalised to Gluetun's identifiers (for example `pia` and
`privateinternetaccess` both map to `private internet access`).
### Configuration
Each VPN provider you want to use gets an entry under `providers:`, carrying its VPN type,
credentials, and how your region aliases map onto Gluetun's server selectors.
```yaml title="unshackle.yaml"
proxy_providers:
gluetun:
providers:
windscribe:
vpn_type: wireguard
credentials:
private_key: YOUR_WIREGUARD_PRIVATE_KEY
addresses: YOUR_WIREGUARD_ADDRESS
preshared_key: YOUR_PRESHARED_KEY
server_countries:
us: US
uk: GB
server_cities: {}
server_hostnames: {}
extra_env: {}
nordvpn:
vpn_type: wireguard
credentials:
private_key: YOUR_WIREGUARD_PRIVATE_KEY
server_countries:
us: US
de: DE
# Global settings (all optional)
base_port: 8888
auto_cleanup: true
container_prefix: unshackle-gluetun
auth_user: null
auth_password: null
verify_ip: true
```
**Per-provider keys:**
| Key | Purpose |
|---|---|
| `vpn_type` | `wireguard` (default) or `openvpn`. |
| `credentials` | The secrets for the tunnel; see the credential rules below. |
| `server_countries` | Maps your region alias (e.g. `us`) to Gluetun's country value (e.g. `US`). |
| `server_cities` | Optional. Maps an alias to a Gluetun city. |
| `server_hostnames` | Optional. Maps an alias directly to a specific server hostname. |
| `extra_env` | Optional. Raw environment variables merged into the container last, for any Gluetun setting not covered above. |
**Credential rules by VPN type:**
=== "WireGuard"
- `private_key` is **always** required.
- `surfshark`, `mullvad`, and `ivpn` also require `addresses`.
- `windscribe` additionally requires **both** `addresses` and `preshared_key`
(`preshared_key` may be an empty string, but the key must be present).
- NordVPN and Proton VPN need only `private_key`.
=== "OpenVPN"
- Requires `username` and `password`.
!!! tip "Which `vpn_type` should I pick?"
WireGuard is the default because it **can be faster**, but it needs the fiddlier
credential setup above (private keys, and for some providers `addresses` and a
preshared key). OpenVPN is the **simplest** to configure (just a username and
password) at the cost of some speed. If you are getting started or hitting credential
trouble, OpenVPN is the path of least resistance; switch to WireGuard when you want the
extra throughput.
**Global keys:**
| Key | Default | Purpose |
|---|---|---|
| `base_port` | `8888` | The first local port to expose proxies on. Additional containers take the next free ports. |
| `auto_cleanup` | `true` | Remove containers on exit. If `false`, containers are only stopped, not deleted. |
| `container_prefix` | `unshackle-gluetun` | Prefix for Docker container names. |
| `auth_user` / `auth_password` | `null` | Optional HTTP-proxy credentials. If set, the local proxy requires auth and the credentials are embedded in the proxy URL. |
| `verify_ip` | `true` | After connecting, verify the container's real exit IP and region. |
### Region selection
For a given query region, Gluetun resolves the server like this:
1. An explicit entry in `server_hostnames`, `server_cities`, or `server_countries` for
that alias always wins.
2. A `us1239`-style query (country code + number) with no explicit mapping is turned
into a provider-specific server hostname (for example `us1239.nordvpn.com` for
NordVPN, `us-1239.prod.surfshark.com` for Surfshark).
3. A bare two-letter code is expanded to the full country name Gluetun expects.
A few providers (Windscribe, VyprVPN, VPN Secure) select by **region** rather than
country, and Windscribe region names are translated automatically. For instance `us`
becomes `US East` and `uk`/`gb` become `United Kingdom`.
### Container lifecycle
Understanding what Gluetun does behind the scenes helps when something goes wrong:
1. **Image check.** `qmcgaw/gluetun:latest` is pulled if it is not already present.
2. **Reuse.** If a container for this exact `provider:region` is already running (from a
concurrent unshackle session), it is adopted instead of starting a new one. Stopped or
dead containers with the same name are pruned first.
!!! note "Expect a slow first request, then instant reuse"
Bringing up a fresh tunnel takes roughly **10 to 30 seconds** for the first request to a
given `provider:region`. This is the VPN handshake and readiness wait, not a hang.
Every subsequent request that reuses that same container is effectively **instant**. A
slow first run is normal.
3. **Port allocation.** A free local port is chosen starting from `base_port`.
4. **Start.** The container is launched bound to `127.0.0.1:{port}`. Credentials are
passed through a temporary, `0600`-permission env-file (never as `-e` flags), which is
overwritten and deleted immediately after.
5. **Readiness wait.** unshackle polls the container logs (up to 60 seconds) until both
the HTTP proxy is listening **and** the VPN reports a completed connection. Fatal
errors like `authentication failed` or `invalid credentials` abort early, and recent
logs are attached to the error to help you diagnose it.
6. **IP verification.** When `verify_ip` is on, unshackle queries the container's exit IP
and checks the country matches what you asked for, raising a *region mismatch* error if
it does not.
7. **Proxy handed back.** You get `http://localhost:{port}` (or with auth credentials if
configured).
!!! note "Cleanup and Ctrl+C"
Containers are cleaned up via an `atexit` handler on normal exit. Gluetun deliberately
does **not** install its own signal handlers, so `Ctrl+C` continues to work as you
expect. If a run is killed hard (e.g. `kill -9`) a container can be left behind; with
`auto_cleanup: true` the next run adopting that name will still work, or you can remove
it manually with `docker rm -f`.
### Troubleshooting
!!! example "Common Gluetun issues"
- **`RuntimeError` about Docker not found.** Install Docker and confirm `docker` is
on your `PATH`.
- **`invalid credentials` / `authentication failed` in the readiness wait.** Your
WireGuard/OpenVPN credentials are wrong or in the wrong fields. Double-check
`private_key`, `addresses`, and (for Windscribe) `preshared_key`.
- **Region mismatch error.** The VPN connected but exited in a different country than
requested. Check your `server_countries` mapping, or set `verify_ip: false` if you
are intentionally using a nearby region.
- **Timeout after 60 seconds.** The tunnel never came up. Inspect the container logs
(`docker logs <container>`) for the underlying VPN error.
## ExpressVPN
ExpressVPN emulates the flow used by the ExpressVPN browser extension: it authenticates
with cookies (or a cached token), obtains proxy-capable locations from ExpressVPN's API,
and returns an authenticated HTTPS proxy. Because it relies on browser cookies rather than
service credentials, setup is slightly more involved than the credential-based providers.
ExpressVPN **auto-loads** if its cookie file exists, so once the cookie is in place you do
not strictly need a YAML block, but you can add one to pin regions or servers.
**Default file locations:**
| File | Default path |
|---|---|
| Browser cookies | `{cookies}/vpn/expressvpn.txt` |
| Token cache | `{cache}/global/expressvpn_tokens.json` |
Where `{cookies}` and `{cache}` are your configured cookie and cache directories. The
cookie file may be a Netscape `cookies.txt`, a JSON cookie array, or a JSON dict. Export
your ExpressVPN session cookies from a logged-in browser into one of those forms.
```yaml title="unshackle.yaml (optional)"
proxy_providers:
expressvpn:
region_map:
us: ny-02 # prefer New York, server 02 for the US
gb: # empty = smart random location
server_map:
myserver: usny-newyork-2
```
**Query forms:**
| Query | Meaning |
|---|---|
| `us` | A smart, random US location. |
| `us-ny` | US, New York. |
| `us-ny-2` / `us-ny2` | US, New York, the 2nd server. |
| `usny-newyork-2` | A full location slug. |
| `hostname.expressprovider.com` | A direct endpoint hostname. |
The resolved proxy is an HTTPS proxy on port `443`. On success, unshackle prints a
location summary such as `(USA - New York, #3 of 5): .214`.
!!! note "Cache safety"
ExpressVPN's token cache is written with `0600` permissions so the refresh token is
never briefly world-readable.
## NordVPN
NordVPN uses your **Service Credentials**, not your normal NordVPN login. Generate them
from the [NordVPN dashboard](https://my.nordaccount.com/dashboard/nordvpn/) under manual
setup; they are a long random username/password pair, not your email and password.
```yaml title="unshackle.yaml"
proxy_providers:
nordvpn:
username: YOUR_SERVICE_USERNAME
password: YOUR_SERVICE_PASSWORD
server_map: # optional: pin specific server IDs
us: 1234
us:seattle: 5678
```
**Required keys:** `username`, `password`. unshackle validates them up front: the
combined username and password must be exactly 48 lowercase alphanumeric characters, and
the username must not contain an `@`. If validation fails, you have almost certainly used
your login credentials instead of your service credentials.
`server_map` is optional and maps a region (or `code:city`) to a pinned NordVPN server ID.
Its main use is tuning: for a given streaming service one specific server often performs
noticeably better than NordVPN's recommended pick, so hard-coding that server for a day or
two is a handy workaround until it degrades and you swap in another.
!!! note "Use `gb`, not `uk`, for the United Kingdom"
Query the UK as `gb`. unshackle uses the Alpha-2 country code `gb` deliberately to stay
consistent with its other regional/country-code systems and avoid the language-vs-
country-code confusion that `uk` invites.
**Query forms:**
| Query | Meaning |
|---|---|
| `us` | A recommended US server. |
| `us1234` | The specific server `us1234`. |
| `us:seattle` | A recommended server in Seattle. |
| `228` | A NordVPN numeric country ID. |
The returned proxy is HTTPS on **port 89** (NordVPN disabled its plain-HTTP proxies on
port 80 in 2021).
```shell
unshackle dl --proxy nordvpn:us:seattle NF 81234567
```
## Proton VPN
Proton VPN supports two authentication methods, and unshackle can work with either. Note
that free Proton accounts work but are limited to Proton's free-tier exit countries.
=== "TV login (recommended)"
A self-sustaining, refreshable session obtained through Proton's TV device-login flow.
This is the only method that can refresh itself headlessly, so it keeps working without
re-exporting anything. Enable it and run a download interactively once:
```yaml title="unshackle.yaml"
proxy_providers:
protonvpn:
enable: true
```
With `enable: true` and an interactive terminal, unshackle prints a code and asks you
to enter it at [account.proton.me/vpn/tv/code](https://account.proton.me/vpn/tv/code).
Once approved, the session is cached and reused automatically.
=== "Exported cookies"
Export your `account.proton.me` session cookies (the `AUTH-<UID>` cookie) to the Proton
cookie file. This method is **access-only**: it cannot refresh, so you must re-export
when it expires.
!!! warning "The `AUTH-<UID>` cookie is HttpOnly"
This is the single most common cause of a failed Proton cookie import: the required
`AUTH-<UID>` cookie is **HttpOnly**, and many one-click cookie-export extensions
silently skip HttpOnly cookies by default. If your import "doesn't work," use an
exporter that explicitly includes HttpOnly cookies.
!!! note "Why this session can't refresh, and lapses after ~24h"
Proton rotates browser refresh tokens and ties them to the live browser, so a
headless refresh is impossible. unshackle therefore deliberately uses only the
cookie's short-lived **access token** and never attempts a refresh, which is why the
session lapses at Proton's ~24h TTL and must be re-exported. This is exactly why TV
login is preferred for set-and-forget use.
**Default file locations:**
| File | Default path |
|---|---|
| Cookies | `{cookies}/vpn/protonvpn.txt` |
| Session cache | `{cache}/vpn/protonvpn.json` |
Proton VPN **auto-loads** if its cookie file exists. Like ExpressVPN, its cached session
is written with `0600` permissions.
!!! note "Cache vs cookie precedence"
If both a TV-login cache and a cookie file exist, the **cache is preferred** unless the
cookie file is newer. Cookies are read fresh every run and never written back to the
cache. So to deliberately force use of the cookie session, just re-save (re-export) the
cookie file so its modification time is newer than the cache.
**Query forms (after the `protonvpn:` prefix):**
| Query | Meaning |
|---|---|
| `us` | A random US server (paid tiers preferred over free). |
| `us12` | Proton US server #12. |
| `us:ny` | A server in New York. |
The returned proxy is HTTPS on **port 4443** (or **port 443** for Secure Core servers).
Secure Core and Tor servers are excluded from selection because they are too slow for a
plain proxy. On success, unshackle prints a summary such as `(Name - City): server-host`.
!!! tip "Proton server numbers aren't sequential"
When pinning a server with `protonvpn:deNN`, don't guess a low number. A country may
expose non-contiguous IDs like `#203`, `#813`, and so on. Copy the number from a
previous run's log line (e.g. `DE#203`) instead.
## Hola
Hola requires no configuration at all. It uses the
[`hola-proxy`](https://github.com/Snawoot/hola-proxy) binary, and **auto-loads** whenever
that binary is found on your `PATH`. If you query Hola but the binary is missing, the
provider raises an error telling you to install it.
```shell title="Install hola-proxy, then just query a country"
unshackle dl --proxy hola:us NF 81234567
```
**Query forms:** a two-letter country code (e.g. `us`, `gb`). unshackle asks `hola-proxy`
for available proxies in that country and picks one at random, returning an HTTP proxy.
!!! warning "Temporary bans"
Hola's free tier can rate-limit or temporarily ban you if queried too aggressively. If
you see a *temporary ban detected* error, wait before retrying. Hola currently uses
only datacenter proxies.
## Surfshark
Surfshark, like NordVPN, uses **Service Credentials** rather than your login. Generate
them from the [Surfshark manual-setup page](https://my.surfshark.com/vpn/manual-setup/main/openvpn).
```yaml title="unshackle.yaml"
proxy_providers:
surfsharkvpn:
username: YOUR_SERVICE_USERNAME
password: YOUR_SERVICE_PASSWORD
server_map: # optional
us: 1234
```
**Required keys:** `username`, `password`, validated with the same 48-character rule as
NordVPN (combined username+password must be 48 lowercase alphanumeric characters, no `@`).
`server_map` optionally pins server IDs per region.
**Query forms:**
| Query | Meaning |
|---|---|
| `us` | A random US server. |
| `us-bos` | A specific server. |
| `us:seattle` | A random server in Seattle. |
The returned proxy is HTTPS on **port 443**.
## Windscribe
Windscribe uses **Service Credentials** from its
[OpenVPN config page](https://windscribe.com/getconfig/openvpn). Only the presence of a
username and password is checked (there is no 48-character rule).
```yaml title="unshackle.yaml"
proxy_providers:
windscribevpn:
username: YOUR_SERVICE_USERNAME
password: YOUR_SERVICE_PASSWORD
server_map: # optional: alias -> hostname
us: us-east.totallyacdn.com
us:seattle: us-seattle.totallyacdn.com
```
**Required keys:** `username`, `password`. `server_map` optionally maps a region (or
`region:city`) directly to a hostname.
**Query forms:**
| Query | Meaning |
|---|---|
| `us` | A random US server. |
| `us150` / `sg007` | A specific numbered server. |
| `us:seattle` | A random server in Seattle. |
The returned proxy is HTTPS on **port 443**.
!!! warning "Windscribe is not loaded by the REST API resolver"
The standalone Windscribe provider (and Gluetun) are loaded by the `dl` command, but
**not** by the proxy resolver used by the REST API and remote-download client. If you
drive downloads through the API and need Windscribe
or Gluetun exits, resolve the proxy on the CLI side or use a different provider for API
jobs. See the developer note below.
## Developer notes
!!! note "Two resolution paths (developers)"
unshackle has **two** proxy-resolution code paths that are intentionally not
identical:
- `unshackle/commands/dl.py` inlines its own logic and loads **all eight** providers,
including Windscribe and Gluetun.
- `unshackle/core/proxies/resolve.py::initialize_proxy_providers()` is the shared
resolver used by the REST API handlers and the remote-service client. It **omits**
Windscribe and Gluetun.
Both paths accept the same query grammar (a direct URI, a `provider:query`, or a bare
country query tried against providers in order), but the set of available providers
differs. Keep this in mind when adding a provider: wiring it into `dl.py` does not make
it available over the API, and vice versa.
!!! note "Implementing a new provider (developers)"
Every provider subclasses `unshackle.core.proxies.proxy.Proxy` and implements three
methods:
- `__init__(self, **kwargs)` receives its `proxy_providers` sub-dict splatted as
keyword arguments, so YAML keys map 1:1 to constructor parameters. Do authentication
and catalog pre-fetching here.
- `__repr__(self)` is a human summary, conventionally `"{n} Countries ({m} Servers)"`.
- `get_proxy(self, query)` returns a Requests-compatible proxy URI, or `None` only
when the query was valid but no proxy exists. Raise on genuine errors or bad queries.
Optionally expose `get_connection_info(query)` (as Gluetun does) or
`last_connection_display()` (as ExpressVPN and Proton VPN do) to have `dl` print a
friendly connection line instead of the raw proxy URL.
## See also
- [Downloading](downloading.md): the full `dl` command guide, including where proxy
flags sit in the download flow.
- [Configuration file](../getting-started/configuration-file.md): the `unshackle.yaml`
reference, including directories for cookies and cache used by the cookie-based
providers.
- [Configuration reference](../reference/api/config.md): the developer-facing
configuration API, including the `proxy_providers` structure.
docs/guide/subtitles.md
+356
View File
@@ -0,0 +1,356 @@
# Subtitles
Subtitles are first-class tracks in unshackle. When you download a title, subtitle
tracks are parsed from the manifest alongside video and audio, downloaded, cleaned up,
optionally converted to the format you want, and then either muxed into the final
Matroska file or written out as sidecar files next to it. The sections below cover the
supported formats, how tracks are normalized after download, how to select and filter
subtitles, and every subtitle-related configuration option.
Most of this is controlled with a handful of `dl` flags and a small `subtitle` block in
your [configuration file](../getting-started/configuration-file.md). See
[Downloading](downloading.md) for how subtitle flags fit into a full download command.
## Supported formats
unshackle understands the following subtitle codecs. The **Format** column is the value
shown in track listings; the **Extension** column is the file extension used on disk.
| Codec (internal name) | Format | Extension | Notes |
| --- | --- | --- | --- |
| `SubRip` | SRT | `srt` | SubRip, the most widely compatible plain-text format |
| `SubStationAlpha` | SSA | `ssa` | SubStation Alpha (styled) |
| `SubStationAlphav4` | ASS | `ass` | Advanced SubStation Alpha (styled) |
| `TimedTextMarkupLang` | TTML | `ttml` | Timed Text Markup Language (IMSC/DFXP) |
| `WebVTT` | VTT | `vtt` | WebVTT |
| `SAMI` | SMI | `smi` | Synchronized Accessible Media Interchange |
| `MicroDVD` | SUB | `sub` | MicroDVD |
| `MPL2` | MPL2 | `mpl2` | MPL2 |
| `TMP` | TMP | `tmp` | TMP |
| `fTTML` | STPP | `stpp` | MPEG-DASH box-encapsulated TTML (IMSC1) |
| `fVTT` | WVTT | `wvtt` | MPEG-DASH box-encapsulated WebVTT |
!!! note "The two boxed formats convert automatically"
`fTTML` (STPP) and `fVTT` (WVTT) are the fragmented, MP4-box-encapsulated forms that
DASH and HLS deliver. You never keep these as-is: right after download they are
unwrapped into their plain counterparts (`fTTML` becomes TTML and `fVTT` becomes
WebVTT), so you always end up with an editable text subtitle. Note that this is a
one-way, source-only relationship: you can convert *from* these boxed formats, but no
backend can produce them as an output target, so they can never be a conversion
destination.
## What happens to a subtitle after download
A subtitle download does more than save a file. Once the bytes are on disk, unshackle runs
format-appropriate clean-up to produce a single, valid file:
- **Encoding**: non-boxed subtitles are forced to UTF-8, and stray `&lrm;` / `&rlm;`
bidirectional entities are unescaped.
- **Segment merging**: when a WebVTT or boxed subtitle arrives as many DASH/HLS
segments, the segments are stitched back into one continuous document with correct
timing.
- **WebVTT sanitization**: merged WebVTT runs through a repair pipeline that clamps
negative timestamps to `00:00:00.000`, removes stray cue identifiers (like `Q0`, `Q1`)
that confuse downstream parsers, and merges multi-line cues that a service split into
separate overlapping cues. By default the original styling and layout are preserved
(see [`preserve_formatting`](#subtitle-configuration)).
- **TTML for muxing**: Matroska cannot carry TTML. If you do not request a specific
output format, any plain TTML track is automatically converted to WebVTT before
muxing (the closest lossless equivalent). Boxed `fTTML` is likewise unwrapped to TTML
first.
You normally do not need to think about any of this. It just produces a clean file.
## Selecting and filtering subtitles
Subtitle selection happens through `dl` flags. The most important is language selection.
| Flag | Purpose |
| --- | --- |
| `-sl`, `--s-lang` | Language(s) wanted for subtitles. Defaults to `all`. |
| `--require-subs` | Require these languages to exist; if present, download **all** subtitles. Cannot be combined with `--s-lang`. |
| `-fs`, `--forced-subs` | Include forced subtitle tracks (excluded by default). |
| `--exact-lang` | Exact language matching, with no regional variants. |
| `-S`, `--subs-only` | Download only subtitle tracks. |
| `-ns`, `--no-subs` | Do not download subtitle tracks at all. |
| `--sub-format` | Output subtitle format / conversion target (see [conversion](#converting-subtitle-formats)). |
| `--skip-subtitle-errors` | Skip a subtitle that fails to download instead of aborting the whole title. |
### Choosing languages
By default (`--s-lang all`) unshackle keeps every subtitle language the service offers.
To narrow it down, pass a comma-separated list of language tags:
```shell title="English and Spanish subtitles only"
unshackle dl --s-lang en,es NF 81234567
```
Language matching is fuzzy by default: `en` will match `en-US`, `en-GB`, and similar
variants. If a requested language is missing but others were found, unshackle continues
with whatever remains and tells you which languages it dropped.
For strict matching, where `es-419` matches only `es-419` and never `es-ES`, add
`--exact-lang`:
```shell title="Only Latin-American Spanish, no other Spanish variants"
unshackle dl --s-lang es-419 --exact-lang DSNP 'https://www.disneyplus.com/…'
```
### Requiring languages before downloading
`--require-subs` is a gate rather than a filter. It downloads **all** available
subtitles, but only if every language you list is present; if any is missing the title
errors out. This is useful for batch jobs where a title without your must-have language
should fail loudly rather than silently produce a partial result.
```shell title="Fail unless both English and French subs exist, then grab everything"
unshackle dl --require-subs en,fr AMZN B0ABCDEFGH
```
!!! warning "`--require-subs` and `--s-lang` are mutually exclusive"
Because `--require-subs` forces downloading of all subtitle languages, it cannot be
combined with `--s-lang`. Choose one or the other.
### Forced subtitles
Forced tracks carry only the important on-screen signs and foreign-dialogue lines meant
to display even when you are watching in your own language. They are **excluded by
default**. Add `-fs` / `--forced-subs` to include them:
```shell title="Include forced tracks alongside full subtitles"
unshackle dl --s-lang en --forced-subs NF 81234567
```
## Track types: forced, SDH, and CC
Every subtitle track carries flags describing what kind of captions it holds. These
flags drive selection, sort order, output filenames, and the Matroska track flags.
- **Forced**: only signs and foreign dialogue; meant to play when the audio language
matches. Off by default; enable with `--forced-subs`.
- **SDH** (Subtitles for the Deaf and Hard-of-Hearing; also called HOH): a full
transcript that includes both dialogue and non-speech sound cues.
- **CC** (Closed Captions): captions originating from an EIA-608/708 broadcast stream,
often uppercase with `>>>` speaker markers and sound cues.
!!! note "SDH and CC are treated together for sorting and stripping"
Within a language, unshackle orders subtitles from fewest to most captions:
**Forced → Normal → SDH/CC**. For the purposes of hearing-impaired handling, SDH and
CC tracks are grouped as the "most captions" variant.
A single track cannot be both CC and SDH, and a forced track cannot also be flagged
CC or SDH. These combinations are rejected as invalid.
### Closed captions embedded in video
Some services do not ship subtitles as separate tracks but instead embed EIA-608/708
captions inside the video stream. After a video track is downloaded and decrypted,
unshackle probes it for these captions and, when found, extracts them into a SubRip
(`SRT`) subtitle flagged as CC. This runs automatically unless you disabled subtitles
(`--no-subs`) or video (`--no-video` / `--video-only` semantics), and it requires the
CCExtractor tool on your `PATH`.
## Automatic SDH stripping
When a language has an SDH track but **no** plain (non-SDH, non-forced) track, unshackle
can generate a clean non-SDH version for you by stripping the hearing-impaired cues:
sound effects, speaker labels, music notes, and similar. This is on by default and
controlled by the [`strip_sdh`](#subtitle-configuration) config key.
The check is deliberately loose: if a close-enough plain track already exists (for
example an `en-GB` non-SDH track when the SDH track is `en-US`), no stripped copy is
created. When a stripped copy *is* created, the original SDH track is kept as well, so
you end up with both.
The stripping engine itself is chosen by the [`sdh_method`](#subtitle-configuration)
config key. The default (`auto`) prefers the `subby` cleaner for SRT, falls back to
SubtitleEdit when it is installed, and otherwise uses the built-in `filter-subs`
approach.
!!! note "Why `convert_before_strip` exists"
`subby`'s SDH stripper only operates on SRT. Handed any other codec it returns
silently without stripping anything, not an error but a no-op that leaves the SDH
cues in place. This is the whole reason for [`convert_before_strip`](#subtitle-configuration):
non-SubtitleEdit engines need the subtitle converted to SRT first, or the
hearing-impaired cues survive the strip untouched.
## Converting subtitle formats
By default unshackle keeps each subtitle in its downloaded format (only auto-converting
TTML to WebVTT for muxing, as noted above). To force a specific output format, use
`--sub-format`:
```shell title="Convert every subtitle to SRT"
unshackle dl --s-lang en,es --sub-format srt NF 81234567
```
`--sub-format` accepts the common format names and aliases, including `srt`, `vtt`,
`ssa`, `ass`, and `ttml`. Pass `original` to explicitly keep the source format and skip
all conversion:
```shell title="Keep whatever the service delivered"
unshackle dl --sub-format original DSNP 'https://www.disneyplus.com/…'
```
Conversion only runs when the source format differs from the target, so
`--sub-format srt` on a track that is already SRT is a no-op.
!!! warning "Down-converting styled subtitles is lossy"
Converting styled SSA/ASS subtitles to SRT throws away positioning, colours, and
italics. unshackle never does this automatically. It only happens when you
explicitly ask for it via `--sub-format`. If you want to preserve styling, convert
to a format that supports it (or keep `original`).
Note that this automatic protection guards **only the default muxed track**: there the
conversion is skipped and the styled original is kept. It does **not** cover sidecars.
A `sidecar_format: srt` will still lossily flatten styled subtitles, dropping
positioning, colours, and styling, and a per-download `--sub-format srt` forces the
muxed track to convert as well. To keep raw styled sidecars, set
[`sidecar_format: original`](#subtitle-configuration).
### How the conversion backend is chosen
Under the hood unshackle picks the highest-fidelity converter that supports the
source→target pair. Which converter runs depends on the
[`conversion_method`](#subtitle-configuration) config key (default `auto`) and the
formats involved:
| Backend | Best at | Requires |
| --- | --- | --- |
| `subtitleedit` | Highest fidelity; preserves positioning and italics | SubtitleEdit installed |
| `subby` | Native SRT output with clean-up | — |
| `pysubs2` | Best fidelity for styled SSA/ASS | — (pure Python, always available) |
| `pycaption` | Last-resort fallback; flattens styling | — |
With `conversion_method: auto`, unshackle ranks these automatically per conversion.
Setting it to a specific value pins that backend as the first choice, falling back to
others only if the pin cannot handle the pair.
!!! note "Config wins over a service's preference"
A service may set a `preferred_conversion_method` on its own tracks (for example when
it ships subtitles that a particular backend handles best). An explicit
`conversion_method` in `unshackle.yaml` always overrides that per-track preference, so
your config takes precedence. This only matters when a service ships a non-default
preference; leave `conversion_method` at `auto` to let the service's hint stand.
## Muxed vs. sidecar output
By default subtitles are muxed **into** the final `.mkv`. You can instead (or
additionally) write them as separate sidecar files next to the video, controlled by the
[`output_mode`](#subtitle-configuration) config key:
=== "mux (default)"
```yaml title="unshackle.yaml"
subtitle:
output_mode: mux
```
Subtitles are embedded in the Matroska file. Nothing is written separately.
=== "sidecar"
```yaml title="unshackle.yaml"
subtitle:
output_mode: sidecar
sidecar_format: srt
```
Subtitles are written as standalone files and **not** muxed into the video (when
there is a video or audio track to sit beside).
=== "both"
```yaml title="unshackle.yaml"
subtitle:
output_mode: both
sidecar_format: srt
```
Subtitles are muxed into the video **and** written as sidecar files.
Sidecar files are named after the output file with the language and flags encoded in the
name, so players and media managers can match them automatically:
```text
Show.S01E01.1080p.WEB.en.srt
Show.S01E01.1080p.WEB.en.forced.srt
Show.S01E01.1080p.WEB.en.sdh.srt
```
The pattern is `{base}.{language}[.forced][.sdh].{extension}`. The sidecar file format
is set by [`sidecar_format`](#subtitle-configuration); use a format name like `srt` or
`vtt`, or `original` to keep each subtitle in its downloaded format.
## Fonts for styled subtitles
Styled SSA/ASS subtitles reference specific fonts by name. When unshackle muxes such a
subtitle it scans it for the fonts it uses and attaches the matching font files to the
Matroska output so the styling renders correctly on playback. If a referenced font is
not available locally, unshackle logs the missing font names and suggests installing the
relevant font packages.
## Subtitle configuration
All of the behaviour above can be defaulted in the `subtitle` block of your
[configuration file](../getting-started/configuration-file.md):
```yaml title="unshackle.yaml"
subtitle:
strip_sdh: true
preserve_formatting: true
conversion_method: auto
sdh_method: auto
convert_before_strip: true
output_mode: mux
sidecar_format: srt
```
| Key | Default | Values | Purpose |
| --- | --- | --- | --- |
| `strip_sdh` | `true` | `true` / `false` | Auto-generate a stripped non-SDH copy when a language has only an SDH track. |
| `preserve_formatting` | `true` | `true` / `false` | Keep original WebVTT styling and layout during sanitization. When `false`, WebVTT is re-encoded more aggressively (merging identical cues, dropping empty ones). |
| `conversion_method` | `auto` | `auto`, `subby`, `pysubs2`, `subtitleedit`, `pycaption` | Which conversion backend to prefer. `auto` ranks them per conversion. |
| `sdh_method` | `auto` | `auto`, `subby`, `subtitleedit`, `filter-subs` | Engine used to strip hearing-impaired cues. |
| `convert_before_strip` | `true` | `true` / `false` | Convert a subtitle to SRT before stripping SDH (for non-SubtitleEdit engines). |
| `output_mode` | `mux` | `mux`, `sidecar`, `both` | Whether subtitles are embedded, written as sidecars, or both. |
| `sidecar_format` | `srt` | a format name (e.g. `srt`, `vtt`) or `original` | Format used for sidecar files. |
!!! tip "SubtitleEdit is optional but recommended"
Several high-fidelity paths (the best-quality conversions and the SubtitleEdit SDH
stripper) only activate when the SubtitleEdit CLI is installed and on your `PATH`.
See [Installation](../getting-started/installation.md) for how to add it. Without it,
unshackle falls back to its built-in Python backends.
---
## Developer reference
!!! note "Audience"
This section is for people extending unshackle or writing services. Everyday users
can stop here.
The subtitle model lives in `unshackle.core.tracks.Subtitle`. A few internals worth
knowing when working with subtitle tracks in code:
- **Flags and validation**: `Subtitle(codec=..., cc=..., sdh=..., forced=...)`. A track
cannot be both `cc` and `sdh`, and `forced` cannot be combined with either; both raise
`ValueError` at construction.
- **`convert(codec, *, forced=False)`**: converts the downloaded file in place through
the backend chain in `subtitle_convert.run_conversion`. `forced=True` (what
`--sub-format` sets) is the only way lossy styled down-converts such as
ASS→SRT are permitted. Raises `NotImplementedError` if no backend supports the pair and
`RuntimeError` if all attempts fail.
- **`strip_hearing_impaired()`**: removes SDH cues using the engine selected by
`config.subtitle["sdh_method"]`.
- **`reverse_rtl()`**: fixes right-to-left sentence-ending positioning via SubtitleEdit
(requires the binary).
- **`parse(data, codec)`**: the central parser, returning a `pycaption.CaptionSet`; it
handles the boxed DASH formats, SAMI, and the WebVTT repair helpers.
- **`extract_fonts(text)`**: returns the set of font names referenced by an ASS/SSA
subtitle, used to decide which fonts to attach at mux time.
The conversion backends (`SubtitleEditBackend`, `SubbyBackend`, `Pysubs2Backend`,
`PycaptionBackend`) each implement a small protocol (`is_available()`, `can_convert()`,
`rank()`, `convert()`) and are tried in rank order, lowest rank first. See
[the Tracks API reference](../reference/api/tracks.md) for the full model.
docs/guide/troubleshooting.md
+303
View File
@@ -0,0 +1,303 @@
# Troubleshooting & Debugging
When a download fails, a service misbehaves, or a key won't resolve, unshackle gives you three tools to find out why:
- **Debug logging**: verbose `DEBUG`-level console output plus a structured, machine-readable log file.
- **[`unshackle env`](#checking-your-environment)**: verify that the external tools unshackle depends on are installed, and see exactly which directories and config file are in use.
- **Targeted toggles**: `debug_keys` and `debug_requests` for the two areas (DRM keys and raw HTTP traffic) that are deliberately quiet by default.
The sections below cover how to enable each of them, how to read what they produce, and which diagnostic fits which failure mode.
---
## Turning on debug logging
There are two independent ways to enable debug logging, and they do slightly different things.
=== "The `-d` / `--debug` flag"
The root-level flag turns on verbose logging for a single invocation:
```console
unshackle -d dl SERVICE TITLE_ID
```
It must come **before** the subcommand, because it is an option on the root `unshackle` command, not on `dl`.
`-d` does three things:
1. Sets the console log level to `DEBUG`, so every module logs its full detail.
2. Enables the structured JSON debug log file (see [below](#the-structured-json-debug-log)).
3. Adds source-path annotations to console log lines (which file and line each message came from).
=== "The `debug` config key"
To make debug logging the default for every run without typing `-d` each time, set it in your `unshackle.yaml`:
```yaml
debug: true
```
or via the [`cfg`](cli-reference.md#cfg) command:
```console
unshackle cfg debug true
```
!!! note "`debug: true` is not identical to `-d`"
The `debug` config key enables the **JSON debug log file**, but it does **not** raise the console to `DEBUG` level and does not add source-path annotations. Those only happen when you pass the `-d` flag. Use the flag when you want to *watch* the verbose output live; use the config key when you just want the log file written every time. Passing `-d` is the most complete option, and you can combine both.
!!! warning "Debug logging can be noisy and revealing"
Debug output is verbose and includes request metadata, service internals, and stack traces. Secrets, URLs, and local paths are automatically redacted (see [Redaction](#what-gets-redacted)), but you should still glance over a log before sharing it publicly.
---
## The structured JSON debug log
Beyond the console output, unshackle writes a structured log in [JSON Lines](https://jsonlines.org/) format: one complete JSON object per line. This file is designed to be filtered, searched, and analysed with standard tooling (`jq`, `grep`, a text editor), and it is safe to attach to a bug report because it is redacted.
### Where it is written
The debug log is created during a `dl` run (and by [`import`](cli-reference.md#import), which wraps `dl`). The path is:
```
<logs directory>/unshackle_debug_<service>_<timestamp>.jsonl
```
For example, `unshackle_debug_NETFLIX_20260703-142530.jsonl`. The `<service>` tag and the timestamp mean each download gets its own file, so runs never overwrite each other.
!!! tip "Find your logs directory"
The logs directory defaults to a `logs` folder inside the package data directory, but it can be relocated with the `directories.logs` config key. To see the exact resolved path on your machine, run [`unshackle env info`](#env-info) and read the **Directories** table.
### What a log entry looks like
Every line is a self-contained JSON object. The first line of a session records the environment:
```json
{"timestamp":"2026-07-03T14:25:30.123456+00:00","session_id":"a1b2c3d4","level":"INFO","operation":"session_start","message":"Debug logging session started","context":{"unshackle_version":"5.3.0","python_version":"3.12.4 ...","platform":"Linux-6.18.5-x86_64","platform_system":"Linux","platform_release":"6.18.5"}}
```
Subsequent lines describe operations: service calls, DRM licence requests, vault lookups, downloader progress, and errors. Common fields include:
| Field | Meaning |
|---|---|
| `timestamp` | UTC ISO-8601 time of the entry. |
| `session_id` | Short random ID shared by every entry in one run, handy for correlating lines. |
| `level` | `DEBUG`, `INFO`, `WARNING`, or `ERROR`. |
| `operation` | The action being logged, e.g. `service_call`, `download_init`, `drm_get_widevine_license`, `vault_get_key`, `session_end`. |
| `message` | Human-readable description (redacted). |
| `service` | Service tag the entry belongs to, when applicable. |
| `context` | Arbitrary structured detail for the operation. |
| `request` / `response` | HTTP method/URL and response metadata for network operations. |
| `duration_ms` | How long a timed operation took. |
| `success` | Whether the operation succeeded. |
| `error` | On failures: the exception `type`, `message`, and a full `traceback` array (all redacted). |
### Reading and filtering the log
Because each line is independent JSON, `jq` is the natural tool:
```console
# Show only error-level entries
jq 'select(.level == "ERROR")' unshackle_debug_NETFLIX_20260703-142530.jsonl
# List every operation and whether it succeeded
jq -c '{operation, success}' unshackle_debug_NETFLIX_20260703-142530.jsonl
# Pull the traceback out of the first error
jq -r 'select(.error) | .error.traceback[]' unshackle_debug_NETFLIX_20260703-142530.jsonl
# Show only DRM / licensing operations
jq 'select(.operation | startswith("drm_"))' unshackle_debug_NETFLIX_20260703-142530.jsonl
```
The session's binary-tool versions (Shaka Packager, mp4decrypt, mkvmerge, FFmpeg, FFprobe) are logged near the start of a `dl` run, which is often the fastest way to confirm a muxing or decryption failure is really a wrong-version problem.
---
## Debugging DRM keys with `debug_keys`
By default, content-decryption **keys are redacted** in the JSON debug log. Any field that looks like a key is written as `[REDACTED]`. This keeps logs shareable. When you are specifically debugging a key-retrieval or decryption problem and need to see the actual KID/KEY values that were fetched, enable `debug_keys`:
```yaml
debug_keys: true
```
```console
unshackle cfg debug_keys true
```
With `debug_keys: true`, the debug log stops masking key fields, so you can confirm which keys came back from a licence server or vault and compare them against what decryption expected.
!!! warning "Keys are sensitive, turn this back off"
`debug_keys` writes real decryption keys into the log file in cleartext. Only enable it while actively diagnosing a key issue, never share a log produced with it on, and set it back to `false` when you're done. Note that `debug_keys` only affects the **JSON debug log file**. It does not change what the console prints.
Fields that are *always* redacted regardless of `debug_keys` (anything whose name contains `password`, `token`, `secret`, `auth`, or `cookie`) stay masked, so credentials never leak even in a keys-on log.
---
## Debugging HTTP traffic with `debug_requests`
When you pass `-d`, unshackle raises its own modules to `DEBUG` but deliberately **silences** the noisy low-level HTTP libraries (`urllib3`, `requests`, `rnet`, `httpx`, `httpcore`, `hpack`, `h2`) back down to `WARNING`. Without this, a single download would bury the useful output under thousands of connection-pool and HTTP/2 frame messages.
If you need that low-level traffic (for example when a request is failing at the TLS or connection layer and you need to see the underlying library's own logs), enable `debug_requests`:
```yaml
debug_requests: true
```
```console
unshackle cfg debug_requests true
```
With `debug_requests: true`, those libraries are left at `DEBUG` level when you run with `-d`, so their request/response chatter appears in the console.
!!! note "`debug_requests` needs `-d` to have any effect"
The `debug_requests` toggle only decides whether the HTTP libraries are silenced when the `-d` flag is active. If you are not running with `-d` (verbose console logging), it changes nothing. Combine them: `unshackle -d dl ...` with `debug_requests: true` in config.
---
## What gets redacted
unshackle redacts logged strings so that debug output is safe to share. Three passes are applied to every logged message, error, and structured value:
- **Secrets**: passwords, tokens, API secrets, auth values, and cookies are replaced with `[REDACTED]`.
- **URLs**: every `http(s)` URL is collapsed to `redacted` (keeping only the file extension, so a manifest still shows as `redacted.mpd` and a segment as `redacted.m4s`). This hides content, CDN, manifest, and licence-server locations while preserving the *type* of resource.
- **Local paths**: install-root, virtualenv, and home-directory prefixes are replaced with tokens, so your username and machine layout don't appear in logs.
Path redaction is controlled by the `redact_paths` config key, which is **on by default**:
```yaml
redact_paths: true # default — mask local base directories in logged paths
```
Set `redact_paths: false` if you are debugging a path problem locally and want to see full, unmasked paths in the console and logs. Secret and URL redaction in the JSON debug log are always applied and are not affected by this toggle.
---
## Checking your environment
Many "download failed" problems are really missing external tools or a config file that isn't where you think it is. The [`unshackle env`](cli-reference.md#env) command group answers both questions before you dig into logs.
### `env check`
`unshackle env check` verifies every external binary unshackle can use and prints a table of what's installed:
```console
unshackle env check
```
It reports each tool's status (`✓` installed / `✗` missing), whether it is **required** or optional, and what it's for. A summary line tells you the installed-vs-total count and, if anything mandatory is absent, exactly which required tools are missing.
The required tools (unshackle will not get far without them) are:
| Tool | Purpose |
|---|---|
| **FFmpeg** | Media processing |
| **FFprobe** | Media analysis |
| **MKVToolNix** (`mkvmerge`) | MKV muxing |
| **mkvpropedit** | MKV metadata |
| **Shaka Packager** | DRM decryption |
Optional tools unlock extra capabilities. For example, `mp4decrypt` (alternative decryptor), `dovi_tool` / `HDR10Plus_tool` (Dolby Vision / HDR10+), `SubtitleEdit` / `CCExtractor` (subtitle conversion and CC extraction), `MPV` / `FFplay` (playback preview), and `git` (service repositories) or `docker` (Gluetun VPN, see [Proxies & VPN](proxies-and-vpn.md)).
!!! tip "First thing to run when something breaks"
If a decryption step errors, a mux fails, or a Dolby Vision title won't process, run `unshackle env check` first. A red `✗` next to the relevant tool is the answer far more often than a bug in unshackle.
### `env info`
`unshackle env info` shows where unshackle is reading configuration from and where every working directory lives:
```console
unshackle env info
```
- If a config file was found, it prints the path it was **loaded from**. If none was found, it lists every location it *would* accept one (see [Config file discovery](#why-is-my-config-not-being-picked-up) below), so you can create the file in the right place.
- It then prints a **Directories** table for every configured path: `downloads`, `temp`, `cache`, `cookies`, `logs`, `exports`, `wvds`, `prds`, the `services` search paths, and more.
This is the authoritative way to answer "where are my logs / downloads / cookies actually going?" The defaults are relative to the installed package, and any `directories:` overrides in your config are reflected here.
### `env clear`
Stale cache or leftover temp files can cause confusing behaviour (an old cached title list, a half-written temp file). `env clear` wipes those directories safely:
```console
unshackle env clear cache # clear the whole cache directory
unshackle env clear cache NETFLIX # clear just one service's cache subdirectory
unshackle env clear temp # clear the temp working directory
```
Each command reports how many files it removed and how much space it freed, then recreates the empty directory.
!!! tip "Cache-related weirdness"
If a service keeps returning stale titles or metadata even after the source changed, clear that service's cache with `unshackle env clear cache <SERVICE>`. See [Vaults](vaults.md) for the separate, persistent key-vault store, which `env clear` does **not** touch.
---
## Common failure modes and how to inspect them
!!! example "A required tool is missing"
**Symptom:** decryption, muxing, or analysis fails early, sometimes with a "binary not found" style error.
**Inspect:** run `unshackle env check` and look for a red `✗` on a **required** row. Install the missing tool and re-run.
!!! example "Decryption produces garbage or fails"
**Symptom:** the download completes but the file won't play, or the decrypt step errors.
**Inspect:** enable `debug_keys` and run with `-d`, then check the JSON debug log's `drm_*` and `vault_*` operations to confirm which keys were fetched and from where. Also confirm Shaka Packager (or `mp4decrypt`) shows a sane version in the `download_init` binary-versions entry. See [DRM & CDM](drm-and-cdm.md).
!!! example "A network request fails at the connection or TLS layer"
**Symptom:** timeouts, connection resets, or handshake errors that unshackle's own logs don't fully explain.
**Inspect:** set `debug_requests: true` and run with `-d` to surface the underlying `urllib3` / `rnet` / `httpx` logs. In the JSON debug log, filter for `service_call` operations to see which request failed. If a proxy is involved, cross-check [Proxies & VPN](proxies-and-vpn.md).
!!! example "The wrong titles/metadata keep coming back"
**Symptom:** a service returns outdated titles even after the source changed.
**Inspect:** the title cache is likely serving stale data. Clear it with `unshackle env clear cache <SERVICE>`, or bypass it for one run using the `dl` no-cache / reset-cache options (see [Downloading](downloading.md) and the [CLI Reference](cli-reference.md#dl)).
!!! example "A config value doesn't seem to apply"
**Symptom:** a setting you edited has no effect.
**Inspect:** run `unshackle env info` to confirm **which** config file is actually loaded, and `unshackle cfg --list` to dump the effective values. It's common to edit a config file in a location unshackle isn't reading from. See below.
---
## Why is my config not being picked up?
unshackle loads the **first** `unshackle.yaml` it finds, in this order:
1. `unshackle.yaml` inside the package's namespace directory.
2. `unshackle.yaml` in the parent of that directory.
3. `unshackle.yaml` in your OS user-config directory: on Linux `~/.config/unshackle/unshackle.yaml`, on Windows `%LOCALAPPDATA%\unshackle\unshackle.yaml`, on macOS `~/Library/Application Support/unshackle/unshackle.yaml`.
If you have config in more than one of these, an earlier one wins and your edits to a later file are silently ignored. `unshackle env info` prints the path that was actually loaded (or, if none exists, lists all three candidate locations so you can create the file correctly). See [Configuration](cli-reference.md#cfg) for editing values with the `cfg` command.
---
!!! note "Developer note: structured logging in code"
Service and command code emits entries through the shared `log_event(...)` helper (and the `timed_operation(...)` context manager for timing a block) from `unshackle.core.utilities`. These are no-ops when debug logging is disabled, so they are always safe to leave in place. When adding a new feature, call `log_event("my_feature_event", message="...", context={...})` rather than reimplementing the "is the logger enabled?" guard. The redaction and JSON-serialisation are handled for you, and any field whose name looks like a key/secret/token is redacted automatically (keys only appear when the run has `debug_keys` enabled).
### Logging conventions when adding entries
The debug log is tuned for **developers troubleshooting pipeline flow** (maximum signal, minimum noise), not for end users. A few conventions keep it that way. They are not enforced by the `log_event` signature, so they are easy to miss and worth stating explicitly.
!!! tip "Log levels are a flow skeleton, not just severity"
Treat `level` as structure, not decoration. Emit exactly **one `INFO` milestone per pipeline stage** so that `jq 'select(.level=="INFO")'` reads back the whole end-to-end flow of a run. Everything internal goes to `DEBUG`; failures go to `ERROR`. Keeping `INFO` sparse is the whole point. It is the line a developer skims first to see where a run got to before it broke.
```console
# Read the pipeline's high-level flow, one line per stage
jq 'select(.level == "INFO")' unshackle_debug_NETFLIX_20260703-142530.jsonl
```
!!! warning "No raw dumps"
Never log a full `Tracks` object, an MPD/manifest body, or a response payload. Log only **counts, ids, sizes, and `safe_display_url(url)`**. This keeps the log low-noise and prevents leaking content or manifest data into the file *before* redaction even runs. Redaction is a backstop, not a licence to dump raw objects.
!!! note "Operation names: `<area>_<event>`, no registry"
Operation names are plain lowercase strings following an `<area>_<event>` convention: `manifest_dash_parse`, `drm_decrypt`, `vault_get_key`, `tool_run`. They are written inline at each call site on purpose: there is **no central registry or enum**, and you should not go hunting for one or create one. The absence is a deliberate design choice, not an omission.
!!! note "Message reads alone; data lives in fields"
Give every entry a single-sentence `message` that stands on its own, and put all structured data (`context`, counts, ids, `duration_ms`) in **fields** rather than baking it into the prose. This separation is what lets the log be both human-scannable and machine-filterable; a message like `"Parsed DASH manifest"` with `context={"video": 4, "audio": 2}` beats `"Parsed manifest with 4 video and 2 audio tracks"`.
!!! tip "Let tool runs log themselves"
For external-tool calls (FFmpeg, mkvmerge, `dovi_tool`, and friends), route through `run_step` / `ffprobe`, which auto-emit a `tool_run` entry capturing the binary's version, return code, and duration consistently. Only fall back to calling `log_tool_run` manually for a direct `subprocess.run` you genuinely can't route through `run_step`. Auto-logging is the preferred path precisely because it captures those fields the same way every time.
docs/guide/vaults.md
+318
View File
@@ -0,0 +1,318 @@
# Key Vaults
A **key vault** is a store of DRM **content keys** that unshackle remembers between downloads. Once a title's keys have been recovered, unshackle saves them to your configured vaults. The next time you download something that reuses those same keys, unshackle finds them in a vault instead of performing another license exchange. The download starts faster and puts less load on the service.
Vaults are entirely optional, but if you download regularly they are one of the biggest quality-of-life improvements you can configure.
!!! tip "Vaults save license calls even on the very first download"
The benefit is not limited to repeat downloads. Within a single run, a service often issues the *same* Key ID and content key for both the video and audio tracks, and for several resolutions or bitrates of the same title. Once unshackle has recovered a key it writes it to your vaults immediately, so the moment the next track presents a KID it has already seen, the key is served from the vault instead of triggering another license exchange. A single fetched key can therefore satisfy many tracks in one run.
## What a vault stores
Every key is stored as a **KID → KEY** pair, grouped by **service**:
- **KID** (Key ID): a 16-byte identifier for a specific encrypted stream, stored as 32 lowercase hex characters with no dashes.
- **KEY**: the 16-byte content key that decrypts that stream, also 32 hex characters.
- **Service**: the canonical service tag (for example `DSNP`, not `DisneyPlus`) that namespaces the key. Each service gets its own logical table.
!!! note "Why keys are matched by KID"
Vaults match keys by KID, never by PSSH/CENC header. A KID does not change unless the underlying video file itself changes, whereas the PSSH box can vary between requests for the very same content. That makes the KID a far more reliable match key.
!!! info "Null keys are ignored everywhere"
A key made entirely of zeroes (`00000000000000000000000000000000`) is treated as "no key". Vault lookups skip it, and every backend raises an error if you try to add one. This prevents a placeholder key from masking a real one.
## How lookups and caching work
When unshackle needs a key during a download it asks your vaults **in the order they are listed in your config**. It returns the first non-null key it finds and stops searching. If a vault raises a permission error or is unreachable, that vault is skipped and the search continues with the next one.
When new keys are recovered from a license, unshackle pushes them to **all** configured vaults, so every vault gradually converges on the same set of keys. Two rules apply to pushing:
- Keys that already exist in a vault are skipped (existing data is never overwritten or deleted).
- A vault marked `no_push: true` receives lookups but is never written to. This is useful for read-only or shared upstream vaults you do not own.
!!! tip "A good two-vault setup"
A common arrangement is a fast local **SQLite** vault plus a shared remote vault (**MySQL**, **HTTP**, or **API**). The local vault answers instantly and works offline; the remote vault lets you share keys across machines or with a group.
## Configuring vaults
Vaults are declared under the `key_vaults` list in your unshackle config file. Each entry is a mapping with two required fields, `type` (which backend to use) and `name` (a label you choose), plus whatever settings that backend needs.
```yaml title="unshackle.yaml"
key_vaults:
- type: SQLite
name: local
path: ~/.local/share/unshackle/vaults/local.db
- type: MySQL
name: shared
host: vault.example.com
database: unshackle
username: reader
password: "s3cret"
```
The `type` value is matched to a vault backend by name. The four backends that ship with unshackle are `SQLite`, `MySQL`, `HTTP`, and `API`, described below.
!!! warning "SQLite vaults are loaded critically"
During a download, a **SQLite** vault that fails to load will **abort the run**. The assumption is that a local database you configured should always be available. `MySQL`, `HTTP`, and `API` vaults are loaded leniently: if one fails to load it is logged and skipped, and the download continues with the remaining vaults.
### `vault_timeout`
Network-backed vaults (`HTTP` and `API`) accept an optional per-vault `timeout` in seconds. If you omit it, unshackle injects the global default:
```yaml title="unshackle.yaml"
vault_timeout: 10.0 # default, in seconds
```
A `timeout` set inline on an individual vault always wins over this global value. The `SQLite` and `MySQL` backends do not take a `timeout` field.
---
## SQLite backend
A locally-accessed SQLite database file. This is the simplest and fastest option and needs no server.
```yaml title="unshackle.yaml"
key_vaults:
- type: SQLite
name: local
path: ~/.local/share/unshackle/vaults/local.db
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `SQLite`. |
| `name` | yes | A label for this vault. |
| `path` | yes | Path to the database file. `~` is expanded. The file (and its tables) are created automatically on first use. |
| `no_push` | no | If `true`, keys are read but never written. Defaults to `false`. |
Each service gets its own table (named after the service tag), created on demand with `kid` and `key_` columns. Connections use WAL journaling with a 30-second busy timeout, so the same database file can be used safely from multiple threads.
!!! warning "Moving the .db file silently forks your vault"
Because the file is created on demand, if you relocate an existing `.db` without updating `path` to match, unshackle will simply create a fresh empty database at the old location, leaving you with two divergent vaults and no error to warn you. Always update the config path whenever you move the file.
!!! tip "Keep a local SQLite vault even when you share a MySQL one"
The SQLite vault is best thought of as an offline-only backup should anything ever happen to a shared `MySQL` vault. Each member of a group should keep their own local SQLite vault alongside the shared database, so no one loses their keys if the shared vault goes down or away.
---
## MySQL backend
A remotely-accessed MySQL/MariaDB database, ideal for sharing keys across several machines. It connects via `pymysql`. MariaDB is recommended over MySQL where you have the choice.
!!! warning "The vault connects directly to the database server"
A `MySQL` vault (like `SQLite`) speaks the database wire protocol directly to the host or IP you configure. It cannot sit behind a PHP API or any other application layer. A common real-world snag: many hosting providers firewall their MySQL server to their own internal network, so a database on shared hosting is frequently unreachable from the outside world even when your credentials are perfectly correct. If connections time out, confirm the server accepts remote connections from your address before suspecting the config.
```yaml title="unshackle.yaml"
key_vaults:
- type: MySQL
name: shared
host: vault.example.com
database: unshackle
username: unshackle
password: "your-password"
port: 3306
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `MySQL`. |
| `name` | yes | A label for this vault. |
| `host` | yes | Database server hostname or IP. |
| `database` | yes | Database (schema) name. |
| `username` | yes | Login user. |
| `no_push` | no | If `true`, keys are read but never written. |
Any additional fields you provide (such as `password`, `port`, or `ssl`) are passed straight through to `pymysql.connect`, so you can use the full range of its connection options.
!!! warning "Account security is on you, not on unshackle"
unshackle enforces none of this. It is operational policy for a database that may be shared with a group. Never connect with the `root` account, not even for yourself. Give every account a password, and never let two users share a username/password. Restrict each account's access to only the `unshackle` database, and grant normal users just `SELECT` and `INSERT`. Reserve the `CREATE` grant for a single trusted user who bootstraps the service tables (see [`kv prepare`](#kv-prepare-pre-create-service-tables)); everyone else can insert new keys but should not be creating tables.
!!! note "Grants determine what the vault can do"
On connection the backend reads its own `SHOW GRANTS`. It requires at least `SELECT` to load at all (a missing `SELECT` grant makes the vault fail to load). Writing keys additionally needs `INSERT`, and creating a new service table needs `CREATE`. A read-only account still works fine as a lookup source. It simply cannot push new keys. Because `MySQL` is loaded leniently during downloads, a permission problem will not abort the run.
---
## HTTP backend
A flexible HTTP vault that speaks one of three wire protocols, selected with `api_mode`. Use this to integrate with a community or self-hosted vault service.
| `api_mode` | Transport | Authentication | Notes |
|------------|-----------|----------------|-------|
| `query` (default) | `GET` with query parameters | `username` + `password` | Full read and write. |
| `json` | `POST` with a JSON payload | `password` used as a token | Read and write per key; cannot bulk-enumerate keys. |
| `decrypt_labs` | `POST` in DecryptLabs format | `api_key` sent as a header | **Read-only**. `no_push` is forced on. |
=== "query mode"
```yaml title="unshackle.yaml"
key_vaults:
- type: HTTP
name: community
host: https://vault.example.com/api
api_mode: query
username: myuser
password: "your-password"
```
`username` is required in query mode. Both the username and password are sent as query parameters on each request.
=== "json mode"
```yaml title="unshackle.yaml"
key_vaults:
- type: HTTP
name: jsonvault
host: https://vault.example.com/rpc
api_mode: json
password: "your-api-token"
```
Requests are `POST`ed as `{"method": ..., "params": ..., "token": ...}` using methods like `GetKey`, `InsertKey`, and `GetServices`. This mode cannot list all keys in bulk, so a `kv copy` from a json vault relies on the remote server's own duplicate handling.
=== "decrypt_labs mode"
```yaml title="unshackle.yaml"
key_vaults:
- type: HTTP
name: decrypt_labs
host: https://api.decryptlabs.example/
api_mode: decrypt_labs
api_key: "your-decrypt-labs-key"
```
A read-only lookup source. The key is sent in a `decrypt-labs-api-key` header and the vault refuses all writes.
| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `HTTP`. |
| `name` | yes | A label for this vault. |
| `host` | yes | Base URL of the vault API. |
| `api_mode` | no | `query` (default), `json`, or `decrypt_labs`. |
| `username` | for `query` | Required in query mode; ignored otherwise. |
| `password` | one of these | Password (query mode) or token (json mode). |
| `api_key` | one of these | Alternative to `password`; used for `decrypt_labs`. If both are given, `api_key` wins. |
| `no_push` | no | Read-only when `true` (always `true` for `decrypt_labs`). |
| `timeout` | no | Request timeout in seconds; defaults to `vault_timeout`. |
You must supply either `password` or `api_key`, or the vault will fail to load.
---
## API backend
A simple RESTful HTTP vault with a fixed, well-defined protocol. Authentication is a bearer token.
```yaml title="unshackle.yaml"
key_vaults:
- type: API
name: myapi
uri: https://vault.example.com/api
token: "your-bearer-token"
```
| Field | Required | Description |
|-------|----------|-------------|
| `type` | yes | Must be `API`. |
| `name` | yes | A label for this vault. |
| `uri` | yes | Base URI of the vault API (a trailing slash is trimmed). |
| `token` | yes | Bearer token, sent as `Authorization: Bearer <token>`. |
| `no_push` | no | If `true`, keys are read but never written. |
| `timeout` | no | Request timeout in seconds; defaults to `vault_timeout`. |
The backend talks to endpoints under `uri` (for example `GET {uri}/{service}/{kid}` to look up a single key) and interprets a numeric `code` field in each JSON response to detect errors such as an invalid token, rate limiting, or an invalid service tag.
!!! warning "This is unshackle's own protocol, not a generic vault client"
The `API` backend expects unshackle's specific HTTP request and response shape. An API or HTTP key-vault endpoint built for a different project or service will generally *not* be compatible. You cannot simply point this at an arbitrary third-party vault API and expect it to work. To bridge another service, put an adapter in front of it that speaks this format, or use the `HTTP` backend if one of its wire protocols matches. When pushing many keys at once it batches requests and automatically shrinks the batch size if the server rejects an over-large request.
!!! tip "DecryptLabs shortcut"
An `API` vault whose `name` contains `decrypt_labs` will have its `token` filled in automatically from a global `decrypt_labs_api_key` in your config if you leave `token` unset. This lets you keep the key in one place.
---
## Managing vaults with `unshackle kv`
The `kv` command group manages the contents of your configured vaults. Every subcommand refers to vaults **by the `name` you gave them** in `key_vaults`; an unknown name is an error.
```console
$ unshackle kv --help
```
### `kv add`: import keys from a file
Add `KID:KEY` pairs to one or more vaults for a given service. The file must contain one pair per line as `HEX:HEX` (32 hex characters each); lines that do not match are ignored.
```console
$ unshackle kv add keys.txt DSNP local shared
```
```text title="keys.txt"
9a04f07998404286ab92e65be0885f95:b1c2d3e4f5a6978012345678deadbeef
0e7cc1a4e2f9411db0f24a3fbb8f8b21:00112233445566778899aabbccddeeff
```
The service argument (`DSNP` above) is normalized to its canonical tag automatically. Keys that already exist in a vault are reported as skipped.
### `kv search`: find a key by KID
Search your vaults for a KID and print any matching key.
```console
$ unshackle kv search 9a04f07998404286ab92e65be0885f95
```
The KID must be 32 hex characters (dashes are stripped for you). By default every configured vault and every service table is scanned; narrow the search with options:
| Option | Description |
|--------|-------------|
| `-s`, `--service` | Limit the search to a single service tag. |
| `-v`, `--vault` | Limit the search to one configured vault by name. |
!!! note
Remote vaults that cannot enumerate their service tables (for example an `HTTP` vault in `json` or `decrypt_labs` mode) can only be searched when you also pass `--service`. Without it, they are skipped.
### `kv copy`: copy keys between vaults
Copy every key from one or more source vaults into a single destination vault. Existing rows are never altered; only genuinely new KIDs are added, and null keys are skipped.
```console
$ unshackle kv copy shared local
```
The first argument is the destination; the rest are sources. You can supply several sources at once:
```console
$ unshackle kv copy local shared community
```
| Option | Description |
|--------|-------------|
| `-s`, `--service` | Only copy the given service. |
| `-l`, `--local-only` | Only copy services you have installed locally, skipping tables for unknown services. |
`--service` and `--local-only` are mutually exclusive.
### `kv sync`: mirror vaults both ways
Ensure two or more vaults each end up with every key the others have. This is effectively a two-way `copy` between each pair, so afterwards all listed vaults hold the same set of keys.
```console
$ unshackle kv sync local shared
```
It accepts the same `--service` and `--local-only` options as `copy`, and requires at least two vaults.
### `kv prepare`: pre-create service tables
Create the per-service tables on table-based vaults (`SQLite`, `MySQL`) ahead of time, for every service you have installed. Vaults that do not use tables (the network-backed `HTTP` and `API` backends) are skipped.
```console
$ unshackle kv prepare shared
```
This is mainly useful with a `MySQL` vault where you want the tables created once by an account that has the `CREATE` grant.
---
## See also
- [Downloading](downloading.md): how keys are recovered and used during a download.