diff --git a/README.md b/README.md index 952d3a78..a7b4c693 100644 --- a/README.md +++ b/README.md @@ -52,4 +52,4 @@ Optional: ## License -[GPL-3.0](LICENSE). Do not use unshackle for content you lack the rights to. Keep the core free and open; keep service code private. Be kind. +[GPL-3.0](LICENSE). Do not use unshackle for content you do not have the rights to. Keep the core free and open. Keep service code private. Be kind. diff --git a/docs/dev/creating-a-service.md b/docs/dev/creating-a-service.md index fb70733e..5da0202e 100644 --- a/docs/dev/creating-a-service.md +++ b/docs/dev/creating-a-service.md @@ -1,16 +1,16 @@ # Creating a Service A **service** is a plugin that teaches unshackle how to talk to one streaming -provider: how to log in, how to look up a title, what video/audio/subtitle -tracks exist, and how to license any DRM. Everything else (track selection, -downloading, decryption, muxing, naming) is handled by the unshackle core. +platform: how to log in, how to look up a title, what video/audio/subtitle +tracks exist, and how to license any DRM. The unshackle core does everything +else (track selection, downloading, decryption, muxing, naming). !!! info "Who this page is for" This is a **developer guide**. If you only want to *use* services, see [downloading](../guide/downloading.md) and - [configuration](../reference/configuration/index.md). Nothing here is required to run - the CLI. Service code is deliberately kept separate from the core so it can - live in its own private repository. + [configuration](../reference/configuration/index.md). You do not need anything here to + operate the CLI. The code of a service is deliberately separate from the + core, so it can live in its own private repository. unshackle ships two reference services. **EXAMPLE**, at `unshackle/services/EXAMPLE/`, is a non-runnable showcase of every framework @@ -20,11 +20,11 @@ kind of showcase for a music service. If your platform has songs, read that one --- -## How a service is discovered +## How unshackle finds a service At startup unshackle scans every path in the `directories.services` config key (a **list**). Each entry is either a local directory of services or a remote -repo spec (a git URL or `owner/repo` shorthand, which is cloned automatically). +repo spec (a git URL or `owner/repo` shorthand, which unshackle clones for you). The default is the bundled `unshackle/services` directory. Within a services directory, **every subfolder that contains an `__init__.py` @@ -39,13 +39,13 @@ services/ Two rules the loader enforces: -1. **The class name must exactly match the folder/tag name.** Folder `EXAMPLE` must +1. **The class name must match the folder/tag name exactly.** Folder `EXAMPLE` must define `class EXAMPLE`. A mismatch raises a `RuntimeError`. 2. **List order is priority.** The first services path to define a given tag - wins; later duplicates are shadowed. Put your own local overrides *last* to + wins, and unshackle ignores the later duplicates. Put your own local overrides *last* to use them as fallbacks, or *first* to override a repo. -Once installed, a service is invoked as a subcommand of `dl`: +After installation, you call a service as a subcommand of `dl`: ```bash unshackle dl EXAMPLE 20914 @@ -56,11 +56,11 @@ unshackle dl EX 20914 # via an ALIAS A service's own settings go in `config.yaml` **inside the service folder** (`directories.services//config.yaml`). This is separate from the global `unshackle.yaml`. Access it in code as `self.config[...]`. Never hardcode - URLs, user agents, or certificates; put them here. + URLs, user agents, or certificates. Put them here. !!! warning "`config.yaml` is shared: keep secrets out of it" Treat `config.yaml` as a checked-in file of **defaults**: it belongs in the - service repo and is shared by everyone who runs the service. Per-user secrets + service repo, and everyone who runs the service shares it. Per-user secrets (API keys, device IDs, account tokens) must **not** live here. Those go in the user's own `unshackle.yaml` under `services.`, which is **merged into `self.config` at runtime**. That checked-in-defaults / per-user-overrides @@ -78,7 +78,7 @@ unshackle dl EX 20914 # via an ALIAS ``` A service loaded from a remote repo lives outside the `unshackle` package, - so the absolute `unshackle.services..*` path does not resolve and the + so Python cannot find the absolute `unshackle.services..*` path and the import fails with `no known parent package`. Relative imports work both locally and from a cloned repo. @@ -104,10 +104,10 @@ Declared at class level to configure framework behaviour: | `DAILY` | `bool` | Set `True` when the catalog is daily/date-based (talk shows, news, sports), so episodes are named by air date. A title's own `daily` flag overrides it. | !!! note "`NO_SUBTITLES` is a convention, not a base-class attribute" - `NO_SUBTITLES` is checked by `dl.py` via `hasattr` and is deliberately **not** - declared on the `Service` base class, so it only exists if your service + `dl.py` uses `hasattr` to find `NO_SUBTITLES`, and the `Service` base class + deliberately does **not** declare it, so it only exists if your service defines it. Setting `NO_SUBTITLES = True` is what makes the pipeline skip - subtitle work for services that never carry subtitles; leave it off otherwise. + subtitle work for services that never carry subtitles. Leave it off otherwise. `GEOFENCE` drives an automatic proxy: on startup, if you did not pass an explicit `--proxy`, the base class does a live IP check and, if your region is @@ -150,14 +150,14 @@ class TrackRequest: !!! note "Read the request, don't second-guess the user" Services may narrow `track_request` for hard technical constraints (e.g. - *"HDR on this service is only delivered as HEVC"*), but must **not** filter + *"this service delivers HDR only as HEVC"*), but must **not** filter the tracks they return by resolution, bitrate, or language. All selection is done by the core after `get_tracks` returns. !!! tip "Override `get_session()` to defeat TLS-fingerprint bot detection" The base `get_session()` returns a plain `requests.Session` (config headers + retry adapter) and **deliberately does not** do TLS impersonation. When a - provider blocks you on TLS fingerprint alone, override it to return + streaming platform blocks you on TLS fingerprint alone, override it to return `session("Chrome131")` (any `rnet.Impersonate` preset). `RnetSession` is a drop-in `requests.Session` replacement (same `.get`/`.post`/cookie-jar interface), so nothing else in the service changes. If instead you need @@ -166,9 +166,9 @@ class TrackRequest: --- -## The methods you implement +## The methods you write -Methods run in the order shown. Only the three abstract ones are mandatory. +unshackle calls the methods in the order shown. Only the three abstract ones are mandatory. ```mermaid graph LR @@ -182,11 +182,11 @@ returns an **instance** of your service. Keep the command `name=` matching the tag. !!! note "`name=` is cosmetic: the directory name is what resolves the command" - `unshackle dl EXAMPLE` is dispatched by the **folder/class name**, not by the - `name=` in `@click.command`. A mismatched `name=` still loads and runs; it - just produces confusing `--help`/usage output that names the wrong command. - That confusing output is the reason to keep it in sync, not any dispatch - requirement. + unshackle chooses the command from the **folder/class name**, not from the + `name=` in `@click.command`. A mismatched `name=` still loads and runs. It + only produces confusing `--help`/usage output that names the wrong command. + That confusing output is the reason to keep it in sync, not any requirement + of the command lookup. ```python @staticmethod @@ -212,7 +212,7 @@ accepted URL/ID formats and options there. !!! tip "CDM-aware behaviour" The resolved CDM is on `ctx.obj.cdm` (may be `None` for DRM-free runs). Use `is_widevine_cdm()` / `is_playready_cdm()` from `unshackle.core.cdm.detect` - to classify it. These correctly handle local *and* remote CDMs, so never + to classify it. These work correctly for local *and* remote CDMs, so never hand-roll an `isinstance` check. Services often pick a device profile or manifest endpoint based on which DRM the CDM speaks. @@ -220,7 +220,7 @@ accepted URL/ID formats and options there. Override to log in with cookies and/or credentials. **Call `super().authenticate()` first**: the base implementation loads the cookie jar into `self.session.cookies` -and stores `self.credential`. Do all token fetching here; it runs before +and stores `self.credential`. Do all token fetching here. It runs before `get_titles`. ```python @@ -237,30 +237,30 @@ def authenticate(self, cookies=None, credential=None) -> None: self.token = cache.data["token"] ``` -Cookies come from files under `directories.cookies`; credentials from the +Cookies come from files under `directories.cookies`. Credentials come from the `credentials` map in `unshackle.yaml`. For interactive prompts (OTP, captcha) use `self.request_input(prompt)`, never a bare `input()`. Under `serve` mode there is no local terminal, so a bare `input()` would hang the server waiting on -stdin that never arrives; `request_input` instead relays the prompt to the +stdin that never arrives. `request_input` instead relays the prompt to the remote client through the attached `InputBridge`. Locally it routes through the shared Rich console (`prompt_user`) so the prompt renders correctly alongside progress and log output. -!!! warning "Key token caches by whatever varies per session" - The cache key above is `tokens_{device}_{profile}` on purpose. A token cache +!!! warning "Make each token cache entry unique" + The cache above uses `tokens_{device}_{profile}` on purpose. A token cache must include every dimension that changes the token (device, profile, or `credential.sha1`), or two profiles/devices will share one cache entry and - stomp each other's tokens. This isn't obvious from the caching API, which - happily lets you use a single flat key. + stomp each other's tokens. This is not obvious from the caching API, which + happily lets you use a single flat name. -!!! warning "Handle the credential carefully" - Use the `Credential` only inside `authenticate()` to obtain tokens; don't +!!! warning "Use the credential carefully" + Use the `Credential` only inside `authenticate()` to get tokens. Do not stash it or the raw cookie jar elsewhere. The base class already caches identity for you. ### `get_titles`: required -Return a titles collection for the given ID. The return type depends on content: +Return a titles collection for the given ID. The return type depends on the kind of title: | Return | Contains | Use for | |---|---|---| @@ -269,8 +269,8 @@ Return a titles collection for the given ID. The return type depends on content: | `Album` | `Song` objects | Music | Every title carries a `language` (its **original recorded language**) and an -arbitrary `data` dict you can use to stash metadata for later methods. At least -one title must be returned, or the ID is treated as invalid. +arbitrary `data` dict you can use to stash metadata for later methods. You must +return at least one title. If you return none, unshackle treats the ID as invalid. ```python def get_titles(self) -> Titles_T: @@ -311,7 +311,7 @@ Common constructor fields: part=None) ``` - Pass `air_date` for daily/sports content to name by date instead of + Pass `air_date` for daily and sports titles to name by date instead of `SxxExx`. Return a `Series([...])` of episodes. Pass `part` only when the service splits one episode into several videos, see [Split episodes](#split-episodes). @@ -354,8 +354,8 @@ Episode(..., season=1, number=1, part=1, name="The Reckoning") Episode(..., season=1, number=1, part=2, name="The Reckoning") ``` -1. `part` counts from 1, contiguous. `part=0` is rejected, and `-w` can only address - parts up to 99. +1. `part` counts from 1, contiguous. unshackle rejects `part=0`, and `-w` can only + address parts up to 99. 2. Every part of one episode shares the same `season` and `number`. 3. Each part is a separate `Episode` with its own unique `id_`. 4. Unsplit episode: leave `part` unset. Never pass `part=1` for a whole episode. @@ -375,13 +375,13 @@ Episode(..., season=1, number=1, part=2, name="The Reckoning") !!! tip "Clean the part out of the name" `Episode` discards any `name` that starts with `Episode `, so a raw - `"Episode 1 (Part 2): The Reckoning"` leaves you with no episode name at all. Strip + `"Episode 1 (Part 2): The Reckoning"` leaves you with no episode name at all. Remove the prefix before you pass it, or pass `None`. Once `part` is set, the filename gains a part token (`S01E01.Part.2`, see [Output & naming](../guide/output-and-naming.md#split-episodes)), `-w S01E01.2` selects a single part while a bare `-w S01E01` selects them all, and the `--list-titles` tree labels -the episode `01.2`. The REST API reports it as an extra `part` key on the serialized +the episode `01.2`. The REST API reports it as an extra `part` field on the serialized title, described in [Endpoints](rest-api/endpoints.md#post-apilist-titles). !!! note "Opt-in and additive" @@ -392,7 +392,7 @@ title, described in [Endpoints](rest-api/endpoints.md#post-apilist-titles). ### `get_tracks`: required Given one title, return a `Tracks` object holding `Video`, `Audio`, and -`Subtitle` tracks. In almost all cases you build these by parsing a manifest, +`Subtitle` tracks. In almost all cases you assemble these by parsing a manifest, not by hand. The manifest parsers live in `unshackle.core.manifests`: @@ -432,11 +432,11 @@ Each exposes `from_url(url, session=...)` (or `from_text(text, url)`) and then !!! warning "Always pass `language=`" Pass the title's original language to `to_tracks(language=...)`. For DASH and - HLS this is **required as a fallback**. If a track's language can't be - derived and no valid fallback was given, `to_tracks` raises `ValueError`. - It's also what lets the parser flag `is_original_lang` on each track (via - `is_close_match`), driving `-l best/all` selection and the filename language - token. + HLS this is **required as a fallback**. If `to_tracks` cannot derive a track's + language and you gave no valid fallback, it raises `ValueError`. + It is also what lets the parser flag `is_original_lang` on each track (through + `is_close_match`), which drives `-l best/all` selection and the filename + language token. Some services deliver a **separate manifest per codec/range**. The base class provides `get_tracks_for_variants(title, fetch_fn)` to fan out over every @@ -459,30 +459,30 @@ for a thorough demonstration. !!! note "Flip HDR10 → HDR10+ by hand when you know the platform embeds it" HDR10+ is a **bitstream (SEI)** feature. The HLS parser intentionally does - not sniff it from the stream, so a manifest that embeds HDR10+ SEI but labels + not sniff it from the bitstream, so a manifest that embeds HDR10+ SEI but labels the variant plain HDR10 will parse as HDR10. The label is the only signal - the parser has, and it's wrong. Only the service author knows the platform - embeds HDR10+, so it's on you to correct `video.range` to HDR10+ here for + the parser has, and it is wrong. Only the service author knows the platform + embeds HDR10+, so it is on you to correct `video.range` to HDR10+ here for those services. !!! warning "`get_tracks` is your only chance to capture license inputs" - The license callbacks (`get_widevine_license`, etc.) fire **much later**, in + The license callbacks (`get_widevine_license` and the others) fire **much later**, in the download/decrypt step, long after `get_tracks` has returned, and they receive **only** `challenge`, `title`, and `track`. Nothing else from the - manifest response reaches them. So anything a license call needs (the license - URL, `dt-custom-data`, a session token) has to be stashed **now**, during + manifest response reaches them. So you must stash anything a license call needs + (the license URL, `dt-custom-data`, a session token) **now**, during `get_tracks`, onto `self.license_data` or `title.data`. This is a - lifecycle-timing consequence you can't infer from the callback signatures. + lifecycle-timing consequence you cannot infer from the callback signatures. -!!! tip "Provide the KID if you cheaply can" - If you can obtain a track's Key ID (32-char hex) without downloading stream - data, set `track.kid`; it speeds up the decryption/key-lookup path. And be +!!! tip "Give the KID if you cheaply can" + If you can get a track's Key ID (32-char hex) without downloading the track + data, set `track.kid`. It speeds up the decryption and key-lookup path. And be sure encrypted tracks carry the correct `drm` so the core licenses them. ### `get_chapters`: required -Return a `Chapters` object (0 or more `Chapter`s). You don't need to number or -sort them; `Chapters` does that automatically and inserts a `00:00:00.000` +Return a `Chapters` object (0 or more `Chapter`s). You do not need to number or +sort them. `Chapters` does that automatically and inserts a `00:00:00.000` marker if missing. A `Chapter` timestamp accepts `"HH:MM:SS[.mmm]"`, an int in milliseconds, or a float in seconds. @@ -515,8 +515,8 @@ def get_chapters(self, title: Title_T) -> Chapters: label applies only to that span. !!! warning "Don't invent chapter names" - Never name chapters `"Chapter 1"` yourself. Leave unnamed markers unnamed; - users who want generic names set `chapter_fallback_name` (e.g. + Never name chapters `"Chapter 1"` yourself. Leave unnamed markers unnamed. + Users who want generic names set `chapter_fallback_name` (for example, `"Chapter {i:02}"`) in their config. --- @@ -526,7 +526,7 @@ def get_chapters(self, title: Title_T) -> Chapters: The core drives licensing. Your job is to answer the license challenges the CDM produces. Mark encrypted tracks with the right DRM in `get_tracks` (the manifest parsers do this automatically for PSSH/`EXT-X-KEY` found in the manifest), then -implement the license callbacks you need. Each receives the challenge plus the +write the license callbacks you need. Each receives the challenge plus the current `title` and `track`: | Method | For | Returns | @@ -558,20 +558,21 @@ def get_widevine_license(self, *, challenge: bytes, title, track): !!! tip "`ClearKeyCENC` (`org.w3.clearkey`) has three escalating integration levels" Pick the lowest level that works for the platform: - 1. **The manifest carries a ``**: implement nothing. The framework + 1. **The manifest carries a ``**: write nothing. The framework POSTs the challenge to that URL itself. - 2. **A custom endpoint or extra headers are needed**: override + 2. **Your service needs a custom endpoint or extra headers**: override `get_clearkey_license` to do the POST yourself. 3. **Keys arrive obfuscated or through a bespoke channel**: fetch and unwrap - the key in `get_tracks` and pre-populate + the content key in `get_tracks` and pre-populate `drm.content_keys[kid] = key_hex` on the track's `ClearKeyCENC`. The non-obvious payoff: when every KID is already keyed, the framework skips the license round-trip entirely. -Things you don't handle: +Things the service does not do: -- **HLS AES-128 (`ClearKey`)** and **DRM-free** content have no license - callback. The key comes from the manifest and is applied directly. +- **HLS AES-128 (`ClearKey`)** tracks and **DRM-free** tracks have no license + callback. The content key comes from the manifest, and unshackle applies it + directly. - **Key vaults** cache `KID:KEY` pairs across runs, so repeat downloads skip the license round-trip entirely. - **Decryption tooling** (shaka-packager or mp4decrypt) and the choice of local @@ -688,7 +689,7 @@ endpoints: license: https://api.myservice.com/v1/license/widevine ``` -Test it with: +Try it with: ```bash unshackle dl MYSVC https://myservice.com/watch/abc123 @@ -699,11 +700,11 @@ unshackle dl MYSVC https://myservice.com/watch/abc123 ## Checklist - [ ] Folder name, class name, and Click `name=` all match the tag exactly. -- [ ] `cli` is a `@staticmethod` returning an instance; `__init__` calls `super().__init__(ctx)`. -- [ ] `get_titles`, `get_tracks`, `get_chapters` implemented; titles carry the original `language`. -- [ ] Tracks come from a manifest parser with `to_tracks(language=title.language)`; no filtering by quality/language. -- [ ] Encrypted tracks are marked with DRM; the license callbacks you need return responses **unmodified**. -- [ ] URLs, user agents, and certificates live in `config.yaml`, read via `self.config[...]`. +- [ ] `cli` is a `@staticmethod` that returns an instance. `__init__` calls `super().__init__(ctx)`. +- [ ] `get_titles`, `get_tracks` and `get_chapters` written. Titles carry the original `language`. +- [ ] Tracks come from a manifest parser with `to_tracks(language=title.language)`. No filtering by quality or language. +- [ ] Encrypted tracks carry their DRM. The license callbacks you need return responses **unmodified**. +- [ ] URLs, user agents, and certificates live in `config.yaml`, read through `self.config[...]`. - [ ] Multi-file services import their own modules relatively (`from .helpers import x`), so they work when loaded from a repo. Read `unshackle/services/EXAMPLE/` end to end. It annotates every feature diff --git a/docs/dev/rest-api/authentication.md b/docs/dev/rest-api/authentication.md index 0f707182..37310abd 100644 --- a/docs/dev/rest-api/authentication.md +++ b/docs/dev/rest-api/authentication.md @@ -1,16 +1,16 @@ # REST API Authentication -The unshackle REST API protects every endpoint (with one exception) behind a single API key. Clients present that key in a request header, and the server checks it against the keys defined in your configuration. This page explains how that check works, how to configure keys, how clients send them, and the extra controls that build on top of the key: per-key service allowlists, per-key CDM device access, and session IP binding. +The unshackle REST API protects every endpoint (with one exception) behind a single API key. Clients present that API key in a request header, and the server examines it against the API keys in your configuration. This page explains how that check works, how to configure API keys, and how clients send them. It also explains the extra controls on top of the API key: per-key service allowlists, per-key CDM device access, and remote session IP binding. -The API is started with the [`unshackle serve`](../../guide/cli-reference.md) command. Authentication is on by default: if you start the server without an API key configured, it refuses to launch. +You start the API with the [`unshackle serve`](../../guide/cli-reference.md) command. Authentication is on by default: if you start the server without an API key configured, it refuses to launch. !!! note "Which server am I running?" - `unshackle serve` can run in two shapes, and each uses a slightly different piece of code to check your key: + `unshackle serve` can operate in two shapes, and each uses a slightly different piece of code to examine your API key: - The **integrated server** (the default) exposes the REST API *and* the pywidevine / pyplayready CDM HTTP endpoints. - The **API-only server** (`--api-only`, also implied by `--remote-only`) exposes only the REST API. - The header, the config, and the behaviour you see as a client are the same in both. The differences are internal and are called out in the developer notes below. + The header, the config, and the behaviour you see as a client are the same in both. The differences are internal, and the developer notes below give them. ## The short version @@ -36,7 +36,7 @@ serve: api_secret: "a-long-random-secret-string" ``` -This value becomes the one and only key that the server accepts. Requests that present it in the `X-Secret-Key` header are allowed; everything else is rejected. +This value becomes the one and only API key that the server accepts. The server allows requests that present it in the `X-Secret-Key` header. The server rejects everything else. !!! warning "The server will not start without a key" If you start `unshackle serve` with no `api_secret` configured, and you have *not* passed `--no-key`, the command fails immediately with: @@ -45,11 +45,11 @@ This value becomes the one and only key that the server accepts. Requests that p API secret key is not configured. Please add 'api_secret' to the 'serve' section in your config. ``` - Choose a long, unpredictable value. The key is the only thing standing between the internet and your CDM and download endpoints. + Choose a long, unpredictable value. The API key is the only thing standing between the internet and your CDM and download endpoints. ### Multiple keys with `users` -For finer control you can define named users, each with their own key. A user entry can also grant CDM device access and restrict which services that key may touch: +For finer control you can define named users, each with their own API key. A user entry can also grant CDM device access and restrict which services that API key may touch: ```yaml title="unshackle.yaml" serve: @@ -65,16 +65,16 @@ serve: devices: ["my_widevine_device"] ``` -The key of each entry under `users` (for example `a-secret-key-for-alice`) *is* the secret that client sends in `X-Secret-Key`. The `username` is a human-readable label used in logs and in job/history metadata. +The config key of each entry under `users` (for example `a-secret-key-for-alice`) *is* the secret that client sends in `X-Secret-Key`. The `username` is a human-readable label used in logs and in job/history metadata. !!! info "How `api_secret` and `users` combine (integrated server)" - On the integrated server, the configured `api_secret` is automatically added as an extra accepted key alongside everyone in `users`. It is granted access to all loaded Widevine and PlayReady devices and is labelled `api_user`. So `api_secret` behaves as an admin-level key in addition to any named users you define. + On the integrated server, the server automatically adds the configured `api_secret` as an extra accepted API key alongside everyone in `users`. It gives that API key access to all loaded Widevine and PlayReady devices, and labels it `api_user`. So `api_secret` behaves as an admin-level API key in addition to any named users you define. - On the API-only server, only `api_secret` is accepted: the single-key model. (See the developer notes for the exact difference.) + On the API-only server, the server accepts only `api_secret`: the single-key model. (See the developer notes for the exact difference.) -### CDM device access per key +### CDM device access per API key -Under the integrated server, a key can only run DRM licensing through a CDM device that its user entry lists: +Under the integrated server, an API key can do DRM licensing only through a CDM device that its user entry lists: | Config key | Grants access to | |---|---| @@ -82,21 +82,21 @@ Under the integrated server, a key can only run DRM licensing through a CDM devi | `playready_devices` | PlayReady `.prd` devices, by name | !!! warning "PlayReady access is opt-in per user" - If a user entry omits `playready_devices`, that key is given **no** PlayReady access, and the server logs a warning at startup. You must list PlayReady devices explicitly for any key that needs them. Widevine and PlayReady device files themselves are auto-loaded from your WVDs and PRDs directories. + If a user entry omits `playready_devices`, the server gives that API key **no** PlayReady access, and logs a warning at startup. You must name the PlayReady devices explicitly for any API key that needs them. Widevine and PlayReady device files themselves are auto-loaded from your WVDs and PRDs directories. !!! note "Device lists *are* the server-side-decryption switch" - There is no separate tier, capability flag, or permission toggle for whether the server will hand back keys (`KID:KEY`) for a session download. The presence of devices on the calling key decides it. With **empty** `devices` and `playready_devices`, the server can only proxy CDM challenges, so the client must run its own CDM. Once those lists are **populated**, the client may request `mode:server_cdm` and receive keys back. Don't go looking for a tier setting; configuring devices is what enables server-side decryption. + There is no separate tier, capability flag, or permission toggle for whether the server will hand back content keys (`KID:KEY`) for a remote session download. The presence of devices on the calling API key decides it. With **empty** `devices` and `playready_devices`, the server can only proxy CDM challenges, so the client must operate its own CDM. Once those lists are **populated**, the client may request `mode:server_cdm` and receive content keys back. Do not look for a tier setting. The devices you configure are what enables server-side decryption. ### Disabling authentication -The `--no-key` flag turns authentication off entirely. Every request is allowed, `X-Secret-Key` is ignored, and no key needs to be configured. +The `--no-key` flag turns authentication off entirely. The server allows every request, ignores `X-Secret-Key`, and needs no API key in your configuration. ```bash unshackle serve --no-key ``` !!! danger "Only use `--no-key` on a trusted, isolated network" - With `--no-key` there is nothing stopping any client that can reach the port from listing your services, starting downloads, or using your CDM devices. Never expose a `--no-key` server to an untrusted network. + With `--no-key`, any client that can reach the port can show your services, start downloads, and use your CDM devices. Never put a `--no-key` server on an untrusted network. ## How clients present credentials @@ -106,18 +106,18 @@ Clients authenticate by sending the secret in a single HTTP request header on ev X-Secret-Key: ``` -The value must exactly match one of the configured keys (`api_secret`, or one of the keys under `users`). There is no login step, no session cookie, and no token exchange for API access. The same static key is sent on each request. +The value must match one of the configured API keys exactly (`api_secret`, or one of the keys under `users`). There is no login step, no HTTP session cookie, and no token exchange for API access. The client sends the same static API key on each request. !!! info "One route also accepts the key as a query parameter" - The job event stream, `GET /api/download/jobs/{job_id}/events`, also accepts the key in a `secret_key` query parameter. This is for browsers: the `EventSource` API cannot set request headers. The exception applies to that one route only, and it works in both server modes. + The job event stream, `GET /api/download/jobs/{job_id}/events`, also accepts the API key in a `secret_key` query parameter. This is for browsers: the `EventSource` API cannot set request headers. The exception applies to that one route only, and it works in both server modes. ```javascript new EventSource(`http://127.0.0.1:8786/api/download/jobs/${jobId}/events?secret_key=${key}`); ``` - The header always wins. If the request has an `X-Secret-Key` header, the server checks that header and ignores the query parameter, so a wrong header is rejected even when the query parameter holds a valid key. + The header always wins. If the request has an `X-Secret-Key` header, the server examines that header and ignores the query parameter. The server rejects a wrong header even when the query parameter holds a valid API key. - A key in a URL is easier to leak than a key in a header, because proxies and browser history can record it. Use the header wherever your client can set one. + An API key in a URL is easier to leak than an API key in a header, because proxies and browser history can record it. Use the header wherever your client can set one. === "curl" @@ -150,11 +150,11 @@ The value must exactly match one of the configured keys (`api_secret`, or one of ``` !!! tip "The reference client sets it for you" - unshackle's own [remote download](../../guide/downloading.md) client (`RemoteService`) sends `X-Secret-Key` automatically, reading the key from your `remote_services` config. It also sends a `User-Agent` of `unshackle/`. You only deal with the header directly when writing your own client. + unshackle's own [remote download](../../guide/downloading.md) client (`RemoteService`) sends `X-Secret-Key` automatically, reading the API key from your `remote_services` config. It also sends a `User-Agent` of `unshackle/`. You only deal with the header directly when writing your own client. ### The health check is always open -One endpoint is exempt from authentication: `GET /api/health`. It never requires a key, so you can use it for liveness and version probes without embedding a secret in your monitoring: +One endpoint is exempt from authentication: `GET /api/health`. It never requires an API key, so you can use it for liveness and version probes without embedding a secret in your monitoring: ```bash title="No key needed" curl http://127.0.0.1:8786/api/health @@ -172,11 +172,11 @@ curl http://127.0.0.1:8786/api/health } ``` -Every other endpoint (including the Swagger UI at `/api/docs/` on the API-only server) requires a valid key. +Every other endpoint (including the Swagger UI at `/api/docs/` on the API-only server) requires a valid API key. ## Authentication failures -When a request is rejected for authentication reasons, the response has an HTTP `401 Unauthorized` status and a small JSON body. +When the server rejects a request for authentication reasons, the response has an HTTP `401 Unauthorized` status and a small JSON body. === "Missing header" @@ -191,7 +191,7 @@ When a request is rejected for authentication reasons, the response has an HTTP === "Wrong key" - Sending a key that is not in your config: + Sending an API key that is not in your config: ```json { @@ -203,17 +203,17 @@ When a request is rejected for authentication reasons, the response has an HTTP !!! warning "Auth errors use a different JSON shape than other errors (developer note)" The rest of the API reports errors with a structured body containing `error_code`, `message`, `timestamp`, and other fields (see [Errors](errors.md)). The authentication middleware is different: it returns `{"status": 401, "message": "..."}`, where `status` is an **integer**, and there is no `error_code` or `timestamp`. - A robust client should treat any `401` as an authentication problem and be prepared to parse *both* shapes rather than assuming the standard error body is always present. + In your client, treat any `401` as an authentication problem and parse *both* shapes, because the standard error body is not always present. -## Building on the key: access controls +## Access controls on top of the API key -Once a request is authenticated, several additional controls decide what that specific key is allowed to do. +After the server authenticates a request, several additional controls decide what that specific API key can do. ### Per-key service allowlists -You can restrict which services a key may use. Two layers combine: +You can restrict which services an API key may use. Two layers combine: -- **Global allowlist**: `serve.services`, applied to every key. +- **Global allowlist**: `serve.services`, applied to every API key. - **Per-key allowlist**: `services` inside that key's `users` entry. The effective allowlist for a request is: @@ -235,11 +235,11 @@ serve: services: ["EXAMPLE1"] # this key ends up allowed only EXAMPLE1 ``` -Endpoints that name a service (listing services, searching, listing titles/tracks, downloading, profiles, history, and remote sessions) all filter against this allowlist. A request for a service outside the key's allowlist is rejected as an invalid service. Service tags are matched after normalization, so casing and aliases resolve consistently. +Endpoints that name a service (listing services, searching, listing titles/tracks, downloading, profiles, history, and remote sessions) all filter against this allowlist. The server rejects a request for a service outside the API key's allowlist as an invalid service. The server matches service tags after normalisation, so casing and aliases give the same result. ### CDM overrides and per-job credentials -Two download-time capabilities are gated behind server config and, unless enabled, are refused with `403 Forbidden` even for an authenticated key: +The server config gates two download-time capabilities. Unless you enable them, the server refuses them with `403 Forbidden`, even for an authenticated API key: | Config key | Default | Controls | |---|---|---| @@ -254,26 +254,26 @@ serve: ``` !!! tip "When to enable these gates" - The two gates are fully independent and can be combined. The intended posture is that a locked-down or public-facing deployment leaves **both** unset (they default off and reject with `403`) so client requests can never steer the server's CDM or feed it their own logins. Enabling them only makes sense for a trusted single-client deployment, where the caller and the operator are effectively the same party. + The two gates are fully independent, and you can combine them. For a locked-down or public-facing deployment, leave **both** unset. They then default off and reject with `403`, so client requests can never steer the server's CDM or feed it their own logins. Enable them only for a trusted single-client deployment, where the caller and the operator are effectively the same party. !!! note "Client-supplied credentials get isolated token caches" - When `allow_job_credentials` is enabled and a download job supplies its own `credential` / `credentials`, each distinct credential is given its **own** isolated token cache on the server. Client-supplied logins are never shared or merged with the server's configured-credential caches, so they cannot cross-contaminate or read the server's own token state, and they don't clobber each other. + When you enable `allow_job_credentials` and a download job supplies its own `credential` / `credentials`, the server gives each distinct credential its **own** isolated token cache. The server never shares or merges client-supplied logins with its configured-credential caches. They cannot cross-contaminate or read the server's own token state, and they do not clobber each other. -See [Downloads](endpoints.md) for how these fields are used on the download endpoint. +See [Downloads](endpoints.md) for how the download endpoint uses these fields. ### Session IP binding (remote sessions) -The remote-download session endpoints (`/api/session/*`) add an extra check on top of the key. When a session is created, the server records the client's IP address. Any later request for that session must come from the **same** IP, or it is rejected with `403 Forbidden`. This means a session is bound both to the key that created it and to the address it was created from. +The remote-download session endpoints (`/api/session/*`) add an extra check on top of the API key. When the server creates a remote session, it records the client's IP address. Any later request for that remote session must come from the **same** IP, or the server rejects it with `403 Forbidden`. Thus the server binds a remote session both to the API key that created it and to the address it came from. ## Transport security (HTTPS) -The API server speaks plain HTTP. Because your key is sent on every request, you should not expose it over an unencrypted connection on an untrusted network. To serve over HTTPS, run a reverse proxy in front of unshackle. The `serve` command can launch Caddy for you: +The API server speaks plain HTTP. Because the client sends your API key on every request, do not use it over an unencrypted connection on an untrusted network. To serve over HTTPS, operate a reverse proxy in front of unshackle. The `serve` command can launch Caddy for you: ```bash unshackle serve --caddy ``` -With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located next to your unshackle config, letting you terminate TLS and reverse-proxy to the API. Configuring Caddy itself (certificates, domains, upstream) is done in that `Caddyfile`. +With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located next to your unshackle config, letting Caddy end the TLS connection and reverse-proxy to the API. Configuring Caddy itself (certificates, domains, upstream) is done in that `Caddyfile`. ## Quick reference @@ -291,7 +291,7 @@ With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located --- -## Developer notes: how the check is implemented +## Developer notes: how the server does the check !!! note "This section is for people writing or debugging the server, not for API consumers." @@ -299,13 +299,13 @@ With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located The header and config are identical between modes, but the code that enforces them differs: -- **API-only server** (`--api-only`, or `--remote-only` which forces API-only). An `api_key_authentication` middleware runs on every request. It lets `GET /api/health` through unconditionally, then reads `X-Secret-Key`. An empty header returns the *"Secret Key is Empty."* body; a header that is not a configured user returns *"Secret Key is Invalid."*. In this mode the accepted-keys map holds your `api_secret` (labelled `api_user`, with no devices) plus every key under `serve.users`, so a per-user key authenticates here exactly as it does on the integrated server, and that user's `services` allowlist applies as usual. With `--no-key`, this middleware is simply not installed and the users map is empty. +- **API-only server** (`--api-only`, or `--remote-only` which forces API-only). An `api_key_authentication` middleware runs on every request. It lets `GET /api/health` through unconditionally, then reads `X-Secret-Key`. An empty header returns the *"Secret Key is Empty."* body. A header that is not a configured user returns *"Secret Key is Invalid."*. In this mode the accepted-keys map holds your `api_secret` (labelled `api_user`, with no devices) plus every key under `serve.users`. A per-user key therefore authenticates here exactly as it does on the integrated server, and that user's `services` allowlist applies as usual. With `--no-key`, this middleware is not installed and the users map is empty. - **Integrated server** (default). A `serve_authentication` middleware wraps pywidevine's authentication (and pyplayready's for `/playready*` paths), and the REST routes live in the same application, guarded by the same check. Here the accepted-keys map is your full `serve` config: `api_secret` (auto-added with access to all loaded devices) plus every entry under `users`, each with its declared `devices` / `playready_devices`. Because the API-only middleware guards *all* non-health paths, the Swagger UI at `/api/docs/` is also behind `X-Secret-Key` there. In `--remote-only` mode the Swagger UI is not mounted at all. -Both middlewares read the caller's key through the shared `request_secret_key()` helper in `unshackle/core/api/handlers.py`, which returns the `X-Secret-Key` header and falls back to the `secret_key` query parameter only when the matched route is the job event stream. The integrated server needs one extra branch for this: pywidevine's `authentication` is header-only, so `serve_authentication` compares the query-parameter key itself when the header is absent, and delegates to pywidevine in every other case. That is why a present-but-wrong header is rejected in both modes. +Both middlewares read the caller's API key through the shared `request_secret_key()` helper in `unshackle/core/api/handlers.py`. That helper returns the `X-Secret-Key` header, and falls back to the `secret_key` query parameter only when the matched route is the job event stream. The integrated server needs one extra branch for this. pywidevine's `authentication` is header-only. So `serve_authentication` compares the API key from the query parameter itself when the header is absent, and delegates to pywidevine in every other case. That is why both modes reject a present-but-wrong header. ### CORS and the `Authorization` header @@ -318,8 +318,8 @@ Access-Control-Allow-Headers: Content-Type, X-Secret-Key, Authorization Access-Control-Max-Age: 3600 ``` -The REST API authenticates with `X-Secret-Key`; `Authorization` is advertised because the co-hosted pywidevine / pyplayready CDM endpoints use it. `PUT` appears in the allowed methods list but no REST route uses it; only `GET`, `POST`, and `DELETE` are wired up. +The REST API authenticates with `X-Secret-Key`. The middleware advertises `Authorization` because the co-hosted pywidevine / pyplayready CDM endpoints use it. `PUT` appears in the allowed-methods list, but no REST route uses it. Only `GET`, `POST`, and `DELETE` have routes. ### Where device access is read -The map of accepted keys lives at `request.app["config"]["users"]`. For DRM operations, a key's permitted Widevine devices come from its `devices` list and PlayReady devices from `playready_devices`; the first configured device is used when a request does not (or may not) override it. A server-wide `config.cdm[]` mapping, if present, takes precedence over the key's device list when resolving which device to use. +The map of accepted API keys lives at `request.app["config"]["users"]`. For DRM operations, an API key's permitted Widevine devices come from its `devices` list, and PlayReady devices from `playready_devices`. The server uses the first configured device when a request does not (or may not) override it. A server-wide `config.cdm[]` mapping, if present, takes precedence over the API key's device list when the server chooses which device to use. diff --git a/docs/dev/rest-api/endpoints.md b/docs/dev/rest-api/endpoints.md index 47520282..0653a356 100644 --- a/docs/dev/rest-api/endpoints.md +++ b/docs/dev/rest-api/endpoints.md @@ -1,25 +1,25 @@ # REST API Endpoints -This is the full endpoint reference for the unshackle REST API. It covers every route: its method and path, the request body it accepts, the response shape it returns, and the status codes it can produce. Routes are grouped into general, download jobs, server management, and remote-dl sessions. +This is the full endpoint reference for the unshackle REST API. It covers every route: its method and path, the request body it accepts, the response shape it returns, and the status codes it can give. This page groups the routes into general, download jobs, server management, and remote-dl sessions. -Start the server with [`unshackle serve`](../../guide/cli-reference.md). By default it binds to `127.0.0.1:8786`, and every route lives under the `/api/` prefix. Authentication, server modes, and the `X-Secret-Key` header are covered in detail on the [Authentication](authentication.md) page. Read that first if you have not already. For an end-to-end walkthrough, see the [Quickstart](quickstart.md). +Start the server with [`unshackle serve`](../../guide/cli-reference.md). By default it binds to `127.0.0.1:8786`, and every route lives under the `/api/` prefix. The [Authentication](authentication.md) page gives authentication, server modes, and the `X-Secret-Key` header in detail. Read that first if you have not already. For an end-to-end walkthrough, see the [Quickstart](quickstart.md). !!! note "Base URL" - All examples below assume a base URL of `http://127.0.0.1:8786`. Replace the host and port to match your own `unshackle serve -h -p ` invocation. + All examples below assume a base URL of `http://127.0.0.1:8786`. Replace the host and port so that they match your own `unshackle serve -h -p ` invocation. !!! tip "Interactive docs" - A live Swagger UI is mounted at **`/api/docs/`** whenever the server is not running in `--remote-only` mode. Its title is "Unshackle REST API" and its version tracks the installed package version. In API-only mode with a key configured, the Swagger UI itself sits behind `X-Secret-Key`; only `/api/health` is exempt. + The server mounts a live Swagger UI at **`/api/docs/`** whenever it does not operate in `--remote-only` mode. Its title is "Unshackle REST API" and its version tracks the installed package version. In API-only mode with an API key configured, the Swagger UI itself sits behind `X-Secret-Key`. Only `/api/health` is exempt. ## Conventions -- **Authentication.** Every request except `GET /api/health` requires the `X-Secret-Key` header when a key is configured. Starting the server with `--no-key` disables the check entirely. See [Authentication](authentication.md). -- **Content type.** Request bodies are JSON (`Content-Type: application/json`); all success responses are JSON unless a `204 No Content` is documented. -- **CORS.** Every response carries permissive CORS headers, and `OPTIONS` preflight requests are answered automatically. +- **Authentication.** Every request except `GET /api/health` requires the `X-Secret-Key` header when you configure an API key. `--no-key` disables the check entirely. See [Authentication](authentication.md). +- **Content type.** Request bodies are JSON (`Content-Type: application/json`). All success responses are JSON unless this page documents a `204 No Content`. +- **CORS.** Every response carries permissive CORS headers, and the server answers `OPTIONS` preflight requests automatically. - **Compression.** JSON responses of 256 bytes or more are gzip-compressed when the client sends `Accept-Encoding: gzip`. -- **Service allowlist.** Service-facing endpoints are filtered by the effective allowlist for your API key (the intersection of the global `serve.services` list and your per-key list). Services you are not allowed to use are treated as unknown. +- **Service allowlist.** The effective allowlist for your API key filters the service-facing endpoints (the intersection of the global `serve.services` list and your per-key list). The server treats services you cannot use as unknown. !!! warning "Error responses have two shapes" - Structured application errors (raised inside a handler) return `{"status": "error", "error_code": "...", "message": "...", "timestamp": "..."}`. The authentication middleware, however, returns a **different** shape: `{"status": 401, "message": "..."}` with an integer `status` and no `error_code`. Clients must handle both. The full error model is documented in the [Error responses](#error-responses) section at the bottom of this page. + Structured application errors (raised inside a handler) return `{"status": "error", "error_code": "...", "message": "...", "timestamp": "..."}`. The authentication middleware, however, returns a **different** shape: `{"status": 401, "message": "..."}` with an integer `status` and no `error_code`. Clients must accept both shapes. The [Error responses](#error-responses) section at the bottom of this page gives the full error model. --- @@ -52,7 +52,7 @@ Health check. **This is the only route exempt from authentication**: you can cal } ``` -`code_hash` fingerprints the framework source the server is running, so it identifies the code itself rather than the release. It is the same value for a git checkout and for an installed package built from that checkout, and it changes if anything under `unshackle/core`, `unshackle/commands`, `unshackle/utils`, or `unshackle/vaults` is edited. Service modules are not included. The value is `null` only when the source files cannot be read. +`code_hash` fingerprints the framework source the server operates, so it names the code itself rather than the release. It is the same value for a git checkout and for an installed package built from that checkout. It changes when you edit anything under `unshackle/core`, `unshackle/commands`, `unshackle/utils`, or `unshackle/vaults`. Service modules are not included. unshackle sets the value to `null` only when it cannot read the source files. If the update check fails (for example, no network), `update_available` and `latest_version` are `null` while `current_version` still reports the installed version. @@ -62,7 +62,7 @@ If the update check fails (for example, no network), `update_available` and `lat ### `GET /api/services` -List the streaming services available on this server, filtered by your allowlist. Each entry describes the service's tag, matching rules, capability flags, and its CLI parameters (useful for building a UI that accepts service-specific options). +Show the streaming services available on this server, filtered by your allowlist. Each entry gives the service's tag, matching rules, capability flags, and its CLI parameters (useful when you make a UI that accepts service-specific options). === "Request" @@ -127,7 +127,7 @@ Field notes: ### `POST /api/search` -Search a service for titles by query string. The service must implement `search()` (see `has_search` above). +Find titles in a service by query string. The service must have a `search()` method (see `has_search` above). **Request body** @@ -175,7 +175,7 @@ Search a service for titles by query string. The service must implement `search( ### `POST /api/list-titles` -Resolve a title ID into its list of titles (episodes, movie, etc.) without downloading anything. +Get the list of titles behind a title ID, for example episodes or a movie, without downloading anything. **Request body** @@ -188,7 +188,7 @@ Resolve a title ID into its list of titles (episodes, movie, etc.) without downl | `no_proxy` | boolean | no | Force-disable proxy. | | `cdm_type` | string | no | Preferred CDM type. | -Service-specific CLI options may also be passed as additional keys. +You can also pass service-specific CLI options as additional keys. === "Request" @@ -221,13 +221,13 @@ Service-specific CLI options may also be passed as additional keys. } ``` -Each serialized title carries `type` (`"episode"`, `"movie"`, or `"other"`), `name`, `id`, `language`, `description`, `date`, and `cover_url`. Episodes and movies add `year`; episodes additionally add `series_title`, `season`, and `number`. +Each serialized title carries `type` (`"episode"`, `"movie"`, or `"other"`), `name`, `id`, `language`, `description`, `date`, and `cover_url`. Episodes and movies add `year`. Episodes also add `series_title`, `season`, and `number`. !!! warning "A song serializes as `"other"`" - A music track has no `type` of its own. It is sent as `"other"`, and the response + A song has no `type` of its own. The server sends it as `"other"`, and the response carries none of its music fields: no artist, no album, no track or disc number, and no - `year`. `name` holds the track name. To read the disc and track numbers of a release, - run the CLI with `--list-titles`. Selecting music tracks over the API still works, + `year`. `name` holds the song name. To read the disc and track numbers of a release, + operate the CLI with `--list-titles`. You can still select songs over the API, through the `wanted` parameter. These keys are sent only when the title carries them, so a title without them serializes exactly as before: @@ -243,8 +243,8 @@ These keys are sent only when the title carries them, so a title without them se !!! info "`part` on a split episode" A service that splits one episode into several separately playable videos reports each one as its own episode. Those titles share a `season` and `number` and add a `part` - integer counting from 1. No other title carries the key, so a client that ignores it - is unaffected. See [Split episodes](../creating-a-service.md#split-episodes). + integer counting from 1. No other title carries the `part` field, so a client can + ignore it safely. See [Split episodes](../creating-a-service.md#split-episodes). | Status | Error code | Meaning | | --- | --- | --- | @@ -257,10 +257,10 @@ These keys are sent only when the title carries them, so a title without them se ### `POST /api/list-tracks` -List the video, audio, and subtitle tracks for a title. For series, you can scope to specific episodes. +Show the video, audio, and subtitle tracks for a title. For series, you can scope to specific episodes. !!! tip "Spotting the original-language audio" - Each audio track carries `is_original`, which marks the track the default `lang` of `orig` selects. Unshackle resolves it with the same language matcher the downloader uses, so it stays correct for regional variants a client cannot separate from tag text alone (`pt` marks `pt-BR` rather than `pt-PT`). Services need not report a title's language, so every track can legitimately come back `false`. + Each audio track carries `is_original`, which marks the track the default `lang` of `orig` selects. unshackle finds it with the same language matcher the downloader uses. Thus it stays correct for regional variants a client cannot separate from tag text alone (`pt` marks `pt-BR` rather than `pt-PT`). Services need not report a title's language, so every track can legitimately come back `false`. **Request body** @@ -274,7 +274,7 @@ List the video, audio, and subtitle tracks for a title. For series, you can scop | `part` | int/string | no | Part index of a split episode (combined with `season` and `episode`). | | `profile`, `proxy`, `no_proxy`, `cdm_type` | - | no | As on `list-titles`. | -When both `season` and `episode` are given, they are combined into a `"{season}x{episode}"` selector. Adding `part` narrows that to `"{season}x{episode}.{part}"`, one part of a [split episode](../creating-a-service.md#split-episodes); without it you get every part of the episode. `part` is read only when `season` and `episode` are both given, and `wanted` takes precedence over all three. To scope a part by range, put it in `wanted` instead (`"S01E01.2"`). +When you give both `season` and `episode`, the server combines them into a `"{season}x{episode}"` selector. `part` narrows that to `"{season}x{episode}.{part}"`, one part of a [split episode](../creating-a-service.md#split-episodes). Without `part` you get every part of the episode. The server reads `part` only when you give both `season` and `episode`, and `wanted` takes precedence over all three. To scope a part by range, put it in `wanted` instead (`"S01E01.2"`). === "Single title / movie: Response `200`" @@ -321,7 +321,7 @@ When both `season` and `episode` are given, they are combined into a `"{season}x } ``` -Tracks are sorted by bitrate, descending. Episodes that fail to resolve are collected into `unavailable_episodes` rather than failing the whole request. Track objects here do **not** include a download `url`; segment URLs are only exposed through the remote-dl session endpoints. +The server sorts tracks by bitrate, descending. Episodes the server cannot get go into `unavailable_episodes`, rather than failing the whole request. Track objects here do **not** include a download `url`. Only the remote session endpoints give segment URLs. The `drm` array on a track is a list of `{type, pssh?, kids?, content_keys?, license_url?}` objects. @@ -337,14 +337,14 @@ The `drm` array on a track is a list of `{type, pssh?, kids?, content_keys?, lic ## Download jobs -Downloads are asynchronous. `POST /api/download` enqueues a job and returns immediately with a `job_id`; the server runs each job in a background worker subprocess. You then poll the job endpoints for progress and results. +Downloads are asynchronous. `POST /api/download` enqueues a job and returns immediately with a `job_id`. The server runs each job in a background job worker subprocess. You then poll the job endpoints for progress and results. !!! info "Job states" - A job moves through `queued` → `downloading` → one of the terminal states `completed`, `failed`, or `cancelled`. Only queued jobs can be prioritized; only terminal jobs can be retried or removed. + A job moves through `queued` → `downloading` → one of the terminal states `completed`, `failed`, or `cancelled`. You can prioritise only queued jobs. You can retry or remove only terminal jobs. ### `POST /api/download` -Create a download job. Requires `service` and `title_id`; every other field is an optional download parameter with a server-side default. On success this returns **`202 Accepted`**, not `200`. +Make a download job. It requires `service` and `title_id`. Every other field is an optional download parameter with a server-side default. On success this returns **`202 Accepted`**, not `200`. === "Request" @@ -418,22 +418,22 @@ Create a download job. Requires `service` and `title_id`; every other field is a | `output_dir` | string | `null` | Override output directory. | | `no_cache` / `reset_cache` | boolean | `false` | Title cache controls. | -**Validation.** Invalid values return `400 INVALID_PARAMETERS`. Notably: `vcodec` must be one of H264/H265/H.264/H.265/AVC/HEVC/VC1/VC-1/VP8/VP9/AV1; `acodec` one of AAC/AC3/EC3/EAC3/DD/DD+/AC4/OPUS/FLAC/ALAC/VORBIS/OGG/DTS; `range` one of SDR/HDR10/HDR10P/DV/HLG/HYBRID (`HDR10+` is accepted); bitrate/worker/download counts must be positive integers; at most one of the `*_only` flags may be set; `no_subs` cannot be combined with `subs_only`, `no_audio` with `audio_only`, or `s_lang` with `require_subs`. +**Validation.** Invalid values return `400 INVALID_PARAMETERS`. `vcodec` must be one of H264/H265/H.264/H.265/AVC/HEVC/VC1/VC-1/VP8/VP9/AV1. `acodec` must be one of AAC/AC3/EC3/EAC3/DD/DD+/AC4/OPUS/FLAC/ALAC/VORBIS/OGG/DTS. `range` must be one of SDR/HDR10/HDR10P/DV/HLG/HYBRID, and `HDR10+` is also valid. The bitrate, download worker, and download counts must be positive integers. You may set at most one of the `*_only` flags. You cannot combine `no_subs` with `subs_only`, `no_audio` with `audio_only`, or `s_lang` with `require_subs`. !!! danger "Rejected parameters" `postscript`, `post_script` and `post_scripts` are never accepted. A body containing any - of them is rejected with `400 INVALID_PARAMETERS`, because a command string arriving from - an HTTP caller would be remote code execution. Post-download hooks are configured only in - `unshackle.yaml`, where they still run for API jobs. See + of them gets `400 INVALID_PARAMETERS`, because a command string that arrives from + an HTTP caller would be remote code execution. You configure post-download hooks only in + `unshackle.yaml`, where they still operate for API jobs. See [Post-download scripts](../../reference/configuration/post-scripts.md). !!! warning "Gated parameters (developer)" - A few keys are accepted but only permitted when the server config opts in, otherwise the request is rejected with `403 FORBIDDEN`: + The server accepts a few keys, but permits them only when the server config opts in. Otherwise the server rejects the request with `403 FORBIDDEN`: - `cdm` (per-request CDM device name): allowed only if `serve.cdm_overrides` is `true` or a list containing that device name. - `credential` (`"user:pass"`) and `credentials` (a map): allowed only if `serve.allow_job_credentials` is truthy (off by default). - Server-side, effective parameters are merged as defaults → `serve:` config overrides → service-specific defaults → your request body. + On the server, unshackle merges the effective parameters in this order: defaults → `serve:` config overrides → service-specific defaults → your request body. | Status | Error code | Meaning | | --- | --- | --- | @@ -445,7 +445,7 @@ Create a download job. Requires `service` and `title_id`; every other field is a ### `GET /api/download/jobs` -List download jobs, with optional filtering and sorting. +Show download jobs, with optional filtering and sorting. **Query parameters** @@ -533,10 +533,10 @@ The `data` field of every event is the same job object that `GET /api/download/j The server sends a `: keep-alive` comment each 15 seconds while the job is quiet, to keep proxies from closing an idle connection. Ignore these lines. -If the job is already in a terminal state, the server sends the `snapshot` event, then the terminal event, and closes the stream immediately. +If the job is already in a terminal state, the server sends the `snapshot` event, then the terminal event, and closes the event stream immediately. !!! tip "Authentication from a browser" - `EventSource` cannot set headers. For this endpoint only, the server also accepts the key in the `secret_key` query parameter. This works in both server modes, the default integrated server and `--api-only`: + `EventSource` cannot set headers. For this endpoint only, the server also accepts the API key in the `secret_key` query parameter. This works in both server modes, the default integrated server and `--api-only`: ```javascript const events = new EventSource( @@ -546,7 +546,7 @@ If the job is already in a terminal state, the server sends the `snapshot` event events.addEventListener("completed", () => events.close()); ``` - The header always wins. Send the key in the query parameter *or* in the header, not both: if an `X-Secret-Key` header is present, the server checks that header and ignores the query parameter. See [Authentication](authentication.md#how-clients-present-credentials). + The header always wins. Send the API key in the query parameter *or* in the header, not both. If an `X-Secret-Key` header is present, the server examines that header and ignores the query parameter. See [Authentication](authentication.md#how-clients-present-credentials). === "Request" @@ -583,7 +583,7 @@ Cancel or remove a job. The behaviour depends on the job's current state: - **Queued or downloading job**: cancelled, returning `200` `{"status": "success", "message": "Job cancelled"}`. !!! warning - A `204` response has no body. Do not attempt to parse JSON from it. + A `204` response has no body. Do not try to parse JSON from it. | Status | Error code | Meaning | | --- | --- | --- | @@ -607,7 +607,7 @@ Remove every terminal (completed/failed/cancelled) job from the manager. No requ Enqueue a **new** job that reuses a terminal job's service, title, and parameters. The service allowlist and parameter gates are re-checked. Returns `202` with the new job's ID. !!! warning "Retry reuses the original, unredacted parameters" - Retry replays the finished job's stored parameters exactly as they were submitted, including any secrets such as `credential`/`credentials`, not the redacted `parameters` view that `GET /api/download/jobs` and `GET /api/history` expose. This is deliberate: it lets a retried job reproduce a credentialed download that the redacted job record alone never could. It also means anyone who can retry a job can re-run it with its original credentials, even though those credentials are masked everywhere else in the API. + A retry replays the finished job's stored parameters exactly as the original client submitted them, including any secrets such as `credential`/`credentials`. It does not use the redacted `parameters` view that `GET /api/download/jobs` and `GET /api/history` give. This is deliberate: it lets a retried job reproduce a credentialed download that the redacted job record alone never could. The API masks those credentials everywhere else. But anyone who can retry a job can operate it again with the original credentials. === "Response `202`" @@ -651,7 +651,7 @@ Endpoints for profiles, effective configuration, download history, environment c ### `GET /api/profiles` -List the named credential profiles configured per service (usable as the `profile` parameter elsewhere). Only services whose credentials are a mapping of profile name to credential are listed; services with a single unnamed credential are omitted. Filtered by your allowlist. +Show the named credential profiles configured per service (usable as the `profile` parameter elsewhere). The response holds only services whose credentials are a mapping of profile name to credential. It leaves out services with a single unnamed credential. Your allowlist filters the response. === "Response `200`" @@ -666,7 +666,7 @@ List the named credential profiles configured per service (usable as the `profil ### `GET /api/config` -Return a read-only, redacted view of the effective server configuration, suitable for a settings page in a UI. Secrets are never emitted: keys matching `secret`, `password`, `token`, `api_key`, or `credential`, and any userinfo in proxy URLs, are masked. +Return a read-only, redacted view of the effective server configuration, suitable for a settings page in a UI. The server never sends secrets: it masks every config key whose name contains `secret`, `password`, `token`, `api_key`, or `credential`, and any userinfo in proxy URLs. === "Response `200`" @@ -695,7 +695,7 @@ Return a read-only, redacted view of the effective server configuration, suitabl ### `GET /api/history` -Read the persisted history of jobs that reached a terminal state, newest first. History is stored in `api_history.jsonl` in the cache directory; corrupt lines are skipped and a missing file yields an empty list. +Read the persisted history of jobs that reached a terminal state, newest first. unshackle stores the history in `api_history.jsonl` in the cache directory. It skips corrupt lines, and a missing file yields an empty list. **Query parameters** @@ -742,12 +742,12 @@ Remove a single persisted history entry. Returns **`204 No Content`** (empty bod ### `POST /api/maintenance/clear-cache` and `POST /api/maintenance/clear-temp` -Delete and recreate the cache or temp directory. Neither takes a request body. Both are blocked with `409 CONFLICT` if any job is currently `downloading` (the `details` include the offending `active_jobs`). +Delete and recreate the cache or temp directory. Neither takes a request body. The server blocks both with `409 CONFLICT` if any job is currently `downloading` (the `details` include the offending `active_jobs`). -The `409` guard is about active file I/O, not an arbitrary lock: an in-flight job may still be *reading* cached title data under the cache directory, and active download workers are *writing* segment files under temp. Removing either directory mid-download would corrupt those running jobs, so the endpoints refuse until nothing is downloading. +The `409` guard is about active file I/O, not an arbitrary lock. An in-flight job may still be *reading* cached title data under the cache directory, and active download workers are *writing* segment files under temp. The removal of either directory mid-download would corrupt those jobs, so the endpoints refuse while a download is in progress. !!! warning "clear-cache also wipes download history" - The persistent history read by [`GET /api/history`](#get-apihistory) lives in `api_history.jsonl` **inside the cache directory**. Because clear-cache deletes and recreates that whole directory, calling it also erases your entire download history, not just cached title data. If you want to keep history, do not clear the cache; remove individual entries with [`DELETE /api/history/{job_id}`](#delete-apihistoryjob_id) instead. + The persistent history read by [`GET /api/history`](#get-apihistory) lives in `api_history.jsonl` **inside the cache directory**. Because clear-cache deletes and recreates that whole directory, calling it also erases your entire download history, not only cached title data. If you want to keep history, do not clear the cache. Remove individual entries with [`DELETE /api/history/{job_id}`](#delete-apihistoryjob_id) instead. === "Response `200`" @@ -762,7 +762,7 @@ The `409` guard is about active file I/O, not an arbitrary lock: an in-flight jo ### `POST /api/maintenance/refresh-services` -git-pull every service repository configured under `directories.services`. `refreshed` is `true` when all repos updated successfully (or none are configured). +git-pull every service repository configured under `directories.services`. `refreshed` becomes `true` when all repos updated successfully, or when you configure no repository. === "Response `200`" @@ -795,11 +795,11 @@ Report the install status of the external binaries unshackle depends on (the sam ## Remote-dl sessions !!! note "Developer feature" - The `/api/session/*` endpoints power unshackle's **remote download** mode, where a thin local client authenticates once against a remote server, then fetches titles, tracks, and segments and proxies DRM licensing back through it. These endpoints are exposed in `--remote-only` mode, and they are what the built-in `RemoteService` client drives. Most end users never call them directly. + The `/api/session/*` endpoints power unshackle's **remote download** mode. A thin local client authenticates once against a remote server, then fetches titles, tracks, and segments, and proxies DRM licensing back through it. The server mounts these endpoints in `--remote-only` mode, and they are what the built-in `RemoteService` client drives. Most end users never call them directly. -Sessions are held in memory. They expire after `serve.session_ttl` (default 300s), except while awaiting interactive authentication input, which uses a 600s window. A session is bound to the IP that created it, so a request from a different IP is rejected with `403 FORBIDDEN`. +The server holds remote sessions in memory. They expire after `serve.session_ttl` (default 300s), except while awaiting interactive authentication input, which uses a 600s window. A remote session binds to the IP that made it, so the server rejects a request from a different IP with `403 FORBIDDEN`. -### Session lifecycle overview +### Remote session lifecycle overview ```text POST /api/session/create → returns session_id, status "authenticating" @@ -813,7 +813,7 @@ DELETE /api/session/{id} → tear down, harvest updated cache ### `POST /api/session/create` -Create a session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required) plus a broad set of optional keys (`additionalProperties` is allowed): `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files), `client_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, and any service CLI options. +Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files), `client_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, and any service CLI options. === "Response `200`" @@ -883,7 +883,7 @@ Submit a response to a pending prompt. Body: `{ "response": "..." }` (required). ### `GET /api/session/{session_id}/titles` -Fetch the titles for an authenticated session. +Fetch the titles for an authenticated remote session. === "Response `200`" @@ -894,11 +894,11 @@ Fetch the titles for an authenticated session. } ``` -Requires an authenticated session; `404 SESSION_NOT_FOUND` otherwise. +Requires an authenticated remote session. Otherwise it returns `404 SESSION_NOT_FOUND`. ### `POST /api/session/{session_id}/tracks` -Get tracks and chapters for a specific title within the session. Body: `{ "title_id": "..." }` (required). Unlike `list-tracks`, track objects here **include** download URLs, along with session headers/cookies and manifest data the client needs to download directly. +Get tracks and chapters for a specific title within the remote session. Body: `{ "title_id": "..." }` (required). Unlike `list-tracks`, track objects here **include** download URLs, along with session headers/cookies and manifest data the client needs to download directly. === "Response `200`" @@ -925,7 +925,7 @@ Get tracks and chapters for a specific title within the session. Body: `{ "title ### `POST /api/session/{session_id}/segments` -Resolve the download URL, DRM info, headers, cookies, and raw track data for selected tracks. Body: `{ "track_ids": ["v-1", "a-1"] }` (required). +Get the download URL, DRM info, headers, cookies, and raw track data for the selected tracks. Body: `{ "track_ids": ["v-1", "a-1"] }` (required). === "Response `200`" @@ -945,7 +945,7 @@ Resolve the download URL, DRM info, headers, cookies, and raw track data for sel ### `POST /api/session/{session_id}/license` -Acquire DRM keys. Two modes are supported via the `mode` field. +Get the content keys for the DRM. The `mode` field selects one of two modes. **Proxy mode** (`mode: "proxy"`, the default) forwards the client's CDM `challenge` to the service's license endpoint and returns the raw license bytes: @@ -994,7 +994,7 @@ Acquire DRM keys. Two modes are supported via the `mode` field. ### `GET /api/session/{session_id}` -Return session metadata and validity. +Return remote session metadata and validity. === "Response `200`" @@ -1009,11 +1009,11 @@ Return session metadata and validity. } ``` -`404 SESSION_NOT_FOUND` if the session does not exist or has expired. +`404 SESSION_NOT_FOUND` if the remote session does not exist or has expired. ### `DELETE /api/session/{session_id}` -Tear down a session. Its input bridge is cancelled, any updated cache files are harvested, and the session plus its cache directory are deleted. +Tear down a remote session. The server cancels its input bridge, harvests any updated cache files, and deletes the remote session and its cache directory. === "Response `200`" @@ -1024,13 +1024,13 @@ Tear down a session. Its input bridge is cancelled, any updated cache files are } ``` -The `cache` field is present only if the session produced cache files. `404 SESSION_NOT_FOUND` if the session does not exist. +The `cache` field is present only if the remote session produced cache files. `404 SESSION_NOT_FOUND` if the remote session does not exist. --- ## Error responses -Every structured error raised inside a handler is serialized to the same JSON envelope: +The server serialises every structured error raised inside a handler to the same JSON envelope: ```json { @@ -1046,7 +1046,7 @@ Every structured error raised inside a handler is serialized to the same JSON en - `details` appears only when non-empty. - `retryable` appears only when the error is retryable (e.g. network errors, rate limiting). -- `debug_info` appears only when the server was started with `--debug-api`. +- `debug_info` appears only when you start the server with `--debug-api`. !!! warning "Authentication errors differ" Errors raised by the API-key middleware use a distinct shape (`{"status": 401, "message": "Secret Key is Empty."}` or `{"status": 401, "message": "Secret Key is Invalid."}`) with an integer `status` and no `error_code`, `timestamp`, or `details`. A malformed JSON body on any POST, by contrast, returns the standard envelope with `error_code: "INVALID_INPUT"`. diff --git a/docs/dev/rest-api/errors.md b/docs/dev/rest-api/errors.md index 8f708793..9453d476 100644 --- a/docs/dev/rest-api/errors.md +++ b/docs/dev/rest-api/errors.md @@ -1,20 +1,20 @@ # Error Responses Every endpoint of the unshackle REST API reports failures in a single, predictable -shape. Whether a request was rejected because a field was missing, a service refused -to authenticate, or the download worker crashed, the server answers with a JSON body +shape. The cause can be a missing field, a service that refuses to authenticate, or a +download worker that crashed. In each case the server answers with a JSON body that carries a machine-readable **error code**, a human-readable **message**, and a matching **HTTP status**. This page documents that format and catalogues every error -code the server can emit. +code the server can send. -If you are building a client against the API, read this page alongside the endpoint -reference. Each endpoint lists the specific codes it raises, and the meanings of -those codes are defined here. +If you write a client against the API, read this page alongside the endpoint +reference. Each endpoint lists the specific codes it raises, and this page gives the +meanings of those codes. !!! note "Where this comes from" - The error format is defined in `unshackle/core/api/errors.py`, and the wrapper + `unshackle/core/api/errors.py` defines the error format, and the wrapper that turns raised errors into responses lives in `unshackle/core/api/routes.py`. - Nothing on this page is invented; it reflects the actual server behaviour. + Nothing on this page is invented. It reflects the actual server behaviour. ## The standard error shape @@ -59,7 +59,7 @@ included when it is non-empty, so a client must treat a missing `details` as "no extra information". Some concrete examples you may encounter: - A validation failure may include the offending field or the list of valid values. -- A "title not found" error on a remote-dl session includes `available_titles`. +- A "title not found" error on a remote session includes `available_titles`. - A maintenance conflict includes `active_jobs`, the jobs blocking the operation. - Categorised exceptions add a `reason` tag, such as `"network_connectivity"` or `"geofence_restriction"`. @@ -68,8 +68,8 @@ extra information". Some concrete examples you may encounter: The `retryable` field appears **only when it is `true`**. It is set for conditions the server believes are transient: network errors, rate limiting, and temporary -service unavailability. When you see it, backing off and retrying is reasonable; when -it is absent, assume the request will keep failing until you change something. +service unavailability. When you see it, you can back off and try the request again. +When it is absent, assume the request will keep failing until you change something. !!! example "A retryable network error" ```json @@ -125,7 +125,7 @@ but the defaults hold for the overwhelming majority of responses. If a code is ever emitted without a mapped status, the server falls back to `500`. In practice every code above has an explicit mapping. -## How errors are produced +## How the server produces errors Understanding where an error comes from helps interpret it. @@ -171,8 +171,8 @@ the server returns `INVALID_INPUT` (400) with the parse error in ## The authentication error exception -There is one place where the response does **not** follow the standard shape: the API -key authentication middleware. When you send a request to a protected endpoint +There is one place where the response does **not** follow the standard shape: the +API key authentication middleware. When you send a request to a protected endpoint without a valid `X-Secret-Key` header, the middleware short-circuits with a legacy body that uses an integer `status` and no `error_code` or `timestamp`. @@ -189,7 +189,7 @@ body that uses an integer `status` and no `error_code` or `timestamp`. ``` !!! warning "Handle both shapes" - A robust client must expect two error formats: the standard structured error + A client must expect two error formats: the standard structured error (string `status` of `"error"`, with `error_code`), and this authentication error (integer `status`, no `error_code`). The `GET /api/health` endpoint is exempt from authentication, so it never returns the key-error shape. @@ -201,13 +201,13 @@ Not every failure or completion has a JSON body. Some endpoints answer with deleting a history entry. !!! warning "Do not parse `204` bodies" - A `204` response has no body. Clients must check the status code before attempting - to decode JSON, or the decode will fail on an empty string. + A `204` response has no body. Clients must examine the status code before they + decode JSON, or the decode will fail on an empty string. ## Debug information -By default, error responses never include tracebacks or internal diagnostics. If the -server was started with the `--debug-api` flag, every structured error gains a +By default, error responses never include tracebacks or internal diagnostics. If you +start the server with the `--debug-api` flag, every structured error gains a `debug_info` object: ```json title="Error with --debug-api enabled" @@ -225,7 +225,7 @@ server was started with the `--debug-api` flag, every structured error gains a The `debug_info` object always carries `exception_type` and, for exception-derived errors, a formatted `traceback`. Some endpoints add extra diagnostics such as -captured worker stderr. +captured job worker stderr. !!! warning "Developer use only" `--debug-api` exposes internal tracebacks in responses. Enable it only for local @@ -235,16 +235,16 @@ captured worker stderr. For anyone writing a consumer of the API, a durable strategy is: -1. **Read the HTTP status first.** It tells you the broad class of outcome and lets - you handle the `204`/no-body and authentication-error cases before touching the +1. **Read the HTTP status first.** It tells you the broad class of outcome, and it + lets you find the `204`/no-body and authentication-error cases before you read the body. 2. **Branch on `error_code`.** For structured errors, this is the stable identifier - to switch on. Do not match on `message`. -3. **Respect `retryable`.** When it is `true`, back off and retry; the reference - client retries `429`, `500`, `502`, `503`, and `504` with exponential backoff. When - absent, fix the request rather than repeating it. -4. **Fall back gracefully.** Unrecognised codes should be surfaced using the - `message` plus the HTTP status. + to switch on. Do not use `message` as the identifier. +3. **Respect `retryable`.** When it is `true`, back off and try the request again. The + reference client retries `429`, `500`, `502`, `503`, and `504` with exponential + backoff. When absent, fix the request rather than repeating it. +4. **Fall back gracefully.** For an unrecognised code, show the `message` and the + HTTP status. !!! note "Reference consumer" unshackle's own remote-download client treats any status of `400` or above as diff --git a/docs/dev/rest-api/index.md b/docs/dev/rest-api/index.md index 5c701958..80c108af 100644 --- a/docs/dev/rest-api/index.md +++ b/docs/dev/rest-api/index.md @@ -1,7 +1,7 @@ # REST API Overview unshackle ships with a built-in HTTP server. Alongside the command-line tool, you -can run `unshackle serve` and drive the same download engine over a REST API: +can operate `unshackle serve` and drive the same download engine over a REST API: searching services, listing titles and tracks, queueing downloads, watching live progress, and browsing history, all as JSON over HTTP. @@ -9,7 +9,7 @@ This page explains what the API is for, how to start the server, where it listen and how the endpoints are organised. Later pages cover each area in depth. !!! info "Who this section is for" - The REST API is aimed at people building automation, dashboards, or remote + The REST API is for people who make automation, dashboards, or remote clients on top of unshackle. If you only download the occasional title, the [command-line tool](../../guide/cli-reference.md) does everything the API does and is simpler to use. Reach for the API when you want a long-running service, a @@ -20,20 +20,20 @@ and how the endpoints are organised. Later pages cover each area in depth. The server exposes the same capabilities as the `dl` command, but as discrete HTTP calls you can script against: -- **Discovery**: list the services available on the server, run a service's search, +- **Discovery**: show the services available on the server, use a service's search, and enumerate the titles and tracks behind a title ID. - **Downloading**: submit a download as a background *job*, then follow its status and live progress (by polling, or by subscribing to its Server-Sent Events stream), cancel it, retry it, or bump it to the front of the queue. - **History and housekeeping**: read the log of finished jobs, inspect the server's - effective (redacted) configuration, check external tool versions, and clear caches + effective (redacted) configuration, examine external tool versions, and clear caches or temp files. - **Remote-download sessions**: a thin local client can authenticate against a service *on the server*, then pull titles, tracks, and segment info back, and proxy DRM licensing. This is how one machine with the credentials and CDMs can serve downloads to another. See [Remote Sessions](remote-sessions.md). -Because every download runs as a queued job in a separate worker process, the API is +Because every download runs as a queued job in a separate job worker process, the API is well suited to running unattended: submit work, come back later, and read the result. ## Running the server @@ -77,27 +77,27 @@ unshackle serve --api-only --host 0.0.0.0 --port 8786 !!! danger "Binding to a public interface" `--no-key` disables authentication completely. Anyone who can reach the port can - search, download, and read your configuration. Only combine it with a host like + find titles, download them, and read your configuration. Only combine it with a host like `127.0.0.1`, or keep the default key-based auth when binding to `0.0.0.0`. Put a TLS-terminating reverse proxy (Caddy, nginx) in front for anything internet-facing. ### Server modes -The set of routes that is actually mounted depends on how you start the server: +The routes the server mounts depend on how you start the server: === "Default (CDM + REST)" - Both the pywidevine/pyplayready CDM endpoints and the full REST API are mounted. - This is the mode used when you run `unshackle serve` with no mode flags. + The server mounts both the pywidevine/pyplayready CDM endpoints and the full REST API. + This is the mode you get when you operate `unshackle serve` with no mode flags. === "`--api-only`" - Only the REST API is mounted; the CDM HTTP endpoints are omitted. Authentication - uses the single configured API key (or none, with `--no-key`). + The server mounts only the REST API. It does not mount the CDM HTTP endpoints. + Authentication uses the single configured API key (or none, with `--no-key`). === "`--remote-only`" - Only the remote-session subset of routes is exposed (health, services, search, + The server mounts only the remote-session subset of routes (health, services, search, and everything under `/api/session/*`). This mode implies `--api-only`, and the interactive Swagger UI is **not** mounted. @@ -115,12 +115,12 @@ health check is `GET http://127.0.0.1:8786/api/health`. !!! tip "Interactive API docs" Unless you started the server with `--remote-only`, an interactive Swagger UI is - available at [`/api/docs/`](http://127.0.0.1:8786/api/docs/). It is generated from - the same route table the server actually serves, so it stays in sync with the - running build. Note that in `--api-only` mode the docs page is itself behind the - API key; only `/api/health` is exempt from authentication. + available at [`/api/docs/`](http://127.0.0.1:8786/api/docs/). The server generates + it from the same route table it serves, so it always matches the version you + operate. Note that in `--api-only` mode the docs page is itself behind the + API key. Only `/api/health` is exempt from authentication. -A quick smoke test that needs no key: +A quick smoke test that needs no API key: ```shell title="Confirm the server is up" curl http://127.0.0.1:8786/api/health @@ -141,7 +141,7 @@ curl http://127.0.0.1:8786/api/health ## Endpoint map Every REST path uses JSON request and response bodies and lives under `/api`. The -table below is a map of the surface; the [Endpoints](endpoints.md) page documents each +table below is a map of the surface. The [Endpoints](endpoints.md) page gives each request and response in full. ### Discovery @@ -216,15 +216,15 @@ curl -X POST http://127.0.0.1:8786/api/search \ -d '{"service": "EXAMPLE", "query": "example"}' ``` -A missing or unknown key returns `401`. The full model, including per-key service -allowlists, is covered in [Authentication](authentication.md). +A missing or unknown API key returns `401`. [Authentication](authentication.md) gives +the full model, including per-key service allowlists. ## Where to go next
-- **[Quick Start](quickstart.md)**: start the server, set a key, and run your first - end-to-end download over the API. +- **[Quick Start](quickstart.md)**: start the server, set an API key, and operate your + first end-to-end download over the API. - **[Authentication](authentication.md)**: the `X-Secret-Key` header, `--no-key`, per-key service allowlists, and CORS. - **[Endpoints](endpoints.md)**: every request body, query parameter, and response diff --git a/docs/dev/rest-api/job-lifecycle.md b/docs/dev/rest-api/job-lifecycle.md index 8aaa5e15..cff548c9 100644 --- a/docs/dev/rest-api/job-lifecycle.md +++ b/docs/dev/rest-api/job-lifecycle.md @@ -1,23 +1,23 @@ # Download Job Lifecycle When you submit a download over the REST API, the server does not block the request -until the file is written. Instead it creates a **job**, hands it back to you with an +until it writes the file. Instead it creates a **job**, hands it back to you with an ID, and works through it in the background. Everything you do afterwards (watching progress, cancelling, retrying, reading the result) is done by referring to that job ID. -This page explains the states a job moves through, how live progress is reported, how -to cancel or retry, and, in clearly marked developer sections, how the worker and -manager behind the queue actually run the download. +This page explains the states a job moves through, how the server reports live +progress, and how to cancel or retry. Clearly marked developer sections tell you how +the job worker and the manager behind the queue operate the download. !!! info "Who this page is for" The first half is for anyone driving the API: the states, the fields you can read, and the calls that move a job around. The **Under the hood** sections near the end - describe the internal worker/manager model and are aimed at contributors and people - debugging the server. You do not need them to use the API. + give the internal job worker and manager model, for contributors and for people + who debug the server. You do not need them to use the API. For the full request and response shape of each endpoint, see -[Endpoints](endpoints.md). For how errors are reported, see +[Endpoints](endpoints.md). For how the server reports errors, see [Error Responses](errors.md). ## The five states @@ -26,15 +26,15 @@ A job is always in exactly one of five states, exposed as the `status` field: | Status | Meaning | |---|---| -| `queued` | Created and waiting for a free worker. No download has started yet. | -| `downloading` | A worker has picked the job up and the download is in progress. | +| `queued` | Created and waiting for a free job worker. No download has started yet. | +| `downloading` | A job worker has picked the job up and the download is in progress. | | `completed` | The download finished successfully; `output_files` lists the results. | | `failed` | The download stopped with an error; `error_message` and `error_code` explain why. | | `cancelled` | The job was cancelled before it could finish, from either the queued or downloading state. | The last three (`completed`, `failed`, and `cancelled`) are **terminal**: once a -job reaches one of them it never changes state again. Terminal jobs can be removed, -retried, or read from history, but they will not resume. +job reaches one of them it never changes state again. You can remove, retry, or read +terminal jobs from history, but they will not resume. ## How a job moves between states @@ -59,7 +59,7 @@ The transitions in prose: - **`queued` → `downloading`** happens automatically. As soon as one of the server's workers is free, it takes the next job off the queue and starts it. You do not - trigger this; you only observe it. + trigger this. You only observe it. - **`downloading` → `completed`** is the happy path: the download engine wrote all the requested files and reported success. - **`downloading` → `failed`** is any error during the download: a service failure, a @@ -72,12 +72,12 @@ The transitions in prose: !!! note "Priority does not change status" Moving a job to the front of the queue with the priority endpoint reorders *when* - it will start, but it stays `queued` until a worker actually picks it up. It is not + it will start, but it stays `queued` until a job worker picks it up. It is not a state change. ## Creating a job -A download is queued with `POST /api/download`. The request needs at least a `service` +You queue a download with `POST /api/download`. The request needs at least a `service` and a `title_id`, plus any download options you want to set. On success the server replies with **`202 Accepted`** and the new job's identity: @@ -101,14 +101,15 @@ curl -X POST http://127.0.0.1:8786/api/download \ } ``` -The job starts in `queued`. Workers auto-start on the first job submitted, so there is -nothing else to switch on. Keep the `job_id`; every later call is addressed to it. +The job starts in `queued`. The job workers auto-start on the first job you submit, so +you switch nothing else on. Keep the `job_id`. Every later call uses it. !!! tip "The full parameter set" `POST /api/download` accepts the same download options as the `dl` command (quality, codecs, language selection, `wanted` episode ranges, and many more), with - server-side defaults filling in anything you omit. The complete list, their - defaults, and validation rules are documented on the [Endpoints](endpoints.md) page. + server-side defaults filling in anything you do not write. The + [Endpoints](endpoints.md) page gives the complete list, their defaults, and the + validation rules. ## Reading a job's status and progress @@ -136,7 +137,7 @@ There are two ways to look at jobs. -H "X-Secret-Key: your-api-secret" ``` - The list can be filtered by `status` and `service`, and ordered with `sort_by` + You can filter the list by `status` and `service`, and order it with `sort_by` (`created_time`, `started_time`, `completed_time`, `progress`, `status`, or `service`) and `sort_order` (`asc` / `desc`). See [Endpoints](endpoints.md) for the query-parameter reference. @@ -179,7 +180,7 @@ The full detail adds `started_time`, `completed_time`, `output_files`, the redac !!! tip "Or subscribe instead of polling" `GET /api/download/jobs/{job_id}/events` pushes the same job object as Server-Sent - Events, and closes the stream when the job ends. See + Events, and closes the event stream when the job ends. See [Endpoints](endpoints.md#get-apidownloadjobsjob_idevents). !!! note "What the percentage means" @@ -201,15 +202,15 @@ The full detail adds `started_time`, `completed_time`, `output_files`, the redac ### Completed A `completed` job has its `output_files` populated with the paths of the muxed -results, and `progress` is `100`. This is the state you are waiting for. +results, and `progress` is `100`. This is the state you want. ### Failed A `failed` job carries the error. `error_message` is the short human message, `error_code` is one of the API's structured error codes (for example `NETWORK_ERROR`, `DRM_ERROR`, `AUTH_FAILED`, or `DOWNLOAD_ERROR`), and `error_details` gives more -context. When the server was started with `--debug-api`, `error_traceback` and -`worker_stderr` are also filled in for diagnosis. +context. When you start the server with `--debug-api`, `error_traceback` and +`worker_stderr` are also filled in, to help you find the cause. !!! tip "Secrets are scrubbed from error text" A service can echo a credential or a proxy URL into its output. The server masks @@ -221,8 +222,8 @@ The error-code catalogue and how each maps to an HTTP status is on the ### Cancelled -A `cancelled` job was stopped on request before finishing. It may have partial output, -but it is not considered complete. See below. +The server stopped a `cancelled` job on request, before it finished. It may have +partial output, but it is not considered complete. See below. ## Cancelling a job @@ -241,7 +242,7 @@ both stops in-flight work and tidies up afterwards. !!! warning "Do not parse the body of a 204" Removing a terminal job returns `204 No Content` with an empty body. A client must - not try to decode JSON from it; check the status code first. + not try to decode JSON from it. Examine the status code first. ```shell title="Cancel a running job" curl -X DELETE http://127.0.0.1:8786/api/download/jobs/b1e6c7de-6d0a-4f2f-9d1e-3a2b4c5d6e7f \ @@ -254,14 +255,15 @@ which returns `{"removed": }`. ## Retrying a job `POST /api/download/jobs/{job_id}/retry` re-runs a finished job by creating a **new** -job with the same service, title, and parameters. The original job is untouched and -stays in its terminal state; the response gives you the new job's ID with `202`. +job with the same service, title, and parameters. The retry does not change the +original job. It stays in its terminal state, and the response gives you the new job's +ID with `202`. -- Only terminal jobs can be retried. Retrying a `queued` or `downloading` job returns +- You can retry only terminal jobs. A retry of a `queued` or `downloading` job returns `409` with error code `CONFLICT`. - A missing job returns `404` (`JOB_NOT_FOUND`). -- The service allowlist and any download gates are re-checked for the new job, exactly - as they were for the original submission, so a retry can be rejected with +- The server examines the service allowlist and any download gates again for the new + job, exactly as for the original submission. Thus the server can reject a retry with `INVALID_SERVICE` or `FORBIDDEN` if permissions changed. ```json title="202 Accepted, a fresh job" @@ -275,38 +277,39 @@ stays in its terminal state; the response gives you the new job's ID with `202`. ## Prioritising a job `POST /api/download/jobs/{job_id}/priority` moves a **queued** job to the front of the -line, so it will be the next one a free worker picks up. +line, so it will be the next one a free job worker picks up. - The job must be `queued`. Prioritising a job in any other state returns `409` - (`CONFLICT`); a missing job returns `404`. + (`CONFLICT`). A missing job returns `404`. - On success the response is `{"job_id": "", "position": "front"}`. -This only reorders the queue; the job's status stays `queued` until a worker starts -it. +This only reorders the queue. The job's status stays `queued` until a job worker +starts it. ## History and retention -Every job that reaches a terminal state is also written once to a persistent history -log, so it survives even after the job is removed from the live list. +The server also writes every job that reaches a terminal state once to a persistent +history log. The job survives there even after the server removes it from the live +list. -- Read it with `GET /api/history` (newest first; supports `limit` and `service` +- Read it with `GET /api/history` (newest first, with `limit` and `service` filters). - Delete a single entry with `DELETE /api/history/{job_id}` (returns `204`). The live in-memory jobs are separate from this on-disk history. A background cleanup task periodically drops old terminal jobs from the live list based on a retention -period; the history file keeps them for later reference. See [Endpoints](endpoints.md) +period. The history file keeps them for later reference. See [Endpoints](endpoints.md) for the history response shape. !!! note "Why the history file lives under the cache directory" - `api_history.jsonl` is placed under the cache directory on purpose, so it stays out - of the config/data tree. It is disposable runtime state, not user configuration or + unshackle puts `api_history.jsonl` under the cache directory on purpose, so it + stays out of the config/data tree. It is disposable runtime state, not user configuration or curated data, which is exactly why clearing the cache is a safe operation against it. --- -## Under the hood: the worker/manager model +## Under the hood: the job worker and manager model !!! info "Developer section" Everything below describes server internals. It is useful for contributors and for @@ -316,18 +319,18 @@ for the history response shape. A single process-wide **download queue manager** owns all jobs. It holds them in memory keyed by ID, backs the wait list with a FIFO queue, and keeps a separate -priority deque for jobs that have been bumped to the front. On the first job, it starts -a fixed pool of worker tasks plus one cleanup task; starting is idempotent, so later -jobs reuse the same pool. +priority deque for the jobs it bumped to the front. On the first job, it starts +a fixed pool of job worker tasks plus one cleanup task. The start is idempotent, so +later jobs reuse the same pool. -- **Concurrency** is capped by `max_concurrent_downloads` (default **2**). That many - jobs can be in `downloading` at once; the rest wait in `queued`. -- **Retention** is governed by `job_retention_hours` (default **24**). An hourly +- **Concurrency**: `max_concurrent_downloads` (default **2**) caps it. That many + jobs can be in `downloading` at once. The rest wait in `queued`. +- **Retention**: `job_retention_hours` (default **24**) controls it. An hourly cleanup task removes terminal jobs whose completion time (or, if never started, creation time) is older than the retention window. !!! note "Two concurrency knobs that are easy to conflate" - `max_concurrent_downloads` caps how many separate download **jobs** run at once. + `max_concurrent_downloads` caps how many separate download **jobs** operate at once. The separate `serve.downloads` setting controls how many **tracks** download in parallel *within* a single job. They are independent, and raising one does not affect the other. @@ -336,64 +339,64 @@ jobs reuse the same pool. The manager reads these as top-level config attributes (`config.max_concurrent_downloads` and `config.download_job_retention_hours`). Those keys are not part of the standard config schema, so in practice the built-in - defaults of **2** and **24** apply. The live values are surfaced by - [`GET /api/config`](endpoints.md). The related `serve.history_limit` (default + defaults of **2** and **24** apply. [`GET /api/config`](endpoints.md) gives the + live values. The related `serve.history_limit` (default **100**) instead controls how many entries the on-disk history file keeps. -Each worker loop drains the priority deque first, then the FIFO queue. A job that was -promoted has its stale FIFO copy skipped so it does not run twice. When a worker takes -a job it flips it to `downloading`, stamps `started_time`, runs it, and on finish -stamps `completed_time` and records history. +Each job worker loop drains the priority deque first, then the FIFO queue. The loop +skips the stale FIFO copy of a promoted job, so it does not do the job twice. When +a job worker takes a job it flips it to `downloading`, stamps `started_time`, does the +download, and on finish stamps `completed_time` and records history. ### Each job runs in its own subprocess -The manager does **not** run the download in the server's event loop or even in the +The manager does **not** operate the download in the server's event loop or even in the server process. For each job it spawns a separate Python subprocess: ```text python -m unshackle.core.api.download_worker ``` -Three temporary files bridge the parent and the worker: +Three temporary files bridge the parent and the job worker: -- **`payload.json`**: written by the manager before launch; contains the job ID, +- **`payload.json`**: the manager writes it before launch. It contains the job ID, service, title ID, and merged parameters. -- **`progress.json`**: written continuously by the worker as the download proceeds. -- **`result.json`**: written by the worker at the end. +- **`progress.json`**: the job worker writes it continuously as the download proceeds. +- **`result.json`**: the job worker writes it at the end. Running each download out-of-process isolates the server from crashes, memory growth, and blocking work inside the download engine, and makes hard cancellation (killing the process) reliable. -The worker's exit codes are `0` for success, `1` for a download error, and `2` for bad +The job worker's exit codes are `0` for success, `1` for a download error, and `2` for bad arguments. Its `result.json` is either `{"status": "success", "output_files": [...]}` or `{"status": "error", "message", "error_details", "error_code", "traceback"}`. If the process exits non-zero, or the result is missing or not a success, the parent marks the -job `failed` and adopts the worker's `error_code` (or derives one by categorising the +job `failed` and adopts the job worker's `error_code` (or derives one by categorising the exception). ### How progress reaches the API -While the worker runs, the manager polls `progress.json` every **0.5 s** and copies the +While the job worker operates, the manager polls `progress.json` every **0.5 s** and copies the fields it finds onto the live `DownloadJob`. That is what the `GET` endpoints return. -Inside the worker, progress is aggregated across all selected tracks. Each track's -completion fraction is weighted by its bitrate, so video and audio dominate the bar and -subtitles barely move it (subtitles get a small fixed weight). Track fractions are kept -**monotonic**: the percentage only ever climbs. The download portion is scaled to fill -`0` to `90`; the remaining `90` to `100` is driven by the download engine as it repackages -and muxes. A track counts as done when it reaches a terminal state such as +The job worker aggregates progress across all selected tracks. It weights each track's +completion fraction by its bitrate, so video and audio dominate the bar and subtitles +barely move it (subtitles get a small fixed weight). Track fractions are kept +**monotonic**: the percentage only ever climbs. The job worker scales the download +portion to fill `0` to `90`. The download engine drives the remaining `90` to `100` as +it repackages and muxes. A track counts as done when it reaches a terminal state such as `Downloaded`, `Decrypted`, or a skipped subtitle. -### How cancellation actually works +### How cancellation works Each job carries a cancellation event. When you `DELETE` a live job: -- If it is **`queued`**, it flips straight to `cancelled`, its history is recorded - immediately (it never reaches a worker), and the worker loop skips it if it later - pops it off the queue. +- If it is **`queued`**, it flips straight to `cancelled`, the manager records its + history immediately (it never reaches a job worker), and the job worker loop skips it + if it later pops it off the queue. - If it is **`downloading`**, the manager sets the cancellation event and flips the job - to `cancelled`. The polling loop notices the event, `terminate()`s the worker + to `cancelled`. The polling loop notices the event, `terminate()`s the job worker subprocess (escalating to `kill()` if it does not exit within 5 seconds), and raises a cancellation so the task unwinds. The subprocess model is what makes this a clean, immediate stop rather than a cooperative one. @@ -403,11 +406,11 @@ Each job carries a cancellation event. When you `DELETE` a live job: Job parameters may carry secrets: a raw `user:pass` credential, or a proxy URL with embedded userinfo. Whenever the manager serializes a job (for an API response or the history file), it masks the sensitive keys (`credential`, `credentials`, `password`, -`token`, `api_key`) and strips userinfo from proxy URLs. Free-text fields that a -service might have leaked a secret into (`error_message`, `error_details`, -`error_traceback`, `worker_stderr`, and worker stdout/stderr in the logs) are scrubbed -of the job's known secret values, longest-match first, before they are ever returned or -logged. +`token`, `api_key`) and strips userinfo from proxy URLs. The manager also scrubs +free-text fields that a service might have leaked a secret into (`error_message`, +`error_details`, `error_traceback`, `worker_stderr`, and job worker stdout/stderr in the +logs). It removes the job's known secret values, longest-match first, before it returns +or logs them. ## See also diff --git a/docs/dev/rest-api/quickstart.md b/docs/dev/rest-api/quickstart.md index d99c8483..567e5dc3 100644 --- a/docs/dev/rest-api/quickstart.md +++ b/docs/dev/rest-api/quickstart.md @@ -1,11 +1,11 @@ # REST API Quick Start -Unshackle ships with a built-in REST API that exposes the same download engine you drive from the command line. It is served by the `unshackle serve` command and lets you queue downloads, poll their progress, browse services, and manage history over HTTP instead of the terminal. +Unshackle ships with a built-in REST API that exposes the same download engine you drive from the command line. The `unshackle serve` command serves it and lets you queue downloads, poll their progress, browse services, and manage history over HTTP instead of the terminal. -The full loop runs end to end: start the server, authenticate, submit a download job, and poll it to completion, using real `curl` commands built directly from the API routes. +The full loop runs end to end: start the server, authenticate, submit a download job, and poll it to completion. The `curl` commands here come directly from the API routes. !!! note "Who this page is for" - This is a **developer** guide. It assumes you are comfortable with HTTP, JSON, and a tool like `curl`. If you just want to download something, the [command line](../../index.md) is simpler. The REST API is meant for building tools, UIs, and automation on top of unshackle. + This is a **developer** guide. It assumes you are comfortable with HTTP, JSON, and a tool like `curl`. If you only want to download something, the [command line](../../index.md) is simpler. The REST API is meant for building tools, UIs, and automation on top of unshackle. --- @@ -30,7 +30,7 @@ You need a working unshackle install with services and CDM devices already confi ## Step 1: Configure an API secret -Every request (except the health check) is authenticated with a secret key sent in the `X-Secret-Key` header. The server reads this key from the `serve` section of your `unshackle.yaml` config. +The server authenticates every request (except the health check) with an API key that you send in the `X-Secret-Key` header. The server reads this API key from the `serve` section of your `unshackle.yaml` config. At minimum you must set `api_secret`: @@ -39,7 +39,7 @@ serve: api_secret: "choose-a-long-random-secret" ``` -If you want the API to be able to license and download DRM-protected content, you also need to grant CDM devices to the key. Configure that under `serve.users`, keyed by the secret: +If you want the API to license and download DRM-protected titles, you must also grant CDM devices to the API key. Configure that under `serve.users`, keyed by the secret: ```yaml title="unshackle.yaml" serve: @@ -55,7 +55,7 @@ serve: If `api_secret` is not set and you do **not** pass `--no-key`, `unshackle serve` refuses to start and raises an error. Treat the secret like a password. Anyone who has it can queue downloads on your machine. !!! tip "Disabling auth for local testing" - Passing `--no-key` to `serve` turns authentication off entirely: every request is allowed and the `X-Secret-Key` header is ignored. This is convenient for a throwaway local test, but never expose a `--no-key` server to a network you do not fully trust. + `--no-key` turns authentication off entirely: the server allows every request and ignores the `X-Secret-Key` header. This is convenient for a throwaway local test, but never put a `--no-key` server on a network you do not fully trust. --- @@ -67,7 +67,7 @@ Start the REST API with the `serve` command: unshackle serve --api-only ``` -By default the server binds to `127.0.0.1:8786`. The `--api-only` flag serves just the REST API and omits the Widevine/PlayReady CDM HTTP endpoints; drop it if you also want to expose those. +By default the server binds to `127.0.0.1:8786`. The `--api-only` flag serves only the REST API and omits the Widevine/PlayReady CDM HTTP endpoints. Drop it when you also want those endpoints. Common options: @@ -90,9 +90,9 @@ Swagger UI available at http://127.0.0.1:8786/api/docs/ ``` !!! note "Swagger UI is also behind auth" - In `--api-only` mode (without `--no-key`), the Swagger UI at `/api/docs/` requires the `X-Secret-Key` header just like every other route. Only `/api/health` is exempt from authentication. + In `--api-only` mode (without `--no-key`), the Swagger UI at `/api/docs/` requires the `X-Secret-Key` header, the same as every other route. Only `/api/health` is exempt from authentication. -For the rest of this page we will assume the server is reachable at `http://127.0.0.1:8786` and store the secret in a shell variable: +For the rest of this page we will assume the server is reachable at `http://127.0.0.1:8786` and store the secret in an environment variable: ```shell export SECRET="choose-a-long-random-secret" @@ -101,9 +101,9 @@ export BASE="http://127.0.0.1:8786" --- -## Step 3: Confirm the server is alive +## Step 3: Make sure that the server is alive -The health endpoint is the one route that never requires a key. Use it to confirm the server is running and to read its version: +The health endpoint is the one route that never requires an API key. Use it to make sure that the server operates, and to read its version: ```shell curl "$BASE/api/health" @@ -127,7 +127,7 @@ The `update_check` block is best effort. If unshackle cannot reach the update so ## Step 4: Authenticate a real request -Every other endpoint needs the `X-Secret-Key` header. A quick way to prove your key works is to list the services the server can see: +Every other endpoint needs the `X-Secret-Key` header. A quick way to prove your API key works is to show the services the server can see: ```shell curl "$BASE/api/services" \ @@ -154,14 +154,14 @@ curl "$BASE/api/services" \ } ``` -If the key is missing or wrong, you get a `401` with a plain body: +If the API key is missing or wrong, you get a `401` with a plain body: ```json title="401 Unauthorized" { "status": 401, "message": "Secret Key is Invalid." } ``` !!! warning "Two error shapes exist" - The authentication middleware returns the short `{"status": 401, "message": "..."}` shape shown above. Every **other** error the API produces uses the richer structured shape (`error_code`, `timestamp`, `details`, ...) described in [Error handling](#error-handling). A robust client should handle both. + The authentication middleware returns the short `{"status": 401, "message": "..."}` shape shown above. Every **other** error the API produces uses the richer structured shape (`error_code`, `timestamp`, `details`, ...) described in [Error handling](#error-handling). Parse both shapes in your client. The `tag` value for the service you want is the identifier you pass as `service` in the calls below. @@ -241,7 +241,7 @@ The request body accepts the full download parameter set. A few of the most usef Validation is strict: invalid codecs, subtitle formats, dynamic-range names, non-positive counts, or mutually exclusive flags (for example both `video_only` and `audio_only`) come back as `400` with error code `INVALID_PARAMETERS`. See the full reference on the [download reference page](endpoints.md) for every accepted field. !!! note "Gated fields" - Passing a per-request CDM (`cdm`) or inline credentials (`credential` / `credentials`) is refused with `403 FORBIDDEN` unless the server operator has explicitly opted in via `serve.cdm_overrides` and `serve.allow_job_credentials`. Both are off by default. + The server refuses a per-request CDM (`cdm`) or inline credentials (`credential` / `credentials`) with `403 FORBIDDEN`, unless the server operator has explicitly opted in with `serve.cdm_overrides` and `serve.allow_job_credentials`. Both are off by default. --- @@ -260,7 +260,7 @@ curl "$BASE/api/download/jobs/$JOB" \ -H "X-Secret-Key: $SECRET" ``` -While the download is running you will see `status: "downloading"` and a `progress` value climbing from `0` toward `100`: +While the download runs you will see `status: "downloading"` and a `progress` value climbing from `0` toward `100`: ```json title="200 OK (in progress)" { @@ -313,14 +313,14 @@ done A few seconds between polls is plenty. The server updates a job's progress roughly twice a second internally, and each poll reads the current snapshot. !!! tip "Or let the server push the updates" - `GET /api/download/jobs/{job_id}/events` streams the same job record as Server-Sent Events, and closes the stream when the job ends. It removes the polling loop above. See [Endpoints](endpoints.md#get-apidownloadjobsjob_idevents). + `GET /api/download/jobs/{job_id}/events` streams the same job record as Server-Sent Events, and closes the event stream when the job ends. It removes the polling loop above. See [Endpoints](endpoints.md#get-apidownloadjobsjob_idevents). ```bash curl -N "$BASE/api/download/jobs/$JOB/events" \ -H "X-Secret-Key: $SECRET" ``` -If a job ends in `failed`, the full record includes `error_message`, `error_code`, and (when the server was started with `--debug-api`) `error_traceback` and `worker_stderr` to help you diagnose it. Sensitive values in these fields are scrubbed before they are returned. +If a job ends in `failed`, the full record includes `error_message`, `error_code`, and (when you start the server with `--debug-api`) `error_traceback` and `worker_stderr` to help you find the cause. The server scrubs sensitive values in these fields before it returns them. --- @@ -333,11 +333,11 @@ curl -X DELETE "$BASE/api/download/jobs/$JOB" \ -H "X-Secret-Key: $SECRET" ``` -- If the job is **terminal** (completed/failed/cancelled) it is removed from the manager and you get **`204 No Content`** with an empty body. -- If the job is still **queued or downloading** it is cancelled instead, and you get `200` with `{"status": "success", "message": "Job cancelled"}`. +- If the job is **terminal** (completed/failed/cancelled), the manager removes it. You get **`204 No Content`** with an empty body. +- If the job is still **queued or downloading**, the manager cancels it instead. You get `200` with `{"status": "success", "message": "Job cancelled"}`. !!! warning "Do not parse JSON on 204" - A `204` response has no body. This applies to terminal-job removal here and to deleting history entries; check the status code before attempting to decode JSON. + A `204` response has no body. This applies to terminal-job removal here and to deleting history entries. Examine the status code before you decode JSON. To clear every finished job at once instead of one at a time: @@ -365,7 +365,7 @@ A handful of endpoints round out job control. All require the `X-Secret-Key` hea | `POST` | `/api/download/jobs/{job_id}/priority` | Move a queued job to the front of the queue | | `POST` | `/api/download/jobs/clear-finished` | Remove all terminal jobs | -Listing supports query parameters for filtering by `status` or `service`, sorting via `sort_by` (`created_time`, `started_time`, `completed_time`, `progress`, `status`, `service`) and `sort_order` (`asc` / `desc`), and requesting full per-job detail with `full=true`: +Listing supports query parameters for filtering by `status` or `service`, sorting with `sort_by` (`created_time`, `started_time`, `completed_time`, `progress`, `status`, `service`) and `sort_order` (`asc` / `desc`), and requesting full per-job detail with `full=true`: ```shell curl "$BASE/api/download/jobs?status=downloading&sort_by=progress&sort_order=desc" \ @@ -390,7 +390,7 @@ curl "$BASE/api/download/jobs?status=downloading&sort_by=progress&sort_order=des } ``` -Retrying reuses the original job's service, title, and parameters to enqueue a brand new job (returning `202` with a fresh `job_id`); it only works on terminal jobs and returns `409 CONFLICT` otherwise. Prioritizing only works on jobs still in the `queued` state. +A retry reuses the original job's service, title, and parameters to enqueue a brand new job, and returns `202` with a fresh `job_id`. A retry works only on terminal jobs, and returns `409 CONFLICT` otherwise. A priority change works only on jobs still in the `queued` state. --- @@ -398,10 +398,10 @@ Retrying reuses the original job's service, title, and parameters to enqueue a b The download loop above is the core of the API, but the same server exposes more: -- **Discovery**: `POST /api/search` finds titles by query, and `POST /api/list-titles` / `POST /api/list-tracks` inspect what a title offers before you commit to downloading it. See [list titles and tracks](endpoints.md). +- **Discovery**: `POST /api/search` finds titles by query, and `POST /api/list-titles` / `POST /api/list-tracks` inspect what a title offers before you commit to downloading it. See [titles and tracks](endpoints.md). - **History**: `GET /api/history` reads a persisted log of jobs that reached a terminal state (newest first), and `DELETE /api/history/{job_id}` removes a single entry. See [history](endpoints.md). -- **Introspection**: `GET /api/profiles` lists configured credential profiles per service, `GET /api/config` returns a redacted view of the effective server config, and `GET /api/env/check` reports which external binaries are installed. -- **Maintenance**: `POST /api/maintenance/clear-cache`, `.../clear-temp`, and `.../refresh-services` perform housekeeping (the cache/temp clears are blocked with `409` while a download is active). +- **Introspection**: `GET /api/profiles` lists configured credential profiles per service, `GET /api/config` returns a redacted view of the effective server config, and `GET /api/env/check` reports which external binaries the machine has. +- **Maintenance**: `POST /api/maintenance/clear-cache`, `.../clear-temp`, and `.../refresh-services` do housekeeping (the server blocks the cache and temp clears with `409` while a download is active). - **Remote sessions**: the `/api/session/*` endpoints drive the thin-client "remote download" flow, where the server authenticates and licenses on behalf of a local client. See [remote sessions](remote-sessions.md). The interactive **Swagger UI at `/api/docs/`** documents every route and lets you try them in the browser (send your `X-Secret-Key` there too). @@ -422,7 +422,7 @@ Apart from the auth middleware's short form, every error the API returns uses a } ``` -`details` appears only when there is something to add, `retryable: true` appears only for transient failures, and a `debug_info` block (exception type and traceback) is included only when the server runs with `--debug-api`. +`details` appears only when the server has more to add. `retryable: true` appears only for transient failures. The server includes a `debug_info` block (exception type and traceback) only when it runs with `--debug-api`. The `error_code` maps to an HTTP status. The ones you will meet most often: diff --git a/docs/dev/rest-api/remote-sessions.md b/docs/dev/rest-api/remote-sessions.md index e68311dd..f1383243 100644 --- a/docs/dev/rest-api/remote-sessions.md +++ b/docs/dev/rest-api/remote-sessions.md @@ -1,6 +1,6 @@ # Remote Service Sessions -A **remote service session** lets one machine do the hard, service-specific work +A **remote session** lets one machine do the hard, service-specific work (authenticating to a streaming service, listing titles, resolving tracks and manifests, and, optionally, DRM licensing) while another machine runs the actual download, decryption, and muxing locally. @@ -9,14 +9,14 @@ This is the mechanism behind unshackle's client/server "remote download" workflo A thin local client (`RemoteService`) talks to an unshackle `serve` instance over HTTP. The server keeps an authenticated service instance alive between requests so the client can authenticate once and then make many follow-up calls against the -same session. +same remote session. !!! note "Why you'd use this" - - The server has service accounts, cookies, or a CDM you don't want to copy to + - The server has service accounts, cookies, or a CDM you do not want to copy to every machine. - - You want to run downloads on a fast/local machine but keep credentials and - licensing centralized. - - A service is region-locked to where the server lives, and the client isn't. + - You want to do downloads on a fast/local machine but keep credentials and + licensing centralised. + - A service is region-locked to where the server lives, and the client is not. --- @@ -43,7 +43,7 @@ selection (downloading segments, decrypting, and muxing) runs locally. The `RemoteService` adapter lives inside the ordinary `dl` command and replaces the old standalone `remote_dl` command. If you have prior scripts or docs that invoke `remote_dl`, use `dl` with a configured remote server instead. The - functionality now rides on `dl` via `RemoteService`. + functionality now rides on `dl` through `RemoteService`. For the auth-facing endpoints, header format, and error shapes referenced below, see [Authentication](authentication.md) and the endpoint reference in @@ -79,18 +79,18 @@ remote_services: !!! warning "Port default" The `unshackle serve` default port is **`8786`** (not `8080`). Use the port - your server actually binds. Some older example snippets show `8080`, which is + your server binds. Some older example snippets show `8080`, which is not the default. ### Selecting a server -If exactly one server is configured, unshackle uses it implicitly: +If you configure exactly one server, unshackle uses it implicitly: ```bash unshackle dl EXAMPLE 0ABCDEF ``` -If **more than one** server is configured, you must pick one with `--server`: +If you configure **more than one** server, you must pick one with `--server`: ```bash unshackle dl --server my_server EXAMPLE 0ABCDEF @@ -104,17 +104,17 @@ you to add the block shown above. ## What happens during a download From your point of view the command looks like an ordinary download. Under the -hood the client walks a session through its lifecycle. +hood the client walks a remote session through its lifecycle. === "1. Create + authenticate" The client calls `POST /api/session/create`, forwarding whatever it can so the - server doesn't have to prompt: + server does not have to prompt: - Local **credentials** for the service/profile (`{username, password, extra?}`) - Local **cookies**, compressed and base64-encoded - - A resolved **proxy**, or, if you didn't set one, your detected - **client region** so the server can auto-proxy to match + - A resolved **proxy**, or, if you did not set one, your detected + **client region** so the server can auto-proxy to match it - Track-selection hints (`range_`, `vcodec`, `quality`, `best_available`) so the server fetches the right manifests - Your local per-service **cache files** (e.g. refreshed tokens) @@ -125,7 +125,7 @@ hood the client walks a session through its lifecycle. === "2. Interactive prompts (if needed)" Some services need a one-time code, PIN, or device confirmation. When the - server's auth flow asks for input, the session enters `pending_input`. The + server's auth flow asks for input, the remote session enters `pending_input`. The client polls, displays the prompt to you locally, collects your answer, and sends it back. The server's auth thread resumes with your response. @@ -136,29 +136,29 @@ hood the client walks a session through its lifecycle. === "3. Titles + tracks" Once authenticated, the client fetches the title list, then the tracks for the - chosen title. The tracks come back with playback URLs, and DASH/ISM manifests - are shipped as compressed XML so the client can re-parse them locally for + chosen title. The tracks come back with playback URLs, and the server ships + DASH/ISM manifests as compressed XML so the client can re-parse them locally for downloading. Any `session_headers` / `session_cookies` the server used are merged into the client's local HTTP session. === "4. Licensing" - For DRM content the client either **proxies** its CDM challenge through the - server, or, if `server_cdm: true`, asks the server to run the full CDM flow + For a DRM-protected title the client either **proxies** its CDM challenge through + the server, or, if `server_cdm: true`, asks the server to do the full CDM flow and return the content keys directly. See [Server-side vs. proxied CDM](#server-side-vs-proxied-cdm) below. === "5. Download + close" The client downloads, decrypts, and muxes locally. On completion it deletes - the session. If the server has updated cache files (e.g. a refreshed token), - they're returned on delete and saved locally so the **next** session can skip - interactive auth. + the remote session. If the server has updated cache files (for example a refreshed + token), the delete returns them and the client saves them locally, so the **next** + remote session can skip interactive auth. !!! tip "Renaming remote titles locally" - You can rename titles for a remote service you don't have installed locally by + You can rename titles for a remote service you do not have installed locally by adding a `title_map` under that service in your client `remote_services..services` - config. The server sends raw titles; your local map wins. + config. The server sends raw titles. Your local map wins. This is deliberate: the server does **no** `title_map` remapping of its own and sends titles exactly as the service returns them. All remapping happens on the @@ -178,26 +178,26 @@ There are two ways DRM keys get resolved, chosen by the client's `server_cdm` fl to `POST /api/session/{id}/license`, the server forwards it to the service, and the raw license bytes come back for your local CDM to parse. - - Keeps your CDM local; the server just relays the license request. + - Keeps your CDM local. The server only relays the license request. - Used when `server_cdm` is `false` (the default). === "Server-side CDM (server_cdm: true)" The **server's CDM** does everything. The client sends track IDs (or a PSSH), the server checks its key vaults, loads the device configured for your API key, - runs the CDM flow, and returns `KID:KEY` pairs directly. No CDM is needed on the - client. + runs the CDM flow, and returns `KID:KEY` pairs directly. The client needs no + CDM. - Enable with `server_cdm: true` in the server entry. - - The server tells the client which DRM type it actually used + - The server tells the client which DRM type it used (`widevine` or `playready`). --- -## Session lifecycle and expiry +## Remote session lifecycle and expiry -Sessions live in an **in-memory store on the server** and expire on a timer. You -generally never touch this directly, but it explains behavior you might observe. +Remote sessions live in an **in-memory store on the server** and expire on a timer. You +generally never touch this directly, but it explains behaviour you might observe. | Behavior | Value | Notes | |---|---|---| @@ -207,9 +207,10 @@ generally never touch this directly, but it explains behavior you might observe. | Cleanup sweep | every 60s | Expired sessions are removed and their input prompts cancelled | !!! note "Auth is never rushed" - A session that is still `authenticating` or waiting on `pending_input` is **not** - subject to the short 300s TTL; it gets the full 600s auth window. This gives you - time to enter an interactive code without the session vanishing mid-prompt. + A remote session that is still `authenticating` or waiting on `pending_input` is + **not** subject to the short 300s TTL. It gets the full 600s auth window. This gives + you time to enter an interactive code without the remote session vanishing + mid-prompt. The TTL and max-session limits are read from server config: @@ -223,17 +224,18 @@ serve: ## Security properties worth knowing -- **API key required.** Every session request carries `X-Secret-Key`; requests - without a valid key are rejected (health check aside). See +- **API key required.** Every remote session request carries `X-Secret-Key`. The + server rejects requests without a valid API key (health check aside). See [Authentication](authentication.md). -- **IP binding.** A session records the creator's IP at create time. If a later - request for the same session comes from a different IP, the server returns - `403 FORBIDDEN`. Sessions are not portable between hosts. -- **Namespaced, isolated cache.** Each session gets its own cache directory, - namespaced by a hash of the API key and the session ID, so sessions can't read - each other's cached tokens. The directory is deleted when the session ends. -- **Secrets are scrubbed from logs.** Session IDs, service tags, and other user - values are sanitized before logging. +- **IP binding.** A remote session records the creator's IP when the server makes it. + If a later request for the same remote session comes from a different IP, the server + returns `403 FORBIDDEN`. Remote sessions are not portable between hosts. +- **Namespaced, isolated cache.** Each remote session gets its own cache directory, + namespaced by a hash of the API key and the remote session ID, so remote sessions + cannot read each other's cached tokens. The server deletes the directory when the + remote session ends. +- **The server redacts secrets in logs.** It redacts remote session IDs, service tags, + and other user values before it writes them to the log. --- @@ -243,18 +245,18 @@ serve: The material below documents the server-side implementation and the HTTP contract. End users configuring a client can stop at the sections above. -### Session endpoints +### Remote session endpoints -All routes are exposed even in `--remote-only` server mode. Paths use the -`session_id` returned by create. +The server mounts all these routes, even in `--remote-only` mode. Paths use the +`session_id` that `create` returns. !!! note "Why `--remote-only` exists" - `--remote-only` narrows the server to just the health, services, search, and - session subset (and emits CORS headers) precisely so it is safe to sit behind + `--remote-only` narrows the server to only the health, services, search, and + remote session subset (and emits CORS headers), precisely so it is safe to sit behind Cloudflare or serve cross-origin browser clients. That trimmed surface, rather than the full `--api-only` mode, is what makes a CORS/Cloudflare-fronted deployment practical: reach for it when the server is public-facing or accessed - from a browser origin, not when you simply want the local HTTP API. + from a browser origin, not when you want only the local HTTP API. | Method | Path | Purpose | |---|---|---| @@ -284,8 +286,8 @@ The response returns **before** authentication finishes: ``` Authentication runs on a background thread (`asyncio.to_thread(authenticate, ...)`). -The session starts in `AUTHENTICATING`; the client must poll the prompt endpoint -until it reaches `authenticated` (or `failed`). +The remote session starts in `AUTHENTICATING`. The client must poll the prompt +endpoint until it reaches `authenticated` (or `failed`). ### Interactive auth: the `InputBridge` @@ -296,7 +298,7 @@ layer. - `AuthStatus` values: `authenticating`, `pending_input`, `authenticated`, `failed`. - `InputBridge.request_input(prompt, timeout=600)` blocks the sync auth thread on a `threading.Event` until `submit_response()` or `cancel()` fires. -- A timeout raises `TimeoutError` and marks the session `FAILED`. +- A timeout raises `TimeoutError` and marks the remote session `FAILED`. - `AUTH_INPUT_TIMEOUT = 600.0` seconds. This is also the TTL granted to `AUTHENTICATING` / `PENDING_INPUT` sessions in the store. @@ -309,30 +311,30 @@ layer. { "status": "failed", "error": "...message..." } ``` -A missing session returns `404 SESSION_NOT_FOUND`; an IP mismatch returns +A missing remote session returns `404 SESSION_NOT_FOUND`. An IP mismatch returns `403 FORBIDDEN`. `POST /api/session/{id}/prompt` takes `{ "response": "..." }` and -returns `{ "status": "accepted" }`; posting with no pending prompt is an +returns `{ "status": "accepted" }`. A post with no pending prompt is an `INVALID_INPUT` error. ### `SessionStore` internals -Source: `unshackle/core/api/session_store.py`. A singleton obtained via +Source: `unshackle/core/api/session_store.py`. A singleton you get from `get_session_store()`. - **Config-driven limits.** `serve.session_ttl` (default `300`) and `serve.max_sessions` (default `100`) are read as properties, so config changes take effect without recreating the store. -- **`create()`** evicts the least-recently-accessed session when at capacity, - then stores a new `SessionEntry` (defaulting `auth_status` to `AUTHENTICATED`; - the create handler overrides it to `AUTHENTICATING`). -- **`get()`** refreshes `last_accessed` via `touch()`. It returns `None` (and - deletes the entry) if an authenticated session has been idle longer than the TTL. - Sessions in `AUTHENTICATING` / `PENDING_INPUT` are exempt from TTL expiry. -- **`cleanup_expired()`** runs every 60s: authenticated sessions expire at `ttl`, - in-flight-auth sessions expire at `AUTH_INPUT_TIMEOUT`. Removing a session - cancels its `InputBridge` and deletes its cache directory (pruning empty parent - dirs up to, but not including, the cache root). -- **`cancel_all_bridges()`** is called on server shutdown to unblock any waiting +- **`create()`** evicts the least-recently-accessed remote session when at capacity, + then stores a new `SessionEntry`. It defaults `auth_status` to `AUTHENTICATED`, and + the `create` handler overrides it to `AUTHENTICATING`. +- **`get()`** refreshes `last_accessed` through `touch()`. It returns `None` (and + deletes the entry) if an authenticated remote session has been idle longer than the + TTL. Remote sessions in `AUTHENTICATING` / `PENDING_INPUT` are exempt from TTL expiry. +- **`cleanup_expired()`** runs every 60s: authenticated remote sessions expire at + `ttl`, in-flight-auth remote sessions expire at `AUTH_INPUT_TIMEOUT`. The removal of + a remote session cancels its `InputBridge` and deletes its cache directory (pruning + empty parent dirs up to, but not including, the cache root). +- The server calls **`cancel_all_bridges()`** on shutdown, to unblock any waiting auth threads. `SessionEntry` fields: @@ -352,22 +354,22 @@ Source: `unshackle/core/api/session_store.py`. A singleton obtained via ### Per-session cache namespacing -The create handler builds a `Cacher` namespaced as: +The `create` handler builds a `Cacher` namespaced as: ```text _sessions/// ``` -Forwarded `cache` files are written into that directory before authentication. +The handler writes forwarded `cache` files into that directory before authentication. On `DELETE`, the handler harvests updated cache files (compressing each with zlib and base64-encoding, **excluding** `titles_*` files) and returns them under a -`cache` key so the client can persist refreshed tokens: +`cache` field, so the client can keep refreshed tokens: ```json { "status": "ok", "cache": { "tokens": "...base64(zlib(bytes))..." } } ``` -### Session info response +### Remote session info response `GET /api/session/{id}`: @@ -384,12 +386,12 @@ and base64-encoding, **excluding** `titles_*` files) and returns them under a !!! warning "`expires_in` is the configured TTL, not time remaining" The `expires_in` value reports the store's configured `session_ttl`, not the - seconds left before this specific session expires. + seconds left before this specific remote session expires. ### Reference client: `RemoteService` Source: `unshackle/core/remote_service.py`. This is the canonical consumer of the -session API and a good template for any client. +remote session API and a good template for any client. - **`RemoteClient.request`** sets `X-Secret-Key` and `User-Agent: unshackle/`, uses a 120s timeout for `POST` and 30s for `GET`/`DELETE`, and treats any @@ -398,10 +400,10 @@ session API and a good template for any client. - **Retries.** The download-side HTTP session mounts an adapter with `Retry(total=5, backoff_factor=0.2, status_forcelist=[429, 500, 502, 503, 504])`. - **Flow.** `authenticate()` → `create` (+ poll `prompt` every 2s up to a 600s - deadline, answering `pending_input` prompts; with stdin closed the client logs an + deadline, answering `pending_input` prompts. With stdin closed the client logs an error and raises `SystemExit(1)` instead of posting an empty answer) → `get_titles()` → `get_tracks()` (merging returned `session_headers`/`session_cookies`, re-parsing `manifests`) → - license via proxy or `server_cdm` → `close()` (`DELETE`, saving any returned + get the licence in proxy mode or `server_cdm` mode → `close()` (`DELETE`, saving any returned `cache`). - **Server resolution.** `resolve_server()` reads `config.remote_services.` into `{url, api_key, services, server_cdm}`, injecting the `server_cdm` flag into @@ -412,20 +414,20 @@ session API and a good template for any client. ## Troubleshooting !!! example "\"Could not connect to remote server ... Is it running?\"" - The client couldn't reach the URL. Confirm `unshackle serve` is running on the - server, the `url`/port in `remote_services` are correct (default port `8786`), + The client could not reach the URL. Make sure that `unshackle serve` operates on + the server, the `url`/port in `remote_services` are correct (default port `8786`), and any firewall or reverse proxy allows the connection. !!! example "`403 FORBIDDEN` mid-download" - The request came from a different IP than the one that created the session. - Sessions are IP-bound. Don't move between networks (or NAT egress IPs) during a + The request came from a different IP than the one that made the remote session. + Remote sessions are IP-bound. Do not move between networks (or NAT egress IPs) during a remote download. !!! example "Auth times out or the prompt never resolves" - Interactive auth allows up to 600s. If you miss that window the session is - marked `failed` and cleaned up; just re-run the download. If the server never - prompts you, check that credentials/cookies forwarded from the client are valid - for the service. + Interactive auth allows up to 600s. If you miss that window, the server marks the + remote session `failed` and cleans it up. Operate the download again. If the server + never prompts you, make sure that the credentials and cookies the client forwarded + are valid for the service. !!! example "\"Multiple remote services configured. Use --server ...\"" You have more than one entry under `remote_services`. Pass `--server ` diff --git a/docs/dev/service-architecture.md b/docs/dev/service-architecture.md index d3fa0f0d..016db234 100644 --- a/docs/dev/service-architecture.md +++ b/docs/dev/service-architecture.md @@ -3,22 +3,22 @@ This page is the conceptual map of the internal object model that a **service author** works against. It ties together the moving parts: the `Service` base class, the `Title` hierarchy a service returns, the `Track` model those titles carry, the -manifest parsers that produce tracks, the DRM hooks that license and decrypt them, -and the request/session objects wired up for you at construction time. +manifest parsers that make tracks, the DRM hooks that license and decrypt them, +and the request and HTTP session objects wired up for you at construction time. !!! info "Who this page is for" - This is a **developer reference**. If you only want to *run* downloads, you never - touch these classes directly; see [Downloading](../guide/downloading.md) and the - [CLI reference](../guide/cli-reference.md) instead. If you are writing or reading a + This is a **developer reference**. If you only want to *download* titles, you never + touch these classes directly. See [Downloading](../guide/downloading.md) and the + [CLI reference](../guide/cli-reference.md) instead. If you write or read a service plugin, this page explains what each object is and how the pieces connect. - For a guided, step-by-step build, see [Creating a Service](creating-a-service.md); - for exact signatures, see the auto-generated [Service API](../reference/api/service.md), + For a guided, step-by-step walkthrough, see [Creating a Service](creating-a-service.md). + For exact signatures, see the auto-generated [Service API](../reference/api/service.md), [Titles API](../reference/api/titles.md), and [Tracks API](../reference/api/tracks.md). ## The shape of a download A service is a translator between one streaming platform and unshackle's generic -pipeline. It never downloads or decrypts anything itself; it just answers questions. +pipeline. It never downloads or decrypts anything itself. It only answers questions. The `dl` command drives the whole flow and only ever calls three required methods on your service, in order: @@ -44,15 +44,15 @@ Tracks.mux → mkvmerge → tagged .mkv (movie / episode) the `Video`/`Audio`/`Subtitle` objects. 3. **`get_chapters(title)`** returns any chapter markers. -Everything after that (track selection, licensing, decryption, repacking, subtitle -conversion, and muxing) is handled by the core using the objects your service -returned. Your job is to produce accurate `Title` and `Track` objects; the framework +The core does everything after that (track selection, licensing, decryption, +repacking, subtitle conversion, and muxing) with the objects your service +returned. Your job is to give accurate `Title` and `Track` objects. The framework does the rest. ## The `Service` base class Every service subclasses `unshackle.core.service.Service`, an abstract base class. -Two kinds of members matter: **class variables** that describe the service, and +Two kinds of members matter: **class variables** that give the details of the service, and **methods** that the pipeline calls. ### Class variables @@ -70,23 +70,23 @@ Declared at the top of your service class, these are static descriptors: ### Required methods -These are abstract; a service will not load unless it implements all three: +These are abstract. A service will not load unless it defines all three: -- `get_titles() -> Titles_T`: resolve the requested ID into a title collection. -- `get_tracks(title) -> Tracks`: build the track set for one title. +- `get_titles() -> Titles_T`: find the title collection for the requested ID. +- `get_tracks(title) -> Tracks`: assemble the track set for one title. - `get_chapters(title) -> Chapters`: return chapter markers (may be empty). ### Optional methods and hooks - **`authenticate(cookies=None, credential=None)`**: the base implementation loads a `CookieJar` into `self.session.cookies` and stores `self.credential`. Override it to - perform a real login handshake. -- **`search()`**: raises `NotImplementedError` by default; implement it to support the - `search` command. + do a real login handshake. +- **`search()`**: raises `NotImplementedError` by default. Write it so that the + `search` command can operate. - **DRM license hooks**: `get_widevine_service_certificate`, `get_widevine_license`, `get_playready_license` (delegates to Widevine by default), and `get_clearkey_license` (returns `None` by default). See [DRM hooks](#drm-hooks) below. -- **`request_input(prompt)`**: prompts the user; routed through the serve-mode input +- **`request_input(prompt)`**: prompts the user. It goes through the serve-mode input bridge or the rich console. - **Event callbacks**: `on_segment_downloaded`, `on_track_downloaded`, `on_track_decrypted`, `on_track_repacked`, `on_track_multiplex` let a service react to @@ -95,12 +95,12 @@ These are abstract; a service will not load unless it implements all three: resolved server-side *ahead of* the normal per-track license flow. Unlike everything else here it is **not declared on the `Service` base class at all**: `dl.py` probes for it with `hasattr` and calls it (once, after track selection) only if your service defines it. Use - it when keys must be fetched or injected in bulk before download, separate from the - per-track license callbacks below. + it when your service must fetch or inject keys in bulk before download, separate from + the per-track license callbacks below. ### What `__init__` wires up for you -You rarely override `__init__`; the base constructor takes the Click context and sets +You rarely override `__init__`. The base constructor takes the Click context and sets up the objects your methods rely on: | Attribute | What it is | @@ -116,7 +116,7 @@ up the objects your methods rely on: The constructor also resolves proxies: it reads `--proxy` / `--proxy-query` / `--proxy-provider`, consults the per-service `proxy_map`, and, if `GEOFENCE` is set and -no explicit proxy was given, does a live IP check and auto-fetches a proxy to +you did not give an explicit proxy, does a live IP check and auto-fetches a proxy to `GEOFENCE[0]` when it detects a geoblock. See [Proxies and VPN](../guide/proxies-and-vpn.md). !!! note "`TrackRequest`" @@ -157,7 +157,8 @@ Each subclass adds its own required fields on top of `id_` and `service`: Movie(id_, service, name, year=None, language=None, data=None, description=None) ``` - `name` is required; `year` is optional but must be a positive integer if given. + You must give `name`. `year` is optional, but it must be a positive integer if you + give it. === "Episode" @@ -167,9 +168,10 @@ Each subclass adds its own required fields on top of `id_` and `service`: description=None, air_date=None) ``` - `title` is the **show** name; `season` and `number` are required (0 is allowed). - An `air_date` is normalized to a `date` so the `{date}` naming token stays clean, - and a placeholder episode `name` like `"Episode 4"` is discarded automatically. + `title` is the **show** name. You must give `season` and `number`, and `0` is a + valid value. unshackle converts an `air_date` to a `date` so the `{date}` naming + token stays clean, and it discards a placeholder episode `name` like `"Episode 4"` + automatically. === "Song" @@ -178,7 +180,7 @@ Each subclass adds its own required fields on top of `id_` and `service`: language=None, data=None, ...) ``` - `name`, `artist`, `album`, and `track` are required and validated. `disc` defaults to + You must give `name`, `artist`, `album`, and `track`, and unshackle validates them. `disc` defaults to 1, and `year` is optional. A single therefore needs no `disc`, and an undated release needs no `year`. @@ -216,17 +218,17 @@ Each subclass adds its own required fields on top of `id_` and `service`: ### Title remapping -After titles are fetched (and after any cache), the framework applies -`remap_titles(titles, title_map)` using the per-service `title_map` config. This does an -**exact-string** rename (`Episode`s are matched on their show `title`, `Movie`s and -`Song`s on their `name`), so a platform's odd naming can be normalized to what your -library expects. Titles are also cache-wrapped via `get_titles_cached()`, which is region- -and account-aware; see the [TitleCacher](#request-and-session-objects) notes. +After the service fetches the titles (and after any cache), the framework applies +`remap_titles(titles, title_map)` with the per-service `title_map` config. This does an +**exact-string** rename (it compares `Episode`s on their show `title`, and `Movie`s and +`Song`s on their `name`), so you can change a platform's odd naming to what your +library expects. `get_titles_cached()` also cache-wraps the titles, and it is region- +and account-aware. See the [TitleCacher](#request-and-session-objects) notes. -The ordering here is deliberate on both sides. `title_map` is applied **after** the title -cache so that editing the map takes effect immediately, without forcing a cache reset, and -**before** metadata enrichment (`--enrich`) so that an explicit enrich override still wins -over a mapped name. +The ordering here is deliberate on both sides. The framework applies `title_map` **after** +the title cache so that editing the map takes effect immediately, without forcing a cache +reset, and **before** metadata enrichment (`--enrich`) so that an explicit enrich override +still wins over a mapped name. ## Tracks and the `Track` model @@ -239,14 +241,14 @@ attachments), and the canonical display/mux order is Video → Audio → Subtitl !!! warning "Track IDs must be deterministic" `Tracks.add` refuses duplicate IDs. Track IDs are CRC32-derived from the track's identity (URL, codec, language, bitrate) and **must be stable across requests**, never - random. The manifest parsers already do this correctly; if you build tracks by hand, + random. The manifest parsers already do this correctly. If you assemble tracks by hand, keep IDs deterministic or `add` will raise. ### The base `Track` `Track` is the base for `Video`, `Audio`, and `Subtitle`. (`Chapter`, `Chapters`, and -`Attachment` are standalone classes, **not** `Track` subclasses.) Every track knows how -it is delivered, via `Track.Descriptor`: +`Attachment` are standalone classes, **not** `Track` subclasses.) Every track records its +delivery method in `Track.Descriptor`, which tells unshackle how to download it: | Descriptor | Delivery | |---|---| @@ -255,10 +257,10 @@ it is delivered, via `Track.Descriptor`: | `DASH` | Dynamic Adaptive Streaming over HTTP. | | `ISM` | Smooth Streaming. | -Key attributes include `url` (a string or list of strings), `language` (a +The main attributes include `url` (a string or list of strings), `language` (a `langcodes.Language`), `drm` (an optional iterable of DRM objects), `needs_repack`, and -descriptor-specific state stashed in `track.data["hls"]` / `track.data["dash"]`. When -`name` is omitted, a human-readable name is derived from the language automatically. +descriptor-specific state stashed in `track.data["hls"]` / `track.data["dash"]`. If you +do not give a `name`, unshackle derives a human-readable name from the language. ### `Video` @@ -268,12 +270,12 @@ closed-caption descriptors. The relevant enums: - **`Video.Codec`**: `AVC` (H.264), `HEVC` (H.265), `VC1`, `VP8`, `VP9`, `AV1`. - **`Video.Range`**: `SDR`, `HLG`, `HDR10`, `HDR10P` (HDR10+), `DV` (Dolby Vision), and `HYBRID`. `HYBRID` is special: it tells the pipeline to select both an HDR10(+) base and - a DV track and merge the DV RPU onto the HDR10 stream with `dovi_tool` (see + a DV track and merge the DV RPU onto the HDR10 track with `dovi_tool` (see [Dolby Vision hybrid](#dolby-vision-and-hybrid)). !!! note "Bitrate units" Both `Video` and `Audio` store `bitrate` in **bits per second** (rounded up). The - string form displays kb/s, and `--vbitrate` and `--abitrate` match on kb/s. + string form displays kb/s, and `--vbitrate` and `--abitrate` compare in kb/s. ### `Audio` @@ -295,11 +297,11 @@ all documented for users in [Subtitles](../guide/subtitles.md). ### Chapters and attachments -- **`Chapter(timestamp, name=None)`**: a single marker; the timestamp accepts +- **`Chapter(timestamp, name=None)`**: a single marker. The timestamp accepts `HH:MM:SS[.mmm]`, integer milliseconds, or float seconds. - **`Chapters`**: a sorted collection that auto-inserts a `00:00:00.000` chapter and reads/writes OGM Simple format. -- **`Attachment`**: a font or image attached to the mux (e.g. fonts an ASS subtitle +- **`Attachment`**: a font or image attached to the mux (for example, fonts an ASS subtitle references), built from a local `path` or a `url`. ### Selection, sorting, and muxing @@ -310,12 +312,12 @@ all documented for users in [Subtitles](../guide/subtitles.md). object multiplexes everything: **`Tracks.mux(title, ...)`** builds and runs an `mkvmerge` command, applying language, default-track, forced, and hearing-impaired flags, attaching chapters and fonts, and writing a `.muxed.mkv` (or `.mka` / `.mks` for -audio- or subtitle-only sets). As a service author you never call `mux` yourself; you -just return correct tracks. +audio- or subtitle-only sets). As a service author you never call `mux` yourself. You +only return correct tracks. ## Manifest parsers -Most services do not build tracks by hand; they hand a manifest to one of three parsers +Most services do not assemble tracks by hand. They give a manifest to one of three parsers in `unshackle.core.manifests`. Each parser has the same three-part contract: | Format | Class | Notes | @@ -328,40 +330,40 @@ in `unshackle.core.manifests`. Each parser has the same three-part contract: 1. **Construct**: `DASH.from_url(url, session=...)` or `DASH.from_text(text, url)` (and likewise for `HLS` / `ISM`). A `url` is always required, even for `from_text`, - because relative segment paths are resolved against it. + because the parser needs it to make relative segment paths absolute. 2. **`to_tracks(...)`**: parses the manifest into a `Tracks` container of `Video`, `Audio`, and `Subtitle` objects, each carrying descriptor-specific state. 3. **`download_track(...)`**: a staticmethod the core calls to fetch, decrypt, and merge - segments. You do not call it; you just return the tracks it will operate on. + segments. You do not call it. You only return the tracks it will operate on. A typical `get_tracks` therefore looks like fetching the manifest text with `self.session` and returning `DASH.from_text(mpd, url).to_tracks(language=title.language)`. !!! tip "Pass the original language" - DASH and HLS `to_tracks` raise `ValueError` if a track's language cannot be derived - and no valid fallback was supplied. Always pass the title's original recorded - language. For HLS this fallback is used directly for the primary track, since HLS - variants carry no language. + DASH and HLS `to_tracks` raise `ValueError` if they cannot derive a track's language + and you gave no valid fallback. Always pass the title's original recorded + language. For HLS, `to_tracks` uses this fallback directly for the primary track, + because HLS variants carry no language. ### Segment-filtering hooks - **HLS**: set `track.OnSegmentFilter(segment)` to drop unwanted segments (ads/bumpers). - A dropped segment is invisible to the whole download: its `EXT-X-KEY` and its - `EXT-X-DISCONTINUITY` are ignored, so a clear ad break inside encrypted content does + A dropped segment is invisible to the whole download: unshackle ignores its `EXT-X-KEY` + and its `EXT-X-DISCONTINUITY`, so a clear ad break inside an encrypted track does not disturb decryption. - **DASH**: pass a `period_filter` callable to `to_tracks` to drop whole periods. - Caching-breadth periods some services emit and non-`content` segment types are dropped automatically. + unshackle automatically drops the caching-breadth periods that some services send, and the non-`content` segment types. ### Format gotchas worth knowing - **DASH** enumerates tracks from only the *first* content period, but downloads segments - from *all* content periods matching the representation. `SegmentTemplate`, - `SegmentList`, and `SegmentBase` addressing and `$Number%0Nd$` padding are handled. + from *all* content periods matching the representation. The parser reads `SegmentTemplate`, + `SegmentList`, and `SegmentBase` addressing, and `$Number%0Nd$` padding. - **HLS** re-fetches the media playlist at download time, recovers `SUPPLEMENTAL-CODECS` from raw text (python-m3u8 drops it) to detect Dolby Vision composite streams, and uses - the media-sequence-number IV for AES-128 keys that omit an explicit IV. -- **ISM** has no segment list and no init segments: segment URLs are synthesized from the - `` fragment timeline, and a `moov`/`ftyp` init box is rebuilt from `CodecPrivateData` + the media-sequence-number IV for AES-128 keys that have no explicit IV. +- **ISM** has no list of segments and no init segments: the parser synthesizes segment URLs + from the `` fragment timeline, and rebuilds a `moov`/`ftyp` init box from `CodecPrivateData` (supported for AVC, HEVC, DV-over-HEVC, AAC, EC-3, and TTML). ## DRM hooks @@ -384,8 +386,8 @@ service to talk to the platform's license server. unshackle recognizes five DRM ### The service-side hooks -The framework supplies the CDM and the decryption tool; your service provides the network -calls that the CDM's challenge must be sent through. Override the hooks that your platform +The framework supplies the CDM and the decryption backend. Your service gives the network +calls that must carry the CDM's challenge. Override the hooks that your platform needs: | Hook | Return | Purpose | @@ -395,29 +397,30 @@ needs: | `get_playready_license(title, track, challenge)` | license | Same for PlayReady; defaults to delegating to Widevine. | | `get_clearkey_license(title, track, challenge)` | dict / str / bytes / `None` | For `ClearKeyCENC`; return `None` to let the framework POST to the manifest `Laurl`. | -You do not open CDM sessions or run shaka-packager yourself. The `Widevine`/`PlayReady` -classes drive the CDM handshake and invoke the decryption tool. Your hook is just the +You do not open CDM sessions or operate shaka-packager yourself. The `Widevine`/`PlayReady` +classes drive the CDM handshake and call the decryption backend. Your hook is only the "send this challenge, give me back the license" step. -### How a CDM is chosen and keys are found +### How unshackle chooses a CDM and finds keys CDM selection is config-driven: the `cdm` map resolves a service tag (and optionally the track quality or DRM type) to a device name, which `load_cdm` instantiates as a local `.wvd`/`.prd` or a remote CDM. Before ever requesting a license, the framework tries an -in-process key cache and the configured **key vaults**; only unresolved KIDs trigger an -actual license request, and newly licensed keys are written back to every vault. This is -why the same key is rarely re-licensed twice. Users configure all of this in -[DRM and CDM](../guide/drm-and-cdm.md) and [Key vaults](../guide/vaults.md); the API -surface is documented in the [DRM API reference](../reference/api/drm.md). +in-process cache of content keys and the configured **key vaults**. Only unresolved KIDs +trigger an actual license request, and the framework writes newly licensed keys back to +every vault. This is why the same content key is rarely re-licensed twice. Users configure +all of this in [DRM and CDM](../guide/drm-and-cdm.md) and +[Key vaults](../guide/vaults.md). The [DRM API reference](../reference/api/drm.md) gives +the API surface. ### Dolby Vision and hybrid -Two DV mechanisms sit on top of the track model. **DV fixup** re-exposes DV signaling on +Two DV mechanisms sit on top of the track model. **DV fixup** re-exposes DV signalling on HLS streams that carry a valid DV RPU but advertise plain HEVC (`dv_compatible_bitstream`). **Hybrid** processing (`Video.Range.HYBRID`) selects an HDR10(+) base and a DV donor, -extracts the RPU with `dovi_tool`, and injects it onto the base layer, producing a single -stream with both HDR10 and DV metadata. Both are automatic once the right tracks are -selected; a service only needs to expose accurate ranges. +extracts the RPU with `dovi_tool`, and injects it onto the base layer, to make a single +track that carries both HDR10 metadata and the DV RPU. Both are automatic once the +pipeline selects the right tracks. A service only needs to give accurate ranges. ## Request and session objects @@ -425,23 +428,23 @@ Everything a service does over the network flows through objects the base constr prepared, all derived from the Click context (`ctx`) that `dl` passes in. - **`self.config`**: your service's slice of `unshackle.yaml`, i.e. - `config.services[]`. This is a plain dict; use it for per-service settings + `config.services[]`. This is a plain dict. Use it for per-service settings like base URLs, an API key, or a `title_map`. See [Configuration](../getting-started/configuration-file.md). - **`self.session`**: a `requests.Session` with `config.headers` applied, a retry policy (`total=5`, backoff, retry on 429/5xx), and any resolved proxy already mounted. Pass this - session into the manifest parsers and DRM calls so headers, cookies, and proxies flow - everywhere. (unshackle also ships an rnet-based TLS-fingerprinting session used by the - newer `session()` factory and configured via the `network` config key.) + HTTP session into the manifest parsers and DRM calls so headers, cookies, and proxies flow + everywhere. (unshackle also ships an rnet-based TLS-fingerprinting HTTP session that the + newer `session()` factory uses and that the `network` config key configures.) - **`self.credential`**: the active `Credential` (`username:password[:extra]`), populated - by `authenticate`. Its `sha1` is used as the account hash for cache keys. + by `authenticate`. unshackle uses its `sha1` as the account hash for cache keys. - **`self.cache`**: a generic `Cacher`: JSON key/value storage under the service's cache dir, with CRC32 integrity checks, versioning, and optional expiry (auto-derived from a JWT `exp` claim when present). Good for auth tokens and API responses. - **`self.title_cache`**: a `TitleCacher` that wraps `get_titles` results, keyed by title ID, region, and account hash. It serves fresh data within `title_cache_time`, falls back - to stale data within `title_cache_max_retention` when an API call fails, and is bypassed - by `--no-cache`. + to stale data within `title_cache_max_retention` when an API call fails, and `--no-cache` + bypasses it. - **`self.track_request`**: the `TrackRequest` describing what the user asked for (codecs, ranges, best-available), for use inside `get_tracks`. @@ -480,7 +483,7 @@ prepared, all derived from the Click context (`ctx`) that `dl` passes in. return self.session.post("https://lic.example.com/wv", data=challenge).content ``` - The directory name (`EXMP`) must exactly match the class name; that stem is the + The directory name (`EXMP`) must match the class name exactly. That stem is the service tag. Put your service in one of the `directories.services` paths. ## Where to go next diff --git a/docs/getting-started/configuration-file.md b/docs/getting-started/configuration-file.md index 572e93c3..dc74c495 100644 --- a/docs/getting-started/configuration-file.md +++ b/docs/getting-started/configuration-file.md @@ -1,13 +1,13 @@ # The Configuration File -Almost everything about how unshackle behaves is controlled by a single YAML file named `unshackle.yaml`: where it saves downloads, which CDM it uses, your credentials, proxies, filename templates, and more. This page covers where that file lives, how unshackle finds it, the shape of its contents, the directory layout unshackle builds around it, and how to edit values from the command line. +A single configuration file named `unshackle.yaml` controls almost everything about how unshackle behaves. It sets where unshackle saves downloads, which CDM it uses, your credentials, proxies, filename templates, and more. This page gives where that file lives, how unshackle finds it, and the shape of its contents. It also gives the directory layout unshackle makes around it, and how to edit values from the command line. !!! note "Where the exhaustive key list lives" This page is an overview. For the complete, key-by-key breakdown of every setting, its type, and its default, see the [Configuration Reference](../reference/configuration/index.md). ## The config filename -The main configuration file is always named `unshackle.yaml`. It is a plain YAML file. You can open and edit it in any text editor, or manage individual values with the [`unshackle cfg`](#editing-the-config-with-unshackle-cfg) command. +The main configuration file is always named `unshackle.yaml`. It uses plain YAML. You can open and edit it in any text editor, or manage individual values with the [`unshackle cfg`](#editing-the-config-with-unshackle-cfg) command. If you have never created one, that is fine: unshackle runs entirely on built-in defaults when no config file exists. The file only needs to contain the keys you want to change from their defaults. @@ -42,13 +42,13 @@ The third location, your per-user config directory, is the recommended place for ``` !!! tip "Not sure which file is being used?" - Run `unshackle env info`. It prints the path the config was loaded from (or tells you none was found), along with every directory unshackle is currently using. + Operate `unshackle env info`. It prints the path unshackle loaded the config from, or tells you that it found none. It also prints every directory unshackle currently uses. -The config is read exactly once, when unshackle starts. If you edit the file, the change takes effect on the next command you run. +The config is read exactly once, when unshackle starts. If you edit the file, the change takes effect on the next command. ## Top-level structure -`unshackle.yaml` is a flat map of top-level keys. Each key configures one area of the program. You only include the keys you want to change; everything else falls back to its default. +`unshackle.yaml` is a flat map of top-level config keys. Each config key configures one area of the program. You include only the config keys you want to change. Everything else falls back to its default. Here is a small, realistic example that sets a download location, a release tag, a default CDM, and one service's credentials: @@ -93,14 +93,14 @@ The top-level keys group loosely into these areas: For the full list with types and defaults, see the [Configuration Reference](../reference/configuration/index.md). !!! warning "Unknown keys are silently ignored" - unshackle does not validate your config against a schema. If you misspell a key, it is simply skipped and the default is used. You will not get an error. Double-check key names (and their nesting) if a setting does not seem to apply. + unshackle does not validate your config against a schema. If you misspell a config key, unshackle skips it and uses the default. You will not get an error. Examine the config key names (and their nesting) if a setting does not seem to apply. ## Setting download defaults (`dl:`) If you always pass the same `dl` flags (a language, a resolution, a codec), put them under a -`dl:` key once and unshackle applies them to every download. Any flag from -[Downloading](../guide/downloading.md) works here; the key is the flag's long name with dashes -turned into underscores (`--best-available` → `best_available`). +`dl:` config key once and unshackle applies them to every download. Any flag from +[Downloading](../guide/downloading.md) works here. The config key is the flag's long name with +dashes turned into underscores (`--best-available` → `best_available`). ```yaml title="unshackle.yaml" dl: @@ -111,7 +111,7 @@ dl: downloads: 2 # two tracks at once ``` -You can still override any of these on the command line for a one-off; an explicit flag +You can still override any of these on the command line for a one-off. An explicit flag always beats the config default. You can also scope defaults to a single service by nesting a `dl:` block under it: @@ -128,11 +128,11 @@ services: Most keys are obvious, but set these exact ones: `range` (`-r`), `list` (`--list`), `tmdb_id` (`--tmdb`), `imdb_id` (`--imdb`), `tvdb_id` (`--tvdb`), `no_atmos` (`--noatmos`), and `output_dir` (`-o`). The [Configuration Reference](../reference/configuration/download.md#dl) has the full list - and every available key. + and every available config key. ## The directory layout -The `directories` key controls where unshackle reads and writes its various files. Each directory has a sensible default, and you can override most of them by giving a new path. Paths support `~` for your home directory. +The `directories` config key controls where unshackle reads and writes its various files. Each directory has a sensible default, and you can override most of them by giving a new path. A path can use `~` for your home directory. ```yaml title="unshackle.yaml" directories: @@ -160,11 +160,11 @@ The directories unshackle uses: | `fonts` | Bundled fonts | Yes | !!! note "Some directories cannot be moved" - unshackle protects five internal entries: `app_dirs`, `core_dir`, `namespace_dir`, `user_configs`, and `data`. If you list any of them under `directories`, unshackle ignores the override. This is intentional; those paths are tied to where the package is installed. + unshackle protects five internal entries: `app_dirs`, `core_dir`, `namespace_dir`, `user_configs`, and `data`. If you write any of them under `directories`, unshackle ignores the override. This is intentional. Those paths come from the package location and your OS user directories. ### The `services` directory is special -Unlike the other entries, `services` is a **list**, and each entry can be either a local folder or a remote repository. This lets you mix your own local service code with services pulled from Git: +Unlike the other entries, `services` is a **list**, and each entry can be either a local folder or a remote repository. This lets you mix your own local services with services pulled from Git: ```yaml title="unshackle.yaml" directories: @@ -174,31 +174,31 @@ directories: - ~/code/local-services # a local folder ``` -unshackle searches entries in the order listed, and **the first source to define a given service tag wins**, so put local folders last if you want them to act as fallbacks rather than overrides. unshackle clones remote repositories on first use and refreshes them at most once a day. See [Creating a Service](../dev/creating-a-service.md) for how service discovery and repositories work. +unshackle examines the entries in the order you wrote them, and **the first source to define a given service tag wins**. Put local folders last if you want them to act as fallbacks and not overrides. unshackle clones remote repositories on first use and refreshes them at most once a day. See [Creating a Service](../dev/creating-a-service.md) for how service discovery and repositories work. -## The `filenames` key +## The `filenames` config key -Alongside `directories`, the `filenames` key lets you override the templates unshackle uses when naming its own working and log files (for example the log filename or the temporary chapters file). Most users never need to touch this. The available names and their default templates are listed in the [Configuration Reference](../reference/configuration/directories.md#filenames). +Alongside `directories`, the `filenames` config key lets you override the templates unshackle uses when it names its own working and log files. Two examples are the log filename and the temporary chapters file. Most users never need to touch this. The [Configuration Reference](../reference/configuration/directories.md#filenames) gives the available names and their default templates. ## Editing the config with `unshackle cfg` You do not have to edit `unshackle.yaml` by hand. The `unshackle cfg` command reads and writes individual values for you, creating the config file (and its parent directory) if it does not exist yet. -**Read a single value** by passing its key. Nested keys use dot notation: +**Read a single value** by passing its config key. Nested config keys use dot notation: ```console $ unshackle cfg tag $ unshackle cfg cdm.default ``` -**Set a value** by passing a key and a value: +**Set a value** by passing a config key and a value: ```console $ unshackle cfg tag MYGRP $ unshackle cfg cdm.default my_device_l3 ``` -unshackle parses the value as a Python literal, so write booleans as `True`/`False` and quote the strings inside a list (`"['en']"`). Anything that is not valid Python literal syntax is stored as a plain string: +unshackle parses the value as a Python literal, so write booleans as `True`/`False` and quote the strings inside a list (`"['en']"`). unshackle stores anything that is not valid Python literal syntax as a plain string: ```console $ unshackle cfg update_checks False @@ -211,19 +211,19 @@ $ unshackle cfg vault_timeout 30 $ unshackle cfg cdm.default --unset ``` -**List everything** currently set with `--list`: +**Show everything** currently set with `--list`: ```console $ unshackle cfg --list ``` -When it writes, `unshackle cfg` targets the config file that was loaded. If none exists yet, it creates `unshackle.yaml` inside the `unshackle` package folder (search location 1), not your OS user-config directory. To keep the config outside the package, create the file at the user-config path yourself first, then `unshackle cfg` writes to it. +When it writes, `unshackle cfg` targets the config file that unshackle loaded. If none exists yet, it makes `unshackle.yaml` inside the `unshackle` package folder (location 1 above), not your OS user-config directory. To keep the config outside the package, make the file at the user-config path yourself first, then `unshackle cfg` writes to it. !!! warning "Editing with `cfg` strips comments" - Because `unshackle cfg` rewrites the whole file when it saves, any comments in `unshackle.yaml` are removed by a write. If you keep important notes as comments, edit the file by hand instead, or keep those notes elsewhere. + Because `unshackle cfg` rewrites the whole file when it saves, a write removes any comments in `unshackle.yaml`. If you keep important notes as comments, edit the file by hand instead, or keep those notes elsewhere. ## Next steps -- Browse the [Configuration Reference](../reference/configuration/index.md) for every key and default. +- Browse the [Configuration Reference](../reference/configuration/index.md) for every config key and default. - Set up your first download in [Downloading](../guide/downloading.md). -- Learn how services are discovered and updated in [Creating a Service](../dev/creating-a-service.md). +- Learn how unshackle discovers and updates services in [Creating a Service](../dev/creating-a-service.md). diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index f993ae91..5b476d9a 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -24,18 +24,18 @@ version it needs, and the external command-line tools it expects to find on your | [dovi_tool](https://github.com/quietvoid/dovi_tool) | ≥ 2.1 | Dolby Vision handling | | [SubtitleEdit](https://github.com/SubtitleEdit/subtitleedit/releases) | ≥ 5.0 | Subtitle conversion (optional) | -The sections below explain each of these and how to confirm your setup is -complete. +The sections below tell you about each of these, and how to make sure that your +setup is complete. -## Python version support +## Python versions unshackle requires **Python 3.11 or newer, up to and including 3.14** (`requires-python = ">=3.11,<3.15"`). Anything older than 3.11 or 3.15 and -later is unsupported and the installer will refuse to build the environment. +later is unsupported, and the installer refuses to assemble the environment. You do not usually need to install a matching Python interpreter by hand, since -`uv` can download and manage one for you. If you are working inside an existing -virtual environment, make sure it is on a supported version. +`uv` can download and manage one for you. If you work inside an existing +virtual environment, make sure that it is on a supported version. ## Installing with uv @@ -54,14 +54,14 @@ unshackle --help ``` !!! tip "Upgrading and removing" - Because it is installed from a git URL, re-run the same - `uv tool install` command (uv will update in place) to pull a newer build, + Because you installed it from a git URL, re-run the same + `uv tool install` command (uv will update in place) to get a newer version, and use `uv tool uninstall unshackle` to remove it. ### Running from a clone If you would rather work from a checkout of the repository (for example to try -an in-development branch or to keep local service code alongside unshackle), use +an in-development branch or to keep your own local services alongside unshackle), use `uv run` inside the clone. This keeps the project's virtual environment active for the duration of the command, so you do not have to activate it yourself. @@ -71,8 +71,8 @@ cd unshackle uv run unshackle --help ``` -Every command shown elsewhere in these docs works the same way from a clone, -just prefix it with `uv run`: +Every command shown elsewhere in these docs works the same way from a clone. +Prefix it with `uv run`: ```shell uv run unshackle env check @@ -81,7 +81,7 @@ uv run unshackle env check !!! note "Developer note" Contributors will also want the development and test dependency groups. The project defines `dev` (linters, type checkers) and `test` (pytest and - friends) groups in `pyproject.toml`; install them with + friends) groups in `pyproject.toml`. Install them with `uv sync --group dev --group test` inside the clone. End users can ignore these entirely. @@ -109,12 +109,12 @@ These must be present for core downloading, decryption, and muxing to work: unshackle decrypts with shaka-packager unless you explicitly configure a different decryptor. If it is missing, protected downloads will fail at the decryption step. See the [configuration reference](../reference/configuration/drm.md#decryption) for the - `decryption` key if you want to switch to `mp4decrypt`. + `decryption` config key if you want to switch to `mp4decrypt`. ### Recommended and optional tools These extend unshackle's capabilities. Install the ones relevant to what you -download. For example, `dovi_tool` only matters if you handle Dolby Vision. +download. For example, `dovi_tool` only matters if you download Dolby Vision video. | Tool | Binary looked up | Why it is needed | | --- | --- | --- | @@ -138,10 +138,10 @@ download. For example, `dovi_tool` only matters if you handle Dolby Vision. ## Verifying your installation -After installing, confirm the command runs and that unshackle can see the tools -it depends on. +After installation, make sure that the command works and that unshackle can see +the tools it depends on. -### Confirm the command is available +### Make sure the command is available ```shell unshackle --help @@ -150,9 +150,9 @@ unshackle --help This prints the list of available commands (`dl`, `env`, `kv`, `serve`, `wvd`, `prd`, `search`, `util`, `cfg`, `import`). If your shell reports that `unshackle` is not found, the `uv tool` bin directory is probably not on -your `PATH`. Run `uv tool update-shell` and open a new terminal. +your `PATH`. Operate `uv tool update-shell` and open a new terminal. -### Check external dependencies +### Examine the external dependencies The `env check` subcommand inspects your environment and reports which tools it found: @@ -187,10 +187,10 @@ unshackle env info ``` If no configuration file exists yet, this command lists the locations unshackle -searches for one, so you know where to create it. See the +searches for one, so you know where to make it. See the [configuration guide](configuration-file.md) for the details of that file. ## Next steps - Set up your [configuration](configuration-file.md) file (`unshackle.yaml`). -- Start [downloading](../guide/downloading.md) content once your tools check out. +- Start [downloading](../guide/downloading.md) titles when `env check` finds every required tool. diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index d0bbdfde..5c453fb7 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -1,8 +1,8 @@ # Quick Start This page takes you from a fresh install to your first downloaded file. You will -create the one piece of configuration unshackle insists on, supply the service -module and CDM it needs to fetch and decrypt media, run a basic `unshackle dl` +make the one piece of configuration unshackle insists on, supply the service +module and CDM it needs to fetch and decrypt media, operate a basic `unshackle dl` command, and learn where the finished file lands. !!! note "Before you start" @@ -16,11 +16,11 @@ command, and learn where the finished file lands. ([step 3](#3-write-a-service)). `EXAMPLE` in this guide stands in for the tag you give yours. -## 1. Check your environment +## 1. Examine your environment unshackle drives several external tools (FFmpeg, MKVToolNix, shaka-packager, and -others) to decrypt, repack, and mux media. Confirm they are visible before you -download anything: +others) to decrypt, repack, and mux media. Make sure that they are visible before +you download anything: ```shell unshackle env check @@ -35,16 +35,18 @@ matter for the features that use them. The summary line at the bottom reports `installed/total` and lists anything required that is still missing, so you know exactly what to install next. -## 2. Create a minimal config +## 2. Make a minimal config -unshackle reads a single YAML file named `unshackle.yaml`. To see where it looks -for that file, and where it *would* accept one if you have not made it yet, run: +unshackle reads a single configuration file named `unshackle.yaml`. To see where it +looks for that file, and where it *would* accept one if you have not made it yet, +use this command: ```shell unshackle env info ``` -If no config exists, this prints the candidate locations. In search order they are: +If no config exists, this prints the candidate locations. In the order unshackle +examines them, they are: 1. `unshackle.yaml` inside the unshackle package folder. 2. `unshackle.yaml` in that folder's parent. @@ -52,10 +54,10 @@ If no config exists, this prints the candidate locations. In search order they a (`~/.config/unshackle/` on Linux, `%LOCALAPPDATA%\unshackle\` on Windows, `~/Library/Application Support/unshackle/` on macOS). -The **first** file that exists wins. Create `unshackle.yaml` in one of those +The **first** file that exists wins. Make `unshackle.yaml` in one of those locations. -### The one key you must set +### The one config key you must set unshackle refuses to start a download unless it knows how to name the output file. That means `output_template` is the one setting a first run genuinely requires. @@ -67,14 +69,14 @@ output_template: series: "{title} {season_episode} {episode_name?} {quality} {source}" ``` -Each `{variable}` is filled in from the title and the tracks you downloaded. The -full set of valid variables, including `resolution`, `video`, `audio`, `hdr`, -`edition`, `tag`, and more, is documented in [Output and Naming](../guide/output-and-naming.md). +unshackle fills in each `{variable}` from the title and the tracks you downloaded. +[Output and Naming](../guide/output-and-naming.md) gives the full set of valid +variables, including `resolution`, `video`, `audio`, `hdr`, `edition`, `tag`, and more. !!! note "Spaces or dots?" unshackle auto-detects your naming style from the template: if the separators - between variables are mostly spaces, it uses spaces; if they are mostly dots, - it produces scene-style `Title.S01E01.1080p` names. Write the template in the + between variables are mostly spaces, it uses spaces. If they are mostly dots, + it makes scene-style `Title.S01E01.1080p` names. Write the template in the style you want the filenames to look. !!! warning "Editing config from the CLI strips comments" @@ -95,8 +97,8 @@ commands. ## 4. Add a CDM for DRM -Most streaming content is encrypted. To fetch decryption keys, unshackle needs a -**CDM**, a Widevine device (`.wvd`) or a PlayReady device (`.prd`). Register a +Most streaming services encrypt their titles. To fetch content keys, unshackle +needs a **CDM**, a Widevine device (`.wvd`) or a PlayReady device (`.prd`). Add a Widevine device you already have with: ```shell @@ -104,8 +106,8 @@ unshackle wvd add /path/to/device.wvd ``` This validates the file and moves it into your WVDs directory. Then point services -at it in your config. The `cdm` map is keyed by service tag, with a `default` that -covers everything else: +at it in your config. The `cdm` map uses the service tag as its config key, with a +`default` that covers everything else: ```yaml title="unshackle.yaml" cdm: @@ -114,20 +116,20 @@ cdm: ``` PlayReady works the same way with `.prd` files created and managed by the -[`prd`](../guide/cli-reference.md#prd) command. If a title is DRM-free, no CDM is needed. +[`prd`](../guide/cli-reference.md#prd) command. If a title is DRM-free, unshackle needs no CDM. !!! tip - Run `unshackle wvd parse my_device` to inspect a device's security level and - contents, and `unshackle env info` to confirm where WVDs and PRDs are stored. + Operate `unshackle wvd parse my_device` to inspect a device's security level + and contents, and `unshackle env info` to show where unshackle stores WVDs and PRDs. -## 5. Provide authentication +## 5. Supply authentication Services that require a login read either **cookies** or **credentials**. - **Cookies**: export the service's cookies to a Netscape-format text file and place it in your cookies directory. unshackle looks for, in order: `cookies/{SERVICE}.txt`, then `cookies/{SERVICE}/{profile}.txt`, then - `cookies/{SERVICE}/default.txt`. So a file at `cookies/EXAMPLE.txt` is picked up + `cookies/{SERVICE}/default.txt`. So unshackle finds a file at `cookies/EXAMPLE.txt` automatically for the `EXAMPLE` service. - **Credentials**: store a username and password per service in your config: @@ -143,21 +145,21 @@ depends on the service module. ## 6. Find a title (optional) -If you have a URL or ID already, skip this. Otherwise, search the service for a -title and note the `id` it prints, since that is what you feed to `dl`: +If you have a URL or ID already, skip this. Otherwise, find a title on the service +and note the `id` it prints, because that is what you feed to `dl`: ```shell unshackle search EXAMPLE "My Show" ``` -You can also list what a service exposes for a given title without downloading: +You can also show what a service gives for a given title without downloading: ```shell unshackle dl --list-titles EXAMPLE 81234567 # show seasons/episodes unshackle dl --list EXAMPLE 81234567 # show available tracks ``` -## 7. Run your first download +## 7. Do your first download A download always has three parts: @@ -175,8 +177,9 @@ A good first command asks for 1080p with English audio: unshackle dl -q 1080 -l en EXAMPLE 81234567 ``` -unshackle will fetch the title, select the tracks matching your flags, acquire -keys through your CDM (and any key vaults), then decrypt, mux, and tag the result. +unshackle will fetch the title, select the tracks that match your flags, and get +the content keys through your CDM (and any key vaults). It then decrypts, muxes, and +tags the result. ### Handy first flags @@ -208,16 +211,17 @@ codec, bitrate, channel-layout, and track-type selection. ## 8. Where the output lands -By default, unshackle writes finished files to the `downloads` directory -(`unshackle env info` shows its exact path; the built-in default is a `downloads` -folder one level above the installed `unshackle` package). Override it per run with `-o`: +By default, unshackle writes finished files to the `downloads` directory. +`unshackle env info` shows its exact path. The built-in default is a `downloads` +folder one level above the installed `unshackle` package. Override it for one +command with `-o`: ```shell unshackle dl -q 1080 -o /mnt/media/incoming EXAMPLE 81234567 ``` - unshackle writes **movies** as a single `.mkv` file named from your `movies` template. -- **TV episodes** are grouped into a per-show / per-season folder (from your +- unshackle groups **TV episodes** into a per-show / per-season folder (from your `series` template) unless you pass `--no-folder`. - unshackle builds the filename from your `output_template` and adds IMDb/TMDB/TVDB IDs to the file's metadata tags when available. @@ -233,12 +237,12 @@ directories: - **[Downloading](../guide/downloading.md)**. The full `dl` command: quality, codecs, languages, track selection, hybrid Dolby Vision, and output control. -- **[Configuration](configuration-file.md)**. Every `unshackle.yaml` key, plus +- **[Configuration](configuration-file.md)**. Every `unshackle.yaml` config key, plus directories, key vaults, proxies, and naming templates. -- **[REST API](../dev/rest-api/index.md)**. Run the `serve` server to drive downloads over - HTTP. +- **[REST API](../dev/rest-api/index.md)**. Operate the `serve` server to drive downloads + over HTTP. !!! tip "You are never far from help" `unshackle --help`, `unshackle dl --help`, and `unshackle --help` - list every option. When in doubt, add `--list` to a `dl` + show every option. When in doubt, add `--list` to a `dl` command to preview what it would do before it downloads anything. diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 45cf4240..3ac0123c 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -1,6 +1,6 @@ # CLI Reference -A complete reference for every `unshackle` command, subcommand, and flag. Commands are grouped by purpose: +A complete reference for every `unshackle` command, subcommand, and flag. This page groups the commands by purpose: - **[`dl`](#dl)**: the download pipeline (the command you use most). - **[`search`](#search)**: find titles on a service. @@ -8,11 +8,11 @@ A complete reference for every `unshackle` command, subcommand, and flag. Comman - **[`cfg`](#cfg)**, **[`env`](#env)**: manage configuration and the environment. - **[`kv`](#kv)**: Key Vault operations. - **[`wvd`](#wvd)**, **[`prd`](#prd)**: Widevine / PlayReady device management. -- **[`serve`](#serve)**: run the local CDM + REST API server. +- **[`serve`](#serve)**: operate the local CDM and REST API server. - **[`util`](#util)**: helper media utilities. !!! tip "Getting help on any command" - Every command accepts `-h`, `--help`, or `-?`. For example `unshackle dl --help` or `unshackle wvd new --help`. Service-specific arguments for `dl` and `search` (such as a title ID or search query) are defined by each service, so check `unshackle dl SERVICE --help` for those. + Every command accepts `-h`, `--help`, or `-?`. For example `unshackle dl --help` or `unshackle wvd new --help`. Each service defines its own arguments for `dl` and `search` (such as a title ID or a query), so examine `unshackle dl SERVICE --help` for those. --- @@ -24,7 +24,7 @@ unshackle [OPTIONS] COMMAND [ARGS]... > unshackle: Modular Movie, TV, and Music Archival Software. -Every invocation prints a banner and, when `update_checks` is enabled in your config, checks for a newer release. +Every invocation prints a banner and, when you enable `update_checks` in your config, checks for a newer release. | Option | Description | |---|---| @@ -45,7 +45,7 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...] ``` - `OPTIONS`: every flag below, parsed at the `dl` level. -- `SERVICE`: a service tag (e.g. `EXAMPLE1`, `EXAMPLE2`). Tags are case-insensitive and honour each service's aliases (`example+`, `EXAMPLE2`, etc. all resolve to the same tag). +- `SERVICE`: a service tag (for example `EXAMPLE1` or `EXAMPLE2`). Tags are case-insensitive and honour each service's aliases (`example+`, `EXAMPLE2`, and the other alias forms all give the same tag). - `SERVICE ARGS`: the title, URL, or ID, plus any service-specific options. These belong to the service, not to `dl`. !!! example "Typical downloads" @@ -61,7 +61,7 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...] ``` !!! note "Config-driven defaults" - Any `dl` flag can be given a default under the `dl:` section of `unshackle.yaml`, and per-service under `services..dl`. Explicit command-line values (and environment values) always win over both; service defaults only fill in options you did not set. See the [configuration file](../getting-started/configuration-file.md) guide. + You can give any `dl` flag a default under the `dl:` section of `unshackle.yaml`, and per-service under `services..dl`. Explicit command-line values (and environment values) always win over both. Service defaults only fill in options you did not set. See the [configuration file](../getting-started/configuration-file.md) guide. ### Quality, codec, bitrate & range @@ -106,7 +106,7 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...] [`subtitle.language_priority`](../reference/configuration/download.md#subtitle) in your configuration file. -The special language tokens `orig`, `all`, and `best` are honoured everywhere a language is expected. +unshackle honours the special language tokens `orig`, `all`, and `best` everywhere it expects a language. ### Title & episode selection @@ -160,21 +160,21 @@ Keep only certain track types, or skip certain track types. Attachments are alwa | `--tvdb-order` | Renumber episodes to a TVDB season order: `official` (aired), `dvd`, `absolute`, `alternate`, or `regional`. Needs `tvdb_api_key`. | !!! note "One ID at a time" - `--tmdb`, `--imdb` and `--tvdb` cannot be combined. Give one and unshackle resolves the - others from it, writing all three to the tags. `--anilist` is outside that rule and pairs + You cannot combine `--tmdb`, `--imdb` and `--tvdb`. Give one and unshackle finds the + others from it, and writes all three to the tags. `--anilist` is outside that rule and pairs with one of them, which is how you attach a western ID to an anime title. - An ID with no provider that can resolve it, such as `--tmdb` with no `tmdb_api_key`, fails - before the download starts. `--anilist` needs no key, so it never fails this check. + An ID with no metadata provider that can find it, such as `--tmdb` with no `tmdb_api_key`, + fails before the download starts. `--anilist` needs no API key, so it never fails this check. #### Episode ordering -A service does not always number a series the way TVDB's aired order does. Some services list -Futurama in TVDB's `alternate` (Streaming) order, for example. `--tvdb-order` works out which +A service does not always number a series the way TVDB's aired order does. Some services use +TVDB's `alternate` (Streaming) order for Futurama, for example. `--tvdb-order` works out which order the service used, then renumbers the episodes into the order you asked for. !!! note "Orders that do not cover the whole series" - An order can omit episodes the service carries; TVDB's `dvd` order for Futurama leaves out - the four movies. Those episodes keep their original numbering. If that would give two + An order can ignore episodes the service carries. TVDB's `dvd` order for Futurama leaves + out the four movies. Those episodes keep their original numbering. If that would give two episodes the same season/episode slot, and so the same filename, unshackle logs an error and keeps the service's numbering unchanged. Pick an order that covers the whole series. @@ -212,19 +212,19 @@ order the service used, then renumbers the episodes into the order you asked for | `--slow` | - | Inter-title delay. Bare `--slow` = 60-120s; `--slow 20-40` = custom range (minimum 20s). | !!! warning "Some flags cannot be combined" - `--require-subs` and `--s-lang`; `--select-titles` and `--wanted`; `--worst` requires `--quality`; `--vbitrate` and `--vbitrate-range` (and the audio equivalents) are mutually exclusive. + `--require-subs` and `--s-lang` are mutually exclusive. `--select-titles` and `--wanted` are mutually exclusive. `--worst` needs `--quality`. `--vbitrate` and `--vbitrate-range` (and the audio equivalents) are mutually exclusive. --- ## `search` -Search a service for titles. Like `dl`, `search` is a group whose subcommands are the installed services, and it reuses `dl`'s authentication, cookie, and proxy machinery. +Find titles on a service. Like `dl`, `search` is a group whose subcommands are the installed services, and it reuses `dl`'s authentication, cookie, and proxy machinery. ``` unshackle search [OPTIONS] SERVICE [QUERY...] ``` -The query syntax is defined per service. Results are printed as a tree of titles with their service IDs. Feed an ID straight into `dl`. +Each service defines its own query syntax. unshackle prints the results as a tree of titles with their service IDs. Feed an ID straight into `dl`. | Option | Description | |---|---| @@ -249,7 +249,7 @@ Reconstruct a download (download → decrypt → mux) from an `--export` JSON fi unshackle import EXPORT_FILE [DL_ARGS...] ``` -Any `dl` options placed after the file are forwarded verbatim, so you can override quality, range, proxy, and so on. +unshackle forwards any `dl` options after the file verbatim, so you can override quality, range, proxy, and so on. !!! example ```shell @@ -257,13 +257,13 @@ Any `dl` options placed after the file are forwarded verbatim, so you can overri unshackle import export.json -r HDR10 --proxy US ``` -The export file must be a valid v2 export (created by a current build of unshackle via `dl --export`) and must contain a `service` tag. +The export file must be a valid v2 export from a current version of unshackle, made with `dl --export`, and must contain a `service` tag. --- ## `cfg` -Read, set, delete, or list configuration values in `unshackle.yaml` without hand-editing YAML. +Read, set, delete, or show configuration values in `unshackle.yaml` without hand-editing YAML. ``` unshackle cfg [KEY] [VALUE] [OPTIONS] @@ -277,7 +277,7 @@ unshackle cfg [KEY] [VALUE] [OPTIONS] | `--list` | List all set configuration values. | !!! warning "Comments are stripped" - Writing a value through `cfg` rewrites the YAML file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error. + Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error. !!! example ```shell @@ -305,7 +305,7 @@ unshackle env COMMAND [ARGS]... unshackle env check ``` -Prints a dependency table (Category / Tool / Status / Required / Purpose) showing which external tools resolve on your `PATH`, and a summary of how many required tools are installed. +Prints a dependency table (Category / Tool / Status / Required / Purpose) that shows which external tools unshackle finds on your `PATH`. It also gives a summary of how many required tools are present. | Category | Tools | |---|---| @@ -316,7 +316,7 @@ Prints a dependency table (Category / Tool / Status / Required / Purpose) showin | Player | FFplay, MPV | | Network | HolaProxy, Caddy, Docker, git | -* required; all others are optional. +* required. All others are optional. ### `env info` @@ -324,7 +324,7 @@ Prints a dependency table (Category / Tool / Status / Required / Purpose) showin unshackle env info ``` -Shows where the config was loaded from (or lists the candidate config locations if none was found) and prints a table of every configured directory (downloads, temp, cache, cookies, logs, exports, WVDs, PRDs, services, and more). +Shows the location unshackle loaded the configuration file from. If it found no file, it shows the candidate locations instead. It then prints a table of every configured directory (downloads, temp, cache, cookies, logs, exports, WVDs, PRDs, services, and more). ### `env theme` @@ -332,11 +332,11 @@ Shows where the config was loaded from (or lists the candidate config locations unshackle env theme ``` -Prints a sample of every available CLI theme: colour swatches, help text with option rows, a track listing with video/audio/subtitle entries, log lines, a progress bar, and the gradient pulse bar. The active theme is marked and each theme's aliases are listed. Set your choice with the [`theme`](../reference/configuration/misc.md#appearance) config key. +Prints a sample of every available CLI theme. Each sample has colour swatches, help text with option rows, and a track listing with video, audio and subtitle entries. It also has log lines, a progress bar, and the gradient pulse bar. unshackle marks the active theme and shows each theme's aliases. Set your choice with the [`theme`](../reference/configuration/misc.md#appearance) config key. ### `env clear` -Clear an environment directory. The directory is emptied and recreated, and the number of files and bytes freed is reported. +Clear an environment directory. unshackle empties the directory, makes it again, and reports the number of files and bytes it freed. | Command | Description | |---|---| @@ -357,7 +357,7 @@ Clear an environment directory. The directory is emptied and recreated, and the ## `kv` -Manage Key Vaults. Vaults are configured under `key_vaults` in `unshackle.yaml`, each with a `name`, a `type` (`sqlite`, `mysql`, `http`, `api`), and type-specific options. Service tags are normalised automatically. +Manage Key Vaults. You configure vaults under `key_vaults` in `unshackle.yaml`, each with a `name`, a `type` (`sqlite`, `mysql`, `http`, `api`), and type-specific options. unshackle normalises service tags automatically. ``` unshackle kv COMMAND [ARGS]... @@ -365,7 +365,7 @@ unshackle kv COMMAND [ARGS]... ### `kv copy TO_VAULT FROM_VAULT...` -Copy keys from one or more source vaults into a destination vault. Rows with matching KIDs are skipped unless the existing row has no key; existing data is never altered or deleted. +Copy content keys from one or more source vaults into a destination vault. unshackle skips rows whose KIDs match, unless the existing row has no content key. unshackle never alters or deletes existing data. | Option | Description | |---|---| @@ -374,15 +374,15 @@ Copy keys from one or more source vaults into a destination vault. Rows with mat ### `kv sync VAULT...` -Ensure two or more vaults hold the same set of keys, essentially a chained bidirectional copy. Requires more than one vault. Accepts the same `--service` / `--local-only` options as `copy`. +Make sure that two or more vaults hold the same set of content keys, essentially a chained bidirectional copy. More than one vault is necessary. Accepts the same `--service` / `--local-only` options as `copy`. ### `kv add FILE SERVICE VAULT...` -Add content keys to one or more vaults for a service. `FILE` contains one `KID:KEY` pair per line (32 hex : 32 hex, UTF-8). Lines that don't match are skipped. +Add content keys to one or more vaults for a service. `FILE` contains one `KID:KEY` pair per line (32 hex : 32 hex, UTF-8). unshackle skips every line that does not match that form. ### `kv search KID` -Search configured vaults for a KID (32 hex characters, no dashes) and report any matching key. +Find a KID (32 hex characters, no dashes) in the configured vaults and report its content key. | Option | Description | |---|---| @@ -390,11 +390,11 @@ Search configured vaults for a KID (32 hex characters, no dashes) and report any | `-v`, `--vault` | Limit the search to a specific configured vault by name. | !!! note - Remote vaults cannot be enumerated without a service, so pass `--service` when searching them. + unshackle cannot enumerate remote vaults without a service, so pass `--service` when you examine them. ### `kv prepare VAULT...` -Create service tables on vaults that use tables, for every installed service, where they don't already exist. +Make service tables on vaults that use tables, for every installed service, where they do not already exist. !!! example ```shell @@ -452,7 +452,7 @@ unshackle prd COMMAND [ARGS]... ### `prd new PATHS...` -Create a new `.prd`. Provide either a single folder containing `zgpriv.dat` (group key) and `bgroupcert.dat` (group certificate), or two file paths (group key, then group certificate). +Make a new `.prd`. Give either a single folder that contains `zgpriv.dat` (group key) and `bgroupcert.dat` (group certificate), or two file paths (group key, then group certificate). | Option | Description | |---|---| @@ -464,11 +464,11 @@ Create a new `.prd`. Provide either a single folder containing `zgpriv.dat` (gro ### `prd reprovision PRD_PATH` -Reprovision an existing device by replacing its leaf certificate and keys. The device must support reprovisioning (version 3 or higher). Accepts the same `-e` / `-s` / `-o` options; defaults to overwriting the device in place. +Reprovision an existing device by replacing its leaf certificate and keys. Only a device of version 3 or higher can do reprovisioning. Accepts the same `-e` / `-s` / `-o` options. By default it overwrites the device in place. ### `prd test DEVICE` -Test a device against the Microsoft PlayReady demo server and print the returned keys. +Do a test of a device against the Microsoft PlayReady demo server and print the returned content keys. | Option | Default | Description | |---|---|---| @@ -509,7 +509,7 @@ unshackle serve [OPTIONS] WVD files in the WVDs directory and PRD files in the PRDs directory are auto-loaded. The REST API lives under `http://:/api/`, with Swagger UI at `/api/docs/` and a health check at `/api/health` (exempt from auth). !!! warning "Configure `api_secret` first" - Unless you pass `--no-key`, `serve.api_secret` must be set in your config. Requests authenticate with the `X-Secret-Key` header. Running with `--no-key` disables authentication entirely and should only be used on a trusted, private network. + Unless you pass `--no-key`, `serve.api_secret` must be set in your config. Requests authenticate with the `X-Secret-Key` header. `--no-key` disables authentication entirely. Only use it on a trusted, private network. ```yaml title="unshackle.yaml" serve: @@ -534,7 +534,7 @@ WVD files in the WVDs directory and PRD files in the PRDs directory are auto-loa ## `util` -Various helper media utilities. When a command is given a directory, it processes every `.mkv`/`.mp4` inside it in natural (S01E01 before S01E10) order. +Various helper media utilities. When you give a command a directory, it processes every `.mkv`/`.mp4` inside it in natural (S01E01 before S01E10) order. ``` unshackle util COMMAND [ARGS]... @@ -569,11 +569,11 @@ Losslessly set the video range flag to full or limited at the bitstream level. ### `util test PATH` -Decode an entire video with FFmpeg and report any corruptions or errors. All streams are tested by default; subtitles cannot be tested. +Decode an entire video with FFmpeg and report any corruptions or errors. By default it does a test of every track. It cannot do a test of subtitles. | Option | Default | Description | |---|---|---| -| `-m`, `--map` | `0` | Test specific streams via FFmpeg's `-map`, e.g. `0:v:0` or `0:a`. | +| `-m`, `--map` | `0` | Do a test of specific tracks with FFmpeg's `-map`, for example `0:v:0` or `0:a`. | !!! example ```shell diff --git a/docs/guide/downloading.md b/docs/guide/downloading.md index 0d9e13f4..3e6b6a9c 100644 --- a/docs/guide/downloading.md +++ b/docs/guide/downloading.md @@ -9,23 +9,23 @@ the download flow itself. !!! note "Where the full flag list lives" This page covers the flags you will use most and explains how they interact. It is not an exhaustive enumeration. For the complete, always-current list of every option - and its exact wording, run `unshackle dl --help`. Every flag shown here can also be + and its exact wording, use `unshackle dl --help`. Every flag shown here can also be given a default in your configuration file. See [Configuration file](../getting-started/configuration-file.md). ## Command shape -A download is always three layers: the root CLI, the `dl` command (which is where all -the flags below are parsed), and a **service tag** subcommand that carries the title -argument. +A download is always three layers: the root CLI, the `dl` command (which is where +unshackle parses all the flags below), and a **service tag** subcommand that carries the +title argument. ```shell unshackle dl [OPTIONS] SERVICE [SERVICE ARGUMENTS] ``` - `unshackle`: the root command. -- `dl`: the download command. **All of the options on this page are parsed here**, so - they go *before* the service tag. +- `dl`: the download command. **unshackle parses all of the options on this page here**, + so they go *before* the service tag. - `SERVICE`: a service tag such as `EXAMPLE1`, `EXAMPLE2`, `EXAMPLE3`. Service tags are case-insensitive and honour each service's aliases (for example `example+` resolves to its real tag). @@ -41,15 +41,15 @@ unshackle dl -q 1080 -v H.265 -r HDR10 --lang en EXAMPLE 'https://www.example.co ``` !!! tip "See what a service accepts" - Because the positional argument is defined by each service, run - `unshackle dl SERVICE --help` (e.g. `unshackle dl EXAMPLE --help`) to see that service's - own argument and any service-specific options. + Because each service defines its own positional argument, use + `unshackle dl SERVICE --help` (for example `unshackle dl EXAMPLE --help`) to see that + service's own argument and any service-specific options. ## The download flow -When you run a download, `dl` moves through the pipeline below. Understanding the order -helps explain why, for example, `--list` stops early and why keys are fetched even with -`--skip-dl`. +When you start a download, `dl` moves through the pipeline below. The order tells you +why, for example, `--list` stops early, and why unshackle fetches the content keys even +with `--skip-dl`. 1. **Setup**: load the DRM CDM, key vaults, proxy providers, cookies, and credentials. 2. **Authenticate**: sign in to the service with your profile's cookies/credentials. @@ -60,7 +60,7 @@ helps explain why, for example, `--list` stops early and why keys are fetched ev 6. **Select tracks**: narrow each type down using your quality, codec, range, language, and bitrate flags. 7. **Download**: pull segments for the selected tracks (concurrently per `--downloads`). -8. **License & decrypt**: acquire content keys (vault → CDM), decrypt each track. +8. **License & decrypt**: get the content keys (vault → CDM), decrypt each track. 9. **Post-process**: extract closed captions, convert subtitles, repack, mux with `mkvmerge`, and move the finished file to your downloads directory. @@ -84,8 +84,8 @@ unshackle dl -q 2160,1080,720 EXAMPLE 81234567 ``` !!! note "16:9 canvas matching" - Resolution matching is by track **height** first. If no track has an exact matching - height, unshackle falls back to a 16:9-canvas match, so `-q 1080` will also match an + Resolution matching is by track **height** first. If no track height matches exactly, + unshackle falls back to a 16:9-canvas match, so `-q 1080` also selects an anamorphic `1920×804` track (computed as `int(width × 9 / 16)`). ### Best-available and worst @@ -94,18 +94,18 @@ By default a missing requested resolution is an error. Two flags change that: | Flag | Behaviour | | --- | --- | -| `--best-available` | If the requested resolution(s) aren't present, continue with the best that *is* available instead of failing. Also softens missing video/audio/subtitle languages and hybrid fallbacks. | +| `--best-available` | If the requested resolution(s) are not present, continue with the best that *is* available instead of failing. Also softens missing video/audio/subtitle languages and hybrid fallbacks. | | `--worst` | Within the specified quality, pick the **lowest** bitrate rendition. **Requires `-q/--quality`.** | ```shell title="Never fail on a missing resolution" unshackle dl -q 2160 --best-available EXAMPLE 0ABC123 ``` -## Video codec and color range +## Video codec and colour range ### Codec -`-v` / `--vcodec` selects one or more video codecs; the default is any codec. It accepts +`-v` / `--vcodec` selects one or more video codecs. The default is any codec. It accepts either enum **names** or their **values**, comma-separated. | Name | Value | @@ -123,9 +123,9 @@ unshackle dl -v H.265 EXAMPLE 81234567 unshackle dl -v hevc,avc EXAMPLE 81234567 ``` -### Color range +### Colour range -`-r` / `--range` selects one or more color ranges; the default is `SDR`. +`-r` / `--range` selects one or more colour ranges. The default is `SDR`. | Range | Meaning | | --- | --- | @@ -141,13 +141,13 @@ unshackle dl -q 2160 -r HDR10,DV EXAMPLE '...' ``` !!! warning "HYBRID requires dovi_tool" - `-r HYBRID` produces a single hybrid stream by injecting the Dolby Vision RPU - metadata onto an HDR10/HDR10+ base layer using **dovi_tool**. + `-r HYBRID` makes a single hybrid track by injecting the Dolby Vision RPU + metadata onto an HDR10/HDR10+ base layer with **dovi_tool**. It requires the `dovi_tool` binary, resolved from unshackle's `binaries/` folder or your `PATH`. The normal case is a DV track plus an HDR10 or HDR10+ base. With HDR10+ and no DV, unshackle converts the HDR10+ metadata to DV instead, which also needs `hdr10plus_tool`. With no DV and no HDR10+, the title fails. - When HDR10+ is present it is preferred over HDR10 as the base layer. + When HDR10+ is present, unshackle prefers it over HDR10 as the base layer. To keep an HDR10+ deliverable *alongside* the hybrid, request both ranges: `-r HYBRID,HDR10P`. `-r HYBRID` on its own muxes only the merged hybrid. @@ -165,13 +165,13 @@ resolution/codec/range/language combination. You can constrain this. | `-ab-range` / `--abitrate-range` | Audio bitrate range in kbps, e.g. `128-256`. | !!! warning "Exact vs range are mutually exclusive" - `--vbitrate` cannot be combined with `--vbitrate-range`, and likewise for the audio + You cannot combine `--vbitrate` with `--vbitrate-range`, and likewise for the audio pair. Pick one form. ### Real bitrate probing Manifest-declared bitrates are sometimes rounded or wrong. Services often advertise a -**peak** or **nominal** bandwidth that is far from the stream's real average. This matters +**peak** or **nominal** bandwidth that is far from the track's real average. This matters because a track's bitrate drives the track listing, the sort order, *and* the `--vbitrate`/`--vbitrate-range` selection above. A bogus declared value therefore makes unshackle pick the wrong track. These flags probe the actual media size to compute a true @@ -182,26 +182,26 @@ bitrate for the top renditions, overriding the manifest value: language). Slower than the video variant because there are more renditions to probe. !!! note "Reading the probed numbers" - A single-file track (DASH `SegmentBase`/`BaseURL`) is measured **exactly**. A + unshackle measures a single-file track (DASH `SegmentBase`/`BaseURL`) **exactly**. A multi-segment track (most HLS) is a **sampled estimate**, normally within a few percent of the true value. For MPEG-TS HLS the probed figure also reads a few percent *above* the demuxed elementary stream, because the segment bytes include container overhead. That is the real *delivered* size, not an over-report or a bug. !!! tip "Why not every rendition is probed" - Probing does not touch every rendition. Only the five highest declared-bitrate - renditions of each quality tier are probed in parallel (video grouped by codec and - range, audio by codec, channels, language, and descriptive flag), and unshackle - extends a group downward while a lower unprobed rendition could still outrank a - probed one. This keeps probing fast even when a service exposes dozens of renditions. - Tracks whose duration can't be determined fall back to `ffprobe`, and a probe failure - is non-fatal: the manifest value is simply left in place. + Probing does not touch every rendition. unshackle probes only the five highest + declared-bitrate renditions of each quality tier, in parallel (video grouped by codec + and range, audio by codec, channels, language, and descriptive flag). It extends a + group downward while a lower unprobed rendition could still outrank a probed one. + This keeps probing fast even when a service exposes dozens of renditions. A track + whose duration unshackle cannot find falls back to `ffprobe`. A probe failure is + non-fatal: the manifest value stays in place. ## Audio codec, channels, and Atmos ### Codec -`-a` / `--acodec` selects one or more audio codecs (comma-separated); the default is any. +`-a` / `--acodec` selects one or more audio codecs (comma-separated). The default is any. | Name | Value | Codec | | --- | --- | --- | @@ -215,8 +215,8 @@ bitrate for the top renditions, overriding the manifest value: | `ALAC` | `ALAC` | Apple Lossless | | `FLAC` | `FLAC` | FLAC | -Names, values, and a few aliases are accepted: `eac3` and `ddp` both resolve to `EC3`, -and `vorbis` resolves to `OGG`. +unshackle accepts names, values, and a few aliases: `eac3` and `ddp` both give `EC3`, +and `vorbis` gives `OGG`. ```shell title="Prefer Dolby Digital Plus, fall back to AAC" unshackle dl -a EC3,AAC EXAMPLE 81234567 @@ -224,9 +224,9 @@ unshackle dl -a EC3,AAC EXAMPLE 81234567 ### Channels and Atmos -- `-c` / `--channels`: desired channel layout, e.g. `5.1` or `2.0`. Matching is by +- `-c` / `--channels`: desired channel layout, for example `5.1` or `2.0`. Matching is by ceiling, so `5.1` implicitly matches a `6.0`-reported layout. -- `-naa` / `--noatmos`: exclude Dolby Atmos audio tracks from selection. +- `-naa` / `--noatmos`: exclude Atmos audio tracks from selection. ```shell title="5.1 audio, no Atmos" unshackle dl -c 5.1 --noatmos EXAMPLE 0ABC123 @@ -250,7 +250,7 @@ unshackle dl -a AAC -ab 320 EXAMPLE 81234567 !!! tip "Two FLAC tracks in one release" A service can offer both a CD FLAC and a hi-res FLAC. The two have the same codec, so - the bitrate is what separates them. Run `--list` to read the bitrates, then give `-ab` + the bitrate is what separates them. Use `--list` to read the bitrates, then give `-ab` the one you want. ## Languages @@ -264,8 +264,8 @@ unshackle dl -a AAC -ab 320 EXAMPLE 81234567 unshackle dl -l orig,en EXAMPLE 81234567 ``` -The special token `orig` is resolved to the title's actual original language everywhere -it is used. You can override each stream type independently: +unshackle changes the special token `orig` to the title's own original language +everywhere you use it. You can override each track type independently: - `-vl` / `--v-lang`: language for **video only** (overrides `-l` for video). Useful when the burned-in video language differs from the audio you want. @@ -284,8 +284,8 @@ Put a `-` in front of a language to remove it from the selection. This works on unshackle dl -sl all,-es EXAMPLE 81234567 ``` -The exclusions are subtracted from the languages the flag would otherwise select, and the -order of the values does not matter. If you give only exclusions, the flag keeps its own +unshackle subtracts the exclusions from the languages the flag would otherwise select, +and the order of the values does not matter. If you give only exclusions, the flag keeps its own default: `-sl -es` is the same as `-sl all,-es`, and `-l -es` is the same as `-l orig,-es`. ```shell title="Only exclusions" @@ -297,7 +297,7 @@ unshackle dl -l -es EXAMPLE 81234567 ``` `-vl` and `-al` have no default of their own: they cascade to `-l`. An override that names -a language replaces `-l` for that stream type, exclusions included. An override that gives +a language replaces `-l` for that track type, exclusions included. An override that gives only exclusions adds them to the ones from `-l`. ```shell title="Cascade" @@ -306,13 +306,13 @@ unshackle dl -l en,-es -vl ja EXAMPLE 81234567 ``` !!! note "Rules for exclusion tokens" - - `orig` is resolved per title, so `-sl -orig` drops the subtitles in the title's - original language. + - `orig` becomes the title's own original language, so `-sl -orig` drops the + subtitles in that language. - `--exact-lang` applies to exclusions with the same matching rules as selection: `-es` removes exactly the tracks that `es` would select. Without it, `-es` also removes `es-419` and `es-ES`. - Tracks with no language tag are never excluded. - - `-all` is rejected. Name the languages you do not want instead. + - unshackle rejects `-all`. Name the languages you do not want instead. - The comma form is the documented style. `-sl -es` also works, but a shell can read a lone `-es` as another option. @@ -330,17 +330,17 @@ subtitle: ``` Subtitles then start with English, Spanish, and French. The languages you leave out follow -alphabetically, after the title's original language. Nothing is removed. Audio works the -same way, but the languages you leave out keep their bitrate and codec order instead of an +alphabetically, after the title's original language. unshackle removes nothing. Audio +works the same way, but the languages you leave out keep their bitrate and codec order instead of an alphabetical one. See [`subtitle.language_priority`](../reference/configuration/download.md#subtitle) for the details. ### Exact vs fuzzy matching -By default language matching is fuzzy: `-l en` also accepts `en-US`, `en-GB`, etc. (up -to a small distance). Pass `--exact-lang` to require an exact match: with it, `-l es-419` -matches only `es-419`, not `es-ES`. +By default language matching is fuzzy: `-l en` also accepts `en-US`, `en-GB`, and the +other regional forms (up to a small distance). Pass `--exact-lang` when only an exact +match is acceptable: with it, `-l es-419` selects only `es-419`, not `es-ES`. !!! tip "The `all` and `best` tokens" For audio, the tokens `all` and `best` bypass the usual one-track-per-language pick @@ -351,8 +351,8 @@ matches only `es-419`, not `es-ES`. ### Selecting subtitle languages -`-sl` / `--s-lang` sets the wanted subtitle language(s). The default is `all`, meaning -every available subtitle language is downloaded. +`-sl` / `--s-lang` sets the wanted subtitle language(s). The default is `all`, so +unshackle downloads every available subtitle language. ```shell title="Only English and Spanish subtitles" unshackle dl -sl en,es EXAMPLE 81234567 @@ -364,9 +364,9 @@ except Spanish, forced Spanish subtitles included. ### Requiring subtitles -`--require-subs` takes a list of languages that **must** exist. If they all exist, *all* -subtitles are downloaded; if any is missing, the title fails. This is useful for gating a -download on the presence of a specific subtitle track. +`--require-subs` takes a list of languages that **must** exist. If they all exist, +unshackle downloads *all* subtitles. If any is missing, the title fails. This is useful +when you want a download only if a specific subtitle track is present. !!! warning "Cannot combine with `--s-lang`" `--require-subs` and `--s-lang` are mutually exclusive. Use one or the other. @@ -374,14 +374,14 @@ download on the presence of a specific subtitle track. ### Forced subtitles and output format - `-fs` / `--forced-subs`: include forced subtitle tracks (signs/foreign dialogue). - Without this flag, forced subs are dropped from selection. + Without this flag, unshackle drops forced subtitle tracks from the selection. - `-fsl` / `--forced-s-lang`: keep forced subtitles only in these languages (implies `-fs`). Works independently of `--s-lang`, so `-sl all -fsl en` grabs every full subtitle but only the English forced track. It accepts exclusions too: `-fsl all,-es` keeps every forced subtitle except the Spanish one, and `-fsl -es` means the same. - `--sub-format`: set the output subtitle format, converting only when necessary. Accepts codec names/values and common aliases (`srt`, `vtt`, `ass`, `ssa`, `ttml`, - etc.), or the literal `original` to keep the source format. + and the other codec aliases), or the literal `original` to keep the source format. | Value | Format | | --- | --- | @@ -397,15 +397,16 @@ unshackle dl --sub-format srt EXAMPLE 81234567 ``` !!! note "SDH stripping happens by default" - When a subtitle track is marked SDH (for the deaf/hard-of-hearing) and there is no - plain same-language subtitle, unshackle produces a stripped, non-SDH variant - automatically. This behaviour is governed by the `subtitle.strip_sdh` config option + When a subtitle track carries the SDH flag (for the deaf/hard-of-hearing) and there + is no plain same-language subtitle, unshackle makes a stripped, non-SDH track + automatically. The `subtitle.strip_sdh` config option controls this behaviour (default on). ## Selecting episodes and tracks -For series, several flags control which episodes are downloaded. The same flags select the -tracks of a music release. By default, **all** episodes and **all** tracks are downloaded. +For series, several flags control which episodes unshackle downloads. The same flags +select the tracks of a music release. By default, unshackle downloads **all** episodes and +**all** tracks. ### Wanted ranges @@ -508,10 +509,11 @@ track on disc 1, which is the number the track list shows for a single-disc rele ### Daily and date-based content Talk shows, news and sports have no official episode numbering, so unshackle names them -by air date. An episode that carries an air date is written as `Show.YYYY.MM.DD` instead of -`SxxExx`, the `{date}` token holds the ISO date, and the season folder becomes the year. +by air date. unshackle writes an episode that carries an air date as `Show.YYYY.MM.DD` +instead of `SxxExx`, the `{date}` token holds the ISO date, and the season folder becomes +the year. -A service that only carries this kind of content sets `DAILY = True` on its class, and a +A service that only carries this kind of title sets `DAILY = True` on its class, and a service can set `air_date` on each episode itself. Add `--daily` to mark any other title as date-based: @@ -520,14 +522,14 @@ unshackle dl --daily --tvdb 73871 --enrich EXAMPLE 81234567 ``` With `--enrich` and a TVDB ID, `--daily` fills in the air date of every episode that has -none. An air date the service already set is kept. Dates before 1970 and dates in the -future are skipped, because TVDB carries placeholder schedule dates for episodes that +none. An air date the service already set is kept. unshackle skips dates before 1970 and +dates in the future, because TVDB carries placeholder schedule dates for episodes that have not aired. Without `--enrich` unshackle has no source to fill from, and says so. -A dated episode answers to its air date in `-w`, as well as to its `SxxExx` key. A date +A dated episode answers to its air date in `-w`, as well as to its `SxxExx` token. A date token is a plain ISO date (`2026-08-11`). A date range uses a colon (`2026-08-01:2026-08-31`), because the dashes in a date are part of the date. A range cannot span more than 1000 days. -Date tokens and `SxxExx` tokens can be mixed in one `-w`. +You can mix date tokens and `SxxExx` tokens in one `-w`. ### Split episodes @@ -545,14 +547,14 @@ A season token such as `S01` covers every episode and every part, so you only ne when you want a part on its own. !!! warning "A part range stays inside one episode" - `S01E01.1-S01E01.3` is valid. `S01E01.1-S01E02.3` is rejected, because the parser - cannot know how many parts episode 1 has and so cannot work out where the range ends. - To span episodes, list the parts you want as separate tokens: + `S01E01.1-S01E01.3` is valid. unshackle rejects `S01E01.1-S01E02.3`, because the + parser cannot know how many parts episode 1 has and so cannot work out where the range + ends. To span episodes, write the parts you want as separate tokens: `-w S01E01.2,S01E02.1`. !!! note "A part of an unsplit episode selects nothing" `-w S01E02.2` on an episode that was never split is deliberately empty: it tells you - the episode has no parts, rather than quietly handing you the whole episode. Run + the episode has no parts, rather than quietly handing you the whole episode. Use `--list-titles` to see which episodes carry parts. ### Other selection flags @@ -589,7 +591,7 @@ or negatively (everything but these). Additional track-type flags: - `-ad` / `--audio-description`: include descriptive (audio-description) tracks, which - are dropped by default. + unshackle drops by default. - `--skip-subtitle-errors`: if a subtitle fails to download, skip it and continue rather than aborting the whole title. Video and audio failures remain fatal. @@ -603,7 +605,7 @@ unshackle dl -nc EXAMPLE 81234567 !!! note You can combine an `*-only` flag with `no-*` flags to fine-tune: `*-only` chooses the - starting set of categories, then `no-*` subtracts from it. Attachments (e.g. fonts) + starting set of categories, then `no-*` subtracts from it. Attachments (for example fonts) are always kept. ## Listing and dry runs @@ -624,9 +626,9 @@ unshackle dl -q 1080 -v H.265 -r HDR10 --list EXAMPLE 81234567 ### Output location and folders -- `-o` / `--output`: override the output directory for this run (otherwise the - configured downloads directory is used). -- `--no-folder`: do not create a per-show folder for TV downloads. +- `-o` / `--output`: override the output directory for this run (otherwise unshackle + uses the configured downloads directory). +- `--no-folder`: do not make a per-show folder for TV downloads. - `--no-source`: remove the service source tag from the filename and path. ```shell title="Send this download somewhere specific" @@ -635,8 +637,8 @@ unshackle dl -o ~/Videos/incoming EXAMPLE 81234567 ### Muxing behaviour -By default, the tracks of a movie or an episode are muxed into a single Matroska (`.mkv`) -file with `mkvmerge`. These flags change how the output is assembled: +By default, `mkvmerge` muxes the tracks of a movie or an episode into a single Matroska +(`.mkv`) file. These flags change how unshackle assembles the output: | Flag | Behaviour | Default source | | --- | --- | --- | @@ -645,7 +647,7 @@ file with `mkvmerge`. These flags change how the output is assembled: | `--merge-video` | Mux video tracks that share a height, range, and codec into one file, so only language varies inside a file. | config `muxing.merge_video` (off) | !!! note "Music is never muxed" - A music track skips the muxer. unshackle keeps the container that the service + A song skips the muxer. unshackle keeps the container that the service delivered, such as `.flac`, `.m4a`, or `.mp3`, and writes the metadata into that file. Read [Music output files](output-and-naming.md#music-output-files). @@ -653,14 +655,14 @@ file with `mkvmerge`. These flags change how the output is assembled: `--postscript ""` runs your own command once per output file, with unshackle's metadata substituted into `{variable}` placeholders. It is repeatable, and it replaces the -`post_scripts` config for that run. No `success` hook runs under `--no-mux`, because that -run writes no muxed output. A `failure` hook still runs if the download itself fails. +`post_scripts` config for that run. No `success` hook operates under `--no-mux`, because +that run writes no muxed output. A `failure` hook still operates if the download fails. ```shell title="Hand each finished file to an uploader" unshackle dl --postscript "python /opt/upload.py {filepath} --service={service}" EXAMPLE 81234567 ``` -For hooks that persist across runs, for the `season` and `run` modes, for `failure` hooks +For hooks that continue across runs, for the `season` and `run` modes, for `failure` hooks and for the full variable list, see [Post-download scripts](../reference/configuration/post-scripts.md). @@ -688,8 +690,8 @@ Two related flags: - `--no-proxy`: force-disable all proxy use for this run. - `--no-proxy-download`: bypass the proxy for **all downloads**. The manifest, - license, and authentication requests still go through the proxy. This is useful when the - proxy is only needed to satisfy geo-checks, not to move the bulk of the data. + license, and authentication requests still go through the proxy. This is useful when you + need the proxy only to satisfy geo-checks, not to move the bulk of the data. ## Performance and caching @@ -715,10 +717,11 @@ unshackle dl -w S01 --slow 30-60 EXAMPLE 81234567 ## Keys, vaults, and export By default, unshackle checks your **key vaults** first and only asks a **CDM** to license -a key when the vault misses. You can force one side or the other: +a content key when the vault misses. You can force one side or the other: - `--cdm-only`: only use the CDM (skip vaults). -- `--vaults-only`: only use key vaults (never license via the CDM); a missing key fails. +- `--vaults-only`: only use key vaults and never license through the CDM. A missing + content key fails. ```shell title="Fetch keys only, no download" unshackle dl --skip-dl EXAMPLE 81234567 @@ -733,8 +736,8 @@ unshackle dl --skip-dl --export EXAMPLE 81234567 ``` !!! note "Region is recorded only with a proxy" - When `--proxy` is used, the export records the region so an import can reproduce the - correct geofence. Without a proxy, no region is stored. + When you use `--proxy`, the export records the region so an import can reproduce the + correct geofence. Without a proxy, the export stores no region. !!! warning "Some DASH and Smooth exports need a title language" An import re-fetches the DASH or ISM manifest and parses it again. Most manifests label @@ -749,7 +752,7 @@ unshackle looks up metadata automatically, but you can override the identifiers tagging and naming. An ID you give is authoritative: unshackle looks it up directly instead of searching by title, and the ID stays in the tags. An ID only settles *which* title this is. To also take that source's title, year and original language, add `--enrich`. Give at most one -of `--tmdb`, `--imdb` and `--tvdb`; unshackle works the rest out from it. To stop the automatic +of `--tmdb`, `--imdb` and `--tvdb`, and unshackle works the rest out from it. To stop the automatic lookups but keep the IDs you give, set [`disable_metadata`](../reference/configuration/misc.md#external-api-keys) in your config. @@ -772,8 +775,9 @@ unshackle dl --imdb tt1375666 --enrich EXAMPLE 81234567 ### What each metadata provider supplies -Providers differ in what they answer with, so the fields `--enrich` can replace depend on -which ID you gave. Every provider except `imdb` only runs when its key is configured: +Metadata providers differ in what they answer with, so the fields `--enrich` can replace +depend on which ID you gave. Every metadata provider except `imdb` and `anilist` only +operates when you set its config key: | Provider | Config key | Title and year | Original language | External IDs it returns | | --- | --- | --- | --- | --- | @@ -788,11 +792,11 @@ Whatever the tag looks like, unshackle normalises it before use, so `ko`, `kor` all end up as the same language. !!! note "AniList only answers for anime" - It is last in the default provider order and returns nothing for a title that is not - anime. The only cost is one search that misses. Put `anilist` earlier in + It is last in the default metadata provider order and returns nothing for a title + that is not anime. The only cost is one search that misses. Put `anilist` earlier in [`metadata_providers`](../reference/configuration/misc.md#external-api-keys) if you mostly - download anime. Which of the three AniList title variants is used is set by - [`anilist_title_language`](../reference/configuration/misc.md#external-api-keys). + download anime. [`anilist_title_language`](../reference/configuration/misc.md#external-api-keys) + sets which of the three AniList title variants unshackle uses. A service can also mark its titles as anime. For those titles unshackle tries AniList first. When AniList has no match, it falls through to the normal order. @@ -801,31 +805,33 @@ Each ID goes to the providers that read that kind of ID, in [`metadata_providers`](../reference/configuration/misc.md#external-api-keys) order: - `--tmdb` reads TMDB. -- `--imdb` reads IMDb, then falls back to OMDb. IMDb needs no key, so this path works out of - the box, and an OMDb key only matters when IMDb has no answer. +- `--imdb` reads IMDb, then falls back to OMDb. IMDb needs no API key, so this path works + out of the box, and an OMDb API key only matters when IMDb has no answer. - `--tvdb` reads TVDB. It needs `tvdb_api_key`. -- `--anilist` reads AniList, which needs no key. It supplies the title, the year and the +- `--anilist` reads AniList, which needs no API key. It supplies the title, the year and the original language, and writes an AniList tag. It knows no TMDB, IMDB or TVDB ID, so pair it with `--tmdb`, `--imdb` or `--tvdb` when you want those tags as well. -Give **one** of `--tmdb`, `--imdb` and `--tvdb`. They cannot be combined, because one ID does -the job on its own: unshackle resolves the other two from it and writes all three to the tags. +Give **one** of `--tmdb`, `--imdb` and `--tvdb`. You cannot combine them, because one ID +does the job on its own: unshackle finds the other two from it and writes all three to the +tags. Passing two is an error rather than a silent choice between them. `--anilist` is the exception and still pairs with one of the three, which is how you tag an anime title with a western ID. -The ID you give must have a provider that can actually resolve it, so unshackle checks that -before it downloads anything. `--tmdb` without `tmdb_api_key`, or an ID whose providers your -`metadata_providers` list leaves out, fails immediately with a message naming what to set. -`--imdb` needs no key, since the `imdb` provider is keyless and in the default order. +The ID you give must have a metadata provider that can find it, so unshackle makes sure of +that before it downloads anything. `--tmdb` without `tmdb_api_key`, or an ID whose metadata +providers your `metadata_providers` list leaves out, fails immediately with a message naming +what to set. `--imdb` needs no API key, because the `imdb` metadata provider is keyless +and in the default order. With `--enrich`, the title, year and original language are all replaced with the external source's values, whether or not the service already filled them in. A field the source does -not answer with is left alone, and unshackle logs which fields those were, so a provider with -a thin record cannot blank out what the service told you. +not answer with is left alone, and unshackle logs which fields those were, so a metadata provider +with a thin record cannot blank out what the service told you. `--enrich` also fills in the absolute episode number of each episode from TVDB's absolute order, for any series that has one and where the service did not supply it. Anime is the -usual beneficiary, but no part of this is limited to anime. This only adds the +usual beneficiary, but nothing here applies only to anime. This only adds the [`{absolute}`](../reference/configuration/output.md#output_template) naming variable. The season and episode numbers are never changed. @@ -836,12 +842,13 @@ With `--daily`, `--enrich` also fills in the air date of each episode from TVDB. Track selection reads the original language, so replacing it changes which audio is treated as the original. That is the point when a service mislabels it, but it means a wrong ID can pick the wrong audio track and not only write a wrong name. Without - `--enrich` the service's own value is kept and track selection is untouched. + `--enrich`, unshackle keeps the service's own value and does not change track + selection. ### Episode ordering -A service does not always number a series the way TVDB's aired order does. Some services list -Futurama in TVDB's `alternate` (Streaming) order, for example, while TVDB's `official` order +A service does not always number a series the way TVDB's aired order does. Some services use +TVDB's `alternate` (Streaming) order for Futurama, for example, while TVDB's `official` order holds back four season-one episodes to the start of season two. `--tvdb-order` works out which order the service used, then renumbers the episodes into the order you asked for: @@ -853,26 +860,26 @@ Available orders are `official` (aired), `dvd`, `absolute`, `alternate`, and `re `tvdb_order` in your config to apply one by default. !!! warning "Orders that do not cover the whole series" - An order can leave out episodes the service carries. TVDB's `dvd` order does not list - Futurama's four movies. Those episodes keep their original numbering. If that would give + An order can leave out episodes the service carries. TVDB's `dvd` order does not + contain Futurama's four movies. Those episodes keep their original numbering. If that would give two episodes the same season/episode slot, and so the same filename, unshackle logs an error and leaves the numbering untouched. Pick an order that covers the whole series. ## Configuration defaults -Every flag on this page can be set as a default in your configuration file so you don't -have to type it each time, and defaults can be scoped per service. See +You can set every flag on this page as a default in your configuration file so you do not +have to type it each time, and you can scope defaults per service. See [Configuration file](../getting-started/configuration-file.md) for the `dl:` block and per-service overrides. !!! tip "Explicit flags always win" - A value you pass on the command line (or via an environment variable) always takes + A value you pass on the command line (or through an environment variable) always takes precedence over both the global `dl:` defaults and any per-service `dl:` overrides. - Config only fills in options you didn't set explicitly. + Config only fills in options you did not set explicitly. ## Quick reference -A condensed lookup of the most-used options. Run `unshackle dl --help` for the complete, +A condensed lookup of the most-used options. Use `unshackle dl --help` for the complete, authoritative list. | Flag | Short | Meaning | @@ -880,7 +887,7 @@ authoritative list. | `--quality` | `-q` | Target resolution(s), e.g. `1080,720`. | | `--vcodec` | `-v` | Video codec(s). | | `--acodec` | `-a` | Audio codec(s). | -| `--range` | `-r` | Color range(s); default `SDR`. | +| `--range` | `-r` | Colour range(s). Default `SDR`. | | `--channels` | `-c` | Audio channel layout. | | `--noatmos` | `-naa` | Exclude Atmos audio. | | `--lang` | `-l` | Video + audio language(s); default `orig`. `-` excludes, e.g. `all,-es`. | diff --git a/docs/guide/drm-and-cdm.md b/docs/guide/drm-and-cdm.md index 27824cf8..5f7328fa 100644 --- a/docs/guide/drm-and-cdm.md +++ b/docs/guide/drm-and-cdm.md @@ -1,16 +1,16 @@ # DRM & CDM Setup -Most commercial streaming content is protected by a Digital Rights Management (DRM) -system. To download and play it back, unshackle needs to obtain the **content keys** +A Digital Rights Management (DRM) system protects most commercial streaming titles. +To download a title and play it back, unshackle needs to get the **content keys** for the encrypted tracks and then decrypt them. This page explains the DRM systems -unshackle supports, how to provide a device so it can talk to a license server, the +unshackle supports, how to install a device so it can talk to a license server, the difference between local and remote CDMs, and how to choose a decryption backend. !!! note "Who this page is for" This is a practical configuration guide for end users of the unshackle CLI. - A few sections are marked **Developer**. They describe internal behaviour that is - useful when writing a service, but you do not need them to download protected - content. + A few sections have a **Developer** mark. They give internal behaviour that is + useful when you write a service, but you do not need them to download a protected + title. ## Key concepts @@ -18,8 +18,8 @@ Before configuring anything, it helps to know the vocabulary. unshackle draws a line between two things: - A **CDM** (Content Decryption Module) is the trusted "black box" that turns a - *license challenge* into *content keys*. It is backed by a **device**, a file - containing the cryptographic identity of a real client (a phone, a browser, etc.). + *license challenge* into *content keys*. A **device** backs it: a file that carries + the cryptographic identity of a real client, such as a phone or a browser. - A **DRM system** is the wrapper around a CDM that parses the protection metadata, drives the challenge → license → keys handshake, and finally invokes an external tool to physically decrypt the downloaded file. @@ -42,7 +42,7 @@ then decrypts the file with those keys. ## Supported DRM systems unshackle implements five DRM systems. Only the first two need a device file and a -license-server handshake; the rest are special cases. +license-server handshake. The rest are special cases. | System | Device file | License flow | Decryption | |--------|-------------|--------------|------------| @@ -54,10 +54,10 @@ license-server handshake; the rest are special cases. !!! warning "Two different ClearKeys" "ClearKey (HLS)" and "ClearKey (CENC)" are entirely different systems that happen - to share a name. The HLS variant is the AES-128 key referenced by an `EXT-X-KEY` - tag and is decrypted in-process. The CENC variant is the W3C EME - `org.w3.clearkey` scheme, delivers keys as a JWK Set, and is decrypted through - shaka-packager or mp4decrypt just like Widevine. Neither involves a device file. + to share a name. The HLS variant is the AES-128 content key that an `EXT-X-KEY` + tag references, and unshackle decrypts it in-process. The CENC variant is the W3C + EME `org.w3.clearkey` scheme. It delivers keys as a JWK Set, and shaka-packager or + mp4decrypt decrypts it like Widevine. Neither involves a device file. ### Widevine vs PlayReady @@ -65,20 +65,20 @@ These are the two "real" DRMs, and in unshackle they behave almost identically: same PSSH/KID handling, the same decryption backends, and the same key-caching flow. The differences that matter to you are: -- **Device file**: Widevine uses a `.wvd`; PlayReady uses a `.prd`. +- **Device file**: Widevine uses a `.wvd`. PlayReady uses a `.prd`. - **Availability**: some services offer only one system, some offer both. When a manifest carries both, unshackle picks the appropriate CDM for each track (see [Widevine and PlayReady on the same title](#widevine-and-playready-on-the-same-title)). -To download a Widevine-protected title you need a `.wvd` device; for PlayReady you +To download a Widevine-protected title you need a `.wvd` device. For PlayReady you need a `.prd` device. You can hold both, and unshackle automatically selects the appropriate one for each track. !!! info "Obtaining device files" A `.wvd` or `.prd` encodes the private keys of a real client device. unshackle - does **not** ship with one and cannot generate the underlying secret material for + does **not** ship with one and cannot make the underlying secret material for you. You must supply your own. The `wvd` and `prd` commands below manage and - provision device files once you have the raw key material. + provision device files once you have the raw cryptographic material. ## Providing a device with the `wvd` and `prd` commands @@ -107,8 +107,8 @@ The `wvd` command group manages `.wvd` files. $ unshackle wvd add ./nexus_6p.wvd ``` - The file is validated and moved into the WVDs directory. You can add several at - once by passing multiple paths. + unshackle validates the file and moves it into the WVDs directory. You can add + several at once by passing multiple paths. === "Inspect a device" @@ -117,8 +117,8 @@ The `wvd` command group manages `.wvd` files. ``` Prints the System ID, Security Level, Type, Flags, and whether a Private Key, - Client ID, and VMP are present. A bare name (no extension) is resolved inside the - WVDs directory; a path with an extension is used as-is. + Client ID, and VMP are present. unshackle finds a bare name (no extension) inside + the WVDs directory. It uses a path with an extension as-is. === "Create from raw material" @@ -166,7 +166,7 @@ The `prd` command group creates and maintains `.prd` files. $ unshackle prd new ./device_folder/ ``` - A fresh leaf certificate is generated and the device is written to the PRDs + unshackle makes a fresh leaf certificate and writes the device to the PRDs directory (or to `-o/--output`). Use `-e/--encryption_key` and `-s/--signing_key` to supply your own ECC keys instead of generating them. @@ -177,8 +177,8 @@ The `prd` command group creates and maintains `.prd` files. ``` Regenerates the leaf certificate and encryption/signing keys of an existing - device in place. The device must support reprovisioning (a group key must be - present, i.e. a version 3 or higher device). + device in place. The device must be able to reprovision (a group key must be + present, that is, a version 3 or higher device). === "Test against Microsoft's demo server" @@ -188,12 +188,12 @@ The `prd` command group creates and maintains `.prd` files. Runs a full challenge/license/keys cycle against Microsoft's public PlayReady test server and prints the returned keys, confirming the device works. - `-sl/--security-level` accepts `150`, `2000`, or `3000` (default `2000`); + `-sl/--security-level` accepts `150`, `2000`, or `3000` (default `2000`). `-c/--ckt` accepts `aesctr` or `aescbc`. ## Selecting which device to use: the `cdm` config -Having a device on disk is not enough; you must tell unshackle which device to use +A device on disk is not enough. You must tell unshackle which device to use for which service. That is the job of the `cdm` section, which maps a service tag to a device name (the file's name without extension). A `default` entry acts as the fallback: @@ -221,8 +221,9 @@ profile. === "By quality" - Keys are matched against the video height. You can use exact heights or - comparisons (`>=1080`, `>720`, `<=576`, `<480`). More specific matches win. + unshackle compares each config key against the video height. You can use exact + heights or comparisons (`>=1080`, `>720`, `<=576`, `<480`). The most specific one + wins. ```yaml cdm: @@ -246,7 +247,8 @@ profile. === "By profile" - Keys are matched against the active profile, with a `default` fallback. + unshackle compares each config key against the active profile, with a `default` + fallback. ```yaml cdm: @@ -257,15 +259,15 @@ profile. ## Local vs remote CDMs -A CDM can run in two places: +A CDM can operate in two places: - A **local CDM** loads a `.wvd` or `.prd` device from your machine and performs the challenge/license/keys handshake locally. This is what the `cdm` mapping above resolves to by default. - A **remote CDM** delegates the CDM operations to an HTTP API. Your device (and its - secrets) live on the remote server; unshackle sends it PSSH/challenge data and + secrets) live on the remote CDM host. unshackle sends it PSSH/challenge data and receives keys back. This is useful for sharing a high-security-level device, using - a hosted key service, or keeping device secrets off your download machine. + a hosted content key service, or keeping device secrets off your download machine. The device name in your `cdm` mapping resolves to a remote CDM whenever it matches the `name` of an entry in the `remote_cdm` list. Otherwise it falls back to a local @@ -273,13 +275,13 @@ The device name in your `cdm` mapping resolves to a remote CDM whenever it match ## Remote CDMs -Remote CDMs are declared as a list under `remote_cdm`. Each entry has a `name` (which +You declare remote CDMs as a list under `remote_cdm`. Each entry has a `name` (which you then reference from the `cdm` mapping) and a shape that depends on its `type`. There are four kinds. ### Widevine remote CDM (pywidevine serve) -With no `type` field and a non-PlayReady device type, the entry is treated as a +With no `type` field and a non-PlayReady device type, unshackle treats the entry as a standard pywidevine "serve" endpoint: ```yaml title="unshackle.yaml" @@ -296,13 +298,14 @@ cdm: default: my_wv_remote ``` -`security_level` defaults to `3000` if omitted. Field names are read -case-insensitively, so `Device Type`/`device_type`, `System ID`/`system_id`, etc. are -all accepted. +`security_level` defaults to `3000` if omitted. unshackle reads field names +case-insensitively, so `Device Type`/`device_type`, `System ID`/`system_id`, and the +other field names are all accepted. ### PlayReady remote CDM -If the device type is `PLAYREADY`, the entry is routed to the pyplayready remote CDM: +If the device type is `PLAYREADY`, unshackle routes the entry to the pyplayready +remote CDM: ```yaml title="unshackle.yaml" remote_cdm: @@ -337,11 +340,11 @@ remote_cdm: decrypt_labs_api_key: your-decrypt-labs-key ``` -The API key is sent as the `decrypt-labs-api-key` header. If you omit `secret` on the -entry, unshackle falls back to the global `decrypt_labs_api_key` value; if neither is -present it raises an error. The `host` defaults to +unshackle sends the API key as the `decrypt-labs-api-key` header. If you do not write +`secret` on the entry, unshackle falls back to the global `decrypt_labs_api_key` value. +If neither is present it raises an error. The `host` defaults to `https://keyxtractor.decryptlabs.com`. PlayReady devices (`SL2`/`SL3`) default to -security level `2000`/`3000` respectively; Widevine devices default to +security level `2000`/`3000` respectively. Widevine devices default to `system_id: 26830`, `security_level: 3`. ### Custom API (`custom_api`) @@ -373,25 +376,30 @@ remote_cdm: The `custom_api` type supports much more: request/response field remapping, transforms (base64/hex/JSON encoding, `kid:key` parsing), conditional parameters, -success/error detection, and vault-backed caching. Its full schema is aimed at -developers integrating a bespoke key server; see the developer note below. +success/error detection, and vault-backed caching. Its full schema is for developers +who integrate a bespoke CDM API. See the developer note below. !!! note "Developer: full `custom_api` schema" - A `custom_api` entry accepts these sub-objects: `device` (`name`, `type`, - `system_id`, `security_level`); `auth` (`type` of `header`/`bearer`/`basic` with - the matching credential fields, plus `custom_headers`); `endpoints` - (`get_request` and `decrypt_response`, each `{path, method, timeout}`); - per-endpoint `request_mapping` (`param_names`, `static_params`, - `conditional_params`, `transforms`, `nested_params`, `exclude_params`) and - `response_mapping` (`fields`, `transforms`, `response_types`, - `success_conditions`, `error_fields`, `key_fields`); `caching` (`enabled`, - `use_vaults`, `check_cached_first`); and `legacy`/`timeout`. PlayReady mode is - inferred when `device.type == PLAYREADY` or the device name is `SL2`/`SL3`. + A `custom_api` entry accepts these sub-objects: + + - `device` (`name`, `type`, `system_id`, `security_level`) + - `auth` (`type` of `header`/`bearer`/`basic` with the matching credential fields, + plus `custom_headers`) + - `endpoints` (`get_request` and `decrypt_response`, each `{path, method, timeout}`) + - per-endpoint `request_mapping` (`param_names`, `static_params`, + `conditional_params`, `transforms`, `nested_params`, `exclude_params`) and + `response_mapping` (`fields`, `transforms`, `response_types`, + `success_conditions`, `error_fields`, `key_fields`) + - `caching` (`enabled`, `use_vaults`, `check_cached_first`) + - `legacy` and `timeout` + + unshackle infers PlayReady mode when `device.type == PLAYREADY` or the device name + is `SL2`/`SL3`. ## Decryption backends -Once the content keys are known, the encrypted file (for Widevine, PlayReady, and -ClearKey-CENC) is decrypted by one of two external tools: +Once unshackle knows the content keys, one of two external tools decrypts the +encrypted file (for Widevine, PlayReady, and ClearKey-CENC): - **shaka-packager** is the default. - **mp4decrypt** is part of the Bento4 tools. @@ -401,7 +409,7 @@ Both must be discoverable in unshackle's bundled `binaries/` directory or on you "executable not found but is required" error. See [dependencies](../getting-started/installation.md) for installation. -You choose the backend with the `decryption` key. It can be a single value applied +You choose the backend with the `decryption` config key. It can be a single value applied globally, or a per-service mapping: === "Global default" @@ -419,31 +427,32 @@ globally, or a per-service mapping: ``` !!! info "How the value is interpreted" - The default is `shaka`. Only the exact value `mp4decrypt` selects mp4decrypt; - **anything else** (including `shaka`) selects shaka-packager. Per-service lookups + The default is `shaka`. Only the exact value `mp4decrypt` selects mp4decrypt. + **Anything else** (including `shaka`) selects shaka-packager. Per-service lookups are case-insensitive, with the `default`/`DEFAULT` entry used as the fallback. -The two backends produce equivalent output for standard CENC content. shaka-packager -is generally preferred; switch a specific service to `mp4decrypt` if you hit a track -that shaka-packager cannot handle. +The two backends give equivalent output for a standard CENC title. In most +conditions shaka-packager is the better backend. Switch a specific service to +`mp4decrypt` if you find a track that shaka-packager cannot decrypt. !!! note "ClearKey (HLS) does not use these tools" - HLS AES-128 ClearKey content is decrypted in-process using AES-CBC and ignores the - `decryption` setting entirely. There is nothing to configure. + unshackle decrypts an HLS AES-128 ClearKey title in-process with AES-CBC, and + ignores the `decryption` setting entirely. You do not need to configure anything. ## ClearKey ### ClearKey over HLS (AES-128) -When an HLS playlist references a key with an `EXT-X-KEY` tag, unshackle fetches the -key directly (from the given URI or an inline `data:` URL) and decrypts each segment -in-process with AES-CBC. There is no device, no CDM, and no license challenge, so -there is nothing to set up. Such tracks are handled automatically during download. +When an HLS playlist references a content key with an `EXT-X-KEY` tag, unshackle +fetches the content key directly (from the given URI or an inline `data:` URL) and +decrypts each segment in-process with AES-CBC. There is no device, no CDM, and no +license challenge, so you do not set anything up. unshackle decrypts such tracks +automatically during download. ### ClearKey over CENC (W3C EME `org.w3.clearkey`) -The CENC ClearKey scheme is a proper license flow, but the "license server" simply -returns the keys as a JWK Set rather than a DRM-wrapped blob. unshackle builds a W3C +The CENC ClearKey scheme is a proper license flow, but the "license server" +returns the keys as a JWK Set and not a DRM-wrapped blob. unshackle builds a W3C EME request of the form: ```json @@ -451,14 +460,14 @@ EME request of the form: ``` It sends this to the manifest's license URL (or to whatever a service's ClearKey -license hook returns) and reads the JSON Web Key set back. No device file is needed. -The decrypted content still goes through shaka-packager or mp4decrypt like any other -CENC track, so the `decryption` setting applies. +license hook returns) and reads the JSON Web Key set back. You do not need a device +file. The decrypted file still goes through shaka-packager or mp4decrypt like any +other CENC track, so the `decryption` setting applies. ## Widevine and PlayReady on the same title A single manifest can advertise both Widevine and PlayReady. unshackle picks the CDM -per track: if a track needs a system that your loaded CDM does not provide, it +per track: if a track needs a system that your loaded CDM cannot supply, it transparently switches to the appropriate device for that track using your `cdm` mapping. If it cannot find a matching device, it fails with a clear message such as: @@ -467,7 +476,7 @@ Title needs a Widevine CDM but SOMESERVICE is configured with PlayReady. ``` Use the `cdm` mapping's [DRM-based selection](#advanced-selection-by-quality-drm-or-profile) -to configure a device for each system; unshackle selects the right one per track +to configure a device for each system. unshackle selects the right one per track automatically. ## Keys, vaults, and caching @@ -477,12 +486,12 @@ unshackle therefore resolves each key ID in this order before ever contacting a license server: 1. Keys already known for the current DRM object. -2. An in-process license-key cache (for the running session). +2. An in-process cache of content keys, for the current `dl` run. 3. Your configured [key vaults](vaults.md). 4. Only if still missing, a live license request. -After a successful license request, the recovered keys are written back to all -configured vaults so future downloads can skip the license server. Two flags let you +After a successful license request, unshackle writes the recovered keys back to all +configured vaults, so future downloads can skip the license server. Two flags let you control this trade-off on the `dl` command: | Flag | Effect | @@ -508,7 +517,7 @@ debug_requests: true ``` !!! warning "Keys and paths in logs" - When key logging is enabled, content keys (KID:key pairs) are written to the logs. + When you enable key logging, unshackle writes content keys (KID:key pairs) to the logs. Treat those logs as sensitive. The unrelated `redact_paths` option (default `true`) masks local directory paths in log output. diff --git a/docs/guide/output-and-naming.md b/docs/guide/output-and-naming.md index 4e88b588..fe13ca59 100644 --- a/docs/guide/output-and-naming.md +++ b/docs/guide/output-and-naming.md @@ -1,10 +1,10 @@ # Output & Naming -This page explains where unshackle writes finished downloads and how it names them. You control naming with **output templates** (the filename) and **folder templates** (the directory the file lands in), using a set of `{variable}` placeholders that unshackle fills in from the media it just downloaded. It also covers muxing options, release-group tagging, Unicode filenames, and how to relocate unshackle's working directories. +This page tells you where unshackle writes finished downloads and how it names them. You control naming with **output templates** (the filename) and **folder templates** (the directory the file lands in). Both take `{variable}` placeholders that unshackle fills in from the downloaded media. This page also covers muxing options, release-group tagging, Unicode filenames, and how to relocate unshackle's working directories. If you have not configured `unshackle.yaml` yet, read [Configuration File](../getting-started/configuration-file.md) first. Everything below lives in that file. -## How an output path is assembled +## How unshackle assembles an output path When a download finishes, unshackle builds the final path from three pieces: @@ -13,9 +13,9 @@ When a download finishes, unshackle builds the final path from three pieces: ``` - **Output directory**: `directories.downloads` from your config, or whatever you pass to `-o/--output` on the command line for a single run. -- **Folder template**: an optional per-title subfolder. TV episodes and music tracks are **always** placed in a folder; movies only get one if you define a `movies` folder template, or a single-string `folder` template that applies to all title kinds (see [Folder templates](#folder-templates)). +- **Folder template**: an optional per-title subfolder. unshackle **always** puts TV episodes and songs in a folder. Movies only get one if you define a `movies` folder template, or a single-string `folder` template that applies to all title kinds (see [Folder templates](#folder-templates)). - **Filename template**: the `output_template` for the title kind (movie, series, or song). -- **Extension**: chosen by the muxer for a movie or an episode: `.mkv` for video, `.mka` for audio-only, `.mks` for subtitle-only. A music track is not muxed. It keeps the container the service delivered, such as `.flac`, `.m4a`, or `.mp3`. Read [Music output files](#music-output-files). +- **Extension**: chosen by the muxer for a movie or an episode: `.mkv` for video, `.mka` for audio-only, `.mks` for subtitle-only. unshackle does not mux a song. A song keeps the container the service delivered, such as `.flac`, `.m4a`, or `.mp3`. Read [Music output files](#music-output-files). !!! example "A typical episode path" ```text @@ -35,7 +35,7 @@ Two command-line flags change this layout for a single run: ## Output templates -Filename templates live under the `output_template` key, keyed by title kind. Each value is a format string built from `{variable}` placeholders. +Filename templates live under the `output_template` config key, keyed by title kind. Each value is a format string built from `{variable}` placeholders. ```yaml title="unshackle.yaml" output_template: @@ -58,20 +58,20 @@ The recognized kinds are: ### Required vs. optional variables -- `{variable}`: **required**. If it is missing from the download's context, formatting raises an error. (Note that most variables are always present in context but resolve to an *empty string* when not applicable, so being "present but empty" is different from "missing".) -- `{variable?}`: **optional**. A trailing `?` marks the variable conditional. When it resolves to empty, unshackle removes the placeholder *and* one adjacent separator (`.`, `-`, or space), so the surrounding text collapses cleanly. +- `{variable}`: **required**. If it is missing from the download's context, formatting raises an error. Note that most variables are always present in context. A variable that does not apply becomes an *empty string*, so "present but empty" is different from "missing". +- `{variable?}`: **optional**. A trailing `?` marks the variable conditional. When it is empty, unshackle removes the placeholder *and* one adjacent separator (`.`, `-`, or space), so the surrounding text collapses cleanly. -!!! tip "Use `?` for anything that isn't always there" - Episode names, editions, tags, HDR labels and audio extras aren't present on every title. Mark them optional so you never get doubled dots or dangling separators: +!!! tip "Use `?` for anything that is not always there" + Episode names, editions, tags, HDR labels and audio extras are not present on every title. Mark them optional so you never get doubled dots or dangling separators: ```yaml series: "{title}.{season_episode}.{episode_name?}.{quality}.{hdr?}.{source}-{tag?}" ``` - For an episode with no on-screen name, `{episode_name?}` and its adjacent dot simply disappear. + For an episode with no on-screen name, `{episode_name?}` and its adjacent dot disappear. ### Separator style (dots vs. spaces) -unshackle looks at the characters you place *between* variables to decide the filename's separator style. If spaces outnumber dots in your template, it sanitizes with spaces; otherwise it uses dots (the scene-style default). This is also applied to the auto-generated date separators for daily/dated content, and to every segment of a nested folder template. +unshackle looks at the characters you place *between* variables to decide the filename's separator style. If spaces outnumber dots in your template, it sanitises with spaces. If not, it uses dots (the scene-style default). unshackle also applies this style to the auto-generated date separators of daily content, and to every segment of a nested folder template. === "Scene style (dots)" @@ -95,7 +95,7 @@ unshackle looks at the characters you place *between* variables to decide the fi ## Template variables -Every variable below is valid in both output and folder templates. Values are derived from the downloaded media (via MediaInfo), from the title metadata, or from your config. Variables that don't apply to a given title resolve to an empty string. +Every variable below is valid in both output and folder templates. unshackle takes the values from the downloaded media (through MediaInfo), from the title metadata, or from your config. A variable that does not apply to a title becomes an empty string. ### Common / video @@ -116,7 +116,7 @@ Every variable below is valid in both output and folder templates. Values are de | `lang_tag` | Result of your `language_tags` rules, if configured | `MULTi` | !!! note "How `language_tags` rules match" - Rules are tried in order, but language comparison is **fuzzy**: a rule condition of `en` matches `en-US`, `en-GB`, and any other `en-*` variant. Within a single rule, **all** conditions must match (AND logic) for it to apply. The first rule that fully matches wins; if none match, `{lang_tag?}` is removed from the filename cleanly. + unshackle tries the rules in order, but language comparison is **fuzzy**: a rule condition of `en` matches `en-US`, `en-GB`, and any other `en-*` form. Within a single rule, **all** conditions must match (AND logic) for it to apply. The first rule that fully matches wins. If none match, unshackle removes `{lang_tag?}` from the filename cleanly. ### Episode-specific @@ -131,17 +131,17 @@ Every variable below is valid in both output and folder templates. Values are de | `date` | ISO air date for daily/dated content | `2024-06-01` | !!! note "Daily & sports content" - When an episode has an air date, unshackle switches to date-based naming automatically: `season` and `season_episode` become the formatted air date, `episode` and `year` are cleared, and `{date}` holds the ISO date. The date's internal separator (dots or spaces) follows your `series` template style. + When an episode has an air date, unshackle switches to date-based naming automatically. `season` and `season_episode` become the formatted air date, unshackle clears `episode` and `year`, and `{date}` holds the ISO date. The date's internal separator (dots or spaces) follows your `series` template style. ### Split episodes {#split-episodes} -A few services split one episode into several separately playable videos. Where a service reports that, the part index is folded into `{episode}` and `{season_episode}`, so the stock `series` template names it with no change to your config. +A few services split one episode into several separately playable videos. Where a service reports that, unshackle folds the part index into `{episode}` and `{season_episode}`, so the stock `series` template names it with no change to your config. ```text Show.Name.S01E01.Part.2.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG.mkv ``` -The part token sits immediately after the episode token, before the quality tags, and the group tag stays last. Its separator follows your template's own style, the same way dated content does, so a spaced template gives `S01E01 Part 2` instead. +The part token sits immediately after the episode token, before the quality tags, and the group tag stays last. Its separator follows your template's own style, the same way daily content does, so a spaced template gives `S01E01 Part 2` instead. The part is left out of the folder name, so every part of an episode lands in the same season folder: @@ -152,10 +152,10 @@ The.Show.S01.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG/ └─ The.Show.S01E02.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG.mkv ``` -The same holds for a folder template of your own: `{season}` never picks up the part, and `{season_episode}` drops it inside a folder template. An episode with no part is named like any other episode. +The same holds for a folder template of your own: `{season}` never picks up the part, and `{season_episode}` drops it inside a folder template. unshackle names an episode with no part like any other episode. !!! note "What media servers do with the part token" - The form above follows scene practice for genuinely split episodes. Kodi stacks the parts back into one playable episode automatically. Jellyfin lists two versions of S01E01, still correct and playable, because its stacking pattern expects the part marker at the end of the name. Plex's scanner is closed-source, so its handling of text after the part token is unverified. + The form above follows scene practice for genuinely split episodes. Kodi stacks the parts back into one playable episode automatically. Jellyfin lists two versions of S01E01, still correct and playable, because its stacking pattern expects the part marker at the end of the name. Plex uses a closed-source scanner, so what it does with text after the part token is not known. For custom templates there is also a standalone [`{part}`](../reference/configuration/output.md#output_template) variable. You rarely want it, see the caveat on that page. @@ -186,11 +186,11 @@ For custom templates there is also a standalone [`{part}`](../reference/configur | `isrc` / `upc` / `label` | ISRC, UPC and record label | !!! warning "Unknown variables are ignored, not errors" - If you use a variable that isn't in the list above, unshackle emits a warning at startup but continues. Double-check spelling. A typo'd `{quailty}` will simply be left in the filename literally or dropped, not filled in. + If you use a variable that is not in the list above, unshackle gives a warning at startup but continues. Double-check spelling. A typo'd `{quailty}` stays in the filename literally, or unshackle drops it. unshackle does not fill it in. ## Folder templates -Folder templates are nested under a special `folder` key inside `output_template`. This key is handled separately from the filename kinds. You can give it a single string (used for all title kinds) or a per-kind map. +Folder templates sit under a special `folder` config key inside `output_template`. unshackle reads this config key separately from the filename kinds. You can give it a single string (for all title kinds) or a per-kind map. === "Single folder template" @@ -213,19 +213,19 @@ Folder templates are nested under a special `folder` key inside `output_template albums: "{artist} - {album} ({year})" ``` -The per-kind folder keys are `movies`, `series`, `songs`, and `albums`. Any other key produces a startup warning. Path separators (`/` or `\`) are allowed in folder templates. Each segment is formatted independently, so you can build nested directory structures like `Show/Season 01`. unshackle reads the separator style from the whole template, so every segment is spaced alike. +The per-kind folder config keys are `movies`, `series`, `songs`, and `albums`. Any other config key gives a startup warning. Folder templates permit path separators (`/` or `\`). unshackle formats each segment independently, so you can assemble nested directory structures like `Show/Season 01`. unshackle reads the separator style from the whole template, so it spaces every segment alike. -**Fallback behavior:** +**Fallback behaviour:** -- A per-kind folder template wins if present; otherwise the single `folder` string is used; otherwise unshackle falls back to a built-in default. +- A per-kind folder template wins if present. If there is none, unshackle uses the single `folder` string. If there is no `folder` string either, unshackle falls back to a built-in default. - Built-in defaults when no folder template is set. Movies get no folder at all unless a `movies` folder template (or a single-string `folder` template) exists. Series fall back to a folder *derived* from the `series` output template (stripping `{episode}`, `{episode_name}`, and collapsing `{season_episode}` down to `{season}`). Music albums fall back to `{artist} - {album} ({year})`. If the song has no year, the folder name is only `{artist} - {album}`. !!! note "Movies are flat by default" - Episodes and songs are always foldered. Movies are written directly into the output directory *unless* you define a `movies` folder template (or a single-string `folder` template, which folders every kind). Set one if you want each movie in its own directory. + Episodes and songs are always foldered. unshackle writes movies directly into the output directory *unless* you define a `movies` folder template (or a single-string `folder` template, which folders every kind). Set one if you want each movie in its own directory. ## Music output files -A music track does not go through the muxer. unshackle moves the downloaded audio file to +A song does not go through the muxer. unshackle moves the downloaded audio file to its final path. It keeps the container that the service delivered. A lossless download arrives as a `.flac` file. An AAC download arrives as a `.m4a` file. unshackle writes no `.mkv` file for music, and the [muxing options](#muxing-options) below do not apply. @@ -248,10 +248,10 @@ track title, artist, album artist, track and disc numbers, date, genre, ISRC, UP copyright, and record label. When `tag` and `tag_group_name` are set, unshackle also writes the group name into a `GROUP` tag. -Cover art is embedded when the service gives an artwork URL. unshackle downloads the image +unshackle embeds cover art when the service gives an artwork URL. It downloads the image and puts it in the file. -Lyrics are written when the service supplies them: a `LYRICS` Vorbis comment for FLAC, a +unshackle writes lyrics when the service supplies them: a `LYRICS` Vorbis comment for FLAC, a `USLT` frame for MP3, and a `©lyr` atom for M4A. Ogg uses the same `LYRICS` field as FLAC. unshackle does not write synchronized lyrics (`SYLT`). @@ -263,7 +263,7 @@ warning and continues. ## Muxing options -Muxing (combining video, audio, subtitle, chapter and attachment tracks into a single Matroska file with `mkvmerge`) is configured under the `muxing` key. Only movies and episodes are muxed. Read [Music output files](#music-output-files) for what a music download does instead. +You configure muxing (the combination of video, audio, subtitle, chapter and attachment tracks into a single Matroska file with `mkvmerge`) under the `muxing` config key. unshackle muxes only movies and episodes. Read [Music output files](#music-output-files) for what a music download does instead. ```yaml title="unshackle.yaml" muxing: @@ -282,29 +282,29 @@ muxing: | `merge_audio` | bool | `true` | Merge audio tracks of the same kind so multiple languages sit in one file. | | `default_language` | map | *(unset)* | Preferred language per track type (`video` / `audio` / `subtitle`). A track in the preferred language is flagged as the default track. | -When no preferred language is matched, unshackle applies sensible defaults: the video default falls back to the title language / original-language track / first track; the audio default is the original-language track; and the subtitle default is a forced track matching the first audio's language. +When no track has the preferred language, unshackle applies sensible defaults. The video default falls back to the title language, then the original-language track, then the first track. The audio default is the original-language track. The subtitle default is a forced track in the first audio's language. !!! note "`default_language` only sets the default-track flag" - `default_language` controls **which track carries the MKV `--default-track` flag**. Nothing else. It does not change track *selection* (that stays with `-l`/`--alang` and friends), and it does not touch which track is marked as original: `--original-flag` still tags the *true* original-audio track. That is the point: you can make your player default to, say, Polish audio on an English-original title without altering the original marker. When the configured language isn't present in the manifest, each track type falls back to its normal default rule described above. + `default_language` controls **which track carries the MKV `--default-track` flag**. Nothing else. It does not change track *selection* (that stays with `-l`/`--alang` and friends). It also does not touch which track carries the original flag: `--original-flag` still tags the *true* original-audio track. That is the point: you can make your player default to, say, Polish audio on an English-original title without altering the original marker. When the configured language is not present in the manifest, each track type falls back to its normal default rule described above. !!! warning "`merge_video` collapses only the language dimension" - `merge_video` groups video tracks by `(resolution, range, codec)` and merges them so that only **language** differs within a single file, with no re-encode and no concatenation. Different resolutions, ranges (SDR / HDR10 / HDR10+ / DV / HYBRID), and codecs (H.264 / H.265) **always** land in separate files. So `-r HYBRID,DV,HDR10,SDR --merge-video` produces one file *per range* (never a single fused file), whereas English + French video of identical resolution, range, and codec yields one file containing both video tracks. `merge_audio` works the same way for audio languages. + `merge_video` groups video tracks by `(resolution, range, codec)` and merges them so that only **language** differs within a single file, with no re-encode and no concatenation. Different resolutions, ranges (SDR / HDR10 / HDR10+ / DV / HYBRID), and codecs (H.264 / H.265) **always** land in separate files. So `-r HYBRID,DV,HDR10,SDR --merge-video` gives one file *per range*, never a single fused file. English and French video of identical resolution, range, and codec gives one file that holds both video tracks. `merge_audio` works the same way for audio languages. A few muxing behaviors are automatic and not configurable: - `--no-date` is always passed for privacy (no timestamps embedded). -- Descriptive audio gets the visually-impaired flag; SDH subtitles get the hearing-impaired flag; forced subtitles get the forced flag. -- Chapters are written from the title's chapter list (see [Chapters](#chapters-naming) below). +- Descriptive audio gets the visually-impaired flag. SDH subtitles get the hearing-impaired flag. Forced subtitles get the forced flag. +- unshackle writes the chapters from the title's chapter list (see [Chapters](#chapters-naming) below). The `--no-mux` flag skips muxing entirely and writes the individual track files, each with a track-type suffix, into the same output/folder structure. ## Tags & group naming -The release-group tag is the `-TAG` portion at the end of scene-style names, produced by the `{tag}` variable. -It can be swapped per release with [`tag_rules`](../reference/configuration/output.md#tag_rules): ordered rules that -match on the release-attribute variables only (`quality`, `resolution`, `hdr`, `source`, `lang_tag`, -`title_type`, and so on), first match wins. The title's own fields, such as `title` and `season_episode`, -cannot be matched. See the [reference page](../reference/configuration/output.md#tag_rules) for the exact list. +The release-group tag is the `-TAG` portion at the end of scene-style names, which the `{tag}` variable gives. +You can swap it per release with [`tag_rules`](../reference/configuration/output.md#tag_rules): ordered rules whose +conditions read the release-attribute variables only (`quality`, `resolution`, `hdr`, `source`, `lang_tag`, +`title_type`, and so on), and the first rule that matches wins. `tag_rules` cannot read the title's own fields, +such as `title` and `season_episode`. See the [reference page](../reference/configuration/output.md#tag_rules) for the exact list. ```yaml title="unshackle.yaml" tag: "MYGROUP" @@ -318,7 +318,7 @@ tag_imdb_tmdb: true | `tag_group_name` | bool | `true` | Write the group name (`config.tag`) into the MKV `Group` metadata tag. | | `tag_imdb_tmdb` | bool | `true` | Look up and embed IMDb / TMDB / TVDB external-ID tags in the MKV metadata (uses `tmdb_api_key` / `tvdb_api_key` / `simkl_client_id` when available). | -The IDs are written as the Matroska `IMDB`, `TMDB`, and `TVDB2` tags. `TVDB2` values carry the +unshackle writes the IDs as the Matroska `IMDB`, `TMDB`, and `TVDB2` tags. `TVDB2` values carry the entity prefix the [Matroska tagging spec](https://www.matroska.org/technical/tagging.html) requires: `series/73871` for a show, `movies/113` for a film. @@ -339,7 +339,7 @@ unshackle dl --repack EXAMPLE "..." ## Unicode filenames -By default unshackle transliterates non-ASCII characters (Korean, Japanese, Chinese, accented Latin, etc.) into ASCII equivalents, which is safest for older tools, DDL, and P2P sharing. Set `unicode_filenames` to keep the native characters. +By default unshackle transliterates non-ASCII characters into ASCII equivalents. Korean, Japanese, Chinese, and accented Latin are examples. This is safest for older tools, DDL, and P2P sharing. Set `unicode_filenames` to keep the native characters. ```yaml title="unshackle.yaml" unicode_filenames: true @@ -350,11 +350,13 @@ unicode_filenames: true | `false` (default) | `기생충` → transliterated ASCII; accents stripped. | | `true` | Native characters preserved in filenames and folders. | -Regardless of this setting, a set of filesystem-unsafe and structural characters is always removed or replaced during sanitization (for example `/` and `;` become ` & `, and characters like `\ * ! ? , ' " < > | $ # ~` are stripped). This is why you should never put path-unsafe characters directly in a template. unshackle warns about templates containing `< > : " / \ | ? *` at startup (folder templates are checked per path segment, so `/` is allowed there as a directory separator). +Regardless of this setting, unshackle always removes or replaces a set of filesystem-unsafe and structural characters during sanitisation. For example, `/` and `;` become ` & `, and unshackle removes characters like `\ * ! ? , ' " < > | $ # ~`. This is why you must never put path-unsafe characters directly in a template. + +unshackle warns at startup about a template that contains `< > : " / \ | ? *`. It examines a folder template per path segment, so you can use `/` there as a directory separator. ## Directory & filename configuration -The `directories` key relocates unshackle's working folders. Only the folders listed here can be moved; core package paths are protected and silently ignored if you try to override them. +The `directories` config key relocates unshackle's working folders. You can move only the folders in the table below. unshackle protects the core package paths and silently ignores them if you try to override them. ```yaml title="unshackle.yaml" directories: @@ -376,11 +378,11 @@ directories: | `services`, `vaults`, `fonts`, `commands` | Search paths for services, key-vault backends, bundled fonts, and CLI commands. | | `wvds`, `prds`, `dcsl` | Widevine devices, PlayReady devices, and DCSL data. | -Paths support `~` expansion. The names `app_dirs`, `core_dir`, `namespace_dir`, `user_configs`, and `data` are protected and cannot be changed. +Paths can hold a `~` for expansion. unshackle protects the names `app_dirs`, `core_dir`, `namespace_dir`, `user_configs`, and `data`, so you cannot change them. ### Filename patterns -The `filenames` key overrides the naming patterns for a handful of internal files. These are **not** your media output names. They are logs, temp files, and per-service configs. +The `filenames` config key overrides the naming patterns for a handful of internal files. These are **not** your media output names. They are logs, temp files, and per-service configs. ```yaml title="unshackle.yaml" filenames: @@ -400,13 +402,13 @@ filenames: ### Chapters naming {#chapters-naming} -When a title has chapters, they are written to an OGM chapters file (`filenames.chapters`) and passed to the muxer. Unnamed chapters use the `chapter_fallback_name` template: +When a title has chapters, unshackle writes them to an OGM chapters file (`filenames.chapters`) and passes it to the muxer. Unnamed chapters use the `chapter_fallback_name` template: ```yaml title="unshackle.yaml" chapter_fallback_name: "Chapter {i:02}" ``` -The fallback supports two placeholders: `{i}` (chapter number, starting at 1) and `{j}` (which increments only for unnamed chapters). Standard format directives like `{i:02}` are allowed. +The fallback can hold two placeholders: `{i}` (chapter number, starting at 1) and `{j}` (which increments only for unnamed chapters). It permits standard format directives like `{i:02}`. ## Removed & migrated options @@ -509,12 +511,12 @@ how the optional (`?`) variables appear and disappear. The.Movie.2024.1080p.EXAMPLE.WEB-DL.DDP5.1.H.264-TAG.mkv ``` - For finer control (e.g. a `SUBBED` tag driven by subtitle languages), use `{lang_tag?}` + For finer control (for example a `SUBBED` tag driven by subtitle languages), use `{lang_tag?}` with [`language_tags` rules](../reference/configuration/download.md#language_tags) instead. === "Daily shows" - Dated content needs no special template. When an episode carries an air date, + Daily content needs no special template. When an episode carries an air date, `{season_episode}` becomes the date automatically and `{episode}` and `{year}` clear: ```yaml @@ -532,8 +534,8 @@ how the optional (`?`) variables appear and disappear. The date separator follows your template style: dots here, spaces if your `series` template uses spaces. - In folder names, `{season}` becomes the air year (`Season 2024` style grouping), and a - `{year?}` in the folder template is skipped when it would just repeat that year. + In folder names, `{season}` becomes the air year (`Season 2024` style grouping), and + unshackle skips a `{year?}` in the folder template when it would only repeat that year. === "Music" @@ -548,8 +550,8 @@ how the optional (`?`) variables appear and disappear. └─ 01.Opening.Track.mka ``` - In `"{track_number}. {title}"` dots and spaces are tied, and ties go to dot style; add - more spaces between variables (as below) if you want space-separated names. + In `"{track_number}. {title}"` the number of dots and spaces is equal, and a tie goes to + dot style. Add more spaces between variables (as below) if you want space-separated names. A multi-disc album with explicit tracks, using more of the music variables. `{disc}` is empty on disc 1, so `{disc?}` and its separator only appear from disc 2 onward: diff --git a/docs/guide/proxies-and-vpn.md b/docs/guide/proxies-and-vpn.md index d2020503..0935df88 100644 --- a/docs/guide/proxies-and-vpn.md +++ b/docs/guide/proxies-and-vpn.md @@ -1,28 +1,28 @@ # Proxies & VPN -Most streaming services are geo-restricted, region-priced, or simply refuse to hand +Most streaming services are geo-restricted, region-priced, or refuse to hand over a manifest from the "wrong" country. unshackle routes its traffic through a proxy so the service sees the location you choose instead of your own. You can point it at a single proxy URL, or configure one of several **proxy providers** (commercial VPNs and self-hosted tunnels), then ask for a country by its two-letter code and let unshackle pick a working server for you. -The `--proxy` flag controls proxying, and a query is resolved to a real proxy URL by the -providers unshackle supports: [basic static proxies](#basic-static-proxies), +The `--proxy` flag controls proxying, and the proxy providers unshackle supports change +a query to a real proxy URL: [basic static proxies](#basic-static-proxies), [Gluetun](#gluetun) (documented in the most detail, since it is the most flexible), [ExpressVPN](#expressvpn), [NordVPN](#nordvpn), [Proton VPN](#proton-vpn), [Hola](#hola), [Surfshark](#surfshark), and [Windscribe](#windscribe). !!! note "Where proxies are configured" - All provider configuration lives under the top-level `proxy_providers:` key in your - `unshackle.yaml`. Each provider has its own sub-key, and the keys inside it map - directly onto that provider's settings. See the + All proxy provider configuration lives under the top-level `proxy_providers:` + config key in your `unshackle.yaml`. Each proxy provider has its own sub-key, and + the config keys inside it map directly onto that proxy provider's settings. See the [Configuration file](../getting-started/configuration-file.md) page for where the - file lives and how it is loaded. + file lives and how unshackle loads it. ## The `--proxy` flag -Proxying is controlled at download time by three options on the `dl` command: +Three options on the `dl` command control proxying at download time: | Option | Effect | |---|---| @@ -43,14 +43,14 @@ unshackle dl --proxy nordvpn:us EXAMPLE 81234567 ``` !!! tip "Set a default proxy in config" - Like every `dl` flag, `--proxy` can be given a default in your configuration so you + Like every `dl` flag, you can give `--proxy` a default in your configuration, so you do not have to pass it on every command. See [Downloading](downloading.md) for how `dl` options and config defaults interact. ### `--no-proxy-download` Segment downloads are usually the largest share of traffic, and a residential-grade VPN -proxy is often the slowest link in the chain. `--no-proxy-download` lets you keep the +proxy is frequently the slowest part of the chain. `--no-proxy-download` lets you keep the proxy where it matters (passing the geo-check for the manifest and the licence) while pulling the files themselves over your normal connection at full speed. @@ -66,8 +66,8 @@ unshackle dl --proxy gb --no-proxy-download EXAMPLE 10a1234 ## How proxy resolution works -The value you pass to `--proxy` is interpreted in one of three ways. unshackle checks -them in order. +unshackle interprets the value you pass to `--proxy` in one of three ways. It tries +them in this order. ```mermaid flowchart TD @@ -78,54 +78,56 @@ flowchart TD D -- no --> F["Ask every provider in order,
first match wins"] ``` -1. **An explicit URI.** Anything shaped like `http://...`, `https://...`, or a `socks...` - URI is used exactly as given. unshackle logs `Using explicit Proxy: ...` and does no - lookup. +1. **An explicit URI.** unshackle uses anything shaped like `http://...`, + `https://...`, or a `socks...` URI exactly as you give it. unshackle logs + `Using explicit Proxy: ...` and does no lookup. 2. **A provider-prefixed query**: `provider:query`, for example `nordvpn:us` or - `gluetun:windscribe:us`. unshackle finds the provider whose name matches the prefix - (case-insensitive) and asks *only* that provider to resolve the remainder. If no such - provider is loaded, or it returns no proxy, the download errors. + `gluetun:windscribe:us`. unshackle finds the proxy provider whose name matches + the prefix (case-insensitive). It then asks *only* that proxy provider to find a + proxy for the remainder. If unshackle loads no such proxy provider, or if that + proxy provider returns no proxy, the download errors. 3. **A bare query**: a region like `us`, `gb`, `us:seattle`, or `us1234`. unshackle - asks each loaded provider **in order** and uses the first proxy any of them returns. + asks each loaded proxy provider **in order** and uses the first proxy any of them + returns. -The query is matched against a region grammar (a two-letter country code, optionally -followed by a server number and/or a `:city` or `-city` part), lowercased, and then -handed to the provider. The exact forms each provider accepts are documented in its -section below. +unshackle compares the query against a region grammar: a two-letter country code, +optionally followed by a server number and/or a `:city` or `-city` part. It then +lowercases the query and hands it to the proxy provider. Its section below gives the +exact forms each proxy provider accepts. ### Provider load order -When you run a download without `--no-proxy`, unshackle instantiates every configured -provider once, in this fixed order, and that same order is used for bare-query +When you do a download without `--no-proxy`, unshackle instantiates every configured +proxy provider once, in this fixed order. It uses that same order for bare-query resolution: **Basic → ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola** -So if you have both Basic and NordVPN configured and run `--proxy us`, a `us` entry in -your Basic config wins because Basic is tried first. To skip ahead to a specific -provider, prefix the query (`--proxy nordvpn:us`). +So if you have both Basic and NordVPN configured and use `--proxy us`, a `us` entry in +your Basic config wins, because unshackle tries Basic first. To skip ahead to a specific +proxy provider, prefix the query (`--proxy nordvpn:us`). !!! note "Auto-loading providers" Most providers only load when you configure them under `proxy_providers:`. Three are - special: **ExpressVPN** and **Proton VPN** also load automatically once their cached - session file exists on disk (even with no YAML), and **Hola** loads automatically + special: **ExpressVPN** and **Proton VPN** also load automatically once their cache + file exists on disk (even with no YAML), and **Hola** loads automatically whenever the `hola-proxy` binary is found on your `PATH`. ### Connection feedback -After a provider resolves a query, unshackle prints a line so you can confirm +After a proxy provider resolves a query, unshackle prints a line so you can see where you came out: - **Gluetun** reports the verified exit IP, country, and city of the container. - **ExpressVPN** and **Proton VPN** print a location summary such as `(USA - New York, #3 of 5): .214`. -- Other providers log the resolved proxy URL. +- Other proxy providers log the resolved proxy URL. ## Basic (static proxies) -The `Basic` provider is pure static configuration, with no accounts and no network calls. You -list proxy URLs per country code, and unshackle serves them back when you query that -country. This is the right choice when you already have proxy endpoints (from a proxy +The `Basic` proxy provider is pure static configuration, with no accounts and no network +calls. You record proxy URLs under each country code, and unshackle serves them back +when you query that country. This is the right choice when you already have proxy endpoints (from a proxy seller, your own servers, or a corporate gateway). Under each two-letter country code you can put either a single proxy string or a list of @@ -148,10 +150,10 @@ proxy_providers: | `us2` | The **2nd** entry in the US list (1-based). | - If a country has a single proxy string, that proxy is always returned. -- If it has a list and you give a plain country code, one is picked at random; append an - index (`de1`, `de2`) to pin a specific entry. -- Each URL is normalised (a missing scheme defaults to `http`) and validated, so a - malformed URI is rejected up front rather than failing mid-download. +- If it has a list and you give a plain country code, unshackle picks one at random. + Append an index (`de1`, `de2`) to pin a specific entry. +- unshackle normalises each URL (a missing scheme defaults to `http`) and validates it, + so it rejects a malformed URI up front and does not fail mid-download. ```shell unshackle dl --proxy de2 EXAMPLE 81234567 @@ -173,13 +175,13 @@ in unshackle itself. !!! warning "Docker is required" Gluetun needs a working [Docker](https://docs.docker.com/get-docker/) installation on - your `PATH`. If Docker is not found, the provider fails to load with an error linking + your `PATH`. If Docker is not found, the proxy provider fails to load with an error linking to the install docs. The container also needs the `NET_ADMIN` capability and access to `/dev/net/tun` to bring up the tunnel. unshackle requests both automatically. ### Query format -Gluetun uses a nested selector: the provider prefix picks Gluetun, and the rest names the +Gluetun uses a nested selector: the proxy provider prefix picks Gluetun, and the rest names the **VPN provider** and the **region**. ``` @@ -194,10 +196,10 @@ unshackle dl --proxy gluetun:windscribe:us EXAMPLE 81234567 unshackle dl --proxy gluetun:nordvpn:de EXAMPLE 81234567 ``` -After unshackle strips the leading `gluetun:`, the provider receives exactly -`provider:region` (two colon-separated parts; anything else is an error). The -`` must be one you have configured under `gluetun.providers`, and friendly -names are normalised to Gluetun's identifiers (for example `pia` and +After unshackle strips the leading `gluetun:`, the proxy provider receives exactly +`provider:region`. It must be two colon-separated parts, and anything else is an error. +The `` must be one you have configured under `gluetun.providers`, and +unshackle normalises friendly names to Gluetun's identifiers (for example `pia` and `privateinternetaccess` both map to `private internet access`). ### Configuration @@ -256,7 +258,7 @@ proxy_providers: - `private_key` is **always** required. - `surfshark`, `mullvad`, and `ivpn` also require `addresses`. - `windscribe` additionally requires **both** `addresses` and `preshared_key` - (`preshared_key` may be an empty string, but the key must be present). + (`preshared_key` may be an empty string, but the config key must be present). - NordVPN and Proton VPN need only `private_key`. === "OpenVPN" @@ -266,10 +268,10 @@ proxy_providers: !!! tip "Which `vpn_type` should I pick?" WireGuard is the default because it **can be faster**, but it needs the fiddlier credential setup above (private keys, and for some providers `addresses` and a - preshared key). OpenVPN is the **simplest** to configure (just a username and - password) at the cost of some speed. If you are getting started or hitting credential - trouble, OpenVPN is the path of least resistance; switch to WireGuard when you want the - extra throughput. + `preshared_key`). OpenVPN is the **simplest** to configure (only a username and + password) at the cost of some speed. If you start out, or if you have credential + trouble, OpenVPN is the path of least resistance. Switch to WireGuard when you want + the extra throughput. **Global keys:** @@ -287,37 +289,38 @@ For a given query region, Gluetun resolves the server like this: 1. An explicit entry in `server_hostnames`, `server_cities`, or `server_countries` for that alias always wins. -2. A `us1239`-style query (country code + number) with no explicit mapping is turned - into a provider-specific server hostname (for example `us1239.nordvpn.com` for - NordVPN, `us-1239.prod.surfshark.com` for Surfshark). -3. A bare two-letter code is expanded to the full country name Gluetun expects. +2. unshackle changes a `us1239`-style query (country code + number) with no explicit + mapping into a provider-specific server hostname (for example `us1239.nordvpn.com` + for NordVPN, `us-1239.prod.surfshark.com` for Surfshark). +3. unshackle expands a bare two-letter code to the full country name Gluetun expects. A few providers (Windscribe, VyprVPN, VPN Secure) select by **region** rather than -country, and Windscribe region names are translated automatically. For instance `us` +country, and unshackle converts Windscribe region names automatically. For instance `us` becomes `US East` and `uk`/`gb` become `United Kingdom`. ### Container lifecycle Understanding what Gluetun does behind the scenes helps when something goes wrong: -1. **Image check.** `qmcgaw/gluetun:latest` is pulled if it is not already present. -2. **Reuse.** If a container for this exact `provider:region` is already running (from a - concurrent unshackle session), it is adopted instead of starting a new one. Stopped or - dead containers with the same name are pruned first. +1. **Image check.** unshackle pulls `qmcgaw/gluetun:latest` if it is not already + present. +2. **Reuse.** If a container for this exact `provider:region` already runs (from a + concurrent unshackle process), unshackle adopts it and does not start a new one. It + removes stopped or dead containers with the same name first. !!! note "Expect a slow first request, then instant reuse" Bringing up a fresh tunnel takes roughly **10 to 30 seconds** for the first request to a given `provider:region`. This is the VPN handshake and readiness wait, not a hang. Every subsequent request that reuses that same container is effectively **instant**. A slow first run is normal. -3. **Port allocation.** A free local port is chosen starting from `base_port`. -4. **Start.** The container is launched bound to `127.0.0.1:{port}`. Credentials are - passed through a temporary, `0600`-permission env-file (never as `-e` flags), which is - overwritten and deleted immediately after. -5. **Readiness wait.** unshackle polls the container logs (up to 60 seconds) until both - the HTTP proxy is listening **and** the VPN reports a completed connection. Fatal - errors like `authentication failed` or `invalid credentials` abort early, and recent - logs are attached to the error to help you diagnose it. +3. **Port allocation.** unshackle chooses a free local port, starting from `base_port`. +4. **Start.** unshackle launches the container bound to `127.0.0.1:{port}`. It passes + credentials through a temporary, `0600`-permission env-file (never as `-e` flags), + and it overwrites and deletes that file immediately after. +5. **Readiness wait.** unshackle polls the container logs (up to 60 seconds) until the + HTTP proxy listens **and** the VPN reports a completed connection. Fatal errors like + `authentication failed` or `invalid credentials` abort early, and unshackle attaches + recent logs to the error to help you diagnose it. 6. **IP verification.** When `verify_ip` is on, unshackle queries the container's exit IP and checks the country matches what you asked for, raising a *region mismatch* error if it does not. @@ -325,23 +328,23 @@ Understanding what Gluetun does behind the scenes helps when something goes wron configured). !!! note "Cleanup and Ctrl+C" - Containers are cleaned up via an `atexit` handler on normal exit. Gluetun deliberately - does **not** install its own signal handlers, so `Ctrl+C` continues to work as you - expect. If a run is killed hard (e.g. `kill -9`) a container can be left behind; with - `auto_cleanup: true` the next run adopting that name will still work, or you can remove - it manually with `docker rm -f`. + unshackle cleans up containers with an `atexit` handler on normal exit. Gluetun + deliberately does **not** install its own signal handlers, so `Ctrl+C` continues to + work as you expect. If something kills a run hard (for example `kill -9`), a container + can stay behind. With `auto_cleanup: true` the next run that adopts that name still + works, or you can remove it manually with `docker rm -f`. ### Troubleshooting !!! example "Common Gluetun issues" - - **`RuntimeError` about Docker not found.** Install Docker and confirm `docker` is - on your `PATH`. + - **`RuntimeError` about Docker not found.** Install Docker and make sure that + `docker` is on your `PATH`. - **`invalid credentials` / `authentication failed` in the readiness wait.** Your WireGuard/OpenVPN credentials are wrong or in the wrong fields. Double-check `private_key`, `addresses`, and (for Windscribe) `preshared_key`. - **Region mismatch error.** The VPN connected but exited in a different country than - requested. Check your `server_countries` mapping, or set `verify_ip: false` if you - are intentionally using a nearby region. + requested. Examine your `server_countries` mapping, or set `verify_ip: false` if + you intentionally use a nearby region. - **Timeout after 60 seconds.** The tunnel never came up. Inspect the container logs (`docker logs `) for the underlying VPN error. @@ -349,10 +352,10 @@ Understanding what Gluetun does behind the scenes helps when something goes wron ExpressVPN mirrors the ExpressVPN Android TV app: it signs in through Express's OAuth 2.0 device-login flow, obtains proxy-capable locations from Express's API, and returns an -authenticated HTTPS proxy. The device login runs once; after that the cached refresh token -keeps the session alive headlessly. +authenticated HTTPS proxy. The device login runs once. After that, the cached refresh +token keeps you signed in headlessly. -Enable it and run a download interactively once: +Enable it and do a download interactively once: ```yaml title="unshackle.yaml" proxy_providers: @@ -361,8 +364,8 @@ proxy_providers: ``` With `enable: true` and an interactive terminal, unshackle prints a code and a URL to -enter it at. Once approved, the tokens are cached and reused automatically, so later runs -need no terminal. ExpressVPN also **auto-loads** once its token cache exists, so you can +enter it at. Once you approve it, unshackle caches the tokens and reuses them +automatically, so later runs need no terminal. ExpressVPN also **auto-loads** once its token cache exists, so you can drop `enable` after the first login, but you can keep a YAML block to pin regions or servers: @@ -398,14 +401,14 @@ The resolved proxy is an HTTPS proxy on port `443`. On success, unshackle prints location summary such as `(USA - New York, #3 of 5): .214`. !!! note "Cache safety" - ExpressVPN's token cache is written with `0600` permissions so the refresh token is - never briefly world-readable. + unshackle writes ExpressVPN's token cache with `0600` permissions, so the refresh + token is never briefly world-readable. ## NordVPN -NordVPN uses your **Service Credentials**, not your normal NordVPN login. Generate them +NordVPN uses your **Service Credentials**, not your normal NordVPN login. Make them from the [NordVPN dashboard](https://my.nordaccount.com/dashboard/nordvpn/) under manual -setup; they are a long random username/password pair, not your email and password. +setup. They are a long random username/password pair, not your email and password. ```yaml title="unshackle.yaml" proxy_providers: @@ -423,9 +426,9 @@ the username must not contain an `@`. If validation fails, you have almost certa your login credentials instead of your service credentials. `server_map` is optional and maps a region (or `code:city`) to a pinned NordVPN server ID. -Its main use is tuning: for a given streaming service one specific server often performs -noticeably better than NordVPN's recommended pick, so hard-coding that server for a day or -two is a handy workaround until it degrades and you swap in another. +Its main use is to tune performance: for a given streaming service one specific server +often performs noticeably better than NordVPN's recommended pick. So you can hard-code +that server for a day or two as a workaround, until it degrades and you swap in another. !!! note "Use `gb`, not `uk`, for the United Kingdom" Query the UK as `gb`. unshackle uses the Alpha-2 country code `gb` deliberately to stay @@ -451,13 +454,13 @@ unshackle dl --proxy nordvpn:us:seattle EXAMPLE 81234567 ## Proton VPN Proton VPN supports two authentication methods, and unshackle can work with either. Note -that free Proton accounts work but are limited to Proton's free-tier exit countries. +that free Proton accounts work, but Proton limits them to its free-tier exit countries. === "TV login (recommended)" - A self-sustaining, refreshable session obtained through Proton's TV device-login flow. - This is the only method that can refresh itself headlessly, so it keeps working without - re-exporting anything. Enable it and run a download interactively once: + Proton's TV device-login flow gives a self-sustaining, refreshable sign-in. This is + the only method that can refresh itself headlessly, so it keeps working without + re-exporting anything. Enable it and do a download interactively once: ```yaml title="unshackle.yaml" proxy_providers: @@ -467,26 +470,26 @@ that free Proton accounts work but are limited to Proton's free-tier exit countr With `enable: true` and an interactive terminal, unshackle prints a code and asks you to enter it at [account.proton.me/vpn/tv/code](https://account.proton.me/vpn/tv/code). - Once approved, the session is cached and reused automatically. + Once you approve it, unshackle caches the sign-in and reuses it automatically. === "Exported cookies" - Export your `account.proton.me` session cookies (the `AUTH-` cookie) to the Proton + Export your `account.proton.me` cookies (the `AUTH-` cookie) to the Proton cookie file. This method is **access-only**: it cannot refresh, so you must re-export when it expires. !!! warning "The `AUTH-` cookie is HttpOnly" This is the single most common cause of a failed Proton cookie import: the required `AUTH-` cookie is **HttpOnly**, and many one-click cookie-export extensions - silently skip HttpOnly cookies by default. If your import "doesn't work," use an + silently skip HttpOnly cookies by default. If your import "does not work," use an exporter that explicitly includes HttpOnly cookies. !!! note "Why this session can't refresh, and lapses after ~24h" Proton rotates browser refresh tokens and ties them to the live browser, so a headless refresh is impossible. unshackle therefore deliberately uses only the - cookie's short-lived **access token** and never attempts a refresh, which is why the - session lapses at Proton's ~24h TTL and must be re-exported. This is exactly why TV - login is preferred for set-and-forget use. + cookie's short-lived **access token** and never attempts a refresh. That is why + the sign-in lapses at Proton's ~24h TTL and you must export it again. This is + exactly why TV login is the better method for set-and-forget use. **Default file locations:** @@ -495,14 +498,14 @@ that free Proton accounts work but are limited to Proton's free-tier exit countr | Cookies | `{cookies}/vpn/protonvpn.txt` | | Session cache | `{cache}/vpn/protonvpn.json` | -Proton VPN **auto-loads** if its cookie file exists. Like ExpressVPN, its cached session -is written with `0600` permissions. +Proton VPN **auto-loads** if its cookie file exists. Like ExpressVPN, unshackle writes +its cache file with `0600` permissions. !!! note "Cache vs cookie precedence" - If both a TV-login cache and a cookie file exist, the **cache is preferred** unless the - cookie file is newer. Cookies are read fresh every run and never written back to the - cache. So to deliberately force use of the cookie session, just re-save (re-export) the - cookie file so its modification time is newer than the cache. + If both a TV-login cache and a cookie file exist, unshackle **prefers the cache** + unless the cookie file is newer. unshackle reads cookies fresh every run and never + writes them back to the cache. So to deliberately force use of the cookie file, + re-save (re-export) it so its modification time is newer than the cache. **Query forms (after the `protonvpn:` prefix):** @@ -513,12 +516,12 @@ is written with `0600` permissions. | `us:ny` | A server in New York. | The returned proxy is HTTPS on **port 4443** (or **port 443** for Secure Core servers). -Secure Core and Tor servers are excluded from selection because they are too slow for a -plain proxy. On success, unshackle prints a summary such as `(Name - City): server-host`. +unshackle excludes Secure Core and Tor servers from selection because they are too slow +for a plain proxy. On success, unshackle prints a summary such as `(Name - City): server-host`. !!! tip "Proton server numbers aren't sequential" - When pinning a server with `protonvpn:deNN`, don't guess a low number. A country may - expose non-contiguous IDs like `#203`, `#813`, and so on. Copy the number from a + When pinning a server with `protonvpn:deNN`, do not guess a low number. A country may + have non-contiguous IDs like `#203`, `#813`, and so on. Copy the number from a previous run's log line (e.g. `DE#203`) instead. ## Hola @@ -526,7 +529,7 @@ plain proxy. On success, unshackle prints a summary such as `(Name - City): serv Hola requires no configuration at all. It uses the [`hola-proxy`](https://github.com/Snawoot/hola-proxy) binary, and **auto-loads** whenever that binary is found on your `PATH`. If you query Hola but the binary is missing, the -provider raises an error telling you to install it. +proxy provider raises an error that tells you to install it. ```shell title="Install hola-proxy, then just query a country" unshackle dl --proxy hola:us EXAMPLE 81234567 @@ -542,7 +545,7 @@ for available proxies in that country and picks one at random, returning an HTTP ## Surfshark -Surfshark, like NordVPN, uses **Service Credentials** rather than your login. Generate +Surfshark, like NordVPN, uses **Service Credentials** rather than your login. Make them from the [Surfshark manual-setup page](https://my.surfshark.com/vpn/manual-setup/main/openvpn). ```yaml title="unshackle.yaml" @@ -571,8 +574,8 @@ The returned proxy is HTTPS on **port 443**. ## Windscribe Windscribe uses **Service Credentials** from its -[OpenVPN config page](https://windscribe.com/getconfig/openvpn). Only the presence of a -username and password is checked (there is no 48-character rule). +[OpenVPN config page](https://windscribe.com/getconfig/openvpn). unshackle only makes +sure that a username and a password are present (there is no 48-character rule). ```yaml title="unshackle.yaml" proxy_providers: @@ -598,11 +601,11 @@ proxy_providers: The returned proxy is HTTPS on **port 443**. !!! warning "Windscribe is not loaded by the REST API resolver" - The standalone Windscribe provider (and Gluetun) are loaded by the `dl` command, but - **not** by the proxy resolver used by the REST API and remote-download client. If you - drive downloads through the API and need Windscribe - or Gluetun exits, resolve the proxy on the CLI side or use a different provider for API - jobs. See the developer note below. + The `dl` command loads the standalone Windscribe proxy provider (and Gluetun), but + the proxy resolver that the REST API and the remote-download client use does **not**. + If you drive downloads through the API and need Windscribe or Gluetun exits, get the + proxy on the CLI side, or use a different proxy provider for API jobs. See the + developer note below. ## Developer notes @@ -618,22 +621,22 @@ The returned proxy is HTTPS on **port 443**. Both paths accept the same query grammar (a direct URI, a `provider:query`, or a bare country query tried against providers in order), but the set of available providers - differs. Keep this in mind when adding a provider: wiring it into `dl.py` does not make - it available over the API, and vice versa. + differs. Keep this in mind when you add a proxy provider: an entry in `dl.py` does not + make it available over the API, and the opposite is also true. !!! note "Implementing a new provider (developers)" - Every provider subclasses `unshackle.core.proxies.proxy.Proxy` and implements three + Every proxy provider subclasses `unshackle.core.proxies.proxy.Proxy` and has three methods: - `__init__(self, **kwargs)` receives its `proxy_providers` sub-dict splatted as keyword arguments, so YAML keys map 1:1 to constructor parameters. Do authentication - and catalog pre-fetching here. + and catalogue pre-fetching here. - `__repr__(self)` is a human summary, conventionally `"{n} Countries ({m} Servers)"`. - `get_proxy(self, query)` returns a Requests-compatible proxy URI, or `None` only when the query was valid but no proxy exists. Raise on genuine errors or bad queries. - Optionally expose `get_connection_info(query)` (as Gluetun does) or - `last_connection_display()` (as ExpressVPN and Proton VPN do) to have `dl` print a + Optionally add `get_connection_info(query)` (as Gluetun does) or + `last_connection_display()` (as ExpressVPN and Proton VPN do) to make `dl` print a friendly connection line instead of the raw proxy URL. ## See also diff --git a/docs/guide/subtitles.md b/docs/guide/subtitles.md index 8efc008a..1c2a49e7 100644 --- a/docs/guide/subtitles.md +++ b/docs/guide/subtitles.md @@ -1,20 +1,21 @@ # Subtitles -Subtitles are first-class tracks in unshackle. When you download a title, subtitle -tracks are parsed from the manifest alongside video and audio, downloaded, cleaned up, -optionally converted to the format you want, and then either muxed into the final -Matroska file or written out as sidecar files next to it. The sections below cover the -supported formats, how tracks are normalized after download, how to select and filter -subtitles, and every subtitle-related configuration option. +Subtitles are first-class tracks in unshackle. When you download a title, unshackle +parses the subtitle tracks from the manifest alongside video and audio. It downloads +them, cleans them up, and optionally converts them to the format you want. It then muxes +them into the final Matroska file, or writes them out as sidecar files next to it. The +sections below cover the supported formats, how unshackle normalizes tracks after +download, how to select and filter subtitles, and every subtitle-related configuration +option. -Most of this is controlled with a handful of `dl` flags and a small `subtitle` block in -your [configuration file](../getting-started/configuration-file.md). See +A handful of `dl` flags and a small `subtitle` block in your +[configuration file](../getting-started/configuration-file.md) control most of this. See [Downloading](downloading.md) for how subtitle flags fit into a full download command. ## Supported formats unshackle understands the following subtitle codecs. The **Format** column is the value -shown in track listings; the **Extension** column is the file extension used on disk. +shown in track listings. The **Extension** column is the file extension used on disk. | Codec (internal name) | Format | Extension | Notes | | --- | --- | --- | --- | @@ -36,30 +37,30 @@ shown in track listings; the **Extension** column is the file extension used on unwrapped into their plain counterparts (`fTTML` becomes TTML and `fVTT` becomes WebVTT), so you always end up with an editable text subtitle. Note that this is a one-way, source-only relationship: you can convert *from* these boxed formats, but no - backend can produce them as an output target, so they can never be a conversion + backend can write them as an output target, so they can never be a conversion destination. ## What happens to a subtitle after download A subtitle download does more than save a file. Once the bytes are on disk, unshackle runs -format-appropriate clean-up to produce a single, valid file: +format-appropriate clean-up to make a single, valid file: -- **Encoding**: non-boxed subtitles are forced to UTF-8, and stray `‎` / `‏` - bidirectional entities are unescaped. +- **Encoding**: unshackle forces non-boxed subtitles to UTF-8, and unescapes stray + `‎` / `‏` bidirectional entities. - **Segment merging**: when a WebVTT or boxed subtitle arrives as many DASH/HLS - segments, the segments are stitched back into one continuous document with correct - timing. + segments, unshackle merges the segments back into one continuous document with + correct timing. - **WebVTT sanitization**: merged WebVTT runs through a repair pipeline that clamps negative timestamps to `00:00:00.000`, removes stray cue identifiers (like `Q0`, `Q1`) that confuse downstream parsers, and merges multi-line cues that a service split into - separate overlapping cues. By default the original styling and layout are preserved + separate overlapping cues. By default unshackle keeps the original styling and layout (see [`preserve_formatting`](#subtitle-configuration)). - **TTML for muxing**: Matroska cannot carry TTML. If you do not request a specific - output format, any plain TTML track is automatically converted to WebVTT before + output format, unshackle automatically converts any plain TTML track to WebVTT before muxing (the closest lossless equivalent). Boxed `fTTML` is likewise unwrapped to TTML first. -You normally do not need to think about any of this. It just produces a clean file. +You normally do not need to think about any of this. It gives a clean file. ## Selecting and filtering subtitles @@ -86,7 +87,7 @@ To narrow it down, pass a comma-separated list of language tags: unshackle dl --s-lang en,es EXAMPLE 81234567 ``` -Language matching is fuzzy by default: `en` will match `en-US`, `en-GB`, and similar +Language matching is fuzzy by default: `en` matches `en-US`, `en-GB`, and similar variants. If a requested language is missing but others were found, unshackle continues with whatever remains and tells you which languages it dropped. @@ -100,9 +101,9 @@ unshackle dl --s-lang es-419 --exact-lang EXAMPLE 'https://www.example.com/...' ### Requiring languages before downloading `--require-subs` is a gate rather than a filter. It downloads **all** available -subtitles, but only if every language you list is present; if any is missing the title -errors out. This is useful for batch jobs where a title without your must-have language -should fail loudly rather than silently produce a partial result. +subtitles, but only if every language you name is present. If any is missing, the title +errors out. This is useful for batch jobs where you want a title without your must-have +language to fail loudly, and not to silently give a partial result. ```shell title="Fail unless both English and French subs exist, then grab everything" unshackle dl --require-subs en,fr EXAMPLE B0ABCDEFGH @@ -115,7 +116,7 @@ unshackle dl --require-subs en,fr EXAMPLE B0ABCDEFGH ### Forced subtitles Forced tracks carry only the important on-screen signs and foreign-dialogue lines meant -to display even when you are watching in your own language. They are **excluded by +to display even when you watch in your own language. They are **excluded by default**. Add `-fs` / `--forced-subs` to include them: ```shell title="Include forced tracks alongside full subtitles" @@ -139,17 +140,17 @@ track, same as plain `-fs`). Every subtitle track carries flags describing what kind of captions it holds. These flags drive selection, sort order, output filenames, and the Matroska track flags. -- **Forced**: only signs and foreign dialogue; meant to play when the audio language - matches. Off by default; enable with `--forced-subs`. -- **SDH** (Subtitles for the Deaf and Hard-of-Hearing; also called HOH): a full +- **Forced**: only signs and foreign dialogue, meant to play when the audio language + matches. Off by default. Enable it with `--forced-subs`. +- **SDH** (Subtitles for the Deaf and Hard-of-Hearing, also called HOH): a full transcript that includes both dialogue and non-speech sound cues. - **CC** (Closed Captions): captions originating from an EIA-608/708 broadcast stream, often uppercase with `>>>` speaker markers and sound cues. !!! note "SDH and CC are treated together for sorting and stripping" unshackle orders subtitles from fewest to most captions: **Forced → Normal → SDH/CC**. - For the purposes of hearing-impaired handling, SDH and CC tracks are grouped as the - "most captions" variant. + For hearing-impaired handling, unshackle groups SDH and CC tracks as the "most + captions" variant. By default this type order is the primary sort, so all forced tracks come first, then all normal tracks, then all SDH/CC tracks, each block sorted by language. To keep each @@ -158,14 +159,14 @@ flags drive selection, sort order, output filenames, and the Matroska track flag [`subtitle.type_priority`](../reference/configuration/download.md#subtitle) to change the Forced → Normal → SDH/CC order itself. -A single track cannot be both CC and SDH, and a forced track cannot also be flagged -CC or SDH. These combinations are rejected as invalid. +A single track cannot be both CC and SDH, and a forced track cannot also carry the CC or +SDH flag. unshackle rejects these combinations as invalid. ### Closed captions embedded in video Some services do not ship subtitles as separate tracks but instead embed EIA-608/708 -captions inside the video stream. After a video track is downloaded and decrypted, -unshackle probes it for these captions and, when found, extracts them into a SubRip +captions inside the video track. After unshackle downloads and decrypts a video track, +it probes the track for these captions and, when it finds them, extracts them into a SubRip (`SRT`) subtitle flagged as CC. This runs automatically unless you disabled subtitles (`--no-subs`) or video (`--no-video` / `--video-only` semantics), and it requires the CCExtractor tool on your `PATH`. @@ -173,7 +174,7 @@ CCExtractor tool on your `PATH`. ## Automatic SDH stripping When a language has an SDH track but **no** plain (non-SDH, non-forced) track, unshackle -can generate a clean non-SDH version for you by stripping the hearing-impaired cues: +can make a clean non-SDH version for you by stripping the hearing-impaired cues: sound effects, speaker labels, music notes, and similar. This is on by default and controlled by the [`strip_sdh`](#subtitle-configuration) config key. @@ -182,9 +183,9 @@ example an `en-GB` non-SDH track when the SDH track is `en-US`), no stripped cop created. When a stripped copy *is* created, the original SDH track is kept as well, so you end up with both. -The stripping engine itself is chosen by the [`sdh_method`](#subtitle-configuration) -config key. The default (`auto`) prefers the `subby` cleaner for SRT, falls back to -SubtitleEdit when it is installed, and otherwise uses the built-in `filter-subs` +The [`sdh_method`](#subtitle-configuration) config key selects the stripping engine +itself. The default (`auto`) prefers the `subby` cleaner for SRT, falls back to +SubtitleEdit when you install it, and otherwise uses the built-in `filter-subs` approach. !!! note "Why `convert_before_strip` exists" @@ -192,7 +193,7 @@ approach. silently without stripping anything, not an error but a no-op that leaves the SDH cues in place. This is the whole reason for [`convert_before_strip`](#subtitle-configuration): non-SubtitleEdit engines need the subtitle converted to SRT first, or the - hearing-impaired cues survive the strip untouched. + hearing-impaired cues stay in the file. ## Converting subtitle formats @@ -218,11 +219,11 @@ Conversion only runs when the source format differs from the target, so !!! warning "Down-converting styled subtitles is lossy" Converting styled SSA/ASS subtitles to SRT throws away positioning, colours, and italics. unshackle never does this automatically. It only happens when you - explicitly ask for it via `--sub-format`. If you want to preserve styling, convert + explicitly ask for it with `--sub-format`. If you want to preserve styling, convert to a format that supports it (or keep `original`). - Note that this automatic protection guards **only the default muxed track**: there the - conversion is skipped and the styled original is kept. It does **not** cover sidecars. + Note that this automatic protection guards **only the default muxed track**: there + unshackle skips the conversion and keeps the styled original. It does **not** cover sidecars. A `sidecar_format: srt` will still lossily flatten styled subtitles, dropping positioning, colours, and styling, and a per-download `--sub-format srt` forces the muxed track to convert as well. To keep raw styled sidecars, set @@ -244,13 +245,13 @@ formats involved: With `conversion_method: auto`, unshackle ranks these automatically per conversion. Setting it to a specific value pins that backend as the first choice, falling back to -others only if the pin cannot handle the pair. +others only if the pin cannot convert the pair. #### What each backend actually preserves The table below comes from round-tripping a subtitle that carries italics, bold, -underline, positioning, and colour through every backend. Each cell lists the -styling that survives; `—` means the backend cannot handle that pair. +underline, positioning, and colour through every backend. Each cell shows the +styling that survives. `—` means that the backend cannot convert that pair. | Conversion | `subtitleedit` | `pysubs2` | `subby` | `pycaption` | | --- | --- | --- | --- | --- | @@ -273,13 +274,13 @@ Reading the table: - **`pysubs2`**: keeps inline italic/underline on every pair, and bold except when writing SRT or WebVTT. It never carries positioning or colour. SSA/ASS is its native model, which makes it the best pick for SSA↔ASS. -- **`subby`**: reads only WebVTT/fVTT/SAMI (never ASS) and is tuned for → SRT, where it - uniquely converts WebVTT cue settings into `{\an8}` positioning. Its +- **`subby`**: reads only WebVTT/fVTT/SAMI (never ASS), and it is best for → SRT, where + it uniquely converts WebVTT cue settings into `{\an8}` positioning. Its `CommonIssuesFixer` may also drop near-duplicate cues. -- **`pycaption`**: strips all styling; last-resort fallback only. +- **`pycaption`**: removes all styling. Last-resort fallback only. !!! tip "What `auto` picks" - `auto` prefers `subtitleedit` first when it is installed, then falls back to `subby` + `auto` prefers `subtitleedit` first when you install it, then falls back to `subby` for → SRT and `pysubs2` otherwise. The one exception comes from the table above: for ASS/SSA → TTML or WebVTT it picks `pysubs2` even when SubtitleEdit is installed, since SubtitleEdit flattens inline styling on those pairs. Most installs @@ -292,11 +293,11 @@ Reading the table: it ships subtitles that a particular backend handles best). An explicit `conversion_method` in `unshackle.yaml` always overrides that per-track preference, so your config takes precedence. This only matters when a service ships a non-default - preference; leave `conversion_method` at `auto` to let the service's hint stand. + preference. Leave `conversion_method` at `auto` to let the service's hint stand. ## Muxed vs. sidecar output -By default subtitles are muxed **into** the final `.mkv`. You can instead (or +By default unshackle muxes subtitles **into** the final `.mkv`. You can instead (or additionally) write them as separate sidecar files next to the video, controlled by the [`output_mode`](#subtitle-configuration) config key: @@ -307,7 +308,7 @@ additionally) write them as separate sidecar files next to the video, controlled output_mode: mux ``` - Subtitles are embedded in the Matroska file. Nothing is written separately. + unshackle embeds the subtitles in the Matroska file. It writes nothing separately. === "sidecar" @@ -317,8 +318,8 @@ additionally) write them as separate sidecar files next to the video, controlled sidecar_format: srt ``` - Subtitles are written as standalone files and **not** muxed into the video (when - there is a video or audio track to sit beside). + unshackle writes the subtitles as standalone files and does **not** mux them into + the video (when there is a video or audio track to sit beside). === "both" @@ -328,10 +329,10 @@ additionally) write them as separate sidecar files next to the video, controlled sidecar_format: srt ``` - Subtitles are muxed into the video **and** written as sidecar files. + unshackle muxes the subtitles into the video **and** writes them as sidecar files. -Sidecar files are named after the output file with the language and flags encoded in the -name, so players and media managers can match them automatically: +unshackle names sidecar files after the output file, with the language and flags encoded +in the name, so players and media managers can match them automatically: ```text Show.S01E01.1080p.WEB.en.srt @@ -339,9 +340,10 @@ Show.S01E01.1080p.WEB.en.forced.srt Show.S01E01.1080p.WEB.en.sdh.srt ``` -The pattern is `{base}.{language}[.forced][.sdh].{extension}`. The sidecar file format -is set by [`sidecar_format`](#subtitle-configuration); use a format name like `srt` or -`vtt`, or `original` to keep each subtitle in its downloaded format. +The pattern is `{base}.{language}[.forced][.sdh].{extension}`. The +[`sidecar_format`](#subtitle-configuration) config key sets the sidecar file format. Use +a format name like `srt` or `vtt`, or `original` to keep each subtitle in its downloaded +format. ## Fonts for styled subtitles @@ -353,7 +355,7 @@ relevant font packages. ## Subtitle configuration -All of the behaviour above can be defaulted in the `subtitle` block of your +You can set defaults for all of the behaviour above in the `subtitle` block of your [configuration file](../getting-started/configuration-file.md): ```yaml title="unshackle.yaml" @@ -381,7 +383,7 @@ subtitle: !!! tip "SubtitleEdit is optional but recommended" Several high-fidelity paths (the best-quality conversions and the SubtitleEdit SDH - stripper) only activate when the SubtitleEdit CLI is installed and on your `PATH`. + stripper) only activate when you install the SubtitleEdit CLI and put it on your `PATH`. See [Installation](../getting-started/installation.md) for how to add it. Without it, unshackle falls back to its built-in Python backends. @@ -397,23 +399,23 @@ The subtitle model lives in `unshackle.core.tracks.Subtitle`. A few internals wo knowing when working with subtitle tracks in code: - **Flags and validation**: `Subtitle(codec=..., cc=..., sdh=..., forced=...)`. A track - cannot be both `cc` and `sdh`, and `forced` cannot be combined with either; both raise + cannot be both `cc` and `sdh`, and you cannot combine `forced` with either. Both raise `ValueError` at construction. - **`convert(codec, *, forced=False)`**: converts the downloaded file in place through the backend chain in `subtitle_convert.run_conversion`. `forced=True` (what - `--sub-format` sets) is the only way lossy styled down-converts such as - ASS→SRT are permitted. Raises `NotImplementedError` if no backend supports the pair and + `--sub-format` sets) is the only way to permit lossy styled down-converts such as + ASS→SRT. Raises `NotImplementedError` if no backend supports the pair and `RuntimeError` if all attempts fail. - **`strip_hearing_impaired()`**: removes SDH cues using the engine selected by `config.subtitle["sdh_method"]`. -- **`reverse_rtl()`**: fixes right-to-left sentence-ending positioning via SubtitleEdit +- **`reverse_rtl()`**: fixes right-to-left sentence-ending positioning with SubtitleEdit (requires the binary). -- **`parse(data, codec)`**: the central parser, returning a `pycaption.CaptionSet`; it +- **`parse(data, codec)`**: the central parser, returning a `pycaption.CaptionSet`. It handles the boxed DASH formats, SAMI, and the WebVTT repair helpers. - **`extract_fonts(text)`**: returns the set of font names referenced by an ASS/SSA subtitle, used to decide which fonts to attach at mux time. The conversion backends (`SubtitleEditBackend`, `SubbyBackend`, `Pysubs2Backend`, -`PycaptionBackend`) each implement a small protocol (`is_available()`, `can_convert()`, -`rank()`, `convert()`) and are tried in rank order, lowest rank first. See +`PycaptionBackend`) each have a small protocol (`is_available()`, `can_convert()`, +`rank()`, `convert()`), and unshackle tries them in rank order, lowest rank first. See [the Tracks API reference](../reference/api/tracks.md) for the full model. diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index 50cf356f..f0fd70a2 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -1,12 +1,12 @@ # Troubleshooting & Debugging -When a download fails, a service misbehaves, or a key won't resolve, unshackle gives you three tools to find out why: +When a download fails, a service misbehaves, or unshackle cannot get a content key, unshackle gives you three tools to find out why: - **Debug logging**: verbose `DEBUG`-level console output plus a structured, machine-readable log file. -- **[`unshackle env`](#checking-your-environment)**: verify that the external tools unshackle depends on are installed, and see exactly which directories and config file are in use. +- **[`unshackle env`](#checking-your-environment)**: make sure that unshackle finds the external tools it depends on, and see exactly which directories and config file are in use. - **Targeted toggles**: `debug_keys` and `debug_requests` for the two areas (DRM keys and raw HTTP traffic) that are deliberately quiet by default. -The sections below cover how to enable each of them, how to read what they produce, and which diagnostic fits which failure mode. +The sections below cover how to enable each of them, how to read what they give, and which diagnostic fits which failure mode. --- @@ -38,27 +38,27 @@ There are two independent ways to enable debug logging, and they do slightly dif debug: true ``` - or via the [`cfg`](cli-reference.md#cfg) command: + or with the [`cfg`](cli-reference.md#cfg) command: ```console unshackle cfg debug true ``` !!! note "`debug: true` is not identical to `-d`" - The `debug` config key enables the **JSON debug log file**, but it does **not** raise the console to `DEBUG` level and does not add source-path annotations. Those only happen when you pass the `-d` flag. Use the flag when you want to *watch* the verbose output live; use the config key when you just want the log file written every time. Passing `-d` is the most complete option, and you can combine both. + The `debug` config key enables the **JSON debug log file**, but it does **not** raise the console to `DEBUG` level and does not add source-path annotations. Those only happen when you pass the `-d` flag. Use the flag when you want to *watch* the verbose output live. Use the config key when you want only the log file written every time. Passing `-d` is the most complete option, and you can combine both. !!! warning "Debug logging can be noisy and revealing" - Debug output is verbose and includes request metadata, service internals, and stack traces. Secrets, URLs, and local paths are automatically redacted (see [Redaction](#what-gets-redacted)), but you should still glance over a log before sharing it publicly. + Debug output is verbose and includes request metadata, service internals, and stack traces. unshackle automatically redacts secrets, URLs, and local paths (see [Redaction](#what-gets-redacted)), but you should still read a log before you share it publicly. --- ## The structured JSON debug log -Beyond the console output, unshackle writes a structured log in [JSON Lines](https://jsonlines.org/) format: one complete JSON object per line. This file is designed to be filtered, searched, and analysed with standard tooling (`jq`, `grep`, a text editor), and it is safe to attach to a bug report because it is redacted. +Beyond the console output, unshackle writes a structured log in [JSON Lines](https://jsonlines.org/) format: one complete JSON object per line. You can filter, examine, and analyse this file with standard tooling (`jq`, `grep`, a text editor). It is safe to attach to a bug report, because unshackle redacts it. ### Where it is written -The debug log is created during a `dl` run (and by [`import`](cli-reference.md#import), which wraps `dl`). The path is: +unshackle makes the debug log during a `dl` run, and during [`import`](cli-reference.md#import), which wraps `dl`. The path is: ``` /unshackle_debug__.jsonl @@ -67,19 +67,19 @@ The debug log is created during a `dl` run (and by [`import`](cli-reference.md#i For example, `unshackle_debug_EXAMPLE_20260703-142530.jsonl`. The `` tag and the timestamp mean each download gets its own file, so runs never overwrite each other. !!! tip "Find your logs directory" - The logs directory defaults to a `logs` folder inside the package data directory, but it can be relocated with the `directories.logs` config key. To see the exact resolved path on your machine, run [`unshackle env info`](#env-info) and read the **Directories** table. + The logs directory defaults to a `logs` folder inside the package data directory, but you can move it with the `directories.logs` config key. To see the exact resolved path on your machine, use [`unshackle env info`](#env-info) and read the **Directories** table. ### What a log entry looks like -Every line is a self-contained JSON object. The first line of a session records the environment: +Every line is a self-contained JSON object. The first line of each log file records the environment: ```json {"timestamp":"2026-07-03T14:25:30.123456+00:00","session_id":"a1b2c3d4","level":"INFO","operation":"session_start","message":"Debug logging session started","context":{"unshackle_version":"5.3.0","unshackle_code_hash":"1d22a1e","python_version":"3.12.4 ...","platform":"Linux-6.18.5-x86_64","platform_system":"Linux","platform_release":"6.18.5"}} ``` -Quote `unshackle_code_hash` in a bug report. It fingerprints the framework source you are running, so it tells us the exact code state even when you did not install from git. A hash that matches no release means the source was edited locally. +Quote `unshackle_code_hash` in a bug report. It fingerprints the framework source you operate, so it tells us the exact code state even when you did not install from git. A hash that matches no release means that someone edited the source locally. -Subsequent lines describe operations: service calls, DRM licence requests, vault lookups, downloader progress, and errors. Common fields include: +Subsequent lines record the operations: service calls, DRM licence requests, vault lookups, downloader progress, and errors. Common fields include: | Field | Meaning | |---|---| @@ -113,13 +113,13 @@ jq -r 'select(.error) | .error.traceback[]' unshackle_debug_EXAMPLE_20260703-142 jq 'select(.operation | startswith("drm_"))' unshackle_debug_EXAMPLE_20260703-142530.jsonl ``` -The session's binary-tool versions (Shaka Packager, mp4decrypt, mkvmerge, FFmpeg, FFprobe) are logged near the start of a `dl` run, which is often the fastest way to confirm a muxing or decryption failure is really a wrong-version problem. +unshackle logs the binary-tool versions (shaka-packager, mp4decrypt, mkvmerge, FFmpeg, FFprobe) near the start of a `dl` run. Often that is the fastest way to make sure that a muxing or decryption failure is really a wrong-version problem. --- ## Debugging DRM keys with `debug_keys` -By default, content-decryption **keys are redacted** in the JSON debug log. Any field that looks like a key is written as `[REDACTED]`. This keeps logs shareable. When you are specifically debugging a key-retrieval or decryption problem and need to see the actual KID/KEY values that were fetched, enable `debug_keys`: +By default, unshackle redacts the **content keys** in the JSON debug log. It writes any field that looks like a content key as `[REDACTED]`. This keeps logs shareable. When you specifically debug a key-retrieval or decryption problem and need to see the actual KID/KEY values unshackle fetched, enable `debug_keys`: ```yaml debug_keys: true @@ -129,10 +129,10 @@ debug_keys: true unshackle cfg debug_keys true ``` -With `debug_keys: true`, the debug log stops masking key fields, so you can confirm which keys came back from a licence server or vault and compare them against what decryption expected. +With `debug_keys: true`, the debug log stops masking content key fields. You can then see which content keys came back from a licence server or vault, and compare them against what decryption expected. !!! warning "Keys are sensitive, turn this back off" - `debug_keys` writes real decryption keys into the log file in cleartext. Only enable it while actively diagnosing a key issue, never share a log produced with it on, and set it back to `false` when you're done. Note that `debug_keys` only affects the **JSON debug log file**. It does not change what the console prints. + `debug_keys` writes real content keys into the log file in cleartext. Only enable it while you diagnose a content key problem. Never share a log written with it on, and set it back to `false` when you are done. Note that `debug_keys` only affects the **JSON debug log file**. It does not change what the console prints. Fields that are *always* redacted regardless of `debug_keys` (anything whose name contains `password`, `token`, `secret`, `auth`, or `cookie`) stay masked, so credentials never leak even in a keys-on log. @@ -142,7 +142,7 @@ Fields that are *always* redacted regardless of `debug_keys` (anything whose nam When you pass `-d`, unshackle raises its own modules to `DEBUG` but deliberately **silences** the noisy low-level HTTP libraries (`urllib3`, `requests`, `rnet`, `httpx`, `httpcore`, `hpack`, `h2`) back down to `WARNING`. Without this, a single download would bury the useful output under thousands of connection-pool and HTTP/2 frame messages. -If you need that low-level traffic (for example when a request is failing at the TLS or connection layer and you need to see the underlying library's own logs), enable `debug_requests`: +Sometimes a request fails at the TLS or connection layer, and you need the underlying library's own logs. If you need that low-level traffic, enable `debug_requests`: ```yaml debug_requests: true @@ -152,34 +152,34 @@ debug_requests: true unshackle cfg debug_requests true ``` -With `debug_requests: true`, those libraries are left at `DEBUG` level when you run with `-d`, so their request/response chatter appears in the console. +With `debug_requests: true`, unshackle leaves those libraries at `DEBUG` level when you operate it with `-d`, so their request/response chatter appears in the console. !!! note "`debug_requests` needs `-d` to have any effect" - The `debug_requests` toggle only decides whether the HTTP libraries are silenced when the `-d` flag is active. If you are not running with `-d` (verbose console logging), it changes nothing. Combine them: `unshackle -d dl ...` with `debug_requests: true` in config. + The `debug_requests` toggle only decides whether unshackle silences the HTTP libraries when the `-d` flag is active. If you do not operate unshackle with `-d` (verbose console logging), it changes nothing. Combine them: `unshackle -d dl ...` with `debug_requests: true` in config. --- ## What gets redacted -unshackle redacts logged strings so that debug output is safe to share. Three passes are applied to every logged message, error, and structured value: +unshackle redacts logged strings so that debug output is safe to share. It applies three passes to every logged message, error, and structured value: -- **Secrets**: passwords, tokens, API secrets, auth values, and cookies are replaced with `[REDACTED]`. -- **URLs**: every `http(s)` URL is collapsed to `redacted` (keeping only the file extension, so a manifest still shows as `redacted.mpd` and a segment as `redacted.m4s`). This hides content, CDN, manifest, and licence-server locations while preserving the *type* of resource. -- **Local paths**: install-root, virtualenv, and home-directory prefixes are replaced with tokens, so your username and machine layout don't appear in logs. +- **Secrets**: unshackle replaces passwords, tokens, API secrets, auth values, and cookies with `[REDACTED]`. +- **URLs**: unshackle collapses every `http(s)` URL to `redacted`. It keeps only the file extension, so a manifest still shows as `redacted.mpd` and a segment as `redacted.m4s`. This hides the CDN, manifest, segment, and licence-server locations, but keeps the *type* of resource. +- **Local paths**: unshackle replaces install-root, virtualenv, and home-directory prefixes with tokens, so your username and machine layout do not appear in logs. -Path redaction is controlled by the `redact_paths` config key, which is **on by default**: +The `redact_paths` config key controls path redaction, and it is **on by default**: ```yaml redact_paths: true # default - mask local base directories in logged paths ``` -Set `redact_paths: false` if you are debugging a path problem locally and want to see full, unmasked paths in the console and logs. Secret and URL redaction in the JSON debug log are always applied and are not affected by this toggle. +Set `redact_paths: false` if you debug a path problem locally and want to see full, unmasked paths in the console and logs. unshackle always applies secret and URL redaction in the JSON debug log, and this toggle does not change that. --- ## Checking your environment -Many "download failed" problems are really missing external tools or a config file that isn't where you think it is. The [`unshackle env`](cli-reference.md#env) command group answers both questions before you dig into logs. +Many "download failed" problems are really missing external tools or a config file that is not where you think it is. The [`unshackle env`](cli-reference.md#env) command group answers both questions before you dig into logs. ### `env check` @@ -189,7 +189,7 @@ Many "download failed" problems are really missing external tools or a config fi unshackle env check ``` -It reports each tool's status (`✓` installed / `✗` missing), whether it is **required** or optional, and what it's for. A summary line tells you the installed-vs-total count and, if anything mandatory is absent, exactly which required tools are missing. +It reports each tool's status (`✓` installed / `✗` missing), whether it is **required** or optional, and what it is for. A summary line tells you the installed-vs-total count and, if anything mandatory is absent, exactly which required tools are missing. The required tools (unshackle will not get far without them) are: @@ -204,20 +204,20 @@ The required tools (unshackle will not get far without them) are: Optional tools unlock extra capabilities. For example, `mp4decrypt` (alternative decryptor), `dovi_tool` / `HDR10Plus_tool` (Dolby Vision / HDR10+), `SubtitleEdit` / `CCExtractor` (subtitle conversion and CC extraction), `MPV` / `FFplay` (playback preview), and `git` (service repositories) or `docker` (Gluetun VPN, see [Proxies & VPN](proxies-and-vpn.md)). !!! tip "First thing to run when something breaks" - If a decryption step errors, a mux fails, or a Dolby Vision title won't process, run `unshackle env check` first. A red `✗` next to the relevant tool is the answer far more often than a bug in unshackle. + If a decryption step errors, a mux fails, or a Dolby Vision title will not process, use `unshackle env check` first. A red `✗` next to the relevant tool is the answer far more often than a bug in unshackle. ### `env info` -`unshackle env info` shows where unshackle is reading configuration from and where every working directory lives: +`unshackle env info` shows where unshackle reads configuration from and where every working directory lives: ```console unshackle env info ``` -- If a config file was found, it prints the path it was **loaded from**. If none was found, it lists every location it *would* accept one (see [Config file discovery](#why-is-my-config-not-being-picked-up) below), so you can create the file in the right place. +- If unshackle found a config file, it prints the path it was **loaded from**. If it found none, it shows every location it *would* accept one (see [Config file discovery](#why-is-my-config-not-being-picked-up) below). You can then make the file in the right place. - It then prints a **Directories** table for every configured path: `downloads`, `temp`, `cache`, `cookies`, `logs`, `exports`, `wvds`, `prds`, the `services` search paths, and more. -This is the authoritative way to answer "where are my logs / downloads / cookies actually going?" The defaults are relative to the installed package, and any `directories:` overrides in your config are reflected here. +This is the authoritative way to answer "where do my logs, downloads and cookies go?" The defaults are relative to the installed package. The table also shows any `directories:` overrides from your config. ### `env clear` @@ -241,17 +241,17 @@ Each command reports how many files it removed and how much space it freed, then !!! example "A required tool is missing" **Symptom:** decryption, muxing, or analysis fails early, sometimes with a "binary not found" style error. - **Inspect:** run `unshackle env check` and look for a red `✗` on a **required** row. Install the missing tool and re-run. + **Inspect:** use `unshackle env check` and look for a red `✗` on a **required** row. Install the missing tool and re-run. !!! example "Decryption produces garbage or fails" - **Symptom:** the download completes but the file won't play, or the decrypt step errors. + **Symptom:** the download completes but the file will not play, or the decrypt step errors. - **Inspect:** enable `debug_keys` and run with `-d`, then check the JSON debug log's `drm_*` and `vault_*` operations to confirm which keys were fetched and from where. Also confirm Shaka Packager (or `mp4decrypt`) shows a sane version in the `download_init` binary-versions entry. See [DRM & CDM](drm-and-cdm.md). + **Inspect:** enable `debug_keys` and operate unshackle with `-d`. Then examine the JSON debug log's `drm_*` and `vault_*` operations to see which content keys unshackle fetched, and from where. Also make sure that shaka-packager (or `mp4decrypt`) shows a sane version in the `download_init` binary-versions entry. See [DRM & CDM](drm-and-cdm.md). !!! example "A network request fails at the connection or TLS layer" - **Symptom:** timeouts, connection resets, or handshake errors that unshackle's own logs don't fully explain. + **Symptom:** timeouts, connection resets, or handshake errors, and unshackle's own logs do not give the full cause. - **Inspect:** set `debug_requests: true` and run with `-d` to surface the underlying `urllib3` / `rnet` / `httpx` logs. In the JSON debug log, filter for `service_call` operations to see which request failed. If a proxy is involved, cross-check [Proxies & VPN](proxies-and-vpn.md). + **Inspect:** set `debug_requests: true` and operate unshackle with `-d` to show the underlying `urllib3` / `rnet` / `httpx` logs. In the JSON debug log, filter for `service_call` operations to see which request failed. If you use a proxy, also read [Proxies & VPN](proxies-and-vpn.md). !!! example "The wrong titles/metadata keep coming back" **Symptom:** a service returns outdated titles even after the source changed. @@ -261,7 +261,7 @@ Each command reports how many files it removed and how much space it freed, then !!! example "A config value doesn't seem to apply" **Symptom:** a setting you edited has no effect. - **Inspect:** run `unshackle env info` to confirm **which** config file is actually loaded, and `unshackle cfg --list` to dump the effective values. It's common to edit a config file in a location unshackle isn't reading from. See below. + **Inspect:** use `unshackle env info` to see **which** config file unshackle loaded, and `unshackle cfg --list` to dump the effective values. It is common to edit a config file in a location unshackle does not read from. See below. --- @@ -273,19 +273,19 @@ unshackle loads the **first** `unshackle.yaml` it finds, in this order: 2. `unshackle.yaml` in the parent of that directory. 3. `unshackle.yaml` in your OS user-config directory: on Linux `~/.config/unshackle/unshackle.yaml`, on Windows `%LOCALAPPDATA%\unshackle\unshackle.yaml`, on macOS `~/Library/Application Support/unshackle/unshackle.yaml`. -If you have config in more than one of these, an earlier one wins and your edits to a later file are silently ignored. `unshackle env info` prints the path that was actually loaded (or, if none exists, lists all three candidate locations so you can create the file correctly). See [Configuration](cli-reference.md#cfg) for editing values with the `cfg` command. +If you have config in more than one of these, an earlier one wins and unshackle silently ignores your edits to a later file. `unshackle env info` prints the path it loaded (or, if none exists, shows all three candidate locations so you can make the file correctly). See [Configuration](cli-reference.md#cfg) for editing values with the `cfg` command. --- !!! note "Developer note: structured logging in code" - Service and command code emits entries through the shared `log_event(...)` helper (and the `timed_operation(...)` context manager for timing a block) from `unshackle.core.utilities`. These are no-ops when debug logging is disabled, so they are always safe to leave in place. When adding a new feature, call `log_event("my_feature_event", message="...", context={...})` rather than reimplementing the "is the logger enabled?" guard. The redaction and JSON-serialisation are handled for you, and any field whose name looks like a key/secret/token is redacted automatically (keys only appear when the run has `debug_keys` enabled). + Service and command code sends entries through the shared `log_event(...)` helper (and the `timed_operation(...)` context manager for timing a block) from `unshackle.core.utilities`. These are no-ops when you disable debug logging, so they are always safe to leave in place. When you add a new feature, call `log_event("my_feature_event", message="...", context={...})`. Do not write the "is the logger enabled?" guard again. unshackle does the redaction and the JSON-serialisation for you, and redacts any field whose name looks like a content key, a secret, or a token. Content keys only appear when the run has `debug_keys` enabled. ### Logging conventions when adding entries -The debug log is tuned for **developers troubleshooting pipeline flow** (maximum signal, minimum noise), not for end users. A few conventions keep it that way. They are not enforced by the `log_event` signature, so they are easy to miss and worth stating explicitly. +The debug log is tuned for **developers troubleshooting pipeline flow** (maximum signal, minimum noise), not for end users. A few conventions keep it that way. The `log_event` signature does not enforce them, so they are easy to miss and worth stating explicitly. !!! tip "Log levels are a flow skeleton, not just severity" - Treat `level` as structure, not decoration. Emit exactly **one `INFO` milestone per pipeline stage** so that `jq 'select(.level=="INFO")'` reads back the whole end-to-end flow of a run. Everything internal goes to `DEBUG`; failures go to `ERROR`. Keeping `INFO` sparse is the whole point. It is the line a developer skims first to see where a run got to before it broke. + Treat `level` as structure, not decoration. Send exactly **one `INFO` milestone per pipeline stage** so that `jq 'select(.level=="INFO")'` reads back the whole end-to-end flow of a run. Everything internal goes to `DEBUG`. Failures go to `ERROR`. A sparse `INFO` is the whole point. It is the line a developer skims first to see where a run got to before it broke. ```console # Read the pipeline's high-level flow, one line per stage @@ -293,13 +293,13 @@ The debug log is tuned for **developers troubleshooting pipeline flow** (maximum ``` !!! warning "No raw dumps" - Never log a full `Tracks` object, an MPD/manifest body, or a response payload. Log only **counts, ids, sizes, and `safe_display_url(url)`**. This keeps the log low-noise and prevents leaking content or manifest data into the file *before* redaction even runs. Redaction is a backstop, not a licence to dump raw objects. + Never log a full `Tracks` object, an MPD/manifest body, or a response payload. Log only **counts, ids, sizes, and `safe_display_url(url)`**. This keeps the log low-noise and prevents leaking title or manifest data into the file *before* redaction even runs. Redaction is a backstop, not a licence to dump raw objects. !!! note "Operation names: `_`, no registry" - Operation names are plain lowercase strings following an `_` convention: `manifest_dash_parse`, `drm_decrypt`, `vault_get_key`, `tool_run`. They are written inline at each call site on purpose: there is **no central registry or enum**, and you should not go hunting for one or create one. The absence is a deliberate design choice, not an omission. + Operation names are plain lowercase strings that follow an `_` convention: `manifest_dash_parse`, `drm_decrypt`, `vault_get_key`, `tool_run`. We write them inline at each call site on purpose. There is **no central registry or enum**. Do not look for one, and do not make one. The absence is a deliberate design choice, not an omission. !!! note "Message reads alone; data lives in fields" - Give every entry a single-sentence `message` that stands on its own, and put all structured data (`context`, counts, ids, `duration_ms`) in **fields** rather than baking it into the prose. This separation is what lets the log be both human-scannable and machine-filterable; a message like `"Parsed DASH manifest"` with `context={"video": 4, "audio": 2}` beats `"Parsed manifest with 4 video and 2 audio tracks"`. + Give every entry a single-sentence `message` that stands on its own. Put all structured data (`context`, counts, ids, `duration_ms`) in **fields**, not in the prose. This separation lets the log be both human-scannable and machine-filterable. A message like `"Parsed DASH manifest"` with `context={"video": 4, "audio": 2}` beats `"Parsed manifest with 4 video and 2 audio tracks"`. !!! tip "Let tool runs log themselves" - For external-tool calls (FFmpeg, mkvmerge, `dovi_tool`, and friends), route through `run_step` / `ffprobe`, which auto-emit a `tool_run` entry capturing the binary's version, return code, and duration consistently. Only fall back to calling `log_tool_run` manually for a direct `subprocess.run` you genuinely can't route through `run_step`. Auto-logging is the preferred path precisely because it captures those fields the same way every time. + For external-tool calls (FFmpeg, mkvmerge, `dovi_tool`, and friends), route through `run_step` / `ffprobe`, which auto-emit a `tool_run` entry capturing the binary's version, return code, and duration consistently. Only fall back to calling `log_tool_run` manually for a direct `subprocess.run` you genuinely cannot route through `run_step`. Auto-logging is the preferred path precisely because it captures those fields the same way every time. diff --git a/docs/index.md b/docs/index.md index 64cdd785..12e9780b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,13 +6,13 @@ well-tagged files, all driven from a single command-line tool, or served over a REST API for automation. It is a fork of [Devine](https://github.com/devine-dl/devine) with DASH, HLS, and -Smooth Streaming (ISM) parsing, both **Widevine** and **PlayReady** DRM support, -and a built-in HTTP server. +Smooth Streaming (ISM) parsing, both **Widevine** and **PlayReady** DRM, and a +built-in HTTP server. !!! warning "Use responsibly" - unshackle is licensed under [GPL-3.0](https://github.com/unshackle-dl/unshackle/blob/main/LICENSE). + The [GPL-3.0](https://github.com/unshackle-dl/unshackle/blob/main/LICENSE) licence covers unshackle. Do not use it for content you do not have the rights to. Keep the core free and - open, keep service code private, and be kind. + open, keep your own services private, and be kind. ## What it does @@ -24,8 +24,8 @@ at a title and it will: - Parse the streaming manifest: **DASH**, **HLS**, or **Smooth Streaming (ISM)**. - Select the tracks you asked for by resolution, codec, bitrate, range, language, and channel layout. -- Acquire decryption keys via a **Widevine** or **PlayReady** CDM, with optional - **key vaults** so keys are reused instead of re-licensed. +- Get the content keys through a **Widevine** or **PlayReady** CDM, with optional + **key vaults** that let unshackle reuse a content key instead of licensing it again. - Decrypt, extract closed captions, repack, and **mux** the result into a Matroska (`.mkv`) file with correct language, forced, and default flags. - Tag and name the output using configurable filename templates. @@ -76,19 +76,19 @@ service's title argument (a URL, ID, or slug). unshackle dl -q 1080 -l en EXAMPLE 81234567 ``` -The `dl` command carries all the quality, language, and output flags; the service -tag (`EXAMPLE` above) selects which streaming service to talk to, and the value after it -is passed to that service. See the [Quickstart](getting-started/quickstart.md) to go from a fresh -install to your first file. +The `dl` command carries all the quality, language, and output flags. The service +tag (`EXAMPLE` above) selects which streaming service to talk to, and unshackle passes +the value after it to that service. See the [Quickstart](getting-started/quickstart.md) +to go from a fresh install to your first file. ## Where to go next - **[Installation](getting-started/installation.md)**. Install unshackle and the external tools it depends on. -- **[Quickstart](getting-started/quickstart.md)**. Configure `unshackle.yaml` and run your first download. +- **[Quickstart](getting-started/quickstart.md)**. Configure `unshackle.yaml` and do your first download. - **[Downloading](guide/downloading.md)**. The full `dl` command guide: quality, codecs, languages, track selection, and output. -- **[Configuration](getting-started/configuration-file.md)**. Where `unshackle.yaml` lives, the directory layout, and how to edit values. Full key list in the [Configuration Reference](reference/configuration/index.md). -- **[REST API](dev/rest-api/index.md)**. Run the `serve` HTTP server and drive downloads programmatically. +- **[Configuration](getting-started/configuration-file.md)**. Where `unshackle.yaml` lives, the directory layout, and how to edit values. Full config key list in the [Configuration Reference](reference/configuration/index.md). +- **[REST API](dev/rest-api/index.md)**. Operate the `serve` HTTP server and drive downloads programmatically. !!! tip "Getting help" - Run `unshackle --help` or `unshackle dl --help` at any time. The community + Operate `unshackle --help` or `unshackle dl --help` at any time. The community [Discord](https://discord.gg/mHYyPaCbFK) is the fastest place to ask questions. diff --git a/docs/reference/api/config.md b/docs/reference/api/config.md index 9e29b85e..6d170f15 100644 --- a/docs/reference/api/config.md +++ b/docs/reference/api/config.md @@ -1,7 +1,7 @@ # Config API Auto-generated reference for the [`Config`][unshackle.core.config.Config] object that models the -`unshackle` configuration file. For a human-friendly walkthrough of every key, see the +`unshackle` configuration file. For a human-friendly walkthrough of every config key, see the [Configuration Reference](../configuration/index.md). ::: unshackle.core.config diff --git a/docs/reference/configuration/directories.md b/docs/reference/configuration/directories.md index ec132f66..dcb4fbe9 100644 --- a/docs/reference/configuration/directories.md +++ b/docs/reference/configuration/directories.md @@ -1,10 +1,10 @@ # Directories { #directories } -The `directories` key is a **dict** mapping directory names to filesystem paths. Only the -names listed below are honoured; a handful are **protected** and cannot be moved (attempts -to override them are silently skipped). All defaults are computed relative to the installed -package, and every user-settable path is passed through `Path(...).expanduser()`, so `~` -works. +The `directories` config key is a **dict** mapping directory names to filesystem paths. +unshackle honours only the names listed below. A handful are **protected** and you cannot +move them, and unshackle silently ignores an override of one. unshackle computes all +defaults relative to the installed package, and passes every user-settable path through +`Path(...).expanduser()`, so `~` works. ```yaml title="unshackle.yaml" directories: @@ -38,9 +38,9 @@ directories: !!! note "The `services` directory is special" `services` may be a **list**, and each entry can be either a local directory or a **repository spec**: a git URL (`https://...`, `ssh://...`, `git@...`, or anything ending in - `.git`) or `owner/repo` shorthand. Repo specs are cloned and updated automatically; plain - paths are used as-is. **List order is priority**: the first source to define a service - tag wins. + `.git`) or `owner/repo` shorthand. unshackle clones and updates repo specs automatically, + and uses plain paths as they are. **List order is priority**: the first source to define a + service tag wins. ```yaml directories: @@ -51,11 +51,11 @@ directories: ``` !!! info "How git-backed service repos are handled" - unshackle's use of git is **read-only on the remote**: it only ever performs a shallow - `clone`, `fetch`, `pull`, and a local `reset`; **nothing is ever pushed**. Private repos - are authenticated through your existing git credential helper, and unshackle stores no + unshackle's use of git is **read-only on the remote**: it only ever does a shallow + `clone`, `fetch`, `pull`, and a local `reset`. **unshackle never pushes anything**. Your + existing git credential helper authenticates private repos, and unshackle stores no tokens of its own. Clones live under `/_repos/____/` - (or the bundled `unshackle/services`), and **nothing is written to the cache directory**. After + (or the bundled `unshackle/services`), and **unshackle writes nothing to the cache directory**. After the first clone unshackle re-pulls **at most once every 24 h**, so it does not touch the network on every run. @@ -74,9 +74,9 @@ directories: ## Filenames { #filenames } -The `filenames` key is a **dict** of templated file/name patterns. Each value is used -verbatim (no path processing). Braced fields like `{time}` and `{service}` are filled in at -runtime. +The `filenames` config key is a **dict** of templated file/name patterns. unshackle uses +each value verbatim (no path processing). It fills in braced fields like `{time}` and +`{service}` at runtime. | Key | Type | Default | Notes | |-----|------|---------|-------| diff --git a/docs/reference/configuration/download.md b/docs/reference/configuration/download.md index 858a55f8..6efe68f3 100644 --- a/docs/reference/configuration/download.md +++ b/docs/reference/configuration/download.md @@ -4,13 +4,14 @@ - **Type:** `dict`  ·  **Default:** `{}` -Persistent defaults for the `dl` command. The block is used as Click's `default_map`, so any -`dl` flag can be given a default here. See [Downloading](../../guide/downloading.md) and the +Persistent defaults for the `dl` command. unshackle uses the block as Click's `default_map`, +so you can give any `dl` flag a default here. See [Downloading](../../guide/downloading.md) and the [CLI Reference](../../guide/cli-reference.md) for what each flag does. -**Key naming.** A key is the flag's long name with dashes replaced by underscores -(`--best-available` → `best_available`, `--sub-format` → `sub_format`). A few keys are named -after the flag's internal destination rather than its visible name; set these exact keys: +**Config key naming.** A config key is the flag's long name with dashes replaced by +underscores (`--best-available` → `best_available`, `--sub-format` → `sub_format`). A few +config keys use the flag's internal destination and not its visible name. Set these exact +config keys: | Flag | Config key | |------|-----------| @@ -27,7 +28,7 @@ after the flag's internal destination rather than its visible name; set these ex [`services..dl`](services.md) override → this global `dl:` block → the built-in default. In other words, config only fills in options you did not set on the command line. -Common keys (a useful subset; every `dl` flag works): +Common config keys. This is a useful subset, and every `dl` flag works: | Key | Type | Default | Sets | |-----|------|---------|------| @@ -89,28 +90,28 @@ dl: !!! note "`serve` uses these same keys" The [`serve`](services.md#serve) block accepts most of the same `dl` option keys at its top level - to set server-wide download defaults for every request it handles. Keys the API does not know - are ignored there (`speed_limit`, for example; `serve` has its own `global_speed_limit`). + to set server-wide download defaults for every request it handles. The API ignores the + config keys it does not know there, `speed_limit` for example. `serve` has its own + `global_speed_limit`. ## `continue_downloads` - **Type:** `bool`  ·  **Default:** `false` -Keep a failed download's completed segment files so the next attempt resumes from where it +Keep a failed download's completed segment files so the next download resumes from where it stopped instead of restarting from byte zero. -Only whole, fully downloaded segments are reused; partially written segments are always -discarded, and no partial-byte (`Range`) resume across runs takes place. Reuse also -requires proof that the remote content is unchanged: the manifest URL, segment count, and -byte ranges must all match the previous run, otherwise the download silently restarts -clean. +unshackle reuses only whole, fully downloaded segments. It always discards partially written +segments, and no partial-byte (`Range`) resume across runs takes place. Reuse is possible only +with proof that the segmentation did not change. The manifest URL, segment count, and byte +ranges must all match the previous download, or the download silently restarts clean. !!! note "What can and cannot resume" - Segmented DASH, ISM, and HLS tracks resume. - Single-file (non-segmented) downloads and DASH tracks collapsed to a single ranged - download never produce segment files, so this option does not apply to them (the + download never make segment files, so this option does not apply to them (the downloader has its own in-flight partial-file handling for those). - - HLS resumes Widevine/PlayReady and unencrypted content only; AES-128 content always + - HLS resumes Widevine/PlayReady and unencrypted titles only. An AES-128 title always restarts. An HLS failure during the post-download merge step also restarts. For a one-off resume without changing config, pass `dl --continue-downloads` on the @@ -124,8 +125,8 @@ continue_downloads: true - **Type:** `dict`  ·  **Default:** `{}` -Subtitle handling: SDH stripping, format conversion, and whether subtitles are muxed in or -written as sidecar files. See [Subtitles](../../guide/subtitles.md) for the full guide. +Subtitle handling: SDH stripping, format conversion, and whether unshackle muxes the subtitles +in or writes them as sidecar files. See [Subtitles](../../guide/subtitles.md) for the full guide. | Sub-key | Type | Default | Description | |---------|------|---------|-------------| @@ -168,7 +169,7 @@ subtitle: | Default | English, Arabic, Bulgarian, Spanish, French, Japanese | | With the list above | English, Spanish, French, Arabic, Bulgarian, Japanese | - Nothing is removed, unlike `-sl`, which drops every subtitle language it does not name. + unshackle removes nothing, unlike `-sl`, which drops every subtitle language it does not name. Use `orig` in the list for the title's original language. A base tag also covers its regional forms, so `en` sorts `en-US` and `en-GB`, but `zh` does **not** cover @@ -245,8 +246,8 @@ muxing: - **Type:** `dict`  ·  **Default:** `{}` Language-tag remapping and the rule engine behind the `{lang_tag}` filename variable. When a -`rules` list is present, each rule is evaluated in order and the first match's `tag` is used. -A rule may test: +`rules` list is present, unshackle evaluates each rule in order and uses the first match's +`tag`. A rule can examine: | Condition | Meaning | |-----------|---------| diff --git a/docs/reference/configuration/drm.md b/docs/reference/configuration/drm.md index b4c2f0f1..21bf4d92 100644 --- a/docs/reference/configuration/drm.md +++ b/docs/reference/configuration/drm.md @@ -31,13 +31,13 @@ cdm: default: chromecdm_l3 ``` -Quality keys support `1080`, `>=1080`, `>720`, `<=576`, `<480` style comparisons; DRM keys -are `widevine` / `playready`. Quality keys are matched first, but only when a video height is -known; if the value is still a dict after that, `widevine`/`playready` are used when present, -otherwise the credential profile name, then `default`. +Quality keys can hold `1080`, `>=1080`, `>720`, `<=576`, `<480` style comparisons. DRM keys +are `widevine` / `playready`. unshackle matches the quality keys first, but only when a video +height is known. If the value is still a dict after that, unshackle uses +`widevine`/`playready` when present, or else the credential profile name, then `default`. -The resolved name is looked up in [`remote_cdm`](#remote_cdm) by `name` first. If no entry -matches, it is loaded as a local device file, in this order: `.prd` in +unshackle finds the resolved name in [`remote_cdm`](#remote_cdm) by `name` first. If no entry +matches, unshackle loads it as a local device file, in this order: `.prd` in `directories.prds`, `.prd` in `directories.wvds`, then `.wvd` in `directories.wvds`. @@ -45,7 +45,7 @@ matches, it is loaded as a local device file, in this order: `.prd` in - **Type:** `list[dict]`  ·  **Default:** `[]` -A list of remote CDM server definitions. Each entry is matched by its `name` (referenced from +A list of remote CDM definitions. unshackle matches each entry by its `name` (referenced from [`cdm`](#cdm)), and its `type` selects the backend: | `type` | Backend | Key fields | @@ -89,7 +89,7 @@ styles: `Device Type`/`device_type`, `System ID`/`system_id`, `Security Level`/` | `system_id` | `26830` (Widevine), `0` (PlayReady) | | | `security_level` | Widevine `3`; PlayReady `2000` for `SL2`, else `3000` | | -PlayReady mode is used when `device_type` is `PLAYREADY` or `device_name` starts with `SL`. +unshackle uses PlayReady mode when `device_type` is `PLAYREADY` or `device_name` starts with `SL`. ### `custom_api` fields { #remote-cdm-custom-api } @@ -105,8 +105,8 @@ PlayReady mode is used when `device_type` is `PLAYREADY` or `device_name` starts | `legacy` | dict | Legacy mode options. | | `timeout` | int | Default `30`. Request timeout in seconds. | -Transforms cover base64/hex/JSON encoding and `kid:key` parsing. PlayReady mode is used when -`device.type` is `PLAYREADY` or the device name is `SL2`/`SL3`. +Transforms cover base64/hex/JSON encoding and `kid:key` parsing. unshackle uses PlayReady mode +when `device.type` is `PLAYREADY` or the device name is `SL2`/`SL3`. ```yaml remote_cdm: @@ -135,12 +135,12 @@ remote_cdm: - **Type:** `str` or `dict`  ·  **Default:** `"shaka"` -Selects the tool used to physically decrypt CENC content. The value is compared -case-insensitively; `"mp4decrypt"` selects Bento4's mp4decrypt, and **anything else** -(including `"shaka"`) selects shaka-packager. When the value is a dict, the service key is -matched case-insensitively and the `default` entry is the fallback. +Selects the tool that physically decrypts CENC-encrypted tracks. unshackle compares the value +case-insensitively. `"mp4decrypt"` selects Bento4's mp4decrypt, and **anything else** +(including `"shaka"`) selects shaka-packager. When the value is a dict, unshackle compares the +service tag case-insensitively and uses the `default` entry as the fallback. -HLS AES-128 ClearKey is decrypted in-process and ignores this setting. +unshackle decrypts HLS AES-128 ClearKey in-process and ignores this setting. === "Global" diff --git a/docs/reference/configuration/example.md b/docs/reference/configuration/example.md index 382cca61..bdf23dd3 100644 --- a/docs/reference/configuration/example.md +++ b/docs/reference/configuration/example.md @@ -1,6 +1,6 @@ # A complete annotated example -The following config sets a handful of common keys. Every key not shown keeps its default. +The following config sets a handful of common config keys. Every config key not shown keeps its default. ```yaml title="unshackle.yaml" directories: diff --git a/docs/reference/configuration/index.md b/docs/reference/configuration/index.md index 57f3ced7..2fceb625 100644 --- a/docs/reference/configuration/index.md +++ b/docs/reference/configuration/index.md @@ -1,17 +1,17 @@ # Configuration Reference -Every key you can set in `unshackle.yaml`, with its name, type, default value, and behaviour, +Every config key you can set in `unshackle.yaml`, with its name, type, default value, and behaviour, split into one page per area. A worked config is on the [annotated example](example.md) page. Where the file lives, how unshackle discovers it, and how to edit it: [The Configuration File](../../getting-started/configuration-file.md). !!! info "How defaults work" - Every key is optional except [`output_template`](output.md), which `dl` requires. - Without it, `dl` aborts with an error. If you omit any other key, unshackle uses - the built-in default listed here. There is **no schema validation**: unknown keys are - silently ignored (they never become settings), and the file is parsed with - `yaml.safe_load`. An empty or missing file means every other key falls back to its + Every config key is optional except [`output_template`](output.md), which `dl` requires. + Without it, `dl` aborts with an error. If you do not write any other config key, unshackle + uses the built-in default listed here. There is **no schema validation**: unshackle + silently ignores unknown config keys (they never become settings), and reads the file with + `yaml.safe_load`. An empty or missing file means every other config key falls back to its default. !!! warning "Two keys behave specially on load" diff --git a/docs/reference/configuration/misc.md b/docs/reference/configuration/misc.md index 5e989b67..a4481de2 100644 --- a/docs/reference/configuration/misc.md +++ b/docs/reference/configuration/misc.md @@ -5,7 +5,7 @@ Smaller configuration areas: external metadata API keys, caching and update chec ## External API keys { #external-api-keys } All are unset by default and enable optional metadata/geolocation features. For which fields -each metadata provider can actually answer with, see +each metadata provider can answer with, see [What each metadata provider supplies](../../guide/downloading.md#what-each-metadata-provider-supplies). | Key | Type | Default | Description | @@ -23,7 +23,7 @@ each metadata provider can actually answer with, see | `ipinfo_api_key` | str | `""` | ipinfo.io API key for IP/region lookups. | !!! note "How `metadata_providers` is applied" - A lookup goes to each provider in turn and stops at the first good match, so the order + A lookup goes to each metadata provider in turn and stops at the first good match, so the order decides which source wins. `metadata_providers` sets that order: ```yaml @@ -37,8 +37,8 @@ each metadata provider can actually answer with, see metadata_providers: [imdb, omdb, simkl, tmdb, tvdb, anilist] ``` - Whatever the order, unshackle skips a provider that has no API key. `imdb` and - `anilist` need no key, so they are always available. Available names are `imdb`, `omdb`, + Whatever the order, unshackle skips a metadata provider that has no API key. `imdb` and + `anilist` need no API key, so they are always available. Available names are `imdb`, `omdb`, `simkl`, `tmdb`, `tvdb`, and `anilist`. The old name `imdbapi` still works and is read as `imdb`. @@ -63,7 +63,7 @@ each metadata provider can actually answer with, see !!! note "What `disable_metadata` turns off" With `disable_metadata: true`, unshackle never contacts a metadata provider on its own. - The title search is dead, and that includes the keyless `imdb` provider. Titles, years and + The title search is dead, and that includes the keyless `imdb` metadata provider. Titles, years and languages then come only from the service. ```yaml @@ -72,18 +72,18 @@ each metadata provider can actually answer with, see An ID you give with `--tmdb`, `--imdb`, `--tvdb`, or `--anilist` is your permission to use that ID. That lookup still works. unshackle looks the ID up directly, cross references the other - IDs from it, and `--enrich` still reads that source. Only the lookups you did not ask for - are stopped. + IDs from it, and `--enrich` still reads that source. unshackle stops only the lookups you + did not ask for. !!! note "The `imdb` provider needs no key" - It replaces the earlier `imdbapi` provider, which read a third-party mirror + It replaces the earlier `imdbapi` metadata provider, which read a third-party mirror (api.tiffara.com) and needed the `imdb_api_enabled` switch. That option is gone: delete it from your config, where it is now ignored. !!! tip "A different order per title kind" Give a map instead of one list when the best source differs by kind. TVDB is strongest on series, TMDB on films. There are two kinds, `tv` and `movie`, and this example sets both - of them and names every provider, so nothing is left to the default: + of them and names every metadata provider, so nothing is left to the default: ```yaml metadata_providers: @@ -91,9 +91,9 @@ each metadata provider can actually answer with, see movie: [tmdb, imdb, omdb, simkl, tvdb] ``` - Each list is independent: it both orders and filters that kind on its own, so a provider - you leave out of `tv` is still used for `movie`. Here TVDB is tried first for a series and - last for a film. + Each list is independent: it both orders and filters that kind on its own, so unshackle + still uses for `movie` a metadata provider you leave out of `tv`. Here unshackle tries TVDB + first for a series and last for a film. You do not need to give both keys, or a full list in each. A kind you leave out uses the default order: @@ -104,11 +104,11 @@ each metadata provider can actually answer with, see ``` `tv` covers every episode and `movie` covers every film. unshackle only looks up - metadata for those two kinds, so any other key, such as `anime` or `music`, is ignored. + metadata for those two kinds, so unshackle ignores any other config key, such as `anime` or `music`. !!! note "`ipinfo_api_key` never touches your service sessions" - The token is only ever sent to `api.ipinfo.io` as a per-request `Authorization` header; it - is **never** attached to your service session, so it cannot leak to a streaming provider. + unshackle sends the token only to `api.ipinfo.io`, as a per-request `Authorization` header. + It **never** goes on your service HTTP session, so it cannot leak to a streaming service. Lookups degrade gracefully through a fallback chain: the authenticated Lite endpoint (higher rate limits, and ASN/continent data, but no city or region) → anonymous ipinfo → `ip-api.in` as a last resort. @@ -160,19 +160,19 @@ calls. | `set_terminal_bg` | bool | `false` | Append the theme's background colour to output styling. | !!! note "Effect of `set_terminal_bg`" - When on, the theme's background colour is appended to the foreground styles, which makes + When on, unshackle appends the theme's background colour to the foreground styles, which makes the full colour palette render correctly on terminals whose *default* background differs from unshackle's theme. When off, such terminals can render the ASCII-art banner and coloured output incorrectly. !!! danger "Key exposure" - With `debug_keys` on, decryption content keys are written to the structured debug log at + With `debug_keys` on, unshackle writes the content keys to the structured debug log at INFO level during the DRM handshake. Treat `debug`, `debug_keys`, and the `unshackle_debug_*.jsonl` files as sensitive, and keep `redact_paths` enabled when sharing logs. `debug_keys` affects **only** content-encryption keys (the `content_key`/`key` fields). Passwords, tokens, cookies, and session tokens are **always** redacted regardless of this - setting, and KIDs, key counts, and other metadata are always logged either way. + setting, and KIDs, content key counts, and other metadata are always logged either way. --- diff --git a/docs/reference/configuration/network.md b/docs/reference/configuration/network.md index 0c5e8d3a..b2bdf856 100644 --- a/docs/reference/configuration/network.md +++ b/docs/reference/configuration/network.md @@ -4,7 +4,7 @@ - **Type:** `dict`  ·  **Default:** `{}` -TLS-fingerprinting and HTTP client settings for the rnet-based session. +TLS-fingerprinting and HTTP client settings for the rnet-based HTTP session. | Sub-key | Type | Default | Description | |---------|------|---------|-------------| @@ -43,7 +43,7 @@ network: - **Type:** `dict`  ·  **Default:** `{}` -Default HTTP headers merged into every session unshackle creates. +Default HTTP headers merged into every HTTP session unshackle makes. ```yaml headers: @@ -52,17 +52,17 @@ headers: !!! warning "Don't set `Accept-Encoding` (and similar) here" Compatibility headers such as `Accept-Encoding` are set by the rnet HTTP backend as part of - its browser-impersonation profile; overriding them breaks the impersonation fingerprint. - This block is for cross-service defaults only (for example `Accept-Language`, - `User-Agent`); per-service headers belong in that service's own config. + its browser-impersonation profile. If you override them, you break the impersonation + fingerprint. This block is for cross-service defaults only (for example `Accept-Language`, + `User-Agent`). Per-service headers belong in that service's own config. ## `proxy_providers` - **Type:** `dict`  ·  **Default:** `{}` -Proxy/VPN provider configuration. Each sub-key names a provider, and its block is passed -straight to that provider's constructor. See [Proxies & VPN](../../guide/proxies-and-vpn.md) for -the full provider guide. Recognised providers and their exit ports: +Proxy/VPN provider configuration. Each sub-key names a proxy provider, and unshackle passes its +block straight to that provider's constructor. See [Proxies & VPN](../../guide/proxies-and-vpn.md) +for the full proxy provider guide. Recognised proxy providers and their exit ports: | Provider | Config key | Credentials | Proxy scheme/port | |----------|-----------|-------------|-------------------| diff --git a/docs/reference/configuration/output.md b/docs/reference/configuration/output.md index 9623a3f2..9ad8f7f5 100644 --- a/docs/reference/configuration/output.md +++ b/docs/reference/configuration/output.md @@ -23,7 +23,7 @@ output_template: series: "{title} ({year})" ``` -**The nested `folder` sub-key** controls output subfolders and is popped out of +**The nested `folder` sub-key** controls output subfolders, and unshackle pops it out of `output_template`: - A **dict** sets per-kind folder templates (kinds: `movies`, `series`, `songs`, `albums`). @@ -53,7 +53,7 @@ folder-kind keys. !!! tip "`{absolute}` counts episodes across seasons" `absolute` holds the absolute episode number, zero-padded to 3 digits (`007`), and is - empty on titles that have none. It is added to the name and never replaces `{season}` + empty on titles that have none. You add it to the name yourself, and it never replaces `{season}` or `{episode}`. Any series with a TVDB absolute order can use it, anime most often. Services can supply it, and [`--enrich`](../../guide/downloading.md#metadata-and-tagging) fills it in from TVDB. @@ -74,8 +74,8 @@ folder-kind keys. - **Type:** `list`  ·  **Default:** `[]` Conditionally replaces [`tag`](#tagging-naming-keys) for a given download. Each rule pairs a `when` -dict of conditions with the `tag` to use when they hold. Rules are evaluated in order and the -first match wins; if no rule matches, the global `tag` is kept. +dict of conditions with the `tag` to use when they hold. unshackle evaluates the rules in +order and the first match wins. If no rule matches, the global `tag` stays. ```yaml title="unshackle.yaml" tag_rules: @@ -88,30 +88,31 @@ tag_rules: Rule semantics: - Every condition in `when` must hold (they are AND-ed). -- A value may be a scalar or a list; a list matches if **any** entry matches. +- A value can be a scalar or a list. A list matches if **any** entry matches. - Comparison is case-insensitive string equality. -- The keys of `when` are the filename template variables that describe the release itself: +- The keys of `when` are the filename template variables for the release itself: `title_type`, `quality`, `resolution`, `source`, `video`, `hdr`, `hfr`, `audio`, `audio_channels`, `audio_full`, `atmos`, `dual`, `multi`, `dubbed`, `edition`, `repack`, - and `lang_tag`. Only these are available. The title's own fields (`title`, `year`, `season`, - `episode`, and so on) are added after the rules run and cannot be matched. -- An unknown key in `when` is a mistake: unshackle logs a warning that names the key and skips - that rule. A rule without conditions or without a `tag` is also warned about and skipped. + and `lang_tag`. Only these are available. unshackle adds the title's own fields (`title`, + `year`, `season`, `episode`, and so on) after it evaluates the rules, so a condition cannot + use them. +- An unknown key in `when` is a mistake: unshackle logs a warning that names the key + and skips that rule. A rule without conditions or without a `tag` is also warned about and skipped. -Put the more specific rules first, because the first match ends the search. +Put the more specific rules first, because unshackle stops at the first match. !!! warning "Match `hdr` on the exact value" Matching is exact, and `hdr` is a composite label for Dolby Vision titles: a DV title with an HDR10 base layer gives `DV.HDR`, and with an HDR10+ base layer `DV.HDR10P`. Plain `DV` - does **not** match those. List every variant you want, as the example above does. Note + does **not** match those. Name every variant you want, as the example above does. Note also that HDR10 gives `HDR`, not `HDR10`. The full set of values is `HDR`, `HDR10P`, `DV`, `DV.HDR`, `DV.HDR10P`, and `HLG`, as listed in the [variable table](../../guide/output-and-naming.md#template-variables). ### Operators -A condition value may start with a comparison operator, which is applied to the rest of the -value: +A condition value can start with a comparison operator, which unshackle applies to the rest of +the value: ```yaml title="unshackle.yaml" tag_rules: diff --git a/docs/reference/configuration/post-scripts.md b/docs/reference/configuration/post-scripts.md index ca14e971..cacf9aba 100644 --- a/docs/reference/configuration/post-scripts.md +++ b/docs/reference/configuration/post-scripts.md @@ -1,12 +1,12 @@ # Post-download scripts { #post-scripts } -Run your own script when a download finishes, with unshackle's metadata passed in as +Operate your own script when a download finishes, with unshackle's metadata passed in as `{variable}` placeholders. Use it to hand a finished file to an uploader, send a notification, or start a library scan. ## `post_scripts` -- **Type:** `list[dict]`  ·  **Default:** `[]` (no scripts run) +- **Type:** `list[dict]`  ·  **Default:** `[]` (unshackle operates no scripts) ```yaml title="unshackle.yaml" post_scripts: @@ -17,7 +17,7 @@ post_scripts: ``` Every entry that matches the event and mode runs, so one download can feed an uploader and -a notifier without a wrapper script. An entry may also be written as a plain string, which +a notifier without a wrapper script. You can also write an entry as a plain string, which is the same as `{command: "...", event: success, mode: file}`. | Key | Default | Meaning | @@ -26,15 +26,15 @@ is the same as `{command: "...", event: success, mode: file}`. | `event` | `success` | `success` when a file was written, `failure` when the download failed | | `mode` | `file` | `file`, `season`, or `run`. See [Modes](#modes) | -An entry whose `event` or `mode` is not one of the listed values is skipped and logs a -warning naming the bad value, once per command. An entry whose `command` cannot be split -into arguments, such as one with an unbalanced quote, is skipped with a warning too. Neither -stops the download or the other entries. +unshackle skips an entry whose `event` or `mode` is not one of the listed values, and logs a +warning that names the bad value, once per command. unshackle also skips an entry whose +`command` it cannot split into arguments, such as one with an unbalanced quote, and logs a +warning. Neither stops the download or the other entries. ## How the command runs -The command is split into arguments **first**, and variables are substituted into those -arguments afterwards. The process runs without a shell. +unshackle splits the command into arguments **first**, and then substitutes the variables +into those arguments. The process runs without a shell. This is what keeps a title like `Bob"; rm -rf ~` harmless: a value can never turn into an extra argument or a shell operator, and you never have to think about quoting. It also @@ -45,12 +45,12 @@ means a path containing spaces always arrives as a single argument on every OS. first stops a value becoming a *new* argument, but a value on its own can still be a whole argument. A bare `{title}` whose value is `--upload-file=/etc/passwd` would arrive as an option to your script, not as text. Write `--title={title}`, not a bare `{title}`, - so a service can never forge a flag. unshackle refuses to run a command when a + so a service can never forge a flag. unshackle refuses to operate a command when a substituted value would become an option-like token (one that starts with `-`), and logs a warning instead. !!! warning "No shell features, and name your interpreter" - Pipes, `&&`, `>` redirection, `~` and globs do not work, because no shell is involved. + Pipes, `&&`, `>` redirection, `~` and globs do not work, because unshackle uses no shell. If you want them, make the shell explicit: `bash -c "..."`. For the same reason there is no interpreter lookup. Write `python /opt/upload.py` @@ -58,7 +58,7 @@ means a path containing spaces always arrives as a single argument on every OS. Python or the one in its own virtual environment. On Windows, when a **fixed** part of an argument contains a space, quote the whole - argument (`"--out=C:\My Dir\{filename}"`), not just the value. Substituted values + argument (`"--out=C:\My Dir\{filename}"`), not only the value. Substituted values never need quoting on any OS. On Windows, do not target a `.bat` or `.cmd` file directly. Windows runs batch files @@ -79,7 +79,7 @@ already started. ## Variables Metadata comes from the same naming context that produced the file's name, built per output -file. So `{quality}` and `{hdr}` always describe the file being passed. Downloading +file. So `{quality}` and `{hdr}` always give the values for the file unshackle passes in. Downloading `-q 1080,2160 -r HDR10,SDR` produces four files, each with its own invocation and its own values. @@ -104,11 +104,11 @@ Every variable from your `output_template` is available here as well, including !!! warning "Empty variables become an empty string" `--tmdb={tmdb}` with no ID resolved hands your script `--tmdb=`, so treat an empty - string as absent. Write your script to handle every variable being empty. + string as absent. Write your script to accept an empty value for every variable. Empty is common, and not only when data is missing: - - Nothing was resolved. `{tmdb}` is empty when no metadata provider found the title, + - No metadata provider found an ID. `{tmdb}` is empty when no metadata provider found the title, the matching API key is missing, or the title is a movie you gave no ID for. - The title has no such field. Movies have no `{season}` or `{episode}`, and music has neither. @@ -153,10 +153,9 @@ track arrived in, such as `.flac`, because a music download is not muxed. ### Tagging IDs { #tagging-ids } -`{tmdb}`, `{imdb}` and `{tvdb}` carry the IDs unshackle is tagging the file with, read at -the moment the hook is dispatched rather than snapshotted at the start of the run. An -episode's TMDB ID is resolved by the title search inside the download loop, so a hook fires -with it already filled in. +`{tmdb}`, `{imdb}` and `{tvdb}` carry the IDs unshackle tags the file with. unshackle reads +them at the moment it sends the hook, and not at the start of the run. The title search inside +the download loop finds an episode's TMDB ID, so a hook fires with it already filled in. They hold an ID you passed with `--tmdb`, `--imdb` or `--tvdb`, plus a TMDB ID the episode search resolved. No such search runs for a movie, so a movie hook gets them empty unless you @@ -165,14 +164,14 @@ passed one. ### Sidecar files `{sidecars}` holds the sidecar subtitle files written beside the output, which are the only -extra files a download leaves next to it. Attachments such as fonts are muxed into the -container and temporary artwork is deleted, so neither appears. Expect `{sidecars}` to be +extra files a download leaves next to it. unshackle muxes attachments such as fonts into the +container and deletes temporary artwork, so neither appears. Expect `{sidecars}` to be empty unless [`subtitle.output_mode`](download.md#subtitle) is `sidecar` or `both`, which is not the default. A music download writes no sidecars at all. Paths are absolute and separated by a newline, because a newline is the only character that cannot appear in a filename on any supported OS. The whole list arrives as a single -argument, so no quoting is needed in your template. +argument, so your template needs no quoting. When one download writes several outputs (`-q 1080,720`), each output's folder gets its own copy of the sidecar files, and each invocation lists the copies sitting beside its own @@ -201,7 +200,7 @@ subs = [p for p in sys.argv[i].removeprefix("--subs=").split("\n") if p] hook fires after that point. This applies to every title type. If one episode of S01 fails, the S01 hook does not fire at all. -The hook is keyed to the folder, not to the season, so one season landing in several +unshackle attaches the hook to the folder, not to the season, so one season landing in several folders (`-q 1080,720` with a quality-dependent folder template) fires it once per folder, each with its own `{folder}`. A movie is a group of its own: a `season` hook fires for a movie's folder as soon as the movie finishes. @@ -269,32 +268,33 @@ entries, which have no `--postscript` equivalent. ## The REST API The API never accepts a post-script command. `postscript`, `post_script` and `post_scripts` -in a `POST /api/download` body are rejected with `400 INVALID_PARAMETERS`, because a command -arriving from an HTTP caller would be remote code execution. +in a `POST /api/download` body get `400 INVALID_PARAMETERS`. A command that arrives from an +HTTP caller would be remote code execution. -Scripts defined in `unshackle.yaml` do run for API jobs, music jobs included. Only the +unshackle does operate the scripts defined in `unshackle.yaml` for API jobs, music jobs +included. Only the command itself may not cross the network. ## Logging -The command that was run is logged twice: at `DEBUG` level on the console, and as a +unshackle logs the command it operated twice: at `DEBUG` level on the console, and as a `post_script_dispatch` entry in the [structured JSON debug log](../../guide/troubleshooting.md#the-structured-json-debug-log) -when that log is enabled. +when that log is on. -Both are masked the same way, honouring [`redact_paths`](misc.md), which is on by default. +unshackle masks both the same way, and honours [`redact_paths`](misc.md), which is on by default. Debug logs get shared in issue reports, and a hook command line is mostly paths. URL credentials and `token=` / `api_key=` query parameters are also masked. !!! warning "Do not hardcode secrets in the command" - An arbitrary secret in the command line, such as `-H "Authorization: Bearer "`, - cannot be recognised and is logged verbatim. Read the secret from a file or an + unshackle cannot recognise an arbitrary secret in the command line, such as + `-H "Authorization: Bearer "`, and logs it verbatim. Read the secret from a file or an environment variable inside your script rather than putting it in the `command`. ## Limitations -- No `success` hooks run under `--no-mux` or `--skip-dl`, because neither writes a muxed +- No `success` hook operates under `--no-mux` or `--skip-dl`, because neither writes a muxed output. A `failure` hook still runs under `--no-mux` if the download itself fails. -- `failure` hooks only run in `file` mode. +- `failure` hooks only operate in `file` mode. - There is no webhook sender. Call `curl` from a script instead. - Script output is not captured. Redirect it inside your own script if you want a log. diff --git a/docs/reference/configuration/services.md b/docs/reference/configuration/services.md index caf3756b..baea5084 100644 --- a/docs/reference/configuration/services.md +++ b/docs/reference/configuration/services.md @@ -7,19 +7,19 @@ Per-service configuration, keyed by the canonical service tag (for example `EXAMPLE1`, `EXAMPLE2`, `EXAMPLE3`). unshackle reads several well-known sub-keys out of each service's block: -- **`proxy_map`**: remaps the region query you passed to `-p/--proxy` for this service. A key is - that query, or `"provider:query"` when the flag also named a provider; the value is the query - asked of the proxy provider instead. It has no effect when `-p` was given a full proxy URL. +- **`proxy_map`**: remaps the region query you passed to `-p/--proxy` for this service. + A config key is that query, or `"provider:query"` when the flag also named a proxy provider. The value is + the query asked of the proxy provider instead. It has no effect when you give `-p` a full proxy URL. - **`title_map`**: an exact-match rename map applied to fetched titles (source name → desired name), so a service that names a title differently from your library still matches. - **`dl`**: per-service download defaults, using the same keys as the global [`dl`](download.md#dl) block. -Individual services may read any additional keys they define. The merged result (the service's -own `config.yaml` with this block layered on top) is handed to the service as `self.config`. +Individual services may read any additional keys they define. unshackle hands the merged result +(the service's own `config.yaml` with this block layered on top) to the service as `self.config`. User- or device-specific values (API keys, account IDs, device attributes) belong in this block -rather than in the service's own `config.yaml`, which holds shared defaults; see +rather than in the service's own `config.yaml`, which holds shared defaults. See [creating a service](../../dev/creating-a-service.md). ```yaml @@ -38,10 +38,10 @@ services: Per-service login credentials. Each value is a `username:password[:extra]` string, the same data as a `[username, password]` (or `[username, password, extra]`) list, or a dict of profile -name to either of those. With the dict form, `-p/--profile` selects the entry, and the -`default` key is used when no profile was given or the named profile is missing. These are -parsed into `Credential` objects; the credential's SHA-1 is also used as an account hash for -cache keys. +name to either of those. With the dict form, `-p/--profile` selects the entry, and unshackle +uses the `default` config key when you give no profile, or when the named profile is missing. +unshackle parses these into `Credential` objects. It also uses the credential's SHA-1 as an +account hash for cache keys. ```yaml credentials: @@ -52,22 +52,22 @@ credentials: ``` !!! tip "Cookies vs credentials" - Cookies are stored as files under [`directories.cookies`](directories.md), not in this key. + unshackle stores cookies as files under [`directories.cookies`](directories.md), not in this config key. A service's `authenticate()` accepts cookies, credentials, or both. ## `firefox_cookies` - **Type:** `dict` keyed by service tag  ·  **Default:** `{}` -Settings for extracting cookies directly from a local Firefox profile. A service block is -expected to provide `hosts` (a list of cookie hostnames; entries shorter than 3 characters -are ignored to prevent dumping the whole cookie store) and an optional `local_storage` -boolean to also pull matching entries from `webappsstore.sqlite`, which only services that -keep auth tokens in localStorage rather than in HTTP cookies need. Extraction is read-only. +Settings for extracting cookies directly from a local Firefox profile. A service block must give +`hosts`, a list of cookie hostnames. unshackle ignores an entry shorter than 3 characters, to +prevent dumping the whole cookie store. With an optional `local_storage` boolean, unshackle also +pulls matching entries from `webappsstore.sqlite`, which only services that keep auth tokens +in localStorage rather than in HTTP cookies need. Extraction is read-only. !!! note "Firefox does not need to be closed" The extractor copies **both** `cookies.sqlite` **and** its WAL file into a `0700` temp - directory, so writes Firefox has not yet flushed to the main DB are included. Extraction + directory, so the copy holds the writes Firefox has not yet flushed to the main DB. Extraction fails only if Firefox holds an exclusive write lock at the instant of the copy. The live profile is not modified. @@ -81,21 +81,21 @@ keep auth tokens in localStorage rather than in HTTP cookies need. Extraction is - **Type:** `dict`  ·  **Default:** `{}` Definitions of remote unshackle service servers, used by the `--remote` mode. Each entry is -named by you (pick it with `--server`, or omit that flag when only one is configured) and gives +named by you (pick it with `--server`, or do not write that flag when you configure only one) and gives the server's `url` (required), an optional `api_key`, an optional `auth_headers` list, an optional `server_cdm` boolean, and an optional `services` sub-dict of per-service local overrides such as `title_map`. `auth_headers` lists extra header names to send the API key in, tried before the defaults `X-Secret-Key` and `X-Api-Key`, which are always appended as fallbacks. unshackle sends the -first name; if the server answers `401`, it retries the same request with the next name, and -keeps the one that works for the rest of the session. Names you list keep your spelling and are -not repeated in the fallbacks, so `auth_headers: ["Authorization", "x-secret-key"]` is tried as -`Authorization`, `x-secret-key`, `X-Api-Key`. +first name. If the server answers `401`, it retries the same request with the next name, and +keeps the one that works for the rest of the HTTP session. Names you give keep your spelling and +are not repeated in the fallbacks, so unshackle tries +`auth_headers: ["Authorization", "x-secret-key"]` as `Authorization`, `x-secret-key`, `X-Api-Key`. In `--remote` mode unshackle turns the server's service list into synthetic CLI -commands that run against it, falling back to the tags in that `services` sub-dict when the -list cannot be fetched. Each synthetic command carries the server-side service's options and +commands that operate against it, falling back to the tags in that `services` sub-dict when +unshackle cannot fetch the list. Each synthetic command carries the server-side service's options and documentation, so `unshackle dl --remote -h` shows the same help text as it does on the server. See [remote sessions](../../dev/rest-api/remote-sessions.md) for the full setup. @@ -105,7 +105,7 @@ full setup. - **Type:** `dict`  ·  **Default:** `{}` Configuration for the `serve` command (the built-in REST API server). The full server guide -is the [REST API](../../dev/rest-api/index.md) section; these are the config keys. +is the [REST API](../../dev/rest-api/index.md) section. These are the config keys. | Sub-key | Type | Default | Description | |---------|------|---------|-------------| @@ -123,15 +123,16 @@ is the [REST API](../../dev/rest-api/index.md) section; these are the config key | `devices` | list | *(auto)* | Widevine devices offered; auto-filled from `directories.wvds`. | | `playready_devices` | list | *(auto)* | PlayReady devices; auto-filled from `directories.prds`. | -Each entry under `users` is keyed by that user's API key and may set its own `services`, +Each entry under `users` uses that user's API key as its name, and can set its own `services`, `devices`, and `playready_devices` allowlists, narrowing the global ones, plus an optional -`username` used as the log label for that key (defaults to a truncated form of the key). A -user with no `playready_devices` key gets no PlayReady access at all, not the global list. +`username` used as the log label for that API key (defaults to a truncated form of the API key). +A user with no `playready_devices` config key gets no PlayReady access at all, not the global +list. -`server_cdm` decides whether the server runs the CDM licensing for that key. It is `false` +`server_cdm` decides whether the server runs the CDM licensing for that API key. It is `false` unless the entry sets it, so a remote client configured with `server_cdm: true` is told to license with its own local CDM instead, and a client that asks anyway gets a `FORBIDDEN` -error. Because a download job always licenses with the server's CDM, a key without +error. Because a download job always licenses with the server's CDM, an API key without `server_cdm` also cannot submit or retry `/api/download` jobs. Keys that have no `users` entry, such as `api_secret`, keep server CDM access. diff --git a/docs/reference/configuration/vaults.md b/docs/reference/configuration/vaults.md index 2ef9cbd1..904aea6d 100644 --- a/docs/reference/configuration/vaults.md +++ b/docs/reference/configuration/vaults.md @@ -6,9 +6,9 @@ The full guide is at [Vaults](../../guide/vaults.md). Two keys configure them. - **Type:** `list[dict]`  ·  **Default:** `[]` -An ordered list of key-vault backends. unshackle queries them in order and reuses content -keys instead of re-licensing. Each entry needs a `type` (the backend module name) and a -`name`, plus backend-specific keys. +An ordered list of key-vault backends. unshackle queries them in order and reuses the +content keys instead of licensing them again. Each entry needs a `type` (the backend module +name) and a `name`, plus backend-specific keys. | `type` | Purpose | Required keys | Notes | |--------|---------|--------------|-------| @@ -32,12 +32,12 @@ key_vaults: ``` !!! note "Per-entry options" - - `no_push: true` makes a vault read-only (keys are fetched but never written to it). + - `no_push: true` makes a vault read-only (unshackle fetches content keys from it but never writes to it). - A vault of `type: API` whose `name` contains `decrypt_labs` auto-fills its `token` from [`decrypt_labs_api_key`](misc.md#external-api-keys) when not set inline. Vault `type` values are case-sensitive module names. - - An all-zero content key (32 zeros) is treated as "no key" everywhere and is never - stored. + - unshackle treats an all-zero content key (32 zeros) as "no key" everywhere and never + stores it. ## `vault_timeout` diff --git a/unshackle/commands/dl.py b/unshackle/commands/dl.py index bc1c9a86..3193174a 100644 --- a/unshackle/commands/dl.py +++ b/unshackle/commands/dl.py @@ -103,7 +103,7 @@ from unshackle.core.vaults import Vaults class SkippedSubtitle(TypedDict): """A subtitle skipped under ``--skip-subtitle-errors``. Accumulated as ``dl.skipped_subtitles``, - one entry per track; ``id`` and ``title`` identify which subtitle of which title was unavailable.""" + one entry per track. ``id`` and ``title`` identify which subtitle of which title was unavailable.""" id: str language: str @@ -136,7 +136,7 @@ def validate_metadata_ids( tvdb_id: Optional[int], anilist_id: Optional[Union[int, str]] = None, ) -> None: - """Reject ID flags that conflict, or that no configured provider could ever resolve. + """Reject ID flags that conflict, or that no configured metadata provider could ever find. --anilist stays outside the mutual exclusion: AniList knows no western IDs, so pairing it with one of the other three is how an anime title gets both. @@ -176,9 +176,9 @@ def validate_metadata_ids( def group_videos_by_variant(videos: list[Video], *, merge: bool) -> list[list[Video]]: """Group video tracks for muxing. - When ``merge`` is True, tracks sharing ``(height, range, codec)`` are grouped into one - file so only language varies within a group; different resolutions, ranges and codecs - stay in separate groups (separate files). When False, each track is its own group + When ``merge`` is True, this function groups tracks that share ``(height, range, codec)`` + into one file, so only language varies within a group. Different resolutions, ranges and + codecs stay in separate groups (separate files). When False, each track is its own group (one file per track, the default behaviour). Group order follows first-seen track order. """ if not merge: @@ -201,7 +201,7 @@ def title_wanted(candidate: Any, wanted: Collection[str]) -> bool: def post_script_group(candidate: Any) -> Any: - """Key a title by the folder its outputs share, for post-scripts in season mode.""" + """Group a title by the folder its outputs share, for post-scripts in season mode.""" if isinstance(candidate, Episode): return ("episode", candidate.title, candidate.season) if isinstance(candidate, Song): @@ -211,8 +211,8 @@ def post_script_group(candidate: Any) -> Any: def apply_service_dl_overrides(ctx: click.Context, service_dl_config: dict[str, Any], log: logging.Logger) -> None: - """Apply ``services..dl`` config onto ``ctx.params``. Explicit CLI/env values win; - defaults and global ``dl:`` default_map values are replaced.""" + """Apply ``services..dl`` config onto ``ctx.params``. Explicit CLI/env values win. + The config replaces defaults and global ``dl:`` default_map values.""" params_by_name = {param.name: param for param in ctx.command.params if param.name} for name, value in normalize_dl_config(service_dl_config).items(): param = params_by_name.get(name) @@ -245,16 +245,16 @@ def download_tracks_in_passes( skip_subtitle_errors: bool, on_subtitle_skipped: Callable[[Subtitle], None], ) -> None: - """Download a title's tracks so a skippable subtitle failure can't corrupt the rest. + """Download a title's tracks so a skippable subtitle failure cannot corrupt the rest. A failed track sets the process-global ``DOWNLOAD_CANCELLED`` event, making other in-flight tracks early-return without raising. With ``skip_subtitle_errors`` set, video/audio download - concurrently first; the subtitles then run one at a time (a concurrent pass would let one + concurrently first. The subtitles then download one at a time (a concurrent pass would let one failure's cancel silently drop the others, unrecorded), with the event cleared before each. - Video/audio failures stay fatal. The event is cleared on entry (stale cancel from a prior - title) and in ``finally`` (never leave it set for later code). + Video/audio failures stay fatal. This function clears the event on entry (stale cancel from a + prior title) and in ``finally`` (never leave it set for later code). - ``run_one(track, index)`` downloads a single track; ``on_subtitle_skipped(track)`` records a + ``run_one(track, index)`` downloads a single track. ``on_subtitle_skipped(track)`` records a subtitle whose download raised (handled here). """ DOWNLOAD_CANCELLED.clear() @@ -379,7 +379,7 @@ class dl: Show package installation suggestions for missing fonts. Args: - missing_fonts: List of font names that couldn't be found + missing_fonts: List of font names that could not be found """ if suggestions := suggest_font_packages(missing_fonts): self.log.info("Install font packages to improve subtitle rendering:") @@ -395,7 +395,7 @@ class dl: target_codec: Optional[Subtitle.Codec] = None, source_path: Optional[Path] = None, ) -> Path: - """Generate sidecar path: {base}.{lang}[.forced][.sdh].{ext}""" + """Make the sidecar path: {base}.{lang}[.forced][.sdh].{ext}""" lang_suffix = str(subtitle.language) if subtitle.language else "und" forced_suffix = ".forced" if subtitle.forced else "" sdh_suffix = ".sdh" if (subtitle.sdh or subtitle.cc) else "" @@ -418,7 +418,7 @@ class dl: """ Output subtitles as sidecar files, converting if needed. - Conversion runs on a temp copy, so each subtitle's own file is left untouched and can still be muxed. + Conversion runs on a temp copy, so each subtitle's own file stays untouched and mkvmerge can still mux it. """ created_paths: list[Path] = [] config.directories.temp.mkdir(parents=True, exist_ok=True) @@ -479,8 +479,8 @@ class dl: def post_script_ids(self) -> dict[str, Any]: """Tagging IDs as they stand right now. - Read per dispatch, never snapshotted: an Episode's TMDB ID is resolved by the - search inside the title loop, so a snapshot taken before it is always empty. + Read at each post-script call, never snapshotted: the search inside the title loop + finds an Episode's TMDB ID, so a snapshot taken before it is always empty. """ return {"tmdb": self.tmdb_id, "imdb": self.imdb_id, "tvdb": self.tvdb_id} @@ -1857,7 +1857,7 @@ class dl: """Pick the best audio per (language, codec) group from a sort_audio-ordered list. "Best" is the first track of each order-preserving candidate filter, so the - sort_audio ranking (descriptive-last, atmos, codec_priority, bitrate) decides the + sort_audio ranking (descriptive-last, Atmos, codec_priority, bitrate) decides the winner. With audio_description both the best standard and best descriptive track per group are kept. "best"/"all" tokens select every language present in sorted order. """ @@ -2191,7 +2191,7 @@ class dl: with console.status("Selecting tracks...", spinner="dots"): def resolve_excludes(excludes: list[str], _title: Title_T = title) -> list[str]: - """Resolve 'orig' against this title. An unresolvable 'orig' excludes nothing.""" + """Change 'orig' to this title's language. An 'orig' with no language excludes nothing.""" resolved: list[str] = [] for token in excludes: value = str(_title.language) if token == "orig" else token @@ -3533,7 +3533,7 @@ class dl: return self.service_anime if per_title is None else bool(per_title) def daily_hint(self, title: Optional[Title_T] = None) -> bool: - """Whether this title is daily/date-based content named by air date.""" + """Whether this title is daily content named by air date.""" per_title = getattr(title, "daily", None) return self.daily or (self.service_daily if per_title is None else bool(per_title)) @@ -3756,9 +3756,9 @@ class dl: def write_export(self, export: Path, title: Title_T, track: AnyTrack, drm: Any = None) -> None: """Write a shareable v2 export usable by ``unshackle import``. - Carries no session/cookies/dl-flags. Region (country code) is stored only when the - export used ``--proxy``, as an import geofence. Each track records only the licensed - DRM system; content keys live once under the track's ``keys``. ``drm`` may be None + Carries no HTTP session, cookies, or dl-flags. The export records the region (country + code) only when the export used ``--proxy``, as an import geofence. Each track records + only the licensed DRM system. Content keys live once under the track's ``keys``. ``drm`` may be None (DRM-free track) or a DRM system without ``to_dict``/``content_keys`` (e.g. ClearKey) - the track, manifest, chapter and attachment info is still exported. """ @@ -4483,8 +4483,8 @@ class dl: """ Get Service Cookies for Profile. - If firefox_cookies is configured for the service, cookies come from Firefox first. If that extraction - raises or returns nothing, the profile's cookie file is used instead and the failure is not logged. + If the config has firefox_cookies for the service, cookies come from Firefox first. If that extraction + raises or returns nothing, unshackle uses the profile's cookie file instead and does not log the failure. """ ff_settings = getattr(config, "firefox_cookies", {}).get(service) if ff_settings: @@ -4551,8 +4551,8 @@ class dl: ) -> Optional[object]: """ Get CDM for a specified service (either Local or Remote CDM). - Now supports quality-based selection when quality is provided. - Raises a ValueError if there's a problem getting a CDM. + Now supports quality-based selection when the caller gives a quality. + Raises a ValueError if there is a problem getting a CDM. """ # A per-request override (set by the REST API per job) takes precedence over the # global config, so a job can select a specific CDM device without mutating shared state. diff --git a/unshackle/commands/env.py b/unshackle/commands/env.py index 2317b8df..7813ad32 100644 --- a/unshackle/commands/env.py +++ b/unshackle/commands/env.py @@ -82,7 +82,7 @@ def get_dependencies() -> list[dict]: def clear_directory(path: Path) -> tuple[int, int]: - """Delete a directory's contents, returning (files_removed, freed_bytes); recreates the dir. + """Delete a directory's contents, returning (files_removed, freed_bytes). Recreates the dir. Skips task directories that belong to a running download. """ diff --git a/unshackle/commands/import.py b/unshackle/commands/import.py index 79ec5bec..ec803bcc 100644 --- a/unshackle/commands/import.py +++ b/unshackle/commands/import.py @@ -24,8 +24,8 @@ class ImportCommand: """ Reconstruct an exported download without re-contacting the service. - Re-fetches the manifest, injects the stored keys, then downloads/decrypts/muxes as a - normal `dl` run. Any `dl` options after the file are forwarded verbatim: + Re-fetches the manifest, injects the stored keys, then downloads/decrypts/muxes exactly + as the `dl` command does. unshackle forwards any `dl` options after the file verbatim: unshackle import export.json -r HDR10 --proxy US """ diff --git a/unshackle/commands/kv.py b/unshackle/commands/kv.py index 1efddff7..670afee8 100644 --- a/unshackle/commands/kv.py +++ b/unshackle/commands/kv.py @@ -102,7 +102,7 @@ def copy( ) -> None: """ Copy data from multiple Key Vaults into a single Key Vault. - Rows with matching KIDs are skipped unless there's no KEY set. + The copy skips rows with matching KIDs, unless the row has no content key. Existing data is not deleted or altered. The `to_vault_name` argument is the key vault you wish to copy data to. @@ -183,9 +183,9 @@ def sync( local_only: bool = False, ) -> None: """ - Ensure multiple Key Vaults copies of all keys as each other. - It's essentially just a bi-way copy between each vault. - To see the precise details of what it's doing between each + Make sure that every Key Vault has copies of all the content keys of the others. + It is essentially a bi-way copy between each vault. + To see the precise details of what it does between each provided vault, see the documentation for the `copy` command. """ if not len(vaults) > 1: @@ -216,9 +216,9 @@ def add(file: Path, service: str, vaults: list[str]) -> None: """ Add new Content Keys to Key Vault(s) by service. - File should contain one key per line in the format KID:KEY (HEX:HEX). + File should contain one content key per line in the format KID:KEY (HEX:HEX). Each line should have nothing else within it except for the KID:KEY. - Encoding is presumed to be UTF8. + unshackle reads the file as UTF8. """ if not file.exists(): raise click.ClickException(f"File provided ({file}) does not exist.") @@ -257,7 +257,7 @@ def add(file: Path, service: str, vaults: list[str]) -> None: def search_vault(vault: Vault, kid: str, services: list[str], log: logging.Logger) -> Optional[tuple[str, str]]: - """Return the (service, key) of the first service table in a vault holding the KID.""" + """Return the service and content key from the first service table in a vault holding the KID.""" def probe(svc: str) -> Optional[tuple[str, str]]: try: @@ -294,13 +294,13 @@ def search_vault(vault: Vault, kid: str, services: list[str], log: logging.Logge ) def search(kid: str, service: Optional[str], vault_name: Optional[str]) -> None: """ - Search configured Key Vault(s) for a KID and report any matching KEY. + Examine configured Key Vault(s) for a KID and report the content key it finds. - KID must be 32 hex characters (no dashes). If --service is omitted, every - service table in each vault is scanned; vaults that cannot list their - tables are probed with every locally installed service tag instead, so - passing --service is much faster. If --vault is omitted, every vault in - the config is searched. + KID must be 32 hex characters (no dashes). If you do not give --service, + unshackle scans every service table in each vault. For a vault that cannot + show its tables, unshackle probes every locally installed service tag + instead, so --service is much faster. If you do not give --vault, unshackle + examines every vault in the config. """ log = logging.getLogger("kv") @@ -355,7 +355,7 @@ def search(kid: str, service: Optional[str], vault_name: Optional[str]) -> None: @kv.command() @click.argument("vaults", nargs=-1, type=click.UNPROCESSED) def prepare(vaults: list[str]) -> None: - """Create Service Tables on Vaults if not yet created.""" + """Make Service Tables on Vaults if they do not exist yet.""" log = logging.getLogger("kv") vaults_ = load_vaults(vaults) diff --git a/unshackle/commands/prd.py b/unshackle/commands/prd.py index e8dcf1a2..f0d81a15 100644 --- a/unshackle/commands/prd.py +++ b/unshackle/commands/prd.py @@ -21,7 +21,7 @@ from unshackle.core.constants import context_settings context_settings=context_settings, ) def prd() -> None: - """Manage creation of PRD (Playready Device) files.""" + """Manage creation of PRD (PlayReady Device) files.""" @prd.command() @@ -49,7 +49,7 @@ def new( signing_key: Optional[Path], output: Optional[Path], ) -> None: - """Create a new .PRD PlayReady Device file. + """Make a new .PRD device file. Accepts either paths to a group key and certificate or a single directory containing ``zgpriv.dat`` and ``bgroupcert.dat``. @@ -159,7 +159,7 @@ def reprovision_device( signing_key: Optional[Path], output: Optional[Path] = None, ) -> None: - """Reprovision a Playready Device (.prd) file.""" + """Reprovision a PlayReady Device (.prd) file.""" if not prd_path.is_file(): raise click.UsageError("prd_path: Not a path to a file, or it doesn't exist.", ctx) @@ -231,7 +231,7 @@ def test( ckt: str, security_level: str, ) -> None: - """Test a Playready Device on the Microsoft demo server.""" + """Do a test of a PlayReady device on the Microsoft demo server.""" if not device.is_file(): raise click.UsageError("device: Not a path to a file, or it doesn't exist.", ctx) diff --git a/unshackle/commands/serve.py b/unshackle/commands/serve.py index 049ccf14..ad932c62 100644 --- a/unshackle/commands/serve.py +++ b/unshackle/commands/serve.py @@ -62,8 +62,8 @@ def serve( \b CDM ENDPOINTS: - - Widevine: /{device}/open, /{device}/close/{session_id}, etc. - - PlayReady: /playready/{device}/open, /playready/{device}/close/{session_id}, etc. + - Widevine: /{device}/open, /{device}/close/{session_id}, and others + - PlayReady: /playready/{device}/open, /playready/{device}/close/{session_id}, and others \b You may serve with Caddy at the same time with --caddy. You can use Caddy diff --git a/unshackle/commands/util.py b/unshackle/commands/util.py index 4443406b..c5e7f9ec 100644 --- a/unshackle/commands/util.py +++ b/unshackle/commands/util.py @@ -69,7 +69,7 @@ def refresh_services() -> None: def crop(path: Path, aspect: str, letter: bool, offset: int, preview: bool) -> None: """ Losslessly crop H.264 and H.265 video files at the bit-stream level. - You may provide a path to a file, or a folder of mkv and/or mp4 files. + You may give a path to a file, or a folder of mkv and/or mp4 files. Note: If you notice that the values you put in are not quite working, try tune -o/--offset. This may be necessary on videos with sub-sampled chroma. @@ -77,8 +77,8 @@ def crop(path: Path, aspect: str, letter: bool, offset: int, preview: bool) -> N Do note that you may not get an ideal lossless cropping result on some cases, again due to sub-sampled chroma. - It's recommended that you try -o about 10 or so pixels and lower it until - you get as close in as possible. Do make sure it's not over-cropping either + Try -o about 10 or so pixels and lower it until you get as close in as + possible. Do make sure it is not over-cropping either as it may go from being 2px away from a perfect crop, to 20px over-cropping again due to sub-sampled chroma. """ @@ -179,10 +179,10 @@ def crop(path: Path, aspect: str, letter: bool, offset: int, preview: bool) -> N def range_(path: Path, full: bool, preview: bool) -> None: """ Losslessly set the Video Range flag to full or limited at the bit-stream level. - You may provide a path to a file, or a folder of mkv and/or mp4 files. + You may give a path to a file, or a folder of mkv and/or mp4 files. If you ever notice blacks not being quite black, and whites not being quite white, - then you're video may have the range set to the wrong value. Flip its range to the + then your video may have the range set to the wrong value. Flip its range to the opposite value and see if that fixes it. """ if not binaries.FFMPEG: @@ -264,12 +264,12 @@ def range_(path: Path, full: bool, preview: bool) -> None: ) def test(path: Path, map_: str) -> None: """ - Decode an entire video and check for any corruptions or errors using FFmpeg. - You may provide a path to a file, or a folder of mkv and/or mp4 files. + Decode an entire video and examine it for any corruptions or errors using FFmpeg. + You may give a path to a file, or a folder of mkv and/or mp4 files. - Tests all streams within the file by default. Subtitles cannot be tested. - You may choose specific streams using the -m/--map parameter. E.g., - '0:v:0' to test the first video stream, or '0:a' to test all audio streams. + Does a test of all tracks in the file by default. You cannot test subtitle + tracks. You may choose specific tracks with the -m/--map parameter. + For example, '0:v:0' for the first video track, or '0:a' for all audio tracks. """ if not binaries.FFMPEG: raise click.ClickException('FFmpeg executable "ffmpeg" not found but is required.') diff --git a/unshackle/commands/wvd.py b/unshackle/commands/wvd.py index 92ff5c5e..0e92d0d6 100644 --- a/unshackle/commands/wvd.py +++ b/unshackle/commands/wvd.py @@ -72,7 +72,7 @@ def delete(names: list[str]) -> None: @click.argument("path", type=Path) def parse(path: Path) -> None: """ - Parse a .WVD Widevine Device file to check information. + Parse a .WVD device file to examine its information. Relative paths are relative to the WVDs directory. """ try: @@ -115,7 +115,7 @@ def parse(path: Path) -> None: @click.argument("out_dir", type=Path, nargs=1) def dump(wvd_paths: list[Path], out_dir: Path) -> None: """ - Extract data from a .WVD Widevine Device file to a folder structure. + Extract data from a .WVD device file to a folder structure. If the path is relative, with no file extension, it will dump the WVD in the WVDs directory. @@ -209,7 +209,7 @@ def new( output: Optional[Path], ) -> None: """ - Create a new .WVD Widevine provision file. + Make a new .WVD Widevine provision file. name: The origin device name of the provided data. e.g. `Nexus 6P`. You do not need to specify the security level, that will be done automatically. @@ -217,7 +217,7 @@ def new( client_id: A binary blob file which follows the Widevine ClientIdentification protobuf schema. file_hashes: A binary blob file with follows the Widevine FileHashes protobuf schema. - Also known as VMP as it's used for VMP (Verified Media Path) assurance. + Also known as VMP, because VMP (Verified Media Path) assurance uses it. """ try: # TODO: Remove need for name, create name based on Client IDs ClientInfo values diff --git a/unshackle/core/api/download_manager.py b/unshackle/core/api/download_manager.py index 746e5317..1cbe6e58 100644 --- a/unshackle/core/api/download_manager.py +++ b/unshackle/core/api/download_manager.py @@ -28,7 +28,7 @@ SENSITIVE_PARAM_KEYS = ("credential", "credentials", "password", "token", "api_k def redact_parameters(parameters: Dict[str, Any]) -> Dict[str, Any]: - """Return a copy of job parameters with secrets masked, safe to serialize.""" + """Return a copy of job parameters with secrets masked, safe to serialise.""" if not isinstance(parameters, dict): return parameters redacted = dict(parameters) @@ -64,7 +64,7 @@ def secret_values(parameters: Dict[str, Any]) -> List[str]: def _redact_text(text: Optional[str], parameters: Dict[str, Any]) -> Optional[str]: """Mask proxy userinfo and any known parameter secrets that leaked into a free-text field - (error message / details / traceback / worker stderr) before it is returned via the API.""" + (error message / details / traceback / job worker stderr) before the API returns the field.""" return redact_text(text, secret_values(parameters)) @@ -175,7 +175,7 @@ def history_path() -> Path: def history_limit() -> int: - """Max history entries to retain (serve.history_limit, default 100; <= 0 means unlimited).""" + """Max history entries to retain (serve.history_limit, default 100). A value of 0 or less means no limit.""" from unshackle.core.config import config try: @@ -227,7 +227,10 @@ def record_job_history(job: DownloadJob) -> None: def read_job_history(limit: int = 100, service: Optional[str] = None) -> List[Dict[str, Any]]: - """Read persisted job history, newest first; corrupt lines are skipped, missing file = empty.""" + """Read persisted job history, newest first. + + This function ignores corrupt lines. A missing file gives no entries. + """ path = history_path() if not path.exists(): return [] @@ -255,10 +258,10 @@ def read_job_history(limit: int = 100, service: Optional[str] = None) -> List[Di def delete_job_history(job_id: str, allowed: Optional[set] = None) -> bool: - """Remove a history entry by job_id (rewriting the file). Returns True if one was deleted. + """Remove a history entry by job_id (rewriting the file). Returns True if it removed one. - When `allowed` is given, an entry outside the caller's service allowlist is treated as - absent (returns False) so it can't be deleted by a restricted key. + When you give `allowed`, this function treats an entry outside the caller's service + allowlist as absent (returns False), so a restricted API key cannot delete the entry. """ path = history_path() if not path.exists(): @@ -302,7 +305,7 @@ def to_enum(values: List[str], enum_cls: type[Enum]) -> List[Enum]: def normalize_sub_format(sub_format_raw: str) -> Any: - """Normalize an API sub_format string to what dl.py expects. + """Normalise an API sub_format string to what dl.py expects. "original" is a sentinel meaning "keep source format" (dl.py skips conversion), so it must pass through unchanged to match the CLI (SubtitleCodecChoice returns the same string). Any @@ -323,10 +326,10 @@ def perform_download( params: Dict[str, Any], progress_callback: Optional[Callable[[Dict[str, Any]], None]] = None, ) -> List[str]: - """Execute the synchronous download logic for a job. + """Do the synchronous download work for a job. - `params` holds the API's string form of the `dl` options and is normalized in place before - dl.result() sees it: vcodec and range names become enums, `slow` accepts "MIN-MAX", a two-item + `params` holds the API's string form of the `dl` options. This function normalizes `params` in + place before dl.result() sees it: vcodec and range names become enums, `slow` accepts "MIN-MAX", a two-item list, or True (which means 60-120), and `wanted` accepts forms such as "S01E01", "S01-S03", "s1e1", or "1x1" as well as the internal "SxE" form. Music takes track numbers in the same field, such as "1-5" or "2x3" for disc 2 track 3. @@ -655,7 +658,7 @@ class DownloadQueueManager: ) def create_job(self, service: str, title_id: str, owner_key: Optional[str] = None, **parameters) -> DownloadJob: - """Create a new download job and add it to the queue.""" + """Make a new download job and add it to the queue.""" job_id = str(uuid.uuid4()) job = DownloadJob( job_id=job_id, @@ -678,11 +681,11 @@ class DownloadQueueManager: return self._jobs.get(job_id) def list_jobs(self) -> List[DownloadJob]: - """List all jobs.""" + """Get all jobs.""" return list(self._jobs.values()) def subscribe(self, job_id: str) -> asyncio.Queue: - """Register a listener for a job's events, returning the queue it receives them on.""" + """Add a listener for a job's events, and return the queue it receives them on.""" queue: asyncio.Queue = asyncio.Queue(maxsize=100) self._subscribers.setdefault(job_id, []).append(queue) return queue @@ -707,7 +710,7 @@ class DownloadQueueManager: self.put(queue, payload) def publish_terminal(self, job: DownloadJob) -> None: - """Publish a job's final state, then close every listener's stream.""" + """Publish a job's final state, then close every listener's event stream.""" self.publish(job, job.status.value) for queue in self._subscribers.pop(job.job_id, []): self.put(queue, None) @@ -724,7 +727,7 @@ class DownloadQueueManager: queue.put_nowait(item) def cancel_job(self, job_id: str) -> bool: - """Cancel a job if it's queued or downloading.""" + """Cancel a job that has the queued or the downloading status.""" job = self._jobs.get(job_id) if not job: return False @@ -810,7 +813,7 @@ class DownloadQueueManager: return len(jobs_to_remove) async def start_workers(self): - """Start worker tasks to process the download queue.""" + """Start job worker tasks to process the download queue.""" if self._workers_started: return @@ -870,7 +873,7 @@ class DownloadQueueManager: await asyncio.gather(*self._active_downloads.values(), return_exceptions=True) async def download_worker(self, worker_name: str): - """Worker task that processes jobs from the queue.""" + """Job worker task that processes jobs from the queue.""" log.debug(f"Download worker {worker_name} started") while not self._shutdown_event.is_set(): @@ -920,7 +923,7 @@ class DownloadQueueManager: log.error(f"Worker {worker_name} error: {e}") async def execute_download(self, job: DownloadJob): - """Execute the actual download for a job.""" + """Do the actual download for a job.""" log.info(f"Executing download for job {job.job_id}") try: @@ -949,7 +952,7 @@ class DownloadQueueManager: raise async def run_download_async(self, job: DownloadJob) -> List[str]: - """Invoke a worker subprocess to execute the download.""" + """Invoke a job worker subprocess to do the download.""" payload = { "job_id": job.job_id, @@ -1110,7 +1113,7 @@ class DownloadQueueManager: pass async def cleanup_worker(self): - """Worker that periodically cleans up old jobs.""" + """Job worker that periodically cleans up old jobs.""" while not self._shutdown_event.is_set(): try: await asyncio.sleep(3600) diff --git a/unshackle/core/api/download_worker.py b/unshackle/core/api/download_worker.py index 45dc17fb..5767b7b0 100644 --- a/unshackle/core/api/download_worker.py +++ b/unshackle/core/api/download_worker.py @@ -1,4 +1,4 @@ -"""Standalone worker process entry point for executing download jobs.""" +"""Standalone job worker process entry point for executing download jobs.""" from __future__ import annotations diff --git a/unshackle/core/api/errors.py b/unshackle/core/api/errors.py index 6e891c6b..4ff65301 100644 --- a/unshackle/core/api/errors.py +++ b/unshackle/core/api/errors.py @@ -54,7 +54,7 @@ class APIError(Exception): error_code: Standardized error code from APIErrorCode enum message: User-friendly error message details: Additional structured error information - retryable: Whether the operation can be retried + retryable: Whether the caller can do the operation again http_status: HTTP status code to return (default based on error_code) """ @@ -108,12 +108,12 @@ def build_error_response( extra_debug_info: dict[str, Any] | None = None, ) -> web.Response: """ - Build a structured JSON error response. + Assemble a structured JSON error response. Args: error: APIError or generic Exception to convert to response debug_mode: Whether to include technical debug information - extra_debug_info: Additional debug info (stderr, stdout, etc.) + extra_debug_info: Additional debug info such as stderr and stdout Returns: aiohttp JSON response with structured error data @@ -172,7 +172,7 @@ def categorize_exception( Args: exc: The exception to categorize - context: Optional context (service name, operation type, etc.) + context: Optional context such as the service tag and the operation type Returns: APIError with appropriate error code and details @@ -277,10 +277,10 @@ def handle_api_exception( extra_debug_info: dict[str, Any] | None = None, ) -> web.Response: """ - Convenience function to categorize an exception and build an error response. + Convenience function to categorize an exception and assemble an error response. Args: - exc: The exception to handle + exc: The exception to categorize context: Optional context information debug_mode: Whether to include debug information extra_debug_info: Additional debug info diff --git a/unshackle/core/api/handlers.py b/unshackle/core/api/handlers.py index e875149e..11a70426 100644 --- a/unshackle/core/api/handlers.py +++ b/unshackle/core/api/handlers.py @@ -114,7 +114,7 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str] inside ``__init__``, so the lightweight ``resolve_server_cdm`` stub is not enough for list_titles / list_tracks / search. Mirrors ``dl.get_cdm`` selection logic but skips the quality-tier shortcuts (no track context yet) and falls back to the stub - if no device is configured or loading fails. + if the configuration has no device, or if loading fails. """ from unshackle.core.cdm import load_cdm from unshackle.core.config import config as app_config @@ -168,10 +168,10 @@ def build_parent_ctx( service_config: dict, extra_params: Optional[Dict[str, Any]] = None, ) -> Any: - """Build a parent click Context for invoking a service.cli via ctx.invoke(). + """Assemble a parent click Context for invoking a service.cli through ctx.invoke(). - The service's CLI callback uses ``ctx.parent.params`` (proxy, range_, vcodec, etc.) - and ``ctx.obj`` (ContextData). Both flow through Click's parent chain. + The service's CLI callback uses ``ctx.parent.params`` (proxy, range_, vcodec, and the + other dl options) and ``ctx.obj`` (ContextData). Both flow through Click's parent chain. """ import click @@ -200,10 +200,10 @@ def instantiate_service( ) -> Any: """Instantiate a service by invoking its click cli through Click. - Click fills option defaults via ``param.get_default()`` and runs type coercion, - so we no longer have to inspect ``__init__`` or stitch defaults by hand. Extra - kwargs are pulled from ``data`` when the key matches a cli option name and is - not in the transport-key blocklist. + Click fills option defaults through ``param.get_default()`` and runs type coercion, + so we no longer have to inspect ``__init__`` or stitch defaults by hand. This function + pulls extra kwargs from the ``data`` entries whose names match a cli option name + and are not in the transport-key blocklist. """ cli_params = getattr(getattr(service_module, "cli", None), "params", []) or [] cli_param_names = {p.name for p in cli_params if hasattr(p, "name") and p.name} @@ -217,9 +217,9 @@ def instantiate_service( def setup_list_service(data: Dict[str, Any], normalized_service: str, profile: Optional[str], title_id: str) -> Any: - """Build and authenticate a service instance for list_titles / list_tracks. + """Assemble and authenticate a service instance for list_titles / list_tracks. - Runs the shared preamble: load yaml → resolve proxy → load CDM → build ctx → + Runs the shared preamble: load yaml → get proxy → load CDM → assemble ctx → instantiate → authenticate. Raises APIError on proxy failure. """ from unshackle.commands.dl import dl @@ -257,7 +257,7 @@ def setup_list_service(data: Dict[str, Any], normalized_service: str, profile: O def get_allowed_services(request: Optional[web.Request] = None) -> Optional[List[str]]: """Get effective service allowlist considering global + per-key config. - Returns None if all services are allowed. + Returns None if no allowlist restricts the services. """ global_allowed = config.serve.get("services") global_set: Optional[set[str]] = None @@ -287,10 +287,10 @@ def get_allowed_services(request: Optional[web.Request] = None) -> Optional[List def server_cdm_allowed(request: Optional[web.Request] = None) -> bool: - """Whether the calling key may have the server run the CDM licensing. + """Whether the calling API key may have the server operate the CDM licensing. - Configured keys opt in with ``server_cdm: true``; keys absent from - ``serve.users`` (the admin secret) keep full access. + Configured API keys opt in with ``server_cdm: true``. An API key absent from + ``serve.users`` (the admin secret) keeps full access. """ if not request: return True @@ -312,7 +312,7 @@ CORS_HEADERS = { def request_secret_key(request: web.Request) -> Optional[str]: - """The caller's key: the X-Secret-Key header, or on the events route the secret_key + """The caller's API key: the X-Secret-Key header, or on the events route the secret_key query param (EventSource cannot send headers).""" key = request.headers.get("X-Secret-Key") if not key: @@ -323,17 +323,17 @@ def request_secret_key(request: web.Request) -> Optional[str]: def caller_key(request: Optional[web.Request] = None) -> str: - """The authenticating key for a request, or 'anonymous' when unauthenticated.""" + """The authenticating API key for a request, or 'anonymous' when unauthenticated.""" if not request: return "anonymous" return request_secret_key(request) or "anonymous" def owns_job(job: Any, request: Optional[web.Request] = None) -> bool: - """Whether the calling key owns this job. + """Whether the calling API key owns this job. - Jobs created without an owner (no-key mode / legacy) stay shared; otherwise a job is - only visible to the key that created it (constant-time compare). + Jobs created without an owner (no-key mode / legacy) stay shared. Otherwise, only the + API key that created the job can see it (constant-time compare). """ import hmac @@ -344,7 +344,7 @@ def owns_job(job: Any, request: Optional[web.Request] = None) -> bool: def validate_service(service_tag: str, request: Optional[web.Request] = None) -> Optional[str]: - """Validate, normalize, and check allowlist for service tag.""" + """Validate and normalise the service tag, and examine the allowlist.""" if not isinstance(service_tag, str): return None try: @@ -448,7 +448,7 @@ def extract_manifests(tracks) -> List[Dict[str, Any]]: Serializes DASH and ISM manifest XML as zlib-compressed base64 strings so the client can reconstruct track.data locally. HLS tracks download - directly from their URL so no manifest serialization is needed. + directly from their URL, so this function does not serialise their manifest. """ import base64 import zlib @@ -494,7 +494,7 @@ def extract_manifests(tracks) -> List[Dict[str, Any]]: def serialize_drm(drm_list) -> Optional[List[Dict[str, Any]]]: - """Serialize DRM objects to JSON-serializable list.""" + """Serialise DRM objects to JSON-serializable list.""" if not drm_list: return None @@ -596,8 +596,8 @@ def original_audio_ids(tracks: List[Audio], title: Title_T) -> set: def serialize_audio_track(track: Audio, include_url: bool = False, is_original: bool = False) -> Dict[str, Any]: """Convert audio track to JSON-serializable dict. - Resolve is_original with original_audio_ids so the flag always agrees with the - track 'orig' would actually download. + Get is_original from original_audio_ids so the flag always agrees with the + track 'orig' would download. """ codec_name = enum_name(track.codec) @@ -636,7 +636,7 @@ def serialize_subtitle_track(track: Subtitle, include_url: bool = False) -> Dict async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle search request.""" + """Answer the request to find titles.""" from unshackle.commands.dl import dl service_tag = data.get("service") @@ -718,7 +718,7 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle list-titles request.""" + """Answer the list-titles request.""" require_fields(data, "service", "title_id") service_tag = data.get("service") title_id = data.get("title_id") @@ -756,7 +756,7 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle list-tracks request.""" + """Answer the list-tracks request.""" require_fields(data, "service", "title_id") service_tag = data.get("service") title_id = data.get("title_id") @@ -1062,13 +1062,13 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]: def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request] = None) -> None: """Enforce serve-config gates on per-job cdm overrides and client-supplied credentials. - A per-request `cdm` selects a server-side device, so it is gated here rather than honoured - blindly. `serve.cdm_overrides` opts in: a list permits only those device names, or `true` + A per-request `cdm` selects a server-side device, so this function gates it here rather than + honour it blindly. `serve.cdm_overrides` opts in: a list permits only those device names, or `true` permits any (for a single trusted client). Unset/false rejects every override. A per-request `credential` (or `credentials` map) authenticates the job with client-supplied secrets instead of the server-side credentials. Gate it behind `serve.allow_job_credentials` - (default off) so a default deployment stays locked to its own credentials; mirrors the CDM gate. - A download job licenses DRM in-process with the server's own CDM, so a key without + (default off) so a default deployment stays locked to its own credentials. This mirrors the + CDM gate. A download job licenses DRM in-process with the server's own CDM, so an API key without ``server_cdm`` cannot submit or retry jobs. """ if not server_cdm_allowed(request): @@ -1097,7 +1097,7 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request async def download_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle download request - create and queue a download job.""" + """Answer the download request: make a download job and add it to the queue.""" from unshackle.core.api.download_manager import get_download_manager require_fields(data, "service", "title_id") @@ -1168,7 +1168,7 @@ async def download_handler(data: Dict[str, Any], request: Optional[web.Request] async def list_download_jobs_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle list download jobs request with optional filtering and sorting.""" + """Answer the request to show download jobs, with optional filtering and sorting.""" from unshackle.core.api.download_manager import get_download_manager try: @@ -1204,7 +1204,7 @@ async def list_download_jobs_handler(data: Dict[str, Any], request: Optional[web reverse = sort_order == "desc" def get_sort_key(job): - """Get the sorting key value, handling None values.""" + """Get the value to sort on, and give a None value a default.""" value = getattr(job, sort_by, None) if value is None: if sort_by in ["created_time", "started_time", "completed_time"]: @@ -1237,7 +1237,7 @@ async def list_download_jobs_handler(data: Dict[str, Any], request: Optional[web async def get_download_job_handler(job_id: str, request: Optional[web.Request] = None) -> web.Response: - """Handle get specific download job request.""" + """Answer the request for one specific download job.""" from unshackle.core.api.download_manager import get_download_manager try: @@ -1331,7 +1331,7 @@ async def download_job_events_handler(job_id: str, request: web.Request) -> web. async def cancel_download_job_handler(job_id: str, request: Optional[web.Request] = None) -> web.Response: - """Handle cancel/remove download job request.""" + """Answer the cancel/remove download job request.""" from unshackle.core.api.download_manager import TERMINAL_STATUSES, get_download_manager try: @@ -1374,7 +1374,7 @@ async def cancel_download_job_handler(job_id: str, request: Optional[web.Request async def clear_finished_download_jobs_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle clear finished download jobs request.""" + """Answer the clear finished download jobs request.""" from unshackle.core.api.download_manager import get_download_manager try: @@ -1395,7 +1395,7 @@ async def clear_finished_download_jobs_handler(request: Optional[web.Request] = async def retry_download_job_handler(job_id: str, request: Optional[web.Request] = None) -> web.Response: - """Handle retry download job request - enqueue a new job with the original's parameters.""" + """Answer the retry download job request: enqueue a new job with the original's parameters.""" from unshackle.core.api.download_manager import TERMINAL_STATUSES, get_download_manager try: @@ -1453,7 +1453,7 @@ async def retry_download_job_handler(job_id: str, request: Optional[web.Request] async def prioritize_download_job_handler(job_id: str, request: Optional[web.Request] = None) -> web.Response: - """Handle prioritize download job request - move a queued job to the front of the queue.""" + """Answer the prioritise download job request: move a queued job to the front of the queue.""" from unshackle.core.api.download_manager import JobStatus, get_download_manager try: @@ -1494,7 +1494,7 @@ CONFIG_SECRET_KEY_RE = re.compile(r"secret|password|token|api_key|credential", r def redact_config(value: Any) -> Any: - """Recursively mask secret-looking keys and URL userinfo; stringify paths.""" + """Recursively mask secret-looking config keys and URL userinfo. Stringify paths.""" if isinstance(value, dict): return { str(k): (REDACTED if v and CONFIG_SECRET_KEY_RE.search(str(k)) else redact_config(v)) @@ -1510,7 +1510,7 @@ def redact_config(value: Any) -> Any: async def profiles_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle list credential profiles request.""" + """Answer the request for the credential profiles.""" try: allowed = get_allowed_services(request) profiles: Dict[str, List[str]] = {} @@ -1534,7 +1534,7 @@ async def profiles_handler(request: Optional[web.Request] = None) -> web.Respons async def server_config_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle read-only effective server config request (secrets redacted).""" + """Answer the read-only effective server config request (secrets redacted).""" from unshackle.core.api.download_manager import get_download_manager try: @@ -1574,7 +1574,7 @@ async def server_config_handler(request: Optional[web.Request] = None) -> web.Re async def download_history_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle persisted download history request.""" + """Answer the persisted download history request.""" from unshackle.core.api.download_manager import read_job_history try: @@ -1641,7 +1641,7 @@ def require_no_active_downloads(operation: str) -> None: async def clear_cache_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle clear cache directory request.""" + """Answer the clear cache directory request.""" from unshackle.commands.env import clear_directory try: @@ -1658,7 +1658,7 @@ async def clear_cache_handler(request: Optional[web.Request] = None) -> web.Resp async def clear_temp_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle clear temp directory request.""" + """Answer the clear temp directory request.""" from unshackle.commands.env import clear_directory try: @@ -1675,7 +1675,7 @@ async def clear_temp_handler(request: Optional[web.Request] = None) -> web.Respo async def refresh_services_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle refresh of service repos configured in directories.services.""" + """Answer the request to refresh the service repos configured in directories.services.""" from unshackle.core.service_repo import is_repo_spec, refresh_repo try: @@ -1703,7 +1703,7 @@ VERSION_RE = re.compile(r"\d+\.\d+(?:\.\d+)*") def binary_version(path: Any) -> Optional[str]: - """Best-effort version probe of a binary; None when nothing parseable.""" + """Best-effort version probe of a binary. Returns None when nothing is parseable.""" import subprocess for flag in ("--version", "-version"): @@ -1720,7 +1720,7 @@ def binary_version(path: Any) -> Optional[str]: async def env_check_handler(request: Optional[web.Request] = None) -> web.Response: - """Handle environment dependency check request.""" + """Answer the environment dependency check request.""" from unshackle.commands.env import get_dependencies def run_checks() -> List[Dict[str, Any]]: @@ -1778,7 +1778,7 @@ def create_service_instance( proxy_providers: list, profile: Optional[str], ) -> Any: - """Create and authenticate a service instance. + """Make and authenticate a service instance. Supports client-sent credentials/cookies (for remote-dl) with fallback to server-local config (for backward compatibility). @@ -1870,10 +1870,10 @@ def create_service_instance( async def session_create_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response: - """Handle session creation: authenticate + get titles + get tracks + get chapters. + """Answer the remote session creation: authenticate + get titles + get tracks + get chapters. This is the main entry point for remote-dl clients. It creates a persistent - session on the server with the authenticated service instance, fetches all + remote session on the server with the authenticated service instance, fetches all titles and tracks, and returns everything the client needs for track selection. """ from unshackle.core.api.session_store import get_session_store @@ -1983,11 +1983,11 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req async def session_titles_handler(session_id: str, request: Optional[web.Request] = None) -> web.Response: - """Get titles for the authenticated session. + """Get titles for the authenticated remote session. - Called after session/create. This is separate from auth so that - interactive auth flows (OTP, captcha) can complete before titles - are fetched. + Called after `session/create`. This is separate from auth so that + interactive auth flows (OTP, captcha) can complete before the server + fetches titles. """ session = await get_validated_session(session_id, request) require_authenticated(session) @@ -2028,11 +2028,11 @@ async def session_titles_handler(session_id: str, request: Optional[web.Request] async def session_tracks_handler( data: Dict[str, Any], session_id: str, request: Optional[web.Request] = None ) -> web.Response: - """Get tracks and chapters for a specific title in the session. + """Get tracks and chapters for a specific title in the remote session. - Called per-title by the client after session/create returns titles. + Called per-title by the client after `session/create` returns titles. This keeps auth separate from track fetching, allowing interactive - auth flows (OTP, captcha) before any tracks are requested. + auth flows (OTP, captcha) before the client requests any tracks. """ session = await get_validated_session(session_id, request) require_authenticated(session) @@ -2155,7 +2155,7 @@ async def session_tracks_handler( async def session_segments_handler( data: Dict[str, Any], session_id: str, request: Optional[web.Request] = None ) -> web.Response: - """Resolve segment URLs for selected tracks. + """Get segment URLs for selected tracks. The client calls this after selecting which tracks to download. Returns segment URLs, init data, DRM info, and any headers/cookies @@ -2237,15 +2237,15 @@ def cdm_type_stub(cdm_type: str) -> SimpleNamespace: def resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]) -> Optional[Any]: - """Resolve CDM for the server context. + """Get the CDM for the server context. Checks the server's own CDM config (``config.cdm[service]``) to determine the CDM type without loading the full CDM object. This - ensures that when ``server_cdm: true`` is used, the server's CDM + ensures that when you use ``server_cdm: true``, the server's CDM determines device selection (e.g. PlayReady vs Widevine). - Falls back to a lightweight stub from *cdm_type* only if no server - CDM is configured for the service. + Falls back to a lightweight stub from *cdm_type* only if the configuration + has no server CDM for the service. """ from unshackle.core.config import config as app_config @@ -2284,7 +2284,7 @@ def detect_cdm_type_for_service(service: str, app_config: Any) -> Optional[str]: def detect_cdm_type(cdm_name: str, app_config: Any) -> Optional[str]: - """Detect CDM type (playready/widevine) from config without loading it. + """Detect CDM type (PlayReady/widevine) from config without loading it. Checks remote_cdm entries and local file extensions to determine the type. """ @@ -2307,7 +2307,7 @@ def detect_cdm_type(cdm_name: str, app_config: Any) -> Optional[str]: def require_authenticated(session: Any) -> None: - """Raise if the session has not finished authenticating.""" + """Raise if the remote session has not finished authenticating.""" if session.auth_status == AuthStatus.FAILED: raise APIError( APIErrorCode.AUTH_FAILED, @@ -2367,7 +2367,7 @@ async def session_prompt_post_handler( async def get_validated_session(session_id: str, request: Optional[web.Request]) -> Any: - """Fetch a session and verify the caller owns it. + """Fetch a remote session and make sure that the caller owns it. Ownership is bound to the authenticating X-Secret-Key rather than the source IP: behind a reverse proxy every caller shares the proxy's address, so the IP check @@ -2401,9 +2401,9 @@ async def get_validated_session(session_id: str, request: Optional[web.Request]) def resolve_handler_proxy(data: Dict[str, Any], normalized_service: str) -> tuple[Optional[str], list]: - """Resolve proxy and initialize providers from API request data. + """Get the proxy and initialise the proxy providers from API request data. - Handles explicit proxy param, provider:country format, and + Handles explicit proxy param, the `provider:country` format, and client_region-based auto-proxy when server region differs. """ proxy_param = data.get("proxy") @@ -2576,9 +2576,9 @@ def load_server_vaults(service_name: str) -> Any: def check_vaults(kids: list, service_name: str) -> Optional[Dict[str, str]]: - """Check server vaults for existing keys matching all KIDs. + """Examine the server vaults for existing content keys that match all KIDs. - Returns a KID:KEY dict if ALL KIDs are found, None otherwise. + Returns a `KID:KEY` dict if ALL KIDs are found, None otherwise. """ from uuid import UUID @@ -2628,7 +2628,7 @@ def handle_single_server_cdm( drm_type: str, request: Optional[web.Request], ) -> Dict[str, str]: - """Handle single-track server_cdm licensing using the DRM class get_content_keys() flow.""" + """Do the single-track server_cdm licensing with the DRM class get_content_keys() flow.""" import base64 from unshackle.core.cdm import load_cdm @@ -2739,13 +2739,13 @@ def handle_proxy_license( async def session_license_handler( data: Dict[str, Any], session_id: str, request: Optional[web.Request] = None ) -> web.Response: - """Handle DRM licensing in proxy or server_cdm mode. + """Do the DRM licensing in proxy or server_cdm mode. Proxy mode (default): forwards client CDM challenge to the service's license endpoint, returns raw license bytes for client-side parsing. - Server-CDM mode (mode="server_cdm"): server uses its own CDM to generate - the challenge, obtain the license, and extract KID:KEY pairs. Supports + Server-CDM mode (mode="server_cdm"): server uses its own CDM to make + the challenge, get the license, and extract `KID:KEY` pairs. Supports batch (track_ids list) and single-track requests. """ import base64 @@ -2910,7 +2910,7 @@ async def session_license_handler( async def session_info_handler(session_id: str, request: Optional[web.Request] = None) -> web.Response: - """Check session validity and get session info.""" + """Make sure that the remote session is valid, and get the remote session info.""" session = await get_validated_session(session_id, request) from unshackle.core.api.session_store import get_session_store @@ -2928,7 +2928,7 @@ async def session_info_handler(session_id: str, request: Optional[web.Request] = async def session_delete_handler(session_id: str, request: Optional[web.Request] = None) -> web.Response: - """Delete a session, return updated cache files, and clean up server-side data.""" + """Delete a remote session, return updated cache files, and clean up server-side data.""" import base64 import zlib diff --git a/unshackle/core/api/input_bridge.py b/unshackle/core/api/input_bridge.py index 8aa398a8..ffdbf526 100644 --- a/unshackle/core/api/input_bridge.py +++ b/unshackle/core/api/input_bridge.py @@ -34,7 +34,7 @@ class InputBridge: """Thread-safe bridge between a sync auth thread and the async HTTP layer. The auth thread calls :meth:`request_input` which blocks until the - remote client submits a response via the HTTP prompt endpoints. + remote client submits a response through the HTTP prompt endpoints. """ _prompt: Optional[str] = field(default=None, init=False, repr=False) @@ -55,8 +55,8 @@ class InputBridge: The string response from the remote client. Raises: - TimeoutError: If no response is received within *timeout*. - RuntimeError: If the bridge was cancelled. + TimeoutError: If the bridge gets no response within *timeout*. + RuntimeError: If the server cancelled the bridge. """ with self._lock: if self._cancelled: @@ -83,7 +83,7 @@ class InputBridge: return response def get_pending_prompt(self) -> Optional[str]: - """Return the current prompt if the auth thread is waiting for input.""" + """Return the current prompt when the auth thread waits for input.""" with self._lock: if self._status == AuthStatus.PENDING_INPUT: return self._prompt @@ -93,7 +93,7 @@ class InputBridge: """Deliver the client's response and unblock the auth thread. Returns: - ``True`` if a prompt was pending and the response was accepted, + ``True`` if a prompt was pending and the bridge accepted the response, ``False`` otherwise. """ with self._lock: diff --git a/unshackle/core/api/routes.py b/unshackle/core/api/routes.py index 45ce109b..ef839e29 100644 --- a/unshackle/core/api/routes.py +++ b/unshackle/core/api/routes.py @@ -138,9 +138,9 @@ async def health(request: web.Request) -> web.Response: @api_handler async def services(request: web.Request) -> web.Response: """ - List available services. + Show the available services. --- - summary: List services + summary: Show services description: Get all available streaming services with their details responses: '200': @@ -203,7 +203,7 @@ async def services(request: web.Request) -> web.Response: format: date-time debug_info: type: object - description: Only present when --debug-api flag is enabled + description: Only present when you use the --debug-api flag """ try: service_tags = Services.get_tags() @@ -332,10 +332,10 @@ async def services(request: web.Request) -> web.Response: @api_handler async def search(request: web.Request) -> web.Response: """ - Search for titles from a service. + Find titles from a service. --- - summary: Search for titles - description: Search for titles by query string from a service + summary: Find titles + description: Find titles by query string from a service requestBody: required: true content: @@ -418,9 +418,9 @@ async def search(request: web.Request) -> web.Response: @api_handler async def list_titles(request: web.Request) -> web.Response: """ - List titles for a service and title ID. + Show titles for a service and title ID. --- - summary: List titles + summary: Show titles description: Get available titles for a service and title ID requestBody: required: true @@ -541,9 +541,9 @@ async def list_titles(request: web.Request) -> web.Response: @api_handler async def list_tracks(request: web.Request) -> web.Response: """ - List tracks for a title, separated by type. + Show tracks for a title, separated by type. --- - summary: List tracks + summary: Show tracks description: Get available video, audio, and subtitle tracks for a title requestBody: required: true @@ -563,7 +563,7 @@ async def list_tracks(request: web.Request) -> web.Response: description: Title identifier wanted: type: string - description: Specific episode/season, or music track ("1-5", "2x3" for disc 2 track 3) (optional) + description: Specific episode/season, or song ("1-5", "2x3" for disc 2 track 3) (optional) proxy: type: string description: Proxy configuration (optional) @@ -591,13 +591,13 @@ async def list_tracks(request: web.Request) -> web.Response: @api_handler async def download(request: web.Request) -> web.Response: """ - Download content based on provided parameters. + Download a title based on the provided parameters. --- - summary: Download content + summary: Download a title description: >- - Download video content based on specified parameters. - 'postscript', 'post_script' and 'post_scripts' are never accepted and return 400; - post-download hooks are configured only in unshackle.yaml. + Download a video title based on the specified parameters. + 'postscript', 'post_script' and 'post_scripts' are never accepted and return 400. + You configure post-download hooks only in unshackle.yaml. requestBody: required: true content: @@ -646,13 +646,13 @@ async def download(request: web.Request) -> web.Response: type: array items: type: string - description: Video color range (SDR, HDR10, HDR10+, HLG, DV, HYBRID) (default - ["SDR"]) + description: Video colour range (SDR, HDR10, HDR10+, HLG, DV, HYBRID) (default - ["SDR"]) channels: type: number description: Audio channels (e.g., 2.0, 5.1, 7.1) (default - None) no_atmos: type: boolean - description: Exclude Dolby Atmos audio tracks (default - false) + description: Exclude Atmos audio tracks (default - false) wanted: type: array items: @@ -699,7 +699,7 @@ async def download(request: web.Request) -> web.Response: description: Use exact language matching (no variants) (default - false) sub_format: type: string - description: Output subtitle format (SRT, VTT, etc.) (default - None) + description: Output subtitle format such as SRT or VTT (default - None) video_only: type: boolean description: Only download video tracks (default - false) @@ -734,7 +734,7 @@ async def download(request: web.Request) -> web.Response: description: Add randomized delay between downloads. `true` for default 60-120s, or `"MIN-MAX"` string (e.g., `"20-40"`). Min must be >= 20 (default - null) split_audio: type: boolean - description: Create separate output files per audio codec instead of merging all audio (default - null) + description: Make separate output files per audio codec instead of merging all audio (default - null) skip_dl: type: boolean description: Skip downloading, only retrieve decryption keys (default - false) @@ -743,7 +743,7 @@ async def download(request: web.Request) -> web.Response: description: Export manifest, track URLs, keys, and subtitles to JSON in the exports directory (default - false) cdm_only: type: boolean - description: Only use CDM for key retrieval (true) or only vaults (false) (default - None) + description: Only use CDM for content key retrieval (true) or only vaults (false) (default - None) proxy: type: string description: Proxy URI or country code (default - None) @@ -755,7 +755,7 @@ async def download(request: web.Request) -> web.Response: description: Bypass proxy for all downloads. Manifest, license, and auth still use proxy (default - false) tag: type: string - description: Set the group tag to be used (default - None) + description: Set the group tag (default - None) tmdb_id: type: integer description: Use this TMDB ID for tagging instead of a title search. Set enrich to also take its title, year and original language. Mutually exclusive with imdb_id and tvdb_id. Needs tmdb_api_key (default - None) @@ -769,7 +769,7 @@ async def download(request: web.Request) -> web.Response: description: Overwrite show title, year and original language with the external source's. Requires one of tmdb_id, imdb_id, tvdb_id or anilist_id (default - false) daily: type: boolean - description: Treat the title as daily/date-based content and fill missing air dates from TVDB. Needs enrich (default - false) + description: Treat the title as daily content and fill missing air dates from TVDB. Needs enrich (default - false) no_folder: type: boolean description: Disable folder creation for TV shows (default - false) @@ -787,7 +787,7 @@ async def download(request: web.Request) -> web.Response: description: Scale per-track segment workers to measured CDN throughput, up to the workers cap (default - false) download_processes: type: integer - description: Split a track's segments across this many processes; only engages for large batches (default - 1) + description: Split a track's segments across this many processes. Only engages for large batches (default - 1) continue_downloads: type: boolean description: Keep completed segment files across runs and resume a previously failed download (default - false) @@ -857,9 +857,9 @@ async def download(request: web.Request) -> web.Response: @api_handler async def download_jobs(request: web.Request) -> web.Response: """ - List all download jobs with optional filtering and sorting. + Show all download jobs with optional filtering and sorting. --- - summary: List download jobs + summary: Show download jobs description: Get list of all download jobs with their status, with optional filtering by status/service and sorting parameters: - name: status @@ -976,8 +976,8 @@ async def download_job_events(request: web.Request) -> web.StreamResponse: `progress` and `status` events, and closes after the terminal `completed`, `failed` or `cancelled` event. Every event carries the same full job object that GET /api/download/jobs/{job_id} returns. A browser EventSource can authenticate with - the `secret_key` query parameter instead of the X-Secret-Key header; when both are - sent the header is used. + the `secret_key` query parameter instead of the X-Secret-Key header. When the client + sends both, the server uses the header. parameters: - name: job_id in: path @@ -1024,7 +1024,7 @@ async def cancel_download_job(request: web.Request) -> web.Response: '204': description: Terminal job removed from the manager '400': - description: Job cannot be cancelled + description: The server cannot cancel the job '404': description: Job not found '500': @@ -1099,9 +1099,9 @@ async def retry_download_job(request: web.Request) -> web.Response: @api_handler async def prioritize_download_job(request: web.Request) -> web.Response: """ - Prioritize download job. + Prioritise download job. --- - summary: Prioritize download job + summary: Prioritise download job description: Move a queued job to the front of the download queue parameters: - name: job_id @@ -1136,14 +1136,15 @@ async def prioritize_download_job(request: web.Request) -> web.Response: @api_handler async def profiles(request: web.Request) -> web.Response: """ - List configured credential profiles per service. + Show the configured credential profiles per service. --- - summary: List credential profiles + summary: Show credential profiles description: > Enumerate named credential profiles configured per service (usable as the `profile` - parameter). Only services whose credentials are a mapping of profile-name to credential - are listed (including a `default` key if present); a service configured with a single - plain (unnamed) credential is omitted entirely. Filtered by the caller's service allowlist. + parameter). This endpoint shows only the services whose credentials are a mapping of + profile-name to credential, and it includes a `default` entry if present. It ignores a + service configured with a single plain (unnamed) credential. Filtered by the caller's + service allowlist. responses: '200': description: Profiles per service @@ -1172,8 +1173,8 @@ async def server_config(request: web.Request) -> web.Response: summary: Get server config description: > Read-only, redacted view of the effective server configuration for display in a UI - settings page. Secrets (api_secret, users, credentials, tokens) are never included; - secret-looking keys inside `dl` are masked. + settings page. This view never includes secrets (api_secret, users, credentials, + tokens), and unshackle masks secret-looking config keys inside `dl`. responses: '200': description: Redacted server configuration @@ -1235,7 +1236,7 @@ async def download_history(request: web.Request) -> web.Response: summary: Get download history description: > Read the persisted job history (jobs that reached a terminal state), newest first. - Corrupt lines in the history file are skipped; a missing file yields an empty list. + This endpoint ignores corrupt lines in the history file. A missing file gives no entries. parameters: - name: limit in: query @@ -1381,8 +1382,8 @@ async def maintenance_refresh_services(request: web.Request) -> web.Response: summary: Refresh service repos description: > Force-sync (git pull) every service repo configured in directories.services. - `refreshed` is true when all repos synced (or none are configured); per-repo - results are listed under `repos`. + `refreshed` is true when all repos synced, or when you configured no repos. The + `repos` field holds the per-repo results. responses: '200': description: Refresh results @@ -1415,7 +1416,7 @@ async def maintenance_refresh_services(request: web.Request) -> web.Response: @api_handler async def env_check(request: web.Request) -> web.Response: """ - Check environment dependencies. + Examine the environment dependencies. --- summary: Environment check description: Report install status of the binaries `env check` inspects, with best-effort versions. @@ -1450,9 +1451,9 @@ async def env_check(request: web.Request) -> web.Response: @api_handler async def session_create(request: web.Request) -> web.Response: """ - Create a remote-dl session. + Make a remote session. --- - summary: Create session + summary: Make remote session description: Authenticate with a service, get titles, tracks, and chapters in one call requestBody: required: true @@ -1485,7 +1486,7 @@ async def session_create(request: web.Request) -> web.Response: additionalProperties: true responses: '200': - description: Session created with titles, tracks, and chapters + description: Remote session created with titles, tracks, and chapters '400': description: Invalid request '401': @@ -1510,10 +1511,10 @@ async def session_create(request: web.Request) -> web.Response: @api_handler async def session_titles(request: web.Request) -> web.Response: """ - Get titles for an authenticated session. + Get titles for an authenticated remote session. --- summary: Get titles - description: Fetch titles from the authenticated service session + description: Fetch titles from the authenticated remote session parameters: - name: session_id in: path @@ -1524,7 +1525,7 @@ async def session_titles(request: web.Request) -> web.Response: '200': description: List of titles '404': - description: Session not found + description: Remote session not found """ session_id = request.match_info["session_id"] try: @@ -1542,7 +1543,7 @@ async def session_tracks(request: web.Request) -> web.Response: Get tracks and chapters for a specific title. --- summary: Get tracks - description: Fetch tracks and chapters for a title in the session + description: Fetch tracks and chapters for a title in the remote session parameters: - name: session_id in: path @@ -1565,7 +1566,7 @@ async def session_tracks(request: web.Request) -> web.Response: '200': description: Tracks and chapters for the title '404': - description: Session or title not found + description: Remote session or title not found """ session_id = request.match_info["session_id"] try: @@ -1587,9 +1588,9 @@ async def session_tracks(request: web.Request) -> web.Response: @api_handler async def session_segments(request: web.Request) -> web.Response: """ - Resolve segment URLs for selected tracks. + Get segment URLs for selected tracks. --- - summary: Resolve segments + summary: Get segments description: Get download URLs, DRM info, and headers for selected tracks parameters: - name: session_id @@ -1610,12 +1611,12 @@ async def session_segments(request: web.Request) -> web.Response: type: array items: type: string - description: List of track IDs to resolve + description: List of track IDs to get segment URLs for responses: '200': description: Segment URLs and DRM info for each track '404': - description: Session or track not found + description: Remote session or track not found """ session_id = request.match_info["session_id"] try: @@ -1671,7 +1672,7 @@ async def session_license(request: web.Request) -> web.Response: '200': description: License response '404': - description: Session or track not found + description: Remote session or track not found """ session_id = request.match_info["session_id"] try: @@ -1693,10 +1694,10 @@ async def session_license(request: web.Request) -> web.Response: @api_handler async def session_info(request: web.Request) -> web.Response: """ - Get session info. + Get remote session info. --- - summary: Session info - description: Check session validity and get metadata + summary: Remote session info + description: Make sure that the remote session is valid, and get metadata parameters: - name: session_id in: path @@ -1705,9 +1706,9 @@ async def session_info(request: web.Request) -> web.Response: type: string responses: '200': - description: Session info + description: Remote session info '404': - description: Session not found + description: Remote session not found """ session_id = request.match_info["session_id"] return await session_info_handler(session_id, request) @@ -1716,10 +1717,10 @@ async def session_info(request: web.Request) -> web.Response: @api_handler async def session_delete(request: web.Request) -> web.Response: """ - Delete a session. + Delete a remote session. --- - summary: Delete session - description: Clean up a remote-dl session + summary: Delete remote session + description: Clean up a remote session parameters: - name: session_id in: path @@ -1728,9 +1729,9 @@ async def session_delete(request: web.Request) -> web.Response: type: string responses: '200': - description: Session deleted + description: Remote session deleted '404': - description: Session not found + description: Remote session not found """ session_id = request.match_info["session_id"] return await session_delete_handler(session_id, request) @@ -1742,7 +1743,7 @@ async def session_prompt_get(request: web.Request) -> web.Response: Poll for pending interactive prompts during authentication. --- summary: Get auth prompt - description: Poll for pending interactive prompts (OTP, device code, PIN) during session authentication + description: Poll for pending interactive prompts (OTP, device code, PIN) during remote session authentication parameters: - name: session_id in: path @@ -1765,9 +1766,9 @@ async def session_prompt_get(request: web.Request) -> web.Response: description: Prompt to display to the user (only when status is pending_input) error: type: string - description: Error message (only when status is failed) + description: Error message (only for the failed status) '404': - description: Session not found + description: Remote session not found """ session_id = request.match_info["session_id"] try: @@ -1818,7 +1819,7 @@ async def session_prompt_submit(request: web.Request) -> web.Response: '400': description: No prompt pending or invalid request '404': - description: Session not found + description: Remote session not found """ session_id = request.match_info["session_id"] try: @@ -1875,7 +1876,7 @@ ROUTES: list[tuple[str, str, Handler, bool]] = [ def setup_routes(app: web.Application, remote_only: bool = False) -> None: - """Setup API routes. When remote_only=True, only expose remote session endpoints.""" + """Setup API routes. When remote_only=True, only the remote session endpoints operate.""" add: dict[str, Callable[..., Any]] = { "GET": app.router.add_get, "POST": app.router.add_post, diff --git a/unshackle/core/api/sanitize.py b/unshackle/core/api/sanitize.py index fd516db0..2a82b3c1 100644 --- a/unshackle/core/api/sanitize.py +++ b/unshackle/core/api/sanitize.py @@ -7,19 +7,19 @@ from typing import Optional def sanitize_log(value: object) -> str: - """Sanitize a value for safe logging by removing newlines and control characters.""" + """Sanitise a value for safe logging by removing newlines and control characters.""" return str(value).replace("\n", "").replace("\r", "").replace("\x00", "") def safe_cache_key(key: object) -> Optional[str]: """Return a bare filename for a peer-supplied cache key, or None if it escapes its directory. - Cache keys are written as ``.json`` inside a fixed directory. A key containing a - path separator, ``..``, or an absolute path would let the peer write outside that - directory, so anything that is not already a plain filename is rejected. + unshackle writes each cache key as ``.json`` inside a fixed directory. A cache key + that holds a path separator, ``..``, or an absolute path would let the peer write outside + that directory, so this function rejects anything that is not already a plain filename. - PureWindowsPath treats both ``/`` and ``\\`` as separators, so a key like - ``..\\..\\secret`` is rejected even on POSIX (where backslash is a plain character). + PureWindowsPath treats both ``/`` and ``\\`` as separators, so this function rejects a + cache key like ``..\\..\\secret`` even on POSIX (where backslash is a plain character). """ text = str(key) name = PureWindowsPath(text).name diff --git a/unshackle/core/api/session_store.py b/unshackle/core/api/session_store.py index 634d7818..3f63540b 100644 --- a/unshackle/core/api/session_store.py +++ b/unshackle/core/api/session_store.py @@ -1,8 +1,9 @@ -"""Server-side session store for remote-dl client-server architecture. +"""Server-side remote session store for the remote-dl client-server architecture. Maintains authenticated service instances between API calls so that -a client can authenticate once and then make multiple requests (list tracks, -resolve segments, proxy license) using the same session. +a client can authenticate once and then make several requests (get the tracks, +get the segment URLs, get the licence in proxy mode) with the same +remote session. """ from __future__ import annotations @@ -24,7 +25,7 @@ log = logging.getLogger("api.session") @dataclass class SessionEntry: - """A single authenticated session with a service.""" + """A single authenticated remote session with a service.""" session_id: str service_tag: str @@ -49,7 +50,7 @@ class SessionEntry: class SessionStore: - """Thread-safe session store with TTL-based expiration.""" + """Thread-safe remote session store with TTL-based expiration.""" def __init__(self) -> None: self._sessions: Dict[str, SessionEntry] = {} @@ -58,7 +59,7 @@ class SessionStore: @property def ttl(self) -> int: - """Session TTL in seconds from config.""" + """Remote session TTL in seconds from config.""" return config.serve.get("session_ttl", 300) @property @@ -72,7 +73,7 @@ class SessionStore: service_instance: Any, session_id: Optional[str] = None, ) -> SessionEntry: - """Create a new session with an authenticated service instance.""" + """Make a new remote session with an authenticated service instance.""" async with self._lock: if len(self._sessions) >= self.max_sessions: oldest_id = min(self._sessions, key=lambda k: self._sessions[k].last_accessed) @@ -90,7 +91,7 @@ class SessionStore: return entry async def get(self, session_id: str) -> Optional[SessionEntry]: - """Get a session by ID, returns None if not found or expired.""" + """Get a remote session by ID, returns None if not found or expired.""" async with self._lock: entry = self._sessions.get(session_id) if entry is None: @@ -107,7 +108,7 @@ class SessionStore: return entry async def delete(self, session_id: str) -> bool: - """Delete a session. Returns True if it existed.""" + """Delete a remote session. Returns True if it existed.""" async with self._lock: entry = self._sessions.pop(session_id, None) if entry: @@ -173,7 +174,7 @@ class SessionStore: @staticmethod def cleanup_cache_dir(cache_tag: Optional[str]) -> None: - """Remove session cache directory and empty parents.""" + """Remove the remote session cache directory and empty parents.""" if not cache_tag: return import shutil @@ -203,7 +204,7 @@ session_store: Optional[SessionStore] = None def get_session_store() -> SessionStore: - """Get or create the global session store singleton.""" + """Get or make the global remote session store singleton.""" global session_store if session_store is None: session_store = SessionStore() diff --git a/unshackle/core/cacher.py b/unshackle/core/cacher.py index 2f102d93..36e18268 100644 --- a/unshackle/core/cacher.py +++ b/unshackle/core/cacher.py @@ -49,7 +49,7 @@ class Cacher: def get(self, key: str, version: int = 1) -> Cacher: """ - Get Cached data for the Service by Key. + Get the cached data of the service by cache key. :param key: the filename to save the data to, should be url-safe. :param version: the config data version you expect to use. :returns: Cache object containing the cached data or None if the file does not exist. @@ -76,7 +76,7 @@ class Cacher: def set(self, data: Any, expiration: Optional[EXP_T] = None) -> Any: """ - Set Cached data for the Service by Key. + Set the cached data of the service by cache key. :param data: absolutely anything including None. :param expiration: when the data expires, optional. Can be ISO 8601, seconds til expiration, unix timestamp, or a datetime object. @@ -110,7 +110,7 @@ class Cacher: @staticmethod def resolve_datetime(timestamp: EXP_T) -> datetime: """ - Resolve multiple formats of a Datetime or Timestamp to an absolute Datetime. + Change a Datetime or a Timestamp in one of several formats to an absolute Datetime. Examples: >>> now = datetime.now() @@ -129,8 +129,8 @@ class Cacher: datetime.datetime(2022, 6, 27, 11, 59, 13, 770208) In the int/float examples you may notice that it did not return now + 3600 seconds - but rather something a bit more than that. This is because it did not resolve 3600 - seconds from the `now` variable but from right now as the function was called. + but rather something a bit more than that. This is because the 3600 seconds start at + the moment the function operates, not at the `now` variable. """ if isinstance(timestamp, datetime): return timestamp diff --git a/unshackle/core/cdm/__init__.py b/unshackle/core/cdm/__init__.py index a76f13ee..e6c2d8c1 100644 --- a/unshackle/core/cdm/__init__.py +++ b/unshackle/core/cdm/__init__.py @@ -4,7 +4,7 @@ CDM helpers and implementations. Keep this module import-light: downstream code frequently imports helpers from `unshackle.core.cdm.detect`, which requires importing this package first. Some CDM implementations pull in optional/heavy dependencies, so we lazily -import them via `__getattr__` (PEP 562). +import them through `__getattr__` (PEP 562). """ from __future__ import annotations diff --git a/unshackle/core/cdm/custom_remote_cdm.py b/unshackle/core/cdm/custom_remote_cdm.py index 50dab66a..2347493a 100644 --- a/unshackle/core/cdm/custom_remote_cdm.py +++ b/unshackle/core/cdm/custom_remote_cdm.py @@ -25,7 +25,7 @@ class MockCertificateChain: class Key: - """Key object compatible with pywidevine.""" + """`Key` object compatible with pywidevine.""" def __init__(self, kid: str, key: str, type_: str = "CONTENT"): if isinstance(kid, str): @@ -52,7 +52,7 @@ class CustomRemoteCDMExceptions: """Raised when session ID is invalid.""" class TooManySessions(Exception): - """Raised when session limit is reached.""" + """Raised when the CDM sessions reach the limit.""" class InvalidInitData(Exception): """Raised when PSSH/init data is invalid.""" @@ -64,7 +64,7 @@ class CustomRemoteCDMExceptions: """Raised when license message is invalid.""" class InvalidContext(Exception): - """Raised when session has no context data.""" + """Raised when the CDM session has no context data.""" class SignatureMismatch(Exception): """Raised when signature verification fails.""" @@ -75,19 +75,19 @@ class CustomRemoteCDM: Highly Configurable Custom Remote CDM implementation. This class provides a maximally flexible CDM interface that can adapt to - ANY CDM API format through YAML configuration alone. It's designed to support - both current and future CDM providers without requiring code changes. + ANY CDM API format through YAML configuration alone. It can hold both current + and future CDM providers without code changes. - Key Features: - - Fully configuration-driven behavior (all logic controlled via YAML) + Main Features: + - Fully configuration-driven behaviour (all logic controlled through YAML) - Pluggable authentication strategies (header, body, bearer, basic, custom) - Flexible endpoint configuration (custom paths, methods, timeouts) - Advanced parameter mapping (rename, add static, conditional, nested) - - Powerful response parsing (deep field access, type detection, transforms) - - Transform engine (base64, hex, JSON, custom key formats) + - Response parsing (deep field access, type detection, transforms) + - Transform engine (base64, hex, JSON, custom content key formats) - Condition evaluation (response type detection, success validation) - Compatible with both Widevine and PlayReady DRM schemes - - Vault integration for intelligent key caching + - Vault integration for intelligent content key caching Configuration Philosophy: - 90% of new CDM providers: YAML config only @@ -95,12 +95,12 @@ class CustomRemoteCDM: - 1% of cases: Add new auth strategy (minimal code) - 0% need to modify core request/response logic - The class is designed to handle diverse API patterns including: + The class adapts to these diverse API patterns: - Different authentication mechanisms (headers vs body vs tokens) - Custom endpoint paths and HTTP methods - Parameter name variations (scheme vs device, init_data vs pssh) - Nested JSON structures in requests/responses - - Various key formats (JSON objects, colon-separated strings, etc.) + - Various content key formats (JSON objects, colon-separated strings, and other forms) - Different response success indicators and error messages - Conditional parameters based on device type or other factors """ @@ -123,18 +123,18 @@ class CustomRemoteCDM: **kwargs, ): """ - Initialize Custom Remote CDM with highly configurable options. + Initialise Custom Remote CDM with highly configurable options. Args: host: Base URL for the CDM API - service_name: Service name for key caching and vault operations - vaults: Vaults instance for local key caching + service_name: Service tag for content key caching and vault operations + vaults: Vaults instance for local content key caching device: Device configuration (name, type, system_id, security_level) auth: Authentication configuration (type, credentials, headers) endpoints: Endpoint configuration (paths, methods, timeouts) request_mapping: Request transformation rules (param names, static params, transforms) response_mapping: Response parsing rules (field locations, type detection, success conditions) - caching: Caching configuration (enabled, use_vaults, etc.) + caching: Caching configuration (enabled, use_vaults, and the other caching keys) legacy: Legacy mode configuration timeout: Default request timeout in seconds **kwargs: Additional configuration options for future extensibility @@ -229,7 +229,7 @@ class CustomRemoteCDM: @property def is_playready(self) -> bool: - """Check if this CDM is in PlayReady mode.""" + """Return True if this CDM is in PlayReady mode.""" return self._is_playready @property @@ -246,7 +246,7 @@ class CustomRemoteCDM: Set the required Key IDs for intelligent caching decisions. This method enables the CDM to make smart decisions about when to request - additional keys via license challenges. When cached keys are available, + additional keys through license challenges. When cached keys are available, the CDM will compare them against the required KIDs to determine if a license request is still needed for missing keys. @@ -254,8 +254,8 @@ class CustomRemoteCDM: kids: List of required Key IDs as UUIDs or hex strings Note: - Should be called by DRM classes (PlayReady/Widevine) before making - license challenge requests to enable optimal caching behavior. + Call this method from the DRM classes (PlayReady/Widevine) before a + license challenge request, to enable optimal caching behaviour. """ self._required_kids = [] for kid in kids: @@ -265,7 +265,7 @@ class CustomRemoteCDM: self._required_kids.append(str(kid).replace("-", "").lower()) def generate_session_id(self) -> bytes: - """Generate a unique session ID.""" + """Make a unique session ID.""" return secrets.token_bytes(16) def get_init_data_from_pssh(self, pssh: Any) -> str: @@ -407,11 +407,11 @@ class CustomRemoteCDM: Args: condition: Condition string (e.g., "message == 'success'") - context: Context dictionary with values to check + context: Context dictionary with the values to examine Returns: True if condition is met, False otherwise. A condition that matches none of the supported - forms is treated as unmet and returns False. + forms counts as unmet, so the method returns False. Supported conditions: - "field == value": Equality check @@ -454,12 +454,12 @@ class CustomRemoteCDM: self, endpoint_name: str, base_params: Dict[str, Any], session: Optional[Dict[str, Any]] = None ) -> Dict[str, Any]: """ - Build request parameters with mapping and transformations. + Assemble request parameters with mapping and transformations. Args: endpoint_name: Name of the endpoint (e.g., "get_request", "decrypt_response") base_params: Base parameters to transform - session: Optional session data for context + session: Optional CDM session data for context Returns: Transformed parameters dictionary @@ -469,7 +469,7 @@ class CustomRemoteCDM: 2. Static parameters (add fixed values) 3. Conditional parameters (add based on conditions) 4. Parameter transforms (apply data transformations) - 5. Nested parameter structure (create nested objects) + 5. Nested parameter structure (make nested objects) 6. Parameter exclusions (remove unwanted params) """ mapping_config = self.request_mapping.get(endpoint_name, {}) @@ -530,14 +530,14 @@ class CustomRemoteCDM: Apply authentication to the HTTP session based on auth configuration. Args: - session: requests.Session to apply authentication to + session: `requests.Session` to apply authentication to Supported auth types: - header: Add authentication header (e.g., x-api-key, Authorization) - - body: Authentication will be added to request body (handled in request building) + - body: the request body carries the authentication (not applied here) - bearer: Add Bearer token to Authorization header - basic: Add HTTP Basic authentication - - query: Authentication will be added to query string (handled in request) + - query: the query string carries the authentication (not applied here) """ auth_type = self.auth_config.get("type", "header") @@ -617,14 +617,14 @@ class CustomRemoteCDM: def parse_keys_from_response(self, endpoint_name: str, response_data: Dict[str, Any]) -> List[Dict[str, Any]]: """ - Parse keys from response data using key field mapping. + Parse content keys from response data with the key field mapping. Args: endpoint_name: Name of the endpoint response_data: Parsed response data Returns: - List of key dictionaries with standardized format + List of content key dictionaries with standardized format """ mapping_config = self.response_mapping.get(endpoint_name, {}) key_fields = mapping_config.get("key_fields", {"kid": "kid", "key": "key", "type": "type"}) @@ -668,7 +668,7 @@ class CustomRemoteCDM: def close(self, session_id: bytes) -> None: """ - Close a CDM session and perform comprehensive cleanup. + Close a CDM session and do comprehensive cleanup. Args: session_id: Session identifier @@ -685,7 +685,7 @@ class CustomRemoteCDM: def get_service_certificate(self, session_id: bytes) -> Optional[bytes]: """ - Get the service certificate for a session. + Get the service certificate for a CDM session. Args: session_id: Session identifier @@ -703,7 +703,7 @@ class CustomRemoteCDM: def set_service_certificate(self, session_id: bytes, certificate: Optional[Union[bytes, str]]) -> str: """ - Set the service certificate for a session. + Set the service certificate for a CDM session. Args: session_id: Session identifier @@ -735,7 +735,7 @@ class CustomRemoteCDM: def has_cached_keys(self, session_id: bytes) -> bool: """ - Check if cached keys are available for the session. + Return True if cached keys are available for the CDM session. Args: session_id: Session identifier @@ -757,7 +757,7 @@ class CustomRemoteCDM: self, session_id: bytes, pssh_or_wrm: Any, license_type: str = "STREAMING", privacy_mode: bool = True ) -> bytes: """ - Generate a license challenge using the custom CDM API. + Make a license challenge with the custom CDM API. This method implements intelligent caching logic that checks vaults first, then attempts to retrieve cached keys from the API, and only makes a @@ -895,10 +895,10 @@ class CustomRemoteCDM: """ Parse license response using the custom CDM API. - Vault keys and cached keys are collected first, then any license key whose key ID is not - already present is appended. Key IDs are matched with dashes stripped and case ignored, so - an existing key wins over a license key for the same ID. If the session already holds final - keys and has no cached keys left to combine, the call is a no-op. + The method collects vault keys and cached keys first, then appends any license key whose + key ID is not already present. It matches key IDs with dashes stripped and case ignored, so + an existing content key wins over a license key for the same ID. If the CDM session already + holds final keys and has no cached keys left to combine, the call is a no-op. Args: session_id: Session identifier @@ -1007,14 +1007,14 @@ class CustomRemoteCDM: def get_keys(self, session_id: bytes, type_: Optional[str] = None) -> List[Key]: """ - Get keys from the session. + Get keys from the CDM session. Args: session_id: Session identifier - type_: Optional key type filter (CONTENT, SIGNING, etc.) + type_: Optional type filter (CONTENT, SIGNING, and the other types) Returns: - List of Key objects + List of `Key` objects Raises: InvalidSession: If session ID is invalid diff --git a/unshackle/core/cdm/decrypt_labs_remote_cdm.py b/unshackle/core/cdm/decrypt_labs_remote_cdm.py index a97234cb..329a5de6 100644 --- a/unshackle/core/cdm/decrypt_labs_remote_cdm.py +++ b/unshackle/core/cdm/decrypt_labs_remote_cdm.py @@ -25,7 +25,7 @@ class MockCertificateChain: class Key: - """Key object compatible with pywidevine.""" + """`Key` object compatible with pywidevine.""" def __init__(self, kid: str, key: str, type_: str = "CONTENT"): if isinstance(kid, str): @@ -52,7 +52,7 @@ class DecryptLabsRemoteCDMExceptions: """Raised when session ID is invalid.""" class TooManySessions(Exception): - """Raised when session limit is reached.""" + """Raised when the CDM sessions reach the limit.""" class InvalidInitData(Exception): """Raised when PSSH/init data is invalid.""" @@ -64,7 +64,7 @@ class DecryptLabsRemoteCDMExceptions: """Raised when license message is invalid.""" class InvalidContext(Exception): - """Raised when session has no context data.""" + """Raised when the CDM session has no context data.""" class SignatureMismatch(Exception): """Raised when signature verification fails.""" @@ -78,15 +78,15 @@ class DecryptLabsRemoteCDM: Decrypt Labs' KeyXtractor API service, enhanced with smart caching logic that minimizes unnecessary license requests. - Key Features: + Main Features: - Compatible with both Widevine and PlayReady DRM schemes - Intelligent caching that compares required vs. available keys - Optimized caching for L1/L2 devices (leverages API auto-optimization) - - Automatic key combination for mixed cache/license scenarios - - Seamless fallback to license requests when keys are missing + - Automatic content key combination for mixed cache/license scenarios + - Fallback to license requests when keys are missing Intelligent Caching System: - 1. DRM classes (PlayReady/Widevine) provide required KIDs via set_required_kids() + 1. DRM classes (PlayReady/Widevine) give the required KIDs through set_required_kids() 2. get_license_challenge() first checks for cached keys 3. For L1/L2 devices, always attempts cached keys first (API optimized) 4. If cached keys satisfy requirements, returns empty challenge (no license needed) @@ -109,14 +109,14 @@ class DecryptLabsRemoteCDM: **kwargs, ): """ - Initialize Decrypt Labs Remote CDM for Widevine and PlayReady schemes. + Initialise Decrypt Labs Remote CDM for Widevine and PlayReady schemes. Args: secret: Decrypt Labs API key (matches config format) host: Decrypt Labs API host URL (matches config format) - device_name: DRM scheme (ChromeCDM, L1, L2 for Widevine; SL2, SL3 for PlayReady) - service_name: Service name for key caching and vault operations - vaults: Vaults instance for local key caching + device_name: DRM system device name. Widevine takes ChromeCDM, L1 or L2, and PlayReady takes SL2 or SL3 + service_name: Service tag for content key caching and vault operations + vaults: Vaults instance for local content key caching device_type: Device type (CHROME, ANDROID, PLAYREADY) - for compatibility system_id: System ID - for compatibility security_level: Security level - for compatibility @@ -169,7 +169,7 @@ class DecryptLabsRemoteCDM: @property def is_playready(self) -> bool: - """Check if this CDM is in PlayReady mode.""" + """Return True if this CDM is in PlayReady mode.""" return self._is_playready @property @@ -186,7 +186,7 @@ class DecryptLabsRemoteCDM: Set the required Key IDs for intelligent caching decisions. This method enables the CDM to make smart decisions about when to request - additional keys via license challenges. When cached keys are available, + additional keys through license challenges. When cached keys are available, the CDM will compare them against the required KIDs to determine if a license request is still needed for missing keys. @@ -194,8 +194,8 @@ class DecryptLabsRemoteCDM: kids: List of required Key IDs as UUIDs or hex strings Note: - Should be called by DRM classes (PlayReady/Widevine) before making - license challenge requests to enable optimal caching behavior. + Call this method from the DRM classes (PlayReady/Widevine) before a + license challenge request, to enable optimal caching behaviour. """ self._required_kids = [] for kid in kids: @@ -205,7 +205,7 @@ class DecryptLabsRemoteCDM: self._required_kids.append(str(kid).replace("-", "").lower()) def generate_session_id(self) -> bytes: - """Generate a unique session ID.""" + """Make a unique session ID.""" return secrets.token_bytes(16) def get_init_data_from_pssh(self, pssh: Any) -> str: @@ -263,7 +263,7 @@ class DecryptLabsRemoteCDM: def close(self, session_id: bytes) -> None: """ - Close a CDM session and perform comprehensive cleanup. + Close a CDM session and do comprehensive cleanup. Args: session_id: Session identifier @@ -280,7 +280,7 @@ class DecryptLabsRemoteCDM: def get_service_certificate(self, session_id: bytes) -> Optional[bytes]: """ - Get the service certificate for a session. + Get the service certificate for a CDM session. Args: session_id: Session identifier @@ -298,7 +298,7 @@ class DecryptLabsRemoteCDM: def set_service_certificate(self, session_id: bytes, certificate: Optional[Union[bytes, str]]) -> str: """ - Set the service certificate for a session. + Set the service certificate for a CDM session. Args: session_id: Session identifier @@ -330,7 +330,7 @@ class DecryptLabsRemoteCDM: def has_cached_keys(self, session_id: bytes) -> bool: """ - Check if cached keys are available for the session. + Return True if cached keys are available for the CDM session. Args: session_id: Session identifier @@ -352,7 +352,7 @@ class DecryptLabsRemoteCDM: self, session_id: bytes, pssh_or_wrm: Any, license_type: str = "STREAMING", privacy_mode: bool = True ) -> bytes: """ - Generate a license challenge using Decrypt Labs API with intelligent caching. + Make a license challenge with the Decrypt Labs API and intelligent caching. This method implements smart caching logic that: 1. First checks local vaults for required keys @@ -364,7 +364,7 @@ class DecryptLabsRemoteCDM: The intelligent caching works as follows: - Local vaults: Always checked first if available - For L1/L2 devices: Always prioritizes cached keys (API automatically optimizes) - - For other devices: Uses cache retry logic based on session state + - For other devices: Uses cache retry logic based on CDM session state - With required KIDs set: Only requests license for missing keys - Without required KIDs: Returns any available cached keys - For PlayReady: Combines vault, cached, and license keys seamlessly @@ -383,7 +383,7 @@ class DecryptLabsRemoteCDM: requests.RequestException: If API request fails Note: - Call set_required_kids() before this method for optimal caching behavior. + Call set_required_kids() before this method for optimal caching behaviour. L1/L2 devices automatically use cached keys when available per API design. Local vault keys are always checked first when vaults are available. """ @@ -546,17 +546,17 @@ class DecryptLabsRemoteCDM: def parse_license(self, session_id: bytes, license_message: Union[bytes, str]) -> None: """ - Parse license response using Decrypt Labs API with intelligent key combination. + Parse license response with the Decrypt Labs API and intelligent content key combination. - For PlayReady content with partial cached keys, this method intelligently + For PlayReady titles with partial cached keys, this method intelligently combines the cached keys with newly obtained license keys, avoiding duplicates while ensuring all required keys are available. - The key combination process: + The content key combination process: 1. Extracts keys from the license response 2. If cached keys exist (PlayReady), combines them with license keys 3. Removes duplicate keys by comparing normalized KIDs - 4. Updates the session with the complete key set + 4. Updates the CDM session with the complete content key set Args: session_id: Session identifier @@ -679,14 +679,14 @@ class DecryptLabsRemoteCDM: def get_keys(self, session_id: bytes, type_: Optional[str] = None) -> List[Key]: """ - Get keys from the session. + Get keys from the CDM session. Args: session_id: Session identifier - type_: Optional key type filter (CONTENT, SIGNING, etc.) + type_: Optional type filter (CONTENT, SIGNING, and the other types) Returns: - List of Key objects + List of `Key` objects Raises: InvalidSession: If session ID is invalid @@ -706,10 +706,10 @@ class DecryptLabsRemoteCDM: """Parse cached keys from API response. Args: - cached_keys_data: List of cached key objects from API + cached_keys_data: List of cached content key objects from API Returns: - List of key dictionaries + List of content key dictionaries """ keys = [] diff --git a/unshackle/core/cdm/detect.py b/unshackle/core/cdm/detect.py index 162413f4..e26bacdd 100644 --- a/unshackle/core/cdm/detect.py +++ b/unshackle/core/cdm/detect.py @@ -5,7 +5,7 @@ from typing import Any def is_remote_cdm(cdm: Any) -> bool: """ - Return True if the CDM instance is backed by a remote/service CDM. + Return True if a remote CDM or a service CDM backs the CDM instance. This is useful for service logic that needs to know whether the CDM runs locally (in-process) vs over HTTP/RPC (remote). @@ -102,11 +102,11 @@ def cdm_location(cdm: Any) -> str: def is_playready_cdm(cdm: Any) -> bool: """ - Return True if the given CDM should be treated as PlayReady. + Return True if unshackle treats the given CDM as PlayReady. This intentionally supports both: - Local PlayReady CDMs (pyplayready.cdm.Cdm) - - Remote/wrapper CDMs (e.g. DecryptLabsRemoteCDM) that expose `is_playready` + - Remote/wrapper CDMs (for example DecryptLabsRemoteCDM) that have `is_playready` """ if cdm is None: @@ -162,10 +162,10 @@ def is_remote_widevine_cdm(cdm: Any) -> bool: def is_widevine_cdm(cdm: Any) -> bool: """ - Return True if the given CDM should be treated as Widevine. + Return True if unshackle treats the given CDM as Widevine. - Note: for remote/wrapper CDMs that expose `is_playready`, Widevine is treated - as the logical opposite. + Note: for remote/wrapper CDMs that have `is_playready`, unshackle treats + Widevine as the logical opposite. """ if cdm is None: diff --git a/unshackle/core/cdm/loader.py b/unshackle/core/cdm/loader.py index ec342518..648c8662 100644 --- a/unshackle/core/cdm/loader.py +++ b/unshackle/core/cdm/loader.py @@ -2,7 +2,7 @@ Instantiates a CDM object (local or remote) given a resolved device name. Name resolution (quality-based, profile-based, DRM-type) is the caller's -responsibility; this module only handles the instantiation step. +responsibility. This module does the instantiation step only. """ from __future__ import annotations diff --git a/unshackle/core/cdm/monalisa/monalisa_cdm.py b/unshackle/core/cdm/monalisa/monalisa_cdm.py index d7dc064b..548e6e51 100644 --- a/unshackle/core/cdm/monalisa/monalisa_cdm.py +++ b/unshackle/core/cdm/monalisa/monalisa_cdm.py @@ -1,8 +1,8 @@ """ MonaLisa CDM - WASM-based Content Decryption Module wrapper. -This module provides key extraction from MonaLisa-protected content using -a WebAssembly module that runs locally via wasmtime. +This module extracts content keys from MonaLisa-protected titles. It uses a +WebAssembly module that runs locally with wasmtime. """ import base64 @@ -25,12 +25,12 @@ logger = logging.getLogger(__name__) class MonaLisaCDM: """ - MonaLisa CDM wrapper for WASM-based key extraction. + MonaLisa CDM wrapper for WASM-based content key extraction. This CDM differs from Widevine/PlayReady in that it does not use a - challenge/response flow with a license server. Instead, the license - (ticket) is provided directly by the service API, and keys are extracted - locally via the WASM module. + challenge/response flow with a license server. Instead, the service API + gives the license (ticket) directly, and the WASM module extracts the + content keys locally. """ DYNAMIC_BASE = 6065008 @@ -50,7 +50,7 @@ class MonaLisaCDM: def __init__(self, device_path: Path): """ - Initialize the MonaLisa CDM. + Initialise the MonaLisa CDM. Args: device_path: Path to the device file (.mld). @@ -108,7 +108,7 @@ class MonaLisaCDM: Session ID (always 1 for MonaLisa). Raises: - RuntimeError: If session initialization fails. + RuntimeError: If CDM session initialization fails. """ try: self.store = wasmtime.Store(self.engine) @@ -179,7 +179,7 @@ class MonaLisaCDM: Dictionary with keys: kid (hex), key (hex), type ("CONTENT"). Raises: - RuntimeError: If session not open or license validation fails. + RuntimeError: If the CDM session is not open, or license validation fails. ValueError: If license_data is empty. """ if not self.instance or not self.memory or self.ctx is None: @@ -233,7 +233,7 @@ class MonaLisaCDM: return {"kid": kid_bytes.hex(), "key": key_bytes.hex(), "type": "CONTENT"} def extract_license_key_bytes(self) -> bytes: - """Extract the 16-byte decryption key from WASM memory.""" + """Extract the 16-byte content key from WASM memory.""" data_ptr = self.memory.data_ptr(self.store) data_len = self.memory.data_len(self.store) @@ -318,7 +318,7 @@ class MonaLisaCDM: mem_ptr[buffer + len(encoded)] = 0 def build_imports(self): - """Build the WASM import stubs required by the MonaLisa module.""" + """Assemble the WASM import stubs the MonaLisa module needs.""" def sys_fcntl64(a, b, c): return 0 diff --git a/unshackle/core/commands.py b/unshackle/core/commands.py index 459f6143..b4cff382 100644 --- a/unshackle/core/commands.py +++ b/unshackle/core/commands.py @@ -35,9 +35,9 @@ def load_command(path: Path) -> object: def load_commands(paths: list[Path]) -> tuple[dict[str, object], list[str]]: """Load every command, returning the good ones plus a list of load errors. - Importing this module must never raise (it runs at CLI startup, before Rich - is installed, so a raise here prints an ugly pre-setup traceback). Instead we - collect failures and surface them once, cleanly, when the CLI is used. + Importing this module must never raise (it runs at CLI startup, before unshackle + installs Rich, so a raise here prints an ugly pre-setup traceback). Instead we + collect failures and surface them once, cleanly, when the user operates the CLI. """ modules: dict[str, object] = {} errors: list[str] = [] diff --git a/unshackle/core/config.py b/unshackle/core/config.py index d0588d1c..d2e288e3 100644 --- a/unshackle/core/config.py +++ b/unshackle/core/config.py @@ -13,12 +13,12 @@ from unshackle.core.utils.collections import ci_get def resolve_decryption(decryption_map: dict, default: str, service: str) -> str: - """Pick the decryption tool for a service (case-insensitive), falling back to default.""" + """Pick the decryption backend for a service (case-insensitive), falling back to default.""" return ci_get(decryption_map, service, default) def resolve_cdm_name(cdm: dict, service: str, override: Any = None) -> Any: - """Resolve a service's top-level CDM entry (case-insensitive), with default fallback.""" + """Find a service's top-level CDM entry (case-insensitive), with default fallback.""" return override or ci_get(cdm, service) or ci_get(cdm, "default") @@ -248,7 +248,7 @@ class Config: warnings.warn(f"Template '{template_type}' is empty") def get_folder_template(self, kind: str) -> str: - """Resolve the folder template for the given title kind. + """Find the folder template for the given title kind. kind: one of "movies", "series", "songs", "albums". Falls back to the legacy single-string folder template, then "". diff --git a/unshackle/core/console.py b/unshackle/core/console.py index 13c4ad63..10448ba0 100644 --- a/unshackle/core/console.py +++ b/unshackle/core/console.py @@ -138,7 +138,7 @@ class ComfyConsole(Console): """A comfy high level console interface. Args: - color_system (str, optional): The color system supported by your terminal, + color_system (str, optional): The colour system supported by your terminal, either ``"standard"``, ``"256"`` or ``"truecolor"``. Leave as ``"auto"`` to autodetect. force_terminal (Optional[bool], optional): Enable/disable terminal control codes, or None to auto-detect terminal. Defaults to None. @@ -154,7 +154,7 @@ class ComfyConsole(Console): width (int, optional): The width of the terminal. Leave as default to auto-detect width. height (int, optional): The height of the terminal. Leave as default to auto-detect height. style (StyleType, optional): Style to apply to all output, or None for no style. Defaults to None. - no_color (Optional[bool], optional): Enabled no color mode, or None to auto-detect. Defaults to None. + no_color (Optional[bool], optional): Enabled no colour mode, or None to auto-detect. Defaults to None. tab_size (int, optional): Number of spaces used to replace a tab character. Defaults to 8. record (bool, optional): Boolean to enable recording of terminal output, required to call :meth:`export_html`, :meth:`export_svg`, and :meth:`export_text`. Defaults to False. @@ -168,7 +168,7 @@ class ComfyConsole(Console): strftime or callable that formats the time. Defaults to "[%X] ". highlighter (HighlighterType, optional): Default highlighter. legacy_windows (bool, optional): Enable legacy Windows mode, or ``None`` to auto-detect. Defaults to ``None``. - safe_box (bool, optional): Restrict box options that don't render on legacy Windows. + safe_box (bool, optional): Restrict box options that do not render on legacy Windows. get_datetime (Callable[[], datetime], optional): Callable that gets the current time as a datetime.datetime object (used by Console.log), or None for datetime.now. get_time (Callable[[], time], optional): Callable that gets the current time in seconds, default uses @@ -260,10 +260,10 @@ class ComfyConsole(Console): speed (float, optional): Speed factor for spinner animation. Defaults to 1.0. refresh_per_second (float, optional): Number of refreshes per second. Defaults to 12.5. pad (Union[int, Tuple[int]]): Padding for top, right, bottom, and left borders. - May be specified with 1, 2, or 4 integers (CSS style). + Give 1, 2, or 4 integers (CSS style). Returns: - Status: A Status object that may be used as a context manager. + Status: A Status object you can use as a context manager. """ status_renderable = super().status( status=status, @@ -436,7 +436,7 @@ def listing_panel(renderable: RenderableType, title: str) -> Panel: def listing_table(title: Optional[str] = None, **kwargs: Any) -> Table: - """Build a table in the style of the rich-click help tables, so listings and help screens match.""" + """Assemble a table in the style of the rich-click help tables, so listings and help screens match.""" kwargs.setdefault("box", box.HORIZONTALS) kwargs.setdefault("show_edge", False) # no top or bottom rule, so the title runs straight into the header kwargs.setdefault("border_style", "dark_gray") diff --git a/unshackle/core/credential.py b/unshackle/core/credential.py index b923d708..3d69cea3 100644 --- a/unshackle/core/credential.py +++ b/unshackle/core/credential.py @@ -62,9 +62,9 @@ class Credential: Format: {username}:{password} Rules: Only one Credential must be in this text contents. - All whitespace before and after all text will be removed. + unshackle removes all whitespace before and after all text. Any whitespace between text will be kept and used. - The credential can be spanned across one or multiple lines as long as it + The credential can span one or multiple lines as long as it abides with all the above rules and the format. Example that follows the format and rules: @@ -81,7 +81,7 @@ class Credential: def load(cls, path: Path) -> Credential: """ Load Credential from a file path. - Use Credential.loads() for loading from text content and seeing the rules and - format expected to be found in the URIs contents. + Use Credential.loads() to load from a text string, and to see the rules and + format expected in the URIs contents. """ return cls.loads(path.read_text("utf8")) diff --git a/unshackle/core/downloaders/requests.py b/unshackle/core/downloaders/requests.py index 01012d61..3dc7669e 100644 --- a/unshackle/core/downloaders/requests.py +++ b/unshackle/core/downloaders/requests.py @@ -121,7 +121,7 @@ def parse_speed_limit(value: Union[str, int, float, None]) -> Optional[float]: Accepts plain numbers (exact bytes/sec) or a decimal suffix with optional /s, case-insensitive: "500k", "5M", "10MB/s", "1.5G", 5000000. Suffixes - match the displayed speeds (5M = 5.0 MB/s); values are bytes, not bits. + match the displayed speeds (5M = 5.0 MB/s). Values are bytes, not bits. "", 0, "off", "none" and "unlimited" mean no limit. """ if value is None: @@ -159,7 +159,7 @@ def set_speed_limit(bytes_per_sec: Optional[float], lock: bool = False) -> None: def adaptive_chunk_size(content_length: int) -> int: - """Pick chunk size based on content length. Benchmarked sweet spot: 512KB-4MB.""" + """Pick the read chunk size from ``content_length``. Benchmarked sweet spot: 512KB-4MB.""" if content_length <= 0: return DEFAULT_CHUNK return min(MAX_CHUNK, max(MIN_CHUNK, content_length // 4)) @@ -176,10 +176,11 @@ class AdaptiveWorkerController: """CDN-aware segment concurrency governor. Pure and synchronous (no threads or sockets), so the policy is unit-testable. - ``update`` is evaluated once per ``tick``, starting once half a tick of throughput - samples exists. The target starts at ``start`` (the cap by default) and follows AIMD - from there: an error burst halves it and takes a one-tick cooldown, a probe upward - that plateaus is reverted, and otherwise it climbs by ``ADAPTIVE_STEP``. Target is + ``update`` evaluates the policy once per ``tick``. It starts as soon as half a tick + of throughput samples exists. The target starts at ``start`` (the cap by default) and + follows AIMD from there: an error burst halves it and takes a one-tick cooldown, the + controller reverts a probe upward that plateaus, and otherwise it climbs by + ``ADAPTIVE_STEP``. Target is always clamped to ``[ADAPTIVE_MIN, cap]``. """ @@ -231,12 +232,12 @@ class AdaptiveWorkerController: return self._first_sample is not None and (now - self._first_sample) >= self.tick / 2 def update(self, now: float, inflight_plus_remaining: Optional[int] = None) -> int: - """Evaluate the policy at most once per tick; return the current target. + """Evaluate the policy at most once per tick. Return the current target. ``inflight_plus_remaining`` (when given) is the count of segments still in flight or not yet started. If it is below the target there is not enough work to saturate the - target, so the low measured speed reflects starvation rather than a plateau, and the - probe/revert step is skipped this tick. Error-burst decreases still apply. Default ``None`` keeps the + target, so the low measured speed reflects starvation rather than a plateau, and + there is no probe/revert step this tick. Error-burst decreases still apply. Default ``None`` keeps the original behaviour for existing callers. """ if self._last_tick is None: @@ -300,10 +301,10 @@ class AdaptiveWorkerController: def retry_sleep(exc: Exception, attempts: int) -> float: """Seconds to wait before retry number ``attempts``. - download() owns segment retries (session-level retries are suppressed), so backoff + download() owns segment retries (the HTTP session itself does not retry), so backoff lives here: honor the server's Retry-After when the failure carries a response, else exponential growth from RETRY_WAIT with jitter so concurrently rate-limited - segments don't retry in lockstep. Capped at session.MAX_BACKOFF either way. + segments do not retry in lockstep. Capped at ``session.MAX_BACKOFF`` either way. """ from unshackle.core.session import MAX_BACKOFF @@ -335,12 +336,12 @@ def retry_sleep(exc: Exception, attempts: int) -> float: def is_requests_session(session: Any) -> bool: - """Check if the session is a standard requests.Session (supports resp.raw).""" + """Whether the HTTP session is a standard requests.Session (it has ``resp.raw``).""" return isinstance(session, Session) def is_rnet_session(session: Any) -> bool: - """Check if the session is an RnetSession (uses resp.stream()).""" + """Whether the HTTP session is an RnetSession (it uses ``resp.stream()``).""" from unshackle.core.session import RnetSession return isinstance(session, RnetSession) @@ -348,7 +349,7 @@ def is_rnet_session(session: Any) -> bool: def is_content_encoded(value: Optional[str]) -> bool: """ - Check a Content-Encoding value for any coding other than absent or `identity`. + Whether a Content-Encoding value names any coding other than absent or `identity`. Reading .raw skips urllib3's decoding, so an encoded body has to go through iter_content or the compressed bytes land on disk as the payload. Codings @@ -390,7 +391,7 @@ def has_range_header(item: dict[str, Any]) -> bool: """True when a URL item's per-request headers already carry a Range. Such an item is itself a byte-range slice of a larger resource (DASH SegmentBase, - HLS EXT-X-BYTERANGE); probing or part-mode would clobber that Range and fetch the + HLS EXT-X-BYTERANGE). Probing or part-mode would clobber that Range and fetch the wrong bytes, so ranged-parallel strategies must skip these items entirely. """ return any(k.lower() == "range" for k in (item.get("headers") or {})) @@ -401,7 +402,7 @@ def range_covers_full_body(headers: MutableMapping[str, Any], content_length: in A server may ignore Range and answer 200 with the whole resource (RFC 9110 permits this). That body is only the right bytes when the slice starts at byte 0 - and spans exactly the full body; anything else (open-ended, multi-range, nonzero + and spans exactly the full body. Anything else (open-ended, multi-range, nonzero start, span mismatch) stays fail-closed. """ value = str(next((v for k, v in headers.items() if k.lower() == "range"), "")) @@ -413,7 +414,7 @@ def range_covers_full_body(headers: MutableMapping[str, Any], content_length: in def parse_content_range(value: Any) -> Optional[tuple[int, int, Optional[int]]]: - """Parse a 206's ``Content-Range`` into ``(start, end, total)``; None when absent or unparseable. + """Parse a 206's ``Content-Range`` into ``(start, end, total)``. Gives None when absent or unparseable. ``total`` is None for an unknown complete length (``*``). None fails closed: a caller must not place bytes at an offset it cannot prove the server agreed to. @@ -428,7 +429,7 @@ def parse_content_range(value: Any) -> Optional[tuple[int, int, Optional[int]]]: def request_range(headers: MutableMapping[str, Any]) -> Optional[tuple[int, Optional[int]]]: """ - The request's ``Range`` as ``(start, end)``; None when absent or not ``bytes=start-...``. + The request's ``Range`` as ``(start, end)``. Gives None when absent or not ``bytes=start-...``. ``end`` is None for an open-ended ``bytes=start-``, which the caller cannot hold a response to. @@ -443,11 +444,12 @@ def request_range(headers: MutableMapping[str, Any]) -> Optional[tuple[int, Opti def force_unblock_stream(stream: Any) -> None: """Best-effort: shut the stream's socket so a thread parked in a blocking read returns now. - A superseded hedge loser can sit in a full-chunk ``raw.read`` on a slow connection; the - batch's wait-gated shutdown (which guarantees handles are released before the merge sweep) - would otherwise drain that whole slow read. ``socket.shutdown`` makes the read return at once - so the loser unwinds and closes its ``.!dev`` handle immediately. Requests/urllib3 only (rnet - exposes no reachable socket); any failure is swallowed and the caller falls back to waiting. + A superseded hedge loser can sit in a full-chunk ``raw.read`` on a slow connection. The + batch's wait-gated shutdown (which guarantees that every thread has closed its file before + the merge sweep) would otherwise drain that whole slow read. ``socket.shutdown`` makes the + read return at once so the loser unwinds and closes its ``.!dev`` handle immediately. + Requests/urllib3 only (rnet exposes no reachable socket). This function ignores any failure, + and the caller falls back to waiting. """ try: stream.raw._fp.fp.raw._sock.shutdown(socket.SHUT_RDWR) @@ -483,10 +485,10 @@ def plan_tail_parts(size: int, spare: int) -> list[tuple[int, int]]: def tail_boost_engages(remaining: int, pending: int, target: int) -> bool: """True when idle workers outnumber the remaining whole segments. - That is the tail condition where running one worker per remaining segment would leave - workers idle, so splitting those segments into range parts keeps throughput up. Gating on - idle capacity (not a fixed segment count) is what lets the boost fire despite ``remaining`` - draining in strides of the worker target. Pure, so the gate is unit-testable. + That is the tail condition where one download worker for each remaining segment would + leave workers idle, so a split of those segments into range parts keeps throughput up. + Gating on idle capacity (not a fixed segment count) is what lets the boost fire despite + ``remaining`` draining in strides of the download worker target. Pure, so the gate is unit-testable. """ spare = target - pending return remaining > 0 and spare >= 2 and remaining <= spare @@ -621,14 +623,14 @@ def download( Download a file with optimized I/O. Supports both requests.Session and RnetSession for TLS fingerprinting. - RnetSession streams natively; requests.Session reads the raw socket, except - on a content-encoded body, which has to go through iter_content to be decoded. + RnetSession streams natively. requests.Session reads the raw socket, except + on a content-encoded body: only iter_content decodes such a body. - Yields the following download status updates while chunks are downloading: + Yields these download status updates while the chunks download: - {total: 123} (there are 123 chunks to download) - {total: None} (there are an unknown number of chunks to download) - - {advance: 1} (one chunk was downloaded) + - {advance: 1} (the downloader finished one chunk) - {downloaded: "10.1 MB/s"} (currently downloading at a rate of 10.1 MB/s) - {file_downloaded: Path(...), written: 1024} (download finished, has the save path and size) @@ -641,24 +643,26 @@ def download( segmented: If downloads are segments or parts of one bigger file. part_offset: Byte offset to write at within a pre-allocated file. When set (with `part_end`), enables part mode for parallel ranged downloads: - no truncate, no skip-if-exists, no control file; emits only `advance` - events; retries resume mid-part via Range. + no truncate, no skip-if-exists, no control file. Part mode emits only + `advance` events, and a retry resumes mid-part with a Range header. part_end: Inclusive end byte of the part. Required when `part_offset` is set. - claimed: Optional predicate checked before every attempt; when it returns True - the download stops silently (another worker already delivered this file). - racing: Optional predicate selecting read granularity per iteration. Requires - `claimed`. read1 (per-network-arrival reads) is used only on iterations where - it returns True (this segment is being hedge-raced, so a superseded loser must - notice `claimed()` mid-stream); otherwise a blocking full-chunk read is used, - avoiding the per-record read tax on non-racing segments. + claimed: Optional predicate checked before every try. When it returns True, + the download stops silently (another download worker already delivered + this file). + racing: Optional predicate that selects the read granularity for each iteration. + Requires `claimed`. The download uses read1 (per-network-arrival reads) only + on the iterations where the predicate returns True, because the batch then + hedge-races this segment and a superseded loser must notice `claimed()` + mid-read. On every other iteration the download does a blocking full-chunk + read, which avoids the per-record read tax on a segment with no race. abort: Optional local cancel signal. When set, the download stops like a cancel (keeps its partial, returns silently, does not raise). Used by ranged-parallel downloads to abort sibling parts without touching the process-global cancel. on_retry: Optional callback invoked with the raised exception before each retry wait (not on cancel or final exhaustion). Used to feed CDN error signals to - the adaptive worker controller. Callback errors are swallowed. - register: Optional callback ``(stream, active)`` invoked with active=True while a - response is being read and active=False when that read ends. Lets the batch reach + the ``AdaptiveWorkerController``. The downloader ignores callback errors. + register: Optional callback ``(stream, active)`` invoked with active=True while the + downloader reads a response, and with active=False when that read ends. Lets the batch reach a superseded loser's socket to unblock its in-flight read at teardown (see force_unblock_stream) instead of waiting the slow read out. kwargs: Any extra keyword arguments to pass to the session.get() call. Use this @@ -986,11 +990,11 @@ def download( def build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]: - """Picklable spec to rebuild ``session`` in a child process; None if not cheaply rebuildable. + """Picklable spec to rebuild ``session`` in a child process. Gives None when it is not cheaply rebuildable. - Live sessions (sockets, TLS state, threads) can't cross a process boundary, so each child - reconstructs its own from cheap state. RnetSession needs a resolvable impersonate preset; - without one it is not cheaply rebuildable and the caller falls back to a single process. + Live sessions (sockets, TLS state, threads) cannot cross a process boundary, so each child + reconstructs its own from cheap state. RnetSession needs a resolvable impersonate preset. + Without one it is not cheaply rebuildable, and the caller falls back to a single process. """ if session is None: return {"kind": "none"} @@ -1016,11 +1020,11 @@ def build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]: def rebuild_session(spec: dict[str, Any], max_workers: int) -> Optional[Any]: - """Reconstruct a session inside a child process from a spec built by ``build_session_spec``. + """Reconstruct an HTTP session inside a child process from a spec built by ``build_session_spec``. - ``max_workers`` sizes the connection pool to what this child actually runs; the rebuilt - session belongs to one child alone, so mounting an adapter here is safe where remounting a - caller-provided session in ``requests()`` is not. + ``max_workers`` sizes the connection pool to what this child runs. The rebuilt HTTP session + belongs to one child alone, so mounting an adapter here is safe where remounting a + caller-provided HTTP session in ``requests()`` is not. """ kind = spec["kind"] if kind == "requests": @@ -1056,9 +1060,9 @@ def rebuild_session(spec: dict[str, Any], max_workers: int) -> Optional[Any]: def mp_worker(queue: Any, kwargs: dict[str, Any]) -> None: """Child entry point (top-level so ``spawn`` can pickle it). - Rebuilds the session, iterates ``requests()`` for its url chunk, and relays every event over - the shared queue. Errors travel as a ``__mp_error__`` sentinel; a ``__mp_done__`` sentinel is - always sent last so the parent knows this child is finished. + Rebuilds the HTTP session, iterates ``requests()`` for its url slice, and relays every event + over the shared queue. Errors travel as a ``__mp_error__`` sentinel. This child always sends a + ``__mp_done__`` sentinel last, so the parent knows that this child is complete. """ try: # spawn re-imports this module, dropping any timing constants the bench patched in @@ -1094,13 +1098,13 @@ def download_multiprocess( processes: int, debug_logger: Any, ) -> Generator[dict[str, Any], None, None]: - """Fan segments across spawned children, each running ``requests()`` on a strided chunk. + """Fan segments across spawned children, each running ``requests()`` on a strided slice. Child ``k`` takes segments ``k, k+processes, k+2*processes, ...``: slow segments tend to cluster by position (a cold edge, a throttled range region), and striding spreads such a cluster across all children instead of trapping it in one child whose few workers would - bound the tail. Children keep their original global indices via ``index_offset`` and - ``index_stride`` so file names are unchanged. The parent owns aggregate progress: it emits + bound the tail. Children keep their original global indices with ``index_offset`` and + ``index_stride``, so the file names do not change. The parent owns aggregate progress: it emits ``total`` once and computes one speed string from written bytes, dropping the children's own ``total`` and speed events. """ @@ -1248,19 +1252,19 @@ def requests( """ Download files with optimized I/O and adaptive chunk sizing. - Supports both requests.Session and RnetSession. When a RnetSession is - provided (e.g. from a service's get_session()), TLS fingerprinting is preserved - on all segment downloads. + Supports both requests.Session and RnetSession. When you give a RnetSession + (for example, from a service's get_session()), every segment download keeps the + TLS fingerprint. - A file already at its computed save path counts as finished from an earlier run. It is reported as - downloaded at its on-disk size without any request, so calling this again resumes a batch from the - segments the earlier run completed. + A file already at its computed save path counts as finished from an earlier run. This function + reports it as downloaded at its on-disk size without any request, so a second call resumes a + batch from the segments the earlier run completed. - Yields the following download status updates while chunks are downloading: + Yields these download status updates while the chunks download: - {total: 123} (there are 123 chunks to download) - {total: None} (there are an unknown number of chunks to download) - - {advance: 1} (one chunk was downloaded) + - {advance: 1} (the downloader finished one chunk) - {downloaded: "10.1 MB/s"} (currently downloading at a rate of 10.1 MB/s) - {file_downloaded: Path(...), written: 1024} (download finished, has the save path and size) @@ -1276,22 +1280,23 @@ def requests( filename: The filename or filename template to use for each file. The variables you can use are `i` for the URL index and `ext` for the URL extension. headers: A mapping of HTTP Header Key/Values to use for all downloads. - cookies: A mapping of Cookie Key/Values or a Cookie Jar to use for all downloads. + cookies: A mapping of Cookie Key/Values or a `CookieJar` to use for all downloads. proxy: An optional proxy URI to route connections through for all downloads. max_workers: The maximum amount of threads to use for downloads. Defaults to min(16, cpu_count + 4). - session: An optional requests.Session or RnetSession to use. If provided, - it will be used directly (preserving TLS fingerprinting). If None, a new - requests.Session with HTTPAdapter connection pooling will be created. + session: An optional requests.Session or RnetSession to use. When you give one, + this function uses it directly, which keeps the TLS fingerprint. With None, + this function makes a new requests.Session with HTTPAdapter connection pooling. adaptive: Opt-in CDN-aware dynamic segment concurrency for segmented (multi-URL) - downloads. When True the per-track worker count starts moderate and ramps up - or backs off based on measured throughput and CDN errors, capped at - max_workers. When False (default) all segments are submitted upfront using a - fixed worker count, matching the non-adaptive behaviour exactly. + downloads. When True the per-track download worker count starts moderate and + ramps up or backs off based on measured throughput and CDN errors, capped at + max_workers. When False (default) this function submits all segments upfront + with a fixed download worker count, which matches the non-adaptive + behaviour exactly. processes: Split a large segment batch across this many spawned child processes to beat the single-interpreter throughput cap (GIL in the ssl read path). Engaged - only when > 1 and there are at least MP_MIN_SEGMENTS urls; otherwise the behaviour - is byte-identical to a single process. Each child runs its own worker pool. + only when > 1 and there are at least MP_MIN_SEGMENTS urls. If not, the behaviour + is byte-identical to a single process. Each child runs its own download worker pool. Ignored while a download speed limit is set: spawned children cannot share the process-global rate budget, so the batch stays in a single process. index_offset: Added to each url's enumerate index (after ``index_stride`` scaling) @@ -1299,7 +1304,7 @@ def requests( its urls' original global indices/names. Internal: set by the multiprocess path, leave at 0 otherwise. index_stride: Multiplies each url's enumerate index before ``index_offset`` is - added, mapping a strided chunk (``urls[k::stride]``) back to global indices. + added, mapping a strided slice (``urls[k::stride]``) back to global indices. Internal: set by the multiprocess path, leave at 1 otherwise. """ if not urls: diff --git a/unshackle/core/drm/clearkey.py b/unshackle/core/drm/clearkey.py index 0fca7110..afb0bd18 100644 --- a/unshackle/core/drm/clearkey.py +++ b/unshackle/core/drm/clearkey.py @@ -25,12 +25,12 @@ TS_SYNC_CHECKS = 4 class ClearKey: - """AES Clear Key DRM System.""" + """AES-128 ClearKey (HLS) DRM system.""" def __init__(self, key: Union[bytes, str], iv: Optional[Union[bytes, str]] = None): """ - Generally IV should be provided where possible. If not provided, it will be - set to \x00 of the same bit-size of the key. + Give the IV where possible. If you do not give an IV, unshackle sets it to + \x00 of the same bit-size as the content key. """ if isinstance(key, str): key = bytes.fromhex(key.replace("0x", "")) @@ -51,13 +51,13 @@ class ClearKey: self._warned_clear: bool = False def warn_clear(self, reason: str) -> None: - """Warn once per track that a segment was passed through undecrypted.""" + """Warn once per track that unshackle passed a segment through undecrypted.""" if not self._warned_clear: log.warning("ClearKey: manifest signaled AES-128 encryption but %s; passing segment through.", reason) self._warned_clear = True def decrypt(self, path: Path) -> None: - """Decrypt a Track with AES Clear Key DRM.""" + """Decrypt a Track with AES-128 ClearKey (HLS) DRM.""" if not path or not path.exists(): raise ValueError("Tried to decrypt a file that does not exist.") @@ -108,9 +108,9 @@ class ClearKey: Load a ClearKey from an M3U(8) Playlist's EXT-X-KEY. Parameters: - m3u_key: A Key object parsed from a m3u(8) playlist using + m3u_key: A `Key` object parsed from a m3u(8) playlist using the `m3u8` library. - session: Optional session used to request external URIs with. + session: Optional HTTP session used to request external URIs with. Useful to set headers, proxies, cookies, and so forth. """ if not isinstance(m3u_key, Key): diff --git a/unshackle/core/drm/clearkey_cenc.py b/unshackle/core/drm/clearkey_cenc.py index d0d1c834..0a1dcad4 100644 --- a/unshackle/core/drm/clearkey_cenc.py +++ b/unshackle/core/drm/clearkey_cenc.py @@ -29,10 +29,10 @@ def b64url_decode(data: str) -> bytes: class ClearKeyCENC: - """W3C EME ClearKey (org.w3.clearkey) DRM System over MPEG-CENC content. + """W3C EME ClearKey (org.w3.clearkey) DRM System over MPEG-CENC tracks. - Distinct from the HLS AES-128 `ClearKey` class: keys here are delivered by a - license server as a JWK Set keyed by KID, and content is standard CENC + Distinct from the HLS AES-128 `ClearKey` class: a license server gives the keys + here as a JWK Set keyed by KID, and the media is standard CENC (decrypted with shaka-packager/mp4decrypt KID:KEY pairs, same as Widevine). """ @@ -70,7 +70,8 @@ class ClearKeyCENC: def to_dict(self) -> dict[str, Any]: """Serialise this DRM instance for export/import (KIDs + license URL). - Content keys are stored once at the export's track level, not duplicated here. + unshackle stores the content keys once at the export's track level, and does not + duplicate them here. """ data: dict[str, Any] = { "system": "ClearKeyCENC", @@ -81,19 +82,19 @@ class ClearKeyCENC: return data def get_license_challenge(self) -> bytes: - """Build the W3C EME ClearKey JSON license request for the unkeyed KIDs.""" + """Assemble the W3C EME ClearKey JSON license request for the unkeyed KIDs.""" kids = [kid for kid in self.kids if kid not in self.content_keys] or self.kids request = {"kids": [b64url_encode_nopad(kid.bytes) for kid in kids], "type": "temporary"} return json.dumps(request).encode("utf8") def get_content_keys(self, *, licence: Callable, session: Optional[Session] = None) -> None: """ - Obtain Content Keys for this DRM Instance from a ClearKey license server. + Get Content Keys for this DRM Instance from a ClearKey license server. - The licence param is expected to be a function and will be provided with the - W3C JSON license request as `challenge`. It may return the JWK Set license as - a dict, JSON str, or bytes. If it returns None and the manifest provided a - Laurl, the challenge is POSTed there directly instead. + The licence param is a function. unshackle gives it the W3C JSON license + request as `challenge`. It may return the JWK Set license as a dict, JSON str, + or bytes. If it returns None and the manifest gave a Laurl, unshackle POSTs the + challenge there directly instead. """ if all(kid in self.content_keys for kid in self.kids): return @@ -183,7 +184,7 @@ class ClearKeyCENC: path: Path to the encrypted file to decrypt Raises: EnvironmentError if the required decryption executable could not be found. - ValueError if the track has not yet been downloaded. + ValueError if unshackle has not yet downloaded the track. SubprocessError if the decryption process returned a non-zero exit code. """ if not self.content_keys: @@ -264,7 +265,7 @@ class ClearKeyCENC: shutil.move(output_path, path) def decrypt_with_shaka_packager(self, path: Path) -> None: - """Decrypt using Shaka Packager""" + """Decrypt with shaka-packager""" if not binaries.ShakaPackager: raise EnvironmentError("Shaka Packager executable not found but is required.") diff --git a/unshackle/core/drm/monalisa.py b/unshackle/core/drm/monalisa.py index 50b06328..afcd4a58 100644 --- a/unshackle/core/drm/monalisa.py +++ b/unshackle/core/drm/monalisa.py @@ -1,7 +1,7 @@ """ MonaLisa DRM System. -A WASM-based DRM system that uses local key extraction and two-stage +A WASM-based DRM system that uses local content key extraction and two-stage segment decryption (ML-Worker binary + AES-ECB). """ @@ -26,20 +26,20 @@ class MonaLisa: MonaLisa DRM System. Unlike Widevine/PlayReady, MonaLisa does not use a challenge/response flow - with a license server. Instead, the PSSH value (ticket) is provided directly - by the service API, and keys are extracted locally via a WASM module. + with a license server. Instead, the service API gives the PSSH value (ticket) + directly, and a WASM module extracts the content keys locally. - Decryption is performed in two stages: + Decryption has two stages: 1. ML-Worker binary: Removes MonaLisa encryption layer (bbts -> ents) - 2. AES-ECB decryption: Final decryption with service-provided key + 2. AES-ECB decryption: Final decryption with the service-supplied content key """ class Exceptions: class TicketNotFound(Exception): - """Raised when no PSSH/ticket data is provided.""" + """Raised when the caller gives no PSSH/ticket data.""" class KeyExtractionFailed(Exception): - """Raised when key extraction from the ticket fails.""" + """Raised when content key extraction from the ticket fails.""" class WorkerNotFound(Exception): """Raised when the ML-Worker binary is not found.""" @@ -55,17 +55,17 @@ class MonaLisa: **kwargs: Any, ): """ - Initialize MonaLisa DRM. + Initialise MonaLisa DRM. Args: ticket: PSSH value from service API (base64 string or raw bytes). - aes_key: AES-ECB key for second-stage decryption (hex string or bytes). + aes_key: AES-ECB content key for second-stage decryption (hex string or bytes). device_path: Path to the CDM device file (.mld). **kwargs: Additional metadata stored in self.data. Raises: TicketNotFound: If ticket/PSSH is empty. - KeyExtractionFailed: If key extraction fails. + KeyExtractionFailed: If content key extraction fails. """ if not ticket: raise MonaLisa.Exceptions.TicketNotFound("No PSSH/ticket data provided.") @@ -112,11 +112,11 @@ class MonaLisa: device_path: Path, ) -> MonaLisa: """ - Create a MonaLisa DRM instance from a PSSH/ticket. + Make a MonaLisa DRM instance from a PSSH/ticket. Args: ticket: PSSH value from service API. - aes_key: AES-ECB key for second-stage decryption. + aes_key: AES-ECB content key for second-stage decryption. device_path: Path to the CDM device file (.mld). Returns: @@ -190,7 +190,7 @@ class MonaLisa: Get content keys in the same format as Widevine/PlayReady. Returns: - Dictionary mapping KID to key hex string. + Dictionary mapping KID to the content key hex string. """ if self._kid and self._key: return {self._kid: self._key} @@ -284,9 +284,9 @@ class MonaLisa: def decrypt(self, _path: Path) -> None: """ - MonaLisa uses per-segment decryption during download via the - on_segment_downloaded callback. By the time this method is called, - the content has already been decrypted and muxed into a container. + MonaLisa uses per-segment decryption during download, through the + on_segment_downloaded callback. When the caller calls this method, + unshackle has already decrypted the media and muxed it into a container. Args: _path: Path to the file (ignored). diff --git a/unshackle/core/drm/playready.py b/unshackle/core/drm/playready.py index e0858dc2..2893efd5 100644 --- a/unshackle/core/drm/playready.py +++ b/unshackle/core/drm/playready.py @@ -253,7 +253,8 @@ class PlayReady: def to_dict(self) -> dict[str, Any]: """Serialise this DRM instance for export/import (PSSH + KIDs). - Content keys are stored once at the export's track level, not duplicated here. + unshackle stores the content keys once at the export's track level, and does not + duplicate them here. """ return { "system": "PlayReady", @@ -372,7 +373,7 @@ class PlayReady: """Return the decoded error string if a revocation HRESULT is present, else None. Reads the code from the raw SOAP body (0x8004C065) - or from an exception message that carries it, so any service is covered. + or from an exception message that carries it, so this method covers any service. """ from unshackle.core.drm.playready_errors import describe, is_revocation @@ -391,7 +392,7 @@ class PlayReady: path: Path to the encrypted file to decrypt Raises: EnvironmentError if the required decryption executable could not be found. - ValueError if the track has not yet been downloaded. + ValueError if unshackle has not yet downloaded the track. SubprocessError if the decryption process returned a non-zero exit code. """ if not self.content_keys: @@ -483,7 +484,7 @@ class PlayReady: shutil.move(output_path, path) def decrypt_with_shaka_packager(self, path: Path) -> None: - """Decrypt using Shaka Packager (original method)""" + """Decrypt with shaka-packager (original method)""" if not binaries.ShakaPackager: raise EnvironmentError("Shaka Packager executable not found but is required.") diff --git a/unshackle/core/drm/playready_errors.py b/unshackle/core/drm/playready_errors.py index 335341fa..f58452da 100644 --- a/unshackle/core/drm/playready_errors.py +++ b/unshackle/core/drm/playready_errors.py @@ -851,7 +851,7 @@ REVOCATION_CODES = frozenset( def is_revocation(code: int) -> bool: - """True if the HRESULT means a device or certificate was revoked (signed or unsigned).""" + """True if the HRESULT means a revoked device or a revoked certificate (signed or unsigned).""" return code is not None and (code & 0xFFFFFFFF) in REVOCATION_CODES diff --git a/unshackle/core/drm/widevine.py b/unshackle/core/drm/widevine.py index 2f72ad76..6d6690e5 100644 --- a/unshackle/core/drm/widevine.py +++ b/unshackle/core/drm/widevine.py @@ -70,14 +70,14 @@ class Widevine: """ Get PSSH and KID from within the Initiation Segment of the Track Data. It also tries to get PSSH and KID from other track data like M3U8 data - as well as through ffprobe. + as well as through FFprobe. - Create a Widevine DRM System object from a track's information. - This should only be used if a PSSH could not be provided directly. + Make a Widevine DRM System object from a track's information. + Use this method only when a PSSH cannot be given directly. It is *rare* to need to use this. - You may provide your own requests session to be able to use custom - headers and more. + You can give your own `requests.Session` to use custom headers + and more. Raises: PSSHNotFound: If the PSSH was not found within the data. @@ -127,7 +127,7 @@ class Widevine: """ Get PSSH and KID from within Initialization Segment Data. - This should only be used if a PSSH could not be provided directly. + Use this method only when a PSSH cannot be given directly. It is *rare* to need to use this. Raises: @@ -184,7 +184,8 @@ class Widevine: def to_dict(self) -> dict[str, Any]: """Serialise this DRM instance for export/import (PSSH + KIDs). - Content keys are stored once at the export's track level, not duplicated here. + unshackle stores the content keys once at the export's track level, and does not + duplicate them here. """ return { "system": "Widevine", @@ -194,9 +195,9 @@ class Widevine: def get_content_keys(self, cdm: WidevineCdm, certificate: Callable, licence: Callable) -> None: """ - Create a CDM Session and obtain Content Keys for this DRM Instance. - The certificate and license params are expected to be a function and will - be provided with the challenge and session ID. + Make a CDM Session and get Content Keys for this DRM Instance. + The certificate and license params are functions. unshackle gives them + the challenge and the session ID. """ for kid in self.kids: if kid in self.content_keys: @@ -263,9 +264,9 @@ class Widevine: def get_NF_content_keys(self, cdm: WidevineCdm, certificate: Callable, licence: Callable) -> None: """ - Create a CDM Session and obtain Content Keys for this DRM Instance. - The certificate and license params are expected to be a function and will - be provided with the challenge and session ID. + Make a CDM Session and get Content Keys for this DRM Instance. + The certificate and license params are functions. unshackle gives them + the challenge and the session ID. """ for kid in self.kids: if kid in self.content_keys: @@ -307,7 +308,7 @@ class Widevine: path: Path to the encrypted file to decrypt Raises: EnvironmentError if the required decryption executable could not be found. - ValueError if the track has not yet been downloaded. + ValueError if unshackle has not yet downloaded the track. SubprocessError if the decryption process returned a non-zero exit code. """ if not self.content_keys: @@ -399,7 +400,7 @@ class Widevine: shutil.move(output_path, path) def decrypt_with_shaka_packager(self, path: Path) -> None: - """Decrypt using Shaka Packager (original method)""" + """Decrypt with shaka-packager (original method)""" if not binaries.ShakaPackager: raise EnvironmentError("Shaka Packager executable not found but is required.") diff --git a/unshackle/core/events.py b/unshackle/core/events.py index d5622b96..a85451ef 100644 --- a/unshackle/core/events.py +++ b/unshackle/core/events.py @@ -31,10 +31,10 @@ class Events: Parameters: event_type: The Events.Type to subscribe to. - callback: The function or lambda to call on event emit. - ephemeral: Unsubscribe the callback from the event on first emit. - Note that this is not thread-safe and may be called multiple - times at roughly the same time. + callback: The function or lambda to call when `emit()` sends the event. + ephemeral: Unsubscribe the callback from the event at the first `emit()` call. + Note that this is not thread-safe, so unshackle can call the + callback several times at roughly the same time. """ [self.__subscriptions, self.__ephemeral][ephemeral][event_type].append(callback) @@ -44,7 +44,7 @@ class Events: Parameters: event_type: The Events.Type to unsubscribe from. - callback: The function or lambda to remove from event emit. + callback: The function or lambda to remove from the `emit()` callbacks. """ if callback in self.__subscriptions[event_type]: self.__subscriptions[event_type].remove(callback) @@ -53,10 +53,10 @@ class Events: def emit(self, event_type: Events.Types, *args: Any, **kwargs: Any) -> None: """ - Emit an Event, executing all subscribed Callbacks. + Send an Event to all subscribed Callbacks. Parameters: - event_type: The Events.Type to emit. + event_type: The Events.Type to send. args: Positional arguments to pass to callbacks. kwargs: Keyword arguments to pass to callbacks. """ diff --git a/unshackle/core/import_service.py b/unshackle/core/import_service.py index adf26a18..8838e303 100644 --- a/unshackle/core/import_service.py +++ b/unshackle/core/import_service.py @@ -48,9 +48,9 @@ def resolve_import_manifest_data( language: Any, title: Title_T, ) -> None: - """Populate ``track.data`` for DASH/ISM tracks rebuilt from export dicts. + """Fill ``track.data`` for DASH/ISM tracks rebuilt from export dicts. - Exports may omit ``manifest_url`` (e.g. when the service never set + Exports can be without ``manifest_url`` (for example, when the service never set ``title.tracks.manifest_url``) or carry one MPD per adaptation set. ``Track.from_dict`` does not serialise manifest XML, so each exported track's ``url`` is re-fetched and matched to a locally parsed representation before download. @@ -98,9 +98,10 @@ def resolve_import_manifest_data( class ImportService: """Reconstructs a download from an export JSON. - Auth and licensing are skipped; tracks are rebuilt from the export and keys injected - directly. ``_server_cdm``/``_server_cdm_type`` keep their underscores: dl.py reads them - via getattr as the server-CDM contract that skips client licensing. + ImportService does not authenticate and does not license. It rebuilds the tracks from + the export and injects the keys directly. ``_server_cdm``/``_server_cdm_type`` keep their + underscores: dl.py reads them through getattr as the server-CDM contract that skips + client licensing. """ ALIASES: tuple[str, ...] = () @@ -142,10 +143,11 @@ class ImportService: @staticmethod def build_session(ctx: click.Context, region: Optional[str] = None) -> requests.Session: - """Session for re-fetching the manifest. + """HTTP session for re-fetching the manifest. - Honours the importer's ``--proxy``; otherwise falls back to the export region as a - geofence. An explicit proxy that fails to resolve raises; a region fallback warns. + Honours the importer's ``--proxy``. Without one, it falls back to the export region + as a geofence. An explicit proxy that unshackle cannot find raises an error. A region + fallback gives a warning. """ session = requests.Session() session.headers.update(config.headers) @@ -201,11 +203,11 @@ class ImportService: """Reconstruct the title's tracks from the export. DASH/ISM: re-fetch and re-parse ``manifest_url`` for the full ladder on that MPD (the importer - picks quality with normal dl flags; keys are injected by KID later), then merge in exported + picks quality with normal dl flags, and injects the keys by KID later), then merge in exported DASH/ISM tracks from other MPDs (e.g. a separate HEVC manifest) and direct-URL side-loads. A service side-loads its own subtitles when the manifest's are the worse copy, so any - direct-URL subtitle in the export means the whole re-parsed subtitle set is dropped in - favour of the exported one. + direct-URL subtitle in the export means unshackle drops the whole re-parsed subtitle + set in favour of the exported one. HLS/URL: rebuild from the stored per-track dicts, since the variant is re-fetched from track.url at download time and a master playlist can hand out a new token on every fetch. """ @@ -340,9 +342,10 @@ class ImportService: def resolve_server_keys(self, title: Title_T) -> None: """Inject exported keys into the selected encrypted tracks by KID (no network). - Called by dl.py after selection. Only encrypted video/audio are touched; encrypted - DASH tracks (no DRM at parse time) get a stub holding the keys, which - DASH.download_track preserves. decrypt() applies the key whose KID matches the media. + dl.py calls this method after selection. It touches only encrypted video and audio + tracks. Encrypted DASH tracks (no DRM at parse time) get a stub holding the keys, + which DASH.download_track preserves. decrypt() applies the content key whose KID + matches the media. """ pool = self.key_pool() if not pool: @@ -364,8 +367,9 @@ class ImportService: def track_is_encrypted(track: Any) -> bool: """True if the track carries DRM or its manifest declares protection. - ISM is checked as well as DASH because ISM.download_track reads only ``track.drm``. A rung - that misses key injection here downloads encrypted and muxes without error. DASH re-derives + This method examines ISM as well as DASH, because ISM.download_track reads only + ``track.drm``. A rung that misses content key injection here downloads encrypted and + muxes without error. DASH re-derives its DRM from the manifest elements, so it survives the same omission. """ if track.drm: diff --git a/unshackle/core/manifests/dash.py b/unshackle/core/manifests/dash.py index 51f28afb..5220429f 100644 --- a/unshackle/core/manifests/dash.py +++ b/unshackle/core/manifests/dash.py @@ -101,8 +101,8 @@ class DASH: Convert an MPEG-DASH document to Video, Audio and Subtitle Track objects. Parameters: - language: The Title's Original Recorded Language. It will also be used as a fallback - track language value if the manifest does not list language information. + language: The Title's Original Recorded Language. unshackle also uses it as a + fallback track language value when the manifest gives no language information. period_filter: Filter out period's within the manifest. Only the first main-content period becomes tracks. Later periods do not add tracks of their @@ -636,7 +636,7 @@ class DASH: ) -> bool: """ Whether every segment is a byte range of one parent resource whose [0, init_len) - prefix is the init segment, so the whole resource can be downloaded in one pass. + prefix is the init segment, so unshackle can download the whole resource in one pass. Returns False for subtitle tracks, an unknown init length, mixed URLs, malformed ranges, and anything but one gapless run in document order starting at the init @@ -671,7 +671,7 @@ class DASH: @staticmethod def is_content_period(period: Element, filtered_period_ids: list[str]) -> bool: - """Check if a period is a valid content period (not an ad, not filtered, not trick mode).""" + """Whether a period is a valid content period (not an ad, not filtered, not trick mode).""" period_id = period.get("id") if period_id and period_id in filtered_period_ids: return False @@ -686,11 +686,11 @@ class DASH: @staticmethod def merge_segment_templates(adaptation_set: Element, representation: Element) -> Optional[Element]: """ - Build the effective SegmentTemplate for a Representation by cascading the + Assemble the effective SegmentTemplate for a Representation by cascading the AdaptationSet > Representation levels (ISO/IEC 23009-1 5.3.9.1). - The Representation-level node, when present, is the base; attributes and the - SegmentTimeline child it does not declare are inherited from the AdaptationSet-level + The Representation-level node, when present, is the base. It takes the attributes + and the SegmentTimeline child that it does not declare from the AdaptationSet-level node. Returns None if no SegmentTemplate exists at either level. """ levels = [node.find("SegmentTemplate") for node in (adaptation_set, representation)] @@ -946,8 +946,8 @@ class DASH: """ Get Language (if any) from the AdaptationSet or Representation. - A fallback language may be provided if no language information could be - retrieved. + The caller can give a fallback language for when the manifest holds no + language information. """ options = [] @@ -1012,7 +1012,7 @@ class DASH: @staticmethod def is_trick_mode(adaptation_set: Element) -> bool: - """Check if contents of Adaptation Set is a Trick-Mode stream.""" + """Whether the Adaptation Set carries Trick-Mode tracks.""" essential_props = adaptation_set.findall("EssentialProperty") supplemental_props = adaptation_set.findall("SupplementalProperty") @@ -1038,7 +1038,7 @@ class DASH: Representation ids are only unique within an AdaptationSet, so the set matching `adaptation_set`'s identity wins over an earlier sibling that reuses the same id - for different content (e.g. audio description vs dialog). + for a different track (for example, audio description vs dialog). """ as_key = DASH.adaptation_set_key(adaptation_set) @@ -1071,7 +1071,7 @@ class DASH: @staticmethod def is_descriptive(adaptation_set: Element) -> bool: - """Check if contents of Adaptation Set is Descriptive.""" + """Whether the Adaptation Set is Descriptive.""" return any( (x.get("schemeIdUri"), x.get("value")) in (("urn:mpeg:dash:role:2011", "descriptive"), ("urn:tva:metadata:cs:AudioPurposeCS:2007", "1")) @@ -1080,7 +1080,7 @@ class DASH: @staticmethod def is_forced(adaptation_set: Element) -> bool: - """Check if contents of Adaptation Set is a Forced Subtitle.""" + """Whether the Adaptation Set is a Forced Subtitle.""" return any( x.get("schemeIdUri") == "urn:mpeg:dash:role:2011" and x.get("value") in ("forced-subtitle", "forced_subtitle") @@ -1089,7 +1089,7 @@ class DASH: @staticmethod def is_sdh(adaptation_set: Element) -> bool: - """Check if contents of Adaptation Set is for the Hearing Impaired.""" + """Whether the Adaptation Set is SDH.""" return any( (x.get("schemeIdUri"), x.get("value")) == ("urn:tva:metadata:cs:AudioPurposeCS:2007", "2") for x in adaptation_set.findall("Accessibility") @@ -1097,7 +1097,7 @@ class DASH: @staticmethod def is_closed_caption(adaptation_set: Element) -> bool: - """Check if contents of Adaptation Set is a Closed Caption Subtitle.""" + """Whether the Adaptation Set is a Closed Caption Subtitle.""" return any( (x.get("schemeIdUri"), x.get("value")) == ("urn:mpeg:dash:role:2011", "caption") for x in adaptation_set.findall("Role") diff --git a/unshackle/core/manifests/hls.py b/unshackle/core/manifests/hls.py index d46b6966..ef5e4747 100644 --- a/unshackle/core/manifests/hls.py +++ b/unshackle/core/manifests/hls.py @@ -750,7 +750,7 @@ class HLS: Parameters: to: The output file with all merged data. via: List of files to merge, in sequence. - delete: Delete the file once it's been merged. + delete: Delete each source file after this function merges it. include_map_data: Whether to include the init map data. """ prepend = include_map_data and map_data and map_data[1] @@ -774,10 +774,10 @@ class HLS: """ Decrypt all segments that uses the currently set DRM. - All segments that will be decrypted with this DRM will be merged together - in sequence, prefixed with the init data (if any), and then deleted. Once - merged they will be decrypted. The merged and decrypted file names state - the range of segments that were used. + This function merges every segment of this DRM together in sequence, adds + the init data (if any) at the start, and then deletes the segment files. It + then decrypts the merged file. The merged and decrypted file names state the + range of segments the file holds. Parameters: include_this_segment: Whether to include the current segment in the @@ -842,8 +842,8 @@ class HLS: """ Merge all segments of the discontinuity. - All segment files for this discontinuity must already be downloaded and - already decrypted (if it needs to be decrypted). + The caller must download every segment file of this discontinuity first. + The caller must also decrypt them, when they need decryption. Parameters: include_this_segment: Whether to include the current segment in the @@ -1131,7 +1131,7 @@ class HLS: def parse_session_data_keys( manifest: M3U8, session: Optional[Union[Session, RnetSession]] = None ) -> list[m3u8.model.Key]: - """Parse `com.apple.hls.keys` session data and return Key objects.""" + """Parse the `com.apple.hls.keys` `EXT-X-SESSION-DATA` entry into `Key` objects.""" keys: list[m3u8.model.Key] = [] for data in getattr(manifest, "session_data", []) or []: @@ -1189,8 +1189,8 @@ class HLS: """ Filter EXT-X-KEY entries to only include those matching the CDM type. - This ensures we select the correct DRM system (Widevine vs PlayReady) - based on what CDM is configured, avoiding license request failures. + This makes sure that we select the correct DRM system (Widevine or PlayReady) + for the CDM in the configuration, so the challenge does not fail. """ playready_urn = f"urn:uuid:{PR_PSSH.SYSTEM_ID}" playready_keyformats = {playready_urn, "com.microsoft.playready"} @@ -1234,15 +1234,15 @@ class HLS: @staticmethod def get_supported_key(keys: list[Union[m3u8.model.SessionKey, m3u8.model.Key]]) -> Optional[m3u8.Key]: """ - Get a support Key System from a list of Key systems. + Get the first DRM system unshackle can use from the given `EXT-X-KEY` entries. - Note that the key systems are chosen in an opinionated order. + Note that unshackle chooses the DRM systems in an opinionated order. - Returns None if one of the key systems is method=NONE, which means all segments - from hence forth should be treated as plain text until another key system is - encountered, unless it's also method=NONE. + Returns None if one of the DRM systems is method=NONE. Then every segment from + that point on is plain text, until another DRM system that is not method=NONE + comes in the playlist. - Raises NotImplementedError if none of the key systems are supported. + Raises NotImplementedError when unshackle can use none of the DRM systems. """ if any(key.method == "NONE" for key in keys): return None @@ -1278,11 +1278,11 @@ class HLS: Convert HLS EXT-X-KEY data to an initialized DRM object. Parameters: - key: m3u8 key system (EXT-X-KEY) object. - session: Optional session used to request AES-128 URIs. + key: The m3u8 `EXT-X-KEY` object. + session: Optional HTTP session used to request AES-128 URIs. Useful to set headers, proxies, cookies, and so forth. - Raises a NotImplementedError if the key system is not supported. + Raises a NotImplementedError when unshackle cannot use the DRM system. """ if not isinstance(session, (Session, RnetSession, type(None))): raise TypeError(f"Expected session to be a {Session} or {RnetSession}, not {type(session)}") @@ -1323,10 +1323,10 @@ class HLS: Convert HLS EXT-X-KEY data to initialized DRM objects. Parameters: - keys: m3u8 key system (EXT-X-KEY) objects. + keys: The m3u8 `EXT-X-KEY` objects. proxy: Optional proxy string used for requesting AES-128 URIs. - Raises a NotImplementedError if none of the key systems are supported. + Raises a NotImplementedError when unshackle can use none of the DRM systems. """ unsupported_keys: list[m3u8.Key] = [] drm_objects: list[DRM_T] = [] diff --git a/unshackle/core/manifests/ism.py b/unshackle/core/manifests/ism.py index 91de896b..271aeb2b 100644 --- a/unshackle/core/manifests/ism.py +++ b/unshackle/core/manifests/ism.py @@ -110,7 +110,7 @@ class ISM: def get_video_range_and_fps(fourcc: str, codec_private_data: str) -> tuple[Video.Range, Optional[float]]: """Derive colour range and fps from the SPS VUI in CodecPrivateData, since Smooth manifests carry neither as attributes. Range soft-fails to - SDR; fps is None for non-HEVC codecs and VUIs without timing info.""" + SDR. fps is None for non-HEVC codecs and VUIs without timing info.""" fourcc = (fourcc or "").upper() try: cpd = bytes.fromhex(codec_private_data or "") diff --git a/unshackle/core/manifests/ism_init.py b/unshackle/core/manifests/ism_init.py index 13fb1183..bd137fc6 100644 --- a/unshackle/core/manifests/ism_init.py +++ b/unshackle/core/manifests/ism_init.py @@ -2,12 +2,12 @@ Synthesize an ISO-BMFF initialization segment (ftyp + moov) for ISM / Smooth Streaming tracks. -Smooth Streaming fragments are bare ``moof`` + ``mdat`` pairs; the server never -sends a ``moov``. The init box must be reconstructed from the manifest's -``CodecPrivateData`` (and, for protected content, the track KID) before a muxer -or decryptor such as shaka-packager can parse the stream. Ported from yt-dlp's -``write_piff_header`` and N_m3u8DL-RE's ``MSSMoovProcessor`` with HEVC, Dolby -Vision, EC-3, TTML and CENC (PIFF) support. +Smooth Streaming fragments are bare ``moof`` + ``mdat`` pairs. The server never +sends a ``moov``. unshackle must reconstruct the init box from the manifest's +``CodecPrivateData`` (and, for a protected track, the track KID) before a muxer +or decryptor such as shaka-packager can parse the track. Ported from yt-dlp's +``write_piff_header`` and N_m3u8DL-RE's ``MSSMoovProcessor``. This port also +writes the HEVC, Dolby Vision, EC-3, TTML and CENC (PIFF) boxes. ``piff_senc_to_cenc`` rewrites the fragments themselves, which neither port does: shaka-packager cannot read the PIFF sample-encryption box, so its payload is @@ -94,7 +94,7 @@ def split_nal_units(codec_private_data: bytes) -> list[bytes]: def remove_emulation_prevention(data: bytes) -> bytes: - """Strip H.26x emulation-prevention bytes (the 0x03 in any 00 00 03 run). + """Remove H.26x emulation-prevention bytes (the 0x03 in any 00 00 03 run). The byte after a consumed escape is data, even another 0x03, so the scan must skip past it rather than re-examine (a naive trailing-window check @@ -142,7 +142,7 @@ class BitReader: def read_hevc_sps_to_bit_depth(r: BitReader) -> tuple[int, int, int, int]: - """Advance reader through bit_depth_chroma_minus8; returns + """Advance reader through bit_depth_chroma_minus8. Returns (chroma_format_idc, bit_depth_luma, bit_depth_chroma, max_sub_layers_minus1).""" r.read_bits(16) # NAL unit header r.read_bits(4) # sps_video_parameter_set_id @@ -202,7 +202,7 @@ def skip_hevc_scaling_list_data(r: BitReader) -> None: def skip_hevc_st_ref_pic_set(r: BitReader, idx: int, num_delta_pocs: list[int]) -> int: - """Skip one st_ref_pic_set() (H.265 7.3.7); returns its NumDeltaPocs.""" + """Skip one st_ref_pic_set() (H.265 7.3.7). Returns its NumDeltaPocs.""" if idx and r.read_bits(1): # inter_ref_pic_set_prediction_flag r.read_bits(1) # delta_rps_sign r.read_ue() # abs_delta_rps_minus1 @@ -223,7 +223,7 @@ def skip_hevc_st_ref_pic_set(r: BitReader, idx: int, num_delta_pocs: list[int]) def parse_hevc_sps_vui(sps_rbsp: bytes) -> tuple[Optional[tuple[int, int, int]], Optional[float]]: """((colour_primaries, transfer_characteristics, matrix_coeffs), fps) from a - de-emulated HEVC SPS VUI; either element is None when absent.""" + de-emulated HEVC SPS VUI. Either element is None when absent.""" r = BitReader(sps_rbsp) _, _, _, max_sub_layers_minus1 = read_hevc_sps_to_bit_depth(r) log2_max_poc_lsb_minus4 = r.read_ue() @@ -287,10 +287,10 @@ HEVC_FOURCCS = frozenset(("HVC1", "HEV1", "HEVC", "H265", "DVHE", "DVH1")) def parse_codec_private_data_vui( fourcc: str, codec_private_data: bytes ) -> tuple[Optional[tuple[int, int, int]], Optional[float]]: - """(colour triple, fps) from the SPS VUI in HEVC CodecPrivateData; (None, None) - when the codec is unsupported or the data is malformed. H.264 is excluded - on purpose: its VUI timing is field-based and often misdeclared, so fps - read from it can't be trusted.""" + """(colour triple, fps) from the SPS VUI in HEVC CodecPrivateData. Gives + (None, None) for a codec this parser does not read, and for malformed data. + This parser excludes H.264 on purpose: its VUI timing is field-based and + often misdeclared, so unshackle cannot trust the fps it reads from there.""" if (fourcc or "").upper() not in HEVC_FOURCCS: return None, None try: @@ -304,18 +304,19 @@ def parse_codec_private_data_vui( def parse_codec_private_data_colour(fourcc: str, codec_private_data: bytes) -> Optional[tuple[int, int, int]]: - """SPS VUI colour triple from HEVC CodecPrivateData; None when the codec - is unsupported, no colour description, or malformed data.""" + """SPS VUI colour triple from HEVC CodecPrivateData. Gives None for a codec + this parser does not read, for data with no colour description, and for + malformed data.""" return parse_codec_private_data_vui(fourcc, codec_private_data)[0] def iter_boxes(data: bytes, start: int, end: int) -> Iterator[tuple[bytes, Optional[bytes], int, int]]: """Yield (type, uuid_usertype, payload_start, box_end) for each child box. - Stops silently at the first box whose declared size cannot be walked. A short - result therefore reports how far the walk got, and a caller counting survivors - should treat it as unverified. box_end may exceed end when the declared size - overruns; clamp it before slicing. + Stops silently at the first box with a declared size the walk cannot follow. + A short result therefore reports how far the walk got, and a caller that counts + survivors must treat it as unverified. box_end may exceed end when the declared + size overruns. Clamp it before slicing. """ offset = start while offset + 8 <= end: @@ -338,7 +339,7 @@ def iter_boxes(data: bytes, start: int, end: int) -> Iterator[tuple[bytes, Optio def find_box(data: bytes, start: int, end: int, target: bytes) -> Optional[tuple[int, int]]: - """Find the first child box of the given type; return (payload_start, end).""" + """Find the first child box of the given type. Returns (payload_start, end).""" for box_type, _, body, box_end in iter_boxes(data, start, end): if box_type == target: return body, box_end @@ -348,9 +349,9 @@ def find_box(data: bytes, start: int, end: int, target: bytes) -> Optional[tuple def read_track_id(fragment: bytes) -> Optional[int]: """Read the track_ID from a fragment's moof/traf/tfhd box, if present. - Smooth fragments declare their own track_ID; the synthesized moov must use + Smooth fragments declare their own track_ID. The synthesized moov must use the same value or the muxer cannot associate samples with the track. The - track_ID sits before any tfhd optional fields, so the flags don't matter. + track_ID sits before any tfhd optional fields, so the flags do not matter. """ moof = find_box(fragment, 0, len(fragment), b"moof") if not moof: @@ -439,8 +440,8 @@ def piff_senc_to_cenc(fragment: bytes, iv_size: int = 8) -> bytes: shaka-packager does not consume the PIFF uuid form. It finds no per-sample IVs, falls through to the constant-IV path, and aborts with "IV cannot be empty". mp4decrypt - accepts either form, and the payloads are identical once the optional override header - is stripped, so emitting 'senc' serves both decrypters. + accepts either form, and the payloads are identical without the optional override + header, so emitting 'senc' serves both decrypters. The rewrite preserves length: bytes reclaimed from the uuid header become a 'free' box. The moof's byte length must stay the same. trun's data_offset resolves against tfhd's @@ -503,7 +504,7 @@ def piff_senc_to_cenc(fragment: bytes, iv_size: int = 8) -> bytes: def build_avcc(codec_private_data: bytes, nal_length_size: int = 4) -> bytes: - """Build an avcC (AVC decoder config) box from SPS+PPS CodecPrivateData.""" + """Assemble an avcC (AVC decoder config) box from SPS+PPS CodecPrivateData.""" nals = split_nal_units(codec_private_data) # Pick parameter sets by H.264 NAL type (low 5 bits): 7 = SPS, 8 = PPS. # Manifests do not guarantee SPS-first ordering. @@ -523,11 +524,12 @@ def build_avcc(codec_private_data: bytes, nal_length_size: int = 4) -> bytes: def build_hvcc(codec_private_data: bytes, nal_length_size: int = 4) -> bytes: """ - Build an hvcC (HEVC decoder config) box from VPS+SPS+PPS CodecPrivateData. + Assemble an hvcC (HEVC decoder config) box from VPS+SPS+PPS CodecPrivateData. - Profile/tier/level bytes are lifted from the SPS profile_tier_level; chroma - format and bit depths are parsed from the SPS so 10-bit/HDR streams signal - correctly (falls back to 8-bit 4:2:0 on malformed SPS data). + The profile/tier/level bytes come from the SPS profile_tier_level. This + function reads the chroma format and the bit depths from the SPS, so 10-bit + and HDR tracks signal correctly. On malformed SPS data it falls back to + 8-bit 4:2:0. """ nals = split_nal_units(codec_private_data) if len(nals) < 3: @@ -586,7 +588,7 @@ def build_hvcc(codec_private_data: bytes, nal_length_size: int = 4) -> bytes: def build_esds(codec_private_data: bytes) -> bytes: - """Build an esds box wrapping the AAC AudioSpecificConfig.""" + """Assemble an esds box that wraps the AAC AudioSpecificConfig.""" asc = codec_private_data # DecoderSpecificInfo (tag 0x05) dsi = u8.pack(0x05) + u8.pack(len(asc)) + asc @@ -608,13 +610,13 @@ def build_esds(codec_private_data: bytes) -> bytes: def build_dec3(codec_private_data: bytes) -> Optional[bytes]: - """Build a dec3 (EC-3 specific) box from Smooth EC-3 CodecPrivateData. + """Assemble a dec3 (EC-3 specific) box from Smooth EC-3 CodecPrivateData. Smooth EC-3 CodecPrivateData ([MS-SSTR] AudioTag 65534) serializes a WAVEFORMATEXTENSIBLE, sometimes the full structure, sometimes only its extension (samples-per-block + channel mask + DD+ SubFormat GUID), with the raw dec3 payload (ETSI TS 102 366 F.6) after the GUID. Returns None - when the GUID is absent; decoders still sync from EC-3 frames in mdat. + when the GUID is absent. Decoders still sync from EC-3 frames in mdat. """ guid_at = codec_private_data.find(DOLBY_DIGITAL_PLUS_GUID) if guid_at != -1 and len(codec_private_data) > guid_at + 16: @@ -623,9 +625,9 @@ def build_dec3(codec_private_data: bytes) -> Optional[bytes]: def synthesize_aac_codec_private_data(fourcc: str, sampling_rate: int, channels: int) -> bytes: - """Generate the AAC AudioSpecificConfig when the manifest omits it. + """Make the AAC AudioSpecificConfig when the manifest omits it. - AACL -> 2-byte AAC-LC config; AACH -> 4-byte HE-AAC (SBR, AOT 5) config + AACL -> 2-byte AAC-LC config. AACH -> 4-byte HE-AAC (SBR, AOT 5) config with the extension sampling frequency at twice the core rate. """ freq = AAC_SAMPLING_FREQUENCY_INDEX.get(sampling_rate, 0x0) @@ -648,10 +650,10 @@ def build_sinf( iv_size: int = 8, constant_iv: Optional[bytes] = None, ) -> bytes: - """Build a sinf protection box (frma + schm cenc + schi/tenc) for CENC. + """Assemble a sinf protection box (frma + schm CENC + schi/tenc) for CENC. - iv_size is the tenc default_Per_Sample_IV_Size (8 or 16). When constant_iv - is given, the per-sample IV size is 0 and the constant IV is appended per + iv_size is the tenc default_Per_Sample_IV_Size (8 or 16). With a constant_iv, + the per-sample IV size is 0 and this function appends the constant IV per ISO/IEC 23001-7 (cbcs-style constant-IV form). """ frma = box(b"frma", original_format) @@ -689,16 +691,17 @@ def build_init_segment( constant_iv: Optional[bytes] = None, ) -> bytes: """ - Build a complete ftyp + moov initialization segment. + Assemble a complete ftyp + moov initialization segment. stream_type: "video" | "audio" | "text". fourcc: Smooth FourCC ("H264"/"AVC1"/"DAVC", "HVC1"/"HEV1"/"HEVC"/"H265", "DVHE"/"DVH1", "AACL"/"AACH"/"AAC", "EC-3", "TTML"/"STPP"/"DFXP"). - Only "HEV1" yields an hev1 sample entry; the rest of that group yield - hvc1, which requires its parameter sets in the sample entry. + Only "HEV1" yields an hev1 sample entry. The rest of that group yield + hvc1, which needs its parameter sets in the sample entry. codec_private_data: hex string from the manifest QualityLevel. nal_length_size: manifest NALUnitLengthField (bytes per NAL length prefix). - kid: 16-byte default key id; when set, the sample entry is wrapped for CENC. + kid: 16-byte default key id. When set, this function wraps the sample entry + for CENC. iv_size / constant_iv: tenc IV form (see build_sinf). """ if stream_type not in ("video", "audio", "text"): diff --git a/unshackle/core/providers/__init__.py b/unshackle/core/providers/__init__.py index 84762e8b..7a98c64d 100644 --- a/unshackle/core/providers/__init__.py +++ b/unshackle/core/providers/__init__.py @@ -28,11 +28,11 @@ DEFAULT_ORDER: tuple[str, ...] = ("imdb", "omdb", "simkl", "tmdb", "tvdb", "anil def provider_order(kind: Optional[str] = None, anime: bool = False) -> list[type[MetadataProvider]]: - """Provider classes in the order configured by `metadata_providers`. + """Metadata provider classes in the order configured by `metadata_providers`. `metadata_providers` is either a flat list applying to both kinds, or a mapping of kind ("tv"/"movie") to its own list. A kind the mapping omits uses `DEFAULT_ORDER`. - An `anime` title puts anilist first; the rest of the order stays behind it to + An `anime` title puts AniList first. The rest of the order stays behind it to fall back on. """ from unshackle.core.config import config @@ -50,12 +50,12 @@ def provider_order(kind: Optional[str] = None, anime: bool = False) -> list[type def get_available_providers() -> list[MetadataProvider]: - """Return instantiated providers that have valid credentials.""" + """Return instantiated metadata providers that have valid credentials.""" return [cls() for cls in provider_order() if cls().is_available()] def get_provider(name: str) -> Optional[MetadataProvider]: - """Get a specific provider by name.""" + """Get one specific metadata provider by name.""" cls = REGISTRY.get(name) if not cls: return None @@ -73,7 +73,7 @@ def search_metadata( cache_account_hash: Optional[str] = None, anime: bool = False, ) -> Optional[MetadataResult]: - """Search all available providers for metadata. Returns best match.""" + """Find metadata in every available metadata provider. Returns the best result.""" from unshackle.core.config import config # the one gate for `disable_metadata`: every automatic lookup reaches a provider through @@ -295,12 +295,12 @@ def resolve_by_ids( cache_account_hash: Optional[str] = None, anime: bool = False, ) -> Optional[MetadataResult]: - """Resolve metadata from user-supplied external IDs, falling back to search only without them. + """Get metadata from user-supplied external IDs, and find the title by name only without them. - A supplied ID is authoritative. It is looked up directly through the providers that - consume its namespace, in `metadata_providers` order for `kind`, and it always survives - into `external_ids` whatever a provider answers. A fuzzy title search only runs when no - ID was supplied at all. + A supplied ID is authoritative. unshackle looks it up directly through the metadata + providers that consume its namespace, in `metadata_providers` order for `kind`, and it + always survives into `external_ids` whatever a metadata provider answers. Without any + supplied ID, unshackle finds the title by a fuzzy name comparison. """ supplied: dict[str, Union[int, str]] = {} if tmdb_id is not None: @@ -366,17 +366,17 @@ ENRICHMENT_AUTHORITY: tuple[str, ...] = ("tmdb", "simkl", "tvdb") def enrichment_providers(kind: Optional[str] = None) -> list[str]: - """Names of configured providers that can resolve an IMDB ID, most trusted first.""" + """Names of the configured metadata providers that can find a title by IMDB ID, most trusted first.""" configured = {cls.NAME for cls in provider_order(kind) if hasattr(cls, "find_by_imdb_id")} return [name for name in ENRICHMENT_AUTHORITY if name in configured] def enrich_ids(result: MetadataResult) -> None: - """Enrich a MetadataResult by cross-referencing IMDB ID with available providers. + """Enrich a MetadataResult by cross-referencing IMDB ID with the available metadata providers. - Queries all available providers, cross-validates tmdb_id as anchor. - If a provider returns a different tmdb_id than the authoritative source, - ALL of that provider's data is dropped (likely resolved to wrong title). + Queries all available metadata providers, cross-validates tmdb_id as anchor. + If a metadata provider returns a different tmdb_id than the authoritative source, + unshackle drops ALL of that metadata provider's data (it likely found the wrong title). """ ids = result.external_ids if not ids.imdb_id: @@ -417,11 +417,11 @@ def validate_enrichments( enrichments: list[tuple[str, ExternalIds]], authority: dict[str, int], ) -> list[tuple[str, ExternalIds]]: - """Drop providers whose tmdb_id conflicts with the authoritative value. + """Drop the metadata providers whose tmdb_id conflicts with the authoritative value. - If providers disagree on tmdb_id, the more authoritative source wins - and ALL data from disagreeing providers is discarded (different tmdb_id - means the provider likely resolved to a different title entirely). + If metadata providers disagree on tmdb_id, the more authoritative source wins. + unshackle discards ALL data from a metadata provider that disagrees, because a + different tmdb_id means that metadata provider likely found a different title. """ from collections import Counter @@ -479,7 +479,7 @@ def external_ids_to_dict(ext: ExternalIds) -> dict: def cached_to_result(cached: dict, provider_name: str, kind: str) -> Optional[MetadataResult]: - """Convert a cached provider dict back to a MetadataResult.""" + """Convert a cached metadata provider dict back to a MetadataResult.""" if provider_name == "tmdb": detail = cached.get("detail", {}) ext_raw = cached.get("external_ids", {}) diff --git a/unshackle/core/providers/_base.py b/unshackle/core/providers/_base.py index 6a89f194..b2cc3554 100644 --- a/unshackle/core/providers/_base.py +++ b/unshackle/core/providers/_base.py @@ -31,7 +31,7 @@ class ExternalIds: @dataclass class MetadataResult: - """Unified metadata result from any provider.""" + """Unified metadata result from any metadata provider.""" title: Optional[str] = None year: Optional[int] = None @@ -71,11 +71,11 @@ class MetadataProvider(metaclass=ABCMeta): @abstractmethod def is_available(self) -> bool: - """Return True if this provider has the credentials/keys it needs.""" + """Return True if this metadata provider has the credentials/keys it needs.""" @abstractmethod def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: - """Search for a title and return metadata, or None on failure/no match.""" + """Find a title and return its metadata, or None on failure or when nothing matches.""" @abstractmethod def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: @@ -95,6 +95,6 @@ def strip_year(s: str) -> str: def fuzzy_match(a: str, b: str, threshold: float = 0.8) -> bool: - """Return True if ``a`` and ``b`` are a close match.""" + """Return True if ``a`` and ``b`` match closely.""" ratio = SequenceMatcher(None, clean(a), clean(b)).ratio() return ratio >= threshold diff --git a/unshackle/core/providers/anilist.py b/unshackle/core/providers/anilist.py index 8b7d3f13..e5f2d9dc 100644 --- a/unshackle/core/providers/anilist.py +++ b/unshackle/core/providers/anilist.py @@ -28,7 +28,7 @@ COUNTRY_TO_LANGUAGE: dict[str, str] = {"JP": "ja", "KR": "ko", "CN": "zh", "TW": def parse_anilist_ref(value: Union[int, str]) -> Optional[tuple[str, int]]: """Split an AniList reference into its namespace and number. - A bare number is an AniList ID; ``mal:12345`` is a MyAnimeList ID, which AniList + A bare number is an AniList ID. ``mal:12345`` is a MyAnimeList ID, which AniList resolves natively. Returns None for anything else. """ text = str(value).strip().lower() diff --git a/unshackle/core/providers/imdb.py b/unshackle/core/providers/imdb.py index 5b42f440..908b7232 100644 --- a/unshackle/core/providers/imdb.py +++ b/unshackle/core/providers/imdb.py @@ -67,7 +67,7 @@ class IMDBProvider(MetadataProvider): return True def graphql(self, operation: str, query: str) -> Optional[dict]: - """Run a persisted query, registering it with one POST the first time it is seen.""" + """Send a persisted query, and add it to the gateway with one POST if the gateway does not know its hash.""" extensions = {"persistedQuery": {"version": 1, "sha256Hash": hashlib.sha256(query.encode()).hexdigest()}} # IMDb's GraphQL gateway decodes "+" literally, so these JSON params must carry no spaces params = { diff --git a/unshackle/core/providers/tmdb.py b/unshackle/core/providers/tmdb.py index 8d33ad4e..40ac6735 100644 --- a/unshackle/core/providers/tmdb.py +++ b/unshackle/core/providers/tmdb.py @@ -128,8 +128,8 @@ class TMDBProvider(MetadataProvider): def find_by_imdb_id(self, imdb_id: str, kind: str) -> Optional[ExternalIds]: """Look up TMDB/TVDB IDs from an IMDB ID using TMDB's /find endpoint. - When the requested kind has no results, the other kind is tried, so the returned - tmdb_kind can differ from the kind asked for. Returns None if the request fails or + When the requested kind has no results, unshackle tries the other kind, so the + returned tmdb_kind can differ from the kind asked for. Returns None if the request fails or nothing matches. """ self.log.debug("Looking up IMDB ID %s on TMDB", imdb_id) diff --git a/unshackle/core/providers/tvdb.py b/unshackle/core/providers/tvdb.py index efa9bb25..9b775d8a 100644 --- a/unshackle/core/providers/tvdb.py +++ b/unshackle/core/providers/tvdb.py @@ -11,7 +11,7 @@ from unshackle.core.providers._base import ExternalIds, MetadataProvider, Metada def primary_language(data: dict) -> Optional[str]: - """Search results spell the key primary_language, the extended record originalLanguage.""" + """Search results spell the field primary_language, the extended record originalLanguage.""" return data.get("primary_language") or data.get("originalLanguage") or None @@ -34,7 +34,7 @@ def parse_int(value: Optional[str]) -> Optional[int]: def ids_from_remote(remote_ids: Optional[list], tvdb_id: Optional[int]) -> ExternalIds: - """Build ExternalIds from a TVDB remote_ids/remoteIds list.""" + """Assemble ExternalIds from a TVDB remote_ids/remoteIds list.""" ext = ExternalIds(tvdb_id=tvdb_id) for entry in remote_ids or []: source = (entry.get("sourceName") or "").lower() @@ -49,7 +49,7 @@ def ids_from_remote(remote_ids: Optional[list], tvdb_id: Optional[int]) -> Exter def pick_match( results: list[dict], search_title: str, year: Optional[int] ) -> tuple[Optional[dict], Optional[str], float]: - """Best title match among results, ignoring any whose year is more than a year out.""" + """The result whose title matches best, ignoring any whose year is more than a year out.""" best: Optional[dict] = None best_title: Optional[str] = None best_ratio = 0.0 @@ -274,7 +274,7 @@ class TVDBProvider(MetadataProvider): return mapping def find_by_imdb_id(self, imdb_id: str, kind: str) -> Optional[ExternalIds]: - """Resolve a TVDB ID from an IMDb ID via /search/remoteid.""" + """Get a TVDB ID from an IMDb ID through /search/remoteid.""" results = self.api_get(f"/search/remoteid/{imdb_id}") if not results: self.log.debug("No TVDB results for IMDB ID %s", imdb_id) diff --git a/unshackle/core/proxies/basic.py b/unshackle/core/proxies/basic.py index aebe98ad..0392afdd 100644 --- a/unshackle/core/proxies/basic.py +++ b/unshackle/core/proxies/basic.py @@ -10,7 +10,7 @@ from unshackle.core.proxies.proxy import Proxy class Basic(Proxy): def __init__(self, **countries: dict[str, Union[str, list[str]]]): - """Basic Proxy Service using Proxies specified in the config.""" + """Basic proxy provider that uses the proxies given in the configuration file.""" self.countries = {k.lower(): v for k, v in countries.items()} def __repr__(self) -> str: diff --git a/unshackle/core/proxies/expressvpn.py b/unshackle/core/proxies/expressvpn.py index a26d1529..e8bc54ac 100644 --- a/unshackle/core/proxies/expressvpn.py +++ b/unshackle/core/proxies/expressvpn.py @@ -24,11 +24,11 @@ class ExpressVPN(Proxy): ExpressVPN HTTPS proxy provider. Mirrors the ExpressVPN Android TV app: an OAuth 2.0 device authorization grant obtains - API tokens (interactive on first run, silent via cached refresh token afterwards), a + API tokens (interactive on first run, silent through a cached refresh token afterwards), a subscription receipt (SRT) resolves proxy-capable locations, and get_proxy() returns an authenticated HTTPS proxy URL (https://cat:@server:443). - Query format (after the provider prefix, e.g. "expressvpn:ca"): + Query format (after the proxy provider prefix, for example "expressvpn:ca"): ca random location in the country (smart connection) us-ny city us-ny-2 / usny2 pinned server by position @@ -56,13 +56,14 @@ class ExpressVPN(Proxy): enable: bool = False, ): """ - Proxy Service using ExpressVPN app proxy credentials. + Proxy provider that uses ExpressVPN app proxy credentials. - region_map maps a country code to an optional city/server preset (e.g. "us": "ny-02") - applied when the CLI query names only the country; empty values mean smart connection. - server_map maps aliases to location slugs or concrete .expressprovider.com hosts. - Tokens are optional: without them, cached tokens are used, else account_json is read. - Set enable: true to run the one-time interactive device login when no session exists. + region_map maps a country code to an optional city/server preset (for example "us": + "ny-02") applied when the CLI query names only the country. Empty values mean a smart + connection. server_map maps aliases to location slugs or concrete + .expressprovider.com hosts. Tokens are optional: without them, unshackle uses the + cached tokens, or else it reads account_json. Set enable: true to do the one-time + interactive device login when no login session exists. """ if region_map is not None and not isinstance(region_map, dict): raise TypeError(f"Expected region_map to be a dict mapping aliases to locations, not '{region_map!r}'.") @@ -109,7 +110,7 @@ class ExpressVPN(Proxy): return "ExpressVPN HTTPS Proxy" def has_silent_session(self) -> bool: - """Whether tokens can be obtained without the interactive device login.""" + """Whether unshackle can get tokens without the interactive device login.""" return bool( self.tokens or self.access_token @@ -194,7 +195,7 @@ class ExpressVPN(Proxy): return self.match_city(pool, city_query) def match_city(self, locations: list[dict], city_query: str) -> Optional[dict]: - """Match a city query by abbreviation ("ny"), exact slug, slug prefix, then substring.""" + """Find the city that matches the query by abbreviation ("ny"), exact slug, slug prefix, then substring.""" city_query = city_query.strip().lower() abbrev, exact, prefix, substring = [], [], [], [] for loc in locations: @@ -337,7 +338,7 @@ class ExpressVPN(Proxy): return response.json() def device_login(self) -> Optional[dict]: - """One-time interactive bootstrap via the OAuth 2.0 device authorization grant (RFC 8628).""" + """One-time interactive bootstrap through the OAuth 2.0 device authorization grant (RFC 8628).""" response = self.request( "POST", f"{self.AUTH_BASE}/device/code", @@ -515,7 +516,7 @@ def jwt_expired(token: Optional[str]) -> bool: def write_private(path: Path, content: str) -> None: - """Write content to path created with owner-only (0600) permissions.""" + """Write text to path. The new file has owner-only (0600) permissions.""" if hasattr(os, "fchmod"): fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) try: diff --git a/unshackle/core/proxies/gluetun.py b/unshackle/core/proxies/gluetun.py index bea26059..d86cf6ad 100644 --- a/unshackle/core/proxies/gluetun.py +++ b/unshackle/core/proxies/gluetun.py @@ -37,7 +37,7 @@ def cleanup_all_gluetun_containers(): def register_cleanup(): - """Register cleanup handlers (only once).""" + """Add the cleanup handlers to atexit, one time only.""" global cleanup_registered with cleanup_lock: if not cleanup_registered: @@ -49,7 +49,7 @@ def register_cleanup(): class Gluetun(Proxy): """ - Dynamic Gluetun VPN-to-HTTP Proxy Provider with multi-provider support. + Dynamic Gluetun VPN-to-HTTP proxy provider that can use more than one VPN provider. Automatically manages Docker containers running Gluetun for WireGuard/OpenVPN VPN connections. Supports multiple VPN providers in a single configuration using query format: provider:region @@ -170,7 +170,7 @@ class Gluetun(Proxy): **kwargs, ): """ - Initialize Gluetun proxy provider with multi-provider support. + Initialise the Gluetun proxy provider that can use more than one VPN provider. Args: providers: Dict of VPN provider configurations @@ -187,7 +187,7 @@ class Gluetun(Proxy): container_prefix: Docker container name prefix (default: "unshackle-gluetun") auth_user: Optional HTTP proxy authentication username auth_password: Optional HTTP proxy authentication password - verify_ip: Automatically verify IP and region after connection (default: True) + verify_ip: Automatically make sure that the IP and region are correct after connection (default: True) """ if not binaries.Docker: raise RuntimeError( @@ -497,7 +497,7 @@ class Gluetun(Proxy): return port def is_port_in_use(self, port: int) -> bool: - """Check if a port is in use on the system or by any Docker container.""" + """Return True if the port is in use on the system or by any Docker container.""" import socket try: @@ -523,7 +523,7 @@ class Gluetun(Proxy): def build_server_hostname(self, provider_name: str, country_code: str, server_num: str) -> str: """ - Build a server hostname for specific server selection. + Assemble a server hostname for specific server selection. Args: provider_name: VPN provider name (e.g., "nordvpn") @@ -550,9 +550,9 @@ class Gluetun(Proxy): def ensure_image_available(self, image: str = "qmcgaw/gluetun:latest") -> bool: """ - Ensure the Gluetun Docker image is available locally. + Make sure that the Gluetun Docker image is available locally. - If the image is not present, it will be pulled. This prevents + If the image is not present, unshackle runs `docker pull` to get it. This prevents the container creation from timing out during the first run. Args: @@ -609,11 +609,11 @@ class Gluetun(Proxy): hostname: Optional[str] = None, ): """ - Create and start a Gluetun Docker container. + Make and start a Gluetun Docker container. - Credentials go into a temporary env-file with restricted permissions and are passed with --env-file, - never as "-e KEY=VALUE" arguments, so they stay out of process listings. The file is overwritten and - deleted once "docker run" returns. + Credentials go into a temporary env-file with restricted permissions. unshackle gives that file + to Docker with --env-file, never as "-e KEY=VALUE" arguments, so the credentials stay out of + process listings. unshackle overwrites and deletes the file once "docker run" returns. """ start_time = time.time() @@ -831,7 +831,7 @@ class Gluetun(Proxy): pass def is_container_running(self, container_name: str) -> bool: - """Check if a Docker container is running.""" + """Return True if the Docker container operates.""" try: result = subprocess.run( [ @@ -858,13 +858,13 @@ class Gluetun(Proxy): def get_existing_container_info(self, container_name: str) -> Optional[dict]: """ - Check if a container exists in Docker and get its info. + Find whether a container exists in Docker, and get its info. - This handles multiple Unshackle sessions - if another session already - created the container, we'll reuse it instead of trying to create a duplicate. + This lets several copies of unshackle share one container - if another copy + already made the container, unshackle uses it again instead of a duplicate. Args: - container_name: Name of the container to check + container_name: Name of the container to examine Returns: Dict with container info if exists and running, None otherwise @@ -1082,11 +1082,11 @@ class Gluetun(Proxy): def verify_container(self, query_key: str, max_retries: int = 3): """ - Verify container's VPN IP and region using ipinfo.io lookup. + Make sure that the container's VPN IP and region are correct with an ipinfo.io lookup. - Uses the shared get_ip_info function with a session configured to use + Uses the shared get_ip_info function with an HTTP session configured to use the Gluetun proxy. Retries with exponential backoff if the network - isn't ready immediately after the VPN connects. + is not ready immediately after the VPN connects. Args: query_key: The container query key (provider:region) @@ -1234,13 +1234,13 @@ class Gluetun(Proxy): Remove a stopped container with the given name if it exists. This prevents "container name already in use" errors when a previous - container wasn't properly cleaned up. + container was not properly cleaned up. Args: - container_name: Name of the container to check and remove + container_name: Name of the container to examine and remove Returns: - True if a container was removed, False otherwise + True if unshackle removed a container, False otherwise """ try: # Check if container exists (running or stopped) @@ -1312,7 +1312,7 @@ class Gluetun(Proxy): pass def build_proxy_uri(self, port: int) -> str: - """Build HTTP proxy URI.""" + """Assemble the HTTP proxy URI.""" if self.auth_user and self.auth_password: return f"http://{self.auth_user}:{self.auth_password}@localhost:{port}" return f"http://localhost:{port}" diff --git a/unshackle/core/proxies/hola.py b/unshackle/core/proxies/hola.py index 292871cd..74dbda6c 100644 --- a/unshackle/core/proxies/hola.py +++ b/unshackle/core/proxies/hola.py @@ -10,7 +10,7 @@ from unshackle.core.proxies.proxy import Proxy class Hola(Proxy): def __init__(self): """ - Proxy Service using Hola's direct connections via the hola-proxy project. + Proxy provider that uses Hola's direct connections through the hola-proxy project. https://github.com/Snawoot/hola-proxy """ self.binary = binaries.HolaProxy diff --git a/unshackle/core/proxies/nordvpn.py b/unshackle/core/proxies/nordvpn.py index 986d330c..a7acdca3 100644 --- a/unshackle/core/proxies/nordvpn.py +++ b/unshackle/core/proxies/nordvpn.py @@ -11,9 +11,9 @@ from unshackle.core.proxies.proxy import Proxy class NordVPN(Proxy): def __init__(self, username: str, password: str, server_map: Optional[dict[str, int]] = None): """ - Proxy Service using NordVPN Service Credentials. + Proxy provider that uses NordVPN Service Credentials. - A username and password must be provided. These are Service Credentials, not your Login Credentials. + You must give a username and password. These are Service Credentials, not your Login Credentials. The Service Credentials can be found here: https://my.nordaccount.com/dashboard/nordvpn/ """ if not username: @@ -45,7 +45,7 @@ class NordVPN(Proxy): """ Get an HTTP(SSL) proxy URI for a NordVPN server. - HTTP proxies under port 80 were disabled on the 15th of Feb, 2021: + NordVPN disabled HTTP proxies under port 80 on the 15th of Feb, 2021: https://nordvpn.com/blog/removing-http-proxies Supports: @@ -112,7 +112,7 @@ class NordVPN(Proxy): return f"https://{self.username}:{self.password}@{hostname}:89" def get_country(self, by_id: Optional[int] = None, by_code: Optional[str] = None) -> Optional[dict]: - """Search for a Country and it's metadata.""" + """Find a Country and its metadata.""" if all(x is None for x in (by_id, by_code)): raise ValueError("At least one search query must be made.") diff --git a/unshackle/core/proxies/proton.py b/unshackle/core/proxies/proton.py index 834362a0..6522a152 100644 --- a/unshackle/core/proxies/proton.py +++ b/unshackle/core/proxies/proton.py @@ -23,12 +23,12 @@ class ProtonVPN(Proxy): """ Proton VPN HTTPS proxy provider. - Authenticates via TV login (a self-sustaining session cached at /vpn/protonvpn.json, the + Authenticates through TV login (a self-sustaining login session cached at /vpn/protonvpn.json, the only refreshable kind) or an exported account.proton.me cookie session (AUTH-, access only, re-export on expiry). Resolves a server by country/city, mints short-lived proxy credentials, and returns an authenticated HTTPS proxy URL (https://user:pass@server:4443). - Query format (after the provider prefix, e.g. "protonvpn:us"): + Query format (after the proxy provider prefix, for example "protonvpn:us"): us random server in the country (paid preferred over free) us12 Proton server #12 (the number shown in the connection log) us:ny city, NordVPN-style @@ -211,7 +211,7 @@ class ProtonVPN(Proxy): return None def api(self, method: str, path: str) -> Optional[requests.Response]: - """Call the Proton API, refreshing the session once on a 401.""" + """Call the Proton API, and refresh the login session one time on a 401.""" if not self.access_token: self.load_session() if not self.access_token: @@ -305,7 +305,10 @@ class ProtonVPN(Proxy): self.tv_login() def tv_login(self) -> bool: - """Fork a TV session and have the user approve the code at proton.me/tv; tokens refresh like the cookie path.""" + """Fork a TV login session and let the user approve the code at proton.me/tv. + + Tokens refresh like the cookie path. + """ headers = { "x-pm-appversion": self.TV_APP_VERSION, "x-pm-locale": "en", @@ -437,7 +440,7 @@ def server_no(server: dict) -> Optional[int]: def write_private(path: Path, content: str) -> None: - """Write content to path created with owner-only (0600) permissions.""" + """Write text to path. The new file has owner-only (0600) permissions.""" if hasattr(os, "fchmod"): fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) try: diff --git a/unshackle/core/proxies/proxy.py b/unshackle/core/proxies/proxy.py index 10e044a7..b8f9875c 100644 --- a/unshackle/core/proxies/proxy.py +++ b/unshackle/core/proxies/proxy.py @@ -21,10 +21,10 @@ class Proxy: @abstractmethod def get_proxy(self, query: str) -> Optional[str]: """ - Get a Proxy URI from the Proxy Service. + Get a proxy URI from the proxy provider. - Only return None if the query was accepted, but no proxy could be returned. - Otherwise, please use exceptions to denote any errors with the call or query. + Return None only when this method accepts the query but can give no proxy. + Otherwise, use exceptions to denote any errors with the call or query. The returned Proxy URI must be a string supported by Python-Requests: '{scheme}://[{user}:{pass}@]{host}:{port}' diff --git a/unshackle/core/proxies/resolve.py b/unshackle/core/proxies/resolve.py index a5290935..83ea9982 100644 --- a/unshackle/core/proxies/resolve.py +++ b/unshackle/core/proxies/resolve.py @@ -13,7 +13,7 @@ log = logging.getLogger("proxies") def initialize_proxy_providers() -> List[Any]: - """Initialize and return available proxy providers from config.""" + """Initialise and return available proxy providers from config.""" proxy_providers: list = [] try: from unshackle.core import binaries @@ -56,7 +56,7 @@ def initialize_proxy_providers() -> List[Any]: def resolve_proxy(proxy: str, proxy_providers: List[Any]) -> Optional[str]: - """Resolve a proxy parameter to an actual proxy URI. + """Change a proxy parameter to a real proxy URI. Accepts: - Direct URI: "https://...", "socks5://..." diff --git a/unshackle/core/proxies/surfsharkvpn.py b/unshackle/core/proxies/surfsharkvpn.py index 9ce235ca..b7d7eaeb 100644 --- a/unshackle/core/proxies/surfsharkvpn.py +++ b/unshackle/core/proxies/surfsharkvpn.py @@ -11,9 +11,9 @@ from unshackle.core.proxies.proxy import Proxy class SurfsharkVPN(Proxy): def __init__(self, username: str, password: str, server_map: Optional[dict[str, int]] = None): """ - Proxy Service using SurfsharkVPN Service Credentials. + Proxy provider that uses SurfsharkVPN Service Credentials. - A username and password must be provided. These are Service Credentials, not your Login Credentials. + You must give a username and password. These are Service Credentials, not your Login Credentials. The Service Credentials can be found here: https://my.surfshark.com/vpn/manual-setup/main/openvpn """ if not username: @@ -94,7 +94,7 @@ class SurfsharkVPN(Proxy): return f"https://{self.username}:{self.password}@{hostname}:443" def get_country(self, by_id: Optional[int] = None, by_code: Optional[str] = None) -> Optional[dict]: - """Search for a Country and it's metadata.""" + """Find a Country and its metadata.""" if all(x is None for x in (by_id, by_code)): raise ValueError("At least one search query must be made.") diff --git a/unshackle/core/proxies/windscribevpn.py b/unshackle/core/proxies/windscribevpn.py index 78792375..8d129c7a 100644 --- a/unshackle/core/proxies/windscribevpn.py +++ b/unshackle/core/proxies/windscribevpn.py @@ -11,13 +11,13 @@ from unshackle.core.proxies.proxy import Proxy class WindscribeVPN(Proxy): def __init__(self, username: str, password: str, server_map: Optional[dict[str, str]] = None): """ - Proxy Service using WindscribeVPN Service Credentials. + Proxy provider that uses WindscribeVPN Service Credentials. - A username and password must be provided. These are Service Credentials, not your Login Credentials. + You must give a username and password. These are Service Credentials, not your Login Credentials. The Service Credentials can be found here: https://windscribe.com/getconfig/openvpn server_map pins a query to a hostname. Its keys are either a country code ("ca") or a country - code with a city ("ca:toronto"), and a pinned hostname is used before any server lookup. + code with a city ("ca:toronto"), and unshackle uses a pinned hostname before any server lookup. """ if not username: raise ValueError("No Username was provided to the WindscribeVPN Proxy Service.") @@ -130,7 +130,7 @@ class WindscribeVPN(Proxy): A random hostname from matching servers, or None if the country has no servers. Raises: - ValueError: If a city was given and no server in that country matches it. + ValueError: If you give a city and no server in that country matches it. """ hostnames = [] diff --git a/unshackle/core/remote_service.py b/unshackle/core/remote_service.py index bd934bd2..138b786f 100644 --- a/unshackle/core/remote_service.py +++ b/unshackle/core/remote_service.py @@ -41,7 +41,7 @@ DEFAULT_AUTH_HEADERS = ["X-Secret-Key", "X-Api-Key"] def redact_secrets(text: str, data: Optional[Dict[str, Any]] = None) -> str: - """Mask URL userinfo and any request-payload secrets before the text is logged.""" + """Mask URL userinfo and any request-payload secrets before unshackle writes the text to the log.""" secrets = [v for k in SENSITIVE_DATA_KEYS if isinstance(v := (data or {}).get(k), str) and v] return redact_text(text, secrets) or "" @@ -68,7 +68,7 @@ class RemoteClient: return self._session def next_auth_header(self) -> bool: - """Move the api key onto the next candidate header. False once they are exhausted.""" + """Move the api key onto the next candidate header. False when no candidate header remains.""" if not self.api_key or self._auth_header_index + 1 >= len(self.auth_headers): return False session = self.session @@ -220,12 +220,12 @@ def build_tracks(data: Dict[str, Any]) -> Tracks: def resolve_manifest_data(tracks: Tracks, manifests: list) -> None: - """Re-parse serialized manifests and populate track.data for downloading. + """Re-parse serialized manifests and fill track.data for downloading. The server serializes DASH and ISM manifest XML as zlib-compressed base64. We decode and decompress locally, re-parse with the appropriate manifest parser, then match each remote track to the locally-parsed track by ID - to copy track.data. HLS is skipped as it re-fetches from track.url. + to copy track.data. We skip HLS, because it re-fetches from track.url. """ import base64 as b64 @@ -283,7 +283,7 @@ def resolve_manifest_data(tracks: Tracks, manifests: list) -> None: def match_track(remote_track: Track, local_tracks: list) -> Optional[Track]: - """Match a remote track to a locally-parsed track by ID or attributes.""" + """Find the locally-parsed track that matches a remote track by ID or by attributes.""" remote_id = str(remote_track.id) for lt in local_tracks: if str(lt.id) == remote_id: @@ -361,7 +361,7 @@ def resolve_auth_headers(svc: dict, server_name: str) -> list[str]: def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]: - """Resolve server URL, API key, and per-service config from remote_services. + """Find the server URL, the API key, and the per-service config in remote_services. The per-service config carries the ``_auth_headers`` candidate list for RemoteClient. """ @@ -647,8 +647,8 @@ class RemoteService: def get_titles_cached(self, title_id: str = None) -> Titles_T: """Apply the client's local title_map to titles fetched from the remote server. - Lets users rename titles for remote services they don't have installed locally. - The server sends raw titles; the client's own ``services..title_map`` wins. + Lets users rename titles for remote services they do not have installed locally. + The server sends raw titles. The client's own ``services..title_map`` wins. """ title_map = (config.services.get(self.service_tag) or {}).get("title_map") or {} return remap_titles(self.get_titles(), title_map) @@ -679,10 +679,10 @@ class RemoteService: return tracks def resolve_server_keys(self, title: Title_T) -> None: - """Resolve DRM keys via server CDM for all tracks on a title. + """Get the DRM content keys through the server CDM for all tracks on a title. Called by dl.py between track selection and download. The server - decides which CDM device to use and tells the client via + decides which CDM device to use and tells the client through server_cdm_type. We send track IDs and the server does the full CDM flow, returning KID:KEY pairs. """ @@ -734,10 +734,11 @@ class RemoteService: @staticmethod def create_drm_stub(drm_type: str, kid_hexes: list[str]) -> Any: - """Create a DRM object stub matching the type the server actually used. + """Make a DRM object stub that matches the type the server used. - For server_cdm mode, this is only used for display; keys are already - resolved. We build a minimal DRM object that holds content_keys. + For server_cdm mode, unshackle uses this stub for display only, because it + already has the content keys. We assemble a minimal DRM object that holds + content_keys. """ from uuid import UUID diff --git a/unshackle/core/search_result.py b/unshackle/core/search_result.py index c9029955..6320e553 100644 --- a/unshackle/core/search_result.py +++ b/unshackle/core/search_result.py @@ -11,16 +11,16 @@ class SearchResult: url: Optional[str] = None, ): """ - A Search Result for any support Title Type. + A search result for any title type a service can give. Parameters: id_: The search result's Title ID. - title: The primary display text, e.g., the Title's Name. - description: The secondary display text, e.g., the Title's Description or + title: The primary display text, for example the Title's Name. + description: The secondary display text, for example the Title's Description or further title information. - label: The tertiary display text. This will typically be used to display - an informative label or tag to the result. E.g., "unavailable", the - title's price tag, region, etc. + label: The tertiary display text. The `search` command shows it after the + title as an informative label or tag, for example "unavailable", the + title's price tag, or the region. url: A hyperlink to the search result or title's page. """ if not isinstance(id_, (str, int)): diff --git a/unshackle/core/service.py b/unshackle/core/service.py index b0905d3d..86b9665b 100644 --- a/unshackle/core/service.py +++ b/unshackle/core/service.py @@ -48,13 +48,13 @@ DEFAULT_TIMEOUT = (CONNECT_TIMEOUT, READ_TIMEOUT) class TimeoutSession(requests.Session): """requests.Session applying DEFAULT_TIMEOUT when the caller passes none. - requests has no native default timeout; without one a stalled connect or - read hangs forever. RnetSession bounds every request via its client's + requests has no native default timeout. Without one, a stalled connect or + read hangs forever. RnetSession bounds every request through its client's connect_timeout/read_timeout, so this mirrors that on the requests path. - A per-request non-None ``timeout=`` wins; an explicit ``timeout=None`` is - still replaced with the default by :class:`TimeoutHTTPAdapter` on the - mounted adapters, so there is no unbounded read, matching RnetSession where - the client-level timeouts always apply. Pass a large timeout instead. + A per-request non-None ``timeout=`` wins. :class:`TimeoutHTTPAdapter` on the + mounted adapters still replaces an explicit ``timeout=None`` with the default, + so there is no unbounded read, as with RnetSession where the client-level + timeouts always apply. Pass a large timeout instead. """ def request(self, *args: Any, **kwargs: Any) -> requests.Response: @@ -66,8 +66,8 @@ class TimeoutHTTPAdapter(HTTPAdapter): """HTTPAdapter applying DEFAULT_TIMEOUT when the caller passes none. Backstops :class:`TimeoutSession` for the ``session.send(prepared)`` path, - which bypasses ``Session.request``; RnetSession bounds those too via its - client, so this keeps parity. A per-request non-None ``timeout=`` wins; + which bypasses ``Session.request``. RnetSession bounds those too through its + client, so this keeps parity. A per-request non-None ``timeout=`` wins. ``None`` (unset, or explicitly passed) gets the default, because the adapter cannot distinguish the two, and rnet has no unbounded mode either. """ @@ -100,7 +100,7 @@ class TrackRequest: def sanitize_proxy_for_log(uri: Optional[str]) -> Optional[str]: """ - Sanitize a proxy URI for logs by redacting any embedded userinfo (username/password). + Sanitise a proxy URI for logs by redacting any embedded userinfo (username/password). Examples: - http://user:pass@host:8080 -> http://REDACTED@host:8080 @@ -140,10 +140,10 @@ def sanitize_proxy_for_log(uri: Optional[str]) -> Optional[str]: class Service(metaclass=ABCMeta): """The Service Base Class. - A Service must implement the abstract methods; the rest are optional overrides that fall back to the base - implementation when a Service does not define them. The main flow runs the session and authentication - methods first, then titles, then tracks and chapters. The license callbacks run later still, during track - download. + A Service must define the abstract methods. The rest are optional overrides that fall back to the base + implementation when a Service does not define them. The main flow operates the HTTP session and + authentication methods first, then titles, then tracks and chapters. The license callbacks operate later + still, during track download. """ ALIASES: tuple[str, ...] = () # list of aliases for the service; alternatives to the service tag. @@ -299,11 +299,11 @@ class Service(metaclass=ABCMeta): The fetch_fn signature should be: (title, codec, range_) -> Tracks - For HYBRID range, fetch_fn is called with HDR10 and DV separately and - the DV video tracks are merged into the HDR10 result. + For HYBRID range, this helper calls fetch_fn with HDR10 and DV separately, + then merges the DV video tracks into the HDR10 result. Args: - title: The title being processed. + title: The title to process. fetch_fn: A callable that fetches tracks for a specific codec/range. """ all_tracks = Tracks() @@ -365,9 +365,9 @@ class Service(metaclass=ABCMeta): @staticmethod def get_session() -> requests.Session: """ - Creates a Python-requests Session, adds common headers + Creates a Python-requests HTTP session, adds common headers from config, cookies, retry handler, and a proxy if available. - :returns: Prepared Python-requests Session + :returns: Prepared Python-requests HTTP session """ session = TimeoutSession() session.headers.update(config.headers) @@ -401,14 +401,14 @@ class Service(metaclass=ABCMeta): Authenticate the Service with Cookies and/or Credentials (Email/Username and Password). This is effectively a login() function. Any API calls or object initializations - needing to be made, should be made here. This will be run before any of the - following abstract functions. + needing to be made, should be made here. unshackle operates this method before + any of the following abstract functions. You should avoid storing or using the Credential outside this function. Make any calls you need for any Cookies, Tokens, or such, then use those. - The Cookie jar should also not be stored outside this function. However, you may load - the Cookie jar into the service session. + Do not store the cookies outside this function either. However, you can load + the cookies into the service HTTP session. """ if cookies is not None: if not isinstance(cookies, CookieJar): @@ -421,7 +421,7 @@ class Service(metaclass=ABCMeta): def request_input(self, prompt: str) -> str: """Request interactive input from the user. - When running locally (CLI), prompts via the shared rich console so the + When running locally (CLI), prompts through the shared rich console so the prompt renders correctly alongside Live progress / log handlers. When running in serve mode with an :class:`InputBridge` attached, delegates to the bridge which relays the prompt to the remote client. @@ -432,12 +432,12 @@ class Service(metaclass=ABCMeta): def search(self) -> Generator[SearchResult, None, None]: """ - Search by query for titles from the Service. + Find titles from the Service by query. - The query must be taken as a CLI argument by the Service class. - Ideally just re-use the title ID argument (i.e. self.title). + The Service class must take the query as a CLI argument. + Ideally re-use the title ID argument (that is, self.title). - Search results will be displayed in the order yielded. + unshackle displays the search results in the order yielded. """ raise NotImplementedError(f"Search functionality has not been implemented by {self.__class__.__name__}") @@ -449,10 +449,10 @@ class Service(metaclass=ABCMeta): :param challenge: The service challenge, providing this to a License endpoint should return the privacy certificate that the service uses. - :param title: The current `Title` from get_titles that is being executed. This is provided in - case it has data needed to be used, e.g. for a HTTP request. + :param title: The current `Title` from get_titles that unshackle processes now. unshackle + gives this in case it holds data you need, for example for an HTTP request. :param track: The current `Track` needing decryption. Provided for same reason as `title`. - :return: The Service Privacy Certificate as Bytes or a Base64 string. Don't Base64 Encode or + :return: The Service Privacy Certificate as Bytes or a Base64 string. Do not Base64 Encode or Decode the data, return as is to reduce unnecessary computations. """ @@ -460,18 +460,18 @@ class Service(metaclass=ABCMeta): """ Get a Widevine License message by sending a License Request (challenge). - This License message contains the encrypted Content Decryption Keys and will be + This License message contains the encrypted content keys and will be read by the Cdm and decrypted. This is a very important request to get correct. A bad, unexpected, or missing - value in the request can cause your key to be detected and promptly banned, - revoked, disabled, or downgraded. + value in the request can cause the service to detect your CDM device. The device + can then be banned, revoked, disabled, or downgraded. :param challenge: The license challenge from the Widevine CDM. - :param title: The current `Title` from get_titles that is being executed. This is provided in - case it has data needed to be used, e.g. for a HTTP request. + :param title: The current `Title` from get_titles that unshackle processes now. unshackle + gives this in case it holds data you need, for example for an HTTP request. :param track: The current `Track` needing decryption. Provided for same reason as `title`. - :return: The License response as Bytes or a Base64 string. Don't Base64 Encode or + :return: The License response as Bytes or a Base64 string. Do not Base64 Encode or Decode the data, return as is to reduce unnecessary computations. """ @@ -481,18 +481,18 @@ class Service(metaclass=ABCMeta): """ Get a PlayReady License message by sending a License Request (challenge). - This License message contains the encrypted Content Decryption Keys and will be + This License message contains the encrypted content keys and will be read by the CDM and decrypted. This is a very important request to get correct. A bad, unexpected, or missing - value in the request can cause your key to be detected and promptly banned, - revoked, disabled, or downgraded. + value in the request can cause the service to detect your CDM device. The device + can then be banned, revoked, disabled, or downgraded. :param challenge: The license challenge from the PlayReady CDM. - :param title: The current `Title` from get_titles that is being executed. This is provided in - case it has data needed to be used, e.g. for a HTTP request. + :param title: The current `Title` from get_titles that unshackle processes now. unshackle + gives this in case it holds data you need, for example for an HTTP request. :param track: The current `Track` needing decryption. Provided for same reason as `title`. - :return: The License response as Bytes or a Base64 string. Don't Base64 Encode or + :return: The License response as Bytes or a Base64 string. Do not Base64 Encode or Decode the data, return as is to reduce unnecessary computations. """ # Delegates license handling to the Widevine license method by default if a service-specific PlayReady implementation is not provided. @@ -504,13 +504,13 @@ class Service(metaclass=ABCMeta): """ Get a W3C ClearKey License (JWK Set) by sending a License Request (challenge). - Used for DASH `org.w3.clearkey` content. No CDM is involved: the challenge is + Used for DASH `org.w3.clearkey` tracks. unshackle uses no CDM here: the challenge is the W3C EME JSON license request, e.g. ``{"kids": [""], "type": "temporary"}``, and the license is a JWK Set, e.g. ``{"keys": [{"kty": "oct", "k": "...", "kid": "..."}]}``. :param challenge: The JSON license request bytes to POST to the license server. - :param title: The current `Title` from get_titles that is being executed. This is provided in - case it has data needed to be used, e.g. for a HTTP request. + :param title: The current `Title` from get_titles that unshackle processes now. unshackle + gives this in case it holds data you need, for example for an HTTP request. :param track: The current `Track` needing decryption. Provided for same reason as `title`. :return: The JWK Set license as a dict, JSON str, or raw bytes. Return None (the default) to let the framework POST the challenge to the manifest-provided Laurl, if any. @@ -527,11 +527,11 @@ class Service(metaclass=ABCMeta): Return a Movies, Series, or Album objects containing Movie, Episode, or Song title objects respectively. The returned data must be for the given title ID, or a spawn of the title ID. - At least one object is expected to be returned, or it will presume an invalid Title ID was - provided. + You must return at least one object. If you do not, unshackle presumes an invalid + Title ID. You can use the `data` dictionary class instance attribute of each Title to store data you may need later on. - This can be useful to store information on each title that will be required like any sub-asset IDs, or such. + This can be useful to store information on each title that you need later, like any sub-asset IDs, or such. """ def get_titles_cached(self, title_id: str = None) -> Titles_T: @@ -596,26 +596,26 @@ class Service(metaclass=ABCMeta): Return a Tracks object, which itself can contain Video, Audio, Subtitle or even Chapters. Tracks.videos, Tracks.audio, Tracks.subtitles, and Track.chapters should be a List of Track objects. - Each Track in the Tracks should represent a Video/Audio Stream/Representation/Adaptation or - a Subtitle file. + Each Track in the Tracks should represent a Video/Audio track, Representation, or + Adaptation, or a Subtitle file. - While one Track should only hold information for one stream/downloadable, try to get as many - unique Track objects per stream type so Stream selection by the root code can give you more - options in terms of Resolution, Bitrate, Codecs, Language, e.t.c. + While one Track should only hold information for one downloadable track, try to get as many + unique Track objects per track type so track selection by the root code can give you more + options in terms of Resolution, Bitrate, Codecs, Language, and such. No decision making or filtering of which Tracks get returned should happen here. It can be considered an error to filter for e.g. resolution, codec, and such. All filtering based on arguments will be done by the root code automatically when needed. - Make sure you correctly mark which Tracks are encrypted or not, and by which DRM System - via its `drm` property. + Make sure you correctly mark which Tracks have encryption, and which DRM System they + use, with the `drm` property. - If you are able to obtain the Track's KID (Key ID) as a 32 char (16 bit) HEX string, provide - it to the Track's `kid` variable as it will speed up the decryption process later on. It may - or may not be needed, that depends on the service. Generally if you can provide it, without - downloading any of the Track's stream data, then do. + If you can get the Track's KID (Key ID) as a 32 char (16 bit) HEX string, give it to the + Track's `kid` variable, as it will speed up the decryption process later on. The service + decides whether it is necessary. Generally if you can give it, without downloading any of + the Track's media data, then do. - :param title: The current `Title` from get_titles that is being executed. + :param title: The current `Title` from get_titles that unshackle processes now. :return: Tracks object containing Video, Audio, Subtitles, and Chapters, if available. """ @@ -625,12 +625,12 @@ class Service(metaclass=ABCMeta): Get Chapters for the Title. Parameters: - title: The current Title from `get_titles` that is being processed. + title: The current Title from `get_titles` that unshackle processes now. You must return a Chapters object containing 0 or more Chapter objects. You do not need to set a Chapter number or sort/order the chapters in any way as - the Chapters class automatically handles all of that for you. If there's no + the Chapters class automatically handles all of that for you. If there is no descriptive name for a Chapter then do not set a name at all. You must not set Chapter names to "Chapter {n}" or such. If you (or the user) @@ -646,7 +646,7 @@ class Service(metaclass=ABCMeta): Parameters: track: The Track object that had a Segment downloaded. - segment: The Path to the Segment that was downloaded. + segment: The Path to the downloaded Segment. """ def on_track_downloaded(self, track: AnyTrack) -> None: @@ -654,7 +654,7 @@ class Service(metaclass=ABCMeta): Called when a Track has finished downloading. Parameters: - track: The Track object that was downloaded. + track: The downloaded Track object. """ def on_track_decrypted(self, track: AnyTrack, drm: DRM_T, segment: Optional[m3u8.Segment] = None) -> None: @@ -662,9 +662,9 @@ class Service(metaclass=ABCMeta): Called when a Track has finished decrypting. Parameters: - track: The Track object that was decrypted. + track: The decrypted Track object. drm: The DRM object it decrypted with. - segment: The HLS segment information that was decrypted. + segment: The decrypted HLS segment information. """ def on_track_repacked(self, track: AnyTrack) -> None: @@ -672,19 +672,19 @@ class Service(metaclass=ABCMeta): Called when a Track has finished repacking. Parameters: - track: The Track object that was repacked. + track: The repacked Track object. """ def on_track_multiplex(self, track: AnyTrack) -> None: """ - Called when a Track is about to be Multiplexed into a Container. + Called immediately before unshackle multiplexes a Track into a Container. - Note: Right now only MKV containers are multiplexed but in the future - this may also be called when multiplexing to other containers like - MP4 via ffmpeg/mp4box. + Note: Right now unshackle multiplexes only MKV containers. In the future + unshackle can also call this when it multiplexes to other containers like + MP4 with FFmpeg/mp4box. Parameters: - track: The Track object that was repacked. + track: The repacked Track object. """ diff --git a/unshackle/core/service_repo.py b/unshackle/core/service_repo.py index 79be54b4..7f71136f 100644 --- a/unshackle/core/service_repo.py +++ b/unshackle/core/service_repo.py @@ -14,7 +14,7 @@ log = logging.getLogger("services") class DirtyServiceRepo(Exception): - """A clone has uncommitted or unpushed local changes; refresh is refused to protect them.""" + """A clone has uncommitted or unpushed local changes. unshackle refuses to refresh it, to protect them.""" def __init__(self, path: Path): self.path = path @@ -45,7 +45,7 @@ def is_repo_spec(value: object) -> bool: def parse_spec(spec: str) -> tuple[str, Optional[str], str]: - """Return (clone_url, branch, dest_name). `@branch` is honored for http/shorthand, not ssh.""" + """Return (clone_url, branch, dest_name). This function honours `@branch` for http/shorthand, not for ssh.""" is_ssh = spec.startswith(("git@", "ssh://")) branch: Optional[str] = None url = spec @@ -58,7 +58,7 @@ def parse_spec(spec: str) -> tuple[str, Optional[str], str]: def slug(url: str) -> str: """Filesystem-safe clone dir name from a repo URL, host included so same owner/repo on - different hosts don't collide, e.g. github.com__owner__repo.""" + different hosts do not collide, e.g. github.com__owner__repo.""" cleaned = re.sub(r"^[a-z]+://", "", url).split("@")[-1].replace(":", "/") cleaned = re.sub(r"\.git$", "", cleaned) return "__".join(p for p in cleaned.split("/") if p) @@ -78,7 +78,7 @@ def repos_base() -> Path: def resolve_service_repo(spec: str, *, ttl: int = DEFAULT_TTL, force: bool = False) -> Optional[Path]: - """Resolve a repo spec to its local clone dir. + """Find the local clone dir of a repo spec. First use clones it. ``force=True`` (manual ``refresh-services``) hard-resets the clone to upstream, discarding local changes. Otherwise a TTL-stale clone is fast-forwarded, but a dirty @@ -136,7 +136,8 @@ def refresh_repo(spec: str) -> tuple[Optional[Path], list[str]]: """Force-sync a repo to upstream and return (clone dir, git-style per-service change lines). Used by the manual ``refresh-services`` command. Change lines look like ``+ TAG (added)`` / - ``~ TAG (modified)`` / ``- TAG (removed)`` plus a shortstat; empty list means already up to date. + ``~ TAG (modified)`` / ``- TAG (removed)`` plus a shortstat. An empty list means the clone is + already up to date. """ url, branch, dest_name = parse_spec(spec) dest = repos_base() / dest_name @@ -208,9 +209,9 @@ def force_sync(dest: Path, stamp: Path) -> None: def is_dirty(dest: Path) -> bool: """True if the clone has uncommitted edits to tracked files or local commits not on its upstream. - Untracked files are intentionally ignored: a `git pull --ff-only` only touches tracked files, so - untracked content (including `__pycache__` that importing the services creates) is never at risk - and must not falsely flag the clone as dirty. + This function intentionally ignores untracked files: a `git pull --ff-only` only touches tracked + files, so untracked files (including `__pycache__` that importing the services creates) are never + at risk and must not falsely flag the clone as dirty. """ git = str(binaries.Git) # modifications to tracked files only (--untracked-files=no skips __pycache__ etc.) diff --git a/unshackle/core/services.py b/unshackle/core/services.py index c318322b..2f1e9c3d 100644 --- a/unshackle/core/services.py +++ b/unshackle/core/services.py @@ -69,10 +69,10 @@ def load_service(path: Path) -> object: def load_services(paths: list[Path]) -> tuple[dict[str, object], list[str]]: """Load every Service, returning the good ones plus a list of load errors. - Importing this module must never raise: it is imported by several commands, - and a failed import is not cached by Python, so raising here would re-run and - re-report for every command. Instead we collect failures and let the caller - surface them once, cleanly, at the point services are actually used. + Importing this module must never raise: several commands import it, and Python + does not cache a failed import, so raising here would re-run and re-report for + every command. Instead we collect failures and let the caller surface them once, + cleanly, when it uses the services. """ modules: dict[str, object] = {} errors: list[str] = [] @@ -95,11 +95,12 @@ SUMMARY_LOGGED = False def check_load_errors() -> None: """Log a one-line load summary (once) and raise a single clean error on failures. - Called when services are actually needed (listing/resolving) so the message - is rendered once by Click, without a traceback and without cascading through - every command that imports this module. Duplicate services (a tag also served by an - earlier, higher-priority source, which is ignored) are summarised here; the full - list - naming the path used and the duplicate ignored - is debug-only. + unshackle calls this when it needs the services (listing/resolving), so Click + renders the message once, without a traceback and without cascading through every + command that imports this module. This function summarises the duplicate services + (a tag that an earlier, higher-priority source also gives, and which unshackle + ignores). The full list - naming the path used and the duplicate ignored - is + debug-only. """ if DIRTY_REPOS: joined = "\n".join(f" - {p}" for p in DIRTY_REPOS) @@ -127,7 +128,7 @@ class Services(click.Group): remote_services_cache: list[dict] | None = None def parse_args(self, ctx: click.Context, args: list[str]) -> list[str]: - """Preprocess --slow to support optional range value before Click parses args.""" + """Preprocess --slow so it can take an optional range value before Click parses args.""" processed = [] i = 0 while i < len(args): @@ -230,7 +231,7 @@ class Services(click.Group): @staticmethod def make_remote_command(tag: str, ctx: click.Context) -> click.Command: - """Create a Click command for a remote service with server-provided options.""" + """Make a Click command for a remote service with server-provided options.""" svc_info = Services.fetch_remote_service_info(tag, ctx) short_help = svc_info.get("url") if svc_info else None help_text = svc_info.get("help") if svc_info else None @@ -268,9 +269,9 @@ class Services(click.Group): @staticmethod def make_import_command(tag: str, ctx: click.Context) -> click.Command: - """Create a synthetic command that yields an ImportService from an export JSON. + """Make a synthetic command that yields an ImportService from an export JSON. - Mirrors how remote services are wired so dl.py's result() runs unchanged. + Mirrors how unshackle wires remote services, so dl.py's result() runs unchanged. """ @click.command(name=tag, short_help="Reconstruct a download from an export JSON.") @@ -317,7 +318,7 @@ class Services(click.Group): """ Get the Service Tag (the name of the service's directory, not one of its aliases) by an Alias. Input value can be of any case-sensitivity. - Original input value is returned if it did not match a service tag. + This method returns the original input value if it does not match a service tag. """ original_value = value value = value.lower() @@ -340,7 +341,7 @@ class Services(click.Group): @staticmethod def get_vault_tag(name: str) -> str: - """Resolve the key-vault namespace tag for a service. + """Find the key-vault namespace tag for a service. Returns the service's VAULT_TAG override when set, otherwise its own tag. Falls back to the resolved tag for non-local services (remote/import). diff --git a/unshackle/core/session.py b/unshackle/core/session.py index bc5ce7cf..1d8ae78b 100644 --- a/unshackle/core/session.py +++ b/unshackle/core/session.py @@ -1,4 +1,4 @@ -"""Session utilities for creating HTTP sessions with TLS fingerprinting via rnet (Rust/BoringSSL).""" +"""Session utilities for creating HTTP sessions with TLS fingerprinting through rnet (Rust/BoringSSL).""" from __future__ import annotations @@ -40,7 +40,7 @@ DEFAULT_IMPERSONATE = rnet.Impersonate.Chrome131 def resolve_impersonate(browser: str) -> rnet.Impersonate: - """Resolve a browser string to an rnet.Impersonate preset. + """Change a browser string to an rnet.Impersonate preset. Accepts exact rnet preset names (e.g. "Chrome131", "OkHttp4_12", "Edge101"). See https://github.com/0x676e67/rnet for the full list of available presets. @@ -200,7 +200,7 @@ class RnetResponse: ) def iter_content(self, chunk_size: Optional[int] = None) -> Iterator[bytes]: - """Re-chunk rnet's variable-size stream into fixed-size pieces.""" + """Re-chunk rnet's variable-size data into fixed-size pieces.""" self._streamed = True if chunk_size is None or chunk_size <= 0: yield from self._resp.stream() @@ -216,7 +216,7 @@ class RnetResponse: yield bytes(buf) def stream(self) -> Iterator[bytes]: - """Direct pass-through of rnet's native stream iterator.""" + """Direct pass-through of rnet's native ``stream()`` iterator.""" self._streamed = True yield from self._resp.stream() @@ -228,7 +228,7 @@ class RnetResponse: class RnetSessionHeaders(CaseInsensitiveDict): - """Dict-like headers that persist to the rnet client via update().""" + """Dict-like headers that write to the rnet client through update().""" def __init__(self, client: Any) -> None: self._client = client @@ -283,7 +283,7 @@ class RnetCookieAdapter(MutableMapping): pass def flush_to_client(self) -> None: - """Push all buffered cookies to the rnet client once it is created.""" + """Push all buffered cookies to the rnet client after unshackle makes it.""" if self._client is None: return for domain, cookies in self._cookies.items(): @@ -298,7 +298,7 @@ class RnetCookieAdapter(MutableMapping): def jar(self) -> CookieJar: """Return a CookieJar with original Cookie objects (requests compat). - Used by ``save_cookies`` in dl.py to persist cookies back to disk. + ``save_cookies`` in dl.py uses this to write cookies back to disk. """ jar = CookieJar() for cookie in self._original_cookies: @@ -371,8 +371,8 @@ class RnetCookieAdapter(MutableMapping): def get_dict(self, domain: Optional[str] = None, path: Optional[str] = None) -> dict[str, str]: """Return cookies as a plain dict (requests RequestsCookieJar compat). - If *domain* is given, only cookies for that domain are returned. - *path* is accepted for API compatibility but ignored (flat storage). + With a *domain*, this method returns only the cookies for that domain. + It accepts *path* for API compatibility but ignores it (flat storage). """ if domain is not None: return dict(self._cookies.get(domain, {})) @@ -382,8 +382,8 @@ class RnetCookieAdapter(MutableMapping): """Return cookies grouped by domain, domain-less ones under ``""``. rnet scopes each cookie to the host of the URL it was set on, so a copy made with - ``get_dict`` can only ever be replayed against one host. Cookies that arrived as a - plain dict are recorded flat only, and are reported under the empty domain to match + ``get_dict`` can only ever replay against one host. This class records cookies that + arrived as a plain dict flat only, and reports them under the empty domain, to match the localhost fallback in :meth:`flush_to_client`. """ grouped = {domain: dict(cookies) for domain, cookies in self._cookies.items() if cookies} @@ -431,7 +431,7 @@ class RnetProxyDict(dict): """Dict-like proxy config that syncs to the rnet client. Accepts ``{"all": url}``, ``{"https": url}``, or ``{"http": url}`` - and applies via rnet's native ``proxies`` parameter (``List[rnet.Proxy]``). + and applies through rnet's native ``proxies`` parameter (``List[rnet.Proxy]``). Supports both lazy (pre-client) and live (post-client) proxy updates. """ @@ -466,11 +466,11 @@ class RnetSession: Drop-in replacement for CurlSession with requests-compatible API. Supports browser impersonation (Chrome, Firefox, Edge, Safari, OkHttp), - retry with exponential backoff, cookie persistence, and proxy support. + retry with exponential backoff, cookie persistence, and proxies. - The client is created lazily on the first request so that headers, - cookies, and proxies can be configured freely before any connection - is established. + unshackle makes the client lazily on the first request, so you can + configure headers, cookies, and proxies freely before it opens any + connection. """ def __init__( @@ -544,10 +544,11 @@ class RnetSession: @property def impersonate_name(self) -> Optional[str]: - """Preset name (e.g. 'Chrome131') for rebuilding this session in another process. + """Preset name (e.g. 'Chrome131') for rebuilding this HTTP session in another process. - rnet enums stringify as 'Impersonate.Chrome131'; return the trailing name, or None - when no preset was set (then the session is not cheaply rebuildable across processes). + rnet enums stringify as 'Impersonate.Chrome131'. Return the trailing name, or None + when no preset was set (then the HTTP session is not cheaply rebuildable across + processes). """ preset = self._client_kwargs.get("impersonate") if preset is None: @@ -556,7 +557,7 @@ class RnetSession: return name or None def ensure_client(self) -> rnet.BlockingClient: - """Lazily create the rnet client on first use, flushing any buffered state.""" + """Lazily make the rnet client on first use, flushing any buffered state.""" if self._client is None: self._client = rnet.BlockingClient(**self._client_kwargs) self.headers._client = self._client @@ -569,8 +570,8 @@ class RnetSession: """Encode params into the URL (rnet ignores the params kwarg). Accepts the same shapes as requests: a mapping, a sequence of pairs, or a - pre-built query string/bytes. A string is appended verbatim (already encoded); - urlencode() would raise TypeError on it. + pre-built query string/bytes. This method appends a string verbatim (already + encoded). urlencode() would raise TypeError on it. """ if not params: return url @@ -617,10 +618,10 @@ class RnetSession: def request(self, method: str, url: str, **kwargs: Any) -> RnetResponse: """Send a request, retrying on the status forcelist and on the caught exception types. - Only methods in allowed_methods are retried; any other method gets a single attempt. - A max_retries kwarg overrides the session retry budget for this call alone, where 0 means - one attempt with no retries. Once the budget is spent, MaxRetriesError is raised with the - last failure as its cause. + This method retries only the methods in allowed_methods. Any other method gets one try. + A max_retries kwarg overrides the HTTP session retry budget for this call alone, where 0 + means one try with no retries. Once the budget is spent, this method raises + MaxRetriesError with the last failure as its cause. """ client = self.ensure_client() method_upper = method.upper() if isinstance(method, str) else str(method).upper() @@ -755,11 +756,11 @@ class RnetSession: return self.request(method, url, **send_kwargs) def mount(self, prefix: str, adapter: Any) -> None: - """No-op; rnet handles TLS and connection pooling natively.""" + """No-op. rnet does TLS and connection pooling natively.""" pass def close(self) -> None: - """No-op; rnet manages its own resources.""" + """No-op. rnet manages its own resources.""" pass @@ -768,7 +769,7 @@ def session( **kwargs: Any, ) -> RnetSession: """ - Create an rnet session with TLS fingerprinting (browser/app impersonation). + Make an rnet HTTP session with TLS fingerprinting (browser/app impersonation). Args: browser: Exact rnet.Impersonate preset name. Examples: @@ -779,7 +780,7 @@ def session( **kwargs: Additional arguments passed to RnetSession constructor. Returns: - RnetSession configured with browser impersonation and retry behavior. + RnetSession configured with browser impersonation and retry behaviour. Examples: session() # Default browser from config diff --git a/unshackle/core/temp.py b/unshackle/core/temp.py index b70d2573..d6d45a99 100644 --- a/unshackle/core/temp.py +++ b/unshackle/core/temp.py @@ -50,7 +50,7 @@ def sweep_task_dirs(root: Path) -> None: @contextmanager def task_temp_dir(task_id: Optional[str] = None) -> Iterator[Path]: - """Point config.directories.temp at a private dir for this run; remove it on any exit.""" + """Point config.directories.temp at a private dir for this task. Remove it on any exit.""" root = config.directories.temp root.mkdir(parents=True, exist_ok=True) sweep_task_dirs(root) @@ -69,7 +69,7 @@ def task_temp_dir(task_id: Optional[str] = None) -> Iterator[Path]: def with_task_temp(fn: Callable[P, R]) -> Callable[P, R]: - """Run the wrapped callable inside its own task temp dir.""" + """Call the wrapped callable inside its own task temp dir.""" @functools.wraps(fn) def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: diff --git a/unshackle/core/themes.py b/unshackle/core/themes.py index dae47478..d8c38280 100644 --- a/unshackle/core/themes.py +++ b/unshackle/core/themes.py @@ -1,8 +1,8 @@ """ -Named CLI color themes. +Named CLI colour themes. Each palette maps the same role keys to colors: `text`/`text2` for body copy, `pink` for the -brand accent, the six ANSI hues, and the gray ramp used for rules and dim details. The active +brand accent, the six ANSI hues, and the grey ramp used for rules and dim details. The active palette drives both the rich Console theme (banner, logs, progress) and the click help screens, so a single `theme` config value restyles the whole CLI. """ @@ -182,7 +182,8 @@ ALIASES = { def resolve_palette(name: object) -> dict[str, str] | None: """Look up a palette by theme name or alias, case-insensitively. - Takes any YAML scalar since config values are unvalidated; non-strings never match. + Takes any YAML scalar, because unshackle does not validate config values. A non-string + never matches a palette name. Returns a copy: console.py extends the active palette in place, and that must not pollute the master definitions. """ @@ -192,7 +193,7 @@ def resolve_palette(name: object) -> dict[str, str] | None: def build_help_config(palette: dict[str, str]) -> RichHelpConfiguration: - """Build the rich-click help styling for a palette.""" + """Assemble the rich-click help styling for a palette.""" return RichHelpConfiguration( style_option=palette["text"], style_switch=palette["green"], diff --git a/unshackle/core/title_cacher.py b/unshackle/core/title_cacher.py index 33a9c155..de12ffbc 100644 --- a/unshackle/core/title_cacher.py +++ b/unshackle/core/title_cacher.py @@ -15,7 +15,7 @@ class TitleCacher: Handles caching of Title objects to reduce redundant API calls. This wrapper provides: - - Region-aware caching to handle geo-restricted content + - Region-aware caching for geo-restricted titles - Automatic fallback to cached data when API calls fail - Cache lifetime extension during failures - Cache hit/miss statistics for debugging @@ -32,7 +32,7 @@ class TitleCacher: self, title_id: str, region: Optional[str] = None, account_hash: Optional[str] = None ) -> str: """ - Generate a unique cache key for title data. + Make a unique cache key for title data. Args: title_id: The title identifier @@ -68,7 +68,7 @@ class TitleCacher: reset_cache: bool = False, ) -> Optional[Titles_T]: """ - Get titles from cache or fetch from API with fallback support. + Get titles from the cache, or fetch them from the API with a fallback. Args: title_id: The title identifier @@ -171,7 +171,7 @@ class TitleCacher: region: Optional[str] = None, account_hash: Optional[str] = None, ) -> Optional[dict]: - """Get cached metadata for any provider.""" + """Get the cached data of any metadata provider.""" if not config.title_cache_enabled or self.no_cache: return None @@ -223,7 +223,7 @@ class TitleCacher: account_hash: Optional[str] = None, ttl_days: int = 7, ) -> None: - """Cache metadata from any provider.""" + """Cache the data of any metadata provider.""" if not config.title_cache_enabled or self.no_cache: return @@ -297,7 +297,7 @@ def get_region_from_proxy(proxy_url: Optional[str]) -> Optional[str]: def get_account_hash(credential) -> Optional[str]: """ - Generate a hash for account identification. + Make a hash for account identification. Args: credential: Credential object diff --git a/unshackle/core/tracks/attachment.py b/unshackle/core/tracks/attachment.py index 3ea7839c..4285d761 100644 --- a/unshackle/core/tracks/attachment.py +++ b/unshackle/core/tracks/attachment.py @@ -28,11 +28,11 @@ class Attachment: session: Optional[AnySession] = None, ): """ - Create a new Attachment. + Make a new Attachment. If providing a path, the file must already exist. If providing a URL, download() fetches the file during the download phase. - Either path or url must be provided. + You must give either path or url. If name is not provided it will use the file name (without extension). If mime_type is not provided, it will try to guess it. @@ -43,7 +43,7 @@ class Attachment: name: Name of the attachment. mime_type: MIME type of the attachment. description: Description of the attachment. - session: Optional requests session to use for downloading. + session: Optional requests HTTP session to download the file with. """ if path is None and url is None: raise ValueError("Either path or url must be provided.") @@ -164,14 +164,14 @@ class Attachment: session: Optional[AnySession] = None, ) -> "Attachment": """ - Create an attachment from a URL. + Make an attachment from a URL. Args: url: URL to download the attachment from. name: Name of the attachment. mime_type: MIME type of the attachment. description: Description of the attachment. - session: Optional requests session to use for downloading. + session: Optional requests HTTP session to download the file with. Returns: Attachment: A new attachment instance. diff --git a/unshackle/core/tracks/audio.py b/unshackle/core/tracks/audio.py index ec35c040..3d10d300 100644 --- a/unshackle/core/tracks/audio.py +++ b/unshackle/core/tracks/audio.py @@ -80,18 +80,18 @@ class Audio(Track): **kwargs: Any, ): """ - Create a new Audio track object. + Make a new Audio track object. Parameters: codec: An Audio.Codec enum representing the audio codec. - If not specified, MediaInfo will be used to retrieve the codec - once the track has been downloaded. + If not specified, unshackle uses MediaInfo to get the codec + after it downloads the track. bitrate: A number or float representing the average bandwidth in bits/s. - Float values are rounded up to the nearest integer. + unshackle rounds float values up to the nearest integer. channels: A number, float, or string representing the number of audio channels. Strings may represent numbers or floats. Expanded layouts like 7.1.1 is not supported. All numbers and strings will be cast to float. - joc: The number of Joint-Object-Coding Channels/Objects in the audio stream. + joc: The number of Joint-Object-Coding Channels/Objects in the audio track. descriptive: Mark this audio as being descriptive audio for the blind. Note: If codec, bitrate, channels, or joc is not specified some checks may be @@ -153,7 +153,7 @@ class Audio(Track): @property def atmos(self) -> bool: - """Return True if the audio track contains Dolby Atmos.""" + """Return True if the audio track contains Atmos.""" if self.joc: return True if isinstance(self.extra, dict): @@ -188,12 +188,12 @@ class Audio(Track): @staticmethod def parse_channels(channels: Union[str, int, float]) -> float: """ - Converts a Channel string to a float representing audio channel count and layout. + Converts a Channel string to a float representing the audio channel layout. E.g. "3" -> "3.0", "2.1" -> "2.1", ".1" -> "0.1". This does not validate channel strings as genuine channel counts or valid layouts. It does not convert the value to assume a sub speaker channel layout, e.g. 5.1->6.0. - It also does not support expanded surround sound channel layout strings like 7.1.2. + It also cannot read expanded surround sound channel layout strings like 7.1.2. """ if isinstance(channels, str): # TODO: Support all possible DASH channel configurations (https://datatracker.ietf.org/doc/html/rfc8216) diff --git a/unshackle/core/tracks/chapter.py b/unshackle/core/tracks/chapter.py index b02c8ca6..20e21e86 100644 --- a/unshackle/core/tracks/chapter.py +++ b/unshackle/core/tracks/chapter.py @@ -10,7 +10,7 @@ TIMESTAMP_FORMAT = re.compile(r"^(?P\d{2}):(?P\d{2}):(?P\d class Chapter: def __init__(self, timestamp: Union[str, int, float], name: Optional[str] = None): """ - Create a new Chapter with a Timestamp and optional name. + Make a new Chapter with a Timestamp and optional name. The timestamp may be in the following formats: - "HH:MM:SS" string, e.g., `25:05:23`. @@ -18,8 +18,8 @@ class Chapter: - a timecode integer in milliseconds, e.g., `90323120` is `25:05:23.120`. - a timecode float in seconds, e.g., `90323.12` is `25:05:23.120`. - If you have a timecode integer in seconds, just multiply it by 1000. - If you have a timecode float in milliseconds (no decimal value), just convert + If you have a timecode integer in seconds, multiply it by 1000. + If you have a timecode float in milliseconds (no decimal value), convert it to an integer. """ if timestamp is None: @@ -70,7 +70,7 @@ class Chapter: @property def named(self) -> bool: - """Check if Chapter is named.""" + """True if the Chapter has a name.""" return bool(self.name) diff --git a/unshackle/core/tracks/chapters.py b/unshackle/core/tracks/chapters.py index 9d75ebcc..b58a6fa7 100644 --- a/unshackle/core/tracks/chapters.py +++ b/unshackle/core/tracks/chapters.py @@ -88,7 +88,7 @@ class Chapters(SortedKeyList, ABC): - {j}: A number starting at 1 that increments any time a Chapter has no name. E.g., `"Chapter {j}"`: "Chapter 1", "Intro", "Chapter 2". - These are formatted with f-strings, directives are supported. + unshackle formats these with f-strings, so format directives work. For example, `"Chapter {i:02}"` will result in `"Chapter 01"`. """ chapters = [] diff --git a/unshackle/core/tracks/dv_fixup.py b/unshackle/core/tracks/dv_fixup.py index e10fc4c7..6706c22b 100644 --- a/unshackle/core/tracks/dv_fixup.py +++ b/unshackle/core/tracks/dv_fixup.py @@ -1,14 +1,14 @@ """ DV exposure for HLS composite HEVC streams. -Some services deliver DV Profile 8.1 in a stream whose primary CODECS is plain -hvc1, with DV advertised only via SUPPLEMENTAL-CODECS. The fMP4 carries valid DV RPU NALs +Some services deliver DV Profile 8.1 in a track whose primary CODECS is plain +hvc1, with DV advertised only in SUPPLEMENTAL-CODECS. The fMP4 carries valid DV RPU NALs but the container does not signal DV, so muxing the MP4 directly produces an MKV that -mediainfo and DV-capable TVs see as plain HDR10/HDR10+. +MediaInfo and DV-capable TVs see as plain HDR10/HDR10+. -The RPU is already valid; only the container's DV signaling is lost. Demuxing the -elementary HEVC stream (ffmpeg -c:v copy) exposes the in-stream RPU to mkvmerge, which then -signals DV in the muxed MKV. No dovi_tool extract/inject round-trip is needed. +The RPU is already valid. Only the container's DV signalling is lost. A demux to an +elementary HEVC file (ffmpeg -c:v copy) exposes the RPU inside it to mkvmerge, which then +signals DV in the muxed MKV. unshackle does not need a dovi_tool extract/inject round-trip. """ from __future__ import annotations @@ -30,7 +30,7 @@ if TYPE_CHECKING: class DVFixup: - """Demux a DV-composite HEVC track to its elementary stream so mkvmerge exposes DV.""" + """Demux a DV-composite HEVC track to an elementary HEVC file so mkvmerge exposes DV.""" def __init__(self, video: "Video") -> None: self.log = logging.getLogger("dv-fixup") @@ -84,7 +84,7 @@ class DVFixup: def apply_dv_fixup(video: "Video") -> None: - """Run DV fixup on `video` if flagged as DV-composite. Updates `video.path` in place + """Do the DV fixup on `video` if flagged as DV-composite. Updates `video.path` in place and deletes the original source file so the standard mux cleanup handles the new path.""" if not getattr(video, "dv_compatible_bitstream", False): return diff --git a/unshackle/core/tracks/hybrid.py b/unshackle/core/tracks/hybrid.py index 4197f51a..7ac1d050 100644 --- a/unshackle/core/tracks/hybrid.py +++ b/unshackle/core/tracks/hybrid.py @@ -200,7 +200,7 @@ class Hybrid: ) def level_5(self, input_video): - """Generate Level 5 active area metadata via crop detection on the HDR10 stream. + """Make Level 5 active area metadata through crop detection on the HDR10 track. This resolves mismatches where DV has no black bars but HDR10 does (or vice versa) by telling the display the correct active area. @@ -377,7 +377,7 @@ class Hybrid: MaxCLL must not exceed the mastering-display peak (some HDR10+ sources ship MaxCLL 10000 on a 1000-nit master), and MaxFALL must not exceed - MaxCLL. A value of 0 means "unknown" and is preserved as-is. + MaxCLL. A value of 0 means "unknown", and this method keeps it as-is. """ if max_mdl and max_cll and max_cll > max_mdl: max_cll = max_mdl @@ -525,7 +525,7 @@ class Hybrid: ) def extract_hdr10plus(self, video): - """Extract HDR10+ metadata from the video stream""" + """Extract HDR10+ metadata from the video track""" if os.path.isfile(config.directories.temp / self.hdr10plus_file): return @@ -573,7 +573,7 @@ class Hybrid: ) def probe_hdr_metadata(self): - """Extract mastering display and content light level metadata from the HDR10 stream via ffprobe. + """Extract mastering display and Content Light Level metadata from the HDR10 track with FFprobe. Returns (max_mdl, min_mdl, max_cll, max_fall) in dovi_tool level6 units: - max_mdl: nits (integer) diff --git a/unshackle/core/tracks/resume.py b/unshackle/core/tracks/resume.py index c2778a47..605a690b 100644 --- a/unshackle/core/tracks/resume.py +++ b/unshackle/core/tracks/resume.py @@ -1,12 +1,12 @@ """Cross-run segment-resume sidecars. -A track's completed segment files may be reused by a later run, but only once -the manifest proves the segmentation is unchanged. The proof is a fingerprint +A later download can reuse a track's completed segment files, but only after +the manifest proves that the segmentation did not change. The proof is a fingerprint of the segmentation stored in a JSON sidecar next to the segment directory. The sidecar is deliberately a sibling of the segment directory, never a file inside it: the manifest parsers merge by globbing the directory with no -extension filter, so any file left in there would be concatenated into the +extension filter, so they would concatenate any file left in there into the final output. """ @@ -42,7 +42,7 @@ def fingerprint( """Stable digest of a track's segmentation. Hashes the manifest URL (sans query), the segment count, and each - segment's URL (sans query) with its byte-range string verbatim; byte + segment's URL (sans query) with its byte-range string verbatim. Byte ranges are the strongest identity signal available and never carry tokens. extra carries protocol-specific identity inputs (HLS: the media_sequence start and post-filter segment count). @@ -75,7 +75,7 @@ def reusable(save_dir: Path, digest: str) -> bool: def write_sidecar(save_dir: Path, digest: str) -> None: - """Record digest so a later run can prove save_dir's segments are reusable.""" + """Record digest so a later download can prove that save_dir's segments are reusable.""" sidecar_path(save_dir).write_text(json.dumps({"version": _VERSION, "digest": digest}), encoding="utf-8") diff --git a/unshackle/core/tracks/subtitle.py b/unshackle/core/tracks/subtitle.py index d9e619c5..c7df3ae0 100644 --- a/unshackle/core/tracks/subtitle.py +++ b/unshackle/core/tracks/subtitle.py @@ -112,55 +112,56 @@ class Subtitle(Track): **kwargs: Any, ): """ - Create a new Subtitle track object. + Make a new Subtitle track object. Parameters: codec: A Subtitle.Codec enum representing the subtitle format. - If not specified, MediaInfo will be used to retrieve the format - once the track has been downloaded. + If not specified, unshackle uses MediaInfo to get the format + after it downloads the track. cc: Closed Caption. - - Intended as if you couldn't hear the audio at all. - - Can have Sound as well as Dialogue, but doesn't have to. - - Original source would be from an EIA-CC encoded stream. Typically all + - Intended as if you could not hear the audio at all. + - Can have Sound as well as Dialogue, but does not have to. + - Original source would be from EIA-CC encoded video. Typically all upper-case characters. Indicators of it being CC without knowing original source: - Extracted with CCExtractor, or - - >>> (or similar) being used at the start of some or all lines, or + - >>> (or similar) at the start of some or all lines, or - All text is uppercase or at least the majority, or - - Subtitles are Scrolling-text style (one line appears, oldest line - then disappears). - Just because you downloaded it as a SRT or VTT or such, doesn't mean it - isn't from an EIA-CC stream. And I wouldn't take the streaming services + - Subtitles use the scrolling-text style (one line appears, oldest + line then disappears). + The fact that you downloaded it as a SRT or VTT or such does not mean it + is not from an EIA-CC stream. And I would not take the streaming services (CC) as gospel either as they tend to get it wrong too. sdh: Deaf or Hard-of-Hearing. Also known as HOH in the UK (EU?). - - Intended as if you couldn't hear the audio at all. - - MUST have Sound as well as Dialogue to be considered SDH. - - It has no "syntax" or "format" but is not transmitted using archaic - forms like EIA-CC streams, would be intended for transmission via - SubRip (SRT), WebVTT (VTT), TTML, etc. - If you can see important audio/sound transcriptions and not just dialogue - and it doesn't have the indicators of CC, then it's most likely SDH. - If it doesn't have important audio/sounds transcriptions it might just be - regular subtitling (you wouldn't mark as CC or SDH). This would be the + - Intended as if you could not hear the audio at all. + - MUST have Sound as well as Dialogue to count as SDH. + - It has no "syntax" or "format", but it is not transmitted in + archaic forms like EIA-CC streams. It is intended for transmission + through SubRip (SRT), WebVTT (VTT), TTML, and the other modern + subtitle formats. + If you can see important audio/sound transcriptions and not only dialogue + and it does not have the indicators of CC, then it is most likely SDH. + If it does not have important audio/sounds transcriptions it might only be + regular subtitling (you would not mark as CC or SDH). This would be the case for most translation subtitles. Like Anime for example. - forced: Typically used if there's important information at some point in time - like watching Dubbed content and an important Sign or Letter is shown + forced: Typically used if there is important information at some point in time + like watching a Dubbed title and an important Sign or Letter is shown or someone talking in a different language. - Forced tracks are recommended by the Matroska Spec to be played if + The Matroska Spec recommends that the player plays Forced tracks if the player's current playback audio language matches a subtitle marked as "forced". - However, that doesn't mean every player works like this but there is + However, that does not mean every player works like this but there is no other way to reliably work with Forced subtitles where multiple - forced subtitles may be in the output file. Just know what to expect + forced subtitles may be in the output file. Know what to expect with "forced" subtitles. - Note: If codec is not specified some checks may be skipped or assume a value. - Specifying as much information as possible is highly recommended. + Note: If codec is not specified, unshackle can skip some checks or assume a value. + Give as much information as possible. Information on Subtitle Types: https://bit.ly/2Oe4fLC (3PlayMedia Blog on SUB vs CC vs SDH). - However, I wouldn't pay much attention to the claims about SDH needing to - be in the original source language. It's logically not true. + However, I would not pay much attention to the claims about SDH needing to + be in the original language. It is logically not true. CC == Closed Captions. Source: Basically every site. SDH = Subtitles for the Deaf or Hard-of-Hearing. Source: Basically every site. @@ -331,23 +332,23 @@ class Subtitle(Track): Fix invalid timestamps in WebVTT files, particularly negative timestamps. Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - Sanitized WebVTT content + Sanitized WebVTT text """ return re.sub(r"(-\d+:\d+:\d+\.\d+)", "00:00:00.000", text) @staticmethod def has_webvtt_cue_identifiers(text: str) -> bool: """ - Check if WebVTT content has cue identifiers that need removal. + Examine WebVTT text for cue identifiers that need removal. Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - True if cue identifiers are detected, False otherwise + True if the text has cue identifiers, False otherwise """ lines = text.split("\n") @@ -366,16 +367,16 @@ class Subtitle(Track): """ Remove WebVTT cue identifiers that can confuse subtitle parsers. - Some services use cue identifiers like "Q0", "Q1", etc. + Some services use cue identifiers such as "Q0" and "Q1" that appear on their own line before the timing line. These can be - incorrectly parsed as part of the previous cue's text content by + incorrectly parsed as part of the previous cue's text by some parsers (like pysubs2). Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - Sanitized WebVTT content with cue identifiers removed + Sanitized WebVTT text with cue identifiers removed """ if not Subtitle.has_webvtt_cue_identifiers(text): return text @@ -424,16 +425,16 @@ class Subtitle(Track): @staticmethod def has_overlapping_webvtt_cues(text: str) -> bool: """ - Check if WebVTT content has overlapping cues that need merging. + Examine WebVTT text for overlapping cues that need merging. Detects cues with start times within 50ms of each other and the same end time, which indicates multi-line subtitles split into separate cues. Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - True if overlapping cues are detected, False otherwise + True if the text has overlapping cues, False otherwise """ timings = [] for line in text.split("\n"): @@ -461,10 +462,10 @@ class Subtitle(Track): line: position (lower percentage = higher on screen = first line). Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - WebVTT content with overlapping cues merged + WebVTT text with overlapping cues merged """ if not Subtitle.has_overlapping_webvtt_cues(text): return text @@ -568,17 +569,18 @@ class Subtitle(Track): @staticmethod def sanitize_webvtt(text: str) -> str: """ - More thorough sanitization of WebVTT files to handle multiple potential issues. + More thorough sanitization of WebVTT files that corrects multiple potential issues. - This is lossy, so use it only as a fallback once normal parsing has failed. Everything before the - WEBVTT header line is discarded and the header itself is reduced to a bare "WEBVTT" line. Negative - timestamps become 00:00:00.000 and timestamps missing the hours field are padded to HH:MM:SS.mmm. + This is lossy, so use it only as a fallback once normal parsing has failed. This method discards + everything before the WEBVTT header line and reduces the header itself to a bare "WEBVTT" line. + Negative timestamps become 00:00:00.000, and it pads timestamps that have no hours field to + HH:MM:SS.mmm. Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - Sanitized WebVTT content + Sanitized WebVTT text """ # Make sure we have a proper WEBVTT header if not text.strip().startswith("WEBVTT"): @@ -619,15 +621,15 @@ class Subtitle(Track): """ Convert this Subtitle to another format. - Backend selection is data-driven (see ``tracks/subtitle_convert.py``): the best - available backend that supports source->target is used, falling back through the - capability chain on failure. The backend can be pinned via the ``conversion_method`` - config key (``auto`` | ``subby`` | ``pysubs2`` | ``subtitleedit`` | ``pycaption``), - or nudged per-service via ``preferred_conversion_method``; an explicit config value - always wins. + Backend selection is data-driven (see ``tracks/subtitle_convert.py``): unshackle + uses the best available backend that can convert source->target, and falls back + through the capability chain on failure. The ``conversion_method`` config key + (``auto`` | ``subby`` | ``pysubs2`` | ``subtitleedit`` | ``pycaption``) pins the + backend, and ``preferred_conversion_method`` nudges it per-service. An explicit + config value always wins. - ``forced`` marks an explicit user request (``--sub-format``). Lossy downconverts of - styled formats (SSA/ASS -> SRT) are skipped unless ``forced`` is True. + ``forced`` marks an explicit user request (``--sub-format``). unshackle skips lossy + downconverts of styled formats (SSA/ASS -> SRT) unless ``forced`` is True. """ from unshackle.core.tracks.subtitle_convert import run_conversion @@ -650,8 +652,8 @@ class Subtitle(Track): (column located from the section's ``Format:`` line, not assumed by index), and - inline ``\\fn`` font overrides inside ``Dialogue`` override blocks. - Leading ``@`` (vertical-writing prefix) is stripped and names are de-duplicated - case-insensitively, preferring a mixed-case spelling over an all-lowercase one. + This method removes a leading ``@`` (vertical-writing prefix), de-duplicates the + names case-insensitively, and prefers a mixed-case spelling over an all-lowercase one. """ names: set[str] = set() name_index = 1 # ASS default Style order: Name, Fontname, ... @@ -746,10 +748,10 @@ class Subtitle(Track): Remove or fix corrupted WebVTT lines, particularly those with invalid timestamps. Parameters: - text: The WebVTT content as string + text: The WebVTT text as string Returns: - Sanitized WebVTT content with corrupted lines removed + Sanitized WebVTT text with corrupted lines removed """ lines = text.splitlines() sanitized_lines = [] @@ -785,7 +787,7 @@ class Subtitle(Track): Segmented VTT when merged may have the WEBVTT headers part of the next caption as they were not separated far enough from the previous caption and ended up - being considered as caption text rather than the header for the next segment. + as caption text rather than the header for the next segment. """ if isinstance(data, bytes): data = try_ensure_utf8(data).decode("utf8") @@ -845,10 +847,10 @@ class Subtitle(Track): def merge_segmented_wvtt(data: bytes, period_start: float = 0.0) -> tuple[CaptionList, Optional[str]]: """ Convert Segmented DASH WebVTT cues into a pycaption Caption List. - Also returns an ISO 639-2 alpha-3 language code if available. + Also returns an ISO 639-2 alpha-3 language tag if available. Code ported originally by xhlove to Python from shaka-player. - Has since been improved upon by rlaphoenix using pymp4 and + rlaphoenix has since improved it with pymp4 and pycaption functions. """ captions = CaptionList() @@ -975,9 +977,9 @@ class Subtitle(Track): def strip_hearing_impaired(self) -> None: """ - Strip captions for hearing impaired (SDH). + Remove the SDH captions. - The SDH stripping method is determined by the 'sdh_method' setting in config: + The 'sdh_method' setting in config gives the SDH stripping method: - 'auto' (default): Tries subby first, then SubtitleEdit, then filter-subs - 'subby': Uses subby's SDHStripper - 'subtitleedit': Uses SubtitleEdit when available @@ -1086,7 +1088,7 @@ class Subtitle(Track): def reverse_rtl(self) -> None: """ Reverse RTL (Right to Left) Start/End on Captions. - This can be used to fix the positioning of sentence-ending characters. + Use this to fix the positioning of sentence-ending characters. """ if not self.path or not self.path.exists(): raise ValueError("You must download the subtitle track first.") diff --git a/unshackle/core/tracks/subtitle_convert.py b/unshackle/core/tracks/subtitle_convert.py index c02c1055..a8e54963 100644 --- a/unshackle/core/tracks/subtitle_convert.py +++ b/unshackle/core/tracks/subtitle_convert.py @@ -3,12 +3,12 @@ Subtitle conversion backend registry. Routing is data-driven: each backend declares which (source -> target) codec pairs it can read/write, whether it is available in the current environment, and a preference rank. -``resolve_backends`` filters the registry to the available backends that support the -requested pair and orders them by rank; ``run_conversion`` tries each in turn (a real +``resolve_backends`` filters the registry to the available backends that can convert the +requested pair and orders them by rank. ``run_conversion`` tries each in turn (a real fallback chain) until one succeeds. The public entry point stays ``Subtitle.convert`` / ``Subtitle.strip_hearing_impaired`` in -subtitle.py; this module only holds the selection + conversion logic so subtitle.py keeps +subtitle.py. This module only holds the selection + conversion logic so subtitle.py keeps the codec enum, ``parse``, sanitizers and cue helpers (the collaborators backends reuse). """ @@ -65,7 +65,7 @@ PYSUBS2_FORMATS: dict[Codec, str] = { def is_seconv(binary: object) -> bool: - """True for the SubtitleEdit 5+ CLI (seconv); 4.x binaries need legacy /convert syntax.""" + """True for the SubtitleEdit 5+ CLI (seconv). 4.x binaries need legacy /convert syntax.""" name = str(binary).replace("\\", "/").rsplit("/", 1)[-1] return name.lower().rsplit(".", 1)[0] == "seconv" @@ -81,12 +81,12 @@ def subtitleedit_args( reverse_rtl: bool = False, ) -> list[str]: """ - Build a SubtitleEdit batch-convert command in 5.x (seconv ``--flags``) or 4.x + Assemble a SubtitleEdit batch-convert command in 5.x (seconv ``--flags``) or 4.x (``/convert /flags``) syntax, per ``is_seconv``. - Both name the output ``.``; ``output_folder`` steers placement + Both name the output ``.``. ``output_folder`` steers placement (a bare ``--output-filename`` resolves against the cwd, not the input dir). Overwrite is - always set so re-runs and in-place transforms (SDH/RTL) don't fail on an existing file. + always set so re-runs and in-place transforms (SDH/RTL) do not fail on an existing file. """ if not is_seconv(binary): args = [str(binary), "/convert", str(src), fmt, "/encoding:utf8", "/overwrite"] @@ -280,7 +280,7 @@ REGISTRY: list[SubtitleBackend] = [ def resolve_backends(source: Codec, target: Codec, *, pin: Optional[str] = None) -> list[SubtitleBackend]: - """Available backends that support source->target, ordered by rank. A pin is tried first.""" + """Available backends that can convert source->target, ordered by rank. A pinned backend goes first.""" available = [b for b in REGISTRY if b.is_available() and b.can_convert(source, target)] if pin: pinned = [b for b in available if b.name == pin] @@ -306,8 +306,8 @@ def run_conversion(sub: Subtitle, target: Codec, *, pin: Optional[str] = None, f Convert ``sub`` to ``target`` using the best available backend, falling back through the capability chain on failure. - ``forced`` is True only for explicit user requests (``--sub-format``); lossy downconverts - (styled SubStation -> SRT) are skipped unless forced. + ``forced`` is True only for explicit user requests (``--sub-format``). unshackle skips + lossy downconverts (styled SubStation -> SRT) unless ``forced`` is True. """ if sub.path is None or not sub.path.exists(): raise ValueError("You must download the subtitle track first.") diff --git a/unshackle/core/tracks/track.py b/unshackle/core/tracks/track.py index 08b7726a..81d9db49 100644 --- a/unshackle/core/tracks/track.py +++ b/unshackle/core/tracks/track.py @@ -62,8 +62,9 @@ def direct_session(session: Union[Session, "RnetSession"]) -> Session: def read_top_level_box(path: Path, box_type: bytes) -> Optional[bytes]: """Read a single top-level box from an MP4 file. - Only box headers are walked, so the wanted box is the only payload read into memory. Returns None if the - file has no such top-level box, or if a box header is truncated or declares an impossible size. + This function walks only the box headers, so the wanted box is the only payload it reads into memory. + Returns None if the file has no such top-level box, or if a box header is truncated or declares an + impossible size. """ file_size = path.stat().st_size with path.open("rb") as f: @@ -93,10 +94,10 @@ def read_top_level_box(path: Path, box_type: bytes) -> Optional[bytes]: def has_encrypted_sample_entry(path: Path) -> bool: """True if the MP4's moov still carries an encrypted sample entry (encv/enca). - A faithful decrypt rewrites encv/enca back to the real codec 4CC via frma, so a + A faithful decrypt rewrites encv/enca back to the real codec 4CC through frma, so a survivor means the decrypt tool skipped/failed the track. This is a 4CC scan - scoped to the moov box (where sample entries live), not a structural parse; - scoping avoids chance byte collisions in mdat. Any read error -> False. + scoped to the moov box (where sample entries live), not a structural parse. + The scoping avoids chance byte collisions in mdat. Any read error -> False. """ try: moov = read_top_level_box(path, b"moov") @@ -152,19 +153,19 @@ def assert_fragments_decrypted(path: Path) -> None: skips, so a survivor marks a silent skip. One cause is a tfhd.sample_description_index dangling past the stsd entry count, which mp4decrypt reports as success (exit 0, no stderr) while leaving the payload as ciphertext. Unlike has_encrypted_sample_entry - (moov-scoped) this is fragment-scoped; the moov comes out clean in that failure. + (moov-scoped) this is fragment-scoped. The moov comes out clean in that failure. - Only a standard senc counts as evidence. Some DASH content ships a PIFF uuid beside it, - and a decrypter detaches just the atom it consumed, so that uuid outlives a good - decrypt; counting it would condemn a healthy file. + Only a standard senc counts as evidence. Some DASH titles ship a PIFF uuid beside it, + and a decrypter detaches only the atom it consumed, so that uuid outlives a good + decrypt. Counting it would condemn a healthy file. The guarantee runs in one direction: a survivor proves a skip, while a clean pass is only as good as the decrypter's own behaviour. shaka-packager (the default `decryption` backend) remuxes into a single fragment and drops senc whether or not it - decrypted, so this cannot fire on shaka output. The walk is unbuffered and reads only - box headers; it seeks over mdat and never reads it. A malformed box size aborts the + decrypted, so this cannot fire on shaka output. The walk does not use a buffer and reads + only box headers. It seeks over mdat and never reads it. A malformed box size aborts the walk with a warning and returns normally, so a file this function cannot parse is - never escalated into a raise. The logged warning is the only signal of that; a caller + never escalated into a raise. The logged warning is the only signal of that. A caller sees the same silent return it gets from a verified clean file. """ surviving = 0 @@ -239,7 +240,7 @@ class DownloadContext: cdm: Optional[object] = None def ensure_session(self) -> Union[Session, "RnetSession"]: - """Return the session, or a new ``Session`` if none was set.""" + """Return the HTTP session, or a new ``Session`` if none was set.""" session = self.session if not session: session = Session() @@ -369,7 +370,7 @@ class Track: """ Arbitrary track data dictionary. - A defaultdict is used with a dict as the factory for easier + This uses a defaultdict with a dict as the factory for easier nested saving and safer exists-checks. Reserved keys: @@ -387,8 +388,8 @@ class Track: - segment_durations: list[int] - A list of each segment's duration. You should not add, change, or remove any data within reserved keys. - You may use their data but do note that the values of them may change - or be removed at any point. + You may use their data, but note that these values can change or be removed + at any point. """ return self._data @@ -414,8 +415,8 @@ class Track: ): """Download and optionally Decrypt this Track. - A URL-descriptor Video or Audio track with no `drm` set has its DRM probed from the init data of the - stream and stored on the track, so a service need not declare `drm` itself. + For a URL-descriptor Video or Audio track with no `drm` set, unshackle probes the DRM from the + track's init data and stores it on the track, so a service need not declare `drm` itself. """ from unshackle.core.manifests import DASH, HLS, ISM @@ -647,7 +648,7 @@ class Track: def to_dict(self) -> dict[str, Any]: """Serialise the track for export/import (identity/URL/descriptor/language). - DRM is not serialised here; the export writer attaches the licensed DRM + keys. + DRM is not serialised here. The export writer attaches the licensed DRM + keys. Subclasses add their own codec/quality fields. """ data: dict[str, Any] = { @@ -665,7 +666,7 @@ class Track: @staticmethod def base_kwargs_from_dict(data: dict[str, Any]) -> dict[str, Any]: - """Build the shared Track constructor kwargs from a ``to_dict()`` payload. + """Assemble the shared Track constructor kwargs from a ``to_dict()`` payload. DRM is not reconstructed here: ``to_dict`` does not serialise it, and the import flow attaches the licensed DRM + content keys separately. @@ -719,20 +720,20 @@ class Track: """ Probe the DRM encryption Key ID (KID) for this specific track. - It currently supports finding the Key ID by probing the track's stream - with ffprobe for `enc_key_id` data, as well as for mp4 `tenc` (Track - Encryption) boxes. + It can find the Key ID by probing the track with FFprobe for + `enc_key_id` data, as well as for mp4 `tenc` (Track Encryption) + boxes. It explicitly ignores PSSH information like the `PSSH` box, as the box is likely to contain multiple Key IDs that may or may not be for this specific track. To retrieve the initialization segment, this method calls :meth:`get_init_segment` - with the positional and keyword arguments. The return value of `get_init_segment` - is then used to determine the Key ID. + with the positional and keyword arguments. This method then uses the return value + of `get_init_segment` to find the Key ID. Returns: - The Key ID as a UUID object, or None if the Key ID could not be determined. + The Key ID as a UUID object, or None if unshackle cannot find the Key ID. """ if not init_data: init_data = self.get_init_segment(*args, **kwargs) @@ -758,13 +759,13 @@ class Track: def load_drm_if_needed(self, service=None) -> bool: """ - Load DRM information for this track if it was deferred during parsing. + Load DRM information for this track if the parser deferred it. Args: service (Service | None): Service instance that can fetch track-specific DRM info Returns: - True if DRM was loaded or already present, False if failed + True if the DRM loaded or is already present, False if the load failed """ if not getattr(self, "needs_drm_loading", False): return bool(self.drm) @@ -840,13 +841,13 @@ class Track: session: Optional[Session] = None, ) -> bytes: """ - Get the Track's Initial Segment Data Stream. + Get the Track's initial segment data. - HLS and DASH tracks must explicitly provide a URL to the init segment or file. - Providing the byte-range for the init segment is recommended where possible. + HLS and DASH tracks must explicitly give a URL to the init segment or file. + Give the byte-range for the init segment where possible. - If `byte_range` is not set, it will make a HEAD request and check the size of - the file. If the size could not be determined, it will download up to the first + If `byte_range` is not set, it will make a HEAD request and examine the size of + the file. If it cannot find the size, it will download up to the first 20KB only, which should contain the entirety of the init segment. You may override this by changing the `maximum_size`. @@ -854,12 +855,12 @@ class Track: seems to work well across the board. Parameters: - maximum_size: Size to assume as the content length if byte-range is not - used, the content size could not be determined, or the content size - is larger than it. A value of 20000 (20KB) or higher is recommended. + maximum_size: Size to assume as the response body length if byte-range is + not used, if unshackle cannot find the body size, or if the body size + is larger than it. Use a value of 20000 (20KB) or higher. url: Explicit init map or file URL to probe from. byte_range: Range of bytes to download from the explicit or implicit URL. - session: Session context, e.g., authorization and headers. + session: HTTP session context, for example authorization and headers. """ if not isinstance(maximum_size, int): raise TypeError(f"Expected maximum_size to be an {int}, not {type(maximum_size)}") @@ -919,12 +920,12 @@ class Track: def repackage(self, bsf_v: Optional[str] = None) -> bool: """Remux the track with ffmpeg ``-c copy``. - When ``bsf_v`` is given it is folded into the same pass as ``-bsf:v`` (used to - normalize video VUI colour metadata without a second full-file remux). Repackaging - is mandatory; the bitstream filter is best-effort. If the combined run fails (and it - isn't the AAC-retry case) it is retried once without ``bsf_v`` so the remux still - succeeds. Returns True if the requested ``bsf_v`` was applied (always False when - ``bsf_v`` is None, since nothing was requested). + A given ``bsf_v`` goes into the same pass as ``-bsf:v``, which normalises video VUI + colour metadata without a second full-file remux. Repackaging is mandatory. The + bitstream filter is best-effort: if the combined pass fails, and it is not the + AAC-retry case, unshackle tries it again once without ``bsf_v`` so the remux still + succeeds. Returns True if unshackle applied the requested ``bsf_v`` (always False + when ``bsf_v`` is None, because the caller requested nothing). """ if not self.path or not self.path.exists(): raise ValueError("Cannot repackage a Track that has not been downloaded.") diff --git a/unshackle/core/tracks/tracks.py b/unshackle/core/tracks/tracks.py index 8c5c6aab..90c2e289 100644 --- a/unshackle/core/tracks/tracks.py +++ b/unshackle/core/tracks/tracks.py @@ -132,7 +132,7 @@ class Tracks: **kwargs: Any, ) -> None: """ - Ensure terminal status states render as a fully completed bar. + Make sure that terminal status states render as a fully completed bar. Some downloaders can report completed slightly below total before emitting the final "Downloaded" state. @@ -190,7 +190,7 @@ class Tracks: return tree, progress_callables def exists(self, by_id: Optional[str] = None, by_url: Optional[Union[str, list[str]]] = None) -> bool: - """Check if a track already exists by various methods.""" + """Examine whether a track already exists, by various methods.""" if by_id: return any(x.id == by_id for x in self) if by_url: @@ -204,7 +204,7 @@ class Tracks: ], warn_only: bool = False, ) -> None: - """Add a provided track to its appropriate array and ensuring it's not a duplicate.""" + """Add a provided track to its appropriate array and ensuring it is not a duplicate.""" if isinstance(tracks, Tracks): if tracks.manifest_url and not self.manifest_url: self.manifest_url = tracks.manifest_url @@ -296,7 +296,7 @@ class Tracks: ) -> None: """ Sort subtitle tracks by various track attributes to a common P2P standard. - You may optionally provide a sequence of languages to prioritize to the top. + You may optionally give a sequence of languages to prioritise to the top. Section Order: - by_language groups prioritized to top, and ascending alphabetically @@ -310,9 +310,9 @@ class Tracks: (Least to most captions expected in the subtitle) type_priority overrides the Type Order with an explicit ranking of "forced", - "normal", and "sdh" (cc counts as sdh); unlisted types fall to the end. + "normal", and "sdh" (cc counts as sdh). Unlisted types fall to the end. - group_by sets the major key. "type" (default) keeps every forced track together, + group_by sets the major sort order. "type" (default) keeps every forced track together, then every normal, then every SDH, each block ascending by language. "language" groups by language instead, so Finnish sits next to Finnish SDH, with the Type Order applied inside each language. @@ -377,8 +377,9 @@ class Tracks: def merge_video_selections(*groups: list[Video]) -> list[Video]: """Concatenate video selections, dropping duplicates (by track id, order-preserving). - A DV track can be chosen as both the hybrid ingredient (lowest) and an explicit - deliverable; without dedup the same track would be muxed/downloaded twice. + A caller can choose a DV track as both the hybrid ingredient (lowest) and an + explicit deliverable. Without dedup, unshackle would mux and download the same + track twice. """ merged: list[Video] = [] for group in groups: @@ -393,9 +394,10 @@ class Tracks: ) -> tuple[list[Video], list[Video]]: """Split videos into hybrid-ingredient candidates and the standalone-deliverable pool. - HDR10/HDR10+/DV tracks are hybrid ingredients; they only enter the standalone - pool when their range was explicitly requested alongside HYBRID, so e.g. - `-r HYBRID` muxes only the hybrid while `-r HYBRID,HDR10P` also delivers HDR10+. + HDR10/HDR10+/DV tracks are hybrid ingredients. They only enter the standalone + pool when the user explicitly requested their range alongside HYBRID, so for + example `-r HYBRID` muxes only the hybrid while `-r HYBRID,HDR10P` also delivers + HDR10+. """ ingredient_ranges = (Video.Range.HDR10, Video.Range.HDR10P, Video.Range.DV) hybrid_candidates = [v for v in videos if v.range in ingredient_ranges] @@ -406,8 +408,8 @@ class Tracks: def flag_hybrid_ingredients(hybrid_selected: list[Video], non_hybrid_selected: list[Video]) -> None: """Mark tracks selected only as hybrid ingredients so the standalone mux loop skips them. - A track that was also picked as an explicit deliverable (same track in both - selections) stays unflagged and is muxed standalone alongside the hybrid. + A track that the caller also selected as an explicit deliverable (same track in both + selections) stays unflagged, and the standalone mux loop muxes it alongside the hybrid. """ for video in hybrid_selected: if video not in non_hybrid_selected: @@ -492,21 +494,21 @@ class Tracks: Multiplex all the Tracks into a Matroska Container file. A failed mux does not raise. mkvmerge's exit code and error lines come back alongside the - output path, and the caller must check them. + output path, and the caller must examine them. Parameters: title: Set the Matroska Container file title. Usually displayed in players instead of the filename if set. delete: Delete all track files after multiplexing. - progress: Update a rich progress bar via `completed=...`. This must be the - progress object's update() func, pre-set with task id via functools.partial. - audio_expected: Whether audio is expected in the output. Used to determine - if embedded audio metadata should be added. + progress: Update a rich progress bar with `completed=...`. This must be the + progress object's update() func, pre-set with task id by functools.partial. + audio_expected: Whether the output must have audio. unshackle uses this to + decide if it adds embedded audio metadata. title_language: The title's intended language. Used to select the best video track for audio metadata when multiple video tracks exist. skip_subtitles: Skip muxing subtitle tracks into the container. - output_path: Explicit destination for the muxed container. When None (default) the - path is derived from the first track, so callers muxing several track groups that + output_path: Explicit destination for the muxed container. When None (default) + unshackle derives the path from the first track, so callers muxing several track groups that share a video list must pass distinct paths to avoid clobbering each other. """ if self.videos and not self.audio and audio_expected: diff --git a/unshackle/core/tracks/video.py b/unshackle/core/tracks/video.py index 0fbef2bc..4f187a0c 100644 --- a/unshackle/core/tracks/video.py +++ b/unshackle/core/tracks/video.py @@ -103,20 +103,20 @@ class Video(Track): """ Convert CICP (Coding-Independent Code Points) values to Video Range. - CICP is defined in ITU-T H.273 and ISO/IEC 23091-2 for signaling video - color properties independently of the compression codec. These values are + ITU-T H.273 and ISO/IEC 23091-2 define CICP for signalling video + colour properties independently of the compression codec. These values are used across AVC (H.264), HEVC (H.265), VVC, AV1, and other modern codecs. The enum values (Primaries, Transfer, Matrix) match the official specifications: - ITU-T H.273: Coding-independent code points for video signal type identification - - ISO/IEC 23091-2: Information technology — Coding-independent code points — Part 2: Video + - ISO/IEC 23091-2: Information technology - Coding-independent code points - Part 2: Video - H.264 Table E-3 (Colour Primaries) and Table E-4 (Transfer Characteristics) - H.265 Table E.3 and E.4 (identical to H.264) Note: Value 0 = "Reserved" and Value 2 = "Unspecified" per specification. While both effectively mean "unknown" in practice, the distinction matters for - spec compliance. Value 2 was added based on user feedback (GitHub issue) and - verified against FFmpeg's AVColorPrimaries/AVColorTransferCharacteristic enums. + spec compliance. Value 2 was added after user feedback (GitHub issue) and matches + FFmpeg's AVColorPrimaries/AVColorTransferCharacteristic enums. Sources: - https://www.itu.int/rec/T-REC-H.273 @@ -234,25 +234,24 @@ class Video(Track): **kwargs: Any, ) -> None: """ - Create a new Video track object. + Make a new Video track object. Parameters: codec: A Video.Codec enum representing the video codec. - If not specified, MediaInfo will be used to retrieve the codec - once the track has been downloaded. - range_: A Video.Range enum representing the video color range. + If not specified, unshackle uses MediaInfo to get the codec + after it downloads the track. + range_: A Video.Range enum representing the video colour range. Defaults to SDR if not specified. bitrate: A number or float representing the average bandwidth in bytes/s. - Float values are rounded up to the nearest integer. + unshackle rounds float values up to the nearest integer. width: The horizontal resolution of the video. height: The vertical resolution of the video. fps: A number, float, or string representing the frames/s of the video. Strings may represent numbers, floats, or a fraction (num/den). All strings will be cast to either a number or float. - Note: If codec, bitrate, width, height, or fps is not specified some checks - may be skipped or assume a value. Specifying as much information as possible - is highly recommended. + Note: If codec, bitrate, width, height, or fps is not specified, unshackle can + skip some checks or assume a value. Give as much information as possible. """ super().__init__(*args, **kwargs) @@ -362,7 +361,7 @@ class Video(Track): ) def change_color_range(self, range_: int) -> None: - """Change the Video's Color Range to Limited (0) or Full (1).""" + """Change the Video's Colour Range to Limited (0) or Full (1).""" if not self.path or not self.path.exists(): raise ValueError("Cannot change the color range flag on a Video that has not been downloaded.") if not self.codec: @@ -414,7 +413,7 @@ class Video(Track): def vui_bsf(self) -> Optional[str]: """Return the ``-bsf:v`` value that rewrites SPS VUI colour metadata to match ``self.range``. - Returns None when no rewrite is needed: SDR/DV/HYBRID ranges, non-AVC/HEVC codecs, a + Returns None when a rewrite is not necessary: SDR/DV/HYBRID ranges, non-AVC/HEVC codecs, a missing file, or when the bitstream already ships the correct colour metadata. Otherwise returns a ``{h264,hevc}_metadata=...`` string suitable for ffmpeg ``-bsf:v``. """ @@ -477,8 +476,8 @@ class Video(Track): Some services ship HDR10/HLG bitstreams with stale BT.709 VUI, which makes downstream tools mis-classify the file. The manifest-derived range is the - source of truth. Skips SDR, DV, and HYBRID. Returns True if the bitstream - was rewritten. + source of truth. Skips SDR, DV, and HYBRID. Returns True if unshackle rewrote + the bitstream. """ bsf = self.vui_bsf() if bsf is None: @@ -595,19 +594,19 @@ class Video(Track): def extract_c608(self) -> list[Subtitle]: """ - Extract Apple-Style c608 box (CEA-608) subtitle using ccextractor. + Extract Apple-Style c608 box (CEA-608) subtitle using CCExtractor. - This isn't much more than a wrapper to the track.ccextractor function. - All this does, is actually check if a c608 box exists and only if so - does it actually call ccextractor. + This is not much more than a wrapper to the track.ccextractor function. + All this does is examine if a c608 box exists, and only then + does it call CCExtractor. - Even though there is a possibility of more than one c608 box, only one - can actually be extracted. It is also very possible this + Even though there is a possibility of more than one c608 box, unshackle + can extract only one. It is also very possible this needs to be done before any decryption as the decryption may destroy some of the metadata. - TODO: Need a test file with more than one c608 box to add support for - more than one CEA-608 extraction. + TODO: Need a test file with more than one c608 box, before unshackle can + extract more than one CEA-608 track. """ if not self.path: raise ValueError("You must download the track first.") @@ -638,9 +637,9 @@ class Video(Track): Remove EIA-CC data from Bitstream while keeping SEI data. This works by removing all NAL Unit's with the Type of 6 from the bistream - and then re-adding SEI data (effectively a new NAL Unit with just the SEI data). - Only bitstreams with x264 encoding information is currently supported due to the - obscurity on the MDAT mp4 box structure. Therefore, we need to use hacky regex. + and then re-adding SEI data (effectively a new NAL Unit with only the SEI data). + unshackle can do this only for bitstreams with x264 encoding information, because of + the obscurity on the MDAT mp4 box structure. Therefore, we need to use hacky regex. """ if not self.path or not self.path.exists(): raise ValueError("Cannot clean a Track that has not been downloaded.") diff --git a/unshackle/core/update_checker.py b/unshackle/core/update_checker.py index d18a50ea..ede03f77 100644 --- a/unshackle/core/update_checker.py +++ b/unshackle/core/update_checker.py @@ -11,11 +11,11 @@ import requests class UpdateChecker: """ - Check for available updates from the GitHub repository. + Examine the GitHub repository for available updates. - This class provides functionality to check for newer versions of the application - by querying the GitHub releases API. It includes rate limiting, caching, and - both synchronous and asynchronous interfaces. + This class examines the GitHub releases API for newer versions of the application. + It includes rate limiting, caching, and both synchronous and asynchronous + interfaces. Attributes: REPO_URL: GitHub API URL for latest release @@ -56,7 +56,7 @@ class UpdateChecker: @staticmethod def parse_version(version_string: str) -> str: """ - Parse and normalize version string by removing 'v' prefix. + Parse and normalise version string by removing 'v' prefix. Args: version_string: Raw version string from API @@ -117,13 +117,13 @@ class UpdateChecker: @classmethod def should_check_for_updates(cls, check_interval: int = DEFAULT_CHECK_INTERVAL) -> bool: """ - Check if enough time has passed since the last update check. + Examine whether enough time has passed since the last update check. Args: check_interval: Time in seconds between checks (default: 24 hours) Returns: - True if we should check for updates, False otherwise + True when it is time to do an update check, False otherwise """ cache_data = cls.load_cache_data() @@ -142,7 +142,7 @@ class UpdateChecker: Args: latest_version: The latest version found, if any - current_version: The current version being used + current_version: The current version in use """ cache_file = cls.get_cache_file() @@ -198,7 +198,7 @@ class UpdateChecker: @classmethod async def check_for_updates(cls, current_version: str) -> Optional[str]: """ - Check if there's a newer version available on GitHub. + Examine GitHub for a newer version. Args: current_version: The current version string (e.g., "1.1.0") @@ -223,7 +223,7 @@ class UpdateChecker: @classmethod def get_cached_update_info(cls, current_version: str) -> Optional[str]: """ - Check if there's a cached update available for the current version. + Examine the cache for an update for the current version. Args: current_version: The current version string diff --git a/unshackle/core/utilities.py b/unshackle/core/utilities.py index a3698d1a..41b5c001 100644 --- a/unshackle/core/utilities.py +++ b/unshackle/core/utilities.py @@ -119,14 +119,14 @@ def import_module_by_path(path: Path) -> ModuleType: def sanitize_filename(filename: str, spacer: str = ".") -> str: """ - Sanitize a string to be filename safe. + Sanitise a string to be filename safe. The spacer is safer to be a '.' for older DDL and p2p sharing spaces. - This includes web-served content via direct links and such. + This includes files served on the web through direct links and such. - Set `unicode_filenames: true` in config to preserve native language - characters (Korean, Japanese, Chinese, etc.) instead of transliterating - them to ASCII equivalents. + Set `unicode_filenames: true` in config to preserve the characters of the + original language (for example Korean, Japanese, or Chinese) instead of + transliterating them to ASCII equivalents. """ if not config.unicode_filenames: filename = unidecode(filename) @@ -145,7 +145,7 @@ def sanitize_filename(filename: str, spacer: str = ".") -> str: def is_close_match(language: Union[str, Language], languages: Sequence[Union[str, Language, None]]) -> bool: - """Check if a language is a close match to any of the provided languages.""" + """Examine whether a language closely matches any of the given languages.""" languages = [x for x in languages if x] if not languages: return False @@ -153,7 +153,7 @@ def is_close_match(language: Union[str, Language], languages: Sequence[Union[str def is_exact_match(language: Union[str, Language], languages: Sequence[Union[str, Language, None]]) -> bool: - """Check if a language is an exact match to any of the provided languages.""" + """Examine whether a language exactly matches any of the given languages.""" languages = [x for x in languages if x] if not languages: return False @@ -164,8 +164,8 @@ def partition_exclusions(tokens: Optional[Sequence[str]]) -> tuple[list[str], li """ Split selection tokens into wanted values and values excluded with a leading '-'. - Both sides keep the order they were written in and drop later repeats. A bare '-' - carries no value and is skipped. + Both sides keep the order the user wrote them in and drop later repeats. A bare '-' + carries no value, so this function ignores it. Example: >>> partition_exclusions(["all", "-es"]) @@ -209,7 +209,7 @@ def keep_forced_subtitle( forced_s_lang: Sequence[str], exact: bool = False, ) -> bool: - """The -fsl filter: non-forced tracks always pass, forced tracks only if they match forced_s_lang.""" + """The -fsl filter: non-forced tracks always pass, forced tracks only when they match forced_s_lang.""" if not forced: return True if not forced_s_lang: @@ -222,7 +222,7 @@ def embedded_audio_langs(videos: Sequence[Any], keep_videos: bool) -> list[str]: """ Return the audio languages carried inside video tracks rather than beside them. - A muxed stream keeps its audio in the video track, which a service declares by setting + A muxed video keeps its audio in the video track, which a service declares by setting ``data["audio_language"]``. That audio is only available while the video is kept, so dropping the video (``--audio-only``, ``--no-video``) drops the language with it. """ @@ -237,7 +237,7 @@ def find_missing_langs( *, exact: bool = False, ) -> list[str]: - """Return requested language tokens with no match in available languages.""" + """Return the requested language tokens that match no available language.""" match_func = is_exact_match if exact else is_close_match skip = {"all", "best", "orig"} return [tok for tok in requested if tok not in skip and not match_func(tok, available)] @@ -259,7 +259,8 @@ def matching_languages( In exact mode CLDR tag distance is not sufficient on its own: it rates a base language and its "paradigm" regional variant as the same language (distance 0 for en/en-US, pt/pt-BR), so it cannot separate en-US from en. Follow RFC 4647 Lookup: prefer the string-equal tag, and - fall back to the distance match only when no such tag is present (e.g. zh matches cmn). + fall back to the distance rating only when no such tag is present (for example, zh matches + cmn). """ tags = {str(x) for x in available if x} try: @@ -277,8 +278,8 @@ def resolve_sort_langs(tokens: Sequence[str], original: Optional[Union[str, Lang """ Prepare language tokens for a Tracks.sort_* by_language argument. - "orig" becomes the title language, or is dropped when the title has none. "all" and - "best" pass through, because the sort methods resolve them against the tracks. The + "orig" becomes the title language. This function drops it when the title has none. + "all" and "best" pass through, because the sort methods find them among the tracks. The result keeps the first position of each language and drops later repeats. """ resolved: list[str] = [] @@ -317,7 +318,7 @@ def get_boxes(data: bytes, box_type: bytes, as_bytes: bool = False) -> Box: # t - For each box found, the function updates the search offset to skip past the current box to avoid finding the same box multiple times - The function handles validation errors for certain box types (e.g., tenc) - - The size field is located 4 bytes before the box type identifier + - The size field sits 4 bytes before the box type identifier """ # using slicing to get to the wanted box is done because parsing the entire box and recursively # scanning through each box and its children often wouldn't scan far enough to reach the wanted box. @@ -366,8 +367,8 @@ def ap_case(text: str, keep_spaces: bool = False, stop_words: tuple[str] = None) Parameters: text: The text string to title case with AP/APA style. - keep_spaces: To keep the original whitespace, or to just use a normal space. - This would only be needed if you have special whitespace between words. + keep_spaces: To keep the original whitespace, or to use a normal space. + You need this only if you have special whitespace between words. stop_words: Override the default stop words with your own ones. """ if not text: @@ -500,9 +501,10 @@ def time_elapsed_since(start: float) -> str: def try_ensure_utf8(data: bytes) -> bytes: """ - Try to ensure that the given data is encoded in UTF-8. + Try to make sure that the given data is in UTF-8. - Gzip or zlib compressed input is decompressed first, detected by magic bytes. + This function first decompresses gzip or zlib compressed input, which it detects + by magic bytes. Callers hand in bytes read from files and from responses they fetched themselves, so there is no Content-Encoding header left to consult. @@ -510,7 +512,7 @@ def try_ensure_utf8(data: bytes) -> bytes: data: Input data that may or may not yet be UTF-8 or another encoding. Returns the input data encoded in UTF-8 if successful. If unable to detect the - encoding of the input data, then the original data is returned as-received. + encoding of the input data, then this function returns the original data as-received. """ if data[:2] == b"\x1f\x8b": try: @@ -546,8 +548,8 @@ def get_free_port() -> int: """ Get an available port to use between a-b (inclusive). - The port is freed as soon as this has returned, therefore, it - is possible for the port to be taken before you try to use it. + This function frees the port as soon as it returns. Thus something else + can take the port before you use it. """ with contextlib.closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s: s.bind(("", 0)) @@ -642,7 +644,7 @@ def scan_font_directory(font_dir: Path, fonts: dict[str, Path], log: logging.Log Args: font_dir: Directory to scan - fonts: Dictionary to populate with found fonts + fonts: Dictionary to fill with found fonts log: Logger instance for error reporting """ font_files = list(font_dir.rglob("*.ttf")) + list(font_dir.rglob("*.otf")) @@ -721,7 +723,7 @@ FONT_ALIASES = { def find_case_insensitive(font_name: str, fonts: dict[str, Path]) -> Optional[Path]: """ - Find font by case-insensitive name match. + Find a font by a case-insensitive name. Args: font_name: Font family name to find @@ -739,20 +741,20 @@ def find_case_insensitive(font_name: str, fonts: dict[str, Path]) -> Optional[Pa def find_font_with_fallbacks(font_name: str, system_fonts: dict[str, Path]) -> Optional[Path]: """ - Find a font by name with intelligent fallback matching. + Find a font by name, with intelligent fallbacks. Tries multiple strategies in order: - 1. Exact match (case-sensitive) - 2. Case-insensitive match + 1. Exact name (case-sensitive) + 2. Exact name (case-insensitive) 3. Alias lookup (Windows → Linux font equivalents) - 4. Partial/prefix match + 4. Partial or prefix name Args: font_name: The requested font family name (e.g., "Arial", "Times New Roman") system_fonts: Dictionary of available fonts (family name → path) Returns: - Path to the matched font file, or None if no match found + Path to the matched font file, or None when no font matches """ if not system_fonts: return None @@ -804,10 +806,10 @@ def suggest_font_packages(missing_fonts: list[str]) -> dict[str, list[str]]: Suggest system packages to install for missing fonts. Args: - missing_fonts: List of font family names that couldn't be found + missing_fonts: List of font family names that could not be found Returns: - Dictionary mapping package names to lists of fonts they would provide + Dictionary mapping package names to lists of fonts they would give """ suggestions = {} @@ -859,17 +861,17 @@ class DebugLogger: Structured JSON debug logger for unshackle. Outputs JSON Lines format where each line is a complete JSON object. - This makes it easy to parse, filter, and analyze logs programmatically. + This makes it easy to parse, filter, and analyse logs programmatically. """ def __init__(self, log_path: Optional[Path] = None, enabled: bool = False, log_keys: bool = False): """ - Initialize the debug logger. + Initialise the debug logger. Args: - log_path: Path to the log file. If None, logging is disabled. - enabled: Whether debug logging is enabled. - log_keys: Whether to log decryption keys (for debugging key issues). + log_path: Path to the log file. None turns logging off. + enabled: True to turn debug logging on. + log_keys: True to log the content keys (to debug content key issues). """ self.enabled = enabled and log_path is not None self.log_path = log_path @@ -883,7 +885,7 @@ class DebugLogger: self.log_session_start() def log_session_start(self): - """Log the start of a new session with environment information.""" + """Log the ``session_start`` entry, with environment information.""" import platform from unshackle.core import __code_hash__, __version__ @@ -921,7 +923,7 @@ class DebugLogger: Args: level: Log level (DEBUG, INFO, WARNING, ERROR) - operation: Name of the operation being performed + operation: Name of the operation message: Human-readable message context: Additional context information service: Service tag @@ -979,7 +981,7 @@ class DebugLogger: def sanitize_data(self, data: Any) -> Any: """ - Sanitize data for JSON serialization. + Sanitise data for JSON serialization. Handles complex objects and removes sensitive information. """ if data is None: @@ -1042,7 +1044,7 @@ class DebugLogger: **kwargs: Additional context Returns: - Operation ID that can be used to log the end of the operation + Operation ID you can use to log the end of the operation """ op_id = str(uuid4())[:8] self.log( @@ -1082,9 +1084,9 @@ class DebugLogger: Log a service API call. Args: - method: HTTP method (GET, POST, etc.) + method: HTTP method (for example GET or POST) url: Request URL - **kwargs: Additional request details (headers, body, etc.) + **kwargs: Additional request details (for example headers or body) """ message = kwargs.pop("message", "") level = kwargs.pop("level", "DEBUG") @@ -1094,10 +1096,10 @@ class DebugLogger: def log_drm_operation(self, drm_type: str, operation: str, **kwargs): """ - Log a DRM operation (PSSH extraction, license request, key retrieval). + Log a DRM operation (PSSH extraction, license request, content key retrieval). Args: - drm_type: DRM type (Widevine, PlayReady, etc.) + drm_type: DRM type (for example Widevine or PlayReady) operation: DRM operation name **kwargs: Additional context. Pass ``message=``/``level=`` to override the defaults. """ @@ -1111,7 +1113,7 @@ class DebugLogger: Args: vault_name: Name of the vault - operation: Vault operation (get_key, add_key, etc.) + operation: Vault operation (for example get_key or add_key) **kwargs: Additional context. Pass ``message=``/``level=`` to override the defaults. """ message = kwargs.pop("message", None) or f"Vault {vault_name}: {operation}" @@ -1160,7 +1162,7 @@ def get_debug_logger() -> Optional[DebugLogger]: def log_event(operation: str, *, level: str = "DEBUG", message: str = "", **kwargs: Any) -> None: - """Emit a single structured debug-log entry. No-op when debug logging is disabled. + """Send a single structured debug-log entry. No-op when debug logging is off. The canonical one-shot logging primitive. Replaces the ``if dl := get_debug_logger(): dl.log(...)`` guard boilerplate. To add logging to a new @@ -1176,7 +1178,7 @@ def timed_operation(operation: str, *, level: str = "DEBUG", message: str = "", """Time a block and log ``operation`` once it finishes, with ``duration_ms``. Logs at ``level`` with ``success=True`` on normal exit, or at ERROR with the exception and - ``success=False`` if the block raises (then re-raises). No-op when debug logging is disabled, + ``success=False`` if the block raises (then re-raises). No-op when debug logging is off, so it is always safe to wrap a block in it. Example:: @@ -1214,12 +1216,12 @@ def timed_operation(operation: str, *, level: str = "DEBUG", message: str = "", def init_debug_logger(log_path: Optional[Path] = None, enabled: bool = False, log_keys: bool = False): """ - Initialize the global debug logger. + Initialise the global debug logger. Args: log_path: Path to the log file - enabled: Whether debug logging is enabled - log_keys: Whether to log decryption keys (for debugging key issues) + enabled: True to turn debug logging on + log_keys: True to log the content keys (to debug content key issues) """ global debug_logger if debug_logger: diff --git a/unshackle/core/utils/bitrate.py b/unshackle/core/utils/bitrate.py index c85ad748..e518e09f 100644 --- a/unshackle/core/utils/bitrate.py +++ b/unshackle/core/utils/bitrate.py @@ -56,10 +56,11 @@ def measure_real_bitrate( leaky-bucket ceiling, not an average). This measures the true bitrate (bits/sec) from real media byte sizes and durations using ``bytes * 8 / sec``. - Single-file tracks are measured exactly. Segmented tracks probe up to - ``samples`` segments spread across the track and extrapolate; byte-range - segments need no request. Returns bits/sec, or ``None`` if it cannot be - measured. Never raises; a probe failure must not abort a download. + This function measures a single-file track exactly. For a segmented track it + probes a maximum of ``samples`` segments spread across the track and + extrapolates. A byte-range segment needs no request. Returns bits/sec, or + ``None`` when unshackle cannot measure the bitrate. Never raises. A probe + failure must not abort a download. """ from unshackle.core.tracks.track import Track @@ -125,13 +126,13 @@ def apply_real_bitrates( """ Probe real bitrates and overwrite ``track.bitrate`` for the tracks worth probing. - Probing every rendition is slow when a service exposes dozens. Tracks are - grouped by ``group_key`` (a quality tier), and only the ``per_group`` highest - declared-bitrate tracks per group are probed, in parallel. Each group is then - extended downward: while the lowest probed bitrate in a group sits below the - next unprobed track's declared bitrate (so that track could outrank a probed - one), the next track is probed too, until the probed set is safely above the - rest. Unprobed tracks keep their manifest-declared bitrate. + Probing every rendition is slow when a service exposes dozens. This function + groups tracks by ``group_key`` (a quality tier) and probes only the + ``per_group`` highest declared-bitrate tracks of each group, in parallel. It + then extends each group downward: while the lowest probed bitrate in a group + sits below the next unprobed track's declared bitrate (so that track could + outrank a probed one), it probes the next track too, until the probed set is + safely above the rest. Unprobed tracks keep their manifest-declared bitrate. """ groups: defaultdict[Hashable, list["Track"]] = defaultdict(list) for track in tracks: @@ -179,14 +180,14 @@ def probe_batch( def dedupe(segments: list[Segment]) -> list[Segment]: """ - Collapse segments that address the same bytes so each object is measured once. + Collapse segments that address the same bytes, to measure each object one time only. Manifests sometimes wrap a single file in several segment entries sharing one URL, with no byte range (a ``SegmentTemplate`` whose media pattern has no ``$Number$``) or with the same range. Each resolves to the whole file, so - counting them all would multiply the size by the segment count. Segments - sharing the same ``(url, byte_range)`` are merged into one entry whose duration - is the sum they cover. Distinct byte ranges of one file (different offsets) are + counting them all would multiply the size by the segment count. This function + merges the segments that share the same ``(url, byte_range)`` into one entry + whose duration is the sum they cover. Distinct byte ranges of one file (different offsets) are kept individual so their sizes still add up to the full track. """ merged: OrderedDict[tuple[str, Optional[str]], Segment] = OrderedDict() @@ -211,7 +212,7 @@ def pick_samples(segments: list[Segment], samples: int) -> list[Segment]: def probe_size(segment: Segment, session: Union[Session, RnetSession]) -> Optional[int]: - """Return a segment's byte size via HEAD, falling back to a ranged GET. Validates status.""" + """Return a segment's byte size with a HEAD request, falling back to a ranged GET. Validates status.""" try: res = session.head(segment.url, allow_redirects=True, timeout=PROBE_TIMEOUT) if getattr(res, "status_code", 0) in (200, 206): @@ -257,10 +258,10 @@ def uniform_segments( total_duration: Optional[float], ) -> list[Segment]: """ - Build Segments giving each an equal share of the total duration. + Assemble Segments giving each an equal share of the total duration. Used for DASH: ``DASH.get_period_segments`` returns timeline *start times* - rather than per-segment durations, so they cannot be trusted. Segment lengths + rather than per-segment durations, so unshackle cannot trust them. Segment lengths are near-uniform in practice, so the track duration (from ``mediaPresentationDuration``) split evenly is both correct and timeline-safe. """ @@ -384,10 +385,11 @@ def ffprobe_duration(url: str, session: Union[Session, RnetSession], *, log: log """ Read a single-file track's duration (seconds) without a manifest. - The bundled ffprobe segfaults on network input, so the file's ``moov`` box is - fetched over HTTP with the session (keeping the service's proxy/headers) and - piped to ffprobe as local bytes. The head of the file is tried first (VOD is - usually faststart), then the tail as a fallback for moov-at-end files. + The bundled FFprobe segfaults on network input, so unshackle fetches the + file's ``moov`` box over HTTP with the HTTP session (keeping the service's + proxy/headers) and pipes it to FFprobe as local bytes. This function tries the + head of the file first (VOD is usually faststart), then the tail as a fallback + for moov-at-end files. """ head = ranged_get(url, session, f"bytes=0-{MOOV_PROBE_BYTES - 1}") duration = probe_bytes_duration(head, log) @@ -414,7 +416,7 @@ def ranged_get(url: str, session: Union[Session, RnetSession], byte_range: str) def probe_bytes_duration(data: Optional[bytes], log: logging.Logger) -> Optional[float]: - """Pipe media bytes to ffprobe and return the format/stream duration in seconds.""" + """Pipe media bytes to FFprobe and return the container or track duration in seconds.""" if not data: return None ffprobe_bin = str(FFProbe) if FFProbe else "ffprobe" diff --git a/unshackle/core/utils/click_types.py b/unshackle/core/utils/click_types.py index 939ceb33..1496fe16 100644 --- a/unshackle/core/utils/click_types.py +++ b/unshackle/core/utils/click_types.py @@ -187,8 +187,8 @@ class SeasonRange(click.ParamType): def token_sides(self, match: re.Match) -> tuple[int, Optional[int], Optional[int]]: """Read one side of a token as (season, episode, part), episode/part None when absent. - A music disc x track token shares this key space, disc reading as the season and - track as the episode. An omitted disc is disc 1, not every disc: a bare number is + A music disc x track token shares this (season, episode, part) space, disc reading + as the season and track as the episode. An omitted disc is disc 1, not every disc: a bare number is the number the tracklist shows for a single-disc release, and widening it would silently add tracks on a multi-disc release. """ @@ -207,16 +207,17 @@ class SeasonRange(click.ParamType): """ Parse multiple tokens or ranged tokens as '{s}x{e}' strings. - An episode split into separately playable parts is addressed as '{s}x{e}.{p}'. - A part range must stay inside one episode, since how many parts an episode has - is not knowable here. A part-qualified exclusion cannot be removed from the - computed keys (they are base keys), so it becomes a '!' key resolved at match time. + You address an episode split into separately playable parts as '{s}x{e}.{p}'. + A part range must stay inside one episode, because how many parts an episode has + is not knowable here. unshackle cannot remove a part-qualified exclusion from the + computed keys, because those are base keys. The exclusion becomes a '!' entry that + unshackle applies later, when it matches an episode to the wanted keys. - Dated content is addressed by ISO air date. A date range uses ':' only, because + You address an episode by its ISO air date. A date range uses ':' only, because a date's own '-' separators are not a range separator. A music release uses the same keys, its disc reading as the season and its track - as the episode. A track is addressed as '{d}x{t}', or by its number alone, which + as the episode. You address a track as '{d}x{t}', or by its number alone, which is a track on disc 1. Supports exclusioning by putting a `-` before the token. @@ -393,10 +394,10 @@ class AudioCodecList(click.ParamType): class MultipleChoice(click.Choice): """ - The multiple choice type allows multiple values to be checked against - a fixed set of supported values. + The multiple choice type takes several values. Each value must be in a fixed + set of permitted values. - It internally uses and is based off of click.Choice. + It internally uses and extends click.Choice. """ name = "multiple_choice" @@ -428,8 +429,8 @@ class MultipleChoice(click.Choice): Parameters: ctx: Invocation context for this command. - param: The parameter that is requesting completion. - incomplete: Value being completed. May be empty. + param: The parameter that requests completion. + incomplete: The value to complete. Can be empty. """ incomplete = incomplete.rsplit(",")[-1] return super(self).shell_complete(ctx, param, incomplete) diff --git a/unshackle/core/utils/dovi.py b/unshackle/core/utils/dovi.py index d9f31738..f1672265 100644 --- a/unshackle/core/utils/dovi.py +++ b/unshackle/core/utils/dovi.py @@ -35,7 +35,7 @@ def extract_rpu( status: Optional[str] = "Extracting DV RPU...", label: str = "dovi_tool extract-rpu", ) -> bytes: - """Extract DV RPU NALs from a raw HEVC stream. `mode=None` skips the -m flag (untouched).""" + """Extract DV RPU NALs from a raw HEVC bitstream. `mode=None` skips the -m flag (untouched).""" tool = require_dovi_tool() args: list = [tool] if mode is not None: @@ -52,7 +52,7 @@ def inject_rpu( status: Optional[str] = "Re-injecting DV RPU...", label: str = "dovi_tool inject-rpu", ) -> bytes: - """Inject a DV RPU back into a raw HEVC stream, producing DV-signaled output.""" + """Inject a DV RPU back into a raw HEVC bitstream, producing DV-signaled output.""" tool = require_dovi_tool() return run_step( [tool, "inject-rpu", "-i", source, "--rpu-in", rpu, "-o", output], @@ -106,7 +106,7 @@ def generate_from_hdr10plus( status: Optional[str] = "Generating DV RPU from HDR10+ metadata...", label: str = "dovi_tool generate", ) -> bytes: - """Build a DV RPU from extracted HDR10+ metadata + an extra JSON descriptor.""" + """Assemble a DV RPU from extracted HDR10+ metadata + an extra JSON descriptor.""" tool = require_dovi_tool() return run_step( [tool, "generate", "-j", extra_json, "--hdr10plus-json", hdr10plus_json, "-o", output], @@ -117,7 +117,7 @@ def generate_from_hdr10plus( def extract_rpu_with_fallback(source: Path, output: Path, *, label: str = "dovi_tool extract-rpu") -> bytes: - """Try `-m 3` first; on MAX_PQ_LUMINANCE error, retry untouched (no -m). Returns stderr. + """Try `-m 3` first. On a MAX_PQ_LUMINANCE error, retry untouched (no -m). Returns stderr. Used when the caller wants automatic normalization but cannot abort if the source rejects mode-3 conversion. diff --git a/unshackle/core/utils/firefox_cookie_extractor.py b/unshackle/core/utils/firefox_cookie_extractor.py index 1bc70cb2..a58f86d2 100644 --- a/unshackle/core/utils/firefox_cookie_extractor.py +++ b/unshackle/core/utils/firefox_cookie_extractor.py @@ -85,7 +85,7 @@ def get_local_storage_data(profile_path: Path, hosts: List[str], tmp_dir_path: P def get_firefox_cookies(service_settings: dict) -> Optional[CookieJar]: """ Extracts cookies and optionally localStorage from Firefox based on provided host patterns. - Implements security hardening and data consistency (WAL support) as per PR review. + Implements security hardening and data consistency (it can copy the WAL log) as per PR review. """ raw_hosts = service_settings.get("hosts", []) # Filter empty or dangerously short hosts to prevent full store dumps (Finding #3) diff --git a/unshackle/core/utils/ip_info.py b/unshackle/core/utils/ip_info.py index ea75d7ee..b6fa52ca 100644 --- a/unshackle/core/utils/ip_info.py +++ b/unshackle/core/utils/ip_info.py @@ -23,7 +23,7 @@ log = logging.getLogger("ip_info") class RateLimited(Exception): - """Raised by a provider fetcher when the upstream returns 429.""" + """Raised by a geolocation API fetcher when the upstream returns 429.""" def normalize( @@ -37,7 +37,7 @@ def normalize( as_name: str = "", continent_code: str = "", ) -> Optional[dict]: - """Build the canonical IP-info dict, or None if no country code is present.""" + """Assemble the canonical IP-info dict, or None if no country code is present.""" code = country_code.strip() if not code: return None @@ -94,13 +94,14 @@ def parse_ip_api_in(data: dict) -> Optional[dict]: def lookup_session(source: Optional[requests.Session]) -> requests.Session: """ - Build a plain, retry-free requests session for IP geolocation. + Assemble a plain, retry-free `requests` HTTP session for IP geolocation. - Geolocation needs no TLS fingerprinting, so we skip the impersonated rnet - session and the base session's urllib3 retry loop. Both retry 429 internally, - which hides the response and defeats fast provider handover. With a bare session - a 429 comes straight back so we can move to the next provider immediately. Only - the proxy is carried over so proxied lookups still report the proxy's exit IP. + Geolocation needs no TLS fingerprinting, so we skip the impersonated rnet HTTP + session, and the urllib3 retry loop of the base HTTP session. Both retry 429 + internally, which hides the response and defeats fast handover to the next + geolocation API. With a bare HTTP session a 429 comes straight back, so we can + move to the next API immediately. This function carries over the proxy only, so + proxied lookups still report the proxy's exit IP. """ sess = requests.Session() proxies = getattr(source, "proxies", None) @@ -139,7 +140,7 @@ def fetch_ipinfo(session: requests.Session) -> Optional[dict]: def fetch_ip_api_in(session: requests.Session) -> Optional[dict]: - """ip-api.in has no /me endpoint, so resolve IP via ipify first, then look it up.""" + """ip-api.in has no /me endpoint, so get the IP from ipify first, then look it up.""" ip_resp = session.get("https://api.ipify.org", timeout=REQUEST_TIMEOUT) if ip_resp.status_code == 429: raise RateLimited() @@ -185,20 +186,21 @@ def get_ip_info( cached: bool = False, ) -> Optional[dict]: """ - Look up IP/geolocation info via ipinfo.io (Lite when `ipinfo_api_key` configured) + Look up IP/geolocation info through ipinfo.io (Lite when `ipinfo_api_key` configured) with fallback to ip-api.in. Live lookups return a dict with `ip`, `country` (lowercase ISO2), `country_code` (uppercase ISO2), `region`, `city`, `org`, `asn`, `as_name`, `continent_code` and `_provider`. Cached lookups return only `country`/`country_code` (see GEO_CACHE_KEYS). - Returns None if every provider fails. + Returns None if every geolocation API fails. Args: - session: Optional requests session. If a proxied session is passed, the - returned info reflects the proxy's exit IP. Auth headers for ipinfo - are sent per-request; never mutated onto session.headers. + session: Optional `requests` HTTP session. If you pass a proxied session, + the returned info reflects the proxy's exit IP. unshackle sends the + ipinfo auth headers per-request, and never mutates them onto + session.headers. cached: When True, read/write a 24h Cacher-backed entry. Use only for - local IP lookups, never with a proxied session. + local IP lookups, never with a proxied HTTP session. """ cache = None if cached: diff --git a/unshackle/core/utils/language_tags.py b/unshackle/core/utils/language_tags.py index a8b0ccd1..ac1d6552 100644 --- a/unshackle/core/utils/language_tags.py +++ b/unshackle/core/utils/language_tags.py @@ -19,8 +19,8 @@ def evaluate_language_tag( ) -> str: """Evaluate language tag rules against selected tracks. - Rules are evaluated in order; the first matching rule's tag is returned. - Returns empty string if no rules match. + This function evaluates the rules in order. It returns the tag of the first rule + that matches, or an empty string if no rule matches. Args: rules: List of rule dicts from config, each with conditions and a ``tag``. @@ -28,7 +28,7 @@ def evaluate_language_tag( subtitle_languages: Languages of the selected subtitle tracks. Returns: - The tag string from the first matching rule, or ``""`` if none match. + The tag string from the first rule that matches, or ``""`` if none match. """ for rule in rules: tag = rule.get("tag") @@ -48,7 +48,7 @@ def rule_matches( audio_languages: Sequence[Language], subtitle_languages: Sequence[Language], ) -> bool: - """Check if all conditions in a rule are satisfied.""" + """Return True when every condition in the rule matches the tracks.""" has_condition = False audio_lang = rule.get("audio") diff --git a/unshackle/core/utils/post_scripts.py b/unshackle/core/utils/post_scripts.py index ffb4d369..20757238 100644 --- a/unshackle/core/utils/post_scripts.py +++ b/unshackle/core/utils/post_scripts.py @@ -1,7 +1,7 @@ -"""Run user-defined scripts after a download completes. +"""Operate user-defined scripts after a download completes. -A command is tokenized once, then variables are substituted into the resulting argv -tokens and the process is spawned with ``shell=False``. Substituting after tokenizing +unshackle tokenizes a command one time, then substitutes the variables into the +resulting argv tokens, then spawns the process with ``shell=False``. Substituting after tokenizing is what makes a title like ``Bob"; rm -rf ~`` harmless: a value can never become a new token or a shell operator. The cost is that the command has no shell features and no interpreter lookup, so users name the interpreter themselves (``python upload.py``). @@ -37,7 +37,7 @@ _warned_entries: set[str] = set() def tokenize(command: str) -> list[str]: - """Split a command string into argv tokens, before any variable is substituted.""" + """Split a command string into argv tokens, before unshackle substitutes any variable.""" if os.name == "nt": def unquote(token: str) -> str: @@ -51,7 +51,7 @@ def tokenize(command: str) -> list[str]: def substitute(tokens: Sequence[str], context: dict[str, str]) -> list[str]: - """Replace every ``{variable}`` inside each token; unknown or empty ones become ''.""" + """Replace every ``{variable}`` inside each token. Unknown or empty ones become ''.""" unknown: set[str] = set() def replace(match: re.Match[str]) -> str: @@ -92,13 +92,13 @@ def build_context( ids: Optional[dict[str, Any]] = None, error: str = "", ) -> dict[str, str]: - """Build the variable set for one output file. + """Assemble the variable set for one output file. Metadata comes from the same naming context that produced the file's name, so - ``{quality}`` and ``{hdr}`` always describe *this* file. That is what lets - ``-q 1080,2160 -r HDR10,SDR`` label its four outputs correctly. Season and episode - are handed over as plain numbers rather than the filename's ``S01E05`` form: the - spacer and padding belong to the filename template, not to the data. + ``{quality}`` and ``{hdr}`` always give the values of *this* file. That is what lets + ``-q 1080,2160 -r HDR10,SDR`` label its four outputs correctly. unshackle gives the + season and the episode as plain numbers rather than in the filename's ``S01E05`` + form: the spacer and padding belong to the filename template, not to the data. """ context: dict[str, str] = {} @@ -152,7 +152,7 @@ def season_context(context: dict[str, str], folder: Path) -> dict[str, str]: def _entries(event: str, mode: str, overrides: Sequence[str] = ()) -> Iterator[str]: - """Commands configured for this event and mode; --postscript replaces the config list.""" + """Commands configured for this event and mode. --postscript replaces the config list.""" if overrides: if event == "success" and mode == "file": yield from overrides diff --git a/unshackle/core/utils/redact.py b/unshackle/core/utils/redact.py index 3350e1bb..1be79afe 100644 --- a/unshackle/core/utils/redact.py +++ b/unshackle/core/utils/redact.py @@ -20,7 +20,7 @@ SENSITIVE_QUERY_PARAM_RE = re.compile(r"(?i)\b(password|passwd|pwd|token|api_key def redact_text(text: Optional[str], secrets: Iterable[str] = ()) -> Optional[str]: """ Mask URL userinfo, secret-bearing query parameters, and any known secret - strings inside a free-text value before it is logged or serialized. + strings inside a free-text value before unshackle logs or serializes it. """ if not isinstance(text, str) or not text: return text @@ -33,7 +33,7 @@ def redact_text(text: Optional[str], secrets: Iterable[str] = ()) -> Optional[st def safe_display_url(url: str) -> str: - """Rebuild a URL from its scheme/host/port/path only, so userinfo can never be logged.""" + """Rebuild a URL from its scheme/host/port/path only, so unshackle can never log the userinfo.""" parts = urlsplit(url) netloc = parts.hostname or "" if parts.port: @@ -42,7 +42,7 @@ def safe_display_url(url: str) -> str: def path_bases() -> list[tuple[str, str]]: - """Local base directories to strip from logged paths, longest-first. + """Local base directories to remove from logged paths, longest-first. Anchors everything left of the unshackle install/root (and the venv / home dir) so an absolute path like ``/home/me/Projects/unshackle-live/temp`` logs as ``/temp``, @@ -90,7 +90,7 @@ def path_redaction_enabled() -> bool: def redact_path(text: Optional[str]) -> Optional[str]: """Replace local base-directory prefixes (install root, venv, home) in ``text`` with tokens. - Idempotent and cheap; only touches strings that actually contain a known base dir, so URLs + Idempotent and cheap. It touches only the strings that contain a known base dir, so URLs and relative paths pass through unchanged. Disabled by ``config.redact_paths: false``. """ if not isinstance(text, str) or not text or not path_redaction_enabled(): @@ -121,7 +121,7 @@ def collapse_url(match: "re.Match[str]") -> str: def redact_url(text: Optional[str]) -> Optional[str]: """Collapse every http(s) URL in ``text`` to ``redacted[.ext]``. - Hides content/CDN/manifest/segment/api locations (host + path + query) from shareable debug + Hides media/CDN/manifest/segment/api locations (host + path + query) from shareable debug logs while keeping the file extension so the manifest/segment type stays visible (e.g. ``redacted.mpd``). Non-URL strings pass through unchanged. """ diff --git a/unshackle/core/utils/selector.py b/unshackle/core/utils/selector.py index 73fba581..fe73c082 100644 --- a/unshackle/core/utils/selector.py +++ b/unshackle/core/utils/selector.py @@ -31,16 +31,16 @@ class Selector: collapse_on_start: bool = False, ): """ - Initialize the Selector. + Initialise the Selector. Args: options: List of strings to select from. cursor_style: Rich style for the highlighted cursor item. text_style: Rich style for normal items. page_size: Number of items to show per page. - minimal_count: Minimum number of items that must be selected. + minimal_count: Minimum number of items you must select. dependencies: Dictionary mapping parent index to list of child indices. - collapse_on_start: If True, child items are hidden initially. + collapse_on_start: If True, the Selector hides child items initially. """ self.options = options self.cursor_style = cursor_style @@ -219,7 +219,7 @@ class Selector: def toggle_expand_all(self): """ Toggles expansion state of ALL headers. - If all are expanded -> Collapse all. + If every header shows its children -> Collapse all. Otherwise -> Expand all. """ if not self.dependencies: @@ -242,7 +242,7 @@ class Selector: def get_input_windows(self): """ Captures and parses keyboard input on Windows systems using msvcrt. - Returns command strings like 'UP', 'DOWN', 'ENTER', etc. + Returns command strings such as 'UP', 'DOWN', or 'ENTER'. """ key = msvcrt.getch() # Ctrl+C (0x03) or ESC (0x1b) @@ -293,7 +293,7 @@ class Selector: def get_input_unix(self): """ Captures and parses keyboard input on Unix/Linux systems using click.getchar(). - Returns command strings like 'UP', 'DOWN', 'ENTER', etc. + Returns command strings such as 'UP', 'DOWN', or 'ENTER'. """ char = click.getchar() # Ctrl+C @@ -406,9 +406,9 @@ def select_multiple( options: List of options to display. minimal_count: Minimum number of selections required. page_size: Number of items per page. - return_indices: If True, returns indices; otherwise returns the option strings. - cursor_style: Style color for the cursor. - collapse_on_start: If True, child items are hidden initially. + return_indices: If True, returns indices. If False, returns the option strings. + cursor_style: Style colour for the cursor. + collapse_on_start: If True, the Selector hides child items initially. """ selector = Selector( options=options, diff --git a/unshackle/core/utils/sslciphers.py b/unshackle/core/utils/sslciphers.py index e3eb210f..d62c9073 100644 --- a/unshackle/core/utils/sslciphers.py +++ b/unshackle/core/utils/sslciphers.py @@ -8,9 +8,9 @@ class SSLCiphers(HTTPAdapter): """ Custom HTTP Adapter to change the TLS Cipher set and security requirements. - Security Level may optionally be provided. A level above 0 must be used at all times. - A list of Security Levels and their security is listed below. Usually 2 is used by default. - Do not set the Security level via @SECLEVEL in the cipher list. + You can optionally give a Security Level. Always use a level above 0. The list of + Security Levels and their security is below. Usually the default is 2. + Do not set the Security level with @SECLEVEL in the cipher list. Level 0: Everything is permitted. This retains compatibility with previous versions of OpenSSL. diff --git a/unshackle/core/utils/subprocess.py b/unshackle/core/utils/subprocess.py index 1eb0c3e8..2cfac7c2 100644 --- a/unshackle/core/utils/subprocess.py +++ b/unshackle/core/utils/subprocess.py @@ -17,10 +17,10 @@ def log_tool_run( duration_ms: Optional[float] = None, **context: object, ) -> None: - """Emit a structured ``tool_run`` debug-log entry for an external tool invocation. + """Send a structured ``tool_run`` debug-log entry for an external tool invocation. - Central helper so every binary call (ffmpeg, mkvpropedit, dovi_tool, etc.) logs the - same shape. No-op when debug logging is disabled. + Central helper so every binary call (FFmpeg, mkvpropedit, dovi_tool, and the other + tools) logs the same shape. No-op when debug logging is off. """ dl = get_debug_logger() if not dl: @@ -41,7 +41,7 @@ def log_tool_run( def ffprobe(uri: Union[bytes, Path]) -> dict: - """Use ffprobe on the provided data to get stream information.""" + """Use FFprobe on the provided data to get its track information (``-show_streams``).""" if not binaries.FFProbe: raise EnvironmentError('FFProbe executable "ffprobe" not found but is required.') @@ -88,10 +88,10 @@ def run_step( output: Optional[Path] = None, label: str = "subprocess step", ) -> bytes: - """Run a CLI step that writes to `output` (when provided). Returns stderr bytes. + """Operate a CLI step that writes to `output` (when provided). Returns stderr bytes. Raises RuntimeError with the stderr tail when the process exits non-zero, or when - `output` is given and does not exist / is empty after the run. + you give `output` and it does not exist or is empty after the step. """ if output is not None: output.unlink(missing_ok=True) diff --git a/unshackle/core/utils/tag_rules.py b/unshackle/core/utils/tag_rules.py index 53636087..9f314933 100644 --- a/unshackle/core/utils/tag_rules.py +++ b/unshackle/core/utils/tag_rules.py @@ -26,7 +26,8 @@ NUMBER = re.compile(r"\d+(?:\.\d+)?") def evaluate_tag_rules(rules: Any, context: dict[str, Any]) -> Optional[str]: """Evaluate tag rules against a filename template context. - Rules are evaluated in order; the first matching rule's tag is returned. + unshackle evaluates the rules in order. It returns the tag of the first rule + that matches. Conditions are AND-ed, compared case-insensitively, and a list matches if any entry matches. A value may lead with a comparison operator, e.g. ">=2160" or "!=DV". @@ -37,7 +38,7 @@ def evaluate_tag_rules(rules: Any, context: dict[str, Any]) -> Optional[str]: context: The already-built filename template context. Returns: - The tag string from the first matching rule, or ``None`` if none match. + The tag string from the first rule that matches, or ``None`` if none match. """ if not isinstance(rules, list): log.warning("tag_rules must be a list of rules, ignoring: %r", rules) @@ -76,19 +77,19 @@ def evaluate_tag_rules(rules: Any, context: dict[str, Any]) -> Optional[str]: def has_bool(expected: Any) -> bool: - """Check for a YAML boolean, which is an unquoted word the user meant as text.""" + """Examine the expected value for a YAML boolean, which is an unquoted word the user meant as text.""" values = expected if isinstance(expected, list) else [expected] return any(isinstance(x, bool) for x in values) def matches(expected: Any, actual: Any) -> bool: - """Check one condition, where a list matches on any entry.""" + """Examine one condition, where a list matches if any one entry matches.""" values = expected if isinstance(expected, list) else [expected] return any(matches_one(x, actual) for x in values) def matches_one(expected: Any, actual: Any) -> bool: - """Check a single value, which may carry a leading comparison operator.""" + """Examine a single value, which can carry a leading comparison operator.""" text = "" if expected is None else str(expected) for op in sorted(OPS, key=len, reverse=True): if text.startswith(op): diff --git a/unshackle/core/utils/tags.py b/unshackle/core/utils/tags.py index 93ba2d1f..ed0293df 100644 --- a/unshackle/core/utils/tags.py +++ b/unshackle/core/utils/tags.py @@ -67,7 +67,7 @@ def apply_tags(path: Path, tags: dict[str, str]) -> None: def build_tags_from_ids(ids: ExternalIds, kind: str) -> dict[str, str]: - """Build standard MKV tags from external IDs.""" + """Assemble standard MKV tags from external IDs.""" tags: dict[str, str] = {} if ids.imdb_id: tags["IMDB"] = ids.imdb_id diff --git a/unshackle/core/utils/template_formatter.py b/unshackle/core/utils/template_formatter.py index 33c77a68..da63b6a2 100644 --- a/unshackle/core/utils/template_formatter.py +++ b/unshackle/core/utils/template_formatter.py @@ -28,7 +28,7 @@ class TemplateFormatter: """ def __init__(self, template: str, spacer: str | None = None): - """Initialize the template formatter. + """Initialise the template formatter. Args: template: Template string with variables in {variable} format diff --git a/unshackle/core/utils/webvtt.py b/unshackle/core/utils/webvtt.py index 5e84acc3..8a302ea0 100644 --- a/unshackle/core/utils/webvtt.py +++ b/unshackle/core/utils/webvtt.py @@ -23,14 +23,15 @@ def merge_segmented_webvtt(vtt_raw: str, segment_durations: Optional[list[int]] Timing lines, cue settings and payload are kept verbatim, so inline formatting (span tags, entities, ASS-style overrides) survives untouched. - No timestamp offset is applied: segments are expected to carry absolute cue - times already (same guard as shaka-player and N_m3u8DL-RE; re-offsetting - double-shifts such streams). segment_durations/timescale are accepted for - call-site compatibility and only matter if offsetting is ever reintroduced. + This function applies no timestamp offset: the segments must already carry absolute + cue times. This is the same guard as shaka-player and N_m3u8DL-RE, because + re-offsetting double-shifts such documents. This function accepts + segment_durations/timescale for call-site compatibility. They only matter if + someone reintroduces offsetting. - Unique STYLE/REGION blocks are kept (players that don't support them ignore - them); cue identifiers and NOTE blocks are dropped; cues with a malformed - timing line are skipped (downstream sanitizers catch worse). + This function keeps unique STYLE/REGION blocks (players that cannot read them ignore + them). It drops cue identifiers and NOTE blocks. It skips a cue with a malformed + timing line (downstream sanitizers catch worse). """ # each kept cue: [start, end, settings, payload, normalized payload for dedupe] cues: list[list[str]] = [] diff --git a/unshackle/core/vault.py b/unshackle/core/vault.py index e7e53424..d62f61d9 100644 --- a/unshackle/core/vault.py +++ b/unshackle/core/vault.py @@ -14,15 +14,15 @@ class Vault(metaclass=ABCMeta): @abstractmethod def get_key(self, kid: Union[UUID, str], service: str) -> Optional[str]: """ - Get Key from Vault by KID (Key ID) and Service. + Get the content key from the Vault by KID (Key ID) and Service. - It does not get Key by PSSH as the PSSH can be different depending on it's implementation, - or even how it was crafted. Some PSSH values may also actually be a CENC Header rather - than a PSSH MP4 Box too, which makes the value even more confusingly different. + It does not get the content key by PSSH, as the PSSH can be different depending on its + implementation, or even on how it was made. Some PSSH values can also be a CENC + Header rather than a PSSH MP4 Box, which makes the value even more confusingly different. However, the KID never changes unless the video file itself has changed too, meaning the - key for the presumed-matching KID wouldn't work, further proving matching by KID is - superior. + content key for the presumed-matching KID would not work, further proving matching by KID + is superior. """ @abstractmethod @@ -37,8 +37,8 @@ class Vault(metaclass=ABCMeta): def add_keys(self, service: str, kid_keys: dict[Union[UUID, str], str]) -> int: """ Add Multiple Content Keys with Key IDs for Service to the Vault. - Pre-existing Content Keys are ignored/skipped. - Raises PermissionError if the user has no permission to create the table. + The Vault ignores pre-existing Content Keys. + Raises PermissionError if the user has no permission to make the table. """ @abstractmethod diff --git a/unshackle/core/vaults.py b/unshackle/core/vaults.py index 936bd29a..e878e717 100644 --- a/unshackle/core/vaults.py +++ b/unshackle/core/vaults.py @@ -111,9 +111,9 @@ class Vaults: def add_keys(self, kid_keys: dict[Union[UUID, str], str]) -> int: """ - Add multiple KID:KEYs to all Vaults. Duplicate Content Keys are skipped. - PermissionErrors when the user cannot create Tables are absorbed and ignored. - Vaults with no_push=True are skipped. + Add multiple KID:KEYs to all Vaults. The Vaults skip duplicate Content Keys. + This method absorbs and ignores the PermissionError a Vault raises when it cannot make Tables. + This method also skips Vaults with no_push=True. """ success = 0 for vault in self.vaults: diff --git a/unshackle/services/EXAMPLE/__init__.py b/unshackle/services/EXAMPLE/__init__.py index 5e9ead8b..1189d9f9 100644 --- a/unshackle/services/EXAMPLE/__init__.py +++ b/unshackle/services/EXAMPLE/__init__.py @@ -26,8 +26,8 @@ from unshackle.core.utilities import is_close_match class EXAMPLE(Service): """ Reference service for domain.com - a deliberately exhaustive showcase of - EVERYTHING an unshackle service can touch. It is NOT meant to run against a - real API; it exists so a new service author can see one canonical example of + EVERYTHING an unshackle service can touch. It is NOT meant to operate against + a real API. It exists so a new service author can see one canonical example of every framework feature in one place. Music is the one exception: it has its own reference service, MUSIC_EXAMPLE. @@ -58,9 +58,9 @@ class EXAMPLE(Service): get_tracks DASH variant fan-out (default) + HLS/ISM alternates fetch_dash_manifest range override, HDR10+ flip, DV-composite, Atmos, descriptive audio, channel fixups, cover-art attachment, - VUI normalize, bitrate-window awareness + VUI normalise, bitrate-window awareness get_chapters Chapters() with named + unnamed markers - get_widevine_* service cert + license (per-segment PSSH via `track`) + get_widevine_* service certificate + license (per-segment PSSH through `track`) get_playready_license PlayReady challenge POST get_clearkey_license DASH org.w3.clearkey JWK Set POST (Laurl fallback) """ diff --git a/unshackle/services/MUSIC_EXAMPLE/__init__.py b/unshackle/services/MUSIC_EXAMPLE/__init__.py index fcd1f844..ac80f85e 100644 --- a/unshackle/services/MUSIC_EXAMPLE/__init__.py +++ b/unshackle/services/MUSIC_EXAMPLE/__init__.py @@ -29,8 +29,8 @@ from unshackle.core.tracks import Audio, Chapters, Tracks class MUSIC_EXAMPLE(Service): """ Reference music service for musicdomain.com - the music counterpart of the - EXAMPLE service. Like EXAMPLE it is NOT meant to run against a real API; it - exists so a new music-service author can see the whole music surface in one + EXAMPLE service. Like EXAMPLE it is NOT meant to operate against a real API. + It exists so a new music-service author can see the whole music surface in one place. Version: 1.0.0 @@ -49,7 +49,7 @@ class MUSIC_EXAMPLE(Service): artists separately: https://musicdomain.com/album/20914 -> album/20914 - -d / --device selects a client profile block from config.yaml. A mobile - profile is usually entitled to fewer formats than a desktop one. + profile usually gets fewer formats than a desktop one. - --lyrics costs one extra request per track, so it is opt-in. The tagger writes what it returns into the track's lyrics tag. - -w / --wanted narrows a release to some of its tracks: `-w 1-5` takes tracks @@ -61,7 +61,7 @@ class MUSIC_EXAMPLE(Service): music_from_album build_music_from_songs, the idiomatic collection builder build_song the full Song field set, and what `data` feeds get_tracks ONE Audio track per codec+bitrate the source offers - get_chapters empty; a song has no chapters + get_chapters empty, because a song has no chapters on_track_downloaded restore the real container suffix before tagging There is no music-specific Service API. A `Song` is an ordinary `Title` on the @@ -258,9 +258,9 @@ class MUSIC_EXAMPLE(Service): def music_from_album(self, release_id: str) -> Music: """An ALBUM, EP or SINGLE. All three come back from one endpoint here. - `build_music_from_songs` is the idiomatic way to build a release. It reads the + `build_music_from_songs` is the idiomatic way to assemble a release. It reads the year, the track and disc totals, the artwork and the total duration back off - the Songs, so each of those facts is stated once, in build_song, instead of + the Songs, so build_song states each of those facts once instead of twice. Construct `Music` directly only when the header must say something the songs do not imply, as the discography branch above does. """ @@ -289,8 +289,8 @@ class MUSIC_EXAMPLE(Service): return response.json() def build_song(self, release: dict, track: dict) -> Song: - """One Song per track. `name`, `artist`, `album` and `track` are required; - everything else is optional and simply left out when the API cannot say. + """One Song per track. `name`, `artist`, `album` and `track` are necessary. + Everything else is optional, and the service leaves it out when the API cannot say. Music APIs are inconsistent about the shape of every one of these fields, which is why `unshackle.core.music` ships normalizers. Use them instead of writing @@ -389,9 +389,9 @@ class MUSIC_EXAMPLE(Service): 24-bit FLAC from a 16-bit FLAC, and it is what sort_audio ranks them by. MIGRATING FROM THE OLD MODEL: a music service used to return exactly one Audio - track and run its own quality fallback inside an overridden Audio.download(). + track and do its own quality fallback inside an overridden Audio.download(). Do not. Nothing downstream can see past a single track, so the user cannot ask - for the CD master, cannot list what exists, and cannot tell what they got. + for the CD master, cannot see what exists, and cannot tell what they got. """ reason = (title.data or {}).get("unavailable_reason") if reason: diff --git a/unshackle/vaults/API.py b/unshackle/vaults/API.py index de467eb8..af74b62e 100644 --- a/unshackle/vaults/API.py +++ b/unshackle/vaults/API.py @@ -237,7 +237,7 @@ class Exceptions: """Authentication Error Occurred, is your token valid? Do you have permission to make this call?""" class TooManyRequests(Exception): - """Rate Limited; Sent too many requests in a given amount of time.""" + """Rate Limited. Sent too many requests in a given amount of time.""" class PageInvalid(Exception): """Requested page does not exist.""" diff --git a/unshackle/vaults/HTTP.py b/unshackle/vaults/HTTP.py index 41abd826..0cc43615 100644 --- a/unshackle/vaults/HTTP.py +++ b/unshackle/vaults/HTTP.py @@ -17,7 +17,7 @@ class InsertResult(Enum): class HTTP(Vault): """ - Key Vault using HTTP API with support for multiple API modes. + Key Vault using HTTP API that can operate in several API modes. Supported modes: - query: Uses GET requests with query parameters @@ -37,7 +37,7 @@ class HTTP(Vault): timeout: float = 10.0, ): """ - Initialize HTTP Vault. + Initialise HTTP Vault. Args: name: Vault name @@ -346,7 +346,7 @@ class HTTP(Vault): def set_title(self, title: str): """ - Set a title to be used for the next key insertions. + Set a title for the next content key insertions. This is optional and will be sent with add_key requests if available. """ self.current_title = title @@ -355,7 +355,7 @@ class HTTP(Vault): self, service: str, kid: Union[UUID, str], key: str, title: Optional[str] = None ) -> InsertResult: """ - Insert a key and return detailed result information. + Insert a content key and return detailed result information. This method provides more granular feedback than the standard add_key method. Available in both API modes. """ diff --git a/unshackle/vaults/MySQL.py b/unshackle/vaults/MySQL.py index 77570d1c..4dccdfe1 100644 --- a/unshackle/vaults/MySQL.py +++ b/unshackle/vaults/MySQL.py @@ -13,8 +13,8 @@ class MySQL(Vault): def __init__(self, name: str, host: str, database: str, username: str, no_push: bool = False, **kwargs): """ - All extra arguments provided via **kwargs will be sent to pymysql.connect. - This can be used to provide more specific connection information. + All extra arguments given through **kwargs will be sent to pymysql.connect. + Use this to give more specific connection information. """ super().__init__(name, no_push) self.slug = f"{host}:{database}:{username}" @@ -193,7 +193,7 @@ class MySQL(Vault): return names[0] if names else None def has_table(self, name: str) -> bool: - """Check if the Vault has a Table with the specified name.""" + """Return True if the Vault has a Table with the specified name.""" conn = self.conn_factory.get() cursor = conn.cursor() @@ -207,7 +207,7 @@ class MySQL(Vault): cursor.close() def create_table(self, name: str): - """Create a Table with the specified name if not yet created.""" + """Make a Table with the specified name if it does not exist yet.""" if self.has_table(name): return @@ -234,7 +234,7 @@ class MySQL(Vault): cursor.close() def get_permissions(self) -> list: - """Get and parse Grants to a more easily usable list tuple array.""" + """Get and parse Grants to a more usable list tuple array.""" conn = self.conn_factory.get() cursor = conn.cursor() @@ -256,7 +256,7 @@ class MySQL(Vault): cursor.close() def has_permission(self, operation: str, database: Optional[str] = None, table: Optional[str] = None) -> bool: - """Check if the current connection has a specific permission.""" + """Return True if the current connection has a specific permission.""" grants = [x for x in self.permissions if x[0] == ["*"] or operation.upper() in x[0]] if grants and database: grants = [x for x in grants if x[1][0] in (database, "*")] diff --git a/unshackle/vaults/SQLite.py b/unshackle/vaults/SQLite.py index 7b03274e..ab293ef9 100644 --- a/unshackle/vaults/SQLite.py +++ b/unshackle/vaults/SQLite.py @@ -166,7 +166,7 @@ class SQLite(Vault): cursor.close() def has_table(self, name: str) -> bool: - """Check if the Vault has a Table with the specified name.""" + """Return True if the Vault has a Table with the specified name.""" conn = self.conn_factory.get() cursor = conn.cursor() @@ -177,7 +177,7 @@ class SQLite(Vault): cursor.close() def create_table(self, name: str): - """Create a Table with the specified name if not yet created.""" + """Make a Table with the specified name if it does not exist yet.""" if self.has_table(name): return