diff --git a/CHANGELOG.md b/CHANGELOG.md index 50224540..070b4423 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm This changelog is automatically generated using [git-cliff](https://git-cliff.org). -## [Unreleased] +## [5.4.0] - 2026-08-15 ### Features @@ -19,6 +19,35 @@ This changelog is automatically generated using [git-cliff](https://git-cliff.or - *dl*: Animate merge/decrypt/mux stages with gradient pulse bars - *tags*: Add imdb_api_enabled toggle, off by default - *tags*: Add OMDb metadata provider with omdb_api_key config +- *cdm*: Stamp remote CDM DRM type at load for reliable detection +- *ism*: Harden Smooth parsing and derive fps/codec from spec fields +- Original-language aware DUAL/MULTi tagging and new DUBBED variable +- *drm*: Add PlayReady HRESULT decoder +- *drm*: Detect revoked PlayReady devices from license responses +- *drm*: Auto-repack tracks with a residual encrypted sample entry +- *dl*: Extend --select-titles to movie listings +- Show git commit hash alongside the version +- Identify the running code by source hash +- *dl*: Add --speed-limit and serve global_speed_limit +- *providers*: Add TheTVDB v4 as a metadata provider +- *tracks*: Add subtitle.group_by to keep a language with its variants +- *temp*: Give each download task its own temp directory +- *dl*: Add -fsl/--forced-s-lang to filter forced subtitles by language +- *tracks*: Add language_priority to sort preferred languages first +- *vaults*: Support custom headers on API vault +- *dl*: Add tag_rules to conditionally swap the group tag +- *titles*: Add optional Episode.part, selected with -w s1e1.2 +- *enrich*: Fill in the original language from metadata providers +- *dl*: Allow --tvdb as an --enrich metadata source +- *dl*: Exclude languages with a - prefix on -l, -vl, -al, -sl and -fsl +- *metadata*: [**breaking**] Authoritative id resolver and keyless imdb provider +- *metadata*: [**breaking**] Replace animeapi with a first-class anilist provider +- *dl*: Add absolute episode numbers filled from tvdb absolute order +- *dl*: Add daily date-based episodes with tvdb air-date fill +- *api*: Stream download job progress over server-sent events +- *remote*: Accept serve.users keys in api-only mode and add client auth-header fallback +- *serve*: Gate server CDM licensing behind a per-key server_cdm opt-in +- *cli*: Add named color themes and rich-formatted help ### Bug Fixes @@ -36,23 +65,69 @@ This changelog is automatically generated using [git-cliff](https://git-cliff.or - *gitignore*: Add memory directory to .gitignore - *dl*: Prevent missing tracks error when excluding track types - *subs*: Stop segmented WebVTT merge from stripping formatting +- *proxies*: Show ExpressVPN country/location counts at startup +- *tracks*: Keep regional language variants distinct in exact selection +- *services*: Register modules in sys.modules so repo-cloned services can self-import and remove old benchmark that should have been deleted +- Skip decrypt on mislabeled-clear segments +- Harden ClearKey/CENC decryption correctness +- *import*: Stop losing side-loaded subtitles and blaming the manifest URL +- *ism*: Emit two stsd entries so sample_description_index 2 resolves +- *ism*: Repair synthesized init and verify fragments decrypted +- *core*: Strip trailing spaces and dots in template formatting to prevent WinError 3 +- *templates*: Use one separator style across a whole folder path +- *lang*: Surface orig in errors and flag the original audio track over the API +- *tracks*: Ignore leftover PIFF uuid when verifying decryption +- *tracks*: Size the no-proxy download session's connection pool +- *import*: Scope the merge-back and detect ISM protection +- *dl*: Honour --no-proxy-download for attachments +- *tools*: Pass -nostdin to all ffmpeg calls +- *subtitles*: Strip X-TIMESTAMP-MAP so WebVTT cues are not offset on conversion +- *downloader*: Decode content-encoded bodies instead of writing raw +- *downloader*: Surface failed downloads instead of returning silently +- *dl*: Warn when --enrich source is missing title, year, or language +- *api*: Carry air_date, absolute, anime and daily across the title wire +- *dl*: Treat audio muxed into the video as an available language +- *hls*: Take the key id from EXT-X-KEY when the PSSH carries none +- *dl*: Reset per-title language and range selection each iteration +- *subtitle*: Keep cue-less webvtt instead of failing the download +- *services*: Re-raise ClickException for better error handling in remote services cache +- *dash*: Resolve representations by adaptation set, not id alone +- *kv*: Search remote vaults that cannot list their tables ### Documentation - Restructure documentation into MkDocs Material site - Expand config reference and genericize service names +- *changelog*: Regenerate with git-cliff +- Correct DRM, muxing, and proxy references (+ case-insensitive proxy prefix fix) +- Cleanup and correct documentation +- *core*: Drop service tags from framework comments +- Cover the TVDB provider, episode ordering and provider order +- Split configuration reference into per-area pages +- *api*: Note the - language exclusion prefix in the OpenAPI spec ### Performance Improvements - *downloader*: Cut per-segment overhead, bound stalled connections +- Little improvements here and there +- *pipeline*: Fix pool starvation, unlock vault writes, prep mux +- Speed up the code hash with sha1 and a plain file reader ### Changes - *core*: [**breaking**] Bundle download_track args into DownloadContext +- *tracks*: Honour the best token and share one match rule + +### Maintenance + +- *changelog*: Update entries for new features and bug fixes +- *env*: Drop ML-Worker from the dependency check +- *changelog*: Update changelog ### Builds - *deps*: Cap pycaption below 2.2.27 +- *mypy*: Enable explicit_package_bases ## [5.3.0] - 2026-07-01 @@ -818,7 +893,7 @@ This changelog is automatically generated using [git-cliff](https://git-cliff.or - Reorganize Planned Features section in README for clarity - Improve track selection logic in dl.py -[unreleased]: https://github.com/unshackle-dl/unshackle/compare/5.3.0..HEAD +[5.4.0]: https://github.com/unshackle-dl/unshackle/compare/5.3.0..5.4.0 [5.3.0]: https://github.com/unshackle-dl/unshackle/compare/5.2.0..5.3.0 [5.2.0]: https://github.com/unshackle-dl/unshackle/compare/5.1.0..5.2.0 [5.1.0]: https://github.com/unshackle-dl/unshackle/compare/5.0.0..5.1.0 diff --git a/docs/dev/creating-a-service.md b/docs/dev/creating-a-service.md index 1c3d5f9b..91547c3a 100644 --- a/docs/dev/creating-a-service.md +++ b/docs/dev/creating-a-service.md @@ -99,6 +99,8 @@ Declared at class level to configure framework behaviour: | `VAULT_TAG` | `Optional[str]` | Overrides the key-vault namespace so sibling services can share one vault. Default `None` (use the service's own tag). | | `AUTH_METHODS` | `Optional[tuple[str, ...]]` | Auth methods accepted (`"cookies"` / `"credentials"`). When `None`, the REST `/services` endpoint infers them from `authenticate()`. | | `NO_SUBTITLES` | `bool` | Set `True` on a service with no subtitle tracks to skip subtitle handling entirely. | +| `ANIME` | `bool` | Set `True` when the catalog is anime, so metadata lookups prefer AniList. A title's own `anime` flag overrides it. | +| `DAILY` | `bool` | Set `True` when the catalog is daily/date-based (talk shows, news, sports), so episodes are named by air date. A title's own `daily` flag overrides it. | !!! note "`NO_SUBTITLES` is a convention, not a base-class attribute" `NO_SUBTITLES` is checked by `dl.py` via `hasattr` and is deliberately **not** @@ -240,7 +242,7 @@ use `self.request_input(prompt)`, never a bare `input()`. Under `serve` mode there is no local terminal, so a bare `input()` would hang the server waiting on stdin that never arrives; `request_input` instead relays the prompt to the remote client through the attached `InputBridge`. Locally it routes through the -shared Rich console (`console.input`) so the prompt renders correctly alongside +shared Rich console (`prompt_user`) so the prompt renders correctly alongside progress and log output. !!! warning "Key token caches by whatever varies per session" @@ -427,22 +429,22 @@ Each exposes `from_url(url, session=...)` (or `from_text(text, url)`) and then token. Some services deliver a **separate manifest per codec/range**. The base class -provides `_get_tracks_for_variants(title, fetch_fn)` to fan out over every +provides `get_tracks_for_variants(title, fetch_fn)` to fan out over every codec×range in the `TrackRequest`, including `HYBRID` (fetch HDR10 + DV and merge) and `--best-available` skip-on-error handling: ```python def get_tracks(self, title: Title_T) -> Tracks: - def _fetch_variant(title, codec, range_) -> Tracks: + def fetch_variant(title, codec, range_) -> Tracks: vcodec = "H265" if codec == Video.Codec.HEVC else "H264" - return self._fetch_dash_manifest(title, vcodec=vcodec, range_=range_) - return self._get_tracks_for_variants(title, _fetch_variant) + return self.fetch_dash_manifest(title, vcodec=vcodec, range_=range_) + return self.get_tracks_for_variants(title, fetch_variant) ``` After parsing, you may **correct** track metadata the manifest gets wrong: stamp the real `video.range`, fix odd audio channel counts, mark descriptive audio, add hand-built subtitles or an `Attachment` (e.g. cover art). Do *not* -filter for resolution/bitrate. See `_fetch_dash_manifest` in the EXAMPLE service +filter for resolution/bitrate. See `fetch_dash_manifest` in the EXAMPLE service for a thorough demonstration. !!! note "Flip HDR10 → HDR10+ by hand when you know the platform embeds it" diff --git a/docs/dev/rest-api/authentication.md b/docs/dev/rest-api/authentication.md index 6cf5b644..0f707182 100644 --- a/docs/dev/rest-api/authentication.md +++ b/docs/dev/rest-api/authentication.md @@ -108,6 +108,17 @@ X-Secret-Key: The value must exactly match one of the configured keys (`api_secret`, or one of the keys under `users`). There is no login step, no session cookie, and no token exchange for API access. The same static key is sent on each request. +!!! info "One route also accepts the key as a query parameter" + The job event stream, `GET /api/download/jobs/{job_id}/events`, also accepts the key in a `secret_key` query parameter. This is for browsers: the `EventSource` API cannot set request headers. The exception applies to that one route only, and it works in both server modes. + + ```javascript + new EventSource(`http://127.0.0.1:8786/api/download/jobs/${jobId}/events?secret_key=${key}`); + ``` + + The header always wins. If the request has an `X-Secret-Key` header, the server checks that header and ignores the query parameter, so a wrong header is rejected even when the query parameter holds a valid key. + + A key in a URL is easier to leak than a key in a header, because proxies and browser history can record it. Use the header wherever your client can set one. + === "curl" ```bash @@ -269,6 +280,7 @@ With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located | Item | Value | |---|---| | Auth header | `X-Secret-Key` | +| Query-parameter alternative | `?secret_key=` on `GET /api/download/jobs/{job_id}/events` only | | Default bind address | `127.0.0.1:8786` | | Exempt endpoint | `GET /api/health` | | Failure status | `401 Unauthorized` | @@ -287,12 +299,14 @@ With `--caddy`, unshackle also starts `caddy run` using the `Caddyfile` located The header and config are identical between modes, but the code that enforces them differs: -- **API-only server** (`--api-only`, or `--remote-only` which forces API-only). An `api_key_authentication` middleware runs on every request. It lets `GET /api/health` through unconditionally, then reads `X-Secret-Key`. An empty header returns the *"Secret Key is Empty."* body; a header that is not a configured user returns *"Secret Key is Invalid."*. In this mode the accepted-keys map contains exactly one entry: your `api_secret` (labelled `api_user`, with no devices). With `--no-key`, this middleware is simply not installed and the users map is empty. +- **API-only server** (`--api-only`, or `--remote-only` which forces API-only). An `api_key_authentication` middleware runs on every request. It lets `GET /api/health` through unconditionally, then reads `X-Secret-Key`. An empty header returns the *"Secret Key is Empty."* body; a header that is not a configured user returns *"Secret Key is Invalid."*. In this mode the accepted-keys map holds your `api_secret` (labelled `api_user`, with no devices) plus every key under `serve.users`, so a per-user key authenticates here exactly as it does on the integrated server, and that user's `services` allowlist applies as usual. With `--no-key`, this middleware is simply not installed and the users map is empty. - **Integrated server** (default). A `serve_authentication` middleware wraps pywidevine's authentication (and pyplayready's for `/playready*` paths), and the REST routes live in the same application, guarded by the same check. Here the accepted-keys map is your full `serve` config: `api_secret` (auto-added with access to all loaded devices) plus every entry under `users`, each with its declared `devices` / `playready_devices`. Because the API-only middleware guards *all* non-health paths, the Swagger UI at `/api/docs/` is also behind `X-Secret-Key` there. In `--remote-only` mode the Swagger UI is not mounted at all. +Both middlewares read the caller's key through the shared `request_secret_key()` helper in `unshackle/core/api/handlers.py`, which returns the `X-Secret-Key` header and falls back to the `secret_key` query parameter only when the matched route is the job event stream. The integrated server needs one extra branch for this: pywidevine's `authentication` is header-only, so `serve_authentication` compares the query-parameter key itself when the header is absent, and delegates to pywidevine in every other case. That is why a present-but-wrong header is rejected in both modes. + ### CORS and the `Authorization` header A `cors_middleware` runs on every response in both modes (and answers `OPTIONS` preflight requests with an empty response). It emits: diff --git a/docs/dev/rest-api/endpoints.md b/docs/dev/rest-api/endpoints.md index 05721df4..1f04afd9 100644 --- a/docs/dev/rest-api/endpoints.md +++ b/docs/dev/rest-api/endpoints.md @@ -223,6 +223,16 @@ Service-specific CLI options may also be passed as additional keys. Each serialized title carries `type` (`"episode"`, `"movie"`, or `"other"`), `name`, `id`, `language`, `description`, `date`, and `cover_url`. Episodes and movies add `year`; episodes additionally add `series_title`, `season`, and `number`. +These keys are sent only when the title carries them, so a title without them serializes exactly as before: + +| Key | Type | On | Meaning | +| --- | --- | --- | --- | +| `part` | integer | episode | Part index of a [split episode](../creating-a-service.md#split-episodes), counting from 1. | +| `air_date` | string | episode | ISO air date of [dated content](../../guide/downloading.md#daily-and-date-based-content). unshackle names the episode by date instead of `SxxExx`. | +| `absolute` | integer | episode | Absolute episode number across all seasons. | +| `daily` | boolean | episode | The episode is daily/date-based. Set from the title, or from the service's `DAILY` class attribute. | +| `anime` | boolean | episode, movie | The title is anime, so metadata lookups prefer AniList. Set from the title, or from the service's `ANIME` class attribute. | + !!! info "`part` on a split episode" A service that splits one episode into several separately playable videos reports each one as its own episode. Those titles share a `season` and `number` and add a `part` @@ -368,7 +378,7 @@ Create a download job. Requires `service` and `title_id`; every other field is a | `range` | string[] | `["SDR"]` | Dynamic range(s). | | `channels` | number | `null` | Audio channel count. | | `no_atmos` | boolean | `false` | Exclude Atmos tracks. | -| `wanted` | string[] | `[]` | Episode/season selectors. Accepts the part form, `"S01E01.2"`. | +| `wanted` | string[] | `[]` | Episode/season selectors. Accepts the part form, `"S01E01.2"`, and the air-date form, `"2026-08-11"` or `"2026-08-01:2026-08-31"`. | | `latest_episode` | boolean | `false` | Only the newest episode. | | `lang` / `v_lang` / `a_lang` / `s_lang` | string[] | `["orig"]` / `[]` / `[]` / `["all"]` | Language filters. | | `require_subs` | string[] | `[]` | Required subtitle languages. | @@ -392,9 +402,10 @@ Create a download job. Requires `service` and `title_id`; every other field is a | `best_available` | boolean | `false` | Fall back to best available. | | `repack` | boolean | `false` | Add REPACK tag. | | `tag` | string | `null` | Release group tag. | -| `tmdb_id` / `imdb_id` / `tvdb_id` / `animeapi_id` | - | `null` | External ID overrides for tagging. | +| `tmdb_id` / `imdb_id` / `tvdb_id` / `anilist_id` | - | `null` | External ID overrides. Each resolves its metadata directly instead of by a title search, and is used for tagging. Set `enrich` to also take the title, year and original language. Give at most one of `tmdb_id`, `imdb_id` and `tvdb_id`, since unshackle resolves the others from it. Sending two returns `400`. `anilist_id` still combines with one of them. `tmdb_id` and `tvdb_id` must be positive integers, `imdb_id` must look like `tt1375666`, and `anilist_id` must be a positive integer or a string like `mal:12345`, or the request returns `400`. An ID whose provider is unconfigured, such as `tmdb_id` with no `tmdb_api_key`, fails the job rather than returning `400`. `anilist_id` needs no key. | | `tvdb_order` | `official`, `dvd`, `absolute`, `alternate`, `regional` | `null` | Renumber episodes to a TVDB season order. Falls back to the `tvdb_order` config option. | -| `enrich` | boolean | `false` | Override title/year from external source. | +| `enrich` | boolean | `false` | Overwrite title, year and original language with the external source's. Needs one of `tmdb_id`, `imdb_id`, `tvdb_id` or `anilist_id`. Without one the job fails instead of returning `400`. | +| `daily` | boolean | `false` | Treat the title as daily/date-based content and fill missing episode air dates from TVDB. The fill needs `enrich` and a TVDB ID. An air date the service already set is kept. | | `output_dir` | string | `null` | Override output directory. | | `no_cache` / `reset_cache` | boolean | `false` | Title cache controls. | @@ -481,6 +492,73 @@ Fetch a single job with full details (equivalent to a `full=true` list entry). | `200` | - | Job returned. | | `404` | `JOB_NOT_FOUND` | No such job. | +### `GET /api/download/jobs/{job_id}/events` + +Stream the job's progress as [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events). Use this instead of polling `GET /api/download/jobs/{job_id}`. + +The response has the content type `text/event-stream`. Each event has this format: + +``` +event: +data: + +``` + +The `data` field of every event is the same job object that `GET /api/download/jobs/{job_id}` returns. + +| Event | When it is sent | +| --- | --- | +| `snapshot` | Immediately, as the first event. Gives the job's current state. | +| `status` | The job leaves the queue and starts to download. | +| `progress` | The worker's progress record changed. The server reads that record every 0.5 seconds, so you get at most two of these events each second, and none while the record stays the same. | +| `completed` | The job finished. The server then closes the stream. | +| `failed` | The job failed. The server then closes the stream. | +| `cancelled` | The job was cancelled. The server then closes the stream. | + +The server sends a `: keep-alive` comment each 15 seconds while the job is quiet, to keep proxies from closing an idle connection. Ignore these lines. + +If the job is already in a terminal state, the server sends the `snapshot` event, then the terminal event, and closes the stream immediately. + +!!! tip "Authentication from a browser" + `EventSource` cannot set headers. For this endpoint only, the server also accepts the key in the `secret_key` query parameter. This works in both server modes, the default integrated server and `--api-only`: + + ```javascript + const events = new EventSource( + `http://127.0.0.1:8786/api/download/jobs/${jobId}/events?secret_key=${key}` + ); + events.addEventListener("progress", (e) => console.log(JSON.parse(e.data).progress)); + events.addEventListener("completed", () => events.close()); + ``` + + The header always wins. Send the key in the query parameter *or* in the header, not both: if an `X-Secret-Key` header is present, the server checks that header and ignores the query parameter. See [Authentication](authentication.md#how-clients-present-credentials). + +=== "Request" + + ```bash + curl -N -H "X-Secret-Key: $KEY" \ + "http://127.0.0.1:8786/api/download/jobs/b0f7c8e2-.../events" + ``` + +=== "Response `200`" + + ``` + event: snapshot + data: {"job_id":"b0f7c8e2-...","status":"downloading","progress":42.5,...} + + event: progress + data: {"job_id":"b0f7c8e2-...","status":"downloading","progress":48.0,...} + + : keep-alive + + event: completed + data: {"job_id":"b0f7c8e2-...","status":"completed","progress":100.0,...} + ``` + +| Status | Error code | Meaning | +| --- | --- | --- | +| `200` | - | Event stream started. | +| `404` | `JOB_NOT_FOUND` | No such job. | + ### `DELETE /api/download/jobs/{job_id}` Cancel or remove a job. The behaviour depends on the job's current state: diff --git a/docs/dev/rest-api/index.md b/docs/dev/rest-api/index.md index d6f9c40c..5c701958 100644 --- a/docs/dev/rest-api/index.md +++ b/docs/dev/rest-api/index.md @@ -22,8 +22,9 @@ calls you can script against: - **Discovery**: list the services available on the server, run a service's search, and enumerate the titles and tracks behind a title ID. -- **Downloading**: submit a download as a background *job*, then poll its status and - live progress, cancel it, retry it, or bump it to the front of the queue. +- **Downloading**: submit a download as a background *job*, then follow its status and + live progress (by polling, or by subscribing to its Server-Sent Events stream), + cancel it, retry it, or bump it to the front of the queue. - **History and housekeeping**: read the log of finished jobs, inspect the server's effective (redacted) configuration, check external tool versions, and clear caches or temp files. @@ -161,6 +162,7 @@ request and response in full. | `GET` | `/api/download/jobs` | List jobs, with filtering and sorting. | | `POST` | `/api/download/jobs/clear-finished` | Remove all finished jobs. | | `GET` | `/api/download/jobs/{job_id}` | Full detail for one job. | +| `GET` | `/api/download/jobs/{job_id}/events` | Stream one job's progress as Server-Sent Events. | | `DELETE` | `/api/download/jobs/{job_id}` | Cancel a running job, or remove a finished one. | | `POST` | `/api/download/jobs/{job_id}/retry` | Re-queue a finished job's parameters as a new job. | | `POST` | `/api/download/jobs/{job_id}/priority` | Move a queued job to the front. | diff --git a/docs/dev/rest-api/job-lifecycle.md b/docs/dev/rest-api/job-lifecycle.md index 56b9c761..8aaa5e15 100644 --- a/docs/dev/rest-api/job-lifecycle.md +++ b/docs/dev/rest-api/job-lifecycle.md @@ -177,6 +177,11 @@ The full detail adds `started_time`, `completed_time`, `output_files`, the redac done ``` +!!! tip "Or subscribe instead of polling" + `GET /api/download/jobs/{job_id}/events` pushes the same job object as Server-Sent + Events, and closes the stream when the job ends. See + [Endpoints](endpoints.md#get-apidownloadjobsjob_idevents). + !!! note "What the percentage means" `progress` climbs from `0` toward `90` while tracks download, then the final stretch to `100` covers post-download work such as decryption cleanup, repackaging, and diff --git a/docs/dev/rest-api/quickstart.md b/docs/dev/rest-api/quickstart.md index da02a045..e06e1aff 100644 --- a/docs/dev/rest-api/quickstart.md +++ b/docs/dev/rest-api/quickstart.md @@ -310,7 +310,15 @@ done ``` !!! tip "Polling cadence" - A few seconds between polls is plenty. The server writes progress updates roughly twice a second internally, but there is no push/streaming channel; you read the current snapshot each time you ask. + A few seconds between polls is plenty. The server updates a job's progress roughly twice a second internally, and each poll reads the current snapshot. + +!!! tip "Or let the server push the updates" + `GET /api/download/jobs/{job_id}/events` streams the same job record as Server-Sent Events, and closes the stream when the job ends. It removes the polling loop above. See [Endpoints](endpoints.md#get-apidownloadjobsjob_idevents). + + ```bash + curl -N "$BASE/api/download/jobs/$JOB/events" \ + -H "X-Secret-Key: $SECRET" + ``` If a job ends in `failed`, the full record includes `error_message`, `error_code`, and (when the server was started with `--debug-api`) `error_traceback` and `worker_stderr` to help you diagnose it. Sensitive values in these fields are scrubbed before they are returned. diff --git a/docs/dev/rest-api/remote-sessions.md b/docs/dev/rest-api/remote-sessions.md index 56a89ecf..e68311dd 100644 --- a/docs/dev/rest-api/remote-sessions.md +++ b/docs/dev/rest-api/remote-sessions.md @@ -391,14 +391,15 @@ and base64-encoding, **excluding** `titles_*` files) and returns them under a Source: `unshackle/core/remote_service.py`. This is the canonical consumer of the session API and a good template for any client. -- **`RemoteClient._request`** sets `X-Secret-Key` and `User-Agent: unshackle/`, +- **`RemoteClient.request`** sets `X-Secret-Key` and `User-Agent: unshackle/`, uses a 120s timeout for `POST` and 30s for `GET`/`DELETE`, and treats any `status_code >= 400` as fatal: it logs `Server error []: ` and raises `SystemExit(1)`. - **Retries.** The download-side HTTP session mounts an adapter with `Retry(total=5, backoff_factor=0.2, status_forcelist=[429, 500, 502, 503, 504])`. - **Flow.** `authenticate()` → `create` (+ poll `prompt` every 2s up to a 600s - deadline, answering `pending_input` prompts) → `get_titles()` → `get_tracks()` + deadline, answering `pending_input` prompts; with stdin closed the client logs an + error and raises `SystemExit(1)` instead of posting an empty answer) → `get_titles()` → `get_tracks()` (merging returned `session_headers`/`session_cookies`, re-parsing `manifests`) → license via proxy or `server_cdm` → `close()` (`DELETE`, saving any returned `cache`). diff --git a/docs/dev/service-architecture.md b/docs/dev/service-architecture.md index 59598417..4819ffe4 100644 --- a/docs/dev/service-architecture.md +++ b/docs/dev/service-architecture.md @@ -64,6 +64,8 @@ Declared at the top of your service class, these are static descriptors: | `GEOFENCE` | `tuple[str, ...]` | ISO region codes the service requires; empty means no geofence. The first entry is treated as the "main region" for automatic proxy selection. | | `VAULT_TAG` | `Optional[str]` | Overrides the key-vault namespace; defaults to the service's own tag. | | `AUTH_METHODS` | `Optional[tuple[str, ...]]` | Accepted auth methods (`"cookies"` / `"credentials"`). When `None`, the REST `/services` endpoint infers them from `authenticate()`. | +| `ANIME` | `bool` | The catalog is anime, so metadata lookups prefer AniList. A title's own `anime` flag overrides it. | +| `DAILY` | `bool` | The catalog is daily/date-based, so episodes are named by air date. A title's own `daily` flag overrides it. | ### Required methods diff --git a/docs/guide/cli-reference.md b/docs/guide/cli-reference.md index 40cd3b55..4560c244 100644 --- a/docs/guide/cli-reference.md +++ b/docs/guide/cli-reference.md @@ -90,13 +90,13 @@ unshackle dl [OPTIONS] SERVICE [SERVICE ARGS...] | Flag | Default | Description | |---|---|---| -| `-l`, `--lang` | `orig` | Language(s) for **both** video and audio. `orig` = the title's original language; e.g. `orig,en`. | -| `-vl`, `--v-lang` | - | Video-only language (overrides `-l` for video). | -| `-al`, `--a-lang` | - | Audio-only language (overrides `-l` for audio). | -| `-sl`, `--s-lang` | `all` | Subtitle language(s). | +| `-l`, `--lang` | `orig` | Language(s) for **both** video and audio. `orig` = the title's original language; e.g. `orig,en`. A `-` prefix excludes, e.g. `all,-es`. | +| `-vl`, `--v-lang` | - | Video-only language (overrides `-l` for video). A `-` prefix excludes. | +| `-al`, `--a-lang` | - | Audio-only language (overrides `-l` for audio). A `-` prefix excludes. | +| `-sl`, `--s-lang` | `all` | Subtitle language(s). A `-` prefix excludes, e.g. `all,-es`. | | `--require-subs` | - | Required subtitle langs; keeps **all** subs only if these exist. **Cannot combine with `--s-lang`.** | | `-fs`, `--forced-subs` | off | Include forced subtitle tracks. | -| `-fsl`, `--forced-s-lang` | none | Language(s) wanted for forced subtitles; implies `-fs`. | +| `-fsl`, `--forced-s-lang` | none | Language(s) wanted for forced subtitles; implies `-fs`. A `-` prefix excludes. | | `--exact-lang` | off | Exact matching only: `-l es-419` matches `es-419`, not `es-ES`. Applies to selection and to sort order. | | `--sub-format` | - | Output subtitle format (`SRT`/`srt`, `VTT`/`webvtt`, `ASS`/`ssa`, `TTML`, `SMI`, ...), or `original` to keep the source format. | @@ -112,7 +112,7 @@ The special language tokens `orig`, `all`, and `best` are honoured everywhere a | Flag | Description | |---|---| -| `-w`, `--wanted` | Wanted episodes, e.g. `S01-S05,S07`, `S01E01-S02E03`. Supports exclusions with a leading `-` (e.g. `-S03`). For a [split episode](downloading.md#split-episodes), `.N` picks one part (`S01E01.2`), a range must stay inside the episode (`S01E01.1-S01E01.3`), and `S01E01` on its own takes every part. | +| `-w`, `--wanted` | Wanted episodes, e.g. `S01-S05,S07`, `S01E01-S02E03`. Supports exclusions with a leading `-` (e.g. `-S03`). For a [split episode](downloading.md#split-episodes), `.N` picks one part (`S01E01.2`), a range must stay inside the episode (`S01E01.1-S01E01.3`), and `S01E01` on its own takes every part. For [dated content](downloading.md#daily-and-date-based-content), a token can also be an ISO air date (`2026-08-11`) or a date range with a colon (`2026-08-01:2026-08-31`). | | `--select-titles` | Interactively select what to download: episodes of a series, or films when a title has more than one. **Cannot combine with `-w`.** | | `--latest-episode` | Download only the single most recent episode. | | `--list-titles` | List titles only; do not download. | @@ -150,13 +150,21 @@ Keep only certain track types, or skip certain track types. Attachments are alwa | Flag | Description | |---|---| -| `--tmdb` | TMDB ID (integer). | -| `--imdb` | IMDb ID, e.g. `tt1375666`. | -| `--tvdb` | TVDB ID (integer). Skips the series lookup that `--tvdb-order` would otherwise do. | -| `--animeapi` | AnimeAPI ID, e.g. `mal:12345` or `anilist:98765` (defaults to MAL). Back-fills TMDB/IMDb/TVDB. | -| `--enrich` | Override show title/year from the external source. **Requires** one of `--tmdb`, `--imdb`, or `--animeapi`. | +| `--tmdb` | TMDB ID (integer). Used for the tags. Skips the title search. `--enrich` reads it too. Needs `tmdb_api_key`. | +| `--imdb` | IMDb ID, e.g. `tt1375666`. Used for the tags. Skips the title search. `--enrich` reads it too. Needs no key. | +| `--tvdb` | TVDB ID (integer). Used for the tags. Skips the series lookup that `--tvdb-order` would otherwise do. `--enrich` reads it too. Needs `tvdb_api_key`. | +| `--anilist` | AniList ID (integer), e.g. `--anilist 21`. A MyAnimeList ID is accepted as `mal:12345` and resolved to the AniList entry. Used for the tags. Skips the title search. `--enrich` reads it too. Needs no key. | +| `--enrich` | Overwrite show title, year and original language with the external source's. **Requires** one of `--tmdb`, `--imdb`, `--tvdb`, or `--anilist`. | +| `--daily` | Treat the title as daily/date-based content and fill missing episode air dates from TVDB. The fill needs `--enrich` and a TVDB ID. See [Daily and date-based content](downloading.md#daily-and-date-based-content). | | `--tvdb-order` | Renumber episodes to a TVDB season order: `official` (aired), `dvd`, `absolute`, `alternate`, or `regional`. Needs `tvdb_api_key`. | +!!! note "One ID at a time" + `--tmdb`, `--imdb` and `--tvdb` cannot be combined. Give one and unshackle resolves the + others from it, writing all three to the tags. `--anilist` is outside that rule and pairs + with one of them, which is how you attach a western ID to an anime title. + An ID with no provider that can resolve it, such as `--tmdb` with no `tmdb_api_key`, fails + before the download starts. `--anilist` needs no key, so it never fails this check. + #### Episode ordering A service does not always number a series the way TVDB's aired order does. Some services list @@ -300,7 +308,7 @@ Prints a dependency table (Category / Tool / Status / Required / Purpose) showin | Category | Tools | |---|---| | Core | FFmpeg*, FFprobe*, MKVToolNix*, mkvpropedit* | -| DRM | shaka-packager*, mp4decrypt, ML-Worker | +| DRM | shaka-packager*, mp4decrypt | | HDR | dovi_tool, HDR10Plus_tool | | Subtitle | SubtitleEdit, CCExtractor | | Player | FFplay, MPV | @@ -316,6 +324,14 @@ unshackle env info Shows where the config was loaded from (or lists the candidate config locations if none was found) and prints a table of every configured directory (downloads, temp, cache, cookies, logs, exports, WVDs, PRDs, services, and more). +### `env theme` + +``` +unshackle env theme +``` + +Prints a sample of every available CLI theme: colour swatches, help text with option rows, a track listing with video/audio/subtitle entries, log lines, a progress bar, and the gradient pulse bar. The active theme is marked and each theme's aliases are listed. Set your choice with the [`theme`](../reference/configuration/misc.md#appearance) config key. + ### `env clear` Clear an environment directory. The directory is emptied and recreated, and the number of files and bytes freed is reported. @@ -329,6 +345,7 @@ Clear an environment directory. The directory is emptied and recreated, and the ```shell unshackle env check unshackle env info + unshackle env theme unshackle env clear cache unshackle env clear cache EXAMPLE # one service only unshackle env clear temp diff --git a/docs/guide/downloading.md b/docs/guide/downloading.md index 5a79f664..f6a41145 100644 --- a/docs/guide/downloading.md +++ b/docs/guide/downloading.md @@ -254,6 +254,47 @@ it is used. You can override each stream type independently: unshackle dl -al en EXAMPLE 81234567 ``` +### Excluding languages + +Put a `-` in front of a language to remove it from the selection. This works on `-l`, +`-vl`, `-al`, `-sl`, and `-fsl`, in the same way as it works on `-w` / `--wanted`. + +```shell title="Every subtitle language except Spanish" +unshackle dl -sl all,-es EXAMPLE 81234567 +``` + +The exclusions are subtracted from the languages the flag would otherwise select, and the +order of the values does not matter. If you give only exclusions, the flag keeps its own +default: `-sl -es` is the same as `-sl all,-es`, and `-l -es` is the same as `-l orig,-es`. + +```shell title="Only exclusions" +# every subtitle language except Spanish and French +unshackle dl -sl -es,-fr EXAMPLE 81234567 + +# the original audio language, unless it is Spanish +unshackle dl -l -es EXAMPLE 81234567 +``` + +`-vl` and `-al` have no default of their own: they cascade to `-l`. An override that names +a language replaces `-l` for that stream type, exclusions included. An override that gives +only exclusions adds them to the ones from `-l`. + +```shell title="Cascade" +# video keeps the Japanese track; only the audio drops Spanish +unshackle dl -l en,-es -vl ja EXAMPLE 81234567 +``` + +!!! note "Rules for exclusion tokens" + - `orig` is resolved per title, so `-sl -orig` drops the subtitles in the title's + original language. + - `--exact-lang` applies to exclusions with the same matching rules as selection: + `-es` removes exactly the tracks that `es` would select. Without it, `-es` also + removes `es-419` and `es-ES`. + - Tracks with no language tag are never excluded. + - `-all` is rejected. Name the languages you do not want instead. + - The comma form is the documented style. `-sl -es` also works, but a shell can read + a lone `-es` as another option. + ### Sort order `-l` and `-sl` **select** languages: naming some removes the rest. To keep every language @@ -296,6 +337,10 @@ every available subtitle language is downloaded. unshackle dl -sl en,es EXAMPLE 81234567 ``` +To keep the default and drop one language, use the `-` prefix described in +[Excluding languages](#excluding-languages). `-sl all,-es` keeps every subtitle language +except Spanish, forced Spanish subtitles included. + ### Requiring subtitles `--require-subs` takes a list of languages that **must** exist. If they all exist, *all* @@ -311,7 +356,8 @@ download on the presence of a specific subtitle track. Without this flag, forced subs are dropped from selection. - `-fsl` / `--forced-s-lang`: keep forced subtitles only in these languages (implies `-fs`). Works independently of `--s-lang`, so `-sl all -fsl en` grabs every full - subtitle but only the English forced track. + subtitle but only the English forced track. It accepts exclusions too: `-fsl all,-es` + keeps every forced subtitle except the Spanish one, and `-fsl -es` means the same. - `--sub-format`: set the output subtitle format, converting only when necessary. Accepts codec names/values and common aliases (`srt`, `vtt`, `ass`, `ssa`, `ttml`, etc.), or the literal `original` to keep the source format. @@ -381,6 +427,43 @@ with `-` to exclude it. unshackle dl -w S01,-S01E01.2 EXAMPLE 81234567 ``` +=== "Air dates" + + ```shell + # One day + unshackle dl -w 2026-08-11 EXAMPLE 81234567 + + # A range of days, colon separated + unshackle dl -w 2026-08-01:2026-08-31 EXAMPLE 81234567 + + # August, but not the 15th + unshackle dl -w 2026-08-01:2026-08-31,-2026-08-15 EXAMPLE 81234567 + ``` + +### Daily and date-based content + +Talk shows, news and sports have no official episode numbering, so unshackle names them +by air date. An episode that carries an air date is written as `Show.YYYY.MM.DD` instead of +`SxxExx`, the `{date}` token holds the ISO date, and the season folder becomes the year. + +A service that only carries this kind of content sets `DAILY = True` on its class, and a +service can set `air_date` on each episode itself. Add `--daily` to mark any other title +as date-based: + +```shell +unshackle dl --daily --tvdb 73871 --enrich EXAMPLE 81234567 +``` + +With `--enrich` and a TVDB ID, `--daily` fills in the air date of every episode that has +none. An air date the service already set is kept. Dates before 1970 and dates in the +future are skipped, because TVDB carries placeholder schedule dates for episodes that +have not aired. Without `--enrich` unshackle has no source to fill from, and says so. + +A dated episode answers to its air date in `-w`, as well as to its `SxxExx` key. A date +token is a plain ISO date (`2026-08-11`). A date range uses a colon (`2026-08-01:2026-08-31`), +because the dashes in a date are part of the date. A range cannot span more than 1000 days. +Date tokens and `SxxExx` tokens can be mixed in one `-w`. + ### Split episodes A few services split one episode into several separately playable videos. Where a service @@ -577,21 +660,98 @@ unshackle dl --skip-dl --export EXAMPLE 81234567 ## Metadata and tagging unshackle looks up metadata automatically, but you can override the identifiers used for -tagging and naming: +tagging and naming. An ID you give is authoritative: unshackle looks it up directly instead of +searching by title, and the ID stays in the tags. An ID only settles *which* title this is. +To also take that source's title, year and original language, add `--enrich`. Give at most one +of `--tmdb`, `--imdb` and `--tvdb`; unshackle works the rest out from it. To stop the automatic +lookups but keep the IDs you give, set +[`disable_metadata`](../reference/configuration/misc.md#external-api-keys) in your config. | Flag | Example | Purpose | | --- | --- | --- | -| `--tmdb` | `--tmdb 27205` | Use this TMDB ID instead of automatic lookup. | -| `--imdb` | `--imdb tt1375666` | Use this IMDb ID. | -| `--tvdb` | `--tvdb 73871` | Use this TVDB ID instead of looking the series up. | -| `--animeapi` | `--animeapi mal:12345` | Resolve via AnimeAPI (`mal:`/`anilist:` prefix; defaults to MAL). | -| `--enrich` | - | Override the show title and year from an external source. **Requires** one of `--tmdb`, `--imdb`, or `--animeapi`. | +| `--tmdb` | `--tmdb 27205` | Use this TMDB ID instead of an automatic search. Needs `tmdb_api_key`. | +| `--imdb` | `--imdb tt1375666` | Use this IMDb ID instead of an automatic search. Needs no key. | +| `--tvdb` | `--tvdb 73871` | Use this TVDB ID instead of looking the series up. Needs `tvdb_api_key`. | +| `--anilist` | `--anilist 21` | Use this AniList ID instead of an automatic search. `mal:12345` is also accepted. Needs no key. | +| `--enrich` | - | Overwrite the show title, year and original language with the external source's. **Requires** one of `--tmdb`, `--imdb`, `--tvdb`, or `--anilist`. | | `--tvdb-order` | `--tvdb-order dvd` | Renumber episodes to a TVDB season order. Needs `tvdb_api_key`. | -```shell title="Force the right IMDb match and enrich the title" +```shell title="Force the right IMDb match, and keep the service's own naming" +unshackle dl --imdb tt1375666 EXAMPLE 81234567 +``` + +```shell title="Force the match and take IMDb's title, year and language too" unshackle dl --imdb tt1375666 --enrich EXAMPLE 81234567 ``` +### What each metadata provider supplies + +Providers differ in what they answer with, so the fields `--enrich` can replace depend on +which ID you gave. Every provider except `imdb` only runs when its key is configured: + +| Provider | Config key | Title and year | Original language | External IDs it returns | +| --- | --- | --- | --- | --- | +| TMDB | `tmdb_api_key` | yes | yes, alpha-2 such as `ko` | IMDb, TMDB, TVDB | +| TVDB | `tvdb_api_key` | yes | yes, alpha-3 such as `kor` | TMDB, TVDB, sometimes IMDb | +| OMDb | `omdb_api_key` | yes | yes, as an English name such as `Korean` | IMDb | +| IMDb | *(no key needed)* | yes | yes, alpha-2 such as `ko` | IMDb | +| SIMKL | `simkl_client_id` | yes | no, it publishes a country and no language | IMDb, TMDB, TVDB | +| AniList | *(no key needed)* | yes | yes, worked out from the country of origin, such as `ja` | AniList | + +Whatever the tag looks like, unshackle normalises it before use, so `ko`, `kor` and `Korean` +all end up as the same language. + +!!! note "AniList only answers for anime" + It is last in the default provider order and returns nothing for a title that is not + anime. The only cost is one search that misses. Put `anilist` earlier in + [`metadata_providers`](../reference/configuration/misc.md#external-api-keys) if you mostly + download anime. Which of the three AniList title variants is used is set by + [`anilist_title_language`](../reference/configuration/misc.md#external-api-keys). + + A service can also mark its titles as anime. For those titles unshackle tries AniList + first. When AniList has no match, it falls through to the normal order. + +Each ID goes to the providers that read that kind of ID, in +[`metadata_providers`](../reference/configuration/misc.md#external-api-keys) order: + +- `--tmdb` reads TMDB. +- `--imdb` reads IMDb, then falls back to OMDb. IMDb needs no key, so this path works out of + the box, and an OMDb key only matters when IMDb has no answer. +- `--tvdb` reads TVDB. It needs `tvdb_api_key`. +- `--anilist` reads AniList, which needs no key. It supplies the title, the year and the + original language, and writes an AniList tag. It knows no TMDB, IMDB or TVDB ID, so pair it + with `--tmdb`, `--imdb` or `--tvdb` when you want those tags as well. + +Give **one** of `--tmdb`, `--imdb` and `--tvdb`. They cannot be combined, because one ID does +the job on its own: unshackle resolves the other two from it and writes all three to the tags. +Passing two is an error rather than a silent choice between them. `--anilist` is the exception +and still pairs with one of the three, which is how you tag an anime title with a western ID. + +The ID you give must have a provider that can actually resolve it, so unshackle checks that +before it downloads anything. `--tmdb` without `tmdb_api_key`, or an ID whose providers your +`metadata_providers` list leaves out, fails immediately with a message naming what to set. +`--imdb` needs no key, since the `imdb` provider is keyless and in the default order. + +With `--enrich`, the title, year and original language are all replaced with the external +source's values, whether or not the service already filled them in. A field the source does +not answer with is left alone, and unshackle logs which fields those were, so a provider with +a thin record cannot blank out what the service told you. + +`--enrich` also fills in the absolute episode number of each episode from TVDB's absolute +order, for any series that has one and where the service did not supply it. Anime is the +usual beneficiary, but no part of this is limited to anime. This only adds the +[`{absolute}`](../reference/configuration/output.md#output_template) naming variable. The +season and episode numbers are never changed. + +With `--daily`, `--enrich` also fills in the air date of each episode from TVDB. See +[Daily and date-based content](#daily-and-date-based-content). + +!!! warning "`--enrich` replaces the original language, which affects more than the filename" + Track selection reads the original language, so replacing it changes which audio is + treated as the original. That is the point when a service mislabels it, but it means a + wrong ID can pick the wrong audio track and not only write a wrong name. Without + `--enrich` the service's own value is kept and track selection is untouched. + ### Episode ordering A service does not always number a series the way TVDB's aired order does. Some services list @@ -637,14 +797,15 @@ authoritative list. | `--range` | `-r` | Color range(s); default `SDR`. | | `--channels` | `-c` | Audio channel layout. | | `--noatmos` | `-naa` | Exclude Atmos audio. | -| `--lang` | `-l` | Video + audio language(s); default `orig`. | -| `--a-lang` | `-al` | Audio-only language override. | -| `--v-lang` | `-vl` | Video-only language override. | -| `--s-lang` | `-sl` | Subtitle language(s); default `all`. | +| `--lang` | `-l` | Video + audio language(s); default `orig`. `-` excludes, e.g. `all,-es`. | +| `--a-lang` | `-al` | Audio-only language override. `-` excludes. | +| `--v-lang` | `-vl` | Video-only language override. `-` excludes. | +| `--s-lang` | `-sl` | Subtitle language(s); default `all`. `-` excludes, e.g. `all,-es`. | | `--forced-subs` | `-fs` | Include forced subtitles. | -| `--forced-s-lang` | `-fsl` | Forced subtitle language(s); implies `-fs`. | +| `--forced-s-lang` | `-fsl` | Forced subtitle language(s); implies `-fs`. `-` excludes. | | `--sub-format` | | Output subtitle format. | -| `--wanted` | `-w` | Episode/season range. | +| `--wanted` | `-w` | Episode/season range, or an air date. | +| `--daily` | | Fill missing air dates from TVDB during `--enrich`. | | `--select-titles` | | Interactively pick episodes or films. | | `--latest-episode` | | Only the newest episode. | | `--video-only` / `--audio-only` / `--subs-only` | `-V` / `-A` / `-S` | Restrict track types. | @@ -658,5 +819,5 @@ authoritative list. | `--list` / `--list-titles` / `--skip-dl` | | Dry runs. | | `--cdm-only` / `--vaults-only` | | Key source control. | | `--export` | | Export track info and keys to JSON. | -| `--tmdb` / `--imdb` / `--tvdb` / `--animeapi` / `--enrich` | | Metadata overrides. | +| `--tmdb` / `--imdb` / `--tvdb` / `--anilist` / `--enrich` | | Metadata overrides. | | `--tvdb-order` | | Renumber episodes to a TVDB season order. | diff --git a/docs/guide/output-and-naming.md b/docs/guide/output-and-naming.md index abaed437..6a162529 100644 --- a/docs/guide/output-and-naming.md +++ b/docs/guide/output-and-naming.md @@ -127,6 +127,7 @@ Every variable below is valid in both output and folder templates. Values are de | `season_episode` | Combined | `S01E05` | | `episode_name` | Episode title | `Pilot` | | `part` | Part index of a [split episode](#split-episodes), empty otherwise | `2` | +| `absolute` | Absolute episode number, zero-padded to 3 digits, empty when unknown | `007` | | `date` | ISO air date for daily/dated content | `2024-06-01` | !!! note "Daily & sports content" diff --git a/docs/guide/subtitles.md b/docs/guide/subtitles.md index 96159d23..8efc008a 100644 --- a/docs/guide/subtitles.md +++ b/docs/guide/subtitles.md @@ -67,10 +67,10 @@ Subtitle selection happens through `dl` flags. The most important is language se | Flag | Purpose | | --- | --- | -| `-sl`, `--s-lang` | Language(s) wanted for subtitles. Defaults to `all`. | +| `-sl`, `--s-lang` | Language(s) wanted for subtitles. Defaults to `all`. A `-` prefix excludes, e.g. `all,-es`. | | `--require-subs` | Require these languages to exist; if present, download **all** subtitles. Cannot be combined with `--s-lang`. | | `-fs`, `--forced-subs` | Include forced subtitle tracks (excluded by default). | -| `-fsl`, `--forced-s-lang` | Language(s) wanted for forced subtitles; implies `-fs`. | +| `-fsl`, `--forced-s-lang` | Language(s) wanted for forced subtitles; implies `-fs`. A `-` prefix excludes. | | `--exact-lang` | Exact language matching, with no regional variants. | | `-S`, `--subs-only` | Download only subtitle tracks. | | `-ns`, `--no-subs` | Do not download subtitle tracks at all. | diff --git a/docs/reference/configuration/download.md b/docs/reference/configuration/download.md index b85b27f3..c66547c0 100644 --- a/docs/reference/configuration/download.md +++ b/docs/reference/configuration/download.md @@ -18,7 +18,7 @@ after the flag's internal destination rather than its visible name; set these ex | `--list` | `list` | | `--tmdb` | `tmdb_id` | | `--imdb` | `imdb_id` | -| `--animeapi` | `animeapi_id` | +| `--anilist` | `anilist_id` | | `-naa` / `--noatmos` | `no_atmos` | | `-o` / `--output` | `output_dir` | | `--cdm-only` / `--vaults-only` | `cdm_only` | @@ -31,9 +31,9 @@ Common keys (a useful subset; every `dl` flag works): | Key | Type | Default | Sets | |-----|------|---------|------| -| `lang` | list | `["orig"]` | Video/audio language(s); `orig` = original language. | -| `a_lang` / `v_lang` | list | `[]` | Audio- / video-only language override. | -| `s_lang` | list | `["all"]` | Subtitle language(s). | +| `lang` | list | `["orig"]` | Video/audio language(s); `orig` = original language. A `-` prefix excludes, e.g. `[all, -es]`. | +| `a_lang` / `v_lang` | list | `[]` | Audio- / video-only language override. A `-` prefix excludes. | +| `s_lang` | list | `["all"]` | Subtitle language(s). A `-` prefix excludes, e.g. `[all, -es]`. | | `quality` | list | `[]` (best) | Resolution(s), e.g. `[1080]`. | | `vcodec` | list | `[]` (any) | Video codec(s), e.g. `[H265]`. | | `acodec` | list | `[]` (any) | Audio codec(s), e.g. `[EC3]`. | @@ -41,7 +41,7 @@ Common keys (a useful subset; every `dl` flag works): | `channels` | float | *(unset)* | Audio channels, e.g. `6` for 5.1. | | `sub_format` | str | *(unset)* | Convert subtitles to this format (`srt`, `vtt`, `original`, ...). | | `forced_subs` | bool | `false` | Include forced subtitle tracks. | -| `forced_s_lang` | list | `[]` | Forced subtitle language(s); implies `forced_subs`. | +| `forced_s_lang` | list | `[]` | Forced subtitle language(s); implies `forced_subs`. A `-` prefix excludes. | | `no_subs` / `no_audio` / `no_chapters` | bool | `false` | Skip that track type. | | `downloads` | int | `1` | Tracks downloaded concurrently. | | `workers` | int | *(downloader default)* | Threads per track. | @@ -61,6 +61,18 @@ dl: downloads: 2 ``` +!!! note "Language exclusions in YAML" + The language keys accept the same `-` exclusion tokens as the command line. Give them + as a list or as one comma-separated string. + + ```yaml + dl: + s_lang: [all, -es] + a_lang: "orig,-es" + ``` + + See [Excluding languages](../../guide/downloading.md#excluding-languages) for the rules. + !!! note "Per-service overrides" A `dl` block nested under a service in [`services`](services.md) takes the same keys and applies to that service only. diff --git a/docs/reference/configuration/index.md b/docs/reference/configuration/index.md index c435059a..eff698ef 100644 --- a/docs/reference/configuration/index.md +++ b/docs/reference/configuration/index.md @@ -36,7 +36,8 @@ Where the file lives, how unshackle discovers it, and how to edit it: | [DRM & CDM](drm.md) | `cdm`, `remote_cdm`, `decryption` | | [Network & proxy](network.md) | `network`, `headers`, `proxy_providers` | | [Key vaults](vaults.md) | `key_vaults`, `vault_timeout` | -| [External API keys](misc.md#external-api-keys) | `imdb_api_enabled`, `omdb_api_key`, `tmdb_api_key`, `tvdb_api_key`, `tvdb_pin`, `tvdb_order`, `simkl_client_id`, `metadata_providers`, `decrypt_labs_api_key`, `ipinfo_api_key` | +| [External API keys](misc.md#external-api-keys) | `omdb_api_key`, `tmdb_api_key`, `tvdb_api_key`, `tvdb_pin`, `tvdb_order`, `simkl_client_id`, `metadata_providers`, `anilist_title_language`, `disable_metadata`, `decrypt_labs_api_key`, `ipinfo_api_key` | | [Caching & updates](misc.md#caching-updates) | `title_cache_enabled`, `title_cache_time`, `title_cache_max_retention`, `update_checks`, `update_check_interval` | +| [Appearance](misc.md#appearance) | `theme` | | [Logging, privacy & debug](misc.md#logging-privacy-debug) | `redact_paths`, `debug`, `debug_keys`, `debug_requests`, `set_terminal_bg` | | [Deprecated & removed](misc.md#deprecated-removed-keys) | `curl_impersonate`, `downloader`, `scene_naming` | diff --git a/docs/reference/configuration/misc.md b/docs/reference/configuration/misc.md index ce3b5c28..5e989b67 100644 --- a/docs/reference/configuration/misc.md +++ b/docs/reference/configuration/misc.md @@ -4,18 +4,21 @@ Smaller configuration areas: external metadata API keys, caching and update chec ## External API keys { #external-api-keys } -All are unset by default and enable optional metadata/geolocation features. +All are unset by default and enable optional metadata/geolocation features. For which fields +each metadata provider can actually answer with, see +[What each metadata provider supplies](../../guide/downloading.md#what-each-metadata-provider-supplies). | Key | Type | Default | Description | |-----|------|---------|-------------| -| `imdb_api_enabled` | bool | `false` | Use the free IMDxAPI (api.tiffara.com) metadata provider, which needs no API key. Off by default since the site has been unreliable. | -| `omdb_api_key` | str | `""` | [OMDb](https://www.omdbapi.com/) API key for IMDb metadata lookups; a more reliable alternative to IMDxAPI. Free keys are available on the OMDb site. | +| `omdb_api_key` | str | `""` | [OMDb](https://www.omdbapi.com/) API key for IMDb metadata lookups; a fallback for when the keyless `imdb` provider has no answer. Free keys are available on the OMDb site. | | `tmdb_api_key` | str | `""` | TMDB API key for metadata enrichment and external-ID tags. | | `simkl_client_id` | str | `""` | SIMKL client ID for metadata lookups; an alternative/fallback source to TMDB. | -| `tvdb_api_key` | str | `""` | [TheTVDB v4](https://thetvdb.com/api-information) API key; a fallback source to TMDB, strongest on TV series. Free keys are available on the TVDB site. | +| `tvdb_api_key` | str | `""` | [TheTVDB v4](https://thetvdb.com/api-information) API key; a fallback source to TMDB, strongest on TV series. Also the source `--enrich --tvdb` reads. Free keys are available on the TVDB site. | | `tvdb_pin` | str | `""` | Subscriber PIN, only needed for a user-supported TVDB key. Leave empty for a normal project key. | | `tvdb_order` | str | `""` | Default for `--tvdb-order`: renumber episodes to a TVDB season order (`official`, `dvd`, `absolute`, `alternate`, `regional`). Empty keeps the numbering the service gives. | -| `metadata_providers` | list | *(see below)* | Metadata providers to use, in the order they are tried. | +| `metadata_providers` | list or map | *(see below)* | Metadata providers to use, in the order they are tried. | +| `anilist_title_language` | str | `english` | Which AniList title variant to use: `english`, `romaji`, or `native`. | +| `disable_metadata` | bool | `false` | Set to `true` to stop all automatic metadata lookups. An ID you give yourself is still looked up. | | `decrypt_labs_api_key` | str | `""` | Global Decrypt Labs API key (used by remote CDM / vault). | | `ipinfo_api_key` | str | `""` | ipinfo.io API key for IP/region lookups. | @@ -31,12 +34,77 @@ All are unset by default and enable optional metadata/geolocation features. ignored with a warning. Leave it unset for the default order: ```yaml - metadata_providers: [imdbapi, omdb, simkl, tmdb, tvdb] + metadata_providers: [imdb, omdb, simkl, tmdb, tvdb, anilist] ``` - Whatever the order, a provider is skipped when its API key is missing (or, for - `imdbapi`, when `imdb_api_enabled` is `false`). Available names are - `imdbapi`, `omdb`, `simkl`, `tmdb`, and `tvdb`. + Whatever the order, unshackle skips a provider that has no API key. `imdb` and + `anilist` need no key, so they are always available. Available names are `imdb`, `omdb`, + `simkl`, `tmdb`, `tvdb`, and `anilist`. The old name `imdbapi` still works and is read as + `imdb`. + + `anilist` is last in the default order. It only answers for anime and returns nothing for + anything else, so it is harmless there. Move it earlier if you mostly download anime. + + The order applies to a title search and to a direct lookup of an ID you give with + `--tmdb`, `--imdb`, `--tvdb`, or `--anilist`. A supplied ID always goes to the providers + that read that kind of ID, and it never falls back to a title search. + +!!! note "Which AniList title `anilist_title_language` picks" + AniList holds up to three titles for a work: the English one, the romaji transliteration, + and the native one. This option says which of them unshackle uses: + + ```yaml + anilist_title_language: romaji + ``` + + A work does not always have all three. When the one you asked for is unset on AniList's + side, unshackle falls back in the order english → romaji → native. It takes the first one + that is set, so you always get a title. + +!!! note "What `disable_metadata` turns off" + With `disable_metadata: true`, unshackle never contacts a metadata provider on its own. + The title search is dead, and that includes the keyless `imdb` provider. Titles, years and + languages then come only from the service. + + ```yaml + disable_metadata: true + ``` + + An ID you give with `--tmdb`, `--imdb`, `--tvdb`, or `--anilist` is your permission to use + that ID. That lookup still works. unshackle looks the ID up directly, cross references the other + IDs from it, and `--enrich` still reads that source. Only the lookups you did not ask for + are stopped. + +!!! note "The `imdb` provider needs no key" + It replaces the earlier `imdbapi` provider, which read a third-party mirror + (api.tiffara.com) and needed the `imdb_api_enabled` switch. That option is gone: delete it + from your config, where it is now ignored. + +!!! tip "A different order per title kind" + Give a map instead of one list when the best source differs by kind. TVDB is strongest on + series, TMDB on films. There are two kinds, `tv` and `movie`, and this example sets both + of them and names every provider, so nothing is left to the default: + + ```yaml + metadata_providers: + tv: [tvdb, tmdb, simkl, imdb, omdb] + movie: [tmdb, imdb, omdb, simkl, tvdb] + ``` + + Each list is independent: it both orders and filters that kind on its own, so a provider + you leave out of `tv` is still used for `movie`. Here TVDB is tried first for a series and + last for a film. + + You do not need to give both keys, or a full list in each. A kind you leave out uses the + default order: + + ```yaml + metadata_providers: + tv: [tvdb, tmdb] # movies keep the default order + ``` + + `tv` covers every episode and `movie` covers every film. unshackle only looks up + metadata for those two kinds, so any other key, such as `anime` or `music`, is ignored. !!! note "`ipinfo_api_key` never touches your service sessions" The token is only ever sent to `api.ipinfo.io` as a per-request `Authorization` header; it @@ -73,6 +141,14 @@ calls. --- +## Appearance { #appearance } + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `theme` | str | `catppuccin-mocha` | Named colour theme for all CLI output: banner, logs, progress bars, and help screens. One of `catppuccin-mocha` (aliases `default`, `catppuccin`, `mocha`), `dracula`, `nord`, `gruvbox`, `one-dark` (alias `onedark`), `tokyo-night` (alias `tokyonight`), `monokai`, `flexoki`, `solarized-dark` (alias `solarized`). Case-insensitive; underscores work in place of hyphens. Unknown names fall back to the default with a warning. Preview them all with `unshackle env theme`. | + +--- + ## Logging, privacy & debug { #logging-privacy-debug } | Key | Type | Default | Description | @@ -107,3 +183,4 @@ calls. | `curl_impersonate` | **Deprecated** → use [`network`](network.md) | Emits a `DeprecationWarning`; still honoured only if `network` is absent. | | `downloader` | **Deprecated** | Any value other than `"requests"` emits a `DeprecationWarning`; the value is otherwise ignored (the unified requests downloader is always used). | | `scene_naming` | **Removed** | If set to any non-null value, unshackle exits with an error directing you to configure [`output_template`](output.md) instead. | +| `imdb_api_enabled` | **Removed** | Ignored. It gated the old `imdbapi` provider, which the keyless [`imdb`](#external-api-keys) provider replaces. Safe to delete from your config. | diff --git a/docs/reference/configuration/output.md b/docs/reference/configuration/output.md index 2a58d909..9623a3f2 100644 --- a/docs/reference/configuration/output.md +++ b/docs/reference/configuration/output.md @@ -35,7 +35,7 @@ folder-kind keys. ??? example "All valid template variables" `title`, `year`, `season`, `episode`, `season_episode`, `episode_name`, `part`, - `date`, `quality`, `resolution`, `source`, `tag`, `track_number`, `artist`, + `absolute`, `date`, `quality`, `resolution`, `source`, `tag`, `track_number`, `artist`, `album_artist`, `album`, `disc`, `track_total`, `disc_total`, `release_type`, `genre`, `explicit`, `isrc`, `upc`, `label`, `audio`, `audio_channels`, `audio_full`, `atmos`, `dual`, `multi`, `dubbed`, `video`, `hdr`, `hfr`, `edition`, `repack`, `lang_tag`, @@ -51,6 +51,14 @@ folder-kind keys. `{season_episode}`, and note that it is always empty in a folder template, since folders do not carry the part. +!!! tip "`{absolute}` counts episodes across seasons" + `absolute` holds the absolute episode number, zero-padded to 3 digits (`007`), and is + empty on titles that have none. It is added to the name and never replaces `{season}` + or `{episode}`. Any series with a TVDB absolute order can use it, anime most often. + Services can supply it, and + [`--enrich`](../../guide/downloading.md#metadata-and-tagging) fills it in from TVDB. + Use `{absolute?}` so the name stays clean when it is unknown. + ## Tagging & naming keys | Key | Type | Default | Description | diff --git a/docs/reference/configuration/services.md b/docs/reference/configuration/services.md index 3ba1043a..caf3756b 100644 --- a/docs/reference/configuration/services.md +++ b/docs/reference/configuration/services.md @@ -82,11 +82,22 @@ keep auth tokens in localStorage rather than in HTTP cookies need. Extraction is Definitions of remote unshackle service servers, used by the `--remote` mode. Each entry is named by you (pick it with `--server`, or omit that flag when only one is configured) and gives -the server's `url` (required), an optional `api_key` sent as `X-Secret-Key`, an optional -`server_cdm` boolean, and an optional `services` sub-dict of per-service local overrides such -as `title_map`. In `--remote` mode unshackle turns the server's service list into synthetic CLI +the server's `url` (required), an optional `api_key`, an optional `auth_headers` list, an +optional `server_cdm` boolean, and an optional `services` sub-dict of per-service local +overrides such as `title_map`. + +`auth_headers` lists extra header names to send the API key in, tried before the defaults +`X-Secret-Key` and `X-Api-Key`, which are always appended as fallbacks. unshackle sends the +first name; if the server answers `401`, it retries the same request with the next name, and +keeps the one that works for the rest of the session. Names you list keep your spelling and are +not repeated in the fallbacks, so `auth_headers: ["Authorization", "x-secret-key"]` is tried as +`Authorization`, `x-secret-key`, `X-Api-Key`. + +In `--remote` mode unshackle turns the server's service list into synthetic CLI commands that run against it, falling back to the tags in that `services` sub-dict when the -list cannot be fetched. See [remote sessions](../../dev/rest-api/remote-sessions.md) for the +list cannot be fetched. Each synthetic command carries the server-side service's options and +documentation, so `unshackle dl --remote -h` shows the same help text as it does on the +server. See [remote sessions](../../dev/rest-api/remote-sessions.md) for the full setup. ## `serve` @@ -117,6 +128,13 @@ Each entry under `users` is keyed by that user's API key and may set its own `se `username` used as the log label for that key (defaults to a truncated form of the key). A user with no `playready_devices` key gets no PlayReady access at all, not the global list. +`server_cdm` decides whether the server runs the CDM licensing for that key. It is `false` +unless the entry sets it, so a remote client configured with `server_cdm: true` is told to +license with its own local CDM instead, and a client that asks anyway gets a `FORBIDDEN` +error. Because a download job always licenses with the server's CDM, a key without +`server_cdm` also cannot submit or retry `/api/download` jobs. Keys that have no `users` +entry, such as `api_secret`, keep server CDM access. + ```yaml serve: api_secret: change-me @@ -128,6 +146,8 @@ serve: users: a1b2c3d4: # this user's API key services: [EXAMPLE1] # may only use EXAMPLE1 + e5f6a7b8: + server_cdm: true # this key may have the server do the licensing ``` !!! note "`dl` keys inside `serve`" diff --git a/pyproject.toml b/pyproject.toml index f6379265..d144c355 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "unshackle" -version = "5.3.0" +version = "5.5.0" description = "Modular Movie, TV, and Music Archival Software." authors = [{ name = "unshackle team" }] requires-python = ">=3.11,<3.15" @@ -64,12 +64,12 @@ dependencies = [ "pycountry>=24.6.1", "language-data>=1.4.0", "wasmtime>=41.0.0", - "animeapi-py>=0.6.0", "rnet>=2.4.2", "bandit>=1.9.4", "jsonpath-ng>=1.8.0", "mutagen>=1.47.0,<2", "defusedxml>=0.7.1", + "rich-click>=1.9.8", ] [project.urls] diff --git a/scripts/bench_downloader.py b/scripts/bench_downloader.py index a18eaf63..67834417 100644 --- a/scripts/bench_downloader.py +++ b/scripts/bench_downloader.py @@ -464,7 +464,7 @@ def _procs_notice(procs: int, n_urls: int, session: Optional[Any]) -> None: f"[yellow]--procs {procs} will NOT engage: {n_urls} segments < MP_MIN_SEGMENTS " f"({dl_mod.MP_MIN_SEGMENTS}); the single-process path runs instead[/]" ) - elif dl_mod._build_session_spec(session) is None: + elif dl_mod.build_session_spec(session) is None: console.print( f"[yellow]--procs {procs} will NOT engage: session not rebuildable in child processes; " f"the single-process path runs instead[/]" diff --git a/scripts/live_segmentbase_ab.py b/scripts/live_segmentbase_ab.py index 3159fb8c..e38d284b 100644 --- a/scripts/live_segmentbase_ab.py +++ b/scripts/live_segmentbase_ab.py @@ -2,7 +2,7 @@ Drives the real DASH pipeline against public SegmentBase test manifests and reports wall time, size, throughput and output validity (ffprobe). On branches that have the -single-URL collapse (DASH._collapsible_single_url) it also reports whether the +single-URL collapse (DASH.collapsible_single_url) it also reports whether the collapse fired; on checkouts without it (e.g. dev) the same script runs the classic per-segment + merge path, so running it from both checkouts gives an A/B comparison: @@ -39,7 +39,7 @@ guard_results: list[bool] = [] def install_guard_spy() -> bool: """Record collapse decisions when this checkout has the guard; no-op otherwise.""" - orig = getattr(DASH, "_collapsible_single_url", None) + orig = getattr(DASH, "collapsible_single_url", None) if orig is None: return False @@ -48,7 +48,7 @@ def install_guard_spy() -> bool: guard_results.append(result) return result - DASH._collapsible_single_url = staticmethod(spy) # type: ignore[method-assign] + DASH.collapsible_single_url = staticmethod(spy) # type: ignore[method-assign] return True diff --git a/tests/core/test_adaptive_workers.py b/tests/core/test_adaptive_workers.py index 9a830143..1f45aad6 100644 --- a/tests/core/test_adaptive_workers.py +++ b/tests/core/test_adaptive_workers.py @@ -15,11 +15,11 @@ from unshackle.core.downloaders.requests import ( TAIL_BOOST_MIN_SEGMENT_SIZE, TAIL_BOOST_PART_SIZE, AdaptiveWorkerController, - _has_range_header, - _plan_tail_parts, - _split_ranges, - _tail_boost_engages, + has_range_header, + plan_tail_parts, requests, + split_ranges, + tail_boost_engages, ) from unshackle.core.tracks.track import DownloadContext @@ -179,7 +179,7 @@ def test_tail_guard_still_halves_on_error_burst() -> None: def test_plan_tail_parts_full_coverage_no_gaps() -> None: size = 20 * 1024 * 1024 - parts = _plan_tail_parts(size, spare=8) + parts = plan_tail_parts(size, spare=8) assert parts, "expected a split for a large segment with spare capacity" assert parts[0][0] == 0 assert parts[-1][1] == size - 1 @@ -191,27 +191,27 @@ def test_plan_tail_parts_full_coverage_no_gaps() -> None: def test_plan_tail_parts_respects_spare_cap() -> None: size = 100 * 1024 * 1024 # would want ~25 parts at 4MB, but spare caps it - parts = _plan_tail_parts(size, spare=3) + parts = plan_tail_parts(size, spare=3) assert len(parts) <= 3 assert parts[0][0] == 0 and parts[-1][1] == size - 1 def test_plan_tail_parts_part_size_bounded() -> None: size = 40 * 1024 * 1024 - parts = _plan_tail_parts(size, spare=16) + parts = plan_tail_parts(size, spare=16) # never more parts than ~ceil(size / TAIL_BOOST_PART_SIZE) assert len(parts) <= -(-size // TAIL_BOOST_PART_SIZE) def test_plan_tail_parts_skips_small_or_no_spare() -> None: - assert _plan_tail_parts(TAIL_BOOST_MIN_SEGMENT_SIZE - 1, spare=8) == [] # too small - assert _plan_tail_parts(20 * 1024 * 1024, spare=1) == [] # not enough spare workers - assert _plan_tail_parts(20 * 1024 * 1024, spare=0) == [] + assert plan_tail_parts(TAIL_BOOST_MIN_SEGMENT_SIZE - 1, spare=8) == [] # too small + assert plan_tail_parts(20 * 1024 * 1024, spare=1) == [] # not enough spare workers + assert plan_tail_parts(20 * 1024 * 1024, spare=0) == [] def test_split_ranges_covers_exactly_no_gaps_no_overlaps() -> None: for size in (1, 100, 4 * 1024 * 1024, 16 * 1024 * 1024 + 7, 64 * 1024 * 1024): - parts = _split_ranges(size, max_parts=8, part_target=4 * 1024 * 1024) + parts = split_ranges(size, max_parts=8, part_target=4 * 1024 * 1024) assert parts[0][0] == 0 and parts[-1][1] == size - 1 for (_, prev_end), (start, _) in zip(parts, parts[1:]): assert start == prev_end + 1 @@ -220,37 +220,37 @@ def test_split_ranges_covers_exactly_no_gaps_no_overlaps() -> None: def test_split_ranges_respects_max_parts_and_target() -> None: - assert len(_split_ranges(100 * 1024 * 1024, max_parts=4, part_target=16 * 1024 * 1024)) == 4 # capped - assert len(_split_ranges(10 * 1024 * 1024, max_parts=8, part_target=16 * 1024 * 1024)) == 1 # below one target - assert _split_ranges(1, max_parts=8, part_target=16 * 1024 * 1024) == [(0, 0)] + assert len(split_ranges(100 * 1024 * 1024, max_parts=4, part_target=16 * 1024 * 1024)) == 4 # capped + assert len(split_ranges(10 * 1024 * 1024, max_parts=8, part_target=16 * 1024 * 1024)) == 1 # below one target + assert split_ranges(1, max_parts=8, part_target=16 * 1024 * 1024) == [(0, 0)] def test_tail_boost_engages_on_idle_capacity() -> None: # engages at the tail: idle workers outnumber the remaining whole segments - assert _tail_boost_engages(remaining=4, pending=2, target=8) # spare 6 >= 4 - assert _tail_boost_engages(remaining=6, pending=6, target=16) # spare 10 >= 6 + assert tail_boost_engages(remaining=4, pending=2, target=8) # spare 6 >= 4 + assert tail_boost_engages(remaining=6, pending=6, target=16) # spare 10 >= 6 def test_tail_boost_engages_when_remaining_is_a_full_stride() -> None: # remaining sits at 6 (one worker stride) while the target has climbed to 16, so 14 workers # are idle. idle capacity far exceeds the remaining segments, so the boost must engage. - assert _tail_boost_engages(remaining=6, pending=2, target=16) + assert tail_boost_engages(remaining=6, pending=2, target=16) def test_tail_boost_does_not_engage_early_or_saturated() -> None: - assert not _tail_boost_engages(remaining=54, pending=2, target=6) # early: 54 > spare 4 - assert not _tail_boost_engages(remaining=1, pending=8, target=8) # saturated: spare 0 - assert not _tail_boost_engages(remaining=0, pending=2, target=8) # nothing left - assert not _tail_boost_engages(remaining=5, pending=10, target=8) # over-subscribed: spare < 0 + assert not tail_boost_engages(remaining=54, pending=2, target=6) # early: 54 > spare 4 + assert not tail_boost_engages(remaining=1, pending=8, target=8) # saturated: spare 0 + assert not tail_boost_engages(remaining=0, pending=2, target=8) # nothing left + assert not tail_boost_engages(remaining=5, pending=10, target=8) # over-subscribed: spare < 0 def test_has_range_header_detects_byte_range_slices() -> None: # items carrying a Range header are byte-range slices; ranged-parallel must skip them - assert _has_range_header({"url": "u", "headers": {"Range": "bytes=0-99"}}) - assert _has_range_header({"url": "u", "headers": {"range": "bytes=0-99"}}) # case-insensitive - assert not _has_range_header({"url": "u", "headers": {"Authorization": "Bearer x"}}) - assert not _has_range_header({"url": "u", "headers": {}}) - assert not _has_range_header({"url": "u"}) + assert has_range_header({"url": "u", "headers": {"Range": "bytes=0-99"}}) + assert has_range_header({"url": "u", "headers": {"range": "bytes=0-99"}}) # case-insensitive + assert not has_range_header({"url": "u", "headers": {"Authorization": "Bearer x"}}) + assert not has_range_header({"url": "u", "headers": {}}) + assert not has_range_header({"url": "u"}) def test_download_context_adaptive_default_false() -> None: diff --git a/tests/core/test_anilist_provider.py b/tests/core/test_anilist_provider.py new file mode 100644 index 00000000..cc24e682 --- /dev/null +++ b/tests/core/test_anilist_provider.py @@ -0,0 +1,278 @@ +"""AniList provider: search, ID lookup, title-variant config and derived fields.""" + +from __future__ import annotations + +from typing import Any, Optional + +import pytest + +from unshackle.core.config import config +from unshackle.core.providers.anilist import AniListProvider, parse_anilist_ref + +ONE_PIECE = { + "id": 21, + "idMal": 21, + "title": {"romaji": "ONE PIECE", "english": "One Piece", "native": "ONE PIECE"}, + "format": "TV", + "startDate": {"year": 1999}, + "countryOfOrigin": "JP", +} + +ONE_PIECE_MOVIE = { + "id": 459, + "idMal": 459, + "title": {"romaji": "ONE PIECE (Movie)", "english": "One Piece: The Movie", "native": "ONE PIECE (Movie)"}, + "format": "MOVIE", + "startDate": {"year": 2000}, + "countryOfOrigin": "JP", +} + + +class FakeResponse: + def __init__(self, body: dict) -> None: + self._body = body + + def json(self) -> dict: + return self._body + + +def _provider(monkeypatch: pytest.MonkeyPatch, body: dict) -> tuple[AniListProvider, list[dict]]: + """An AniListProvider whose every POST answers `body`; returns the sent payloads.""" + provider = AniListProvider() + sent: list[dict] = [] + + def fake_post(url: str, json: Optional[dict] = None, **kwargs: Any) -> FakeResponse: + sent.append(json or {}) + return FakeResponse(body) + + monkeypatch.setattr(provider, "_session", type("S", (), {"post": staticmethod(fake_post)})()) + return provider, sent + + +@pytest.fixture(autouse=True) +def default_title_language(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(config, "anilist_title_language", "english") + + +# ---------- reference parsing ---------- + + +@pytest.mark.parametrize( + ("value", "expected"), + [ + (21, ("id", 21)), + ("21", ("id", 21)), + (" 21 ", ("id", 21)), + ("mal:21", ("mal", 21)), + ("MAL:21", ("mal", 21)), + ("mal:007", ("mal", 7)), + ("tt1375666", None), + ("anilist:21", None), + ("mal:", None), + ("", None), + (0, None), + ("0", None), + ("mal:0", None), + (-3, None), + ("21.5", None), + ("²", None), # superscript two passes isdigit() but crashes int() + ], +) +def test_parse_anilist_ref(value: Any, expected: Optional[tuple[str, int]]) -> None: + assert parse_anilist_ref(value) == expected + + +# ---------- search ---------- + + +def test_search_returns_the_best_match(monkeypatch: pytest.MonkeyPatch) -> None: + provider, sent = _provider(monkeypatch, {"data": {"Page": {"media": [ONE_PIECE, ONE_PIECE_MOVIE]}}}) + + result = provider.search("one piece", None, "tv") + + assert result is not None + assert result.title == "One Piece" + assert result.year == 1999 + assert result.kind == "tv" + assert result.original_language == "ja" + assert result.external_ids.anilist_id == 21 + assert result.source == "anilist" + assert sent[0]["variables"] == {"search": "one piece"} + + +def test_search_prefers_the_requested_year(monkeypatch: pytest.MonkeyPatch) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Page": {"media": [ONE_PIECE, ONE_PIECE_MOVIE]}}}) + + result = provider.search("one piece", 2000, "movie") + + assert result is not None + assert result.year == 2000 + assert result.kind == "movie" + assert result.external_ids.anilist_id == 459 + + +def test_search_prefers_the_requested_kind_without_a_year(monkeypatch: pytest.MonkeyPatch) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Page": {"media": [ONE_PIECE, ONE_PIECE_MOVIE]}}}) + + result = provider.search("one piece", None, "movie") + + assert result is not None + assert result.kind == "movie" + assert result.external_ids.anilist_id == 459 + + +def test_search_for_tv_is_not_shadowed_by_a_movie(monkeypatch: pytest.MonkeyPatch) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Page": {"media": [ONE_PIECE_MOVIE, ONE_PIECE]}}}) + + result = provider.search("one piece", None, "tv") + + assert result is not None + assert result.kind == "tv" + assert result.external_ids.anilist_id == 21 + + +def test_search_falls_back_when_no_node_matches_the_kind(monkeypatch: pytest.MonkeyPatch) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Page": {"media": [ONE_PIECE]}}}) + + result = provider.search("one piece", None, "movie") + + assert result is not None + assert result.external_ids.anilist_id == 21 + + +def test_search_of_a_non_anime_title_returns_none(monkeypatch: pytest.MonkeyPatch) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Page": {"media": []}}}) + assert provider.search("The Walking Dead", None, "tv") is None + + +def test_graphql_errors_are_a_miss_not_a_raise(monkeypatch: pytest.MonkeyPatch) -> None: + provider, _ = _provider(monkeypatch, {"errors": [{"message": "Not Found."}], "data": {"Media": None}}) + assert provider.search("one piece", None, "tv") is None + assert provider.get_by_id(21, "tv") is None + + +# ---------- id lookup ---------- + + +def test_get_by_id_queries_the_anilist_id(monkeypatch: pytest.MonkeyPatch) -> None: + provider, sent = _provider(monkeypatch, {"data": {"Media": ONE_PIECE}}) + + result = provider.get_by_id(21, "tv") + + assert result is not None + assert result.title == "One Piece" + assert sent[0]["variables"] == {"id": 21} + + +def test_get_by_id_resolves_a_mal_reference_natively(monkeypatch: pytest.MonkeyPatch) -> None: + provider, sent = _provider(monkeypatch, {"data": {"Media": ONE_PIECE}}) + + result = provider.get_by_id("mal:21", "tv") + + assert result is not None + assert sent[0]["variables"] == {"idMal": 21} + assert result.external_ids.anilist_id == 21 + + +def test_get_by_id_rejects_a_foreign_reference(monkeypatch: pytest.MonkeyPatch) -> None: + provider, sent = _provider(monkeypatch, {"data": {"Media": ONE_PIECE}}) + assert provider.get_by_id("tt1375666", "tv") is None + assert sent == [] + + +def test_get_external_ids_needs_no_lookup_for_a_bare_id(monkeypatch: pytest.MonkeyPatch) -> None: + provider, sent = _provider(monkeypatch, {"data": {"Media": ONE_PIECE}}) + + assert provider.get_external_ids(21, "tv").anilist_id == 21 + assert sent == [] + assert provider.get_external_ids("mal:21", "tv").anilist_id == 21 + assert sent[0]["variables"] == {"idMal": 21} + + +# ---------- title variant ---------- + + +@pytest.mark.parametrize( + ("configured", "expected"), + [("english", "One Piece"), ("romaji", "ONE PIECE"), ("native", "ONE PIECE"), ("klingon", "One Piece")], +) +def test_title_language_config(monkeypatch: pytest.MonkeyPatch, configured: str, expected: str) -> None: + monkeypatch.setattr(config, "anilist_title_language", configured) + provider, _ = _provider(monkeypatch, {"data": {"Media": ONE_PIECE}}) + + result = provider.get_by_id(21, "tv") + + assert result is not None + assert result.title == expected + + +def test_title_language_falls_back_when_the_variant_is_unset(monkeypatch: pytest.MonkeyPatch) -> None: + node = {**ONE_PIECE, "title": {"romaji": "Koisuru ONE PIECE", "english": None, "native": "恋するワンピース"}} + provider, _ = _provider(monkeypatch, {"data": {"Media": node}}) + + result = provider.get_by_id(21, "tv") + + assert result is not None + assert result.title == "Koisuru ONE PIECE" + + +def test_a_title_with_no_variant_at_all_is_a_miss(monkeypatch: pytest.MonkeyPatch) -> None: + node = {**ONE_PIECE, "title": {"romaji": None, "english": None, "native": None}} + provider, _ = _provider(monkeypatch, {"data": {"Media": node}}) + assert provider.get_by_id(21, "tv") is None + + +# ---------- derived fields ---------- + + +@pytest.mark.parametrize( + ("country", "language"), + [("JP", "ja"), ("KR", "ko"), ("CN", "zh"), ("TW", "zh"), ("US", None), (None, None)], +) +def test_country_of_origin_maps_to_a_language( + monkeypatch: pytest.MonkeyPatch, country: Optional[str], language: Optional[str] +) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Media": {**ONE_PIECE, "countryOfOrigin": country}}}) + + result = provider.get_by_id(21, "tv") + + assert result is not None + assert result.original_language == language + + +@pytest.mark.parametrize( + ("format_", "kind"), [("MOVIE", "movie"), ("TV", "tv"), ("ONA", "tv"), ("SPECIAL", "tv"), (None, "tv")] +) +def test_format_maps_to_kind(monkeypatch: pytest.MonkeyPatch, format_: Optional[str], kind: str) -> None: + provider, _ = _provider(monkeypatch, {"data": {"Media": {**ONE_PIECE, "format": format_}}}) + + result = provider.get_by_id(21, "tv") + + assert result is not None + assert result.kind == kind + + +def test_the_provider_needs_no_api_key() -> None: + assert AniListProvider().is_available() is True + assert AniListProvider.REQUIRES_KEY is False + assert AniListProvider.ID_KIND == "anilist" + + +# ---------- cache round-trip ---------- + + +def test_a_cached_node_rebuilds_with_the_read_time_title_language(monkeypatch: pytest.MonkeyPatch) -> None: + """The raw node is cached; which title variant it yields follows the config at read time.""" + from unshackle.core.providers import cached_to_result + + node = {**ONE_PIECE, "title": {"romaji": "WAN PIISU", "english": "One Piece", "native": "ワンピース"}} + + monkeypatch.setattr(config, "anilist_title_language", "native") + result = cached_to_result(node, "anilist", "tv") + + assert result is not None + assert result.title == "ワンピース" + assert result.year == 1999 + assert result.kind == "tv" + assert result.external_ids.anilist_id == 21 + assert result.original_language == "ja" diff --git a/tests/core/test_cdm_detect.py b/tests/core/test_cdm_detect.py index 612f2a1c..55c0736f 100644 --- a/tests/core/test_cdm_detect.py +++ b/tests/core/test_cdm_detect.py @@ -24,7 +24,7 @@ from unshackle.core.cdm.detect import ( is_remote_widevine_cdm, is_widevine_cdm, ) -from unshackle.core.cdm.loader import _stamp_remote +from unshackle.core.cdm.loader import stamp_remote pytestmark = pytest.mark.unit @@ -125,7 +125,7 @@ def test_custom_remote_cdm_classification(device, expect_pr): def test_stamp_remote_sets_attrs_and_returns_same_object(): obj = SimpleNamespace() - out = _stamp_remote(obj, "playready") + out = stamp_remote(obj, "playready") assert out is obj assert obj.drm == "playready" assert obj.is_remote_cdm is True @@ -136,14 +136,14 @@ def test_stamp_remote_is_best_effort_on_unsettable_object(): __slots__ = () obj = Slotted() - out = _stamp_remote(obj, "widevine") # must not raise + out = stamp_remote(obj, "widevine") # must not raise assert out is obj assert not hasattr(obj, "drm") def test_load_remote_decryptlabs_playready_is_stamped(): cdm_api = {"name": "dl-pr", "type": "decrypt_labs", "secret": "x", "device_name": "SL3000"} - cdm = loader_mod._load_remote_cdm(dict(cdm_api), "dl-pr", "AMZN", None) + cdm = loader_mod.load_remote_cdm(dict(cdm_api), "dl-pr", "AMZN", None) assert cdm.drm == "playready" assert cdm.is_remote_cdm is True assert is_remote_playready_cdm(cdm) @@ -151,7 +151,7 @@ def test_load_remote_decryptlabs_playready_is_stamped(): def test_load_remote_decryptlabs_widevine_is_stamped(): cdm_api = {"name": "dl-wv", "type": "decrypt_labs", "secret": "x", "device_name": "ChromeCDM"} - cdm = loader_mod._load_remote_cdm(dict(cdm_api), "dl-wv", "AMZN", None) + cdm = loader_mod.load_remote_cdm(dict(cdm_api), "dl-wv", "AMZN", None) assert cdm.drm == "widevine" assert is_remote_widevine_cdm(cdm) @@ -165,7 +165,7 @@ def test_load_remote_native_playready_is_stamped(monkeypatch): monkeypatch.setattr(prmod, "RemoteCdm", FakePlayReadyRemote) cdm_api = {"name": "pr", "Device Type": "PLAYREADY", "host": "h", "secret": "s", "device_name": "d"} - cdm = loader_mod._load_remote_cdm(dict(cdm_api), "pr", "AMZN", None) + cdm = loader_mod.load_remote_cdm(dict(cdm_api), "pr", "AMZN", None) assert isinstance(cdm, FakePlayReadyRemote) assert cdm.drm == "playready" assert cdm.is_remote_cdm is True @@ -181,7 +181,7 @@ def test_load_remote_native_widevine_is_stamped(monkeypatch): monkeypatch.setattr(wvmod, "RemoteCdm", FakeWidevineRemote) cdm_api = {"name": "wv", "Device Type": "ANDROID", "host": "h", "secret": "s", "device_name": "d"} - cdm = loader_mod._load_remote_cdm(dict(cdm_api), "wv", "AMZN", None) + cdm = loader_mod.load_remote_cdm(dict(cdm_api), "wv", "AMZN", None) assert isinstance(cdm, FakeWidevineRemote) assert cdm.drm == "widevine" assert is_remote_widevine_cdm(cdm) diff --git a/tests/core/test_clearkey.py b/tests/core/test_clearkey.py index f3df8116..437fd6e9 100644 --- a/tests/core/test_clearkey.py +++ b/tests/core/test_clearkey.py @@ -88,7 +88,7 @@ def test_warns_once_per_instance(tmp_path, caplog) -> None: assert len(warnings) == 1 -def _data_key(raw: bytes, base64_encoded: bool = True) -> Key: +def data_key(raw: bytes, base64_encoded: bool = True) -> Key: if base64_encoded: uri = "data:text/plain;base64," + base64.b64encode(raw).decode("ascii") else: @@ -97,24 +97,24 @@ def _data_key(raw: bytes, base64_encoded: bool = True) -> Key: def test_data_uri_key_decodes_and_validates_length() -> None: - drm = ClearKey.from_m3u_key(_data_key(KEY)) + drm = ClearKey.from_m3u_key(data_key(KEY)) assert drm.key == KEY def test_data_uri_key_wrong_length_rejected() -> None: with pytest.raises(ValueError, match="Unexpected Length"): - ClearKey.from_m3u_key(_data_key(b"\x00" * 8)) + ClearKey.from_m3u_key(data_key(b"\x00" * 8)) def test_raw_data_uri_key_materialises_and_validates() -> None: # A raw (non-base64) data: payload must be taken as literal key bytes, not hex-parsed. - drm = ClearKey.from_m3u_key(_data_key(KEY, base64_encoded=False)) + drm = ClearKey.from_m3u_key(data_key(KEY, base64_encoded=False)) assert drm.key == KEY def test_raw_data_uri_key_wrong_length_rejected() -> None: with pytest.raises(ValueError, match="Unexpected Length"): - ClearKey.from_m3u_key(_data_key(b"\x01" * 8, base64_encoded=False)) + ClearKey.from_m3u_key(data_key(b"\x01" * 8, base64_encoded=False)) def test_data_uri_split_tolerates_commas_in_payload() -> None: diff --git a/tests/core/test_click_types.py b/tests/core/test_click_types.py index 4581816a..fe2c95b7 100644 --- a/tests/core/test_click_types.py +++ b/tests/core/test_click_types.py @@ -7,7 +7,13 @@ import click import pytest from unshackle.core.tracks.subtitle import Subtitle -from unshackle.core.utils.click_types import QUALITY_LIST, SLOW_DELAY_RANGE, SeasonRange, SubtitleCodecChoice +from unshackle.core.utils.click_types import ( + LANGUAGE_RANGE, + QUALITY_LIST, + SLOW_DELAY_RANGE, + SeasonRange, + SubtitleCodecChoice, +) choice = SubtitleCodecChoice(Subtitle.Codec) @@ -60,6 +66,23 @@ def test_slow_delay_range_accepts_bool(value, expected): assert SLOW_DELAY_RANGE.convert(value, None, None) == expected +# --- LanguageRange ---------------------------------------------------------- + + +@pytest.mark.parametrize( + "value,expected", + [ + ("all,-es", ["all", "-es"]), + ("-es", ["-es"]), + ("all, -es ; -fr", ["all", "-es", "-fr"]), + (["all", "-es"], ["all", "-es"]), + ], +) +def test_language_range_passes_exclusion_tokens_through(value, expected): + """The '-' prefix is resolved at filter time, so the type must keep the token intact.""" + assert LANGUAGE_RANGE.convert(value) == expected + + # --- SeasonRange ------------------------------------------------------------ @@ -131,3 +154,44 @@ def test_exclusion_removes_duplicate_keys(): def test_bad_part_tokens_fail(token): with pytest.raises(click.UsageError): parse(token) + + +# --- SeasonRange date tokens ------------------------------------------------ + + +def test_single_date_token_is_its_own_key(): + assert parse("2026-08-11") == ["2026-08-11"] + + +def test_date_range_expands_every_day_inclusive(): + assert sorted(parse("2026-08-01:2026-08-03")) == ["2026-08-01", "2026-08-02", "2026-08-03"] + + +def test_date_range_crosses_a_month(): + assert sorted(parse("2026-01-30:2026-02-02")) == ["2026-01-30", "2026-01-31", "2026-02-01", "2026-02-02"] + + +def test_date_exclusion_removes_the_day(): + assert sorted(parse("2026-08-01:2026-08-03", "-2026-08-02")) == ["2026-08-01", "2026-08-03"] + + +def test_dates_mix_with_season_episode_tokens(): + assert sorted(parse("S01E01", "2026-08-11")) == ["1x1", "2026-08-11"] + + +def test_comma_separated_dates_convert(): + assert sorted(SeasonRange().convert("2026-08-11, 2026-08-12")) == ["2026-08-11", "2026-08-12"] + + +@pytest.mark.parametrize( + "token", + [ + "2026-13-01", # not a real month + "2026-02-30", # not a real day + "2026-08-03:2026-08-01", # reversed range + "2020-01-01:2026-01-01", # over MAX_DATE_SPAN + ], +) +def test_bad_date_tokens_fail(token): + with pytest.raises(click.UsageError): + parse(token) diff --git a/tests/core/test_code_hash.py b/tests/core/test_code_hash.py index 68ee6d92..759b8e6e 100644 --- a/tests/core/test_code_hash.py +++ b/tests/core/test_code_hash.py @@ -70,5 +70,5 @@ def test_unreadable_source_yields_empty_string() -> None: def test_unwalkable_source_yields_empty_string(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: """A dir os.walk cannot enter must yield "", not a valid-looking hash of fewer files.""" (tmp_path / "__main__.py").write_bytes(b"") - monkeypatch.setattr(unshackle.core, "_PKG", tmp_path) + monkeypatch.setattr(unshackle.core, "PKG", tmp_path) assert code_hash() == "" diff --git a/tests/core/test_dash_collapse.py b/tests/core/test_dash_collapse.py index 6ffaa218..82155ab0 100644 --- a/tests/core/test_dash_collapse.py +++ b/tests/core/test_dash_collapse.py @@ -1,4 +1,4 @@ -"""Offline unit tests for DASH._collapsible_single_url, the guard that decides when a +"""Offline unit tests for DASH.collapsible_single_url, the guard that decides when a byte-range track (SegmentBase / ranged SegmentList) can be downloaded as one whole resource instead of per-segment files + merge.""" @@ -12,7 +12,7 @@ URL = "https://cdn/media.mp4" def collapsible(segments, init_len: Optional[int], is_subtitle: bool = False) -> bool: - return DASH._collapsible_single_url(is_subtitle, segments, init_len) + return DASH.collapsible_single_url(is_subtitle, segments, init_len) def test_segmentbase_single_range_eligible(): diff --git a/tests/core/test_dash_collapse_integration.py b/tests/core/test_dash_collapse_integration.py index 778aa888..ae9fb93e 100644 --- a/tests/core/test_dash_collapse_integration.py +++ b/tests/core/test_dash_collapse_integration.py @@ -1,6 +1,6 @@ """Integration coverage for the CONSUMING side of the DASH single-URL byte-range collapse. -The pure predicate ``DASH._collapsible_single_url`` is unit-tested in +The pure predicate ``DASH.collapsible_single_url`` is unit-tested in ``test_dash_collapse.py``; this file drives the real ``DASH.download_track`` code path that acts on it. A minimal MPD is parsed with ``DASH.from_text().to_tracks()`` (so the track's ``data["dash"]`` is built by real parser code, not hand-faked) and downloaded against a @@ -92,7 +92,7 @@ def _base(srv) -> str: def _collapse_mpd(srv) -> str: # SegmentList on one media.mp4: init is bytes [0, INIT_LEN), media ranges cover the rest - # contiguously -> _collapsible_single_url returns the URL and download_track collapses. + # contiguously -> collapsible_single_url returns the URL and download_track collapses. return f""" {_base(srv)} diff --git a/tests/core/test_disable_metadata.py b/tests/core/test_disable_metadata.py new file mode 100644 index 00000000..90ada3fc --- /dev/null +++ b/tests/core/test_disable_metadata.py @@ -0,0 +1,67 @@ +"""The `disable_metadata` config switch stops automatic lookups but keeps supplied IDs working.""" + +from __future__ import annotations + +from typing import Optional, Union + +import pytest + +from unshackle.core import providers +from unshackle.core.config import config +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult + + +class FakeProvider(MetadataProvider): + """Stands in for a real provider; fails the test if it is searched.""" + + def __init__(self, name: str, result: Optional[MetadataResult] = None) -> None: + self.NAME = name + self._result = result + self.asked: list[Union[int, str]] = [] + super().__init__() + + def is_available(self) -> bool: + return True + + def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: + raise AssertionError("no provider may be searched while disable_metadata is set") + + def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: + self.asked.append(provider_id) + return self._result + + def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: + return ExternalIds() + + +@pytest.fixture(autouse=True) +def disabled(monkeypatch: pytest.MonkeyPatch) -> None: + """Turn metadata off and keep every test off the network.""" + monkeypatch.setattr(config, "disable_metadata", True) + monkeypatch.setattr(config, "metadata_providers", []) + monkeypatch.setattr(providers, "fetch_external_ids", lambda *a, **kw: ExternalIds()) + monkeypatch.setattr(providers, "enrich_ids", lambda result: None) + + +def test_search_metadata_returns_none(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(providers, "get_provider", lambda name: FakeProvider(name)) + + assert providers.search_metadata("Futurama", 1999, "tv") is None + + +def test_supplied_id_still_resolves(monkeypatch: pytest.MonkeyPatch) -> None: + tmdb = FakeProvider("tmdb", MetadataResult(title="Inception", year=2010, kind="movie")) + monkeypatch.setattr(providers, "get_provider", lambda name: {"tmdb": tmdb}.get(name)) + + result = providers.resolve_by_ids(27205, kind="movie") + + assert result is not None + assert result.title == "Inception" + assert result.external_ids.tmdb_id == 27205 + assert tmdb.asked == [27205] + + +def test_no_ids_returns_none(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(providers, "get_provider", lambda name: FakeProvider(name)) + + assert providers.resolve_by_ids(title="Futurama", year=1999, kind="tv") is None diff --git a/tests/core/test_downloader_resume.py b/tests/core/test_downloader_resume.py index 33b0d15c..686ee15f 100644 --- a/tests/core/test_downloader_resume.py +++ b/tests/core/test_downloader_resume.py @@ -190,9 +190,9 @@ def test_byte_range_slice_segments_never_probed_or_boosted(tmp_path: Path, monke host, port = httpd.server_address probed: list[str] = [] - real_probe = dl._probe_ranged + real_probe = dl.probe_ranged monkeypatch.setattr( - dl, "_probe_ranged", lambda url, session, **kw: probed.append(url) or real_probe(url, session, **kw) + dl, "probe_ranged", lambda url, session, **kw: probed.append(url) or real_probe(url, session, **kw) ) url = f"http://{host}:{port}/media.mp4" diff --git a/tests/core/test_dual_multi_naming.py b/tests/core/test_dual_multi_naming.py index 6e385275..b5b8f839 100644 --- a/tests/core/test_dual_multi_naming.py +++ b/tests/core/test_dual_multi_naming.py @@ -47,7 +47,7 @@ def count_mode(monkeypatch): def ctx(language, languages) -> dict: - return make_movie(language)._build_base_template_context(make_media_info(languages)) + return make_movie(language).build_base_template_context(make_media_info(languages)) def test_strict_dual_with_original(strict_mode): diff --git a/tests/core/test_enrich_language.py b/tests/core/test_enrich_language.py new file mode 100644 index 00000000..74104717 --- /dev/null +++ b/tests/core/test_enrich_language.py @@ -0,0 +1,80 @@ +"""Original-language parsing for the metadata a supplied ID resolves.""" + +from __future__ import annotations + +import pytest + +from unshackle.commands.dl import parse_language +from unshackle.core.providers.imdb import primary_language as imdb_language +from unshackle.core.providers.omdb import primary_language as omdb_language +from unshackle.core.providers.tvdb import primary_language as tvdb_language + + +@pytest.mark.parametrize( + ("tag", "expected"), + [ + ("ja", "ja"), # TMDB alpha-2 + ("jpn", "ja"), # IMDb alpha-3 + ("pt-BR", "pt-BR"), + (None, None), + ("", None), + ("und", None), + ("xx-not-a-tag", None), + ], +) +def test_parse_language(tag: str | None, expected: str | None) -> None: + result = parse_language(tag) + assert (str(result) if result else None) == expected + + +def test_imdb_language_takes_the_first_spoken_language() -> None: + data = {"spokenLanguages": {"spokenLanguages": [{"id": "ja", "text": "Japanese"}, {"id": "en", "text": "English"}]}} + assert imdb_language(data) == "ja" + + +def test_imdb_language_is_alpha_2() -> None: + """IMDb's GraphQL language ids are alpha-2.""" + data = {"spokenLanguages": {"spokenLanguages": [{"id": "ko", "text": "Korean"}]}} + assert parse_language(imdb_language(data)) is not None + assert str(parse_language(imdb_language(data))) == "ko" + + +@pytest.mark.parametrize( + "data", + [ + {}, + {"spokenLanguages": None}, # search hits can come back with the whole block null + {"spokenLanguages": {}}, + {"spokenLanguages": {"spokenLanguages": []}}, + {"spokenLanguages": {"spokenLanguages": None}}, + ], +) +def test_imdb_language_missing(data: dict) -> None: + assert imdb_language(data) is None + + +@pytest.mark.parametrize( + ("names", "expected"), + [ + ("Korean", "ko"), + ("Korean, English", "ko"), # OMDb lists the main language first + ("", None), + ("None", None), # OMDb's literal placeholder for unknown + ("Not A Language", None), + ], +) +def test_omdb_language_from_english_name(names: str, expected: str | None) -> None: + assert omdb_language({"Language": names}) == expected + + +@pytest.mark.parametrize( + ("data", "expected"), + [ + ({"primary_language": "kor"}, "kor"), # /search + ({"originalLanguage": "kor"}, "kor"), # /series/{id}/extended + ({"primary_language": ""}, None), + ({}, None), + ], +) +def test_tvdb_language_key_variants(data: dict, expected: str | None) -> None: + assert tvdb_language(data) == expected diff --git a/tests/core/test_episode_absolute.py b/tests/core/test_episode_absolute.py new file mode 100644 index 00000000..52b21f35 --- /dev/null +++ b/tests/core/test_episode_absolute.py @@ -0,0 +1,215 @@ +"""Absolute episode numbering: Episode.absolute, the {absolute} token, and the enrich fill.""" + +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + +from unshackle.core import providers +from unshackle.core.config import config +from unshackle.core.titles.episode import Episode, Series + + +class DummyService: + pass + + +class StubMediaInfo: + """Minimal MediaInfo stand-in: the template context only reads track lists.""" + + video_tracks: list = [] + audio_tracks: list = [] + + +def make_episode(**overrides) -> Episode: + kwargs = dict( + id_="ep-0001", + service=DummyService, + title="The Show", + season=1, + number=1, + name="The Reckoning", + ) + kwargs.update(overrides) + return Episode(**kwargs) + + +@pytest.fixture +def reset_template(): + saved = (config.output_template, config.folder_templates, config.folder_template, config.tag) + config.output_template = {"series": "{title}.{season_episode}.{absolute}.{source}-{tag}"} + config.folder_templates = {} + config.folder_template = "" + config.tag = "GROUP" + yield config + config.output_template, config.folder_templates, config.folder_template, config.tag = saved + + +# --- validation ------------------------------------------------------------- + + +def test_absolute_defaults_to_none(): + assert make_episode().absolute is None + + +def test_int_absolute_is_kept(): + assert make_episode(absolute=27).absolute == 27 + + +def test_digit_string_absolute_coerces(): + assert make_episode(absolute="27").absolute == 27 + + +@pytest.mark.parametrize("bad", [0, -1]) +def test_non_positive_absolute_rejected(bad): + with pytest.raises(ValueError): + make_episode(absolute=bad) + + +@pytest.mark.parametrize("bad", ["x", "1.5", 1.5, []]) +def test_non_int_absolute_rejected(bad): + with pytest.raises(TypeError): + make_episode(absolute=bad) + + +def test_bool_absolute_rejected(): + """bool is an int subclass; True would render as '001' in filenames.""" + with pytest.raises(TypeError): + make_episode(absolute=True) + + +def test_old_cache_entry_reads_as_absolute_less(): + """jsonpickle restores by populating __dict__; an older entry has no 'absolute' key.""" + restored = Episode.__new__(Episode) + restored.__dict__.update(title="The Show", season=1, number=1, name=None, year=None) + assert restored.absolute is None + + +# --- naming ----------------------------------------------------------------- + + +def test_absolute_token_is_three_wide(reset_template): + assert make_episode(absolute=7).build_template_context(StubMediaInfo())["absolute"] == "007" + + +def test_absolute_token_does_not_truncate_past_three_digits(reset_template): + assert make_episode(absolute=1042).build_template_context(StubMediaInfo())["absolute"] == "1042" + + +def test_absolute_token_is_empty_without_a_value(reset_template): + assert make_episode().build_template_context(StubMediaInfo())["absolute"] == "" + + +def test_filename_renders_the_absolute_number(reset_template): + assert make_episode(absolute=7).get_filename(StubMediaInfo()) == "The.Show.S01E01.007.DummyService-GROUP" + + +def test_optional_absolute_token_drops_out(reset_template): + reset_template.output_template = {"series": "{title}.{season_episode}.{absolute?}.{source}-{tag}"} + assert make_episode().get_filename(StubMediaInfo()) == "The.Show.S01E01.DummyService-GROUP" + assert make_episode(absolute=7).get_filename(StubMediaInfo()) == "The.Show.S01E01.007.DummyService-GROUP" + + +def test_absolute_is_a_known_template_variable(reset_template, recwarn): + reset_template.validate_output_templates() + assert not [w for w in recwarn.list if "absolute" in str(w.message)] + + +def test_str_and_sorting_ignore_the_absolute_number(): + assert str(make_episode(absolute=7)) == "The Show S01E01 The Reckoning" + series = Series([make_episode(number=2, absolute=1), make_episode(number=1, absolute=2)]) + assert [e.number for e in series] == [1, 2] + + +# --- the enrich fill -------------------------------------------------------- + + +class _FakeProvider: + def __init__(self, mapping: dict, source_order: str = "official") -> None: + self.mapping = mapping + self.source_order = source_order + self.orders: list = [] + + def detect_order(self, _id, _keys) -> str: + return self.source_order + + def get_order_map(self, _id, order, source_order=None) -> dict: + self.orders.append(order) + return self.mapping + + +def fill( + monkeypatch: pytest.MonkeyPatch, episodes: list, mapping: dict, tvdb_id=73871, provider=True, order="official" +): + from unshackle.commands.dl import dl + + fake = _FakeProvider(mapping, order) if provider else None + monkeypatch.setattr(providers, "get_provider", lambda _name: fake) + logged: list = [] + logger = SimpleNamespace( + debug=lambda *a, **k: logged.append(("debug", a)), + info=lambda *a, **k: logged.append(("info", a)), + warning=lambda *a, **k: logged.append(("warning", a)), + ) + stub = SimpleNamespace(tvdb_id=tvdb_id, log=logger) + titles = Series(episodes) + dl.fill_absolute_numbers(stub, titles, tvdb_id) + return titles, fake, logged + + +MAPPING = {(1, 1): (1, 1, "One"), (1, 2): (1, 2, "Two"), (2, 1): (1, 3, "Three")} + + +def test_fill_populates_missing_absolute_numbers(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=1, number=1), make_episode(season=1, number=2), make_episode(season=2, number=1)] + titles, fake, _ = fill(monkeypatch, episodes, MAPPING) + assert [t.absolute for t in titles] == [1, 2, 3] + assert fake.orders == ["absolute"] + + +def test_a_service_already_in_absolute_order_fills_from_its_own_numbers(monkeypatch: pytest.MonkeyPatch) -> None: + """get_order_map returns {} for the same-order case; the numbers are their own fill.""" + episodes = [make_episode(season=1, number=26), make_episode(season=1, number=27)] + titles, fake, _ = fill(monkeypatch, episodes, {}, order="absolute") + assert [t.absolute for t in titles] == [26, 27] + assert fake.orders == [] + + +def test_fill_never_touches_season_or_number(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=2, number=1)] + titles, _, _ = fill(monkeypatch, episodes, MAPPING) + assert [(t.season, t.number) for t in titles] == [(2, 1)] + + +def test_a_service_set_absolute_number_wins(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=1, number=1, absolute=99), make_episode(season=1, number=2)] + titles, _, _ = fill(monkeypatch, episodes, MAPPING) + assert [t.absolute for t in titles] == [99, 2] + + +def test_an_unmapped_episode_keeps_no_absolute_number(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=9, number=9)] + titles, _, _ = fill(monkeypatch, episodes, MAPPING) + assert titles[0].absolute is None + + +def test_fill_skips_silently_without_an_api_key(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=1, number=1)] + titles, _, logged = fill(monkeypatch, episodes, MAPPING, provider=False) + assert titles[0].absolute is None + assert [level for level, _ in logged] == ["debug"] + + +def test_fill_skips_silently_without_a_tvdb_id(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=1, number=1)] + titles, _, logged = fill(monkeypatch, episodes, MAPPING, tvdb_id=None) + assert titles[0].absolute is None + assert [level for level, _ in logged] == ["debug"] + + +def test_fill_skips_silently_when_tvdb_has_no_absolute_order(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(season=1, number=1)] + titles, _, logged = fill(monkeypatch, episodes, {}) + assert titles[0].absolute is None + assert [level for level, _ in logged] == ["debug"] diff --git a/tests/core/test_episode_air_date.py b/tests/core/test_episode_air_date.py index d14a4629..693898c2 100644 --- a/tests/core/test_episode_air_date.py +++ b/tests/core/test_episode_air_date.py @@ -47,7 +47,7 @@ def reset_template(): def test_air_date_replaces_season_episode_dotted(reset_template): ep = make_episode(air_date=date(2024, 6, 30)) - ctx = ep._build_template_context(StubMediaInfo()) + ctx = ep.build_template_context(StubMediaInfo()) assert ctx["season_episode"] == "2024.06.30" assert ctx["season"] == "2024.06.30" assert ctx["episode"] == "" @@ -58,19 +58,19 @@ def test_air_date_replaces_season_episode_dotted(reset_template): def test_air_date_uses_space_separator(reset_template): reset_template.output_template = {"series": "{title} {season_episode} {quality}-{tag}"} ep = make_episode(air_date=date(2024, 6, 30)) - assert ep._build_template_context(StubMediaInfo())["season_episode"] == "2024 06 30" + assert ep.build_template_context(StubMediaInfo())["season_episode"] == "2024 06 30" def test_iso_string_air_date_coerced(reset_template): ep = make_episode(air_date="2024-06-30") assert isinstance(ep.air_date, date) - assert ep._build_template_context(StubMediaInfo())["season_episode"] == "2024.06.30" + assert ep.build_template_context(StubMediaInfo())["season_episode"] == "2024.06.30" def test_year_always_dropped_in_file_when_dated(reset_template): """The air date is the sole date in the filename; a distinct year is dropped too (deterministic).""" ep = make_episode(year=2010, air_date=date(2013, 10, 30)) - ctx = ep._build_template_context(StubMediaInfo()) + ctx = ep.build_template_context(StubMediaInfo()) assert ctx["year"] == "" assert ctx["season_episode"] == "2013.10.30" @@ -78,7 +78,7 @@ def test_year_always_dropped_in_file_when_dated(reset_template): def test_yearless_title_takes_year_from_date(reset_template): """A title with no year still names cleanly; the date supplies the year.""" ep = make_episode(year=None, air_date=date(2026, 6, 28)) - ctx = ep._build_template_context(StubMediaInfo()) + ctx = ep.build_template_context(StubMediaInfo()) assert ctx["year"] == "" assert ctx["season_episode"] == "2026.06.28" @@ -87,13 +87,13 @@ def test_datetime_normalized_to_date(reset_template): """A datetime (subclass of date) must not leak its time into the {date} token.""" ep = make_episode(air_date=datetime(2024, 6, 30, 13, 0, 0)) assert type(ep.air_date) is date - assert ep._build_template_context(StubMediaInfo())["date"] == "2024-06-30" + assert ep.build_template_context(StubMediaInfo())["date"] == "2024-06-30" def test_no_air_date_is_unchanged(reset_template): """Regression: without air_date, naming is exactly as before (presentation-only field).""" ep = make_episode() - ctx = ep._build_template_context(StubMediaInfo()) + ctx = ep.build_template_context(StubMediaInfo()) assert ctx["season_episode"] == "S2024E181" assert ctx["season"] == "S2024" assert ctx["episode"] == "E181" @@ -107,3 +107,45 @@ def test_dated_folder_groups_by_year(reset_template): folder = make_episode(air_date=date(2024, 6, 30)).get_filename(StubMediaInfo(), folder=True) assert "2024" in folder assert "S2024" not in folder # season folder is the year, not the faked season + + +# --- selection by date ------------------------------------------------------ + + +def test_a_dated_episode_matches_its_iso_date_key(): + assert make_episode(air_date=date(2024, 6, 30)).matches_wanted({"2024-06-30"}) + + +def test_a_dated_episode_still_matches_its_season_episode_key(): + assert make_episode(air_date=date(2024, 6, 30)).matches_wanted({"2024x181"}) + + +def test_a_date_key_does_not_match_another_day(): + assert not make_episode(air_date=date(2024, 6, 30)).matches_wanted({"2024-07-01"}) + + +def test_a_season_episode_exclusion_beats_the_date_key(): + assert not make_episode(air_date=date(2024, 6, 30)).matches_wanted({"2024-06-30", "!2024x181"}) + + +def test_an_undated_episode_is_unaffected_by_date_keys(): + episode = make_episode() + assert not episode.matches_wanted({"2024-06-30"}) + assert episode.matches_wanted({"2024x181"}) + + +# --- the air-date guard ----------------------------------------------------- + + +def test_a_pre_1970_air_date_is_rejected(): + with pytest.raises(ValueError): + make_episode(air_date=date(1969, 12, 31)) + + +def test_a_future_air_date_is_accepted(): + future = date.today().replace(year=date.today().year + 1) + assert make_episode(air_date=future).air_date == future + + +def test_an_unparseable_air_date_string_is_kept_as_is(): + assert make_episode(air_date="soon").air_date == "soon" diff --git a/tests/core/test_episode_part.py b/tests/core/test_episode_part.py index b4d1e749..d166c31f 100644 --- a/tests/core/test_episode_part.py +++ b/tests/core/test_episode_part.py @@ -144,7 +144,7 @@ def test_spaced_template_uses_spaces(reset_template): def test_template_context_leaves_the_season_token_alone(reset_template): - ctx = make_episode(part=2)._build_template_context(StubMediaInfo()) + ctx = make_episode(part=2).build_template_context(StubMediaInfo()) assert ctx["season"] == "S01" assert ctx["episode"] == "E01.Part.2" assert ctx["season_episode"] == "S01E01.Part.2" @@ -152,7 +152,7 @@ def test_template_context_leaves_the_season_token_alone(reset_template): def test_part_token_is_empty_without_a_part(reset_template): - assert make_episode()._build_template_context(StubMediaInfo())["part"] == "" + assert make_episode().build_template_context(StubMediaInfo())["part"] == "" def test_parts_share_one_season_folder(reset_template): diff --git a/tests/core/test_fill_air_dates.py b/tests/core/test_fill_air_dates.py new file mode 100644 index 00000000..101f4b1c --- /dev/null +++ b/tests/core/test_fill_air_dates.py @@ -0,0 +1,166 @@ +"""The --daily air-date fill: dl.fill_air_dates and the daily_hint gate.""" + +from __future__ import annotations + +from datetime import date, timedelta +from types import SimpleNamespace + +import pytest + +from unshackle.core import providers +from unshackle.core.titles.episode import Episode, Series + + +class DummyService: + pass + + +def make_episode(**overrides) -> Episode: + kwargs = dict( + id_="ep-0001", + service=DummyService, + title="The Show", + season=1, + number=1, + ) + kwargs.update(overrides) + return Episode(**kwargs) + + +class _FakeProvider: + def __init__(self, episodes: list, source_order: str = "official") -> None: + self.episodes = episodes + self.source_order = source_order + + def detect_order(self, _id, _keys) -> str: + return self.source_order + + def get_episodes(self, _id, _order) -> list: + return self.episodes + + +def fill( + monkeypatch: pytest.MonkeyPatch, + titles: list, + episodes: list, + tvdb_id=73871, + provider=True, + daily=True, + service_daily=False, +): + from unshackle.commands.dl import dl + + fake = _FakeProvider(episodes) if provider else None + monkeypatch.setattr(providers, "get_provider", lambda _name: fake) + logged: list = [] + logger = SimpleNamespace( + debug=lambda *a, **k: logged.append(("debug", a)), + info=lambda *a, **k: logged.append(("info", a)), + warning=lambda *a, **k: logged.append(("warning", a)), + ) + stub = SimpleNamespace(tvdb_id=tvdb_id, log=logger, daily=daily, service_daily=service_daily) + stub.daily_hint = lambda title=None: dl.daily_hint(stub, title) + series = Series(titles) + dl.fill_air_dates(stub, series, tvdb_id) + return series, logged + + +def tvdb_episode(season: int, number: int, aired) -> dict: + return {"seasonNumber": season, "number": number, "aired": aired} + + +LISTING = [tvdb_episode(1, 1, "2026-08-10"), tvdb_episode(1, 2, "2026-08-11")] + + +def test_fill_populates_missing_air_dates(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(number=1), make_episode(number=2)] + series, logged = fill(monkeypatch, episodes, LISTING) + assert [t.air_date for t in series] == [date(2026, 8, 10), date(2026, 8, 11)] + assert [level for level, _ in logged] == ["info"] + + +def test_a_service_set_air_date_wins(monkeypatch: pytest.MonkeyPatch) -> None: + episodes = [make_episode(number=1, air_date="1999-01-02"), make_episode(number=2)] + series, _ = fill(monkeypatch, episodes, LISTING) + assert [t.air_date for t in series] == [date(1999, 1, 2), date(2026, 8, 11)] + + +def test_fill_never_touches_season_or_number(monkeypatch: pytest.MonkeyPatch) -> None: + series, _ = fill(monkeypatch, [make_episode(number=2)], LISTING) + assert [(t.season, t.number) for t in series] == [(1, 2)] + + +def test_an_unmapped_episode_keeps_no_air_date(monkeypatch: pytest.MonkeyPatch) -> None: + series, _ = fill(monkeypatch, [make_episode(season=9, number=9)], LISTING) + assert series[0].air_date is None + + +@pytest.mark.parametrize("aired", [None, "", "not-a-date", "2026-13-40"]) +def test_an_unusable_aired_value_is_skipped(monkeypatch: pytest.MonkeyPatch, aired) -> None: + series, _ = fill(monkeypatch, [make_episode(number=1)], [tvdb_episode(1, 1, aired)]) + assert series[0].air_date is None + + +def test_a_pre_1970_date_is_skipped(monkeypatch: pytest.MonkeyPatch) -> None: + series, _ = fill(monkeypatch, [make_episode(number=1)], [tvdb_episode(1, 1, "1900-01-01")]) + assert series[0].air_date is None + + +def test_a_future_placeholder_date_is_skipped(monkeypatch: pytest.MonkeyPatch) -> None: + future = (date.today() + timedelta(days=30)).isoformat() + series, _ = fill(monkeypatch, [make_episode(number=1)], [tvdb_episode(1, 1, future)]) + assert series[0].air_date is None + + +def test_tomorrow_still_fills(monkeypatch: pytest.MonkeyPatch) -> None: + tomorrow = date.today() + timedelta(days=1) + series, _ = fill(monkeypatch, [make_episode(number=1)], [tvdb_episode(1, 1, tomorrow.isoformat())]) + assert series[0].air_date == tomorrow + + +def test_a_datetime_stamped_aired_value_keeps_the_date(monkeypatch: pytest.MonkeyPatch) -> None: + series, _ = fill(monkeypatch, [make_episode(number=1)], [tvdb_episode(1, 1, "2026-08-10T20:00:00Z")]) + assert series[0].air_date == date(2026, 8, 10) + + +def test_fill_skips_silently_without_an_api_key(monkeypatch: pytest.MonkeyPatch) -> None: + series, logged = fill(monkeypatch, [make_episode(number=1)], LISTING, provider=False) + assert series[0].air_date is None + assert [level for level, _ in logged] == ["debug"] + + +def test_fill_skips_silently_without_a_tvdb_id(monkeypatch: pytest.MonkeyPatch) -> None: + series, logged = fill(monkeypatch, [make_episode(number=1)], LISTING, tvdb_id=None) + assert series[0].air_date is None + assert [level for level, _ in logged] == ["debug"] + + +def test_fill_skips_silently_when_tvdb_lists_no_episodes(monkeypatch: pytest.MonkeyPatch) -> None: + series, logged = fill(monkeypatch, [make_episode(number=1)], []) + assert series[0].air_date is None + assert [level for level, _ in logged] == ["debug"] + + +def test_nothing_is_filled_when_no_title_is_daily(monkeypatch: pytest.MonkeyPatch) -> None: + series, logged = fill(monkeypatch, [make_episode(number=1)], LISTING, daily=False) + assert series[0].air_date is None + assert logged == [] + + +def test_the_service_daily_flag_fills_without_the_cli_flag(monkeypatch: pytest.MonkeyPatch) -> None: + series, _ = fill(monkeypatch, [make_episode(number=1)], LISTING, daily=False, service_daily=True) + assert series[0].air_date == date(2026, 8, 10) + + +def test_a_per_title_daily_false_is_skipped_while_others_fill(monkeypatch: pytest.MonkeyPatch) -> None: + first, second = make_episode(number=1), make_episode(number=2) + first.daily = False + series, _ = fill(monkeypatch, [first, second], LISTING, daily=False, service_daily=True) + assert [t.air_date for t in series] == [None, date(2026, 8, 11)] + + +def test_the_cli_flag_overrides_a_per_title_daily_false(monkeypatch: pytest.MonkeyPatch) -> None: + episode = make_episode(number=1) + episode.daily = False + series, _ = fill(monkeypatch, [episode], LISTING, daily=True) + assert series[0].air_date == date(2026, 8, 10) diff --git a/tests/core/test_imdb_provider.py b/tests/core/test_imdb_provider.py new file mode 100644 index 00000000..e8a06b3f --- /dev/null +++ b/tests/core/test_imdb_provider.py @@ -0,0 +1,183 @@ +"""IMDb GraphQL provider: persisted-query flow, field mapping, and search constraints.""" + +from __future__ import annotations + +import json +from typing import Any, Optional + +import pytest + +from unshackle.core.providers.imdb import IMDBProvider, escape_graphql + + +class _Response: + def __init__(self, payload: dict) -> None: + self._payload = payload + + def json(self) -> dict: + return self._payload + + +def node( + imdb_id: str = "tt3581920", + text: str = "The Last of Us", + year: Optional[int] = 2023, + title_type: str = "tvSeries", + languages: Any = ({"id": "en", "text": "English"}, {"id": "id", "text": "Indonesian"}), +) -> dict: + return { + "id": imdb_id, + "titleText": {"text": text}, + "originalTitleText": {"text": text}, + "releaseYear": {"year": year} if year else None, + "titleType": {"id": title_type}, + "spokenLanguages": {"spokenLanguages": list(languages)} if languages is not None else None, + } + + +class _FakeSession: + """Records GET/POST calls and replays queued payloads.""" + + def __init__(self, responses: list[dict]) -> None: + self._responses = responses + self.gets: list[dict] = [] + self.posts: list[dict] = [] + + def get(self, url: str, params: dict, headers: dict, timeout: int) -> _Response: + self.gets.append(params) + return _Response(self._responses.pop(0)) + + def post(self, url: str, json: dict, headers: dict, timeout: int) -> _Response: # noqa: A002 + self.posts.append(json) + return _Response(self._responses.pop(0)) + + +def _provider(monkeypatch: pytest.MonkeyPatch, responses: list[dict]) -> tuple[IMDBProvider, _FakeSession]: + p = IMDBProvider() + session = _FakeSession(responses) + monkeypatch.setattr(type(p), "session", property(lambda _self: session)) + return p, session + + +NOT_FOUND = {"errors": [{"message": "PersistedQueryNotFound", "extensions": {"code": "PERSISTED_QUERY_NOT_FOUND"}}]} + + +def test_is_available_without_any_key() -> None: + assert IMDBProvider().is_available() is True + + +def test_cache_hit_uses_one_get(monkeypatch: pytest.MonkeyPatch) -> None: + p, session = _provider(monkeypatch, [{"data": {"title": node()}}]) + result = p.get_by_id("tt3581920", "tv") + assert result is not None and result.title == "The Last of Us" + assert len(session.gets) == 1 and session.posts == [] + + +def test_persisted_query_miss_falls_back_to_one_post(monkeypatch: pytest.MonkeyPatch) -> None: + p, session = _provider(monkeypatch, [NOT_FOUND, {"data": {"title": node()}}]) + result = p.get_by_id("tt3581920", "tv") + assert result is not None and result.external_ids.imdb_id == "tt3581920" + assert len(session.gets) == 1 and len(session.posts) == 1 + # the POST registers the query by sending its text, keyed by the same hash the GET sent + assert "query UnshackleTitle" in session.posts[0]["query"] + sent_hash = json.loads(session.gets[0]["extensions"])["persistedQuery"]["sha256Hash"] + assert session.posts[0]["extensions"]["persistedQuery"]["sha256Hash"] == sent_hash + + +def test_get_params_carry_no_spaces(monkeypatch: pytest.MonkeyPatch) -> None: + """IMDb's GraphQL gateway decodes "+" literally, so spaced JSON would come back as a 400.""" + p, session = _provider(monkeypatch, [{"data": {"title": node()}}]) + p.get_by_id("tt3581920", "tv") + assert " " not in session.gets[0]["extensions"] + assert " " not in session.gets[0]["variables"] + + +def test_get_by_id_maps_every_field(monkeypatch: pytest.MonkeyPatch) -> None: + p, _ = _provider(monkeypatch, [{"data": {"title": node()}}]) + result = p.get_by_id("tt3581920", "tv") + assert result is not None + assert (result.title, result.year, result.kind) == ("The Last of Us", 2023, "tv") + assert result.original_language == "en" + assert result.source == "imdb" + assert result.external_ids.imdb_id == "tt3581920" + + +@pytest.mark.parametrize("title_type", ["movie", "tvSeries", "videoGame"]) +def test_caller_kind_wins_over_the_response_title_type(monkeypatch: pytest.MonkeyPatch, title_type: str) -> None: + """kind stays the caller's, so it cannot desync from the tmdb_kind resolve_by_ids seeds.""" + p, _ = _provider(monkeypatch, [{"data": {"title": node(title_type=title_type)}}]) + result = p.get_by_id("tt1375666", "tv") + assert result is not None and result.kind == "tv" + + +@pytest.mark.parametrize("payload", [{"data": {"title": None}}, {"errors": [{"message": "boom"}]}, {"data": {}}]) +def test_get_by_id_returns_none_on_no_answer(monkeypatch: pytest.MonkeyPatch, payload: dict) -> None: + p, _ = _provider(monkeypatch, [payload]) + assert p.get_by_id("tt3581920", "tv") is None + + +def edges(*nodes: dict) -> dict: + return {"data": {"advancedTitleSearch": {"edges": [{"node": {"title": n}} for n in nodes]}}} + + +def test_search_uses_its_own_operation_and_one_get_on_a_cache_hit(monkeypatch: pytest.MonkeyPatch) -> None: + p, session = _provider(monkeypatch, [edges(node())]) + result = p.search("The Last of Us", 2023, "tv") + assert result is not None and result.external_ids.imdb_id == "tt3581920" + assert session.gets[0]["operationName"] == "UnshackleTitleSearch" + assert session.posts == [] + + +def test_search_query_text_carries_the_constraints(monkeypatch: pytest.MonkeyPatch) -> None: + p, session = _provider(monkeypatch, [NOT_FOUND, edges(node())]) + p.search("The Last of Us", 2023, "tv") + query = session.posts[0]["query"] + assert 'searchTerm: "The Last of Us"' in query + assert 'start: "2023-01-01", end: "2023-12-31"' in query + assert '["tvSeries", "tvMiniSeries"]' in query or '["tvSeries","tvMiniSeries"]' in query + + +def test_search_omits_the_year_range_when_unknown(monkeypatch: pytest.MonkeyPatch) -> None: + p, session = _provider(monkeypatch, [NOT_FOUND, edges(node(title_type="movie"))]) + p.search("Inception", None, "movie") + query = session.posts[0]["query"] + assert "releaseDateConstraint" not in query + assert '["movie", "tvMovie"]' in query or '["movie","tvMovie"]' in query + + +def test_search_picks_the_closest_title_not_the_first(monkeypatch: pytest.MonkeyPatch) -> None: + noise = node(imdb_id="tt27478132", text="Making of: The Last of Us") + exact = node() + p, _ = _provider(monkeypatch, [edges(noise, exact)]) + result = p.search("The Last of Us", 2023, "tv") + assert result is not None and result.external_ids.imdb_id == "tt3581920" + + +def test_search_rejects_a_title_that_is_not_a_close_match(monkeypatch: pytest.MonkeyPatch) -> None: + p, _ = _provider(monkeypatch, [edges(node(imdb_id="tt999", text="Something Else Entirely"))]) + assert p.search("The Last of Us", 2023, "tv") is None + + +def test_search_survives_a_null_language_block(monkeypatch: pytest.MonkeyPatch) -> None: + p, _ = _provider(monkeypatch, [edges(node(languages=None))]) + result = p.search("The Last of Us", 2023, "tv") + assert result is not None and result.original_language is None + + +def test_search_returns_none_when_nothing_comes_back(monkeypatch: pytest.MonkeyPatch) -> None: + p, _ = _provider(monkeypatch, [edges()]) + assert p.search("The Last of Us", 2023, "tv") is None + + +def test_escape_graphql_closes_the_string_injection() -> None: + assert escape_graphql('a"b\\c') == 'a\\"b\\\\c' + + +def test_search_term_with_a_quote_stays_inside_the_string(monkeypatch: pytest.MonkeyPatch) -> None: + p, session = _provider(monkeypatch, [NOT_FOUND, edges(node(text='The "Best" Show'))]) + p.search('The "Best" Show', None, "tv") + assert 'searchTerm: "The \\"Best\\" Show"' in session.posts[0]["query"] + + +def test_get_external_ids_echoes_the_id() -> None: + assert IMDBProvider().get_external_ids("tt1375666", "movie").imdb_id == "tt1375666" diff --git a/tests/core/test_ism_drm_systemid.py b/tests/core/test_ism_drm_systemid.py new file mode 100644 index 00000000..56615968 --- /dev/null +++ b/tests/core/test_ism_drm_systemid.py @@ -0,0 +1,55 @@ +"""ProtectionHeader SystemID parsing. + +MS-SSTR manifests commonly write the GUID in registry format +("{EDEF8BA9-...}"); a bare-GUID comparison silently drops the DRM. +""" + +from __future__ import annotations + +from uuid import UUID + +import pytest +from pywidevine import PSSH + +from unshackle.core.drm import Widevine +from unshackle.core.manifests.ism import ISM + +WIDEVINE_PSSH_B64 = PSSH.new(key_ids=["00112233445566778899aabbccddeeff"], system_id=PSSH.SystemId.Widevine).dumps() + +MANIFEST = """ + + + {pssh} + + + + + + +""" + + +@pytest.mark.parametrize( + "system_id", + [ + "EDEF8BA9-79D6-4ACE-A3C8-27DCD51D21ED", + "{EDEF8BA9-79D6-4ACE-A3C8-27DCD51D21ED}", + ], +) +def test_get_drm_resolves_braced_and_bare_system_ids(system_id): + manifest = MANIFEST.format(system_id=system_id, pssh=WIDEVINE_PSSH_B64) + ism = ISM.from_text(manifest, "https://example.test/Manifest") + drm = ISM.get_drm(ism.manifest.xpath(".//ProtectionHeader")) + assert len(drm) == 1 + assert isinstance(drm[0], Widevine) + assert drm[0].kid == UUID("00112233-4455-6677-8899-aabbccddeeff") + + +def test_get_drm_warns_on_unparseable_header(caplog): + manifest = MANIFEST.format(system_id="EDEF8BA9-79D6-4ACE-A3C8-27DCD51D21ED", pssh="!!!not-base64!!!") + ism = ISM.from_text(manifest, "https://example.test/Manifest") + with caplog.at_level("WARNING", logger="ISM"): + drm = ISM.get_drm(ism.manifest.xpath(".//ProtectionHeader")) + assert drm == [] + assert any("unparseable Widevine" in r.message for r in caplog.records) diff --git a/tests/core/test_ism_init.py b/tests/core/test_ism_init.py index 0b9378eb..4d149468 100644 --- a/tests/core/test_ism_init.py +++ b/tests/core/test_ism_init.py @@ -581,7 +581,7 @@ def test_hvcc_profile_tier_level_is_nonzero(): assert level_idc != 0 -def _piff_fragment(flags: int = 0x0, override: bool = False) -> bytes: +def piff_fragment(flags: int = 0x0, override: bool = False) -> bytes: """Minimal moof+mdat fragment carrying sample encryption in the PIFF uuid box.""" ivs = b"".join(bytes([i]) * 8 for i in range(1, 3)) senc_payload = struct.pack(">I", 2) + ivs @@ -599,7 +599,7 @@ def _piff_fragment(flags: int = 0x0, override: bool = False) -> bytes: def test_piff_senc_rewritten_to_cenc_senc(): - frag = _piff_fragment() + frag = piff_fragment() out = piff_senc_to_cenc(frag) # trun's data_offset is moof-relative; shrinking the box misaims decryption assert len(out) == len(frag) @@ -612,7 +612,7 @@ def test_piff_senc_rewritten_to_cenc_senc(): def test_piff_senc_override_header_stripped(): - frag = _piff_fragment(override=True) + frag = piff_fragment(override=True) out = piff_senc_to_cenc(frag) assert len(out) == len(frag) assert PIFF_SENC_UUID not in out @@ -624,7 +624,7 @@ def test_piff_senc_override_header_stripped(): def test_piff_subsample_flag_survives_override_strip(): # losing 0x2 is silent: no subsample map, so clear NAL headers get encrypted too - frag = _piff_fragment(flags=0x2, override=True) + frag = piff_fragment(flags=0x2, override=True) out = piff_senc_to_cenc(frag) senc = out.index(b"senc") flags = int.from_bytes(out[senc + 5 : senc + 8], "big") @@ -648,7 +648,7 @@ def test_piff_rewrite_skips_traf_that_already_has_senc(): def test_piff_rewrite_covers_every_moof_and_traf(): - frag = _piff_fragment() + _piff_fragment() + frag = piff_fragment() + piff_fragment() out = piff_senc_to_cenc(frag) assert len(out) == len(frag) assert PIFF_SENC_UUID not in out @@ -683,8 +683,8 @@ def traf_children_tile_exactly(frag: bytes) -> bool: def test_piff_rewrite_keeps_traf_children_tiling_exactly(): # a mis-sized senc still "decrypts", just off the wrong bytes, so assert the sizes too - assert traf_children_tile_exactly(_piff_fragment()) - for frag in (_piff_fragment(), _piff_fragment(flags=0x2), _piff_fragment(flags=0x2, override=True)): + assert traf_children_tile_exactly(piff_fragment()) + for frag in (piff_fragment(), piff_fragment(flags=0x2), piff_fragment(flags=0x2, override=True)): assert traf_children_tile_exactly(piff_senc_to_cenc(frag)), "rewritten traf no longer tiles" @@ -700,7 +700,7 @@ def test_piff_rewrite_refuses_override_iv_size_mismatch(): def test_piff_rewrite_never_grows_output_on_malformed_boxes(): # A declared size running past the buffer would extend the output over the mdat. - good = _piff_fragment() + good = piff_fragment() truncations = [good[:n] for n in range(8, len(good), 7)] oversized = bytearray(good) struct.pack_into(">I", oversized, 0, len(good) + 4096) # moof claims more than exists @@ -709,7 +709,7 @@ def test_piff_rewrite_never_grows_output_on_malformed_boxes(): assert len(out) == len(frag), f"output grew: {len(frag)} -> {len(out)}" -def _text_fragment(seq: int, begin: str, end: str, text: str, sdi: int) -> bytes: +def text_fragment(seq: int, begin: str, end: str, text: str, sdi: int) -> bytes: """A single-sample fTTML moof+mdat, mirroring what a Smooth text stream serves.""" payload = ( "" @@ -771,7 +771,7 @@ def test_pymp4_consumes_whole_init_and_reaches_fragments(): init = build_init_segment( stream_type="text", fourcc="TTML", codec_private_data="", duration=6_000_000_000, language="eng" ) - data = init + _text_fragment(1, "00:00:01.000", "00:00:02.000", "first", sdi=1) + data = init + text_fragment(1, "00:00:01.000", "00:00:02.000", "first", sdi=1) types = [b.type for b in MP4.parse_stream(BytesIO(data))] assert types == [b"ftyp", b"moov", b"moof", b"mdat"] @@ -784,8 +784,8 @@ def test_fttml_subtitle_parse_yields_cues(): ) data = ( init - + _text_fragment(1, "00:00:01.000", "00:00:02.000", "first", sdi=1) - + _text_fragment(2, "00:00:03.000", "00:00:04.000", "second", sdi=2) + + text_fragment(1, "00:00:01.000", "00:00:02.000", "first", sdi=1) + + text_fragment(2, "00:00:03.000", "00:00:04.000", "second", sdi=2) ) caption_set = Subtitle.parse(data, Subtitle.Codec.fTTML) cues = [c for lang in caption_set.get_languages() for c in caption_set.get_captions(lang)] diff --git a/tests/core/test_ism_parse.py b/tests/core/test_ism_parse.py index 0f0aa28b..80bb27fc 100644 --- a/tests/core/test_ism_parse.py +++ b/tests/core/test_ism_parse.py @@ -28,6 +28,17 @@ def test_placeholder_spellings_repeat_and_audiotag_fallback() -> None: assert audio.codec == Audio.Codec.AAC # AudioTag 255 means AACL +def test_duration_fallback_when_next_fragment_omits_time() -> None: + # A d-less whose successor also omits t must fall back to the manifest + # duration (int(None) used to raise TypeError past the except clause). + sparse = MANIFEST.replace('', '') + tracks = ISM.from_text(sparse, "https://cdn.example/x.ism/manifest").to_tracks() + assert tracks.audio[0].data["ism"]["segments"] == [ + "https://cdn.example/x.ism/QualityLevels(128000)/Fragments(audio=0)", + "https://cdn.example/x.ism/QualityLevels(128000)/Fragments(audio=60000000)", + ] + + def test_missing_language_without_fallback_raises() -> None: # MS-SSTR Language is optional; video StreamIndexes commonly omit it. no_lang = MANIFEST.replace(' Language="en"', "") diff --git a/tests/core/test_lang_exclusions.py b/tests/core/test_lang_exclusions.py new file mode 100644 index 00000000..8dd8be4a --- /dev/null +++ b/tests/core/test_lang_exclusions.py @@ -0,0 +1,80 @@ +"""Tests for the `-` exclusion prefix on the language flags (-l, -vl, -al, -sl, -fsl).""" + +from __future__ import annotations + +import pytest +from langcodes import Language + +from unshackle.core.utilities import excluded_language_tags, partition_exclusions +from unshackle.core.utils.click_types import LANGUAGE_RANGE + +# --- partition_exclusions --------------------------------------------------- + + +@pytest.mark.parametrize( + "tokens,expected", + [ + (["all", "-es"], (["all"], ["es"])), + (["-es"], ([], ["es"])), + (["en", "ja"], (["en", "ja"], [])), + (["-es", "-fr", "en"], (["en"], ["es", "fr"])), + ([], ([], [])), + (None, ([], [])), + ], +) +def test_partition_splits_on_the_leading_dash(tokens, expected): + assert partition_exclusions(tokens) == expected + + +def test_partition_keeps_the_written_order_and_drops_repeats(): + assert partition_exclusions(["ja", "en", "ja", "-es", "-es"]) == (["ja", "en"], ["es"]) + + +@pytest.mark.parametrize("token", ["-", " - ", "", " "]) +def test_partition_skips_tokens_with_no_value(token): + assert partition_exclusions(["en", token]) == (["en"], []) + + +def test_partition_trims_whitespace_around_a_token(): + assert partition_exclusions([" en ", "- es "]) == (["en"], ["es"]) + + +def test_partition_reads_the_tokens_language_range_produces(): + assert partition_exclusions(LANGUAGE_RANGE.convert("all,-es;-fr")) == (["all"], ["es", "fr"]) + + +# --- excluded_language_tags --------------------------------------------------- + + +def test_close_match_excludes_the_regional_variants(): + assert excluded_language_tags(["es"], ["es-419", "es-ES", "en-US"]) == {"es-419", "es-ES"} + + +def test_exact_match_prefers_the_string_equal_tag(): + # same string preference as selection: an exact 'es' only takes 'es' when it exists + assert excluded_language_tags(["es"], ["es", "es-ES", "es-419"], exact=True) == {"es"} + + +def test_exact_match_falls_back_to_the_paradigm_variant(): + # no literal 'es' available, so 'es' resolves the way an exact '-sl es' selection would + assert excluded_language_tags(["es"], ["es-ES", "es-419"], exact=True) == {"es-ES"} + + +def test_exact_match_keeps_an_unnamed_regional_variant(): + assert excluded_language_tags(["es-ES"], ["es-ES", "es-419"], exact=True) == {"es-ES"} + + +def test_language_objects_are_accepted(): + assert excluded_language_tags(["es"], [Language.get("es")]) == {"es"} + + +def test_an_untagged_track_is_never_excluded(): + assert excluded_language_tags(["es"], [None, "en-US"]) == set() + + +def test_no_exclusions_excludes_nothing(): + assert excluded_language_tags([], ["es-419", "en-US"]) == set() + + +def test_an_unmatched_language_is_kept(): + assert excluded_language_tags(["es", "fr"], ["en-US", "ja-JP"]) == set() diff --git a/tests/core/test_metadata_id_validation.py b/tests/core/test_metadata_id_validation.py new file mode 100644 index 00000000..db13890e --- /dev/null +++ b/tests/core/test_metadata_id_validation.py @@ -0,0 +1,183 @@ +"""Input validation for the --tmdb / --imdb / --tvdb flags.""" + +from __future__ import annotations + +import click +import pytest + +from unshackle.commands.dl import validate_metadata_ids +from unshackle.core.api.handlers import validate_download_parameters +from unshackle.core.config import config + + +@pytest.fixture(autouse=True) +def default_order(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(config, "metadata_providers", []) + + +def with_keys(monkeypatch: pytest.MonkeyPatch, **keys: str) -> None: + for name in ("tmdb_api_key", "tvdb_api_key", "omdb_api_key", "simkl_client_id"): + monkeypatch.setattr(config, name, keys.get(name, "")) + + +# ---------- one ID at a time ---------- + + +@pytest.mark.parametrize( + ("tmdb", "imdb", "tvdb"), + [(27205, "tt1375666", None), (27205, None, 73871), (None, "tt1375666", 73871), (27205, "tt1375666", 73871)], +) +def test_two_ids_are_refused(tmdb: int | None, imdb: str | None, tvdb: int | None) -> None: + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(tmdb, imdb, tvdb) + assert "cannot be used together" in str(exc.value) + + +def test_no_ids_is_fine() -> None: + assert validate_metadata_ids(None, None, None) is None + + +def test_imdb_alone_passes_on_the_keyless_provider(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch) + assert validate_metadata_ids(None, "tt1375666", None) is None + + +# ---------- the ID must have a provider that can resolve it ---------- + + +def test_tmdb_without_its_key_names_the_key(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(27205, None, None) + assert "--tmdb" in str(exc.value) and "tmdb_api_key" in str(exc.value) + + +def test_tvdb_without_its_key_names_the_key(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(None, None, 73871) + assert "--tvdb" in str(exc.value) and "tvdb_api_key" in str(exc.value) + + +def test_tmdb_with_its_key_passes(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch, tmdb_api_key="k") + assert validate_metadata_ids(27205, None, None) is None + + +def test_a_provider_filtered_out_of_the_order_points_at_the_config(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch, tmdb_api_key="k") + monkeypatch.setattr(config, "metadata_providers", ["imdb", "omdb"]) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(27205, None, None) + assert "metadata_providers" in str(exc.value) + + +def test_imdb_is_unresolvable_when_the_order_drops_both_of_its_providers(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch) + monkeypatch.setattr(config, "metadata_providers", ["tmdb", "tvdb"]) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(None, "tt1375666", None) + assert "metadata_providers" in str(exc.value) + assert "imdb" in str(exc.value) and "omdb" in str(exc.value) + + +def test_one_kind_keeping_the_provider_is_enough(monkeypatch: pytest.MonkeyPatch) -> None: + """A per-kind order can drop tvdb for movies and keep it for tv, so this must not error.""" + with_keys(monkeypatch, tvdb_api_key="k") + monkeypatch.setattr(config, "metadata_providers", {"tv": ["tvdb"], "movie": ["tmdb"]}) + assert validate_metadata_ids(None, None, 73871) is None + + +# ---------- the REST path rejects the same combinations ---------- + + +@pytest.mark.parametrize( + "payload", + [ + {"tmdb_id": 27205, "imdb_id": "tt1375666"}, + {"tmdb_id": 27205, "tvdb_id": 73871}, + {"imdb_id": "tt1375666", "tvdb_id": 73871}, + ], +) +def test_api_refuses_two_ids(payload: dict) -> None: + err = validate_download_parameters(payload) + assert err and "multiple external IDs" in err + + +@pytest.mark.parametrize("payload", [{"tmdb_id": 27205}, {"imdb_id": "tt1375666"}, {"tvdb_id": 73871}, {}]) +def test_api_accepts_one_id(payload: dict) -> None: + assert validate_download_parameters(payload) is None + + +def test_api_still_reports_a_malformed_id_first() -> None: + err = validate_download_parameters({"tmdb_id": "12345", "imdb_id": "tt1"}) + assert err and "positive integer" in err + + +# ---------- --anilist stays outside the mutual exclusion ---------- + + +@pytest.mark.parametrize("value", [21, "21", "mal:21"]) +def test_anilist_alone_passes_without_any_key(monkeypatch: pytest.MonkeyPatch, value: object) -> None: + with_keys(monkeypatch) + assert validate_metadata_ids(None, None, None, value) is None + + +def test_anilist_pairs_with_one_other_id(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch) + assert validate_metadata_ids(None, "tt1375666", None, 21) is None + + +def test_anilist_does_not_excuse_two_western_ids(monkeypatch: pytest.MonkeyPatch) -> None: + with_keys(monkeypatch) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(27205, "tt1375666", None, 21) + assert "cannot be used together" in str(exc.value) + + +@pytest.mark.parametrize("value", ["tt1375666", "anilist:21", "mal:", "abc", "-3"]) +def test_a_malformed_anilist_value_is_refused(monkeypatch: pytest.MonkeyPatch, value: str) -> None: + with_keys(monkeypatch) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(None, None, None, value) + assert "--anilist" in str(exc.value) + + +@pytest.mark.parametrize("payload", [{"anilist_id": 21}, {"anilist_id": "mal:21"}, {"anilist_id": "21"}]) +def test_api_accepts_an_anilist_id(payload: dict) -> None: + assert validate_download_parameters(payload) is None + + +def test_api_accepts_anilist_alongside_one_western_id() -> None: + assert validate_download_parameters({"imdb_id": "tt1375666", "anilist_id": 21}) is None + + +def test_api_still_refuses_two_western_ids_with_anilist() -> None: + err = validate_download_parameters({"tmdb_id": 27205, "imdb_id": "tt1375666", "anilist_id": 21}) + assert err and "multiple external IDs" in err + + +@pytest.mark.parametrize("value", [0, -3, True, "mal:", "tt1375666", "anilist:21", 1.5]) +def test_api_refuses_a_malformed_anilist_id(value: object) -> None: + err = validate_download_parameters({"anilist_id": value}) + assert err and "anilist_id" in err + + +@pytest.mark.parametrize("value", ["0", "mal:0", "²", "21.5"]) +def test_a_non_positive_or_non_decimal_anilist_value_is_refused(monkeypatch: pytest.MonkeyPatch, value: str) -> None: + with_keys(monkeypatch) + with pytest.raises(click.UsageError) as exc: + validate_metadata_ids(None, None, None, value) + assert "--anilist" in str(exc.value) + + +@pytest.mark.parametrize("payload", [{"anilist_id": "MAL:21"}, {"anilist_id": "007"}, {"anilist_id": " 21 "}]) +def test_api_accepts_what_the_cli_accepts(payload: dict) -> None: + """Both surfaces share parse_anilist_ref, so their accept sets cannot drift.""" + assert validate_download_parameters(payload) is None + + +@pytest.mark.parametrize("value", ["0", "mal:0", "²"]) +def test_api_refuses_what_the_cli_refuses(value: str) -> None: + err = validate_download_parameters({"anilist_id": value}) + assert err and "anilist_id" in err diff --git a/tests/core/test_resolve_by_ids.py b/tests/core/test_resolve_by_ids.py new file mode 100644 index 00000000..a9821c52 --- /dev/null +++ b/tests/core/test_resolve_by_ids.py @@ -0,0 +1,259 @@ +"""Precedence of user-supplied external IDs in providers.resolve_by_ids.""" + +from __future__ import annotations + +from typing import Optional, Union + +import pytest + +from unshackle.core import providers +from unshackle.core.config import config +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult + + +class FakeProvider(MetadataProvider): + """Stands in for a real provider; records the ID it was asked for.""" + + def __init__(self, name: str, result: Optional[MetadataResult] = None, raises: bool = False) -> None: + self.NAME = name + self._result = result + self._raises = raises + self.asked: list[Union[int, str]] = [] + super().__init__() + + def is_available(self) -> bool: + return True + + def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: + raise AssertionError("resolve_by_ids must not search when an ID was supplied") + + def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: + self.asked.append(provider_id) + if self._raises: + raise ValueError("provider is down") + return self._result + + def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: + return ExternalIds() + + +@pytest.fixture(autouse=True) +def isolate(monkeypatch: pytest.MonkeyPatch) -> None: + """Keep every test off the network and on the default provider order.""" + monkeypatch.setattr(config, "metadata_providers", []) + monkeypatch.setattr(providers, "fetch_external_ids", lambda *a, **kw: ExternalIds()) + monkeypatch.setattr(providers, "enrich_ids", lambda result: None) + monkeypatch.setattr( + providers, + "search_metadata", + lambda *a, **kw: pytest.fail("search_metadata must not run when an ID was supplied"), + ) + + +def patch_providers(monkeypatch: pytest.MonkeyPatch, available: dict[str, FakeProvider]) -> None: + monkeypatch.setattr(providers, "get_provider", lambda name: available.get(name)) + + +def test_supplied_ids_survive_a_providers_own_answer(monkeypatch: pytest.MonkeyPatch) -> None: + # tmdb answers with a different imdb id; the one the user gave must win + tmdb = FakeProvider( + "tmdb", + MetadataResult(title="Inception", year=2010, kind="movie", external_ids=ExternalIds(imdb_id="tt9999999")), + ) + patch_providers(monkeypatch, {"tmdb": tmdb}) + + result = providers.resolve_by_ids(27205, "tt1375666", kind="movie") + + assert result is not None + assert result.external_ids.tmdb_id == 27205 + assert result.external_ids.tmdb_kind == "movie" + assert result.external_ids.imdb_id == "tt1375666" + + +def test_imdb_falls_through_to_omdb(monkeypatch: pytest.MonkeyPatch) -> None: + imdb = FakeProvider("imdb", raises=True) + omdb = FakeProvider("omdb", MetadataResult(title="Inception", year=2010, kind="movie")) + patch_providers(monkeypatch, {"imdb": imdb, "omdb": omdb}) + + result = providers.resolve_by_ids(imdb_id="tt1375666", kind="movie") + + assert result is not None + assert result.title == "Inception" + assert imdb.asked == ["tt1375666"] + assert omdb.asked == ["tt1375666"] + + +def test_all_providers_down_synthesizes_a_result(monkeypatch: pytest.MonkeyPatch) -> None: + patch_providers(monkeypatch, {}) + + result = providers.resolve_by_ids(tvdb_id=73871, title="Futurama", year=1999, kind="tv") + + assert result is not None + assert result.title == "Futurama" + assert result.year == 1999 + assert result.external_ids.tvdb_id == 73871 + + +def test_no_ids_searches(monkeypatch: pytest.MonkeyPatch) -> None: + calls: list[tuple] = [] + monkeypatch.setattr(providers, "search_metadata", lambda *a, **kw: calls.append(a) or None) + patch_providers(monkeypatch, {}) + + providers.resolve_by_ids(title="Futurama", year=1999, kind="tv") + + assert calls and calls[0][:3] == ("Futurama", 1999, "tv") + + +def test_no_ids_and_no_title_returns_none(monkeypatch: pytest.MonkeyPatch) -> None: + patch_providers(monkeypatch, {}) + assert providers.resolve_by_ids(kind="tv") is None + + +def test_per_kind_order_puts_tvdb_first_for_tv(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(config, "metadata_providers", {"tv": ["tvdb", "tmdb", "imdb", "omdb"]}) + tvdb = FakeProvider("tvdb", MetadataResult(title="Futurama", year=1999, kind="tv")) + tmdb = FakeProvider("tmdb", MetadataResult(title="Wrong Show", year=2020, kind="tv")) + patch_providers(monkeypatch, {"tvdb": tvdb, "tmdb": tmdb}) + + result = providers.resolve_by_ids(615, tvdb_id=73871, kind="tv") + + assert result is not None + assert result.title == "Futurama" + assert tmdb.asked == [] + # movies keep the default order, so tmdb is consulted before tvdb there + assert [cls.NAME for cls in providers.provider_order("movie")] == list(providers.DEFAULT_ORDER) + + +def test_flat_list_config_applies_to_both_kinds(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(config, "metadata_providers", ["tvdb", "tmdb"]) + assert [cls.NAME for cls in providers.provider_order("tv")] == ["tvdb", "tmdb"] + assert [cls.NAME for cls in providers.provider_order("movie")] == ["tvdb", "tmdb"] + assert [cls.NAME for cls in providers.provider_order()] == ["tvdb", "tmdb"] + + +# ---------- the anilist namespace ---------- + + +def test_an_anilist_id_reaches_the_anilist_provider(monkeypatch: pytest.MonkeyPatch) -> None: + anilist = FakeProvider( + "anilist", + MetadataResult(title="One Piece", year=1999, kind="tv", external_ids=ExternalIds(anilist_id=21)), + ) + tmdb = FakeProvider("tmdb", MetadataResult(title="Wrong Show", year=2020, kind="tv")) + patch_providers(monkeypatch, {"anilist": anilist, "tmdb": tmdb}) + + result = providers.resolve_by_ids(anilist_id=21, kind="tv") + + assert result is not None + assert result.title == "One Piece" + assert result.external_ids.anilist_id == 21 + assert anilist.asked == [21] + assert tmdb.asked == [] + + +def test_a_supplied_anilist_id_survives_a_provider_miss(monkeypatch: pytest.MonkeyPatch) -> None: + patch_providers(monkeypatch, {}) + + result = providers.resolve_by_ids(anilist_id="21", title="One Piece", kind="tv") + + assert result is not None + assert result.external_ids.anilist_id == 21 + + +def test_a_mal_reference_carries_only_what_the_lookup_answered(monkeypatch: pytest.MonkeyPatch) -> None: + anilist = FakeProvider("anilist", MetadataResult(title="One Piece", external_ids=ExternalIds(anilist_id=21))) + patch_providers(monkeypatch, {"anilist": anilist}) + + result = providers.resolve_by_ids(anilist_id="mal:21", kind="tv") + + assert result is not None + assert anilist.asked == ["mal:21"] + assert result.external_ids.anilist_id == 21 + + +def test_a_mal_reference_that_misses_leaves_no_anilist_id(monkeypatch: pytest.MonkeyPatch) -> None: + patch_providers(monkeypatch, {}) + + result = providers.resolve_by_ids(anilist_id="mal:21", title="One Piece", kind="tv") + + assert result is not None + assert result.external_ids.anilist_id is None + + +def test_anilist_combines_with_a_western_id(monkeypatch: pytest.MonkeyPatch) -> None: + tmdb = FakeProvider("tmdb", MetadataResult(title="One Piece", year=1999, kind="tv")) + anilist = FakeProvider("anilist", MetadataResult(title="One Piece", external_ids=ExternalIds(anilist_id=21))) + patch_providers(monkeypatch, {"tmdb": tmdb, "anilist": anilist}) + + result = providers.resolve_by_ids(37854, anilist_id=21, kind="tv") + + assert result is not None + assert result.external_ids.tmdb_id == 37854 + assert result.external_ids.anilist_id == 21 + + +# ---------- the anime hint ---------- + + +def test_anime_puts_anilist_first_with_the_rest_behind_it() -> None: + ordered = [cls.NAME for cls in providers.provider_order("tv", anime=True)] + assert ordered[0] == "anilist" + assert ordered[1:] == [name for name in providers.DEFAULT_ORDER if name != "anilist"] + + +def test_without_the_hint_anilist_stays_last() -> None: + assert [cls.NAME for cls in providers.provider_order("tv")] == list(providers.DEFAULT_ORDER) + assert providers.DEFAULT_ORDER[-1] == "anilist" + + +def test_the_hint_cannot_add_a_provider_the_config_left_out(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(config, "metadata_providers", ["tmdb", "tvdb"]) + assert [cls.NAME for cls in providers.provider_order("tv", anime=True)] == ["tmdb", "tvdb"] + + +def test_an_anime_hint_falls_through_to_the_normal_order_on_a_miss(monkeypatch: pytest.MonkeyPatch) -> None: + """anilist is asked first but knows no imdb ids, so imdb still answers.""" + anilist = FakeProvider("anilist", None) + imdb = FakeProvider("imdb", MetadataResult(title="One Piece", year=1999, kind="tv")) + patch_providers(monkeypatch, {"anilist": anilist, "imdb": imdb}) + + result = providers.resolve_by_ids(imdb_id="tt0388629", kind="tv", anime=True) + + assert result is not None + assert result.title == "One Piece" + # the imdb id is in no namespace anilist consumes, so it is never asked + assert anilist.asked == [] + assert imdb.asked == ["tt0388629"] + + +def test_the_anime_hint_reaches_search(monkeypatch: pytest.MonkeyPatch) -> None: + calls: list[dict] = [] + monkeypatch.setattr(providers, "search_metadata", lambda *a, **kw: calls.append({"args": a}) or None) + patch_providers(monkeypatch, {}) + + providers.resolve_by_ids(title="One Piece", year=1999, kind="tv", anime=True) + + assert calls and calls[0]["args"][-1] is True + + +def test_all_supplied_ids_survive_together(monkeypatch: pytest.MonkeyPatch) -> None: + """anilist_id combined with a western trio member: every supplied ID reaches external_ids.""" + tvdb = FakeProvider( + "tvdb", + MetadataResult(title="One Piece", year=1999, kind="tv", external_ids=ExternalIds(tvdb_id=81797)), + ) + patch_providers(monkeypatch, {"tvdb": tvdb}) + + result = providers.resolve_by_ids(tvdb_id=81797, anilist_id="mal:21", imdb_id="tt0388629", kind="tv") + + assert result is not None + assert result.external_ids.tvdb_id == 81797 + assert result.external_ids.imdb_id == "tt0388629" + # a mal: ref the provider never resolved contributes no anilist_id; a bare id + # would have been forced back in + assert result.external_ids.anilist_id is None + + result = providers.resolve_by_ids(tvdb_id=81797, anilist_id=21, kind="tv") + assert result is not None + assert result.external_ids.tvdb_id == 81797 + assert result.external_ids.anilist_id == 21 diff --git a/tests/core/test_service_dl_overrides.py b/tests/core/test_service_dl_overrides.py index 25321aa5..1d5184ce 100644 --- a/tests/core/test_service_dl_overrides.py +++ b/tests/core/test_service_dl_overrides.py @@ -150,3 +150,24 @@ def test_language_range_yaml_shapes(value, expected): ctx = make_ctx() apply_service_dl_overrides(ctx, {"v_lang": value}, log) assert ctx.params["v_lang"] == expected + + +@pytest.mark.parametrize( + "value,expected", + [ + ("all,-es", ["all", "-es"]), + ("-es", ["-es"]), + (["all", "-es"], ["all", "-es"]), + ], +) +def test_language_exclusion_tokens_survive_the_override_convert(value, expected): + """A service dl override is re-converted, so its exclusion tokens must reach the filter stage intact.""" + ctx = make_ctx() + apply_service_dl_overrides(ctx, {"v_lang": value}, log) + assert ctx.params["v_lang"] == expected + + +def test_cli_exclusion_beats_service_config(): + ctx = make_ctx(["-vl", "-es"]) + apply_service_dl_overrides(ctx, {"v_lang": ["en"]}, log) + assert ctx.params["v_lang"] == ["-es"] diff --git a/tests/core/test_service_repo.py b/tests/core/test_service_repo.py index 05ee84f1..5d90d3b9 100644 --- a/tests/core/test_service_repo.py +++ b/tests/core/test_service_repo.py @@ -66,11 +66,11 @@ def test_repos_base_uses_configured_local_dir(monkeypatch, tmp_path): def test_is_stale(tmp_path): stamp = tmp_path / "stamp" - assert service_repo._is_stale(stamp, ttl=10) is True # missing → stale + assert service_repo.is_stale(stamp, ttl=10) is True # missing → stale stamp.write_text(str(time.time())) - assert service_repo._is_stale(stamp, ttl=10) is False # fresh + assert service_repo.is_stale(stamp, ttl=10) is False # fresh stamp.write_text(str(time.time() - 100)) - assert service_repo._is_stale(stamp, ttl=10) is True # old + assert service_repo.is_stale(stamp, ttl=10) is True # old def test_clone_invokes_git(monkeypatch, tmp_path): @@ -110,7 +110,7 @@ def test_fresh_clone_skips_pull(monkeypatch, tmp_path): assert service_repo.resolve_service_repo("a/b") == dest -def _git_stub(porcelain=b"", ahead=b"0"): +def git_stub(porcelain=b"", ahead=b"0"): """Fake subprocess.run for git: dirty checks return given output, pull succeeds.""" def run(args, **kw): @@ -126,20 +126,20 @@ def _git_stub(porcelain=b"", ahead=b"0"): def test_is_dirty_uncommitted(monkeypatch): monkeypatch.setattr(service_repo.binaries, "Git", "git") - monkeypatch.setattr(service_repo.subprocess, "run", _git_stub(porcelain=b" M EXAMPLE/__init__.py\n")) - assert service_repo._is_dirty(Path("/x")) is True + monkeypatch.setattr(service_repo.subprocess, "run", git_stub(porcelain=b" M EXAMPLE/__init__.py\n")) + assert service_repo.is_dirty(Path("/x")) is True def test_is_dirty_commits_ahead(monkeypatch): monkeypatch.setattr(service_repo.binaries, "Git", "git") - monkeypatch.setattr(service_repo.subprocess, "run", _git_stub(porcelain=b"", ahead=b"2")) - assert service_repo._is_dirty(Path("/x")) is True + monkeypatch.setattr(service_repo.subprocess, "run", git_stub(porcelain=b"", ahead=b"2")) + assert service_repo.is_dirty(Path("/x")) is True def test_is_dirty_clean(monkeypatch): monkeypatch.setattr(service_repo.binaries, "Git", "git") - monkeypatch.setattr(service_repo.subprocess, "run", _git_stub(porcelain=b"", ahead=b"0")) - assert service_repo._is_dirty(Path("/x")) is False + monkeypatch.setattr(service_repo.subprocess, "run", git_stub(porcelain=b"", ahead=b"0")) + assert service_repo.is_dirty(Path("/x")) is False def test_is_dirty_ignores_untracked_files(monkeypatch): @@ -153,7 +153,7 @@ def test_is_dirty_ignores_untracked_files(monkeypatch): return subprocess.CompletedProcess(args, 0, out, b"") monkeypatch.setattr(service_repo.subprocess, "run", run) - service_repo._is_dirty(Path("/x")) + service_repo.is_dirty(Path("/x")) status_args = next(a for a in seen if a[3] == "status") assert "--untracked-files=no" in status_args @@ -165,7 +165,7 @@ def test_auto_refresh_refuses_dirty_clone(monkeypatch, tmp_path): service_repo.stamp_for(dest).write_text("0") # stale → refresh due monkeypatch.setattr(service_repo, "repos_base", lambda: tmp_path) monkeypatch.setattr(service_repo.binaries, "Git", "git") - monkeypatch.setattr(service_repo.subprocess, "run", _git_stub(porcelain=b" M f\n")) + monkeypatch.setattr(service_repo.subprocess, "run", git_stub(porcelain=b" M f\n")) with pytest.raises(service_repo.DirtyServiceRepo) as ei: service_repo.resolve_service_repo("a/b") assert ei.value.path == dest @@ -201,7 +201,7 @@ def test_changed_services_groups_by_top_dir(monkeypatch): return subprocess.CompletedProcess(args, 0, out, b"") monkeypatch.setattr(service_repo.subprocess, "run", run) - lines = service_repo._changed_services(Path("/x"), "aaa", "bbb") + lines = service_repo.changed_services(Path("/x"), "aaa", "bbb") assert "+ NEW (added)" in lines assert "~ OLD (modified)" in lines assert "- GONE (removed)" in lines @@ -215,7 +215,7 @@ def test_local_dirty_services_lists_edited_dirs(monkeypatch): "run", lambda *a, **k: subprocess.CompletedProcess(a, 0, b"CR/__init__.py\nCR/config.yaml\nNF/__init__.py\n", b""), ) - assert service_repo._local_dirty_services(Path("/x")) == ["CR", "NF"] + assert service_repo.local_dirty_services(Path("/x")) == ["CR", "NF"] def test_refresh_reports_discarded_local_edits(monkeypatch, tmp_path): @@ -241,7 +241,7 @@ def test_changed_services_up_to_date(monkeypatch): monkeypatch.setattr(service_repo.binaries, "Git", "git") # same HEAD before/after → no diff computed at all monkeypatch.setattr(service_repo.subprocess, "run", lambda *a, **k: pytest.fail("no git diff when HEAD unchanged")) - assert service_repo._changed_services(Path("/x"), "same", "same") == [] + assert service_repo.changed_services(Path("/x"), "same", "same") == [] @pytest.mark.parametrize("order", [["a", "b"], ["b", "a"]]) diff --git a/tests/core/test_tag_rules_integration.py b/tests/core/test_tag_rules_integration.py index 32e3e5fd..fda75cc8 100644 --- a/tests/core/test_tag_rules_integration.py +++ b/tests/core/test_tag_rules_integration.py @@ -61,7 +61,7 @@ def clean_config(monkeypatch: pytest.MonkeyPatch) -> None: def context_for(rules: list[dict[str, Any]], monkeypatch: pytest.MonkeyPatch) -> dict: monkeypatch.setattr(config, "tag_rules", rules) - return make_movie()._build_base_template_context(make_media_info(["en"])) + return make_movie().build_base_template_context(make_media_info(["en"])) def test_matching_rule_replaces_the_tag(monkeypatch: pytest.MonkeyPatch) -> None: @@ -90,7 +90,7 @@ def test_rules_run_after_lang_tag(monkeypatch: pytest.MonkeyPatch) -> None: movie = make_movie() movie.tracks = TrackStub(audio=[types.SimpleNamespace(language=Language.get("ja"))], subtitles=[]) # type: ignore[assignment] - context = movie._build_base_template_context(make_media_info(["ja"])) + context = movie.build_base_template_context(make_media_info(["ja"])) assert context["lang_tag"] == "SUBBED" assert context["tag"] == "SUBGRP" @@ -113,7 +113,7 @@ def test_a_quality_rule_matches_the_built_context(monkeypatch: pytest.MonkeyPatc ) media_info = types.SimpleNamespace(video_tracks=[video], audio_tracks=[make_audio("en")]) - context = make_movie()._build_base_template_context(media_info) + context = make_movie().build_base_template_context(media_info) assert context["quality"] == "2160p" assert context["video"] == "H.265" diff --git a/tests/core/test_tvdb.py b/tests/core/test_tvdb.py index c50711a8..571ffaa3 100644 --- a/tests/core/test_tvdb.py +++ b/tests/core/test_tvdb.py @@ -6,8 +6,8 @@ import pytest from unshackle.core import providers from unshackle.core.config import config -from unshackle.core.providers.tvdb import TVDBProvider, _ids_from_remote, _pick_match -from unshackle.core.utils.tags import _build_tags_from_ids +from unshackle.core.providers.tvdb import TVDBProvider, ids_from_remote, pick_match +from unshackle.core.utils.tags import build_tags_from_ids REMOTE_IDS = [ {"id": "tt0149460", "type": 2, "sourceName": "IMDB"}, @@ -31,19 +31,19 @@ DVD = [ def test_ids_from_remote() -> None: - ext = _ids_from_remote(REMOTE_IDS, 73871) + ext = ids_from_remote(REMOTE_IDS, 73871) assert ext.imdb_id == "tt0149460" assert ext.tmdb_id == 615 assert ext.tvdb_id == 73871 -def _stub(p: TVDBProvider, monkeypatch: pytest.MonkeyPatch) -> None: +def stub(p: TVDBProvider, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(p, "get_episodes", lambda _id, order: {"official": AIRED, "dvd": DVD}.get(order, [])) def test_order_map_joins_on_episode_id(monkeypatch: pytest.MonkeyPatch) -> None: p = TVDBProvider() - _stub(p, monkeypatch) + stub(p, monkeypatch) mapping = p.get_order_map(73871, "dvd") assert mapping[(1, 9)] == (1, 9, "Hell Is Other Robots") @@ -55,7 +55,7 @@ def test_order_map_joins_on_episode_id(monkeypatch: pytest.MonkeyPatch) -> None: def test_order_map_reverses_with_source_order(monkeypatch: pytest.MonkeyPatch) -> None: p = TVDBProvider() - _stub(p, monkeypatch) + stub(p, monkeypatch) mapping = p.get_order_map(73871, "official", source_order="dvd") assert mapping[(1, 10)] == (2, 1, "A Flight to Remember") @@ -67,13 +67,13 @@ def test_order_map_is_noop_when_orders_match() -> None: def test_order_map_empty_when_order_missing(monkeypatch: pytest.MonkeyPatch) -> None: p = TVDBProvider() - _stub(p, monkeypatch) + stub(p, monkeypatch) assert p.get_order_map(73871, "alternate") == {} def test_detect_order_picks_the_order_the_numbering_matches(monkeypatch: pytest.MonkeyPatch) -> None: p = TVDBProvider() - _stub(p, monkeypatch) + stub(p, monkeypatch) # a service may number a series in a non-aired order, so detection must not assume aired assert p.detect_order(73871, [(1, 10), (1, 11)]) == "dvd" assert p.detect_order(73871, [(2, 1), (2, 2), (2, 3)]) == "official" @@ -94,27 +94,27 @@ BSG = [ def test_pick_match_prefers_the_right_era() -> None: # exact title match on the 1978 series must not beat the era the caller asked for - best, _, _ = _pick_match(BSG, "Battlestar Galactica", 2004) + best, _, _ = pick_match(BSG, "Battlestar Galactica", 2004) assert best["tvdb_id"] == "73545" - best, _, _ = _pick_match(BSG, "Battlestar Galactica", 1978) + best, _, _ = pick_match(BSG, "Battlestar Galactica", 1978) assert best["tvdb_id"] == "71173" def test_pick_match_without_a_year_scores_on_title_alone() -> None: - best, _, _ = _pick_match(BSG, "Battlestar Galactica", None) + best, _, _ = pick_match(BSG, "Battlestar Galactica", None) assert best["tvdb_id"] == "71173" def test_pick_match_returns_nothing_when_no_year_fits() -> None: - assert _pick_match(BSG, "Battlestar Galactica", 1995)[0] is None + assert pick_match(BSG, "Battlestar Galactica", 1995)[0] is None def test_tvdb2_tag_follows_the_matroska_spec() -> None: # TVDB2 IDs are prefixed by entity type: matroska.org/technical/tagging.html from unshackle.core.providers import ExternalIds - assert _build_tags_from_ids(ExternalIds(tvdb_id=73871), "tv")["TVDB2"] == "series/73871" - assert _build_tags_from_ids(ExternalIds(tvdb_id=113), "movie")["TVDB2"] == "movies/113" + assert build_tags_from_ids(ExternalIds(tvdb_id=73871), "tv")["TVDB2"] == "series/73871" + assert build_tags_from_ids(ExternalIds(tvdb_id=113), "movie")["TVDB2"] == "movies/113" def test_provider_order_defaults(monkeypatch: pytest.MonkeyPatch) -> None: @@ -131,10 +131,10 @@ def test_provider_order_is_configurable(monkeypatch: pytest.MonkeyPatch) -> None def test_enrichment_authority_is_fixed(monkeypatch: pytest.MonkeyPatch) -> None: # search order must not demote TMDB below SIMKL for cross-validation tie-breaks monkeypatch.setattr(config, "metadata_providers", []) - assert providers._enrichment_providers() == ["tmdb", "simkl", "tvdb"] + assert providers.enrichment_providers() == ["tmdb", "simkl", "tvdb"] # `metadata_providers` filters the set but does not re-rank the trust order monkeypatch.setattr(config, "metadata_providers", ["tvdb", "simkl"]) - assert providers._enrichment_providers() == ["simkl", "tvdb"] + assert providers.enrichment_providers() == ["simkl", "tvdb"] def test_cached_tvdb_result_keeps_enriched_ids() -> None: @@ -144,7 +144,7 @@ def test_cached_tvdb_result_keeps_enriched_ids() -> None: "year": "1999", "_enriched_ids": {"imdb_id": "tt0149460", "tmdb_id": 615}, } - result = providers._cached_to_result(cached, "tvdb", "tv") + result = providers.cached_to_result(cached, "tvdb", "tv") assert result is not None assert result.external_ids.tvdb_id == 73871 assert result.external_ids.imdb_id == "tt0149460" @@ -152,15 +152,36 @@ def test_cached_tvdb_result_keeps_enriched_ids() -> None: # remote_ids stay authoritative over the enriched copies cached["remote_ids"] = REMOTE_IDS cached["_enriched_ids"] = {"imdb_id": "tt9999999", "tmdb_id": 1} - result = providers._cached_to_result(cached, "tvdb", "tv") + result = providers.cached_to_result(cached, "tvdb", "tv") assert result.external_ids.imdb_id == "tt0149460" assert result.external_ids.tmdb_id == 615 +@pytest.mark.parametrize(("kind", "path"), [("tv", "/series/73871/extended"), ("movie", "/movies/73871/extended")]) +def test_get_by_id_supplies_what_enrich_needs(monkeypatch: pytest.MonkeyPatch, kind: str, path: str) -> None: + p = TVDBProvider() + seen: list[str] = [] + + def fake_get(request_path: str, params: dict) -> dict: + seen.append(request_path) + return { + "name": "Cowboy Bebop", + "year": "1998", + "originalLanguage": "jpn", + "remoteIds": [{"sourceName": "IMDB", "id": "tt0213338"}], + } + + monkeypatch.setattr(p, "api_get", fake_get) + result = p.get_by_id(73871, kind) + assert seen == [path] + assert (result.title, result.year, result.original_language) == ("Cowboy Bebop", 1998, "jpn") + assert result.external_ids.imdb_id == "tt0213338" + + def test_get_episodes_fails_closed_on_a_truncated_listing(monkeypatch: pytest.MonkeyPatch) -> None: p = TVDBProvider() pages: dict[int, object] = {0: {"episodes": [{"id": n} for n in range(500)]}, 1: None} - monkeypatch.setattr(p, "_get", lambda _path, params: pages[params["page"]]) + monkeypatch.setattr(p, "api_get", lambda _path, params: pages[params["page"]]) # page 1 fails mid-listing: a partial list would renumber wrongly assert p.get_episodes(73871, "official") == [] assert p._episodes == {} @@ -168,7 +189,7 @@ def test_get_episodes_fails_closed_on_a_truncated_listing(monkeypatch: pytest.Mo def test_get_episodes_does_not_cache_an_empty_listing(monkeypatch: pytest.MonkeyPatch) -> None: p = TVDBProvider() - monkeypatch.setattr(p, "_get", lambda _path, params: None) + monkeypatch.setattr(p, "api_get", lambda _path, params: None) assert p.get_episodes(73871, "official") == [] assert p._episodes == {} @@ -191,7 +212,7 @@ class _FakeProvider: return self.mapping -def _renumber(monkeypatch: pytest.MonkeyPatch, episodes: list, mapping: dict): +def renumber(monkeypatch: pytest.MonkeyPatch, episodes: list, mapping: dict): from types import SimpleNamespace from unshackle.commands.dl import dl @@ -208,7 +229,7 @@ def _renumber(monkeypatch: pytest.MonkeyPatch, episodes: list, mapping: dict): return dl.apply_tvdb_order(stub, Series(episodes)), errors -def _ep(season: int, number: int, part=None): +def ep(season: int, number: int, part=None): from unshackle.core.titles.episode import Episode return Episode(id_=f"{season}x{number}.{part}", service=_Svc, title="Show", season=season, number=number, part=part) @@ -216,17 +237,17 @@ def _ep(season: int, number: int, part=None): def test_parts_of_one_episode_renumber_together(monkeypatch: pytest.MonkeyPatch) -> None: """Three parts share one (season, number) slot, so they must not read as a collision.""" - episodes = [_ep(1, 1, 1), _ep(1, 1, 2), _ep(1, 1, 3), _ep(1, 2)] + episodes = [ep(1, 1, 1), ep(1, 1, 2), ep(1, 1, 3), ep(1, 2)] mapping = {(1, 1): (2, 1, "Part-ful"), (1, 2): (2, 2, "Whole")} - titles, errors = _renumber(monkeypatch, episodes, mapping) + titles, errors = renumber(monkeypatch, episodes, mapping) assert errors == [] assert [(t.season, t.number, t.part) for t in titles] == [(2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 2, None)] def test_a_real_collision_still_bails(monkeypatch: pytest.MonkeyPatch) -> None: """Dedupe must not disable the guard: two distinct episodes on one slot still refuses.""" - episodes = [_ep(1, 1), _ep(1, 2)] + episodes = [ep(1, 1), ep(1, 2)] mapping = {(1, 2): (1, 1, "Clash")} # (1, 1) is unmapped and keeps the slot (1, 2) moves onto - titles, errors = _renumber(monkeypatch, episodes, mapping) + titles, errors = renumber(monkeypatch, episodes, mapping) assert errors # refused, with the "two episodes each" error assert [(t.season, t.number) for t in titles] == [(1, 1), (1, 2)] # numbering untouched diff --git a/tests/downloaders/test_cancel_hardening.py b/tests/downloaders/test_cancel_hardening.py index 811531d5..17922fb8 100644 --- a/tests/downloaders/test_cancel_hardening.py +++ b/tests/downloaders/test_cancel_hardening.py @@ -2,7 +2,7 @@ Pins two teardown-safety invariants: -- ``_dispatch_parts`` must NOT finalize when the process-global ``DOWNLOAD_CANCELLED`` +- ``dispatch_parts`` must NOT finalize when the process-global ``DOWNLOAD_CANCELLED`` fires mid-download: part workers return silently keeping partials, so every future completes without error, and finalizing would strip the ``.!dev`` control file and pass off a hole-filled pre-truncated file as complete on the next run. @@ -104,7 +104,7 @@ def test_dispatch_parts_does_not_finalize_on_global_cancel(server, tmp_path): canceller = threading.Thread(target=_cancel_when_started) canceller.start() events = list( - dl._dispatch_parts( + dl.dispatch_parts( url=_url(server), save_path=save_path, session=Session(), @@ -121,7 +121,7 @@ def test_dispatch_parts_does_not_finalize_on_global_cancel(server, tmp_path): def test_backoff_wait_exits_promptly_on_batch_abort(server, tmp_path, monkeypatch): server.always_status = 503 - monkeypatch.setattr(dl, "_retry_sleep", lambda exc, attempts: 30.0) + monkeypatch.setattr(dl, "retry_sleep", lambda exc, attempts: 30.0) abort = threading.Event() def _consume() -> None: diff --git a/tests/downloaders/test_content_encoding.py b/tests/downloaders/test_content_encoding.py index 6e3f7b49..816a593a 100644 --- a/tests/downloaders/test_content_encoding.py +++ b/tests/downloaders/test_content_encoding.py @@ -6,7 +6,7 @@ import threading import pytest import requests as rq -from unshackle.core.downloaders.requests import _is_content_encoded, download +from unshackle.core.downloaders.requests import download, is_content_encoded @pytest.mark.parametrize( @@ -35,7 +35,7 @@ from unshackle.core.downloaders.requests import _is_content_encoded, download ], ) def test_is_content_encoded(value, expected): - assert _is_content_encoded(value) is expected + assert is_content_encoded(value) is expected PLAIN = b"WEBVTT\n\n" + b"".join(f"{i:05d} cue text\n".encode() for i in range(2000)) @@ -51,9 +51,9 @@ class _EncodingServer: self.sock.bind(("127.0.0.1", 0)) self.sock.listen(8) self.url = f"http://127.0.0.1:{self.sock.getsockname()[1]}/sub.vtt" - threading.Thread(target=self._serve, daemon=True).start() + threading.Thread(target=self.serve, daemon=True).start() - def _serve(self): + def serve(self): while True: try: conn, _ = self.sock.accept() @@ -146,7 +146,7 @@ class _FakeRnetSession: def test_rnet_encoded_body_discards_a_partial_instead_of_range_resuming(tmp_path, monkeypatch): dl = importlib.import_module("unshackle.core.downloaders.requests") - monkeypatch.setattr(dl, "_is_rnet_session", lambda s: isinstance(s, _FakeRnetSession)) + monkeypatch.setattr(dl, "is_rnet_session", lambda s: isinstance(s, _FakeRnetSession)) session = _FakeRnetSession(PLAIN) save_path = tmp_path / "sub.vtt" save_path.with_name("sub.vtt.!dev").write_bytes(PLAIN[:5000]) diff --git a/tests/downloaders/test_download_hardening.py b/tests/downloaders/test_download_hardening.py index 080fd273..a15316f3 100644 --- a/tests/downloaders/test_download_hardening.py +++ b/tests/downloaders/test_download_hardening.py @@ -99,7 +99,7 @@ def test_small_segments_never_probed(server, tmp_path, monkeypatch): def no_probe(*args, **kw): raise AssertionError("tail boost probed a small-segment batch") - monkeypatch.setattr(dl, "_probe_ranged", no_probe) + monkeypatch.setattr(dl, "probe_ranged", no_probe) files = _run(server, tmp_path, [{"url": _url(server)} for _ in range(30)], max_workers=4, adaptive=True) assert len(files) == 30 assert all(f.read_bytes() == PARENT for f in files) @@ -149,24 +149,24 @@ def _exc_with_headers(headers): def test_retry_sleep_honors_numeric_retry_after(): - assert dl._retry_sleep(_exc_with_headers({"Retry-After": "3"}), 1) == 3.0 - assert dl._retry_sleep(_exc_with_headers({"Retry-After": "9999"}), 1) == 60.0 # session.MAX_BACKOFF cap + assert dl.retry_sleep(_exc_with_headers({"Retry-After": "3"}), 1) == 3.0 + assert dl.retry_sleep(_exc_with_headers({"Retry-After": "9999"}), 1) == 60.0 # session.MAX_BACKOFF cap def test_retry_sleep_honors_http_date_retry_after(): when = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=10), usegmt=True) - assert 5 <= dl._retry_sleep(_exc_with_headers({"Retry-After": when}), 1) <= 10 + assert 5 <= dl.retry_sleep(_exc_with_headers({"Retry-After": when}), 1) <= 10 def test_retry_sleep_exponential_backoff_with_jitter(): # no response on the failure: exponential from RETRY_WAIT with up to 10% jitter for attempts, base in ((1, dl.RETRY_WAIT), (2, dl.RETRY_WAIT * 2), (3, dl.RETRY_WAIT * 4)): - assert base <= dl._retry_sleep(Exception("reset"), attempts) <= base * 1.1 - assert dl._retry_sleep(Exception("reset"), 30) == 60.0 # capped + assert base <= dl.retry_sleep(Exception("reset"), attempts) <= base * 1.1 + assert dl.retry_sleep(Exception("reset"), 30) == 60.0 # capped def test_retry_sleep_invalid_retry_after_falls_back(): - v = dl._retry_sleep(_exc_with_headers({"Retry-After": "soon"}), 1) + v = dl.retry_sleep(_exc_with_headers({"Retry-After": "soon"}), 1) assert dl.RETRY_WAIT <= v <= dl.RETRY_WAIT * 1.1 @@ -174,13 +174,13 @@ def test_retry_sleep_naive_http_date_treated_as_utc(): # "-0000" makes parsedate_to_datetime return a naive datetime; must clamp, not raise when = format_datetime(datetime.now(timezone.utc).replace(tzinfo=None) + timedelta(seconds=10)) assert when.endswith("-0000") - v = dl._retry_sleep(_exc_with_headers({"Retry-After": when}), 1) + v = dl.retry_sleep(_exc_with_headers({"Retry-After": when}), 1) assert 5 <= v <= 10 def test_retry_sleep_nan_retry_after_falls_back(): # float("nan") parses; a non-finite wait must fall back to the exponential path - v = dl._retry_sleep(_exc_with_headers({"Retry-After": "nan"}), 1) + v = dl.retry_sleep(_exc_with_headers({"Retry-After": "nan"}), 1) assert dl.RETRY_WAIT <= v <= dl.RETRY_WAIT * 1.1 @@ -188,7 +188,7 @@ def test_retry_sleep_honors_retry_after_on_wrapped_cause(): # RnetSession raises MaxRetriesError whose __cause__ HTTPError carries the response exc = Exception("max retries exceeded") exc.__cause__ = _exc_with_headers({"Retry-After": "3"}) - assert dl._retry_sleep(exc, 1) == 3.0 + assert dl.retry_sleep(exc, 1) == 3.0 def test_transient_429_with_retry_after_does_not_abort_track(server, tmp_path, monkeypatch): diff --git a/tests/downloaders/test_failure_surfacing.py b/tests/downloaders/test_failure_surfacing.py new file mode 100644 index 00000000..c43cfc54 --- /dev/null +++ b/tests/downloaders/test_failure_surfacing.py @@ -0,0 +1,187 @@ +import socket +import sys +import threading + +import pytest +import requests as rq + +from unshackle.core.constants import DOWNLOAD_CANCELLED +from unshackle.core.downloaders.requests import download, has_range_header +from unshackle.core.downloaders.requests import requests as requests_downloader + +# the package re-exports the `requests` function, shadowing the module of the same name +downloader = sys.modules[download.__module__] + +BODY = b"".join(f"{i:05d} payload\n".encode() for i in range(500)) + + +class _Server: + """Serves BODY, with a switch for the two failure modes retries have to handle.""" + + def __init__(self, mode): + self.mode = mode + self.sock = socket.socket() + self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.sock.bind(("127.0.0.1", 0)) + self.sock.listen(8) + self.url = f"http://127.0.0.1:{self.sock.getsockname()[1]}/seg.mp4" + self.requests = [] + threading.Thread(target=self.serve, daemon=True).start() + + def serve(self): + while True: + try: + conn, _ = self.sock.accept() + except OSError: + return + try: + request = conn.recv(65536).decode("latin1") + headers = dict( + (name.strip().lower(), value.strip()) + for name, _, value in (line.partition(":") for line in request.split("\r\n")) + if value + ) + self.requests.append(headers) + ranged = "range" in headers + if self.mode == "unavailable": + conn.sendall(b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n") + elif ranged: + conn.sendall( + b"HTTP/1.1 416 Range Not Satisfiable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" + ) + else: + conn.sendall( + f"HTTP/1.1 200 OK\r\nContent-Length: {len(BODY)}\r\nConnection: close\r\n\r\n".encode() + BODY + ) + except Exception: + pass + finally: + conn.close() + + def close(self): + self.sock.close() + + +@pytest.fixture(autouse=True) +def clear_cancel(): + # the batch failure path sets the process-global cancel; leaking it makes every later + # test that waits on the event (the speed limiter) return instantly + DOWNLOAD_CANCELLED.clear() + yield + DOWNLOAD_CANCELLED.clear() + + +@pytest.fixture +def no_retry_wait(monkeypatch): + monkeypatch.setattr(downloader, "RETRY_WAIT", 0) + + +@pytest.fixture +def server(request, no_retry_wait): + server = _Server(request.param) + yield server + server.close() + + +@pytest.mark.parametrize("server", ["unavailable"], indirect=True) +def test_exhausted_retries_raise_instead_of_reporting_success(tmp_path, server): + save_path = tmp_path / "0.mp4" + with pytest.raises(rq.HTTPError): + list(download(url=server.url, save_path=save_path, session=rq.Session())) + assert len(server.requests) == downloader.MAX_ATTEMPTS + assert not save_path.exists() + + +@pytest.mark.parametrize("server", ["unavailable"], indirect=True) +def test_cancelled_download_stays_silent(tmp_path, server, monkeypatch): + # one attempt only, so a cancel check ordered after the exhaustion raise would fail here + monkeypatch.setattr(downloader, "MAX_ATTEMPTS", 1) + save_path = tmp_path / "0.mp4" + DOWNLOAD_CANCELLED.set() + list(download(url=server.url, save_path=save_path, session=rq.Session())) + assert not save_path.exists() + + +@pytest.mark.parametrize("server", ["range_unsatisfiable"], indirect=True) +def test_stale_partial_restarts_clean_on_416(tmp_path, server): + save_path = tmp_path / "0.mp4" + save_path.with_name("0.mp4.!dev").write_bytes(b"\x00" * len(BODY)) + list(download(url=server.url, save_path=save_path, session=rq.Session())) + assert save_path.read_bytes() == BODY + assert [r.get("range") for r in server.requests] == [f"bytes={len(BODY)}-", None] + + +@pytest.mark.parametrize("server", ["range_unsatisfiable"], indirect=True) +def test_slice_segment_never_gets_a_resume_range(tmp_path, server): + save_path = tmp_path / "0.mp4" + save_path.with_name("0.mp4.!dev").write_bytes(b"\x00" * 128) + with pytest.raises(OSError): + list(download(url=server.url, save_path=save_path, session=rq.Session(), headers={"Range": "bytes=0-127"})) + assert [r["range"] for r in server.requests] == ["bytes=0-127"] * downloader.MAX_ATTEMPTS + + +@pytest.mark.parametrize("server", ["range_unsatisfiable"], indirect=True) +def test_identity_encoding_requested_unless_the_caller_set_one(tmp_path, server): + save_path = tmp_path / "0.mp4" + list(download(url=server.url, save_path=save_path, session=rq.Session())) + assert server.requests[-1]["accept-encoding"] == "identity" + + list( + download( + url=server.url, save_path=tmp_path / "1.mp4", session=rq.Session(), headers={"accept-encoding": "gzip"} + ) + ) + assert server.requests[-1]["accept-encoding"] == "gzip" + + +@pytest.mark.parametrize("server", ["unavailable"], indirect=True) +def test_failed_segment_fails_the_batch(tmp_path, server): + # a segment the batch never delivers must raise, not merge short of the segment count + with pytest.raises(rq.HTTPError): + list( + requests_downloader( + urls=[server.url, server.url], + output_dir=tmp_path, + filename="{i:01}.mp4", + max_workers=2, + ) + ) + assert sorted(p.name for p in tmp_path.iterdir() if p.suffix == ".mp4") == [] + + +@pytest.mark.parametrize("server", ["ok"], indirect=True) +def test_batch_larger_than_the_queue_window_downloads_every_segment(tmp_path, server): + # a broken top-up livelocks the drain loop instead of erroring, so consume on a thread and + # let the timeout fail the test + urls = [server.url] * 6 + advanced = 0 + + def consume(): + nonlocal advanced + advanced = sum( + event["advance"] + for event in requests_downloader(urls=urls, output_dir=tmp_path, filename="{i:02}.mp4", max_workers=1) + if "advance" in event + ) + + consumer = threading.Thread(target=consume, daemon=True) + consumer.start() + consumer.join(timeout=30) + assert not consumer.is_alive(), "downloader never finished the batch" + assert advanced == len(urls) + assert sorted(p.name for p in tmp_path.iterdir() if p.suffix == ".mp4") == [f"{i:02}.mp4" for i in range(6)] + + +@pytest.mark.parametrize( + ("item", "expected"), + [ + ({}, False), + ({"headers": None}, False), + ({"headers": {}}, False), + ({"headers": {"Accept": "*/*"}}, False), + ({"headers": {"Range": "bytes=0-127"}}, True), + ({"headers": {"range": "bytes=0-127"}}, True), + ], +) +def test_has_range_header(item, expected): + assert has_range_header(item) is expected diff --git a/tests/downloaders/test_multiprocess.py b/tests/downloaders/test_multiprocess.py index e9ad3bdf..54d67a4e 100644 --- a/tests/downloaders/test_multiprocess.py +++ b/tests/downloaders/test_multiprocess.py @@ -1,6 +1,6 @@ """Multiprocess segment-download invariants. -Pins the spawned-child fan-out path in ``requests()``/``_download_multiprocess``: +Pins the spawned-child fan-out path in ``requests()``/``download_multiprocess``: - Strided assignment plus the ``index_offset``/``index_stride`` filename mapping puts every segment's bytes in the correctly named file (a broken stride mapping misfiles content). @@ -109,13 +109,13 @@ def test_multiprocess_stride_downloads_all_segments_correctly(server, tmp_path): def test_multiprocess_child_crash_raises(server, tmp_path): - # _download_multiprocess has no MP_MIN_SEGMENTS gate, so drive it directly with 2 urls. + # download_multiprocess has no MP_MIN_SEGMENTS gate, so drive it directly with 2 urls. # a dict url without "url" raises KeyError while requests() builds the save path (before # any retry loop), so the child sends __mp_error__ fast; the sibling downloads normally. good = {"url": _url(server, "/seg/0.bin")} bad = {"nourl": True} with pytest.raises(RuntimeError): - for _ in dl._download_multiprocess( + for _ in dl.download_multiprocess( urls=[good, bad], output_dir=tmp_path, filename="seg_{i:04}.bin", @@ -163,7 +163,7 @@ def test_speed_limit_forces_single_process(server, tmp_path, monkeypatch): def _no_mp(**kwargs): raise AssertionError("multiprocess fan-out engaged despite a speed limit") - monkeypatch.setattr(dl, "_download_multiprocess", _no_mp) + monkeypatch.setattr(dl, "download_multiprocess", _no_mp) n = dl.MP_MIN_SEGMENTS urls = [{"url": _url(server, f"/seg/{i}.bin")} for i in range(n)] dl.set_speed_limit(1_000_000_000) # far above the tiny batch, so the test is not slowed diff --git a/tests/downloaders/test_ranged_parallel.py b/tests/downloaders/test_ranged_parallel.py index 9cd7d680..4bfeb65b 100644 --- a/tests/downloaders/test_ranged_parallel.py +++ b/tests/downloaders/test_ranged_parallel.py @@ -9,10 +9,10 @@ Pins the byte-range fan-out that splits one large file across parts: retries rather than writing a wrong-length body into the pre-truncated file; - that failure sets only the ranged download's LOCAL abort event, never the process-global DOWNLOAD_CANCELLED (which would poison sibling tracks); -- ``_probe_ranged`` parses the Content-Range total on a 206 and declines a 200 +- ``probe_ranged`` parses the Content-Range total on a 206 and declines a 200 or a content-encoded 206. -Tests 2/3 exercise ``_dispatch_parts`` directly: the public single-URL path in +Tests 2/3 exercise ``dispatch_parts`` directly: the public single-URL path in ``requests()`` deliberately swallows a ranged failure and falls back to a plain sequential download, so the raise the invariant guarantees is only observable at the dispatch layer. @@ -119,7 +119,7 @@ def test_ranged_part_requires_206(server, tmp_path, monkeypatch): server.mode = "probe_only_206" monkeypatch.setattr(dl, "RETRY_WAIT", 0.01) save_path = tmp_path / "seg_0000.bin" - gen = dl._dispatch_parts( + gen = dl.dispatch_parts( url=_url(server), save_path=save_path, session=Session(), @@ -140,7 +140,7 @@ def test_ranged_part_failure_stays_local(server, tmp_path, monkeypatch): monkeypatch.setattr(dl, "RETRY_WAIT", 0.01) dl.DOWNLOAD_CANCELLED.clear() save_path = tmp_path / "seg_0000.bin" - gen = dl._dispatch_parts( + gen = dl.dispatch_parts( url=_url(server), save_path=save_path, session=Session(), @@ -157,8 +157,8 @@ def test_ranged_part_failure_stays_local(server, tmp_path, monkeypatch): def test_probe_ranged_parses_content_range(server): session = Session() server.mode = "range" - assert dl._probe_ranged(_url(server), session) == (len(PAYLOAD), True) + assert dl.probe_ranged(_url(server), session) == (len(PAYLOAD), True) server.mode = "always_200" - assert dl._probe_ranged(_url(server), session) == (0, False) + assert dl.probe_ranged(_url(server), session) == (0, False) server.mode = "gzip_probe" - assert dl._probe_ranged(_url(server), session) == (0, False) + assert dl.probe_ranged(_url(server), session) == (0, False) diff --git a/tests/downloaders/test_speed_limit.py b/tests/downloaders/test_speed_limit.py index 3d32bb5d..18ce8b25 100644 --- a/tests/downloaders/test_speed_limit.py +++ b/tests/downloaders/test_speed_limit.py @@ -113,11 +113,11 @@ def test_locked_limit_ignores_unlocked_calls(): set_speed_limit(5_000_000, lock=True) set_speed_limit(None) set_speed_limit(1_000_000) - limiter = requests_downloader._speed_limiter + limiter = requests_downloader.speed_limiter assert limiter is not None and limiter.rate == 5_000_000 finally: - requests_downloader._speed_limit_locked = False - requests_downloader._speed_limiter = None + requests_downloader.speed_limit_locked = False + requests_downloader.speed_limiter = None def test_bucket_wait_aborts_on_cancellation(): diff --git a/tests/downloaders/test_tail_boost.py b/tests/downloaders/test_tail_boost.py index 7aa5dd38..c5c1f6c2 100644 --- a/tests/downloaders/test_tail_boost.py +++ b/tests/downloaders/test_tail_boost.py @@ -17,7 +17,7 @@ Determinism: with ``max_workers`` < segment count <= 2*max_workers and adaptive= the leading ``max_workers`` segments submit upfront and the trailing few stay in ``remaining`` (the tail top-up only releases boost-declined indices). The controller starts at the cap and never grows, so as soon as two leaders finish the spare workers -outnumber the tail and ``_tail_boost_engages`` fires. Module timing/size constants are +outnumber the tail and ``tail_boost_engages`` fires. Module timing/size constants are lowered via monkeypatch; the whole batch runs in-process (threads), so the patched values are seen by the workers. """ diff --git a/tests/lang_selection/test_embedded_audio_langs.py b/tests/lang_selection/test_embedded_audio_langs.py new file mode 100644 index 00000000..5148f70f --- /dev/null +++ b/tests/lang_selection/test_embedded_audio_langs.py @@ -0,0 +1,32 @@ +from types import SimpleNamespace + +import pytest + +from unshackle.core.utilities import embedded_audio_langs + + +def video(**data): + return SimpleNamespace(data=data) + + +def test_no_videos(): + assert embedded_audio_langs([], True) == [] + + +def test_video_without_a_declaration(): + assert embedded_audio_langs([video(), video(audio_language="")], True) == [] + + +def test_declared_language_is_returned(): + assert embedded_audio_langs([video(audio_language="zh-CN")], True) == ["zh-CN"] + + +def test_every_declaring_video_counts(): + videos = [video(audio_language="zh-CN"), video(), video(audio_language="th")] + assert embedded_audio_langs(videos, True) == ["zh-CN", "th"] + + +@pytest.mark.parametrize("videos", [[video(audio_language="zh-CN")], [video(audio_language="zh-CN"), video()]]) +def test_dropping_the_video_drops_its_audio(videos): + # --audio-only / --no-video discard the track the audio lives in, so nothing is embedded + assert embedded_audio_langs(videos, False) == [] diff --git a/tests/lang_selection/test_overlapping_audio_dedup.py b/tests/lang_selection/test_overlapping_audio_dedup.py index 7de66f77..c962910d 100644 --- a/tests/lang_selection/test_overlapping_audio_dedup.py +++ b/tests/lang_selection/test_overlapping_audio_dedup.py @@ -101,11 +101,11 @@ REAL_WORLD_AUDIO_LANGS = [ ] -def _mk(*tags): +def mk(*tags): return [Audio("https://x", language=t, id_=t) for t in tags] -def _select(tags, pool, exact=True): +def select(tags, pool, exact=True): return [(t.id, str(t.language)) for t in Tracks.by_language(pool, tags, per_language=1, exact_match=exact)] @@ -124,56 +124,56 @@ def test_distinct_pairs_are_not_distance_zero(base, reg): @pytest.mark.parametrize("base,reg", COLLAPSING_PAIRS) def test_exact_lang_selects_distinct_collapsing_variants(base, reg): # Both tracks present + both tags requested: each tag must route to its own track. - assert _select([base, reg], _mk(base, reg)) == [(base, base), (reg, reg)] + assert select([base, reg], mk(base, reg)) == [(base, base), (reg, reg)] @pytest.mark.parametrize("base,reg", DISTINCT_PAIRS) def test_exact_lang_distinct_regionals_still_work(base, reg): - assert _select([base, reg], _mk(base, reg)) == [(base, base), (reg, reg)] + assert select([base, reg], mk(base, reg)) == [(base, base), (reg, reg)] def test_exact_lang_lenient_base_to_regional_fallback(): # Only the regional present; a base request must still find it when CLDR rates them equal. - assert _select(["en"], _mk("en-US")) == [("en-US", "en-US")] + assert select(["en"], mk("en-US")) == [("en-US", "en-US")] # Alias with no string-equal track must still resolve (zh -> cmn). - assert _select(["zh"], _mk("cmn")) == [("cmn", "cmn")] + assert select(["zh"], mk("cmn")) == [("cmn", "cmn")] def test_exact_lang_stays_strict_for_nonzero_distance(): # `es` does NOT accept `es-419` in exact mode (distance 5), but fuzzy mode does. - assert _select(["es"], _mk("es-419")) == [] - assert _select(["es"], _mk("es-419"), exact=False) == [("es-419", "es-419")] + assert select(["es"], mk("es-419")) == [] + assert select(["es"], mk("es-419"), exact=False) == [("es-419", "es-419")] @pytest.mark.parametrize("base,reg", COLLAPSING_PAIRS) def test_overlapping_tags_never_duplicate_in_fuzzy_mode(base, reg): # Non-exact still collapses base/regional, but must dedupe rather than return a track twice. - selected = Tracks.by_language(_mk(base, reg), [base, reg], per_language=1, exact_match=False) + selected = Tracks.by_language(mk(base, reg), [base, reg], per_language=1, exact_match=False) assert len({t.id for t in selected}) == len(selected) def test_exact_lang_three_way_base_and_two_regionals(): # fr (base, dist 0 to fr-FR) + fr-FR + fr-CA (dist 4): all three must resolve distinctly. - assert _select(["fr", "fr-FR", "fr-CA"], _mk("fr", "fr-FR", "fr-CA")) == [ + assert select(["fr", "fr-FR", "fr-CA"], mk("fr", "fr-FR", "fr-CA")) == [ ("fr", "fr"), ("fr-FR", "fr-FR"), ("fr-CA", "fr-CA"), ] # Base request with no bare-tag track falls back to its paradigm regional, without stealing # the sibling regional (fr -> fr-FR, not fr-CA). - assert _select(["fr", "fr-CA"], _mk("fr-FR", "fr-CA")) == [("fr-FR", "fr-FR"), ("fr-CA", "fr-CA")] + assert select(["fr", "fr-CA"], mk("fr-FR", "fr-CA")) == [("fr-FR", "fr-FR"), ("fr-CA", "fr-CA")] @pytest.mark.parametrize("tag", REAL_WORLD_AUDIO_LANGS) def test_full_language_set_each_tag_resolves_to_itself(tag): # Against a full real-world language pool, every exact request must return its own track, so # that no tag captures another's: ar-EG must resolve to ar-EG, never to ar. - pool = _mk(*REAL_WORLD_AUDIO_LANGS) - assert _select([tag], pool) == [(tag, tag)] + pool = mk(*REAL_WORLD_AUDIO_LANGS) + assert select([tag], pool) == [(tag, tag)] def test_full_language_set_selects_every_track_once(): - pool = _mk(*REAL_WORLD_AUDIO_LANGS) + pool = mk(*REAL_WORLD_AUDIO_LANGS) selected = Tracks.by_language(pool, REAL_WORLD_AUDIO_LANGS, per_language=1, exact_match=True) assert len(selected) == len(REAL_WORLD_AUDIO_LANGS) assert {t.id for t in selected} == set(REAL_WORLD_AUDIO_LANGS) diff --git a/tests/remote/conftest.py b/tests/remote/conftest.py index fc90669b..c4133748 100644 --- a/tests/remote/conftest.py +++ b/tests/remote/conftest.py @@ -65,13 +65,13 @@ def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item item.add_marker(skip_live) -def _free_port() -> int: +def free_port() -> int: with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: s.bind(("127.0.0.1", 0)) return s.getsockname()[1] -def _wait_for_health(url: str, timeout: float = 60.0) -> bool: +def wait_for_health(url: str, timeout: float = 60.0) -> bool: import requests deadline = time.monotonic() + timeout @@ -87,11 +87,11 @@ def _wait_for_health(url: str, timeout: float = 60.0) -> bool: @pytest.fixture(autouse=True) -def _isolated_job_history(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: +def isolated_job_history(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: """Redirect job history writes to tmp_path so tests don't append to the server's api_history.jsonl.""" from unshackle.core.api import download_manager - monkeypatch.setattr(download_manager, "_history_path", lambda: tmp_path / "api_history.jsonl") + monkeypatch.setattr(download_manager, "history_path", lambda: tmp_path / "api_history.jsonl") @pytest.fixture(scope="session") @@ -120,19 +120,19 @@ def server_url(request: pytest.FixtureRequest): if mode == "never": if not explicit: pytest.fail("--spawn-serve=never requires --server-url") - if not _wait_for_health(explicit, timeout=10): + if not wait_for_health(explicit, timeout=10): pytest.fail(f"External serve not reachable at {explicit}") yield explicit return if mode == "auto" and explicit: - if not _wait_for_health(explicit, timeout=10): + if not wait_for_health(explicit, timeout=10): pytest.fail(f"External serve not reachable at {explicit}") yield explicit return # Spawn our own serve. - port = _free_port() + port = free_port() cmd = [ "uv", "run", @@ -153,7 +153,7 @@ def server_url(request: pytest.FixtureRequest): ) url = f"http://127.0.0.1:{port}" try: - if not _wait_for_health(url, timeout=60): + if not wait_for_health(url, timeout=60): proc.terminate() pytest.fail(f"Spawned serve at {url} did not become healthy within 60s") yield url diff --git a/tests/remote/e2e/conftest.py b/tests/remote/e2e/conftest.py index 222ae4fe..d676d921 100644 --- a/tests/remote/e2e/conftest.py +++ b/tests/remote/e2e/conftest.py @@ -10,7 +10,7 @@ from pathlib import Path import pytest -def _load_scenarios(config: pytest.Config) -> dict: +def load_scenarios(config: pytest.Config) -> dict: cached = getattr(config, "_e2e_scenarios_cache", None) if cached is not None: return cached @@ -31,7 +31,7 @@ def _load_scenarios(config: pytest.Config) -> dict: def pytest_generate_tests(metafunc: pytest.Metafunc) -> None: if "service_case" in metafunc.fixturenames: - scenarios = _load_scenarios(metafunc.config) + scenarios = load_scenarios(metafunc.config) params = [pytest.param((tag, conf), id=tag) for tag, conf in (scenarios.get("services") or {}).items()] if not params: params = [ diff --git a/tests/remote/e2e/test_live_download.py b/tests/remote/e2e/test_live_download.py index 730d87d1..cb8dcc35 100644 --- a/tests/remote/e2e/test_live_download.py +++ b/tests/remote/e2e/test_live_download.py @@ -20,7 +20,7 @@ import pytest pytestmark = [pytest.mark.live, pytest.mark.slow] -def _wait_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): +def wait_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): deadline = time.monotonic() + timeout while time.monotonic() < deadline: r = http_session.get(f"{server_url}/api/session/{sid}/titles", timeout=30) @@ -38,14 +38,14 @@ def _wait_titles(http_session, server_url: str, sid: str, timeout: float = 120.0 return None -def _pick_target_title(titles, season: int = 1, episode: int = 1): +def pick_target_title(titles, season: int = 1, episode: int = 1): for t in titles: if t.get("type") == "episode" and t.get("season") == season and t.get("number") == episode: return t return titles[0] if titles else None -def _pick_track_at_height(video_tracks, target_height: int): +def pick_track_at_height(video_tracks, target_height: int): same_height = [v for v in video_tracks if v.get("height") == target_height and v.get("range") == "SDR"] return sorted(same_height, key=lambda v: v.get("bitrate") or 0)[0] if same_height else None @@ -77,10 +77,10 @@ def test_download_manifest_fetch(http_session, server_url: str, service_case) -> sid = r.json()["session_id"] try: - body = _wait_titles(http_session, server_url, sid) + body = wait_titles(http_session, server_url, sid) if not body: pytest.skip(f"{service}: titles timeout") - target = _pick_target_title( + target = pick_target_title( body.get("titles") or [], season=conf.get("target_season", 1), episode=conf.get("target_episode", 1), @@ -94,7 +94,7 @@ def test_download_manifest_fetch(http_session, server_url: str, service_case) -> timeout=240, ) assert tr.status_code == 200, tr.text - track = _pick_track_at_height(tr.json().get("video") or [], target_height) + track = pick_track_at_height(tr.json().get("video") or [], target_height) if not track: pytest.skip(f"{service}: no track at height={target_height}") diff --git a/tests/remote/e2e/test_live_license.py b/tests/remote/e2e/test_live_license.py index 5828ce88..f87e895f 100644 --- a/tests/remote/e2e/test_live_license.py +++ b/tests/remote/e2e/test_live_license.py @@ -20,7 +20,7 @@ import pytest pytestmark = [pytest.mark.live, pytest.mark.slow] -def _wait_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): +def wait_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): deadline = time.monotonic() + timeout while time.monotonic() < deadline: r = http_session.get(f"{server_url}/api/session/{sid}/titles", timeout=30) @@ -38,14 +38,14 @@ def _wait_titles(http_session, server_url: str, sid: str, timeout: float = 120.0 return None -def _pick_target_title(titles, season: int = 1, episode: int = 1): +def pick_target_title(titles, season: int = 1, episode: int = 1): for t in titles: if t.get("type") == "episode" and t.get("season") == season and t.get("number") == episode: return t return titles[0] if titles else None -def _pick_track_at_height(video_tracks, target_height: int): +def pick_track_at_height(video_tracks, target_height: int): """Prefer SDR + AVC at the requested height; smallest bitrate wins.""" same_height = [v for v in video_tracks if v.get("height") == target_height] preferred = [v for v in same_height if v.get("codec") == "AVC" and v.get("range") == "SDR"] @@ -80,10 +80,10 @@ def test_license_server_cdm(http_session, server_url: str, service_case) -> None sid = r.json()["session_id"] try: - body = _wait_titles(http_session, server_url, sid) + body = wait_titles(http_session, server_url, sid) if not body: pytest.skip(f"{service}: titles timeout") - target = _pick_target_title( + target = pick_target_title( body.get("titles") or [], season=conf.get("target_season", 1), episode=conf.get("target_episode", 1), @@ -97,7 +97,7 @@ def test_license_server_cdm(http_session, server_url: str, service_case) -> None timeout=240, ) assert tr.status_code == 200, tr.text - track = _pick_track_at_height(tr.json().get("video") or [], target_height) + track = pick_track_at_height(tr.json().get("video") or [], target_height) if not track: pytest.skip(f"{service}: no track at height={target_height}") diff --git a/tests/remote/e2e/test_live_quality.py b/tests/remote/e2e/test_live_quality.py index 327c5eae..52b59911 100644 --- a/tests/remote/e2e/test_live_quality.py +++ b/tests/remote/e2e/test_live_quality.py @@ -18,7 +18,7 @@ import pytest pytestmark = [pytest.mark.live, pytest.mark.slow] -def _create_session(http_session, server_url: str, service: str, title_id: str) -> str: +def create_session(http_session, server_url: str, service: str, title_id: str) -> str: payload = { "service": service, "title_id": title_id, @@ -33,7 +33,7 @@ def _create_session(http_session, server_url: str, service: str, title_id: str) return r.json()["session_id"] -def _wait_for_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): +def wait_for_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): deadline = time.monotonic() + timeout while time.monotonic() < deadline: r = http_session.get(f"{server_url}/api/session/{sid}/titles", timeout=30) @@ -51,7 +51,7 @@ def _wait_for_titles(http_session, server_url: str, sid: str, timeout: float = 1 return 408, {"message": "auth timeout"} -def _pick_target_title(titles: list[dict[str, Any]], season: int = 1, episode: int = 1) -> Optional[dict[str, Any]]: +def pick_target_title(titles: list[dict[str, Any]], season: int = 1, episode: int = 1) -> Optional[dict[str, Any]]: """Pick the configured target episode; fall back to the first title.""" for t in titles: if t.get("type") == "episode" and t.get("season") == season and t.get("number") == episode: @@ -59,7 +59,7 @@ def _pick_target_title(titles: list[dict[str, Any]], season: int = 1, episode: i return titles[0] if titles else None -def _summarize(video_tracks: list[dict[str, Any]]) -> dict[str, Any]: +def summarize(video_tracks: list[dict[str, Any]]) -> dict[str, Any]: max_height = max((v.get("height") or 0 for v in video_tracks), default=0) max_width = max((v.get("width") or 0 for v in video_tracks), default=0) codecs = sorted({v.get("codec") for v in video_tracks if v.get("codec")}) @@ -76,25 +76,25 @@ def _summarize(video_tracks: list[dict[str, Any]]) -> dict[str, Any]: } -def _resolve_title_id(conf: dict) -> Optional[str]: +def resolve_title_id(conf: dict) -> Optional[str]: """Prefer series_url so S01E01 logic kicks in; fall back to movie title_url.""" return conf.get("series_url") or conf.get("title_url") def test_track_quality_meets_expected(http_session, server_url: str, service_case, capsys) -> None: service, conf = service_case - title_input = _resolve_title_id(conf) + title_input = resolve_title_id(conf) if not title_input: pytest.skip(f"{service}: no title_url/series_url in fixture") - sid = _create_session(http_session, server_url, service, title_input) + sid = create_session(http_session, server_url, service, title_input) try: - code, body = _wait_for_titles(http_session, server_url, sid) + code, body = wait_for_titles(http_session, server_url, sid) if code != 200: pytest.skip(f"{service}: titles fetch failed {code}: {str(body)[:200]}") titles = body.get("titles") or [] - target = _pick_target_title(titles, season=conf.get("target_season", 1), episode=conf.get("target_episode", 1)) + target = pick_target_title(titles, season=conf.get("target_season", 1), episode=conf.get("target_episode", 1)) if not target: pytest.skip(f"{service}: no titles returned") @@ -114,7 +114,7 @@ def test_track_quality_meets_expected(http_session, server_url: str, service_cas tracks = r.json() video = tracks.get("video") or [] - summary = _summarize(video) + summary = summarize(video) # Print summary even when test passes — discovery aid. with capsys.disabled(): print(f"\n[{service}] target={kind} '{target.get('name')}' -> {summary}") diff --git a/tests/remote/e2e/test_live_session_lifecycle.py b/tests/remote/e2e/test_live_session_lifecycle.py index 7ae7e51f..6a361698 100644 --- a/tests/remote/e2e/test_live_session_lifecycle.py +++ b/tests/remote/e2e/test_live_session_lifecycle.py @@ -9,7 +9,7 @@ import pytest pytestmark = [pytest.mark.live, pytest.mark.slow] -def _create_session(http_session, server_url: str, service: str, conf: dict) -> str: +def create_session(http_session, server_url: str, service: str, conf: dict) -> str: payload = {"service": service, "title_id": conf["title_url"]} r = http_session.post(f"{server_url}/api/session/create", json=payload, timeout=120) if r.status_code >= 400: @@ -20,7 +20,7 @@ def _create_session(http_session, server_url: str, service: str, conf: dict) -> return sid -def _wait_for_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): +def wait_for_titles(http_session, server_url: str, sid: str, timeout: float = 120.0): """Poll /titles until auth completes. Returns (status_code, response_json). Server returns 400 + auth_status=authenticating while auth is in-flight, @@ -44,20 +44,20 @@ def _wait_for_titles(http_session, server_url: str, sid: str, timeout: float = 1 return 408, {"message": "timeout waiting for auth"} -def _delete_session(http_session, server_url: str, sid: str) -> None: +def delete_session(http_session, server_url: str, sid: str) -> None: http_session.delete(f"{server_url}/api/session/{sid}", timeout=30) def test_session_create_then_delete(http_session, server_url: str, service_case) -> None: service, conf = service_case - sid = _create_session(http_session, server_url, service, conf) + sid = create_session(http_session, server_url, service, conf) try: r = http_session.get(f"{server_url}/api/session/{sid}", timeout=30) assert r.status_code == 200 info = r.json() assert info.get("session", {}).get("service_tag", service) == service or service in str(info) finally: - _delete_session(http_session, server_url, sid) + delete_session(http_session, server_url, sid) # After delete, info should 404 r2 = http_session.get(f"{server_url}/api/session/{sid}", timeout=30) @@ -66,9 +66,9 @@ def test_session_create_then_delete(http_session, server_url: str, service_case) def test_session_titles_returns_list(http_session, server_url: str, service_case) -> None: service, conf = service_case - sid = _create_session(http_session, server_url, service, conf) + sid = create_session(http_session, server_url, service, conf) try: - code, body = _wait_for_titles(http_session, server_url, sid) + code, body = wait_for_titles(http_session, server_url, sid) if code != 200: pytest.skip(f"{service}: titles fetch failed {code}: {str(body)[:200]}") assert "titles" in body @@ -78,14 +78,14 @@ def test_session_titles_returns_list(http_session, server_url: str, service_case assert "id" in first assert "type" in first finally: - _delete_session(http_session, server_url, sid) + delete_session(http_session, server_url, sid) def test_session_tracks_for_first_title(http_session, server_url: str, service_case) -> None: service, conf = service_case - sid = _create_session(http_session, server_url, service, conf) + sid = create_session(http_session, server_url, service, conf) try: - code, body = _wait_for_titles(http_session, server_url, sid) + code, body = wait_for_titles(http_session, server_url, sid) if code != 200: pytest.skip(f"{service}: titles fetch failed {code}: {str(body)[:200]}") titles = body.get("titles") or [] @@ -104,13 +104,13 @@ def test_session_tracks_for_first_title(http_session, server_url: str, service_c assert "video" in body or "audio" in body, body assert body.get("video") or body.get("audio"), body finally: - _delete_session(http_session, server_url, sid) + delete_session(http_session, server_url, sid) def test_session_delete_idempotent_returns_404_after(http_session, server_url: str, service_case) -> None: service, conf = service_case - sid = _create_session(http_session, server_url, service, conf) - _delete_session(http_session, server_url, sid) + sid = create_session(http_session, server_url, service, conf) + delete_session(http_session, server_url, sid) r = http_session.delete(f"{server_url}/api/session/{sid}", timeout=30) assert r.status_code in (404, 200) # tolerate both depending on server semantics diff --git a/tests/remote/unit/test_api_episode_part.py b/tests/remote/unit/test_api_episode_part.py index f1fb8bf9..fdcf21a5 100644 --- a/tests/remote/unit/test_api_episode_part.py +++ b/tests/remote/unit/test_api_episode_part.py @@ -26,7 +26,7 @@ class _FakeSvc: pass -def _ep(season: int = 1, number: int = 1, part: Optional[int] = None, **overrides: Any) -> Episode: +def ep(season: int = 1, number: int = 1, part: Optional[int] = None, **overrides: Any) -> Episode: base: Dict[str, Any] = dict( id_=f"ep-{season}-{number}-{part or 0}", service=_FakeSvc, @@ -58,7 +58,7 @@ class _FakeService: return _FakeTracks() -def _run_list_tracks(monkeypatch: pytest.MonkeyPatch, titles: List[Episode], **data: Any) -> Dict[str, Any]: +def run_list_tracks(monkeypatch: pytest.MonkeyPatch, titles: List[Episode], **data: Any) -> Dict[str, Any]: """Drive list_tracks_handler end to end with the service layer stubbed out.""" monkeypatch.setattr(handlers, "validate_service", lambda tag, request=None: "FAKE") monkeypatch.setattr(handlers, "setup_list_service", lambda *a, **kw: _FakeService(titles)) @@ -71,7 +71,7 @@ def _run_list_tracks(monkeypatch: pytest.MonkeyPatch, titles: List[Episode], **d return json.loads(response.body) -def _selected(result: Dict[str, Any]) -> List[str]: +def selected(result: Dict[str, Any]) -> List[str]: """Selection-syntax keys of whatever the handler returned, single or multi.""" entries = result["episodes"] if "episodes" in result else [{"title": result["title"]}] keys = [] @@ -86,7 +86,7 @@ def _selected(result: Dict[str, Any]) -> List[str]: def test_partless_episode_json_has_no_part_key() -> None: - d = serialize_title(_ep(part=None)) + d = serialize_title(ep(part=None)) assert "part" not in d assert list(d.keys()) == [ "type", @@ -105,26 +105,26 @@ def test_partless_episode_json_has_no_part_key() -> None: def test_partful_episode_json_carries_part_last() -> None: - d = serialize_title(_ep(part=2)) + d = serialize_title(ep(part=2)) assert d["part"] == 2 # additive: `part` is appended, every pre-existing key keeps its position - assert list(d.keys())[:11] == list(serialize_title(_ep(part=None)).keys()) + assert list(d.keys())[:11] == list(serialize_title(ep(part=None)).keys()) assert list(d.keys())[11] == "part" def test_part_stays_out_of_the_name_field() -> None: - """`name` is a reconstruction input for _build_title, not a display string.""" - assert serialize_title(_ep(part=2))["name"] == "Pilot" - assert serialize_title(_ep(part=3, name=None))["name"] == "Episode 01" + """`name` is a reconstruction input for build_title, not a display string.""" + assert serialize_title(ep(part=2))["name"] == "Pilot" + assert serialize_title(ep(part=3, name=None))["name"] == "Episode 01" def test_remote_build_title_round_trip() -> None: - from unshackle.core.remote_service import _build_title + from unshackle.core.remote_service import build_title - rebuilt = _build_title(serialize_title(_ep(1, 4, 2)), "FAKE", "fallback-id") + rebuilt = build_title(serialize_title(ep(1, 4, 2)), "FAKE", "fallback-id") assert (rebuilt.season, rebuilt.number, rebuilt.part, rebuilt.name) == (1, 4, 2, "Pilot") - partless = _build_title(serialize_title(_ep(1, 4, None)), "FAKE", "fallback-id") + partless = build_title(serialize_title(ep(1, 4, None)), "FAKE", "fallback-id") assert partless.part is None @@ -143,32 +143,32 @@ def test_movie_json_unaffected() -> None: @pytest.fixture def split_titles() -> List[Episode]: """E1 split into three parts, E2 unsplit.""" - return [_ep(1, 1, 1), _ep(1, 1, 2), _ep(1, 1, 3), _ep(1, 2, None)] + return [ep(1, 1, 1), ep(1, 1, 2), ep(1, 1, 3), ep(1, 2, None)] def test_rest_bare_episode_key_selects_every_part(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, wanted="S01E01") - assert _selected(result) == ["1x1.1", "1x1.2", "1x1.3"] + result = run_list_tracks(monkeypatch, split_titles, wanted="S01E01") + assert selected(result) == ["1x1.1", "1x1.2", "1x1.3"] def test_rest_part_key_selects_exactly_one(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, wanted="S01E01.2") - assert _selected(result) == ["1x1.2"] + result = run_list_tracks(monkeypatch, split_titles, wanted="S01E01.2") + assert selected(result) == ["1x1.2"] def test_rest_negative_part_key_excludes_only_that_part(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, wanted=["S01", "-S01E01.2"]) - assert _selected(result) == ["1x1.1", "1x1.3", "1x2"] + result = run_list_tracks(monkeypatch, split_titles, wanted=["S01", "-S01E01.2"]) + assert selected(result) == ["1x1.1", "1x1.3", "1x2"] def test_rest_partless_episode_unchanged(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, wanted="S01E02") - assert _selected(result) == ["1x2"] + result = run_list_tracks(monkeypatch, split_titles, wanted="S01E02") + assert selected(result) == ["1x2"] assert "part" not in result["title"] def test_rest_part_of_unsplit_episode_selects_nothing(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, wanted="S01E02.1") + result = run_list_tracks(monkeypatch, split_titles, wanted="S01E02.1") assert result["status"] == 404 @@ -176,28 +176,28 @@ def test_rest_part_of_unsplit_episode_selects_nothing(monkeypatch, split_titles) def test_discrete_fields_without_part_select_every_part(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, season=1, episode=1) - assert _selected(result) == ["1x1.1", "1x1.2", "1x1.3"] + result = run_list_tracks(monkeypatch, split_titles, season=1, episode=1) + assert selected(result) == ["1x1.1", "1x1.2", "1x1.3"] def test_discrete_fields_with_part_select_one(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, season=1, episode=1, part=3) - assert _selected(result) == ["1x1.3"] + result = run_list_tracks(monkeypatch, split_titles, season=1, episode=1, part=3) + assert selected(result) == ["1x1.3"] def test_discrete_part_accepts_string(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, season=1, episode=1, part="3") - assert _selected(result) == ["1x1.3"] + result = run_list_tracks(monkeypatch, split_titles, season=1, episode=1, part="3") + assert selected(result) == ["1x1.3"] def test_discrete_no_match_error_reports_the_part_qualified_key(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, season=1, episode=1, part=9) + result = run_list_tracks(monkeypatch, split_titles, season=1, episode=1, part=9) assert result["status"] == 404 assert result["details"]["wanted"] == "1x1.9" def test_discrete_partless_error_key_unchanged(monkeypatch, split_titles) -> None: - result = _run_list_tracks(monkeypatch, split_titles, season=9, episode=9) + result = run_list_tracks(monkeypatch, split_titles, season=9, episode=9) assert result["details"]["wanted"] == "9x9" @@ -211,15 +211,15 @@ def test_part_is_a_transport_key_not_a_service_option() -> None: def test_rest_multi_episode_order_is_stable_across_parts(monkeypatch) -> None: - titles = [_ep(1, 2, None), _ep(1, 1, 3), _ep(1, 1, 1), _ep(1, 1, 2)] - result = _run_list_tracks(monkeypatch, titles, wanted="S01") - assert _selected(result) == ["1x1.1", "1x1.2", "1x1.3", "1x2"] + titles = [ep(1, 2, None), ep(1, 1, 3), ep(1, 1, 1), ep(1, 1, 2)] + result = run_list_tracks(monkeypatch, titles, wanted="S01") + assert selected(result) == ["1x1.1", "1x1.2", "1x1.3", "1x2"] # ---------- download_manager: already-internal wanted sniff ---------- -_SNIFF = r"^!?\d+x\d+(\.\d+)?$" +SNIFF = r"^!?\d+x\d+(\.\d+)?$" @pytest.mark.parametrize( @@ -237,7 +237,7 @@ _SNIFF = r"^!?\d+x\d+(\.\d+)?$" ], ) def test_download_manager_sniff_regex(wanted: List[str], internal: bool) -> None: - needs_conversion = any(not re.match(_SNIFF, w) for w in wanted) + needs_conversion = any(not re.match(SNIFF, w) for w in wanted) assert needs_conversion is (not internal) @@ -248,12 +248,12 @@ def test_sniff_regex_matches_the_one_in_download_manager() -> None: import unshackle.core.api.download_manager as dm source = Path(dm.__file__).read_text(encoding="utf8") - assert f'needs_conversion = any(not re.match(r"{_SNIFF}", w) for w in wanted_raw)' in source + assert f'needs_conversion = any(not re.match(r"{SNIFF}", w) for w in wanted_raw)' in source def test_perform_download_skips_reconversion_for_preparsed_part_keys(monkeypatch) -> None: """A pre-parsed list with part and negative keys must not reach parse_tokens.""" - from unshackle.core.api.download_manager import _perform_download + from unshackle.core.api.download_manager import perform_download from unshackle.core.utils.click_types import SeasonRange calls = [] @@ -262,14 +262,14 @@ def test_perform_download_skips_reconversion_for_preparsed_part_keys(monkeypatch params: Dict[str, Any] = {"wanted": ["1x1", "1x1.2", "!1x1.3"]} with pytest.raises(BaseException): # blows up later on the unknown service; the wanted block runs first - _perform_download("job-1", "NoSuchServiceXYZ", "t1", params) + perform_download("job-1", "NoSuchServiceXYZ", "t1", params) assert calls == [] assert params["wanted"] == ["1x1", "1x1.2", "!1x1.3"] def test_perform_download_still_converts_cli_style_tokens(monkeypatch) -> None: - from unshackle.core.api.download_manager import _perform_download + from unshackle.core.api.download_manager import perform_download from unshackle.core.utils.click_types import SeasonRange calls = [] @@ -277,7 +277,7 @@ def test_perform_download_still_converts_cli_style_tokens(monkeypatch) -> None: params: Dict[str, Any] = {"wanted": "S01E01"} with pytest.raises(BaseException): - _perform_download("job-2", "NoSuchServiceXYZ", "t1", params) + perform_download("job-2", "NoSuchServiceXYZ", "t1", params) assert calls == [("S01E01",)] assert params["wanted"] == ["1x1"] @@ -325,7 +325,7 @@ def test_rest_filter_agrees_with_cli_filter(monkeypatch, split_titles, wanted_to if t.matches_wanted(wanted) ] - result = _run_list_tracks(monkeypatch, split_titles, wanted=list(wanted_tokens)) - rest_selected = [] if result.get("status") == 404 else _selected(result) + result = run_list_tracks(monkeypatch, split_titles, wanted=list(wanted_tokens)) + rest_selected = [] if result.get("status") == 404 else selected(result) assert sorted(rest_selected) == sorted(cli_selected) diff --git a/tests/remote/unit/test_api_title_fields.py b/tests/remote/unit/test_api_title_fields.py new file mode 100644 index 00000000..874eadac --- /dev/null +++ b/tests/remote/unit/test_api_title_fields.py @@ -0,0 +1,227 @@ +"""The title fields that cross the remote wire: air_date, absolute, anime and daily. + +Covers serialize_title's conditional keys, the service-class ANIME/DAILY stamp, and the +serialize_title / title_to_meta round trips through remote_service.build_title. +""" + +from __future__ import annotations + +import asyncio +import json +from datetime import date +from typing import Any, Dict, List, Optional + +import pytest +from langcodes import Language + +from unshackle.commands.dl import dl +from unshackle.core.api import handlers +from unshackle.core.api.handlers import serialize_title, stamp_service_flags +from unshackle.core.remote_service import build_title +from unshackle.core.titles.episode import Episode +from unshackle.core.titles.movie import Movie + +pytestmark = pytest.mark.unit + + +class _FakeSvc: + pass + + +def ep(**overrides: Any) -> Episode: + base: Dict[str, Any] = dict( + id_="ep-0001", + service=_FakeSvc, + title="My Show", + season=1, + number=1, + name="Pilot", + year=2024, + language=None, + ) + base.update(overrides) + return Episode(**base) + + +def _movie(**overrides: Any) -> Movie: + base: Dict[str, Any] = dict(id_="movie-0001", service=_FakeSvc, name="Film", year=2024, language=Language.get("en")) + base.update(overrides) + return Movie(**base) + + +# ---------- serialize_title: the new conditional keys ---------- + + +NEW_KEYS = ("air_date", "absolute", "daily", "anime") + + +def test_a_plain_episode_gains_no_new_keys() -> None: + d = serialize_title(ep()) + assert not [k for k in NEW_KEYS if k in d] + + +def test_a_plain_movie_gains_no_new_keys() -> None: + d = serialize_title(_movie()) + assert not [k for k in NEW_KEYS if k in d] + + +def test_air_date_is_serialized_as_an_iso_string() -> None: + assert serialize_title(ep(air_date=date(2026, 8, 11)))["air_date"] == "2026-08-11" + + +def test_an_unparseable_air_date_is_serialized_as_text() -> None: + assert serialize_title(ep(air_date="soon"))["air_date"] == "soon" + + +def test_absolute_is_serialized() -> None: + assert serialize_title(ep(absolute=57))["absolute"] == 57 + + +@pytest.mark.parametrize("value", [True, False]) +def test_per_title_daily_and_anime_are_serialized_both_ways(value: bool) -> None: + episode = ep() + episode.daily = value + episode.anime = value + d = serialize_title(episode) + assert (d["daily"], d["anime"]) == (value, value) + + +def test_a_movie_carries_the_anime_flag_but_never_daily() -> None: + movie = _movie() + movie.anime = True + d = serialize_title(movie) + assert d["anime"] is True + assert "daily" not in d + + +# ---------- the service-class ANIME/DAILY stamp ---------- + + +class _AnimeDailySvc: + ANIME = True + DAILY = True + + +class _PlainSvc: + pass + + +def test_the_stamp_fills_both_flags_from_the_service_class() -> None: + d = stamp_service_flags(serialize_title(ep()), _AnimeDailySvc()) + assert (d["anime"], d["daily"]) == (True, True) + + +def test_the_stamp_never_overrides_a_per_title_false() -> None: + episode = ep() + episode.anime = False + episode.daily = False + d = stamp_service_flags(serialize_title(episode), _AnimeDailySvc()) + assert (d["anime"], d["daily"]) == (False, False) + + +def test_a_plain_service_stamps_nothing() -> None: + d = stamp_service_flags(serialize_title(ep()), _PlainSvc()) + assert not [k for k in NEW_KEYS if k in d] + + +def test_a_movie_is_never_stamped_daily() -> None: + d = stamp_service_flags(serialize_title(_movie()), _AnimeDailySvc()) + assert d["anime"] is True + assert "daily" not in d + + +class _FakeService: + ANIME = True + DAILY = True + + def __init__(self, titles: List[Episode]) -> None: + self._titles = titles + + def get_titles(self) -> List[Episode]: + return self._titles + + +def test_list_titles_stamps_the_service_flags(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(handlers, "validate_service", lambda tag, request=None: "FAKE") + monkeypatch.setattr(handlers, "setup_list_service", lambda *a, **kw: _FakeService([ep()])) + response = asyncio.run(handlers.list_titles_handler({"service": "FAKE", "title_id": "t1"})) + title = json.loads(response.body)["titles"][0] + assert (title["anime"], title["daily"]) == (True, True) + + +# ---------- round trips through build_title ---------- + + +def round_trip(episode: Episode, stamp: Optional[Any] = None) -> Episode: + info = serialize_title(episode) + if stamp is not None: + info = stamp_service_flags(info, stamp) + return build_title(info, "FAKE", "fallback-id") + + +def test_serialize_round_trip_keeps_the_new_fields() -> None: + episode = ep(air_date=date(2026, 8, 11), absolute=57) + episode.daily = True + episode.anime = True + rebuilt = round_trip(episode) + assert rebuilt.air_date == date(2026, 8, 11) + assert rebuilt.absolute == 57 + assert (rebuilt.daily, rebuilt.anime) == (True, True) + + +def test_serialize_round_trip_leaves_unset_flags_inheriting() -> None: + rebuilt = round_trip(ep()) + assert rebuilt.air_date is None + assert rebuilt.absolute is None + assert (rebuilt.daily, rebuilt.anime) == (None, None) + + +def test_the_stamped_service_flags_survive_the_round_trip() -> None: + rebuilt = round_trip(ep(), stamp=_AnimeDailySvc()) + assert (rebuilt.daily, rebuilt.anime) == (True, True) + + +def test_a_per_title_false_survives_the_round_trip() -> None: + episode = ep() + episode.daily = False + rebuilt = round_trip(episode) + assert rebuilt.daily is False + + +def test_a_movie_round_trip_keeps_the_anime_flag() -> None: + movie = _movie() + movie.anime = True + rebuilt = build_title(serialize_title(movie), "FAKE", "fallback-id") + assert isinstance(rebuilt, Movie) + assert rebuilt.anime is True + + +# ---------- round trips through title_to_meta (the export/import path) ---------- + + +def test_title_to_meta_omits_the_new_keys_for_a_plain_episode() -> None: + meta = dl.title_to_meta(ep()) + assert not [k for k in ("absolute", "daily", "anime") if k in meta] + + +def test_title_to_meta_round_trip_keeps_the_new_fields() -> None: + episode = ep(air_date=date(2026, 8, 11), absolute=57) + episode.daily = True + episode.anime = True + rebuilt = build_title(dl.title_to_meta(episode), "FAKE", "fallback-id") + assert rebuilt.air_date == date(2026, 8, 11) + assert rebuilt.absolute == 57 + assert (rebuilt.daily, rebuilt.anime) == (True, True) + + +def test_title_to_meta_round_trip_leaves_unset_flags_inheriting() -> None: + rebuilt = build_title(dl.title_to_meta(ep()), "FAKE", "fallback-id") + assert rebuilt.absolute is None + assert (rebuilt.daily, rebuilt.anime) == (None, None) + + +def test_title_to_meta_round_trip_keeps_the_movie_anime_flag() -> None: + movie = _movie() + movie.anime = True + rebuilt = build_title(dl.title_to_meta(movie), "FAKE", "fallback-id") + assert rebuilt.anime is True diff --git a/tests/remote/unit/test_compression.py b/tests/remote/unit/test_compression.py index a7837dca..9139f22d 100644 --- a/tests/remote/unit/test_compression.py +++ b/tests/remote/unit/test_compression.py @@ -19,7 +19,7 @@ class _FakeReq: self.headers = {"Accept-Encoding": accept_encoding} -def _run(coro): +def run(coro): return asyncio.run(coro) @@ -30,7 +30,7 @@ def test_skips_when_client_does_not_accept_gzip() -> None: return web.json_response({"data": "x" * 4096}) req = _FakeReq(accept_encoding="identity") - resp = _run(compression_middleware(req, handler)) + resp = run(compression_middleware(req, handler)) assert resp.headers.get("Content-Encoding") != "gzip" assert resp.body == body_json or len(resp.body) >= len(body_json) - 8 @@ -39,7 +39,7 @@ def test_skips_when_body_below_threshold() -> None: async def handler(req): # noqa: ARG001 return web.json_response({"hi": "x"}) - resp = _run(compression_middleware(_FakeReq(), handler)) + resp = run(compression_middleware(_FakeReq(), handler)) assert resp.headers.get("Content-Encoding") != "gzip" @@ -47,7 +47,7 @@ def test_skips_non_json_response() -> None: async def handler(req): # noqa: ARG001 return web.Response(body=b"x" * 4096, content_type="text/plain") - resp = _run(compression_middleware(_FakeReq(), handler)) + resp = run(compression_middleware(_FakeReq(), handler)) assert resp.headers.get("Content-Encoding") != "gzip" @@ -57,7 +57,7 @@ def test_compresses_large_json_when_accepted() -> None: async def handler(req): # noqa: ARG001 return web.json_response(big) - resp = _run(compression_middleware(_FakeReq(), handler)) + resp = run(compression_middleware(_FakeReq(), handler)) assert resp.headers.get("Content-Encoding") == "gzip" decompressed = gzip.decompress(resp.body) assert json.loads(decompressed) == big diff --git a/tests/remote/unit/test_download_gates_redaction.py b/tests/remote/unit/test_download_gates_redaction.py index 15658475..91668ab3 100644 --- a/tests/remote/unit/test_download_gates_redaction.py +++ b/tests/remote/unit/test_download_gates_redaction.py @@ -12,9 +12,9 @@ from unshackle.core.api import handlers from unshackle.core.api.download_manager import ( DownloadJob, JobStatus, - _redact_parameters, _redact_text, - _secret_values, + redact_parameters, + secret_values, ) from unshackle.core.api.errors import APIError, APIErrorCode @@ -34,7 +34,7 @@ def test_redact_parameters_masks_secrets_and_proxy_userinfo(): "proxy": "http://bob:secret@proxy.example:8080", "quality": "1080p", } - red = _redact_parameters(params) + red = redact_parameters(params) assert red["credential"] == "***" assert red["password"] == "***" assert red["token"] == "***" @@ -45,11 +45,11 @@ def test_redact_parameters_masks_secrets_and_proxy_userinfo(): def test_redact_parameters_masks_credentials_dict(): - assert _redact_parameters({"credentials": {"default": "u:p"}})["credentials"] == "***" + assert redact_parameters({"credentials": {"default": "u:p"}})["credentials"] == "***" def test_secret_values_includes_password_half_and_dict_values(): - secrets = _secret_values({"credential": "user:hunter2", "credentials": {"d": "alice:wonder"}}) + secrets = secret_values({"credential": "user:hunter2", "credentials": {"d": "alice:wonder"}}) assert "user:hunter2" in secrets # full credential assert "hunter2" in secrets # password half of user:pass assert "alice:wonder" in secrets # value from the credentials map @@ -98,10 +98,10 @@ def stub_handler(monkeypatch): forbidden request raises APIError *before* the try block and an allowed one is caught inside it.""" monkeypatch.setattr(handlers, "validate_service", lambda tag, request=None: tag) - def _boom(*_args, **_kwargs): + def boom(*_args, **_kwargs): raise _PastGate() - monkeypatch.setattr(handlers.Services, "load", _boom) + monkeypatch.setattr(handlers.Services, "load", boom) return monkeypatch diff --git a/tests/remote/unit/test_download_manager.py b/tests/remote/unit/test_download_manager.py index bc973042..deb3d377 100644 --- a/tests/remote/unit/test_download_manager.py +++ b/tests/remote/unit/test_download_manager.py @@ -127,17 +127,17 @@ def test_normalize_sub_format_preserves_original_sentinel() -> None: None (as to_enum(["original"]) yields, since there is no such enum member), dl.py would fall into the default branch and re-encode TTML -> WebVTT, contradicting the CLI. """ - from unshackle.core.api.download_manager import _normalize_sub_format + from unshackle.core.api.download_manager import normalize_sub_format from unshackle.core.tracks import Subtitle from unshackle.core.utils.click_types import SubtitleCodecChoice # API path keeps the sentinel, case-insensitively. - assert _normalize_sub_format("original") == "original" - assert _normalize_sub_format("ORIGINAL") == "original" + assert normalize_sub_format("original") == "original" + assert normalize_sub_format("ORIGINAL") == "original" # The CLI produces the exact same sentinel value, so dl.py's `== "original"` branch handles both. assert SubtitleCodecChoice(Subtitle.Codec).convert("original") == "original" # A real codec still maps to its enum member; unknowns collapse to None. - assert _normalize_sub_format("srt") is Subtitle.Codec.SubRip - assert _normalize_sub_format("nonsense") is None + assert normalize_sub_format("srt") is Subtitle.Codec.SubRip + assert normalize_sub_format("nonsense") is None diff --git a/tests/remote/unit/test_handlers_serialize.py b/tests/remote/unit/test_handlers_serialize.py index 7924d886..0fe340d2 100644 --- a/tests/remote/unit/test_handlers_serialize.py +++ b/tests/remote/unit/test_handlers_serialize.py @@ -5,8 +5,10 @@ from __future__ import annotations import pytest from langcodes import Language +from unshackle.core.api.errors import APIError, APIErrorCode from unshackle.core.api.handlers import ( sanitize_log, + search_handler, serialize_audio_track, serialize_drm, serialize_subtitle_track, @@ -27,7 +29,7 @@ class _FakeSvc: pass -def _video(**overrides) -> Video: +def video(**overrides) -> Video: base = dict( url="https://example.com/v.mpd", language=Language.get("en"), @@ -44,7 +46,7 @@ def _video(**overrides) -> Video: return Video(**base) -def _audio(**overrides) -> Audio: +def audio(**overrides) -> Audio: base = dict( url="https://example.com/a.mpd", language=Language.get("en"), @@ -60,7 +62,7 @@ def _audio(**overrides) -> Audio: return Audio(**base) -def _subtitle(**overrides) -> Subtitle: +def subtitle(**overrides) -> Subtitle: base = dict( url="https://example.com/s.vtt", language=Language.get("en"), @@ -142,7 +144,7 @@ def test_serialize_title_episode_unnamed_falls_back_to_number() -> None: def test_serialize_video_track_basic() -> None: - d = serialize_video_track(_video()) + d = serialize_video_track(video()) assert d["id"] == "video-001" assert d["codec"] == "AVC" assert d["bitrate"] == 5000 # kbps @@ -155,12 +157,12 @@ def test_serialize_video_track_basic() -> None: def test_serialize_video_track_include_url() -> None: - d = serialize_video_track(_video(), include_url=True) + d = serialize_video_track(video(), include_url=True) assert d["url"] == "https://example.com/v.mpd" def test_serialize_audio_track_basic() -> None: - d = serialize_audio_track(_audio()) + d = serialize_audio_track(audio()) assert d["id"] == "audio-001" assert d["codec"] == "AAC" assert d["bitrate"] == 128 @@ -169,7 +171,7 @@ def test_serialize_audio_track_basic() -> None: def test_serialize_subtitle_track_basic() -> None: - d = serialize_subtitle_track(_subtitle(forced=True)) + d = serialize_subtitle_track(subtitle(forced=True)) assert d["id"] == "sub-001" assert d["codec"] == "WebVTT" assert d["forced"] is True @@ -287,3 +289,20 @@ def test_validate_download_params_accepts_valid_values() -> None: ) is None ) + + +async def test_search_missing_service_raises_invalid_input() -> None: + # MISSING_SERVICE was never an APIErrorCode member, so this raised AttributeError + # and routes.py surfaced a 500 instead of the intended 400. + with pytest.raises(APIError) as exc: + await search_handler({"query": "dune"}, None) + assert exc.value.error_code == APIErrorCode.INVALID_INPUT + assert APIError.default_http_status(exc.value.error_code) == 400 + + +async def test_search_unknown_service_raises_invalid_service() -> None: + # get_tag echoes unknown tags back, so the old `not normalized_service` guard + # never fired and Services.get_path leaked a KeyError as a 500. + with pytest.raises(APIError) as exc: + await search_handler({"service": "NOPE_THIS_IS_NOT_REAL_", "query": "x"}, None) + assert exc.value.error_code == APIErrorCode.INVALID_SERVICE diff --git a/tests/remote/unit/test_progress_sink.py b/tests/remote/unit/test_progress_sink.py index 7a7e7313..2d1175c9 100644 --- a/tests/remote/unit/test_progress_sink.py +++ b/tests/remote/unit/test_progress_sink.py @@ -45,7 +45,7 @@ class Subtitle: self.bitrate = None -def _noop(**kwargs): +def noop(**kwargs): pass @@ -66,7 +66,7 @@ def test_weight_video_over_audio_over_subtitle(): def test_weighting_makes_video_dominate_progress(): updates: list[dict] = [] video, sub = Video(bitrate=4_000_000), Subtitle() - cbs = build_job_progress_callables([video, sub], [_noop, _noop], updates.append) + cbs = build_job_progress_callables([video, sub], [noop, noop], updates.append) # subtitle fully done, video untouched -> progress is tiny (subtitle barely weighted) cbs[1](downloaded="Downloaded") @@ -80,7 +80,7 @@ def test_weighting_makes_video_dominate_progress(): def test_active_tracks_labels_reported_and_cleared_on_done(): updates: list[dict] = [] - cbs = build_job_progress_callables([Video(2160, "DV"), Audio("en-US", "2.0")], [_noop, _noop], updates.append) + cbs = build_job_progress_callables([Video(2160, "DV"), Audio("en-US", "2.0")], [noop, noop], updates.append) cbs[0](total=100, completed=10) # video downloading assert updates[-1]["active_tracks"] == ["video 2160p DV"] @@ -121,7 +121,7 @@ def test_aggregate_progress_is_monotonic_with_counts(): def test_all_tracks_done_reaches_download_ceiling(): # Downloads fill up to the ceiling; dl.result drives muxing the rest of the way to 100. updates: list[dict] = [] - cbs = build_job_progress_callables([Audio(bitrate=1000), Audio(bitrate=1000)], [_noop, _noop], updates.append) + cbs = build_job_progress_callables([Audio(bitrate=1000), Audio(bitrate=1000)], [noop, noop], updates.append) cbs[0](total=10, completed=10, downloaded="Downloaded") assert updates[-1]["progress"] < DOWNLOAD_PROGRESS_CEILING @@ -136,7 +136,7 @@ def test_finished_track_does_not_dip_when_callable_reused_for_decrypt(): """A track hits 100% (then decrypt reuses the callable with completed=0); the aggregate must hold, never dip - even before the terminal 'Downloaded'/'Decrypted' string arrives.""" updates: list[dict] = [] - cbs = build_job_progress_callables([Video(bitrate=1000), Video(bitrate=1000)], [_noop, _noop], updates.append) + cbs = build_job_progress_callables([Video(bitrate=1000), Video(bitrate=1000)], [noop, noop], updates.append) cbs[0](total=100, completed=100) # download hits 100% BEFORE any terminal string -> 50% cbs[0](total=200, completed=0) # decrypt phase resets counts, still no terminal string @@ -151,7 +151,7 @@ def test_finished_track_does_not_dip_when_callable_reused_for_decrypt(): def test_skipped_subtitle_counts_as_done(): updates: list[dict] = [] - cbs = build_job_progress_callables([Subtitle()], [_noop], updates.append) + cbs = build_job_progress_callables([Subtitle()], [noop], updates.append) cbs[0](downloaded="[yellow]SKIPPED") assert updates[-1]["completed_tracks"] == 1 assert updates[-1]["progress"] == pytest.approx(DOWNLOAD_PROGRESS_CEILING) diff --git a/tests/remote/unit/test_remote_client.py b/tests/remote/unit/test_remote_client.py index 4f6ef202..9ee379d9 100644 --- a/tests/remote/unit/test_remote_client.py +++ b/tests/remote/unit/test_remote_client.py @@ -85,6 +85,65 @@ def test_4xx_raises_systemexit_with_logged_error(client: RemoteClient, caplog: p assert "no such session" in caplog.text +@responses.activate +def test_401_falls_back_to_next_auth_header(client: RemoteClient) -> None: + seen: list[tuple[str | None, str | None]] = [] + + def cb(request): + seen.append((request.headers.get("X-Secret-Key"), request.headers.get("X-Api-Key"))) + if request.headers.get("X-Api-Key") == "secret-xyz": + return (200, {}, json.dumps({"status": "ok"})) + return (401, {}, json.dumps({"message": "Secret Key is Invalid."})) + + responses.add_callback(responses.GET, "http://srv:8786/api/services", callback=cb) + + assert client.get("/api/services") == {"status": "ok"} + assert seen == [("secret-xyz", None), (None, "secret-xyz")] + + # the winning header is sticky: no second attempt on the next request + seen.clear() + assert client.get("/api/services") == {"status": "ok"} + assert seen == [(None, "secret-xyz")] + + +@responses.activate +def test_non_401_error_does_not_fall_back(client: RemoteClient, caplog: pytest.LogCaptureFixture) -> None: + responses.add( + responses.GET, + "http://srv:8786/api/services", + json={"error_code": "FORBIDDEN", "message": "nope"}, + status=403, + ) + with caplog.at_level("ERROR"), pytest.raises(SystemExit): + client.get("/api/services") + assert len(responses.calls) == 1 + assert client.session.headers.get("X-Secret-Key") == "secret-xyz" + + +@responses.activate +def test_401_on_every_header_exits(client: RemoteClient, caplog: pytest.LogCaptureFixture) -> None: + responses.add( + responses.GET, + "http://srv:8786/api/services", + json={"error_code": "UNAUTHORIZED", "message": "Secret Key is Invalid."}, + status=401, + ) + with caplog.at_level("ERROR"), pytest.raises(SystemExit): + client.get("/api/services") + assert len(responses.calls) == 2 + assert "Secret Key is Invalid." in caplog.text + + +@responses.activate +def test_single_configured_auth_header_does_not_retry() -> None: + c = RemoteClient(server_url="http://srv:8786", api_key="k", auth_headers=["X-Api-Key"]) + responses.add(responses.GET, "http://srv:8786/api/services", json={"message": "nope"}, status=401) + with pytest.raises(SystemExit): + c.get("/api/services") + assert len(responses.calls) == 1 + assert c.session.headers.get("X-Api-Key") == "k" + + @responses.activate def test_connection_error_raises_systemexit(client: RemoteClient) -> None: import requests diff --git a/tests/remote/unit/test_remote_resolve.py b/tests/remote/unit/test_remote_resolve.py index ae31e77b..9c48cc98 100644 --- a/tests/remote/unit/test_remote_resolve.py +++ b/tests/remote/unit/test_remote_resolve.py @@ -1,11 +1,11 @@ -"""Unit tests for resolve_server / _resolve_proxy in unshackle.core.remote_service.""" +"""Unit tests for resolve_server / resolve_proxy_arg in unshackle.core.remote_service.""" from __future__ import annotations import click import pytest -from unshackle.core.remote_service import _resolve_proxy, resolve_server +from unshackle.core.remote_service import resolve_proxy_arg, resolve_server pytestmark = pytest.mark.unit @@ -49,6 +49,43 @@ def multi_remote_services(monkeypatch: pytest.MonkeyPatch) -> None: ) +def test_resolve_server_default_auth_headers(single_remote_service) -> None: + _, _, services = resolve_server("primary") + assert services["_auth_headers"] == ["X-Secret-Key", "X-Api-Key"] + + +def test_resolve_server_custom_auth_headers_extend_defaults(monkeypatch: pytest.MonkeyPatch) -> None: + from unshackle.core import remote_service as rs + + monkeypatch.setattr( + rs.config, "remote_services", {"p": {"url": "https://p:8080", "auth_headers": ["Authorization"]}} + ) + _, _, services = resolve_server("p") + assert services["_auth_headers"] == ["Authorization", "X-Secret-Key", "X-Api-Key"] + + +def test_resolve_server_auth_headers_dedup_is_case_insensitive(monkeypatch: pytest.MonkeyPatch) -> None: + from unshackle.core import remote_service as rs + + monkeypatch.setattr( + rs.config, + "remote_services", + {"p": {"url": "https://p:8080", "auth_headers": ["Authorization", "x-secret-key"]}}, + ) + _, _, services = resolve_server("p") + assert services["_auth_headers"] == ["Authorization", "x-secret-key", "X-Api-Key"] + + +@pytest.mark.parametrize("bad", ["X-Api-Key", [], [""], ["X-Api-Key", 3]]) +def test_resolve_server_bad_auth_headers_raises(monkeypatch: pytest.MonkeyPatch, bad: object) -> None: + from unshackle.core import remote_service as rs + + monkeypatch.setattr(rs.config, "remote_services", {"p": {"url": "https://p:8080", "auth_headers": bad}}) + with pytest.raises(click.ClickException) as exc: + resolve_server("p") + assert "auth_headers" in str(exc.value.message) + + def test_resolve_server_no_config_raises_click(empty_remote_services) -> None: with pytest.raises(click.ClickException) as exc: resolve_server(None) @@ -89,8 +126,8 @@ def test_resolve_server_multi_with_name(multi_remote_services) -> None: def test_resolve_proxy_none_returns_none() -> None: - assert _resolve_proxy(None) is None - assert _resolve_proxy("") is None + assert resolve_proxy_arg(None) is None + assert resolve_proxy_arg("") is None def test_resolve_proxy_passes_through_value(monkeypatch: pytest.MonkeyPatch) -> None: @@ -99,7 +136,7 @@ def test_resolve_proxy_passes_through_value(monkeypatch: pytest.MonkeyPatch) -> monkeypatch.setattr(resolve_mod, "initialize_proxy_providers", lambda: []) monkeypatch.setattr(resolve_mod, "resolve_proxy", lambda arg, providers: f"http://proxy/{arg}") - assert _resolve_proxy("us") == "http://proxy/us" + assert resolve_proxy_arg("us") == "http://proxy/us" def test_resolve_proxy_value_error_becomes_click(monkeypatch: pytest.MonkeyPatch) -> None: @@ -113,5 +150,5 @@ def test_resolve_proxy_value_error_becomes_click(monkeypatch: pytest.MonkeyPatch monkeypatch.setattr(resolve_mod, "resolve_proxy", boom) with pytest.raises(click.ClickException) as exc: - _resolve_proxy("xx") + resolve_proxy_arg("xx") assert "no such country" in str(exc.value.message) diff --git a/tests/remote/unit/test_remote_service_helpers.py b/tests/remote/unit/test_remote_service_helpers.py index 81665c37..55cf5ab9 100644 --- a/tests/remote/unit/test_remote_service_helpers.py +++ b/tests/remote/unit/test_remote_service_helpers.py @@ -7,14 +7,14 @@ from enum import Enum import pytest from unshackle.core.remote_service import ( - _build_title, - _build_tracks, - _deserialize_audio, - _deserialize_subtitle, - _deserialize_video, - _enum_get, - _match_track, - _reconstruct_drm, + build_title, + build_tracks, + deserialize_audio, + deserialize_subtitle, + deserialize_video, + enum_get, + match_track, + reconstruct_drm, ) from unshackle.core.titles.episode import Episode from unshackle.core.titles.movie import Movie @@ -29,19 +29,19 @@ class _Color(Enum): def test_enum_get_known() -> None: - assert _enum_get(_Color, "RED") is _Color.RED + assert enum_get(_Color, "RED") is _Color.RED def test_enum_get_unknown_returns_default() -> None: - assert _enum_get(_Color, "PURPLE", default=_Color.BLUE) is _Color.BLUE + assert enum_get(_Color, "PURPLE", default=_Color.BLUE) is _Color.BLUE def test_enum_get_none_returns_default() -> None: - assert _enum_get(_Color, None, default=_Color.RED) is _Color.RED + assert enum_get(_Color, None, default=_Color.RED) is _Color.RED def test_deserialize_video_minimal() -> None: - v = _deserialize_video({"id": "video-1", "codec": "AVC", "width": 1920, "height": 1080, "bitrate": 5000}) + v = deserialize_video({"id": "video-1", "codec": "AVC", "width": 1920, "height": 1080, "bitrate": 5000}) assert isinstance(v, Video) assert v.id == "video-1" assert v.codec is Video.Codec.AVC @@ -52,12 +52,12 @@ def test_deserialize_video_minimal() -> None: def test_deserialize_video_unknown_codec_falls_back_to_none() -> None: - v = _deserialize_video({"id": "v2", "codec": "MADE_UP", "width": 0, "height": 0}) + v = deserialize_video({"id": "v2", "codec": "MADE_UP", "width": 0, "height": 0}) assert v.codec is None def test_deserialize_audio_atmos_flag_sets_joc() -> None: - a = _deserialize_audio({"id": "a1", "codec": "AAC", "atmos": True, "channels": 6, "bitrate": 256}) + a = deserialize_audio({"id": "a1", "codec": "AAC", "atmos": True, "channels": 6, "bitrate": 256}) assert isinstance(a, Audio) assert a.joc == 1 assert a.channels == 6 @@ -65,34 +65,34 @@ def test_deserialize_audio_atmos_flag_sets_joc() -> None: def test_deserialize_audio_no_atmos() -> None: - a = _deserialize_audio({"id": "a2", "codec": "AAC", "channels": 2}) + a = deserialize_audio({"id": "a2", "codec": "AAC", "channels": 2}) assert a.joc == 0 def test_deserialize_subtitle_forced_flag() -> None: - s = _deserialize_subtitle({"id": "s1", "codec": "WebVTT", "language": "en", "forced": True}) + s = deserialize_subtitle({"id": "s1", "codec": "WebVTT", "language": "en", "forced": True}) assert isinstance(s, Subtitle) assert s.forced is True assert s.sdh is False def test_deserialize_subtitle_sdh_flag() -> None: - s = _deserialize_subtitle({"id": "s2", "codec": "WebVTT", "language": "en", "sdh": True}) + s = deserialize_subtitle({"id": "s2", "codec": "WebVTT", "language": "en", "sdh": True}) assert s.sdh is True assert s.forced is False def test_reconstruct_drm_empty() -> None: - assert _reconstruct_drm(None) == [] - assert _reconstruct_drm([]) == [] + assert reconstruct_drm(None) == [] + assert reconstruct_drm([]) == [] def test_reconstruct_drm_skips_entries_without_pssh() -> None: - assert _reconstruct_drm([{"type": "widevine"}]) == [] + assert reconstruct_drm([{"type": "widevine"}]) == [] def test_reconstruct_drm_invalid_pssh_silently_dropped() -> None: - assert _reconstruct_drm([{"type": "widevine", "pssh": "not-real-pssh"}]) == [] + assert reconstruct_drm([{"type": "widevine", "pssh": "not-real-pssh"}]) == [] def test_build_tracks_aggregates() -> None: @@ -102,33 +102,33 @@ def test_build_tracks_aggregates() -> None: "subtitles": [{"id": "s", "codec": "WebVTT", "language": "en"}], "attachments": [], } - t = _build_tracks(data) + t = build_tracks(data) assert len(t.videos) == 1 assert len(t.audio) == 1 assert len(t.subtitles) == 1 def test_match_track_by_id() -> None: - a = _deserialize_video({"id": "v1", "codec": "AVC", "width": 1920, "height": 1080}) - b = _deserialize_video({"id": "v2", "codec": "AVC", "width": 1280, "height": 720}) - remote = _deserialize_video({"id": "v2", "codec": "AVC", "width": 1280, "height": 720}) - assert _match_track(remote, [a, b]) is b + a = deserialize_video({"id": "v1", "codec": "AVC", "width": 1920, "height": 1080}) + b = deserialize_video({"id": "v2", "codec": "AVC", "width": 1280, "height": 720}) + remote = deserialize_video({"id": "v2", "codec": "AVC", "width": 1280, "height": 720}) + assert match_track(remote, [a, b]) is b def test_match_track_by_attributes_when_id_missing() -> None: - local = _deserialize_video({"id": "X", "codec": "AVC", "width": 1920, "height": 1080, "language": "en"}) - remote = _deserialize_video({"id": "Y", "codec": "AVC", "width": 1920, "height": 1080, "language": "en"}) - assert _match_track(remote, [local]) is local + local = deserialize_video({"id": "X", "codec": "AVC", "width": 1920, "height": 1080, "language": "en"}) + remote = deserialize_video({"id": "Y", "codec": "AVC", "width": 1920, "height": 1080, "language": "en"}) + assert match_track(remote, [local]) is local def test_match_track_no_candidates_returns_none() -> None: - remote = _deserialize_video({"id": "X", "codec": "AVC", "width": 1, "height": 1}) - assert _match_track(remote, []) is None + remote = deserialize_video({"id": "X", "codec": "AVC", "width": 1, "height": 1}) + assert match_track(remote, []) is None def test_build_title_movie() -> None: info = {"type": "movie", "id": "movie-0001", "name": "Foo", "year": 2024, "language": "en"} - title = _build_title(info, "ATV", "fallback") + title = build_title(info, "ATV", "fallback") assert isinstance(title, Movie) assert title.id == "movie-0001" assert title.name == "Foo" @@ -145,7 +145,7 @@ def test_build_title_episode() -> None: "year": 2024, "language": "en", } - title = _build_title(info, "ATV", "fallback") + title = build_title(info, "ATV", "fallback") assert isinstance(title, Episode) assert title.season == 1 assert title.number == 2 @@ -153,5 +153,5 @@ def test_build_title_episode() -> None: def test_build_title_falls_back_to_id_when_missing() -> None: - title = _build_title({"type": "movie", "name": "x"}, "ATV", "fallback-id") + title = build_title({"type": "movie", "name": "x"}, "ATV", "fallback-id") assert title.id == "fallback-id" diff --git a/tests/remote/unit/test_routes.py b/tests/remote/unit/test_routes.py index 95e0e2bc..39498f97 100644 --- a/tests/remote/unit/test_routes.py +++ b/tests/remote/unit/test_routes.py @@ -21,20 +21,20 @@ pytestmark = pytest.mark.unit def make_app(): """Factory that builds an aiohttp app for tests.""" - def _factory(remote_only: bool = False, with_auth_middleware: bool = False): + def factory(remote_only: bool = False, with_auth_middleware: bool = False): middlewares = [cors_middleware, compression_middleware] if with_auth_middleware: - middlewares.insert(1, _no_key_required_auth()) + middlewares.insert(1, no_key_required_auth()) app = web.Application(middlewares=middlewares) app["config"] = {"users": {}} app["debug_api"] = False setup_routes(app, remote_only=remote_only) return app - return _factory + return factory -def _no_key_required_auth(): +def no_key_required_auth(): """Mirror serve.py's api_key_authentication middleware: required X-Secret-Key on every endpoint except /api/health.""" @@ -52,13 +52,13 @@ def _no_key_required_auth(): return mw -def _collect_paths(app: web.Application) -> list[tuple[str, str]]: +def collect_paths(app: web.Application) -> list[tuple[str, str]]: return sorted({(r.method, r.resource.canonical) for r in app.router.routes()}) def test_setup_routes_full_mode_wires_all_endpoints(make_app) -> None: app = make_app(remote_only=False) - paths = _collect_paths(app) + paths = collect_paths(app) expected = { ("GET", "/api/health"), ("GET", "/api/services"), @@ -84,7 +84,7 @@ def test_setup_routes_full_mode_wires_all_endpoints(make_app) -> None: def test_setup_routes_remote_only_excludes_list_and_download(make_app) -> None: app = make_app(remote_only=True) - paths = set(_collect_paths(app)) + paths = set(collect_paths(app)) assert ("POST", "/api/list-titles") not in paths assert ("POST", "/api/list-tracks") not in paths assert ("POST", "/api/download") not in paths @@ -121,7 +121,7 @@ async def test_api_handler_maps_apierror(aiohttp_client) -> None: client = await aiohttp_client(app) resp = await client.get("/boom") - assert resp.status == 404 # NOT_FOUND -> 404 per errors.APIError._default_http_status + assert resp.status == 404 # NOT_FOUND -> 404 per errors.APIError.default_http_status body = await resp.json() assert body["status"] == "error" assert body["error_code"] == "NOT_FOUND" @@ -142,10 +142,10 @@ async def test_cors_preflight_returns_headers(make_app, aiohttp_client) -> None: async def test_health_endpoint_responds_ok(make_app, aiohttp_client, monkeypatch: pytest.MonkeyPatch) -> None: from unshackle.core.api import routes as routes_mod - async def _no_update(_): + async def no_update(_): return None - monkeypatch.setattr(routes_mod.UpdateChecker, "check_for_updates", _no_update) + monkeypatch.setattr(routes_mod.UpdateChecker, "check_for_updates", no_update) app = make_app(remote_only=True) client = await aiohttp_client(app) resp = await client.get("/api/health") @@ -158,10 +158,10 @@ async def test_health_endpoint_responds_ok(make_app, aiohttp_client, monkeypatch async def test_health_bypasses_api_key_auth_middleware(make_app, aiohttp_client, monkeypatch) -> None: from unshackle.core.api import routes as routes_mod - async def _no_update(_): + async def no_update(_): return None - monkeypatch.setattr(routes_mod.UpdateChecker, "check_for_updates", _no_update) + monkeypatch.setattr(routes_mod.UpdateChecker, "check_for_updates", no_update) app = make_app(remote_only=True, with_auth_middleware=True) client = await aiohttp_client(app) diff --git a/tests/remote/unit/test_serve_cli.py b/tests/remote/unit/test_serve_cli.py index 5ae0165c..cc5ab6d6 100644 --- a/tests/remote/unit/test_serve_cli.py +++ b/tests/remote/unit/test_serve_cli.py @@ -2,6 +2,8 @@ from __future__ import annotations +import re + import pytest from click.testing import CliRunner @@ -50,7 +52,9 @@ def test_serve_api_only_with_no_widevine_rejected(runner: CliRunner, monkeypatch result = runner.invoke(serve, ["--api-only", "--no-widevine", "--no-key"]) assert result.exit_code != 0 - assert "Cannot use --api-only" in (result.output or str(result.exception)) + # strip ANSI: rich-click forces color under GITHUB_ACTIONS and styles --flags mid-sentence + output = re.sub(r"\x1b\[[0-9;]*m", "", result.output or str(result.exception)) + assert "Cannot use --api-only" in output def test_serve_no_key_without_api_secret_does_not_require_secret( @@ -69,6 +73,61 @@ def test_serve_no_key_without_api_secret_does_not_require_secret( assert result.exit_code == 0, result.output +async def test_api_only_accepts_serve_users_keys( + runner: CliRunner, monkeypatch: pytest.MonkeyPatch, aiohttp_client +) -> None: + """In api-only keyed mode the accepted keys are api_secret plus every `serve.users` key.""" + from aiohttp import web + + captured: dict = {} + + def capture_app(app, **kwargs): + captured["app"] = app + + monkeypatch.setattr(web, "run_app", capture_app) + from unshackle.core.config import config as cfg + + monkeypatch.setattr( + cfg, + "serve", + {"api_secret": "master-key", "users": {"user-key": {"username": "alice", "services": ["EXAMPLE"]}}}, + ) + + result = runner.invoke(serve, ["--api-only", "--remote-only"]) + assert result.exit_code == 0, result.output + + app = captured["app"] + assert set(app["config"]["users"]) == {"master-key", "user-key"} + assert app["config"]["users"]["user-key"]["username"] == "alice" + + client = await aiohttp_client(app) + # an unrouted path still passes through the auth middleware first + assert (await client.get("/api/unknown", headers={"X-Secret-Key": "user-key"})).status == 404 + assert (await client.get("/api/unknown", headers={"X-Secret-Key": "master-key"})).status == 404 + assert (await client.get("/api/unknown", headers={"X-Secret-Key": "nope"})).status == 401 + + +async def test_api_only_coerces_non_str_user_keys( + runner: CliRunner, monkeypatch: pytest.MonkeyPatch, aiohttp_client +) -> None: + """A yaml users key that parses as int must still authenticate (compare_digest needs str).""" + from aiohttp import web + + captured: dict = {} + monkeypatch.setattr(web, "run_app", lambda app, **kwargs: captured.update(app=app)) + from unshackle.core.config import config as cfg + + monkeypatch.setattr(cfg, "serve", {"api_secret": "master-key", "users": {123456: {"username": "bob"}}}) + + result = runner.invoke(serve, ["--api-only", "--remote-only"]) + assert result.exit_code == 0, result.output + assert "123456" in captured["app"]["config"]["users"] + + client = await aiohttp_client(captured["app"]) + assert (await client.get("/api/unknown", headers={"X-Secret-Key": "123456"})).status == 404 + assert (await client.get("/api/unknown", headers={"X-Secret-Key": "master-key"})).status == 404 + + def test_serve_without_no_key_requires_api_secret(runner: CliRunner, monkeypatch: pytest.MonkeyPatch) -> None: from unshackle.core.config import config as cfg diff --git a/tests/remote/unit/test_server_cdm_gate.py b/tests/remote/unit/test_server_cdm_gate.py new file mode 100644 index 00000000..b2864297 --- /dev/null +++ b/tests/remote/unit/test_server_cdm_gate.py @@ -0,0 +1,56 @@ +"""Only keys that opt in may have the server run the CDM licensing.""" + +from types import SimpleNamespace + +import pytest + +from unshackle.core.api import handlers +from unshackle.core.api.errors import APIError, APIErrorCode + +pytestmark = pytest.mark.unit + +USERS = { + "tier1key": {"username": "tier1"}, + "tier3key": {"username": "tier3", "server_cdm": True}, +} + +ENTITLEMENTS = [ + ("tier3key", True), + ("tier1key", False), + ("adminsecret", True), # no users entry, e.g. api_secret + (None, True), +] + + +@pytest.fixture +def serve_users(monkeypatch): + monkeypatch.setattr(handlers.config, "serve", {"users": USERS}, raising=False) + monkeypatch.setattr(handlers, "request_secret_key", lambda request: request.headers.get("X-Secret-Key")) + + +def request(key): + return SimpleNamespace(headers={"X-Secret-Key": key} if key else {}) + + +@pytest.mark.parametrize(("key", "allowed"), ENTITLEMENTS) +def test_server_cdm_allowed(serve_users, key, allowed): + assert handlers.server_cdm_allowed(request(key)) is allowed + + +def test_no_request_allows(serve_users): + assert handlers.server_cdm_allowed(None) is True + + +@pytest.mark.parametrize(("key", "allowed"), ENTITLEMENTS) +def test_download_gate_follows_entitlement(serve_users, key, allowed): + # A download job licenses with the server CDM, so submission obeys the same gate. + if allowed: + handlers.enforce_download_gates({}, request(key)) + else: + with pytest.raises(APIError) as ei: + handlers.enforce_download_gates({}, request(key)) + assert ei.value.error_code == APIErrorCode.FORBIDDEN + + +def test_download_gate_without_request(serve_users): + handlers.enforce_download_gates({}) diff --git a/tests/remote/unit/test_session_store.py b/tests/remote/unit/test_session_store.py index c05499f8..0ebff67c 100644 --- a/tests/remote/unit/test_session_store.py +++ b/tests/remote/unit/test_session_store.py @@ -69,7 +69,7 @@ async def test_cleanup_expired_drops_old_authenticated(store: SessionStore, monk from datetime import datetime, timedelta, timezone entry = await store.create("ATV", _FakeService()) - entry.last_accessed = datetime.now(timezone.utc) - timedelta(seconds=store._ttl + 100) + entry.last_accessed = datetime.now(timezone.utc) - timedelta(seconds=store.ttl + 100) removed = await store.cleanup_expired() assert removed == 1 assert store.session_count == 0 @@ -118,7 +118,7 @@ async def test_get_session_store_returns_singleton() -> None: async def test_max_sessions_evicts_oldest(store: SessionStore, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr(type(store), "_max_sessions", property(lambda _: 2)) + monkeypatch.setattr(type(store), "max_sessions", property(lambda _: 2)) await store.create("A", _FakeService(), session_id="a") await asyncio.sleep(0.01) diff --git a/tests/tracks/test_encrypted_sample_entry.py b/tests/tracks/test_encrypted_sample_entry.py index 3874d2ce..9a9c70d2 100644 --- a/tests/tracks/test_encrypted_sample_entry.py +++ b/tests/tracks/test_encrypted_sample_entry.py @@ -14,73 +14,73 @@ from pathlib import Path from unshackle.core.tracks.track import has_encrypted_sample_entry -def _box(fmt: bytes, payload: bytes = b"") -> bytes: +def box(fmt: bytes, payload: bytes = b"") -> bytes: return struct.pack(">I", 8 + len(payload)) + fmt + payload -def _sample_entry(fmt: bytes, payload: bytes = b"") -> bytes: +def sample_entry(fmt: bytes, payload: bytes = b"") -> bytes: body = b"\x00" * 6 + struct.pack(">H", 1) + payload # reserved + data_reference_index return struct.pack(">I", 8 + len(body)) + fmt + body -def _stsd(entries: list[bytes]) -> bytes: +def stsd(entries: list[bytes]) -> bytes: body = b"\x00" * 4 + struct.pack(">I", len(entries)) + b"".join(entries) # version/flags + count - return _box(b"stsd", body) + return box(b"stsd", body) -def _moov(*children: bytes) -> bytes: - stbl = _box(b"stbl", b"".join(children)) - return _box(b"moov", _box(b"trak", _box(b"mdia", _box(b"minf", stbl)))) +def moov(*children: bytes) -> bytes: + stbl = box(b"stbl", b"".join(children)) + return box(b"moov", box(b"trak", box(b"mdia", box(b"minf", stbl)))) -FTYP = _box(b"ftyp", b"isom\x00\x00\x02\x00isomiso2") +FTYP = box(b"ftyp", b"isom\x00\x00\x02\x00isomiso2") # stray [size][encv] byte run as compressed A/V payload could randomly contain STRAY_ENCV = b"\xde\xad" * 4 + struct.pack(">I", 100) + b"encv" + b"\xbe\xef" * 8 -def _write(tmp_path: Path, name: str, *boxes: bytes) -> Path: +def write(tmp_path: Path, name: str, *boxes: bytes) -> Path: f = tmp_path / name f.write_bytes(b"".join(boxes)) return f def test_dual_stsd_encv_and_avc1_detected(tmp_path: Path) -> None: - sinf = _box(b"sinf", _box(b"frma", b"avc1")) - avc1 = _sample_entry(b"avc1", b"\x00" * 70) - encv = _sample_entry(b"encv", b"\x00" * 70 + sinf) - f = _write(tmp_path, "dual.mp4", FTYP, _moov(_stsd([avc1, encv])), _box(b"mdat", b"\x00" * 32)) + sinf = box(b"sinf", box(b"frma", b"avc1")) + avc1 = sample_entry(b"avc1", b"\x00" * 70) + encv = sample_entry(b"encv", b"\x00" * 70 + sinf) + f = write(tmp_path, "dual.mp4", FTYP, moov(stsd([avc1, encv])), box(b"mdat", b"\x00" * 32)) assert has_encrypted_sample_entry(f) is True def test_single_avc1_not_detected(tmp_path: Path) -> None: - avc1 = _sample_entry(b"avc1", b"\x00" * 70) - f = _write(tmp_path, "clear.mp4", FTYP, _moov(_stsd([avc1])), _box(b"mdat", b"\x00" * 32)) + avc1 = sample_entry(b"avc1", b"\x00" * 70) + f = write(tmp_path, "clear.mp4", FTYP, moov(stsd([avc1])), box(b"mdat", b"\x00" * 32)) assert has_encrypted_sample_entry(f) is False def test_stray_encv_in_mdat_not_detected(tmp_path: Path) -> None: # clean moov, mdat happens to contain a plausible [size][encv] run -> must NOT flag - avc1 = _sample_entry(b"avc1", b"\x00" * 70) - f = _write(tmp_path, "stray.mp4", FTYP, _moov(_stsd([avc1])), _box(b"mdat", STRAY_ENCV)) + avc1 = sample_entry(b"avc1", b"\x00" * 70) + f = write(tmp_path, "stray.mp4", FTYP, moov(stsd([avc1])), box(b"mdat", STRAY_ENCV)) assert has_encrypted_sample_entry(f) is False def test_leading_mdat_does_not_mask_real_encv(tmp_path: Path) -> None: # progressive layout (mdat before moov) with a stray run must not hide a genuine encv - enca = _sample_entry(b"enca", b"\x00" * 20 + _box(b"sinf", _box(b"frma", b"mp4a"))) - f = _write(tmp_path, "masked.mp4", FTYP, _box(b"mdat", STRAY_ENCV), _moov(_stsd([enca]))) + enca = sample_entry(b"enca", b"\x00" * 20 + box(b"sinf", box(b"frma", b"mp4a"))) + f = write(tmp_path, "masked.mp4", FTYP, box(b"mdat", STRAY_ENCV), moov(stsd([enca]))) assert has_encrypted_sample_entry(f) is True def test_text_containing_enca_in_moov_not_detected(tmp_path: Path) -> None: # "enca" inside a metadata string has no plausible box size before it -> must NOT flag - avc1 = _sample_entry(b"avc1", b"\x00" * 70) - udta = _box(b"udta", b"data is encapsulated here") - f = _write(tmp_path, "text.mp4", FTYP, _box(b"moov", _stsd([avc1]) + udta)) + avc1 = sample_entry(b"avc1", b"\x00" * 70) + udta = box(b"udta", b"data is encapsulated here") + f = write(tmp_path, "text.mp4", FTYP, box(b"moov", stsd([avc1]) + udta)) assert has_encrypted_sample_entry(f) is False def test_missing_or_bogus_file(tmp_path: Path) -> None: assert has_encrypted_sample_entry(tmp_path / "nope.mp4") is False - bogus = _write(tmp_path, "bogus.bin", b"\x00\x01\x02\x03not-an-mp4") + bogus = write(tmp_path, "bogus.bin", b"\x00\x01\x02\x03not-an-mp4") assert has_encrypted_sample_entry(bogus) is False diff --git a/tests/tracks/test_fragment_decrypted.py b/tests/tracks/test_fragment_decrypted.py index 6396b4fa..7dc5adda 100644 --- a/tests/tracks/test_fragment_decrypted.py +++ b/tests/tracks/test_fragment_decrypted.py @@ -19,54 +19,54 @@ from unshackle.core.manifests.ism_init import PIFF_SENC_UUID from unshackle.core.tracks.track import assert_fragments_decrypted -def _box(fmt: bytes, payload: bytes = b"") -> bytes: +def box(fmt: bytes, payload: bytes = b"") -> bytes: return struct.pack(">I", 8 + len(payload)) + fmt + payload -def _senc(sample_count: int = 4) -> bytes: - return _box(b"senc", b"\x00" * 4 + struct.pack(">I", sample_count) + b"\x00" * (8 * sample_count)) +def senc(sample_count: int = 4) -> bytes: + return box(b"senc", b"\x00" * 4 + struct.pack(">I", sample_count) + b"\x00" * (8 * sample_count)) -def _piff_senc(sample_count: int = 4) -> bytes: +def piff_senc(sample_count: int = 4) -> bytes: body = PIFF_SENC_UUID + b"\x00" * 4 + struct.pack(">I", sample_count) + b"\x00" * (8 * sample_count) - return _box(b"uuid", body) + return box(b"uuid", body) -def _moof(*children: bytes, sdi: int = 1) -> bytes: - tfhd = _box(b"tfhd", b"\x00\x00\x00\x02" + struct.pack(">II", 1, sdi)) # sample-description-index-present - return _box(b"moof", _box(b"mfhd", b"\x00" * 8) + _box(b"traf", tfhd + b"".join(children))) +def moof(*children: bytes, sdi: int = 1) -> bytes: + tfhd = box(b"tfhd", b"\x00\x00\x00\x02" + struct.pack(">II", 1, sdi)) # sample-description-index-present + return box(b"moof", box(b"mfhd", b"\x00" * 8) + box(b"traf", tfhd + b"".join(children))) -FTYP = _box(b"ftyp", b"isom\x00\x00\x02\x00isomiso2") -MOOV = _box(b"moov", _box(b"mvex", _box(b"trex", b"\x00" * 24))) -MDAT = _box(b"mdat", b"\x00" * 64) +FTYP = box(b"ftyp", b"isom\x00\x00\x02\x00isomiso2") +MOOV = box(b"moov", box(b"mvex", box(b"trex", b"\x00" * 24))) +MDAT = box(b"mdat", b"\x00" * 64) # a compressed payload can contain anything, including plausible-looking box runs -STRAY = _box(b"mdat", struct.pack(">I", 64) + b"senc" + b"\xbe\xef" * 8 + PIFF_SENC_UUID) +STRAY = box(b"mdat", struct.pack(">I", 64) + b"senc" + b"\xbe\xef" * 8 + PIFF_SENC_UUID) -def _write(tmp_path: Path, name: str, *boxes: bytes) -> Path: +def write(tmp_path: Path, name: str, *boxes: bytes) -> Path: f = tmp_path / name f.write_bytes(b"".join(boxes)) return f def test_all_fragments_decrypted_passes(tmp_path: Path) -> None: - f = _write(tmp_path, "clean.mp4", FTYP, MOOV, _moof(), MDAT, _moof(sdi=2), MDAT) + f = write(tmp_path, "clean.mp4", FTYP, MOOV, moof(), MDAT, moof(sdi=2), MDAT) assert_fragments_decrypted(f) def test_surviving_senc_raises(tmp_path: Path) -> None: # the real bug: fragment 1 (index 1) decrypted, the rest (index 2) silently skipped - boxes = [FTYP, MOOV, _moof(), MDAT] + boxes = [FTYP, MOOV, moof(), MDAT] for _ in range(3): - boxes += [_moof(_senc(), sdi=2), MDAT] - f = _write(tmp_path, "skipped.mp4", *boxes) + boxes += [moof(senc(), sdi=2), MDAT] + f = write(tmp_path, "skipped.mp4", *boxes) with pytest.raises(ValueError, match=r"3/4 fragment"): assert_fragments_decrypted(f) def test_surviving_cenc_senc_raises(tmp_path: Path) -> None: - f = _write(tmp_path, "cenc.mp4", FTYP, MOOV, _moof(_senc()), MDAT) + f = write(tmp_path, "cenc.mp4", FTYP, MOOV, moof(senc()), MDAT) with pytest.raises(ValueError, match=r"1/1 fragment"): assert_fragments_decrypted(f) @@ -76,59 +76,59 @@ def test_leftover_piff_uuid_after_senc_removed_does_not_raise(tmp_path: Path) -> # consumed senc, detached senc/saiz/saio, and left the uuid on a fully decrypted fragment. boxes = [FTYP, MOOV] for _ in range(3): - boxes += [_moof(_piff_senc()), MDAT] - f = _write(tmp_path, "piff_leftover.mp4", *boxes) + boxes += [moof(piff_senc()), MDAT] + f = write(tmp_path, "piff_leftover.mp4", *boxes) assert_fragments_decrypted(f) def test_stray_senc_bytes_in_mdat_ignored(tmp_path: Path) -> None: - f = _write(tmp_path, "stray.mp4", FTYP, MOOV, _moof(), STRAY, _moof(sdi=2), STRAY) + f = write(tmp_path, "stray.mp4", FTYP, MOOV, moof(), STRAY, moof(sdi=2), STRAY) assert_fragments_decrypted(f) def test_senc_text_without_plausible_size_ignored(tmp_path: Path) -> None: # "senc" inside a free-box string has no valid box size before it - f = _write(tmp_path, "text.mp4", FTYP, MOOV, _moof(_box(b"free", b"presence sencoded here")), MDAT) + f = write(tmp_path, "text.mp4", FTYP, MOOV, moof(box(b"free", b"presence sencoded here")), MDAT) assert_fragments_decrypted(f) def test_senc_bytes_inside_trun_payload_ignored(tmp_path: Path) -> None: # a trun sample entry spelling "senc" after a size-looking duration; only a byte scan bites - trun = _box(b"trun", b"\x00\x00\x03\x01" + struct.pack(">I", 2) + struct.pack(">I", 512) + b"senc") - f = _write(tmp_path, "trun.mp4", FTYP, MOOV, _moof(trun), MDAT) + trun = box(b"trun", b"\x00\x00\x03\x01" + struct.pack(">I", 2) + struct.pack(">I", 512) + b"senc") + f = write(tmp_path, "trun.mp4", FTYP, MOOV, moof(trun), MDAT) assert_fragments_decrypted(f) def test_empty_senc_on_clear_fragment_ignored(tmp_path: Path) -> None: # sample_count 0 marks a fragment with no protected samples, so the guard ignores it - f = _write(tmp_path, "empty.mp4", FTYP, MOOV, _moof(_senc(sample_count=0)), MDAT) + f = write(tmp_path, "empty.mp4", FTYP, MOOV, moof(senc(sample_count=0)), MDAT) assert_fragments_decrypted(f) -def _senc_override(sample_count: int) -> bytes: +def senc_override(sample_count: int) -> bytes: # flags&1 => AlgorithmID(3) + IV_size(1) + KID(16) precede sample_count head = b"\x00\x00\x00\x01" + b"\x00\x00\x08" + b"\x08" + b"\xaa" * 16 - return _box(b"senc", head + struct.pack(">I", sample_count) + b"\x00" * (8 * sample_count)) + return box(b"senc", head + struct.pack(">I", sample_count) + b"\x00" * (8 * sample_count)) def test_override_flag_shifts_sample_count_offset(tmp_path: Path) -> None: - clear = _write(tmp_path, "ovr_clear.mp4", FTYP, MOOV, _moof(_senc_override(0)), MDAT) + clear = write(tmp_path, "ovr_clear.mp4", FTYP, MOOV, moof(senc_override(0)), MDAT) assert_fragments_decrypted(clear) - f = _write(tmp_path, "ovr.mp4", FTYP, MOOV, _moof(_senc_override(5)), MDAT) + f = write(tmp_path, "ovr.mp4", FTYP, MOOV, moof(senc_override(5)), MDAT) with pytest.raises(ValueError, match=r"1/1 fragment"): assert_fragments_decrypted(f) def test_senc_outside_traf_ignored(tmp_path: Path) -> None: # a decrypted moof whose only "senc" sits in a sibling box outside any traf - f = _write(tmp_path, "sibling.mp4", FTYP, MOOV, _box(b"moof", _box(b"mfhd", b"\x00" * 8) + _senc()), MDAT) + f = write(tmp_path, "sibling.mp4", FTYP, MOOV, box(b"moof", box(b"mfhd", b"\x00" * 8) + senc()), MDAT) assert_fragments_decrypted(f) def test_senc_nested_under_non_traf_ignored(tmp_path: Path) -> None: # senc two levels down under a non-traf container: only traf children count - meta = _box(b"meta", _senc()) - f = _write(tmp_path, "nested.mp4", FTYP, MOOV, _box(b"moof", _box(b"mfhd", b"\x00" * 8) + meta), MDAT) + meta = box(b"meta", senc()) + f = write(tmp_path, "nested.mp4", FTYP, MOOV, box(b"moof", box(b"mfhd", b"\x00" * 8) + meta), MDAT) assert_fragments_decrypted(f) @@ -136,17 +136,17 @@ def test_undersized_largesize_box_does_not_desync_walk(tmp_path: Path, caplog: p # size==1 reads a 16-byte header, so a largesize of 12 clears the 8-byte floor but # still stalls the cursor onto payload; only a header-aware floor catches it runt = struct.pack(">I", 1) + b"free" + struct.pack(">Q", 12) - f = _write(tmp_path, "runt.mp4", FTYP, MOOV, runt, _moof(_senc()), MDAT, _moof(_senc()), MDAT) + f = write(tmp_path, "runt.mp4", FTYP, MOOV, runt, moof(senc()), MDAT, moof(senc()), MDAT) with caplog.at_level("WARNING"): assert_fragments_decrypted(f) assert "malformed box size" in caplog.text def test_largesize_and_zero_size_boxes_walked(tmp_path: Path) -> None: - frag = _moof(_senc()) + frag = moof(senc()) large = struct.pack(">I", 1) + b"mdat" + struct.pack(">Q", 16 + 64) + b"\x00" * 64 trailing = struct.pack(">I", 0) + b"mdat" + b"\x00" * 64 # size 0 => runs to EOF - f = _write(tmp_path, "large.mp4", FTYP, MOOV, frag, large, trailing) + f = write(tmp_path, "large.mp4", FTYP, MOOV, frag, large, trailing) with pytest.raises(ValueError, match=r"1/1 fragment"): assert_fragments_decrypted(f) @@ -154,7 +154,7 @@ def test_largesize_and_zero_size_boxes_walked(tmp_path: Path) -> None: def test_malformed_box_size_warns_and_does_not_raise(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None: # failing open is deliberate, but the abort must be logged so it cannot pass as clean overrun = struct.pack(">I", 1 << 30) + b"free" + b"\x00" * 8 - f = _write(tmp_path, "overrun.mp4", FTYP, MOOV, overrun, _moof(_senc()), MDAT) + f = write(tmp_path, "overrun.mp4", FTYP, MOOV, overrun, moof(senc()), MDAT) with caplog.at_level("WARNING"): assert_fragments_decrypted(f) assert "malformed box size" in caplog.text @@ -162,6 +162,6 @@ def test_malformed_box_size_warns_and_does_not_raise(tmp_path: Path, caplog: pyt def test_missing_or_bogus_file_is_noop(tmp_path: Path) -> None: assert_fragments_decrypted(tmp_path / "nope.mp4") - assert_fragments_decrypted(_write(tmp_path, "bogus.bin", b"\x00\x01\x02\x03not-an-mp4")) - truncated = _write(tmp_path, "cut.mp4", FTYP, MOOV, _moof(_senc())[:20]) + assert_fragments_decrypted(write(tmp_path, "bogus.bin", b"\x00\x01\x02\x03not-an-mp4")) + truncated = write(tmp_path, "cut.mp4", FTYP, MOOV, moof(senc())[:20]) assert_fragments_decrypted(truncated) diff --git a/tests/tracks/test_mux_embedded_audio_metadata.py b/tests/tracks/test_mux_embedded_audio_metadata.py new file mode 100644 index 00000000..83191c97 --- /dev/null +++ b/tests/tracks/test_mux_embedded_audio_metadata.py @@ -0,0 +1,52 @@ +import pytest + +from unshackle.core import binaries +from unshackle.core.tracks import Tracks, Video + + +def make_video(track_id: str, **data) -> Video: + video = Video( + id_=track_id, + url=f"https://example.test/{track_id}.m3u8", + language="zh", + codec=Video.Codec.AVC, + range_=Video.Range.SDR, + width=1920, + height=1080, + bitrate=5_000_000, + ) + video.data.update(data) + return video + + +def mux_until_mkvmerge_check(tracks: Tracks) -> None: + # the embedded-audio metadata pass runs before the mkvmerge lookup, so a missing + # binary stops the mux right after the state under test has been written + with pytest.raises(RuntimeError, match="MKVToolNix"): + tracks.mux("Test Title") + + +def test_names_undeclared_video(monkeypatch): + monkeypatch.setattr(binaries, "MKVToolNix", None) + video = make_video("v1") + tracks = Tracks(video) + + mux_until_mkvmerge_check(tracks) + + assert video.needs_repack + assert video.data["audio_language"] == "zh" + assert video.data["audio_language_name"] == "Chinese" + + +def test_service_declaration_is_preserved(monkeypatch): + monkeypatch.setattr(binaries, "MKVToolNix", None) + declared = make_video("v1", audio_language="th", audio_language_name="Thai [Original]") + undeclared = make_video("v2") + tracks = Tracks([declared, undeclared]) + + mux_until_mkvmerge_check(tracks) + + assert declared.data["audio_language"] == "th" + assert declared.data["audio_language_name"] == "Thai [Original]" + assert not declared.needs_repack + assert undeclared.data["audio_language"] == "zh" diff --git a/tests/tracks/test_subtitle_convert.py b/tests/tracks/test_subtitle_convert.py index c7cf3a4d..aa220dde 100644 --- a/tests/tracks/test_subtitle_convert.py +++ b/tests/tracks/test_subtitle_convert.py @@ -50,7 +50,7 @@ Dialogue: 0,0:00:03.00,0:00:04.00,Default,,0,0,0,,World @pytest.fixture(autouse=True) -def _no_subtitleedit(monkeypatch): +def no_subtitleedit(monkeypatch): """Default every test to a SubtitleEdit-less environment; tests opt in when needed.""" monkeypatch.setattr(binaries, "SubtitleEdit", None) diff --git a/tests/tracks/test_subtitle_sdh_strip.py b/tests/tracks/test_subtitle_sdh_strip.py new file mode 100644 index 00000000..6a2b2ae2 --- /dev/null +++ b/tests/tracks/test_subtitle_sdh_strip.py @@ -0,0 +1,70 @@ +"""``strip_hearing_impaired`` fallback path (``tracks/subtitle.py``). + +Pins the filter-subs fallback's warn-and-continue branch: an invalid subtitle file must +log a warning and return, not crash (it previously raised AttributeError via a +nonexistent ``self.log``). +""" + +from __future__ import annotations + +import pytest + +from unshackle.core import binaries +from unshackle.core.config import config +from unshackle.core.tracks.subtitle import Subtitle + +SRT_SAMPLE = """1 +00:00:01,000 --> 00:00:02,000 +Hello +""" + + +@pytest.fixture() +def srt_track(tmp_path): + path = tmp_path / "sub.srt" + path.write_text(SRT_SAMPLE, encoding="utf-8") + sub = Subtitle(url="https://example.test/x", language="en", codec=Subtitle.Codec.SubRip) + sub.path = path + return sub + + +def test_invalid_subtitle_warns_and_continues(srt_track, monkeypatch, caplog): + import unshackle.core.tracks.subtitle as subtitle_mod + + monkeypatch.setattr(binaries, "SubtitleEdit", None) + monkeypatch.setattr(config, "subtitle", {"sdh_method": "auto"}, raising=False) + + def broken(*args, **kwargs): + raise RuntimeError("subby unavailable") + + monkeypatch.setattr(subtitle_mod, "CommonIssuesFixer", broken) + + def invalid(path): + raise OSError(f"{path} is not valid subtitle file") + + monkeypatch.setattr(subtitle_mod, "Subtitles", invalid) + + with caplog.at_level("WARNING", logger="Subtitle"): + srt_track.strip_hearing_impaired() + + assert any("Failed to strip SDH" in r.message for r in caplog.records) + + +def test_other_oserror_still_raises(srt_track, monkeypatch): + import unshackle.core.tracks.subtitle as subtitle_mod + + monkeypatch.setattr(binaries, "SubtitleEdit", None) + monkeypatch.setattr(config, "subtitle", {"sdh_method": "auto"}, raising=False) + + def broken(*args, **kwargs): + raise RuntimeError("subby unavailable") + + monkeypatch.setattr(subtitle_mod, "CommonIssuesFixer", broken) + + def denied(path): + raise PermissionError("permission denied") + + monkeypatch.setattr(subtitle_mod, "Subtitles", denied) + + with pytest.raises(PermissionError): + srt_track.strip_hearing_impaired() diff --git a/tests/tracks/test_subtitle_skip.py b/tests/tracks/test_subtitle_skip.py index 02c0c868..52080c5d 100644 --- a/tests/tracks/test_subtitle_skip.py +++ b/tests/tracks/test_subtitle_skip.py @@ -77,7 +77,7 @@ class Harness: @pytest.fixture(autouse=True) -def _clear_event(): +def clear_event(): DOWNLOAD_CANCELLED.clear() yield DOWNLOAD_CANCELLED.clear() diff --git a/unshackle/commands/dl.py b/unshackle/commands/dl.py index 2eb9429e..f4466439 100644 --- a/unshackle/commands/dl.py +++ b/unshackle/commands/dl.py @@ -16,6 +16,7 @@ from collections.abc import Iterable from concurrent import futures from concurrent.futures import ThreadPoolExecutor from copy import deepcopy +from datetime import date, timedelta from functools import partial from http.cookiejar import CookieJar, MozillaCookieJar from itertools import product @@ -27,8 +28,9 @@ from uuid import UUID import click import yaml from click.core import ParameterSource -from langcodes import Language +from langcodes import Language, tag_is_valid from pymediainfo import MediaInfo +from rich import box from rich.console import Group from rich.padding import Padding from rich.panel import Panel @@ -59,7 +61,8 @@ from unshackle.core.music import ( write_music_manifest, write_music_metadata, ) -from unshackle.core.providers.tvdb import SEASON_TYPES +from unshackle.core.providers.anilist import parse_anilist_ref +from unshackle.core.providers.tvdb import SEASON_TYPES, parse_int from unshackle.core.proxies import Basic, ExpressVPN, Gluetun, Hola, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN from unshackle.core.service import Service from unshackle.core.services import Services @@ -74,6 +77,8 @@ from unshackle.core.tracks.hybrid import Hybrid from unshackle.core.tracks.track import assert_fragments_decrypted, has_encrypted_sample_entry from unshackle.core.utilities import ( as_requested, + embedded_audio_langs, + excluded_language_tags, find_font_with_fallbacks, find_missing_langs, get_debug_logger, @@ -83,6 +88,7 @@ from unshackle.core.utilities import ( is_exact_match, keep_forced_subtitle, log_event, + partition_exclusions, resolve_sort_langs, suggest_font_packages, time_elapsed_since, @@ -119,11 +125,65 @@ class SkippedSubtitle(TypedDict): DL_OPTION_ALIASES = {"range": "range_", "list": "list_"} +def parse_language(tag: Optional[str]) -> Optional[Language]: + """A provider's original-language tag as a Language, or None if it is absent or not a real tag.""" + tag = (tag or "").strip() + if not tag or not tag_is_valid(tag) or tag.startswith("und"): + return None + return Language.get(tag) + + def normalize_dl_config(dl_config: dict[str, Any]) -> dict[str, Any]: """Map aliased config keys (e.g. ``range``) to their Click parameter names (``range_``).""" return {DL_OPTION_ALIASES.get(key, key): value for key, value in dl_config.items()} +ID_PROVIDER_KEYS = {"tmdb": "tmdb_api_key", "tvdb": "tvdb_api_key", "omdb": "omdb_api_key"} + + +def validate_metadata_ids( + tmdb_id: Optional[int], + imdb_id: Optional[str], + tvdb_id: Optional[int], + anilist_id: Optional[Union[int, str]] = None, +) -> None: + """Reject ID flags that conflict, or that no configured provider could ever resolve. + + --anilist stays outside the mutual exclusion: AniList knows no western IDs, so pairing + it with one of the other three is how an anime title gets both. + """ + if anilist_id and not parse_anilist_ref(anilist_id): + raise click.UsageError("--anilist takes an AniList ID like 21, or a MyAnimeList ID like mal:21.") + + supplied = [ + (flag, value) for flag, value in (("--tmdb", tmdb_id), ("--imdb", imdb_id), ("--tvdb", tvdb_id)) if value + ] + if len(supplied) > 1: + raise click.UsageError( + f"{', '.join(flag for flag, _ in supplied)} cannot be used together. " + "Give one ID and unshackle resolves the others from it." + ) + if not supplied: + return + + flag, _ = supplied[0] + id_kind = flag.lstrip("-") + ordered = {cls.NAME: cls for kind in ("tv", "movie") for cls in providers.provider_order(kind)} + consumers = [cls for cls in ordered.values() if cls.ID_KIND == id_kind] + if any(cls().is_available() for cls in consumers): + return + + if consumers: + keys = sorted({ID_PROVIDER_KEYS[cls.NAME] for cls in consumers if cls.NAME in ID_PROVIDER_KEYS}) + raise click.UsageError( + f"{flag} needs the {consumers[0].NAME} provider. Set {' or '.join(keys)} in your config." + ) + names = sorted({cls.NAME for cls in providers.REGISTRY.values() if cls.ID_KIND == id_kind}) + raise click.UsageError( + f"{flag} needs the {' or '.join(names)} provider, which your metadata_providers config leaves out." + ) + + def group_videos_by_variant(videos: list[Video], *, merge: bool) -> list[list[Video]]: """Group video tracks for muxing. @@ -524,7 +584,7 @@ class dl: "--lang", type=LANGUAGE_RANGE, default="orig", - help="Language(s) wanted for Video and Audio (comma-separated). Use 'orig' to select the original language, e.g. 'orig,en' for both original and English.", + help="Language(s) wanted for Video and Audio (comma-separated). Use 'orig' to select the original language, e.g. 'orig,en' for both original and English. Prefix a value with '-' to exclude it, e.g. 'all,-es'.", ) @click.option( "--latest-episode", @@ -537,16 +597,22 @@ class dl: "--v-lang", type=LANGUAGE_RANGE, default=[], - help="Language wanted for Video. You would use this if the video language doesn't match the audio.", + help="Language wanted for Video. You would use this if the video language doesn't match the audio. Prefix a value with '-' to exclude it, e.g. 'all,-es'.", ) @click.option( "-al", "--a-lang", type=LANGUAGE_RANGE, default=[], - help="Language wanted for Audio, overrides -l/--lang for audio tracks.", + help="Language wanted for Audio, overrides -l/--lang for audio tracks. Prefix a value with '-' to exclude it, e.g. 'all,-es'.", + ) + @click.option( + "-sl", + "--s-lang", + type=LANGUAGE_RANGE, + default=["all"], + help="Language wanted for Subtitles. Prefix a value with '-' to exclude it, e.g. 'all,-es'.", ) - @click.option("-sl", "--s-lang", type=LANGUAGE_RANGE, default=["all"], help="Language wanted for Subtitles.") @click.option( "--require-subs", type=LANGUAGE_RANGE, @@ -559,7 +625,7 @@ class dl: "--forced-s-lang", type=LANGUAGE_RANGE, default=[], - help="Languages wanted for forced subtitles, implies -fs. Keeps forced subs only in these languages.", + help="Languages wanted for forced subtitles, implies -fs. Keeps forced subs only in these languages. Prefix a value with '-' to exclude it, e.g. 'all,-es'.", ) @click.option( "--exact-lang", @@ -601,14 +667,16 @@ class dl: "tmdb_id", type=int, default=None, - help="Use this TMDB ID for tagging instead of automatic lookup.", + help="Use this TMDB ID for tagging instead of an automatic search. Add --enrich to also take " + "its title, year and original language.", ) @click.option( "--tvdb", "tvdb_id", type=int, default=None, - help="Use this TVDB ID for --tvdb-order instead of looking the series up automatically.", + help="Use this TVDB ID for --tvdb-order and tagging instead of looking the series up " + "automatically. Add --enrich to also take its title, year and original language.", ) @click.option( "--tvdb-order", @@ -618,24 +686,36 @@ class dl: "a series' Streaming Order is 'alternate'. Defaults to the tvdb_order config option.", ) @click.option( - "--animeapi", - "animeapi_id", + "--anilist", + "anilist_id", type=str, default=None, - help="Anime database ID via AnimeAPI (e.g. mal:12345, anilist:98765). Defaults to MAL if no prefix.", + help="Use this AniList ID for tagging instead of an automatic search, or a MyAnimeList ID as mal:12345. " + "Add --enrich to also take its title, year and original language. Combines with one of " + "--tmdb, --imdb or --tvdb, which AniList does not know.", ) @click.option( "--enrich", is_flag=True, default=False, - help="Override show title and year from external source. Requires --tmdb, --imdb, or --animeapi.", + help=( + "Overwrite the show title, year and original language with the external source's. " + "Requires --tmdb, --imdb, --tvdb, or --anilist." + ), + ) + @click.option( + "--daily", + is_flag=True, + default=False, + help="Treat the title as daily/date-based content and fill missing air dates from TVDB during --enrich.", ) @click.option( "--imdb", "imdb_id", type=str, default=None, - help="Use this IMDB ID (e.g. tt1375666) for tagging instead of automatic lookup.", + help="Use this IMDB ID (e.g. tt1375666) for tagging instead of an automatic search. Add --enrich " + "to also take its title, year and original language.", ) @click.option( "--sub-format", @@ -810,8 +890,9 @@ class dl: imdb_id: Optional[str] = None, tvdb_id: Optional[int] = None, tvdb_order: Optional[str] = None, - animeapi_id: Optional[str] = None, + anilist_id: Optional[Union[int, str]] = None, enrich: bool = False, + daily: bool = False, output_dir: Optional[Path] = None, *_: Any, **__: Any, @@ -850,36 +931,34 @@ class dl: imdb_id = ctx.params.get("imdb_id", imdb_id) tvdb_id = ctx.params.get("tvdb_id", tvdb_id) tvdb_order = ctx.params.get("tvdb_order", tvdb_order) - animeapi_id = ctx.params.get("animeapi_id", animeapi_id) + anilist_id = ctx.params.get("anilist_id", anilist_id) enrich = ctx.params.get("enrich", enrich) + daily = ctx.params.get("daily", daily) output_dir = ctx.params.get("output_dir", output_dir) self.profile = profile self.proxy_requested = bool(proxy) + validate_metadata_ids(tmdb_id, imdb_id, tvdb_id, anilist_id) self.tmdb_id = tmdb_id self.imdb_id = imdb_id self.tvdb_id = tvdb_id + self.anilist_id = anilist_id self.tvdb_order = (tvdb_order or config.tvdb_order or "").lower() if self.tvdb_order and self.tvdb_order not in SEASON_TYPES: raise click.UsageError(f"tvdb_order must be one of {', '.join(SEASON_TYPES)}, not {self.tvdb_order!r}") self.enrich = enrich - self.animeapi_title: Optional[str] = None + self.daily = daily self.output_dir = output_dir + self.service_anime = False + self.service_daily = False - if animeapi_id: - from unshackle.core.utils.animeapi import resolve_animeapi + if self.enrich and not (self.tmdb_id or self.imdb_id or self.tvdb_id or self.anilist_id): + raise click.UsageError( + "--enrich requires --tmdb, --imdb, --tvdb, or --anilist to provide a metadata source." + ) - anime_title, anime_ids = resolve_animeapi(animeapi_id) - self.animeapi_title = anime_title - if not self.tmdb_id and anime_ids.tmdb_id: - self.tmdb_id = anime_ids.tmdb_id - if not self.imdb_id and anime_ids.imdb_id: - self.imdb_id = anime_ids.imdb_id - if not self.tvdb_id and anime_ids.tvdb_id: - self.tvdb_id = anime_ids.tvdb_id - - if self.enrich and not (self.tmdb_id or self.imdb_id or self.animeapi_title): - raise click.UsageError("--enrich requires --tmdb, --imdb, or --animeapi to provide a metadata source.") + if self.daily and not self.enrich: + self.log.warning("--daily fills air dates from TVDB only with --enrich. A service can set them itself.") # Initialize debug logger with service name if debug logging is enabled if config.debug or logging.root.level == logging.DEBUG: @@ -904,7 +983,7 @@ class dl: "tag": tag, "tmdb_id": self.tmdb_id, "imdb_id": self.imdb_id, - "animeapi_id": animeapi_id, + "anilist_id": anilist_id, "enrich": enrich, "cli_params": { k: v @@ -916,7 +995,7 @@ class dl: "tag", "tmdb_id", "imdb_id", - "animeapi_id", + "anilist_id", "enrich", ] }, @@ -1343,18 +1422,57 @@ class dl: ) -> None: self.tmdb_searched = False self.search_source = None + self.service_anime = bool(getattr(service, "ANIME", False)) + self.service_daily = bool(getattr(service, "DAILY", False)) self.server_cdm = getattr(service, "_server_cdm", False) - self._remote_service = service if self.server_cdm else None + self._remote_service = service if hasattr(service, "_server_cdm") else None start_time = time.time() - if forced_s_lang: + lang, lang_excl = partition_exclusions(lang) + v_lang, v_lang_excl = partition_exclusions(v_lang) + a_lang, a_lang_excl = partition_exclusions(a_lang) + s_lang, s_lang_excl = partition_exclusions(s_lang) + forced_s_lang, fsl_excl = partition_exclusions(forced_s_lang) + for flag, excludes in ( + ("-l/--lang", lang_excl), + ("-vl/--v-lang", v_lang_excl), + ("-al/--a-lang", a_lang_excl), + ("-sl/--s-lang", s_lang_excl), + ("-fsl/--forced-s-lang", fsl_excl), + ): + if any(token.lower() == "all" for token in excludes): + self.log.error(f"{flag}: 'all' cannot be excluded. Name the languages to exclude instead.") + sys.exit(1) + # exclusion-only use falls back to the flag's own default; -vl/-al have none, they cascade to -l + if lang_excl and not lang: + lang = ["orig"] + if s_lang_excl and not s_lang: + s_lang = ["all"] + # an override that names languages replaces -l entirely, so it must not inherit its exclusions + video_excl = list(dict.fromkeys(v_lang_excl + ([] if v_lang else lang_excl))) + audio_excl = list(dict.fromkeys(a_lang_excl + ([] if a_lang else lang_excl))) + + ctx = getattr(service, "ctx", None) + parent_params = ctx.parent.params if ctx and ctx.parent else None + if parent_params is not None: + # services read the raw ctx params, they must never see a '-xx' token + for name, includes, excludes in ( + ("lang", lang, lang_excl), + ("v_lang", v_lang, v_lang_excl), + ("a_lang", a_lang, a_lang_excl), + ("s_lang", s_lang, s_lang_excl), + ("forced_s_lang", forced_s_lang, fsl_excl), + ): + if excludes and name in parent_params: + parent_params[name] = includes + + if forced_s_lang or fsl_excl: # services read forced_subs from raw ctx params, so the implication must be pushed there too forced_subs = True if "all" in forced_s_lang: forced_s_lang = [] - ctx = getattr(service, "ctx", None) - if ctx and ctx.parent: - ctx.parent.params["forced_subs"] = True + if parent_params is not None: + parent_params["forced_subs"] = True if skip_dl: DOWNLOAD_LICENCE_ONLY.set() @@ -1533,45 +1651,53 @@ class dl: enrich_title: Optional[str] = None enrich_year: Optional[int] = None + enrich_lang: Optional[Language] = None - if self.animeapi_title: - enrich_title = self.animeapi_title + enrich_result = providers.resolve_by_ids( + self.tmdb_id, + self.imdb_id, + self.tvdb_id, + self.anilist_id, + kind=kind, + title_cacher=title_cacher, + cache_title_id=cache_title_id, + cache_region=cache_region, + cache_account_hash=cache_account_hash, + anime=self.anime_hint(sample_title), + ) + if enrich_result: + enrich_title = enrich_result.title + enrich_year = enrich_result.year + enrich_lang = parse_language(enrich_result.original_language) - if self.tmdb_id: - if not enrich_title: - enrich_title = providers.get_title_by_id( - self.tmdb_id, kind, title_cacher, cache_title_id, cache_region, cache_account_hash - ) - enrich_year = providers.get_year_by_id( - self.tmdb_id, kind, title_cacher, cache_title_id, cache_region, cache_account_hash - ) - elif self.imdb_id: - imdbapi = providers.get_provider("imdbapi") - if imdbapi: - imdb_result = imdbapi.get_by_id(self.imdb_id, kind) - if imdb_result: - if not enrich_title: - enrich_title = imdb_result.title - enrich_year = imdb_result.year + if enrich_lang: + self.log.info(f"Original language from metadata: {enrich_lang}") + if not (enrich_title or enrich_year or enrich_lang): + self.log.warning("--enrich found no metadata. Is the provider's API key configured?") + elif missing := [ + name + for name, value in (("title", enrich_title), ("year", enrich_year), ("language", enrich_lang)) + if not value + ]: + self.log.warning(f"--enrich source gave no {', '.join(missing)}. Those fields are unchanged.") - if enrich_title or enrich_year: - if isinstance(titles, (Series, Movies)): - for t in titles: - if enrich_title: - if isinstance(t, Episode): - t.title = enrich_title - else: - t.name = enrich_title - if enrich_year and not t.year: - t.year = enrich_year - else: + if enrich_title or enrich_year or enrich_lang: + for t in titles if isinstance(titles, (Series, Movies)) else [titles]: if enrich_title: - if isinstance(titles, Episode): - titles.title = enrich_title + if isinstance(t, Episode): + t.title = enrich_title else: - titles.name = enrich_title - if enrich_year and not titles.year: - titles.year = enrich_year + t.name = enrich_title + if enrich_year: + t.year = enrich_year + if enrich_lang: + t.language = enrich_lang + + if isinstance(titles, Series): + enrich_tvdb_id = self.tvdb_id or (enrich_result.external_ids.tvdb_id if enrich_result else None) + self.fill_absolute_numbers(titles, enrich_tvdb_id) + if any(self.daily_hint(t) for t in titles): + self.fill_air_dates(titles, enrich_tvdb_id) if self.tvdb_order and isinstance(titles, Series): titles = self.apply_tvdb_order(titles, title_cacher, cache_title_id, cache_region, cache_account_hash) @@ -1661,7 +1787,7 @@ class dl: # Note: Headers are not mapped to actual title indices # Format display name - display_name = ((t.name[:30].rstrip() + "…") if len(t.name) > 30 else t.name) if t.name else None + display_name = ((t.name[:30].rstrip() + "...") if len(t.name) > 30 else t.name) if t.name else None # Apply indentation only for multiple seasons prefix = " " if multiple_seasons else "" @@ -1842,6 +1968,18 @@ class dl: ) sys.exit(1) + song_excl = [ + str(song.language) if t == "orig" else t for t in audio_excl if t != "orig" or song.language + ] + if song_excl: + drop = excluded_language_tags(song_excl, [t.language for t in song.tracks.audio], exact_lang) + song.tracks.select_audio(lambda x: str(x.language) not in drop) + if not song.tracks.audio: + self.log.error( + f"Every Audio Track for {song.name} was excluded by -{', -'.join(song_excl)}..." + ) + sys.exit(1) + audio_languages = a_lang or lang if audio_languages: processed_lang = [] @@ -1945,10 +2083,10 @@ class dl: _progress.update(task_id=task_id, **kwargs) track_table = Table.grid() - track_table.add_row(music_renderer._song_line(song, titles)) + track_table.add_row(music_renderer.song_line(song, titles)) song_plan = music_song_plans.get(id(song)) if song_plan and song_plan.selected: - track_table.add_row(music_renderer._option_line(song_plan.selected), style="text2") + track_table.add_row(music_renderer.option_line(song_plan.selected), style="text2") else: track_table.add_row(str(track)[6:], style="text2") track_table.add_row(progress) @@ -2235,7 +2373,10 @@ class dl: if music_collection_mode: raise click.ClickException("Music collections require grouped audio downloads.") + base_selection = (v_lang, a_lang, s_lang, range_) + for i, title in enumerate(titles): + v_lang, a_lang, s_lang, range_ = base_selection if isinstance(title, Episode) and latest_episode and latest_episode_id: # If --latest-episode is set, only process the latest episode if f"{title.season}x{title.number}" != latest_episode_id: @@ -2264,20 +2405,30 @@ class dl: if isinstance(title, Episode) and not self.tmdb_searched: kind = "tv" tmdb_title: Optional[str] = None - if self.tmdb_id: - tmdb_title = providers.get_title_by_id( - self.tmdb_id, kind, title_cacher, cache_title_id, cache_region, cache_account_hash - ) - else: - result = providers.search_metadata( - title.title, title.year, kind, title_cacher, cache_title_id, cache_region, cache_account_hash - ) - if result and result.title and providers.fuzzy_match(result.title, title.title): + has_ids = bool(self.tmdb_id or self.imdb_id or self.tvdb_id or self.anilist_id) + result = providers.resolve_by_ids( + self.tmdb_id, + self.imdb_id, + self.tvdb_id, + self.anilist_id, + title=title.title, + year=title.year, + kind=kind, + title_cacher=title_cacher, + cache_title_id=cache_title_id, + cache_region=cache_region, + cache_account_hash=cache_account_hash, + anime=self.anime_hint(title), + ) + if result: + if has_ids: + tmdb_title = result.title + if not self.tmdb_id: + self.tmdb_id = result.external_ids.tmdb_id + elif result.title and providers.fuzzy_match(result.title, title.title): self.tmdb_id = result.external_ids.tmdb_id tmdb_title = result.title self.search_source = result.source - else: - self.tmdb_id = None if list_ or list_titles: if self.tmdb_id: console.print( @@ -2291,8 +2442,19 @@ class dl: self.tmdb_searched = True if isinstance(title, Movie) and (list_ or list_titles) and not self.tmdb_id: - movie_result = providers.search_metadata( - title.name, title.year, "movie", title_cacher, cache_title_id, cache_region, cache_account_hash + movie_result = providers.resolve_by_ids( + None, + self.imdb_id, + self.tvdb_id, + self.anilist_id, + title=title.name, + year=title.year, + kind="movie", + title_cacher=title_cacher, + cache_title_id=cache_title_id, + cache_region=cache_region, + cache_account_hash=cache_account_hash, + anime=self.anime_hint(title), ) if movie_result and movie_result.external_ids.tmdb_id: console.print( @@ -2305,7 +2467,7 @@ class dl: else: console.print(Padding("Search -> [bright_black]No match found[/]", (0, 5))) - if self.tmdb_id and getattr(self, "search_source", None) not in ("simkl", "imdbapi"): + if self.tmdb_id and getattr(self, "search_source", None) not in ("simkl", "imdb"): kind = "tv" if isinstance(title, Episode) else "movie" providers.fetch_external_ids( self.tmdb_id, kind, title_cacher, cache_title_id, cache_region, cache_account_hash @@ -2435,7 +2597,7 @@ class dl: exact_match=exact_lang, ) title.tracks.sort_subtitles( - by_language=resolve_sort_langs([*subtitle_priority, *s_lang], title.language), + by_language=resolve_sort_langs([*subtitle_priority, *(s_lang or [])], title.language), type_priority=config.subtitle.get("type_priority"), group_by=config.subtitle.get("group_by"), exact_match=exact_lang, @@ -2443,7 +2605,12 @@ class dl: if list_: available_tracks, _ = title.tracks.tree() - console.print(Padding(Panel(available_tracks, title="Available Tracks"), (0, 5))) + console.print( + Padding( + Panel(available_tracks, title="Available Tracks", box=box.SQUARE, border_style="bright_black"), + (0, 5), + ) + ) continue # Determine which tracks to keep @@ -2495,6 +2662,23 @@ class dl: title.tracks.chapters = [] with console.status("Selecting tracks...", spinner="dots"): + + def resolve_excludes(excludes: list[str], _title: Title_T = title) -> list[str]: + """Resolve 'orig' against this title. An unresolvable 'orig' excludes nothing.""" + resolved: list[str] = [] + for token in excludes: + value = str(_title.language) if token == "orig" else token + if token == "orig" and not _title.language: + continue + if value not in resolved: + resolved.append(value) + return resolved + + video_excl_r = resolve_excludes(video_excl) + audio_excl_r = resolve_excludes(audio_excl) + s_excl_r = resolve_excludes(s_lang_excl) + fsl_excl_r = resolve_excludes(fsl_excl) + if isinstance(title, (Movie, Episode)): # filter video tracks if keep_videos and vcodec: @@ -2541,6 +2725,15 @@ class dl: self.log.error(f"No Video Track in {vbitrate_min}-{vbitrate_max}kbps range...") sys.exit(1) + if keep_videos and video_excl_r: + drop = excluded_language_tags( + video_excl_r, [t.language for t in title.tracks.videos], exact_lang + ) + title.tracks.select_video(lambda x: str(x.language) not in drop) + if not title.tracks.videos: + self.log.error(f"Every Video Track was excluded by -{', -'.join(video_excl_r)}...") + sys.exit(1) + effective_video_lang = v_lang or lang video_languages = [lang for lang in effective_video_lang if lang != "best"] video_multi_lang = ( @@ -2821,6 +3014,12 @@ class dl: fsl = [t for t in forced_s_lang if t != "orig"] if "orig" in forced_s_lang and title.language: fsl.append(str(title.language)) + if keep_subtitles and s_excl_r: + # applies to forced subs too: an excluded language is unwanted in any form + drop = excluded_language_tags( + s_excl_r, [t.language for t in title.tracks.subtitles], exact_lang + ) + title.tracks.select_subtitles(lambda x: str(x.language) not in drop) if keep_subtitles and require_subs: missing_langs = [ lang @@ -2888,6 +3087,11 @@ class dl: ) elif not forced_subs: title.tracks.select_subtitles(lambda x: not x.forced) + if fsl_excl_r: + drop = excluded_language_tags( + fsl_excl_r, [t.language for t in title.tracks.subtitles if t.forced], exact_lang + ) + title.tracks.select_subtitles(lambda x: not (x.forced and str(x.language) in drop)) # filter audio tracks # might have no audio tracks if part of the video, e.g. transport stream hls @@ -2924,6 +3128,14 @@ class dl: if not title.tracks.audio: self.log.error(f"No Audio Track in {abitrate_min}-{abitrate_max}kbps range...") sys.exit(1) + if audio_excl_r: + drop = excluded_language_tags( + audio_excl_r, [t.language for t in title.tracks.audio], exact_lang + ) + title.tracks.select_audio(lambda x: str(x.language) not in drop) + if not title.tracks.audio: + self.log.error(f"Every Audio Track was excluded by -{', -'.join(audio_excl_r)}...") + sys.exit(1) audio_languages = a_lang or lang if audio_languages: processed_lang = [] @@ -2945,10 +3157,12 @@ class dl: if a_orig_token in audio_languages: a_orig_token = None + embedded_langs = embedded_audio_langs(title.tracks.videos, keep_videos) + if not any(tok in processed_lang for tok in ("best", "all")): missing_a_langs = find_missing_langs( processed_lang, - [a.language for a in title.tracks.audio], + [a.language for a in title.tracks.audio] + embedded_langs, exact=exact_lang, ) if missing_a_langs: @@ -2973,8 +3187,9 @@ class dl: title.tracks.audio = select_best_audio( title.tracks.audio, processed_lang, acodec, audio_description, exact_lang ) - if not title.tracks.audio: - # empty only when 'orig' was the sole request and did not resolve + if not title.tracks.audio and not embedded_langs: + # empty when 'orig' was the sole request, did not resolve, and the + # video carries no audio of its own to fall back on self.log.error( f"There's no {as_requested(processed_lang, a_orig_token) or 'orig'} " f"Audio Track, cannot continue..." @@ -3715,7 +3930,15 @@ class dl: shutil.move(muxed_path, final_path) used_final_paths.add(final_path) self.completed_files.append(final_path) - tags.tag_file(final_path, title, self.tmdb_id, self.imdb_id, self.tvdb_id) + tags.tag_file( + final_path, + title, + self.tmdb_id, + self.imdb_id, + self.tvdb_id, + self.anilist_id, + self.anime_hint(title), + ) title_dl_time = time_elapsed_since(dl_start_time) downloaded_label = "Track" if music_mode and isinstance(title, Song) else "Title" @@ -3741,6 +3964,112 @@ class dl: console.print(Padding(f"Processed all titles in [progress.elapsed]{dl_time}", (0, 5, 1, 5))) + def anime_hint(self, title: Optional[Title_T] = None) -> bool: + """Whether metadata lookups for this title should prefer AniList.""" + per_title = getattr(title, "anime", None) + return self.service_anime if per_title is None else bool(per_title) + + def daily_hint(self, title: Optional[Title_T] = None) -> bool: + """Whether this title is daily/date-based content named by air date.""" + per_title = getattr(title, "daily", None) + return self.daily or (self.service_daily if per_title is None else bool(per_title)) + + def fill_air_dates(self, titles: Series, tvdb_id: Optional[int] = None) -> None: + """Fill in missing episode air dates from TVDB. + + Additive only: an air date the service already set is kept. + """ + tvdb = providers.get_provider("tvdb") + if not tvdb: + self.log.debug("Air dates need a tvdb_api_key in your config, skipping.") + return + + if not tvdb_id: + self.log.debug("Could not resolve a TVDB ID, skipping air dates.") + return + + wanted = [t for t in titles if self.daily_hint(t)] + if not wanted: + return + + keys = list(dict.fromkeys((t.season, t.number) for t in titles)) + source_order = tvdb.detect_order(tvdb_id, keys) # type: ignore[attr-defined] + episodes = tvdb.get_episodes(tvdb_id, source_order) # type: ignore[attr-defined] + if not episodes: + self.log.debug("TVDB listed no episodes for series %s, skipping air dates.", tvdb_id) + return + + aired_by_key: dict[tuple[int, int], str] = {} + for episode in episodes: + season, number = parse_int(episode.get("seasonNumber")), parse_int(episode.get("number")) + aired = episode.get("aired") + if season is None or number is None or not aired: + continue + aired_by_key[(season, number)] = str(aired) + + # TVDB carries placeholder schedule dates for unaired episodes. Stamping one would + # name the file with a date the episode never aired on. + limit = date.today() + timedelta(days=1) + + filled = 0 + for title in wanted: + if title.air_date is not None: + continue + aired = aired_by_key.get((title.season, title.number)) + if not aired: + continue + try: + air_date = date.fromisoformat(aired[:10]) + except ValueError: + continue + if air_date.year < 1970 or air_date > limit: + continue + title.air_date = air_date + filled += 1 + + if filled: + self.log.info("Filled air dates for %d episode(s) from TVDB (ID %s).", filled, tvdb_id) + + def fill_absolute_numbers(self, titles: Series, tvdb_id: Optional[int] = None) -> None: + """Fill in missing absolute episode numbers from TVDB's absolute order. + + Additive only: season and number are never written, and an absolute number the + service already set is kept. + """ + tvdb = providers.get_provider("tvdb") + if not tvdb: + self.log.debug("Absolute episode numbers need a tvdb_api_key in your config, skipping.") + return + + if not tvdb_id: + self.log.debug("Could not resolve a TVDB ID, skipping absolute episode numbers.") + return + + keys = list(dict.fromkeys((t.season, t.number) for t in titles)) + source_order = tvdb.detect_order(tvdb_id, keys) # type: ignore[attr-defined] + if source_order == "absolute": + # the service already numbers in absolute order; get_order_map would return the + # empty same-order map, but here the episode numbers are their own fill + mapping = {key: (*key, None) for key in keys} + else: + mapping = tvdb.get_order_map(tvdb_id, "absolute", source_order) # type: ignore[attr-defined] + if not mapping: + self.log.debug("TVDB has no absolute order for series %s, skipping absolute episode numbers.", tvdb_id) + return + + filled = 0 + for title in titles: + if title.absolute is not None: + continue + entry = mapping.get((title.season, title.number)) + if not entry: + continue + title.absolute = entry[1] + filled += 1 + + if filled: + self.log.info("Filled absolute numbers for %d episode(s) from TVDB (ID %s).", filled, tvdb_id) + def apply_tvdb_order( self, titles: Series, @@ -3758,8 +4087,17 @@ class dl: tvdb_id = self.tvdb_id if not tvdb_id: show = titles[0] - result = providers.search_metadata( - show.title, show.year, "tv", title_cacher, cache_title_id, cache_region, cache_account_hash + result = providers.resolve_by_ids( + self.tmdb_id, + self.imdb_id, + None, + title=show.title, + year=show.year, + kind="tv", + title_cacher=title_cacher, + cache_title_id=cache_title_id, + cache_region=cache_region, + cache_account_hash=cache_account_hash, ) tvdb_id = result.external_ids.tvdb_id if result else None if not tvdb_id: @@ -3837,12 +4175,19 @@ class dl: year=title.year, air_date=title.air_date.isoformat() if isinstance(title.air_date, date) else title.air_date, ) + # conditional, so meta for a title without them is unchanged if title.part is not None: meta["part"] = title.part + if title.absolute is not None: + meta["absolute"] = title.absolute + if getattr(title, "daily", None) is not None: + meta["daily"] = title.daily elif isinstance(title, Movie): meta.update(type="movie", name=title.name, year=title.year) else: meta.update(type="movie", name=str(title)) + if getattr(title, "anime", None) is not None: + meta["anime"] = title.anime return meta def write_export(self, export: Path, title: Title_T, track: AnyTrack, drm: Any = None) -> None: @@ -3851,7 +4196,7 @@ class dl: Carries no session/cookies/dl-flags. Region (country code) is stored only when the export used ``--proxy``, as an import geofence. Each track records only the licensed DRM system; content keys live once under the track's ``keys``. ``drm`` may be None - (DRM-free track) or a DRM system without ``to_dict``/``content_keys`` (e.g. ClearKey) — + (DRM-free track) or a DRM system without ``to_dict``/``content_keys`` (e.g. ClearKey) - the track, manifest, chapter and attachment info is still exported. """ with self.EXPORT_LOCK: @@ -3941,7 +4286,8 @@ class dl: if not drm: return - server_cdm = getattr(self, "server_cdm", False) + svc_for_cdm = getattr(self, "_remote_service", None) + server_cdm = getattr(svc_for_cdm, "_server_cdm", getattr(self, "server_cdm", False)) if server_cdm: if not drm.content_keys: diff --git a/unshackle/commands/env.py b/unshackle/commands/env.py index f4060677..f3181dd2 100644 --- a/unshackle/commands/env.py +++ b/unshackle/commands/env.py @@ -46,13 +46,6 @@ def get_dependencies() -> list[dict]: "desc": "DRM decryption", "cat": "DRM", }, - { - "name": "ML-Worker", - "binary": binaries.ML_Worker, - "required": False, - "desc": "DRM licensing", - "cat": "DRM", - }, # HDR Processing {"name": "dovi_tool", "binary": binaries.DoviTool, "required": False, "desc": "Dolby Vision", "cat": "HDR"}, { @@ -195,6 +188,111 @@ def check() -> None: console.print(Padding(" ".join(summary_parts), (1, 2))) +@env.command() +def theme() -> None: + """Preview the available CLI themes.""" + import math + + from rich import box + from rich.color import Color, blend_rgb + from rich.console import Group + from rich.panel import Panel + from rich.rule import Rule + from rich.style import Style + from rich.text import Text + + from unshackle.core.themes import ALIASES, DEFAULT_THEME, PALETTES, resolve_palette + + active = resolve_palette(config.theme) or PALETTES[DEFAULT_THEME] + for name, palette in PALETTES.items(): + text, text2, gray, guide = palette["text"], palette["text2"], palette["gray"], palette["bright_black"] + + header = Text(name, style=palette["pink"]) + if palette == active: + header.append(" (active)", style=palette["green"]) + aliases = sorted(alias for alias, target in ALIASES.items() if target == name) + if aliases: + header.append(f" aliases: {', '.join(aliases)}", style=f"dim {gray}") + + swatch = Text() + for role in ("red", "green", "yellow", "blue", "pink", "cyan", "text", "gray"): + swatch.append("██ ", style=palette[role]) + + docstring = Text() + docstring.append("Service docstrings render like this, ", style=text) + docstring.append("with secondary detail lines like this.", style=text2) + + separator = Text("─" * 60, style=palette["dark_gray"]) + options: list[Text] = [] + rows = ( + ("-q, ", "--quality ", "TEXT ", "Video quality to download", " [default: best]", f"dim {gray}"), + ( + "-r, ", + "--range ", + "[SDR|HDR10|DV] ", + "Video color range", + " [env: RANGE]", + f"dim {palette['yellow']}", + ), + ) + for short, long_opt, metavar, help_text, extra, extra_style in rows: + row = Text() + row.append(short, style=palette["green"]) + row.append(long_opt, style=text) + row.append(metavar, style=palette["yellow"]) + row.append(f" {help_text}", style=text) + row.append(extra, style=extra_style) + options.extend((row, separator)) + + listing = Tree(Text("1 season, S1(1)", style=text), guide_style=guide) + episode = listing.add(Text.assemble(("1. ", palette["blue"]), ("Pilot", text)), guide_style=guide) + track_rows = ( + ("VID", "H.264 | 1920x1080 @ 4523 kb/s"), + ("AUD", "AAC 2.0 | en @ 192 kb/s"), + ("SUB", "WebVTT | en (SDH)"), + ) + for kind, desc in track_rows: + episode.add(Text.assemble((kind, palette["pink"]), (" | ", guide), (desc, text)), guide_style=guide) + tracks = Panel(listing, title="Available Tracks", box=box.SQUARE, border_style=guide, expand=False) + + logs = [] + log_rows = ( + ("INFO", palette["green"], "Downloading 3 tracks"), + ("WARNING", palette["yellow"], "Subtitle cues overlapped, fixed 2"), + ("ERROR", palette["red"], "License request denied (403)"), + ) + for level, level_style, message in log_rows: + line = Text() + line.append("18:00:01 ", style=gray) + line.append(f"{level:<8} ", style=level_style) + line.append(message, style=text if level != "ERROR" else palette["red"]) + logs.append(line) + + progress = Text() + progress.append("Pilot ", style=text) + progress.append("━" * 24, style=palette["green"]) + progress.append("╸", style=palette["green"]) + progress.append("━" * 14, style=guide) + progress.append(" 62%", style=palette["yellow"]) + progress.append(" 12.4 MB/s", style=palette["cyan"]) + progress.append(" 0:00:42", style=text2) + + # one frozen frame of GradientPulseBarColumn's dark_gray-to-pink cosine sweep + lo = Color.parse(palette["dark_gray"]).triplet + hi = Color.parse(palette["pink"]).triplet + assert lo and hi + lut = [Style(color=Color.from_triplet(blend_rgb(lo, hi, i / 31))) for i in range(32)] + pulse = Text("Muxing ", style=text) + for i in range(39): + fade = (math.cos((i - 8) * math.tau / 40.0) + 1) / 2 + pulse.append("━", style=lut[int(fade * 31)]) + + block = Group(header, swatch, Text(), docstring, *options, tracks, *logs, Text(), progress, pulse) + console.print(Padding(block, (1, 2, 1, 2))) + console.print(Rule(style=guide, characters="─")) + console.print() + + @env.command() def info() -> None: """Displays information about the current environment.""" diff --git a/unshackle/commands/kv.py b/unshackle/commands/kv.py index a4e52e0c..eda7c55b 100644 --- a/unshackle/commands/kv.py +++ b/unshackle/commands/kv.py @@ -1,5 +1,6 @@ import logging import re +from concurrent.futures import ThreadPoolExecutor, as_completed from pathlib import Path from typing import Optional @@ -243,6 +244,36 @@ def add(file: Path, service: str, vaults: list[str]) -> None: log.info("Done!") +def search_vault(vault: Vault, kid: str, services: list[str], log: logging.Logger) -> Optional[tuple[str, str]]: + """Return the (service, key) of the first service table in a vault holding the KID.""" + + def probe(svc: str) -> Optional[tuple[str, str]]: + try: + key = vault.get_key(kid, svc) + except Exception as e: + log.debug(f"{vault} [{svc}]: lookup error ({e})") + return None + if key and key.count("0") != len(key): + return svc, key + return None + + if len(services) == 1: + return probe(services[0]) + + with ThreadPoolExecutor(max_workers=8) as pool: + futures = [pool.submit(probe, svc) for svc in services] + try: + for future in as_completed(futures): + found = future.result() + if found: + return found + finally: + for future in futures: + future.cancel() + + return None + + @kv.command() @click.argument("kid", type=str) @click.option("-s", "--service", type=str, default=None, help="Limit search to a specific service tag.") @@ -254,8 +285,10 @@ def search(kid: str, service: Optional[str], vault_name: Optional[str]) -> None: Search configured Key Vault(s) for a KID and report any matching KEY. KID must be 32 hex characters (no dashes). If --service is omitted, every - service table in each vault is scanned. If --vault is omitted, every - vault in the config is searched. + service table in each vault is scanned; vaults that cannot list their + tables are probed with every locally installed service tag instead, so + passing --service is much faster. If --vault is omitted, every vault in + the config is searched. """ log = logging.getLogger("kv") @@ -274,8 +307,9 @@ def search(kid: str, service: Optional[str], vault_name: Optional[str]) -> None: service_tag = Services.get_tag(service) if service else None - hit: Optional[tuple[str, str, str]] = None + hits: list[tuple[str, str, str]] = [] for vault in vaults_: + probed = False if service_tag: services_to_check: list[str] = [service_tag] else: @@ -285,28 +319,25 @@ def search(kid: str, service: Optional[str], vault_name: Optional[str]) -> None: log.debug(f"{vault}: get_services() failed ({e})") services_to_check = [] if not services_to_check: - log.warning(f"{vault}: cannot search without a service (remote vault requires --service). Skipping.") - continue + services_to_check = list(Services.get_tags()) + probed = True + log.debug(f"{vault}: cannot list tables, probing {len(services_to_check)} local service tag(s)") - for svc in services_to_check: - try: - key = vault.get_key(kid_norm, svc) - except Exception as e: - log.debug(f"{vault} [{svc}]: lookup error ({e})") - continue - if key and key.count("0") != len(key): - hit = (vault.name, svc, key) - break - if hit: - break + found = search_vault(vault, kid_norm, services_to_check, log) + if found: + svc_hit, key_hit = found + hits.append((vault.name, f"{svc_hit}?" if probed else svc_hit, key_hit)) - if hit: - vname, svc, key = hit - tree = Tree(Text.assemble((svc, "cyan"), (f"({vname})", "text"), overflow="fold")) - tree.add(f"[text2]{kid_norm}:{key}") - console.print(Padding(tree, (1, 5))) - else: + if not hits: log.info(f"KID {kid_norm} not found in {len(vaults_)} vault(s).") + return + + for svc in dict.fromkeys(svc for _, svc, _ in hits): + tree = Tree(Text.assemble((svc, "cyan"), overflow="fold")) + for vname, hit_svc, key in hits: + if hit_svc == svc: + tree.add(f"[text2]{kid_norm}:{key} from {vname}") + console.print(Padding(tree, (1, 5))) @kv.command() diff --git a/unshackle/commands/serve.py b/unshackle/commands/serve.py index 65ccbde6..3480f9de 100644 --- a/unshackle/commands/serve.py +++ b/unshackle/commands/serve.py @@ -8,6 +8,7 @@ from aiohttp import web from unshackle.core import binaries from unshackle.core.api import cors_middleware, setup_routes, setup_swagger from unshackle.core.api.compression import compression_middleware +from unshackle.core.api.handlers import request_secret_key from unshackle.core.config import config from unshackle.core.constants import context_settings from unshackle.core.downloaders import format_speed, parse_speed_limit, set_speed_limit @@ -130,11 +131,11 @@ def serve( config.serve["playready_devices"].extend(list(config.directories.prds.glob("*.prd"))) @web.middleware - async def api_key_authentication(request: web.Request, handler) -> web.Response: + async def api_key_authentication(request: web.Request, handler) -> web.StreamResponse: """Authenticate API requests using X-Secret-Key header.""" if request.path == "/api/health": return await handler(request) - secret_key = request.headers.get("X-Secret-Key") + secret_key = request_secret_key(request) if not secret_key: return web.json_response({"status": 401, "message": "Secret Key is Empty."}, status=401) # Constant-time compare against every configured key so a valid key can't be @@ -159,6 +160,10 @@ def serve( if global_services: log.info(f"Global service allowlist: {', '.join(global_services)}") users = config.serve.get("users", {}) + if isinstance(users, dict): + # yaml keys can parse as int; hmac.compare_digest and allowlist lookups need str + users = {str(k): v for k, v in users.items()} + config.serve["users"] = users for user_key, user_cfg in users.items() if isinstance(users, dict) else []: user_services = user_cfg.get("services") if isinstance(user_cfg, dict) else None if user_services: @@ -172,7 +177,10 @@ def serve( app["config"] = {"users": {}} else: app = web.Application(middlewares=[cors_middleware, api_key_authentication, compression_middleware]) - app["config"] = {"users": {api_secret: {"devices": [], "username": "api_user"}}} + api_users: dict = {api_secret: {"devices": [], "username": "api_user"}} + if isinstance(users, dict): + api_users.update(users) + app["config"] = {"users": api_users} app["debug_api"] = debug_api # Start session cleanup loop for remote-dl sessions @@ -238,9 +246,14 @@ def serve( def create_serve_authentication(serve_playready_flag: bool): @web.middleware - async def serve_authentication(request: web.Request, handler) -> web.Response: + async def serve_authentication(request: web.Request, handler) -> web.StreamResponse: + secret_key = request_secret_key(request) if serve_playready_flag and request.path in ("/playready", "/playready/"): response = await handler(request) + elif secret_key and not request.headers.get("X-Secret-Key"): + if not any(hmac.compare_digest(secret_key, k) for k in request.app["config"]["users"]): + return web.json_response({"status": 401, "message": "Secret Key is Invalid."}, status=401) + response = await handler(request) else: response = await pywidevine_serve.authentication(request, handler) diff --git a/unshackle/commands/util.py b/unshackle/commands/util.py index d2322028..8365d16d 100644 --- a/unshackle/commands/util.py +++ b/unshackle/commands/util.py @@ -9,7 +9,7 @@ from unshackle.core import binaries from unshackle.core.constants import context_settings -def _natural_sort_key(path: Path) -> list: +def natural_sort_key(path: Path) -> list: """Sort key for natural sorting (S01E01 before S01E10).""" return [int(part) if part.isdigit() else part.lower() for part in re.split(r"(\d+)", path.name)] @@ -83,7 +83,7 @@ def crop(path: Path, aspect: str, letter: bool, offset: int, preview: bool) -> N raise click.ClickException('FFmpeg executable "ffmpeg" not found but is required.') if path.is_dir(): - paths = sorted(list(path.glob("*.mkv")) + list(path.glob("*.mp4")), key=_natural_sort_key) + paths = sorted(list(path.glob("*.mkv")) + list(path.glob("*.mp4")), key=natural_sort_key) else: paths = [path] for video_path in paths: @@ -186,7 +186,7 @@ def range_(path: Path, full: bool, preview: bool) -> None: raise click.ClickException('FFmpeg executable "ffmpeg" not found but is required.') if path.is_dir(): - paths = sorted(list(path.glob("*.mkv")) + list(path.glob("*.mp4")), key=_natural_sort_key) + paths = sorted(list(path.glob("*.mkv")) + list(path.glob("*.mp4")), key=natural_sort_key) else: paths = [path] for video_path in paths: @@ -272,7 +272,7 @@ def test(path: Path, map_: str) -> None: raise click.ClickException('FFmpeg executable "ffmpeg" not found but is required.') if path.is_dir(): - paths = sorted(list(path.glob("*.mkv")) + list(path.glob("*.mp4")), key=_natural_sort_key) + paths = sorted(list(path.glob("*.mkv")) + list(path.glob("*.mp4")), key=natural_sort_key) else: paths = [path] for video_path in paths: diff --git a/unshackle/core/__init__.py b/unshackle/core/__init__.py index 130bca20..65e49af2 100644 --- a/unshackle/core/__init__.py +++ b/unshackle/core/__init__.py @@ -3,11 +3,15 @@ import os from pathlib import Path from typing import Callable, Optional -__version__ = "5.3.0" +from rich_click.patch import patch as _patch_click_help -_PKG = Path(__file__).parent.parent +_patch_click_help() + +__version__ = "5.5.0" + +PKG = Path(__file__).parent.parent # Framework code only. Services are user-swappable, so they are not part of the identity. -_CODE_DIRS = ("core", "commands", "utils", "vaults") +CODE_DIRS = ("core", "commands", "utils", "vaults") def _raise(error: OSError) -> None: @@ -16,9 +20,9 @@ def _raise(error: OSError) -> None: def code_files() -> list[str]: """Framework source paths relative to the package root, in a platform-stable order.""" - pkg = str(_PKG) + pkg = str(PKG) rels = ["__main__.py"] - for name in _CODE_DIRS: + for name in CODE_DIRS: for root, dirs, files in os.walk(os.path.join(pkg, name), onerror=_raise): dirs[:] = [d for d in dirs if d != "__pycache__"] rel = os.path.relpath(root, pkg).replace(os.sep, "/") @@ -37,7 +41,7 @@ def code_hash( (see tools/resolve_code_hash.py). Returns "" when the source cannot be read. """ if read is None: - pkg = str(_PKG) + pkg = str(PKG) def read(rel: str) -> bytes: with open(os.path.join(pkg, rel), "rb") as fh: diff --git a/unshackle/core/api/compression.py b/unshackle/core/api/compression.py index 8b05a607..676c2688 100644 --- a/unshackle/core/api/compression.py +++ b/unshackle/core/api/compression.py @@ -25,10 +25,13 @@ def safe_inflate(data: bytes, max_size: int = MAX_INFLATE_BYTES) -> bytes: @web.middleware -async def compression_middleware(request: web.Request, handler) -> web.Response: +async def compression_middleware(request: web.Request, handler) -> web.StreamResponse: """Compress JSON responses with gzip when the client supports it.""" response = await handler(request) + if not isinstance(response, web.Response): + return response + accept_encoding = request.headers.get("Accept-Encoding", "") if "gzip" not in accept_encoding: return response @@ -37,7 +40,7 @@ async def compression_middleware(request: web.Request, handler) -> web.Response: return response body = response.body - if body is None or len(body) < 256: + if not isinstance(body, (bytes, bytearray)) or len(body) < 256: return response from unshackle.core.config import config diff --git a/unshackle/core/api/download_manager.py b/unshackle/core/api/download_manager.py index 0644bf9b..147133fa 100644 --- a/unshackle/core/api/download_manager.py +++ b/unshackle/core/api/download_manager.py @@ -24,15 +24,15 @@ log = logging.getLogger("download_manager") # Job parameters may carry secrets (a raw "user:pass" credential, a proxy URL with embedded # userinfo). These must never leave the process via the API or logs, so they are masked # wherever parameters are serialized for a response. -_SENSITIVE_PARAM_KEYS = ("credential", "credentials", "password", "token", "api_key") +SENSITIVE_PARAM_KEYS = ("credential", "credentials", "password", "token", "api_key") -def _redact_parameters(parameters: Dict[str, Any]) -> Dict[str, Any]: +def redact_parameters(parameters: Dict[str, Any]) -> Dict[str, Any]: """Return a copy of job parameters with secrets masked, safe to serialize.""" if not isinstance(parameters, dict): return parameters redacted = dict(parameters) - for key in _SENSITIVE_PARAM_KEYS: + for key in SENSITIVE_PARAM_KEYS: if redacted.get(key): redacted[key] = REDACTED proxy = redacted.get("proxy") @@ -41,7 +41,7 @@ def _redact_parameters(parameters: Dict[str, Any]) -> Dict[str, Any]: return redacted -def _secret_values(parameters: Dict[str, Any]) -> List[str]: +def secret_values(parameters: Dict[str, Any]) -> List[str]: """Raw secret strings carried in job parameters, longest first, for scrubbing free text.""" if not isinstance(parameters, dict): return [] @@ -65,7 +65,7 @@ def _secret_values(parameters: Dict[str, Any]) -> List[str]: def _redact_text(text: Optional[str], parameters: Dict[str, Any]) -> Optional[str]: """Mask proxy userinfo and any known parameter secrets that leaked into a free-text field (error message / details / traceback / worker stderr) before it is returned via the API.""" - return redact_text(text, _secret_values(parameters)) + return redact_text(text, secret_values(parameters)) class JobStatus(Enum): @@ -154,10 +154,10 @@ class DownloadJob: if include_full_details: # Error/stderr/traceback are free text a service may have echoed a credential or proxy - # URL into, so scrub them with the same secrets that _redact_parameters masks. + # URL into, so scrub them with the same secrets that redact_parameters masks. result.update( { - "parameters": _redact_parameters(self.parameters), + "parameters": redact_parameters(self.parameters), "started_time": self.started_time.isoformat() if self.started_time else None, "completed_time": self.completed_time.isoformat() if self.completed_time else None, "output_files": self.output_files, @@ -172,14 +172,14 @@ class DownloadJob: return result -def _history_path() -> Path: +def history_path() -> Path: """Path of the persistent job history file (under the cache dir, kept out of the config/data tree).""" from unshackle.core.config import config return config.directories.cache / "api_history.jsonl" -def _history_limit() -> int: +def history_limit() -> int: """Max history entries to retain (serve.history_limit, default 100; <= 0 means unlimited).""" from unshackle.core.config import config @@ -207,14 +207,14 @@ def record_job_history(job: DownloadJob) -> None: "created_time": job.created_time.isoformat(), "completed_time": job.completed_time.isoformat() if job.completed_time else None, "output_files": job.output_files, - "parameters": _redact_parameters(job.parameters), + "parameters": redact_parameters(job.parameters), "error_message": _redact_text(job.error_message, job.parameters), } try: - path = _history_path() + path = history_path() path.parent.mkdir(parents=True, exist_ok=True) line = json.dumps(entry) - limit = _history_limit() + limit = history_limit() # Serialized on the manager's event loop, so append + periodic trim doesn't race. with open(path, "a", encoding="utf-8") as handle: handle.write(line + "\n") @@ -233,7 +233,7 @@ def record_job_history(job: DownloadJob) -> None: def read_job_history(limit: int = 100, service: Optional[str] = None) -> List[Dict[str, Any]]: """Read persisted job history, newest first; corrupt lines are skipped, missing file = empty.""" - path = _history_path() + path = history_path() if not path.exists(): return [] try: @@ -265,7 +265,7 @@ def delete_job_history(job_id: str, allowed: Optional[set] = None) -> bool: When `allowed` is given, an entry outside the caller's service allowlist is treated as absent (returns False) so it can't be deleted by a restricted key. """ - path = _history_path() + path = history_path() if not path.exists(): return False try: @@ -306,7 +306,7 @@ def to_enum(values: List[str], enum_cls: type[Enum]) -> List[Enum]: return [lookup[v.upper()] for v in values if v.upper() in lookup] -def _normalize_sub_format(sub_format_raw: str) -> Any: +def normalize_sub_format(sub_format_raw: str) -> Any: """Normalize an API sub_format string to what dl.py expects. "original" is a sentinel meaning "keep source format" (dl.py skips conversion), so it must @@ -321,7 +321,7 @@ def _normalize_sub_format(sub_format_raw: str) -> Any: return mapped[0] if mapped else None -def _perform_download( +def perform_download( job_id: str, service: str, title_id: str, @@ -380,7 +380,7 @@ def _perform_download( sub_format_raw = params.get("sub_format") if sub_format_raw and isinstance(sub_format_raw, str): - params["sub_format"] = _normalize_sub_format(sub_format_raw) + params["sub_format"] = normalize_sub_format(sub_format_raw) if params.get("export"): params["export"] = bool(params["export"]) @@ -442,8 +442,9 @@ def _perform_download( "imdb_id": params.get("imdb_id"), "tvdb_id": params.get("tvdb_id"), "tvdb_order": params.get("tvdb_order"), - "animeapi_id": params.get("animeapi_id"), + "anilist_id": params.get("anilist_id"), "enrich": params.get("enrich", False), + "daily": params.get("daily", False), "output_dir": Path(params["output_dir"]) if params.get("output_dir") else None, "no_cache": params.get("no_cache", False), "reset_cache": params.get("reset_cache", False), @@ -474,8 +475,9 @@ def _perform_download( imdb_id=params.get("imdb_id"), tvdb_id=params.get("tvdb_id"), tvdb_order=params.get("tvdb_order"), - animeapi_id=params.get("animeapi_id"), + anilist_id=params.get("anilist_id"), enrich=params.get("enrich", False), + daily=params.get("daily", False), output_dir=Path(params["output_dir"]) if params.get("output_dir") else None, ) # Per-request CDM override (a device name in the WVDs dir); get_cdm() takes it first. @@ -643,6 +645,7 @@ class DownloadQueueManager: self._job_temp_files: Dict[str, Dict[str, str]] = {} self._workers_started = False self._shutdown_event = asyncio.Event() + self._subscribers: Dict[str, List[asyncio.Queue]] = {} log.info( f"Initialized download queue manager: max_concurrent={max_concurrent_downloads}, retention_hours={job_retention_hours}" @@ -675,6 +678,48 @@ class DownloadQueueManager: """List all jobs.""" return list(self._jobs.values()) + def subscribe(self, job_id: str) -> asyncio.Queue: + """Register a listener for a job's events, returning the queue it receives them on.""" + queue: asyncio.Queue = asyncio.Queue(maxsize=100) + self._subscribers.setdefault(job_id, []).append(queue) + return queue + + def unsubscribe(self, job_id: str, queue: asyncio.Queue) -> None: + """Drop a listener registered by subscribe().""" + queues = self._subscribers.get(job_id) + if not queues: + return + with suppress(ValueError): + queues.remove(queue) + if not queues: + del self._subscribers[job_id] + + def publish(self, job: DownloadJob, event: str) -> None: + """Fan an event carrying the job's full detail dict out to every listener.""" + queues = self._subscribers.get(job.job_id) + if not queues: + return + payload = {"event": event, "data": job.to_dict(include_full_details=True)} + for queue in queues: + self.put(queue, payload) + + def publish_terminal(self, job: DownloadJob) -> None: + """Publish a job's final state, then close every listener's stream.""" + self.publish(job, job.status.value) + for queue in self._subscribers.pop(job.job_id, []): + self.put(queue, None) + + @staticmethod + def put(queue: asyncio.Queue, item: Optional[Dict[str, Any]]) -> None: + """Enqueue an item, discarding the oldest when a slow listener has filled the queue.""" + try: + queue.put_nowait(item) + except asyncio.QueueFull: + with suppress(asyncio.QueueEmpty): + queue.get_nowait() + with suppress(asyncio.QueueFull): + queue.put_nowait(item) + def cancel_job(self, job_id: str) -> bool: """Cancel a job if it's queued or downloading.""" job = self._jobs.get(job_id) @@ -686,6 +731,7 @@ class DownloadQueueManager: job.cancel_event.set() # Signal cancellation job.completed_time = datetime.now() record_job_history(job) # queued jobs never reach a worker, so persist here + self.publish_terminal(job) log.info(f"Cancelled queued job {sanitize_log(job_id)}") return True elif job.status == JobStatus.DOWNLOADING: @@ -771,10 +817,10 @@ class DownloadQueueManager: # Start worker tasks for i in range(self.max_concurrent_downloads): - asyncio.create_task(self._download_worker(f"worker-{i}")) + asyncio.create_task(self.download_worker(f"worker-{i}")) # Start cleanup task - asyncio.create_task(self._cleanup_worker()) + asyncio.create_task(self.cleanup_worker()) log.info(f"Started {self.max_concurrent_downloads} download workers") @@ -783,6 +829,11 @@ class DownloadQueueManager: log.info("Shutting down download queue manager") self._shutdown_event.set() + for queues in list(self._subscribers.values()): + for queue in queues: + self.put(queue, None) + self._subscribers.clear() + # Cancel all active downloads for task in self._active_downloads.values(): task.cancel() @@ -817,7 +868,7 @@ class DownloadQueueManager: if self._active_downloads: await asyncio.gather(*self._active_downloads.values(), return_exceptions=True) - async def _download_worker(self, worker_name: str): + async def download_worker(self, worker_name: str): """Worker task that processes jobs from the queue.""" log.debug(f"Download worker {worker_name} started") @@ -839,11 +890,12 @@ class DownloadQueueManager: # Start processing the job job.status = JobStatus.DOWNLOADING job.started_time = datetime.now() + self.publish(job, "status") log.info(f"Worker {worker_name} starting job {job.job_id}") # Create download task - download_task = asyncio.create_task(self._execute_download(job)) + download_task = asyncio.create_task(self.execute_download(job)) self._active_downloads[job.job_id] = download_task try: @@ -858,6 +910,7 @@ class DownloadQueueManager: finally: job.completed_time = datetime.now() record_job_history(job) + self.publish_terminal(job) if job.job_id in self._active_downloads: del self._active_downloads[job.job_id] @@ -866,12 +919,12 @@ class DownloadQueueManager: except Exception as e: log.error(f"Worker {worker_name} error: {e}") - async def _execute_download(self, job: DownloadJob): + async def execute_download(self, job: DownloadJob): """Execute the actual download for a job.""" log.info(f"Executing download for job {job.job_id}") try: - output_files = await self._run_download_async(job) + output_files = await self.run_download_async(job) job.status = JobStatus.COMPLETED job.output_files = output_files job.progress = 100.0 @@ -895,7 +948,7 @@ class DownloadQueueManager: log.error(f"Download failed for job {job.job_id}: {e}") raise - async def _run_download_async(self, job: DownloadJob) -> List[str]: + async def run_download_async(self, job: DownloadJob) -> List[str]: """Invoke a worker subprocess to execute the download.""" payload = { @@ -934,6 +987,7 @@ class DownloadQueueManager: stdout_bytes = b"" stderr_bytes = b"" last_progress_stat: Optional[tuple[int, int]] = None # (st_mtime_ns, st_size) of last parse + last_published: Optional[Dict[str, Any]] = None try: while True: @@ -981,6 +1035,9 @@ class DownloadQueueManager: if new_progress != job.progress: job.progress = new_progress log.info(f"Job {job.job_id} progress updated: {job.progress}%") + if progress_data != last_published: + last_published = progress_data + self.publish(job, "progress") except (FileNotFoundError, json.JSONDecodeError, ValueError) as e: log.debug(f"Could not read progress for job {job.job_id}: {e}") @@ -1052,7 +1109,7 @@ class DownloadQueueManager: except OSError: pass - async def _cleanup_worker(self): + async def cleanup_worker(self): """Worker that periodically cleans up old jobs.""" while not self._shutdown_event.is_set(): try: diff --git a/unshackle/core/api/download_worker.py b/unshackle/core/api/download_worker.py index 81cf0072..45dc17fb 100644 --- a/unshackle/core/api/download_worker.py +++ b/unshackle/core/api/download_worker.py @@ -4,25 +4,28 @@ from __future__ import annotations import json import logging +import os import sys import traceback from pathlib import Path from typing import Any, Dict -from .download_manager import _perform_download +from .download_manager import perform_download log = logging.getLogger("download_worker") -def _read_payload(path: Path) -> Dict[str, Any]: +def read_payload(path: Path) -> Dict[str, Any]: with path.open("r", encoding="utf-8") as handle: return json.load(handle) -def _write_result(path: Path, payload: Dict[str, Any]) -> None: +def write_result(path: Path, payload: Dict[str, Any]) -> None: + # Atomic replace: the parent polls this file while it is being rewritten. path.parent.mkdir(parents=True, exist_ok=True) - with path.open("w", encoding="utf-8") as handle: - json.dump(payload, handle) + tmp = path.with_name(path.name + ".tmp") + tmp.write_text(json.dumps(payload), encoding="utf-8") + os.replace(tmp, path) def main(argv: list[str]) -> int: @@ -41,7 +44,7 @@ def main(argv: list[str]) -> int: exit_code = 0 try: - payload = _read_payload(payload_path) + payload = read_payload(payload_path) job_id = payload["job_id"] service = payload["service"] title_id = payload["title_id"] @@ -58,12 +61,12 @@ def main(argv: list[str]) -> int: try: progress_state.update(progress_data) log.info(f"Writing progress update: {progress_data}") - _write_result(progress_path, progress_state) + write_result(progress_path, progress_state) log.info(f"Progress update written to {progress_path}") except Exception as e: log.error(f"Failed to write progress update: {e}") - output_files = _perform_download(job_id, service, title_id, params, progress_callback=progress_callback) + output_files = perform_download(job_id, service, title_id, params, progress_callback=progress_callback) result = {"status": "success", "output_files": output_files} @@ -93,7 +96,7 @@ def main(argv: list[str]) -> int: finally: try: - _write_result(result_path, result) + write_result(result_path, result) except Exception as exc: # noqa: BLE001 - last resort logging log.error(f"Failed to write worker result file: {exc}") diff --git a/unshackle/core/api/errors.py b/unshackle/core/api/errors.py index 42f6c319..7fe2226d 100644 --- a/unshackle/core/api/errors.py +++ b/unshackle/core/api/errors.py @@ -73,10 +73,10 @@ class APIError(Exception): self.message = message self.details = details or {} self.retryable = retryable - self.http_status = http_status or self._default_http_status(error_code) + self.http_status = http_status or self.default_http_status(error_code) @staticmethod - def _default_http_status(error_code: APIErrorCode) -> int: + def default_http_status(error_code: APIErrorCode) -> int: """Map error codes to default HTTP status codes.""" status_map = { # 400 Bad Request diff --git a/unshackle/core/api/handlers.py b/unshackle/core/api/handlers.py index c6d52d45..771c2bc6 100644 --- a/unshackle/core/api/handlers.py +++ b/unshackle/core/api/handlers.py @@ -2,6 +2,7 @@ import asyncio import enum import logging import re +from datetime import date as date_ from pathlib import Path from types import SimpleNamespace from typing import Any, Dict, List, Optional @@ -14,6 +15,7 @@ from unshackle.core.api.input_bridge import AuthStatus, InputBridge from unshackle.core.api.sanitize import safe_cache_key, sanitize_log from unshackle.core.config import config from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_CODEC_MAP +from unshackle.core.providers.anilist import parse_anilist_ref from unshackle.core.proxies.resolve import initialize_proxy_providers, resolve_proxy from unshackle.core.services import Services from unshackle.core.titles import Episode, Movie, Title_T @@ -77,8 +79,9 @@ DEFAULT_DOWNLOAD_PARAMS = { "imdb_id": None, "tvdb_id": None, "tvdb_order": None, - "animeapi_id": None, + "anilist_id": None, "enrich": False, + "daily": False, "output_dir": None, "no_cache": False, "reset_cache": False, @@ -105,7 +108,7 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str] """Load a real CDM object for the given service. Services often touch ``ctx.obj.cdm.security_level`` / ``.device_type`` / ``.system_id`` - inside ``__init__``, so the lightweight ``_resolve_server_cdm`` stub is not enough + inside ``__init__``, so the lightweight ``resolve_server_cdm`` stub is not enough for list_titles / list_tracks / search. Mirrors ``dl.get_cdm`` selection logic but skips the quality-tier shortcuts (no track context yet) and falls back to the stub if no device is configured or loading fails. @@ -127,7 +130,7 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str] cdm_name = cdm_name.get(profile) or cdm_name.get("default") or ci_get(app_config.cdm, "default") if not cdm_name or not isinstance(cdm_name, str): - return _resolve_server_cdm(service, profile, cdm_type) + return resolve_server_cdm(service, profile, cdm_type) try: return load_cdm(cdm_name, service_name=service) @@ -135,7 +138,7 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str] log.warning( f"load_cdm({sanitize_log(cdm_name)!r}) failed for {sanitize_log(service)}: {exc}; using lightweight stub" ) - return _resolve_server_cdm(service, profile, cdm_type) + return resolve_server_cdm(service, profile, cdm_type) def load_service_yaml(normalized_service: str) -> dict: @@ -260,7 +263,7 @@ def get_allowed_services(request: Optional[web.Request] = None) -> Optional[List key_set: Optional[set[str]] = None if request: - secret_key = request.headers.get("X-Secret-Key") + secret_key = request_secret_key(request) if secret_key: users = config.serve.get("users", {}) user_config = users.get(secret_key, {}) @@ -280,9 +283,47 @@ def get_allowed_services(request: Optional[web.Request] = None) -> Optional[List return list(result) +def server_cdm_allowed(request: Optional[web.Request] = None) -> bool: + """Whether the calling key may have the server run the CDM licensing. + + Configured keys opt in with ``server_cdm: true``; keys absent from + ``serve.users`` (the admin secret) keep full access. + """ + if not request: + return True + secret_key = request_secret_key(request) + user_config = config.serve.get("users", {}).get(secret_key) + if user_config is None: + return True + return bool(user_config.get("server_cdm", False)) + + +JOB_EVENTS_ROUTE = "/api/download/jobs/{job_id}/events" + +CORS_HEADERS = { + "Access-Control-Allow-Origin": "*", + "Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, OPTIONS", + "Access-Control-Allow-Headers": "Content-Type, X-Secret-Key, Authorization", + "Access-Control-Max-Age": "3600", +} + + +def request_secret_key(request: web.Request) -> Optional[str]: + """The caller's key: the X-Secret-Key header, or on the events route the secret_key + query param (EventSource cannot send headers).""" + key = request.headers.get("X-Secret-Key") + if not key: + resource = request.match_info.route.resource + if resource is not None and resource.canonical == JOB_EVENTS_ROUTE: + key = request.query.get("secret_key") + return key + + def caller_key(request: Optional[web.Request] = None) -> str: - """The authenticating X-Secret-Key for a request, or 'anonymous' when unauthenticated.""" - return request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous" + """The authenticating key for a request, or 'anonymous' when unauthenticated.""" + if not request: + return "anonymous" + return request_secret_key(request) or "anonymous" def owns_job(job: Any, request: Optional[web.Request] = None) -> bool: @@ -301,6 +342,8 @@ def owns_job(job: Any, request: Optional[web.Request] = None) -> bool: def validate_service(service_tag: str, request: Optional[web.Request] = None) -> Optional[str]: """Validate, normalize, and check allowlist for service tag.""" + if not isinstance(service_tag, str): + return None try: normalized = Services.get_tag(service_tag) service_path = Services.get_path(normalized) @@ -310,7 +353,7 @@ def validate_service(service_tag: str, request: Optional[web.Request] = None) -> if allowed is not None and normalized not in allowed: return None return normalized - except Exception: + except KeyError: return None @@ -325,7 +368,7 @@ def require_fields(data: Dict[str, Any], *names: str) -> None: ) -def _part_key_suffix(part: Optional[int]) -> str: +def part_key_suffix(part: Optional[int]) -> str: """`.2` selection-syntax suffix for a part-ful episode, empty otherwise.""" return f".{part}" if part is not None else "" @@ -344,7 +387,7 @@ def serialize_title(title: Title_T) -> Dict[str, Any]: is_episode = isinstance(title, Episode) episode_part = title.part if isinstance(title, Episode) else None if is_episode: - # no part suffix here: remote_service._build_title rebuilds the Episode from this dict, so a + # no part suffix here: remote_service.build_title rebuilds the Episode from this dict, so a # suffixed name plus the structural `part` below would render the part twice in the filename name = title.name if title.name else f"Episode {title.number:02d}" else: @@ -362,17 +405,42 @@ def serialize_title(title: Title_T) -> Dict[str, Any]: # "other" titles carry no year; only Episode/Movie do. if isinstance(title, (Episode, Movie)): result["year"] = title.year - if is_episode: + if isinstance(title, Episode): result["series_title"] = str(title.title) result["season"] = title.season result["number"] = title.number + # every key below is conditional, so JSON for a title without them is unchanged if episode_part is not None: - result["part"] = episode_part # conditional, so part-less JSON is unchanged + result["part"] = episode_part + if title.air_date is not None: + result["air_date"] = ( + title.air_date.isoformat() if isinstance(title.air_date, date_) else str(title.air_date) + ) + if title.absolute is not None: + result["absolute"] = title.absolute + if getattr(title, "daily", None) is not None: + result["daily"] = title.daily + if isinstance(title, (Episode, Movie)) and getattr(title, "anime", None) is not None: + result["anime"] = title.anime return result -def _extract_manifests(tracks) -> List[Dict[str, Any]]: +def stamp_service_flags(serialized: Dict[str, Any], service_instance: Any) -> Dict[str, Any]: + """Fill anime/daily from the service class for titles that set neither. + + The client rebuilds titles from this JSON against a synthetic service class, so a + class-level ANIME/DAILY would otherwise be lost on the way across. + """ + for key, attr in (("anime", "ANIME"), ("daily", "DAILY")): + if key == "daily" and serialized.get("type") != "episode": + continue + if serialized.get(key) is None and getattr(type(service_instance), attr, False): + serialized[key] = True + return serialized + + +def extract_manifests(tracks) -> List[Dict[str, Any]]: """Extract manifest data from tracks for client-side re-parsing. Serializes DASH and ISM manifest XML as zlib-compressed base64 strings @@ -574,11 +642,13 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = query = data.get("query") if not service_tag: - raise APIError(APIErrorCode.MISSING_SERVICE, "Missing required 'service' field") + raise APIError(APIErrorCode.INVALID_INPUT, "Missing required 'service' field") if not query: raise APIError(APIErrorCode.INVALID_PARAMETERS, "Missing required 'query' field") - normalized_service = Services.get_tag(service_tag) + # get_tag echoes an unknown tag back, so it can never be falsy; validate_service + # is the check that actually resolves the service directory and the allowlist. + normalized_service = validate_service(service_tag, request) if not normalized_service: raise APIError( APIErrorCode.INVALID_SERVICE, @@ -586,14 +656,6 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = details={"service": service_tag}, ) - allowed = get_allowed_services(request) - if allowed is not None and normalized_service not in allowed: - raise APIError( - APIErrorCode.INVALID_SERVICE, - f"Service '{service_tag}' not found", - details={"service": service_tag}, - ) - profile = data.get("profile") proxy_param = data.get("proxy") no_proxy = data.get("no_proxy", False) @@ -676,9 +738,9 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques titles = service_instance.get_titles() if hasattr(titles, "__iter__") and not isinstance(titles, str): - title_list = [serialize_title(t) for t in titles] + title_list = [stamp_service_flags(serialize_title(t), service_instance) for t in titles] else: - title_list = [serialize_title(titles)] + title_list = [stamp_service_flags(serialize_title(titles), service_instance)] return web.json_response({"titles": title_list}) @@ -741,7 +803,7 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques details={"wanted": wanted_param, "service": normalized_service}, ) elif season is not None and episode is not None: - wanted = [f"{season}x{episode}{_part_key_suffix(part)}"] + wanted = [f"{season}x{episode}{part_key_suffix(part)}"] if wanted: # Filter titles based on wanted episodes, similar to how dl.py does it @@ -749,7 +811,7 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques log.debug(f"Filtering {len(titles_list)} titles with {len(wanted)} wanted episodes") for title in titles_list: if isinstance(title, Episode): - episode_key = f"{title.season}x{title.number}{_part_key_suffix(title.part)}" + episode_key = f"{title.season}x{title.number}{part_key_suffix(title.part)}" if title.matches_wanted(wanted): log.debug(f"Episode {episode_key} matches wanted list") matching_titles.append(title) @@ -798,12 +860,12 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques log.debug(f"Successfully got tracks for {title.season}x{title.number}") except SystemExit: # Service calls sys.exit() for unavailable episodes - catch and skip - failed_episodes.append(f"S{title.season}E{title.number:02d}{_part_key_suffix(title.part)}") + failed_episodes.append(f"S{title.season}E{title.number:02d}{part_key_suffix(title.part)}") log.debug(f"Episode {title.season}x{title.number} not available, skipping") continue except (Exception, SystemExit) as e: # Handle other errors gracefully - failed_episodes.append(f"S{title.season}E{title.number:02d}{_part_key_suffix(title.part)}") + failed_episodes.append(f"S{title.season}E{title.number:02d}{part_key_suffix(title.part)}") log.debug(f"Error getting tracks for {title.season}x{title.number}: {e}") continue @@ -926,6 +988,30 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]: if not isinstance(data["downloads"], int) or data["downloads"] <= 0: return "downloads must be a positive integer" + for name in ("tmdb_id", "tvdb_id"): + if data.get(name) is not None: + value = data[name] + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + return f"{name} must be a positive integer" + + if data.get("imdb_id") is not None: + if not isinstance(data["imdb_id"], str) or not re.fullmatch(r"tt\d+", data["imdb_id"]): + return "imdb_id must be an IMDB ID like 'tt1375666'" + + if data.get("anilist_id") is not None: + value = data["anilist_id"] + # same parser as --anilist so both surfaces accept and reject the same inputs + valid = not isinstance(value, bool) and isinstance(value, (int, str)) and parse_anilist_ref(value) + if not valid: + return "anilist_id must be a positive integer, or a MyAnimeList ID like 'mal:21'" + + supplied_ids = [name for name in ("tmdb_id", "imdb_id", "tvdb_id") if data.get(name)] + if len(supplied_ids) > 1: + return ( + f"Cannot use multiple external IDs: {', '.join(supplied_ids)}. " + "Give one ID and unshackle resolves the others from it." + ) + exclusive_flags = [] if data.get("video_only"): exclusive_flags.append("video_only") @@ -959,7 +1045,7 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]: return None -def enforce_download_gates(params: Dict[str, Any]) -> None: +def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request] = None) -> None: """Enforce serve-config gates on per-job cdm overrides and client-supplied credentials. A per-request `cdm` selects a server-side device, so it is gated here rather than honoured @@ -968,7 +1054,15 @@ def enforce_download_gates(params: Dict[str, Any]) -> None: A per-request `credential` (or `credentials` map) authenticates the job with client-supplied secrets instead of the server-side credentials. Gate it behind `serve.allow_job_credentials` (default off) so a default deployment stays locked to its own credentials; mirrors the CDM gate. + A download job licenses DRM in-process with the server's own CDM, so a key without + ``server_cdm`` cannot submit or retry jobs. """ + if not server_cdm_allowed(request): + raise APIError( + APIErrorCode.FORBIDDEN, + "Download jobs license with the server CDM, which is not enabled for this key.", + ) + requested_cdm = params.get("cdm") if requested_cdm: allowed = (config.serve or {}).get("cdm_overrides") @@ -1012,7 +1106,7 @@ async def download_handler(data: Dict[str, Any], request: Optional[web.Request] details={"service": normalized_service, "title_id": title_id}, ) - enforce_download_gates(data) + enforce_download_gates(data, request) try: # Load service module to extract service-specific parameter defaults @@ -1022,7 +1116,7 @@ async def download_handler(data: Dict[str, Any], request: Optional[web.Request] # Extract default values from the service's click command. # Skip None defaults here: this dict overlays into job params; injecting # None for keys like `profile` would clobber serve-config overrides. - # Missing required __init__ params are handled in download_manager._perform_download. + # Missing required __init__ params are handled in download_manager.perform_download. if hasattr(service_module, "cli") and hasattr(service_module.cli, "params"): for param in service_module.cli.params: if hasattr(param, "name") and param.default is not None and not isinstance(param.default, enum.Enum): @@ -1162,6 +1256,71 @@ async def get_download_job_handler(job_id: str, request: Optional[web.Request] = ) +async def download_job_events_handler(job_id: str, request: web.Request) -> web.StreamResponse: + """Stream a download job's progress to the caller as Server-Sent Events.""" + import json + + from unshackle.core.api.download_manager import TERMINAL_STATUSES, get_download_manager + + manager = get_download_manager() + job = manager.get_job(job_id) + + if not job or not owns_job(job, request): + raise APIError( + APIErrorCode.JOB_NOT_FOUND, + "Job not found", + details={"job_id": job_id}, + ) + + response = web.StreamResponse( + headers={ + "Content-Type": "text/event-stream", + "Cache-Control": "no-cache", + # Stops nginx from buffering the stream. + "X-Accel-Buffering": "no", + **CORS_HEADERS, + } + ) + await response.prepare(request) + + async def send(event: str, data: Dict[str, Any]) -> None: + payload = json.dumps(data, separators=(",", ":"), default=str) + await response.write(f"event: {event}\ndata: {payload}\n\n".encode()) + + queue: Optional[asyncio.Queue] = None + get_task: Optional[asyncio.Task] = None + try: + await send("snapshot", job.to_dict(include_full_details=True)) + + if job.status not in TERMINAL_STATUSES: + queue = manager.subscribe(job_id) + if queue is None or job.status in TERMINAL_STATUSES: + await send(job.status.value, job.to_dict(include_full_details=True)) + return response + + while True: + if get_task is None: + get_task = asyncio.create_task(queue.get()) + done, _ = await asyncio.wait({get_task}, timeout=15) + if not done: + await response.write(b": keep-alive\n\n") + continue + item = get_task.result() + get_task = None + if item is None: + break + await send(item["event"], item["data"]) + except (ConnectionResetError, asyncio.CancelledError): + log.debug(f"SSE client disconnected from job {sanitize_log(job_id)}") + finally: + if get_task is not None: + get_task.cancel() + if queue is not None: + manager.unsubscribe(job_id, queue) + + return response + + async def cancel_download_job_handler(job_id: str, request: Optional[web.Request] = None) -> web.Response: """Handle cancel/remove download job request.""" from unshackle.core.api.download_manager import TERMINAL_STATUSES, get_download_manager @@ -1256,7 +1415,7 @@ async def retry_download_job_handler(job_id: str, request: Optional[web.Request] f"Invalid or unavailable service: {job.service}", details={"service": job.service}, ) - enforce_download_gates(job.parameters) + enforce_download_gates(job.parameters, request) await manager.start_workers() @@ -1327,18 +1486,18 @@ async def prioritize_download_job_handler(job_id: str, request: Optional[web.Req # --------------------------------------------------------------------------- -_CONFIG_SECRET_KEY_RE = re.compile(r"secret|password|token|api_key|credential", re.IGNORECASE) +CONFIG_SECRET_KEY_RE = re.compile(r"secret|password|token|api_key|credential", re.IGNORECASE) -def _redact_config(value: Any) -> Any: +def redact_config(value: Any) -> Any: """Recursively mask secret-looking keys and URL userinfo; stringify paths.""" if isinstance(value, dict): return { - str(k): (REDACTED if v and _CONFIG_SECRET_KEY_RE.search(str(k)) else _redact_config(v)) + str(k): (REDACTED if v and CONFIG_SECRET_KEY_RE.search(str(k)) else redact_config(v)) for k, v in value.items() } if isinstance(value, (list, tuple)): - return [_redact_config(v) for v in value] + return [redact_config(v) for v in value] if isinstance(value, Path): return str(value) if isinstance(value, str) and "@" in value: @@ -1355,10 +1514,8 @@ async def profiles_handler(request: Optional[web.Request] = None) -> web.Respons # a plain (non-dict) credential is unnamed; that service is omitted entirely if not isinstance(creds, dict): continue - try: - tag = Services.get_tag(service) - except Exception: - tag = service + # get_tag returns its input unchanged when no service matches; str() covers non-str YAML keys + tag = Services.get_tag(str(service)) if allowed is not None and tag not in allowed: continue profiles[tag] = sorted(str(name) for name in creds) @@ -1385,14 +1542,14 @@ async def server_config_handler(request: Optional[web.Request] = None) -> web.Re service_tags = [t for t in service_tags if t in allowed] payload = { - "dl": _redact_config(config.dl), + "dl": redact_config(config.dl), "serve": { "max_concurrent_downloads": manager.max_concurrent_downloads, "job_retention_hours": manager.job_retention_hours, "history_limit": int(serve_cfg.get("history_limit", 100)), "services": serve_cfg.get("services") or None, "remote_only": bool(serve_cfg.get("remote_only", False)), - "cdm_overrides": _redact_config(serve_cfg.get("cdm_overrides")), + "cdm_overrides": redact_config(serve_cfg.get("cdm_overrides")), "allow_job_credentials": bool(serve_cfg.get("allow_job_credentials", False)), }, "directories": { @@ -1466,7 +1623,7 @@ async def delete_history_handler(job_id: str, request: Optional[web.Request] = N return handle_api_exception(e, context={"operation": "delete_history"}, debug_mode=debug_mode) -def _require_no_active_downloads(operation: str) -> None: +def require_no_active_downloads(operation: str) -> None: """Raise 409 CONFLICT if any job is currently downloading.""" from unshackle.core.api.download_manager import JobStatus, get_download_manager @@ -1484,7 +1641,7 @@ async def clear_cache_handler(request: Optional[web.Request] = None) -> web.Resp from unshackle.commands.env import clear_directory try: - _require_no_active_downloads("clear cache") + require_no_active_downloads("clear cache") _, freed_bytes = await asyncio.to_thread(clear_directory, config.directories.cache) return web.json_response({"cleared": True, "freed_bytes": freed_bytes}) @@ -1501,7 +1658,7 @@ async def clear_temp_handler(request: Optional[web.Request] = None) -> web.Respo from unshackle.commands.env import clear_directory try: - _require_no_active_downloads("clear temp") + require_no_active_downloads("clear temp") _, freed_bytes = await asyncio.to_thread(clear_directory, config.directories.temp) return web.json_response({"cleared": True, "freed_bytes": freed_bytes}) @@ -1538,10 +1695,10 @@ async def refresh_services_handler(request: Optional[web.Request] = None) -> web return handle_api_exception(e, context={"operation": "refresh_services"}, debug_mode=debug_mode) -_VERSION_RE = re.compile(r"\d+\.\d+(?:\.\d+)*") +VERSION_RE = re.compile(r"\d+\.\d+(?:\.\d+)*") -def _binary_version(path: Any) -> Optional[str]: +def binary_version(path: Any) -> Optional[str]: """Best-effort version probe of a binary; None when nothing parseable.""" import subprocess @@ -1552,7 +1709,7 @@ def _binary_version(path: Any) -> Optional[str]: ) except (OSError, subprocess.SubprocessError, ValueError): return None - match = _VERSION_RE.search(proc.stdout or "") or _VERSION_RE.search(proc.stderr or "") + match = VERSION_RE.search(proc.stdout or "") or VERSION_RE.search(proc.stderr or "") if match: return match.group(0) return None @@ -1562,7 +1719,7 @@ async def env_check_handler(request: Optional[web.Request] = None) -> web.Respon """Handle environment dependency check request.""" from unshackle.commands.env import get_dependencies - def _run_checks() -> List[Dict[str, Any]]: + def run_checks() -> List[Dict[str, Any]]: checks = [] for dep in get_dependencies(): binary = dep["binary"] @@ -1570,14 +1727,14 @@ async def env_check_handler(request: Optional[web.Request] = None) -> web.Respon { "name": dep["name"], "installed": binary is not None, - "version": _binary_version(binary) if binary else None, + "version": binary_version(binary) if binary else None, "required": dep["required"], } ) return checks try: - checks = await asyncio.to_thread(_run_checks) + checks = await asyncio.to_thread(run_checks) return web.json_response({"checks": checks}) except APIError: @@ -1614,7 +1771,7 @@ SESSION_TRANSPORT_KEYS = { } -def _create_service_instance( +def create_service_instance( normalized_service: str, title_id: str, data: Dict[str, Any], @@ -1740,7 +1897,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req ) try: - proxy_param, proxy_providers = _resolve_handler_proxy(data, normalized_service) + proxy_param, proxy_providers = resolve_handler_proxy(data, normalized_service) import hashlib import uuid as uuid_mod @@ -1753,7 +1910,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req api_key_hash = hashlib.pbkdf2_hmac("sha256", api_key.encode(), b"unshackle-session-ns", 100_000).hex()[:12] session_cache_tag = f"_sessions/{api_key_hash}/{session_id}/{normalized_service}" - service_instance, cookies, credential = _create_service_instance( + service_instance, cookies, credential = create_service_instance( normalized_service, title_id, data, @@ -1793,7 +1950,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req session.input_bridge = bridge session.auth_status = AuthStatus.AUTHENTICATING - async def _run_auth() -> None: + async def run_auth() -> None: try: await asyncio.to_thread(service_instance.authenticate, cookies, credential) session.auth_status = AuthStatus.AUTHENTICATED @@ -1804,7 +1961,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req session.auth_error = str(e) bridge.status = AuthStatus.FAILED - asyncio.create_task(_run_auth()) + asyncio.create_task(run_auth()) return web.json_response( { @@ -1833,8 +1990,8 @@ async def session_titles_handler(session_id: str, request: Optional[web.Request] interactive auth flows (OTP, captcha) can complete before titles are fetched. """ - session = await _get_validated_session(session_id, request) - _require_authenticated(session) + session = await get_validated_session(session_id, request) + require_authenticated(session) try: service_instance = session.service_instance @@ -1851,7 +2008,7 @@ async def session_titles_handler(session_id: str, request: Optional[web.Request] for t in titles_list: tid = str(t.id) if hasattr(t, "id") else str(id(t)) session.title_map[tid] = t - serialized_titles.append(serialize_title(t)) + serialized_titles.append(stamp_service_flags(serialize_title(t), service_instance)) return web.json_response( { @@ -1879,8 +2036,8 @@ async def session_tracks_handler( This keeps auth separate from track fetching, allowing interactive auth flows (OTP, captcha) before any tracks are requested. """ - session = await _get_validated_session(session_id, request) - _require_authenticated(session) + session = await get_validated_session(session_id, request) + require_authenticated(session) title_id = data.get("title_id") if not title_id: @@ -1919,7 +2076,7 @@ async def session_tracks_handler( video_tracks = sorted(tracks.videos, key=lambda t: t.bitrate or 0, reverse=True) audio_tracks = sorted(tracks.audio, key=lambda t: t.bitrate or 0, reverse=True) - manifests = _extract_manifests(tracks) + manifests = extract_manifests(tracks) svc_session = session.service_instance.session session_headers = dict(svc_session.headers) if hasattr(svc_session, "headers") else {} @@ -1937,7 +2094,7 @@ async def session_tracks_handler( has_pr = bool(user_cfg.get("playready_devices")) service_tag = session.service_tag - config_cdm_type = _detect_cdm_type_for_service(service_tag, app_config) + config_cdm_type = detect_cdm_type_for_service(service_tag, app_config) track_has_wv = any( d.__class__.__name__ == "Widevine" for t in list(tracks.videos) + list(tracks.audio) if t.drm for d in t.drm @@ -1982,6 +2139,7 @@ async def session_tracks_handler( "manifests": manifests, "session_headers": session_headers, "session_cookies": session_cookies, + "server_cdm": server_cdm_allowed(request), "server_cdm_type": server_cdm_type, } ) @@ -2005,7 +2163,7 @@ async def session_segments_handler( Returns segment URLs, init data, DRM info, and any headers/cookies needed for CDN download. """ - session = await _get_validated_session(session_id, request) + session = await get_validated_session(session_id, request) track_ids = data.get("track_ids", []) if not track_ids: @@ -2076,13 +2234,13 @@ async def session_segments_handler( ) -def _cdm_type_stub(cdm_type: str) -> SimpleNamespace: +def cdm_type_stub(cdm_type: str) -> SimpleNamespace: """Lightweight CDM stand-in so ``is_playready_cdm()`` (reads ``.is_playready``) can detect the type without loading a full CDM.""" return SimpleNamespace(is_playready=cdm_type == "playready") -def _resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]) -> Optional[Any]: +def resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]) -> Optional[Any]: """Resolve CDM for the server context. Checks the server's own CDM config (``config.cdm[service]``) to @@ -2105,16 +2263,16 @@ def _resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional cdm_name = cdm_name.get("default") or next(iter(cdm_name.values()), None) if cdm_name and isinstance(cdm_name, str): - detected_type = _detect_cdm_type(cdm_name, app_config) + detected_type = detect_cdm_type(cdm_name, app_config) if detected_type: - return _cdm_type_stub(detected_type) + return cdm_type_stub(detected_type) if cdm_type: - return _cdm_type_stub(cdm_type) + return cdm_type_stub(cdm_type) return None -def _detect_cdm_type_for_service(service: str, app_config: Any) -> Optional[str]: +def detect_cdm_type_for_service(service: str, app_config: Any) -> Optional[str]: """Detect the CDM type configured for a service in config.cdm.""" cdm_name = ci_get(app_config.cdm, service) if not cdm_name: @@ -2125,11 +2283,11 @@ def _detect_cdm_type_for_service(service: str, app_config: Any) -> Optional[str] return "playready" if "playready" in lower_keys else "widevine" cdm_name = cdm_name.get("default") or next(iter(cdm_name.values()), None) if cdm_name and isinstance(cdm_name, str): - return _detect_cdm_type(cdm_name, app_config) + return detect_cdm_type(cdm_name, app_config) return None -def _detect_cdm_type(cdm_name: str, app_config: Any) -> Optional[str]: +def detect_cdm_type(cdm_name: str, app_config: Any) -> Optional[str]: """Detect CDM type (playready/widevine) from config without loading it. Checks remote_cdm entries and local file extensions to determine the type. @@ -2152,7 +2310,7 @@ def _detect_cdm_type(cdm_name: str, app_config: Any) -> Optional[str]: return None -def _require_authenticated(session: Any) -> None: +def require_authenticated(session: Any) -> None: """Raise if the session has not finished authenticating.""" if session.auth_status == AuthStatus.FAILED: raise APIError( @@ -2173,7 +2331,7 @@ async def session_prompt_get_handler(session_id: str, request: Optional[web.Requ Returns the current auth status and any pending prompt that the remote client should display to the user. """ - session = await _get_validated_session(session_id, request) + session = await get_validated_session(session_id, request) if session.auth_status == AuthStatus.AUTHENTICATED: return web.json_response({"status": "authenticated"}) @@ -2198,7 +2356,7 @@ async def session_prompt_post_handler( The remote client calls this after collecting user input (OTP code, PIN, or device-code confirmation) to unblock the server auth thread. """ - session = await _get_validated_session(session_id, request) + session = await get_validated_session(session_id, request) response_text = data.get("response") if response_text is None: @@ -2212,7 +2370,7 @@ async def session_prompt_post_handler( return web.json_response({"status": "accepted"}) -async def _get_validated_session(session_id: str, request: Optional[web.Request]) -> Any: +async def get_validated_session(session_id: str, request: Optional[web.Request]) -> Any: """Fetch a session and verify the caller owns it. Ownership is bound to the authenticating X-Secret-Key rather than the source IP: @@ -2246,7 +2404,7 @@ async def _get_validated_session(session_id: str, request: Optional[web.Request] return session -def _resolve_handler_proxy(data: Dict[str, Any], normalized_service: str) -> tuple[Optional[str], list]: +def resolve_handler_proxy(data: Dict[str, Any], normalized_service: str) -> tuple[Optional[str], list]: """Resolve proxy and initialize providers from API request data. Handles explicit proxy param, provider:country format, and @@ -2276,7 +2434,8 @@ def _resolve_handler_proxy(data: Dict[str, Any], normalized_service: str) -> tup server_ip_info = get_ip_info(None, cached=True) server_region = server_ip_info.get("country", "").lower() if server_ip_info else None - except Exception: + except Exception as e: + log.debug(f"Server region lookup failed: {e!r}") server_region = None if server_region and server_region == client_region.lower(): @@ -2291,7 +2450,7 @@ def _resolve_handler_proxy(data: Dict[str, Any], normalized_service: str) -> tup return proxy_param, proxy_providers -def _find_title_for_track(track_id: str, session: Any) -> Any: +def find_title_for_track(track_id: str, session: Any) -> Any: """Find the title object that owns a given track.""" for t_id, tracks_dict in session.tracks_by_title.items(): if track_id in tracks_dict: @@ -2301,7 +2460,7 @@ def _find_title_for_track(track_id: str, session: Any) -> Any: return None -def _extract_pssh_from_track(track: Any, drm_type: str) -> Optional[str]: +def extract_pssh_from_track(track: Any, drm_type: str) -> Optional[str]: """Extract PSSH base64 string from a track's DRM objects.""" if not track.drm: return None @@ -2321,7 +2480,7 @@ def _extract_pssh_from_track(track: Any, drm_type: str) -> Optional[str]: return pssh_b64 -def _ensure_track_drm(track: Any) -> None: +def ensure_track_drm(track: Any) -> None: """Extract DRM from manifest data if track has none. Supports DASH (ContentProtection elements), HLS (EXT-X-KEY from @@ -2357,24 +2516,27 @@ def _ensure_track_drm(track: Any) -> None: if isinstance(drm, (Widevine, PlayReady)): track.drm = [drm] return - except Exception: + # EXT-X-KEY data is third-party input; skip keys we cannot turn into a DRM object + except Exception as e: + log.debug(f"Skipping HLS key {key}: {e!r}") continue - except Exception: - pass + except Exception as e: + log.debug(f"HLS DRM prefetch failed for {track.id}: {e!r}") # ISM: extract from ProtectionHeader elements if track.data.get("ism"): try: from unshackle.core.manifests import ISM as ISMManifest - stream_index = track.data["ism"].get("stream_index") - if stream_index is not None: - track.drm = ISMManifest.get_drm(stream_index) - except Exception: - pass + manifest = track.data["ism"].get("manifest") + if manifest is not None: + track.drm = ISMManifest.get_drm(manifest.xpath(".//ProtectionHeader")) + # a swallowed error here means the track silently gets no DRM, so it must be loud + except Exception as e: + log.warning(f"ISM DRM extraction failed for {track.id}: {e!r}") -def _resolve_device_name(user_config: dict, drm_type: str, service_tag: str = "") -> str: +def resolve_device_name(user_config: dict, drm_type: str, service_tag: str = "") -> str: """Get the CDM device name, checking service-specific config.cdm first. Resolution order: @@ -2402,7 +2564,7 @@ def _resolve_device_name(user_config: dict, drm_type: str, service_tag: str = "" return device_name -def _load_server_vaults(service_name: str) -> Any: +def load_server_vaults(service_name: str) -> Any: """Load server vaults from config.key_vaults.""" from unshackle.core.config import config as app_config from unshackle.core.services import Services @@ -2420,7 +2582,7 @@ def _load_server_vaults(service_name: str) -> Any: return vaults -def _check_vaults(kids: list, service_name: str) -> Optional[Dict[str, str]]: +def check_vaults(kids: list, service_name: str) -> Optional[Dict[str, str]]: """Check server vaults for existing keys matching all KIDs. Returns a KID:KEY dict if ALL KIDs are found, None otherwise. @@ -2428,7 +2590,7 @@ def _check_vaults(kids: list, service_name: str) -> Optional[Dict[str, str]]: from uuid import UUID try: - vaults = _load_server_vaults(service_name) + vaults = load_server_vaults(service_name) if not vaults.vaults: return None keys: Dict[str, str] = {} @@ -2442,17 +2604,18 @@ def _check_vaults(kids: list, service_name: str) -> Optional[Dict[str, str]]: if keys: log.info(f"Vault hit: {len(keys)} key(s) from server vaults, skipping CDM") return keys - except Exception: - pass + # vault lookup is a best-effort shortcut before the CDM; any failure falls through to licensing + except Exception as e: + log.debug(f"Server vault lookup failed: {e!r}") return None -def _cache_to_vaults(keys: Dict[str, str], service_name: str) -> None: +def cache_to_vaults(keys: Dict[str, str], service_name: str) -> None: """Cache newly obtained keys to server vaults.""" from uuid import UUID try: - vaults = _load_server_vaults(service_name) + vaults = load_server_vaults(service_name) if not vaults.vaults: return @@ -2464,7 +2627,7 @@ def _cache_to_vaults(keys: Dict[str, str], service_name: str) -> None: log.warning(f"Failed to cache keys to vaults: {e}") -def _handle_single_server_cdm( +def handle_single_server_cdm( service: Any, title: Any, track: Any, @@ -2478,10 +2641,10 @@ def _handle_single_server_cdm( from unshackle.core.cdm import load_cdm from unshackle.core.config import config as app_config - _ensure_track_drm(track) + ensure_track_drm(track) if not pssh_b64: - pssh_b64 = _extract_pssh_from_track(track, drm_type) + pssh_b64 = extract_pssh_from_track(track, drm_type) if not pssh_b64: raise APIError(APIErrorCode.INVALID_INPUT, "No PSSH available for server_cdm licensing") @@ -2498,9 +2661,9 @@ def _handle_single_server_cdm( # Gate on the caller's CDM device first so a caller with no device cannot # harvest server-side keys from the vault fallback below. - device_name = _resolve_device_name(user_config, drm_type, service.__class__.__name__) + device_name = resolve_device_name(user_config, drm_type, service.__class__.__name__) - vault_keys = _check_vaults(pr_drm.kids, service.__class__.__name__) + vault_keys = check_vaults(pr_drm.kids, service.__class__.__name__) if vault_keys: return vault_keys @@ -2521,9 +2684,9 @@ def _handle_single_server_cdm( # Gate on the caller's CDM device first so a caller with no device cannot # harvest server-side keys from the vault fallback below. - device_name = _resolve_device_name(user_config, drm_type, service.__class__.__name__) + device_name = resolve_device_name(user_config, drm_type, service.__class__.__name__) - vault_keys = _check_vaults(wv_drm.kids, service.__class__.__name__) + vault_keys = check_vaults(wv_drm.kids, service.__class__.__name__) if vault_keys: return vault_keys @@ -2545,11 +2708,11 @@ def _handle_single_server_cdm( if not keys: raise APIError(APIErrorCode.NO_CONTENT, "Server CDM returned no content keys") - _cache_to_vaults(keys, service.__class__.__name__) + cache_to_vaults(keys, service.__class__.__name__) return keys -def _handle_proxy_license( +def handle_proxy_license( service: Any, title: Any, track: Any, @@ -2594,7 +2757,7 @@ async def session_license_handler( """ import base64 - session = await _get_validated_session(session_id, request) + session = await get_validated_session(session_id, request) track_id = data.get("track_id") track_ids = data.get("track_ids") @@ -2602,6 +2765,12 @@ async def session_license_handler( drm_type = data.get("drm_type", "widevine") mode = data.get("mode", "proxy") + if mode == "server_cdm" and not server_cdm_allowed(request): + raise APIError( + APIErrorCode.FORBIDDEN, + "Server CDM licensing is not enabled for this key. Use a local CDM (proxy mode).", + ) + if mode == "server_cdm" and track_ids: from unshackle.core.config import config as app_config @@ -2612,7 +2781,7 @@ async def session_license_handler( has_pr_device = bool(user_config.get("playready_devices")) service_tag = session.service_tag - config_cdm_type = _detect_cdm_type_for_service(service_tag, app_config) + config_cdm_type = detect_cdm_type_for_service(service_tag, app_config) all_keys: Dict[str, Dict[str, str]] = {} seen_pssh: set[str] = set() @@ -2623,37 +2792,37 @@ async def session_license_handler( if not track: continue - _ensure_track_drm(track) + ensure_track_drm(track) if not track.drm: continue - title = _find_title_for_track(tid, session) + title = find_title_for_track(tid, session) track_drm_type = None pssh_str = None if config_cdm_type == "playready": - pssh_str = _extract_pssh_from_track(track, "playready") + pssh_str = extract_pssh_from_track(track, "playready") if pssh_str: track_drm_type = "playready" if not pssh_str: - pssh_str = _extract_pssh_from_track(track, "widevine") + pssh_str = extract_pssh_from_track(track, "widevine") if pssh_str: track_drm_type = "widevine" elif config_cdm_type == "widevine": - pssh_str = _extract_pssh_from_track(track, "widevine") + pssh_str = extract_pssh_from_track(track, "widevine") if pssh_str: track_drm_type = "widevine" if not pssh_str: - pssh_str = _extract_pssh_from_track(track, "playready") + pssh_str = extract_pssh_from_track(track, "playready") if pssh_str: track_drm_type = "playready" else: if has_wv_device: - pssh_str = _extract_pssh_from_track(track, "widevine") + pssh_str = extract_pssh_from_track(track, "widevine") if pssh_str: track_drm_type = "widevine" if not pssh_str and has_pr_device: - pssh_str = _extract_pssh_from_track(track, "playready") + pssh_str = extract_pssh_from_track(track, "playready") if pssh_str: track_drm_type = "playready" @@ -2669,7 +2838,7 @@ async def session_license_handler( seen_pssh.add(pssh_str) try: - keys = _handle_single_server_cdm(service, title, track, pssh_str, track_drm_type, request) + keys = handle_single_server_cdm(service, title, track, pssh_str, track_drm_type, request) if keys: all_keys[tid] = keys if track_drm_type: @@ -2696,7 +2865,7 @@ async def session_license_handler( ) try: - title = _find_title_for_track(track_id, session) + title = find_title_for_track(track_id, session) service = session.service_instance pssh_b64 = data.get("pssh") @@ -2722,11 +2891,11 @@ async def session_license_handler( track.drm.append(wv_drm) if mode == "server_cdm": - keys = _handle_single_server_cdm(service, title, track, pssh_b64, drm_type, request) + keys = handle_single_server_cdm(service, title, track, pssh_b64, drm_type, request) log.info(f"Server CDM resolved {len(keys)} key(s) for track {sanitize_log(track_id[:12])}") return web.json_response({"keys": keys}) - return _handle_proxy_license(service, title, track, challenge_b64, drm_type) + return handle_proxy_license(service, title, track, challenge_b64, drm_type) except APIError: raise @@ -2749,7 +2918,7 @@ async def session_license_handler( async def session_info_handler(session_id: str, request: Optional[web.Request] = None) -> web.Response: """Check session validity and get session info.""" - session = await _get_validated_session(session_id, request) + session = await get_validated_session(session_id, request) from unshackle.core.api.session_store import get_session_store @@ -2758,7 +2927,7 @@ async def session_info_handler(session_id: str, request: Optional[web.Request] = "session_id": session.session_id, "service": session.service_tag, "valid": True, - "expires_in": get_session_store()._ttl, + "expires_in": get_session_store().ttl, "track_count": len(session.tracks), "title_count": len(session.title_map), } @@ -2773,7 +2942,7 @@ async def session_delete_handler(session_id: str, request: Optional[web.Request] from unshackle.core.api.session_store import get_session_store from unshackle.core.config import config as app_config - session = await _get_validated_session(session_id, request) + session = await get_validated_session(session_id, request) store = get_session_store() if session.input_bridge: @@ -2788,7 +2957,7 @@ async def session_delete_handler(session_id: str, request: Optional[web.Request] if not f.stem.startswith("titles_"): try: cache_data[f.stem] = base64.b64encode(zlib.compress(f.read_bytes())).decode("ascii") - except Exception: + except OSError: pass await store.delete(session_id) diff --git a/unshackle/core/api/routes.py b/unshackle/core/api/routes.py index 3af50bf4..9a841931 100644 --- a/unshackle/core/api/routes.py +++ b/unshackle/core/api/routes.py @@ -11,6 +11,8 @@ from aiohttp_swagger3 import SwaggerDocs, SwaggerInfo, SwaggerUiSettings from unshackle.core import __code_hash__, __version__ from unshackle.core.api.errors import APIError, APIErrorCode, build_error_response, handle_api_exception from unshackle.core.api.handlers import ( + CORS_HEADERS, + JOB_EVENTS_ROUTE, cancel_download_job_handler, clear_cache_handler, clear_finished_download_jobs_handler, @@ -18,6 +20,7 @@ from unshackle.core.api.handlers import ( delete_history_handler, download_handler, download_history_handler, + download_job_events_handler, env_check_handler, get_allowed_services, get_download_job_handler, @@ -56,26 +59,22 @@ async def cors_middleware( else: response = await handler(request) - # Add CORS headers - response.headers["Access-Control-Allow-Origin"] = "*" - response.headers["Access-Control-Allow-Methods"] = "GET, POST, PUT, DELETE, OPTIONS" - response.headers["Access-Control-Allow-Headers"] = "Content-Type, X-Secret-Key, Authorization" - response.headers["Access-Control-Max-Age"] = "3600" + response.headers.update(CORS_HEADERS) return response log = logging.getLogger("api") -# Route handler signature: takes the request, returns a response. -Handler = Callable[[web.Request], Awaitable[web.Response]] +# Route handler signature: takes the request, returns a response (streamed or complete). +Handler = Callable[[web.Request], Awaitable[web.StreamResponse]] def api_handler(handler: Handler) -> Handler: """Wrap a route handler so any raised APIError becomes a structured error response.""" @functools.wraps(handler) - async def wrapper(request: web.Request) -> web.Response: + async def wrapper(request: web.Request) -> web.StreamResponse: try: return await handler(request) except APIError as e: @@ -665,22 +664,22 @@ async def download(request: web.Request) -> web.Response: type: array items: type: string - description: Language for video and audio (use 'orig' for original) (default - ["orig"]) + description: Language for video and audio (use 'orig' for original; a '-' prefix excludes, e.g. ["all", "-es"]) (default - ["orig"]) v_lang: type: array items: type: string - description: Language for video tracks only (default - []) + description: Language for video tracks only (a '-' prefix excludes) (default - []) a_lang: type: array items: type: string - description: Language for audio tracks only (default - []) + description: Language for audio tracks only (a '-' prefix excludes) (default - []) s_lang: type: array items: type: string - description: Language for subtitle tracks (default - ["all"]) + description: Language for subtitle tracks (a '-' prefix excludes, e.g. ["all", "-es"]) (default - ["all"]) require_subs: type: array items: @@ -693,7 +692,7 @@ async def download(request: web.Request) -> web.Response: type: array items: type: string - description: Languages wanted for forced subtitles, implies forced_subs (default - []) + description: Languages wanted for forced subtitles, implies forced_subs (a '-' prefix excludes) (default - []) exact_lang: type: boolean description: Use exact language matching (no variants) (default - false) @@ -758,13 +757,18 @@ async def download(request: web.Request) -> web.Response: description: Set the group tag to be used (default - None) tmdb_id: type: integer - description: Use this TMDB ID for tagging (default - None) - animeapi_id: - type: string - description: Anime database ID via AnimeAPI, e.g. mal:12345 (default - None) + description: Use this TMDB ID for tagging instead of a title search. Set enrich to also take its title, year and original language. Mutually exclusive with imdb_id and tvdb_id. Needs tmdb_api_key (default - None) + anilist_id: + oneOf: + - type: integer + - type: string + description: AniList ID for tagging instead of a title search, or a MyAnimeList ID as the string mal:21. Combines with one of tmdb_id, imdb_id and tvdb_id, which AniList does not know (default - None) enrich: type: boolean - description: Override show title and year from external source (default - false) + description: Overwrite show title, year and original language with the external source's. Requires one of tmdb_id, imdb_id, tvdb_id or anilist_id (default - false) + daily: + type: boolean + description: Treat the title as daily/date-based content and fill missing air dates from TVDB. Needs enrich (default - false) no_folder: type: boolean description: Disable folder creation for TV shows (default - false) @@ -791,10 +795,10 @@ async def download(request: web.Request) -> web.Response: description: Add REPACK tag to the output filename (default - false) imdb_id: type: string - description: Use this IMDB ID (e.g. tt1375666) for tagging (default - None) + description: Use this IMDB ID (e.g. tt1375666) for tagging instead of a title search. Set enrich to also take its title, year and original language. Mutually exclusive with tmdb_id and tvdb_id (default - None) tvdb_id: type: integer - description: Use this TVDB ID for tagging and episode ordering (default - None) + description: Use this TVDB ID for tagging and episode ordering instead of a series lookup. Set enrich to also take its title, year and original language. Mutually exclusive with tmdb_id and imdb_id. Needs tvdb_api_key (default - None) tvdb_order: type: string enum: [official, dvd, absolute, alternate, regional] @@ -951,6 +955,46 @@ async def download_job_detail(request: web.Request) -> web.Response: return await get_download_job_handler(job_id, request) +@api_handler +async def download_job_events(request: web.Request) -> web.StreamResponse: + """ + Stream download job events. + --- + summary: Stream download job events + description: > + Server-Sent Events stream of a job's progress. Emits a `snapshot` event, then + `progress` and `status` events, and closes after the terminal `completed`, `failed` + or `cancelled` event. Every event carries the same full job object that + GET /api/download/jobs/{job_id} returns. A browser EventSource can authenticate with + the `secret_key` query parameter instead of the X-Secret-Key header; when both are + sent the header is used. + parameters: + - name: job_id + in: path + required: true + schema: + type: string + - name: secret_key + in: query + required: false + schema: + type: string + responses: + '200': + description: Event stream + content: + text/event-stream: + schema: + type: string + '404': + description: Job not found + '500': + description: Server error + """ + job_id = request.match_info["job_id"] + return await download_job_events_handler(job_id, request) + + @api_handler async def cancel_download_job(request: web.Request) -> web.Response: """ @@ -1796,6 +1840,7 @@ ROUTES: list[tuple[str, str, Handler, bool]] = [ ("GET", "/api/download/jobs", download_jobs, False), ("POST", "/api/download/jobs/clear-finished", clear_finished_download_jobs, False), ("GET", "/api/download/jobs/{job_id}", download_job_detail, False), + ("GET", JOB_EVENTS_ROUTE, download_job_events, False), ("DELETE", "/api/download/jobs/{job_id}", cancel_download_job, False), ("POST", "/api/download/jobs/{job_id}/retry", retry_download_job, False), ("POST", "/api/download/jobs/{job_id}/priority", prioritize_download_job, False), diff --git a/unshackle/core/api/session_store.py b/unshackle/core/api/session_store.py index 5d990b60..726cb0c7 100644 --- a/unshackle/core/api/session_store.py +++ b/unshackle/core/api/session_store.py @@ -57,12 +57,12 @@ class SessionStore: self._cleanup_task: Optional[asyncio.Task] = None @property - def _ttl(self) -> int: + def ttl(self) -> int: """Session TTL in seconds from config.""" return config.serve.get("session_ttl", 300) # 5 min default @property - def _max_sessions(self) -> int: + def max_sessions(self) -> int: """Max concurrent sessions from config.""" return config.serve.get("max_sessions", 100) @@ -74,9 +74,9 @@ class SessionStore: ) -> SessionEntry: """Create a new session with an authenticated service instance.""" async with self._lock: - if len(self._sessions) >= self._max_sessions: + if len(self._sessions) >= self.max_sessions: oldest_id = min(self._sessions, key=lambda k: self._sessions[k].last_accessed) - log.warning(f"Max sessions reached ({self._max_sessions}), evicting oldest: {oldest_id}") + log.warning(f"Max sessions reached ({self.max_sessions}), evicting oldest: {oldest_id}") del self._sessions[oldest_id] session_id = session_id or str(uuid.uuid4()) @@ -98,8 +98,8 @@ class SessionStore: if entry.auth_status not in (AuthStatus.AUTHENTICATING, AuthStatus.PENDING_INPUT): elapsed = (datetime.now(timezone.utc) - entry.last_accessed).total_seconds() - if elapsed > self._ttl: - log.info(f"Session {sanitize_log(session_id)} expired (elapsed={elapsed:.0f}s, ttl={self._ttl}s)") + if elapsed > self.ttl: + log.info(f"Session {sanitize_log(session_id)} expired (elapsed={elapsed:.0f}s, ttl={self.ttl}s)") del self._sessions[session_id] return None @@ -113,7 +113,7 @@ class SessionStore: if entry: if entry.input_bridge: entry.input_bridge.cancel() - self._cleanup_cache_dir(entry.cache_tag) + self.cleanup_cache_dir(entry.cache_tag) log.info(f"Deleted session {sanitize_log(session_id)}") return True return False @@ -128,13 +128,13 @@ class SessionStore: if entry.auth_status in (AuthStatus.AUTHENTICATING, AuthStatus.PENDING_INPUT): if elapsed > AUTH_INPUT_TIMEOUT: expired.append(sid) - elif elapsed > self._ttl: + elif elapsed > self.ttl: expired.append(sid) for sid in expired: entry = self._sessions.pop(sid) if entry.input_bridge: entry.input_bridge.cancel() - self._cleanup_cache_dir(entry.cache_tag) + self.cleanup_cache_dir(entry.cache_tag) if expired: log.info(f"Cleaned up {len(expired)} expired sessions") return len(expired) @@ -144,7 +144,7 @@ class SessionStore: if self._cleanup_task is not None: return - async def _loop() -> None: + async def loop() -> None: while True: await asyncio.sleep(60) # Check every minute try: @@ -152,7 +152,7 @@ class SessionStore: except Exception: log.exception("Error during session cleanup") - self._cleanup_task = asyncio.create_task(_loop()) + self._cleanup_task = asyncio.create_task(loop()) log.info("Session cleanup loop started") async def stop_cleanup_loop(self) -> None: @@ -172,7 +172,7 @@ class SessionStore: log.info(f"Cancelled bridges for {count} active session(s)") @staticmethod - def _cleanup_cache_dir(cache_tag: Optional[str]) -> None: + def cleanup_cache_dir(cache_tag: Optional[str]) -> None: """Remove session cache directory and empty parents.""" if not cache_tag: return @@ -190,7 +190,7 @@ class SessionStore: try: if parent.is_dir() and not any(parent.iterdir()): parent.rmdir() - except Exception: + except OSError: break @property @@ -200,12 +200,12 @@ class SessionStore: # Singleton instance -_session_store: Optional[SessionStore] = None +session_store: Optional[SessionStore] = None def get_session_store() -> SessionStore: """Get or create the global session store singleton.""" - global _session_store - if _session_store is None: - _session_store = SessionStore() - return _session_store + global session_store + if session_store is None: + session_store = SessionStore() + return session_store diff --git a/unshackle/core/cdm/custom_remote_cdm.py b/unshackle/core/cdm/custom_remote_cdm.py index 5e7562c7..161c2d97 100644 --- a/unshackle/core/cdm/custom_remote_cdm.py +++ b/unshackle/core/cdm/custom_remote_cdm.py @@ -158,7 +158,7 @@ class CustomRemoteCDM: # Get device type enum for compatibility if self.device_type_str: - self.device_type = self._get_device_type_enum(self.device_type_str) + self.device_type = self.get_device_type_enum(self.device_type_str) # Authentication configuration self.auth_config = auth or {"type": "header", "header_name": "Authorization", "key": ""} @@ -229,7 +229,7 @@ class CustomRemoteCDM: if custom_headers: self._http_session.headers.update(custom_headers) - def _get_device_type_enum(self, device_type: str): + def get_device_type_enum(self, device_type: str): """Convert device type string to enum for compatibility.""" device_type_upper = device_type.upper() if device_type_upper == "ANDROID": @@ -276,11 +276,11 @@ class CustomRemoteCDM: else: self._required_kids.append(str(kid).replace("-", "").lower()) - def _generate_session_id(self) -> bytes: + def generate_session_id(self) -> bytes: """Generate a unique session ID.""" return secrets.token_bytes(16) - def _get_init_data_from_pssh(self, pssh: Any) -> str: + def get_init_data_from_pssh(self, pssh: Any) -> str: """Extract init data from various PSSH formats.""" if self.is_playready and self._pssh_b64: return self._pssh_b64 @@ -292,7 +292,7 @@ class CustomRemoteCDM: try: base64.b64decode(dumps_result) return dumps_result - except Exception: + except ValueError: return base64.b64encode(dumps_result.encode("utf-8")).decode("utf-8") else: return base64.b64encode(dumps_result).decode("utf-8") @@ -314,7 +314,7 @@ class CustomRemoteCDM: else: raise ValueError(f"Unsupported PSSH type: {type(pssh)}") - def _get_nested_field(self, data: Dict[str, Any], field_path: str, default: Any = None) -> Any: + def get_nested_field(self, data: Dict[str, Any], field_path: str, default: Any = None) -> Any: """ Get a nested field from a dictionary using dot notation. @@ -327,8 +327,8 @@ class CustomRemoteCDM: Field value or default Examples: - _get_nested_field({"data": {"keys": [1,2,3]}}, "data.keys") -> [1,2,3] - _get_nested_field({"message": "success"}, "message") -> "success" + get_nested_field({"data": {"keys": [1,2,3]}}, "data.keys") -> [1,2,3] + get_nested_field({"message": "success"}, "message") -> "success" """ if not field_path: return default @@ -344,7 +344,7 @@ class CustomRemoteCDM: return current - def _apply_transform(self, value: Any, transform_type: str) -> Any: + def apply_transform(self, value: Any, transform_type: str) -> Any: """ Apply a transformation to a value. @@ -415,7 +415,7 @@ class CustomRemoteCDM: # Unknown transform type - return value unchanged return value - def _evaluate_condition(self, condition: str, context: Dict[str, Any]) -> bool: + def evaluate_condition(self, condition: str, context: Dict[str, Any]) -> bool: """ Evaluate a simple condition against a context. @@ -438,23 +438,23 @@ class CustomRemoteCDM: # Check for existence if " exists" in condition: field = condition.replace(" exists", "").strip() - return self._get_nested_field(context, field) is not None + return self.get_nested_field(context, field) is not None # Check for null comparisons if " == null" in condition: field = condition.replace(" == null", "").strip() - return self._get_nested_field(context, field) is None + return self.get_nested_field(context, field) is None if " != null" in condition: field = condition.replace(" != null", "").strip() - return self._get_nested_field(context, field) is not None + return self.get_nested_field(context, field) is not None # Check for equality if " == " in condition: parts = condition.split(" == ", 1) field = parts[0].strip() expected_value = parts[1].strip().strip("'\"") - actual_value = self._get_nested_field(context, field) + actual_value = self.get_nested_field(context, field) return str(actual_value) == expected_value # Check for inequality @@ -462,13 +462,13 @@ class CustomRemoteCDM: parts = condition.split(" != ", 1) field = parts[0].strip() expected_value = parts[1].strip().strip("'\"") - actual_value = self._get_nested_field(context, field) + actual_value = self.get_nested_field(context, field) return str(actual_value) != expected_value # Unknown condition format - return False return False - def _build_request_params( + def build_request_params( self, endpoint_name: str, base_params: Dict[str, Any], session: Optional[Dict[str, Any]] = None ) -> Dict[str, Any]: """ @@ -523,7 +523,7 @@ class CustomRemoteCDM: if session: context.update(session) - if self._evaluate_condition(condition, context): + if self.evaluate_condition(condition, context): params.update(condition_block.get("params", {})) # 4. Apply parameter transforms @@ -532,7 +532,7 @@ class CustomRemoteCDM: param_name = transform.get("param") transform_type = transform.get("type") if param_name in params: - params[param_name] = self._apply_transform(params[param_name], transform_type) + params[param_name] = self.apply_transform(params[param_name], transform_type) # 5. Handle nested parameter structure nested_params = mapping_config.get("nested_params", {}) @@ -552,7 +552,7 @@ class CustomRemoteCDM: return params - def _apply_authentication(self, session: Session) -> None: + def apply_authentication(self, session: Session) -> None: """ Apply authentication to the HTTP session based on auth configuration. @@ -587,7 +587,7 @@ class CustomRemoteCDM: session.auth = HTTPBasicAuth(username, password) - def _parse_response_data(self, endpoint_name: str, response_data: Dict[str, Any]) -> Dict[str, Any]: + def parse_response_data(self, endpoint_name: str, response_data: Dict[str, Any]) -> Dict[str, Any]: """ Parse response data based on response mapping configuration. @@ -609,7 +609,7 @@ class CustomRemoteCDM: parsed = {} for standard_name, field_path in fields_config.items(): - value = self._get_nested_field(response_data, field_path) + value = self.get_nested_field(response_data, field_path) if value is not None: parsed[standard_name] = value @@ -619,13 +619,13 @@ class CustomRemoteCDM: field_name = transform.get("field") transform_type = transform.get("type") if field_name in parsed: - parsed[field_name] = self._apply_transform(parsed[field_name], transform_type) + parsed[field_name] = self.apply_transform(parsed[field_name], transform_type) # Determine response type response_types = mapping_config.get("response_types", []) for response_type_config in response_types: condition = response_type_config.get("condition", "") - if self._evaluate_condition(condition, parsed): + if self.evaluate_condition(condition, parsed): parsed["_response_type"] = response_type_config.get("type") break @@ -633,7 +633,7 @@ class CustomRemoteCDM: success_conditions = mapping_config.get("success_conditions", []) is_success = True if success_conditions: - is_success = all(self._evaluate_condition(cond, parsed) for cond in success_conditions) + is_success = all(self.evaluate_condition(cond, parsed) for cond in success_conditions) parsed["_is_success"] = is_success # Extract error messages if not successful @@ -641,14 +641,14 @@ class CustomRemoteCDM: error_fields = mapping_config.get("error_fields", ["error", "message", "details"]) error_messages = [] for error_field in error_fields: - error_msg = self._get_nested_field(response_data, error_field) + error_msg = self.get_nested_field(response_data, error_field) if error_msg and error_msg not in error_messages: error_messages.append(str(error_msg)) parsed["_error_message"] = " - ".join(error_messages) if error_messages else "Unknown error" return parsed - def _parse_keys_from_response(self, endpoint_name: str, response_data: Dict[str, Any]) -> List[Dict[str, Any]]: + def parse_keys_from_response(self, endpoint_name: str, response_data: Dict[str, Any]) -> List[Dict[str, Any]]: """ Parse keys from response data using key field mapping. @@ -677,7 +677,7 @@ class CustomRemoteCDM: # Handle string format keys (e.g., "kid:key" format) elif isinstance(keys_data, str): - keys = self._apply_transform(keys_data, "parse_key_string") + keys = self.apply_transform(keys_data, "parse_key_string") return keys @@ -688,7 +688,7 @@ class CustomRemoteCDM: Returns: Session identifier as bytes """ - session_id = self._generate_session_id() + session_id = self.generate_session_id() self._sessions[session_id] = { "service_certificate": None, "keys": [], @@ -817,7 +817,7 @@ class CustomRemoteCDM: session = self._sessions[session_id] session["pssh"] = pssh_or_wrm - init_data = self._get_init_data_from_pssh(pssh_or_wrm) + init_data = self.get_init_data_from_pssh(pssh_or_wrm) # Check vaults for cached keys first if self.use_vaults and self._required_kids: @@ -858,10 +858,10 @@ class CustomRemoteCDM: base_params["service_certificate"] = base64.b64encode(session["service_certificate"]).decode("utf-8") # Transform parameters based on configuration - request_params = self._build_request_params("get_request", base_params, session) + request_params = self.build_request_params("get_request", base_params, session) # Apply authentication - self._apply_authentication(self._http_session) + self.apply_authentication(self._http_session) # Make API request endpoint_config = self.endpoints["get_request"] @@ -875,7 +875,7 @@ class CustomRemoteCDM: # Parse response response_data = response.json() - parsed_response = self._parse_response_data("get_request", response_data) + parsed_response = self.parse_response_data("get_request", response_data) # Check if request was successful if not parsed_response.get("_is_success", False): @@ -887,7 +887,7 @@ class CustomRemoteCDM: # Handle cached keys response if response_type == "cached_keys" or "cached_keys" in parsed_response: - cached_keys = self._parse_keys_from_response("get_request", parsed_response) + cached_keys = self.parse_keys_from_response("get_request", parsed_response) all_available_keys = list(cached_keys) if "vault_keys" in session: @@ -927,7 +927,7 @@ class CustomRemoteCDM: if isinstance(challenge, str): try: challenge = base64.b64decode(challenge) - except Exception: + except ValueError: challenge = challenge.encode("utf-8") session["challenge"] = challenge @@ -976,12 +976,12 @@ class CustomRemoteCDM: else: try: license_message = base64.b64decode(license_message) - except Exception: + except ValueError: license_message = license_message.encode("utf-8") # Build request parameters pssh = session["pssh"] - init_data = self._get_init_data_from_pssh(pssh) + init_data = self.get_init_data_from_pssh(pssh) license_request_b64 = base64.b64encode(session["challenge"]).decode("utf-8") license_response_b64 = base64.b64encode(license_message).decode("utf-8") @@ -994,10 +994,10 @@ class CustomRemoteCDM: } # Transform parameters based on configuration - request_params = self._build_request_params("decrypt_response", base_params, session) + request_params = self.build_request_params("decrypt_response", base_params, session) # Apply authentication - self._apply_authentication(self._http_session) + self.apply_authentication(self._http_session) # Make API request endpoint_config = self.endpoints["decrypt_response"] @@ -1011,7 +1011,7 @@ class CustomRemoteCDM: # Parse response response_data = response.json() - parsed_response = self._parse_response_data("decrypt_response", response_data) + parsed_response = self.parse_response_data("decrypt_response", response_data) # Check if request was successful if not parsed_response.get("_is_success", False): @@ -1019,7 +1019,7 @@ class CustomRemoteCDM: raise requests.RequestException(f"License decrypt error: {error_msg}") # Extract keys from response - license_keys = self._parse_keys_from_response("decrypt_response", parsed_response) + license_keys = self.parse_keys_from_response("decrypt_response", parsed_response) # Combine all keys (vault + cached + license) all_keys = [] diff --git a/unshackle/core/cdm/decrypt_labs_remote_cdm.py b/unshackle/core/cdm/decrypt_labs_remote_cdm.py index 39cf7d33..1f54fa1d 100644 --- a/unshackle/core/cdm/decrypt_labs_remote_cdm.py +++ b/unshackle/core/cdm/decrypt_labs_remote_cdm.py @@ -132,7 +132,7 @@ class DecryptLabsRemoteCDM: self._device_type_str = device_type if device_type: - self.device_type = self._get_device_type_enum(device_type) + self.device_type = self.get_device_type_enum(device_type) self._is_playready = (device_type and device_type.upper() == "PLAYREADY") or ( bool(device_name) and device_name.upper().startswith("SL") @@ -157,7 +157,7 @@ class DecryptLabsRemoteCDM: } ) - def _get_device_type_enum(self, device_type: str): + def get_device_type_enum(self, device_type: str): """Convert device type string to enum for compatibility.""" device_type_upper = device_type.upper() if device_type_upper == "ANDROID": @@ -204,11 +204,11 @@ class DecryptLabsRemoteCDM: else: self._required_kids.append(str(kid).replace("-", "").lower()) - def _generate_session_id(self) -> bytes: + def generate_session_id(self) -> bytes: """Generate a unique session ID.""" return secrets.token_bytes(16) - def _get_init_data_from_pssh(self, pssh: Any) -> str: + def get_init_data_from_pssh(self, pssh: Any) -> str: """Extract init data from various PSSH formats.""" if self.is_playready and self._pssh_b64: return self._pssh_b64 @@ -220,7 +220,7 @@ class DecryptLabsRemoteCDM: try: base64.b64decode(dumps_result) return dumps_result - except Exception: + except ValueError: return base64.b64encode(dumps_result.encode("utf-8")).decode("utf-8") else: return base64.b64encode(dumps_result).decode("utf-8") @@ -249,7 +249,7 @@ class DecryptLabsRemoteCDM: Returns: Session identifier as bytes """ - session_id = self._generate_session_id() + session_id = self.generate_session_id() self._sessions[session_id] = { "service_certificate": None, "keys": [], @@ -395,7 +395,7 @@ class DecryptLabsRemoteCDM: session = self._sessions[session_id] session["pssh"] = pssh_or_wrm - init_data = self._get_init_data_from_pssh(pssh_or_wrm) + init_data = self.get_init_data_from_pssh(pssh_or_wrm) already_tried_cache = session.get("tried_cache", False) if self.vaults and self._required_kids: @@ -470,7 +470,7 @@ class DecryptLabsRemoteCDM: for missing keys. """ cached_keys = data.get("cached_keys", []) - parsed_keys = self._parse_cached_keys(cached_keys) + parsed_keys = self.parse_cached_keys(cached_keys) all_available_keys = list(parsed_keys) if "vault_keys" in session: @@ -588,11 +588,11 @@ class DecryptLabsRemoteCDM: else: try: license_message = base64.b64decode(license_message) - except Exception: + except ValueError: license_message = license_message.encode("utf-8") pssh = session["pssh"] - init_data = self._get_init_data_from_pssh(pssh) + init_data = self.get_init_data_from_pssh(pssh) license_request_b64 = base64.b64encode(session["challenge"]).decode("utf-8") license_response_b64 = base64.b64encode(license_message).decode("utf-8") @@ -620,7 +620,7 @@ class DecryptLabsRemoteCDM: error_msg += f" - Details: {data['details']}" raise requests.RequestException(f"License decrypt error: {error_msg}") - license_keys = self._parse_keys_response(data) + license_keys = self.parse_keys_response(data) all_keys = [] @@ -705,7 +705,7 @@ class DecryptLabsRemoteCDM: return keys - def _parse_cached_keys(self, cached_keys_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + def parse_cached_keys(self, cached_keys_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]: """Parse cached keys from API response. Args: @@ -721,11 +721,12 @@ class DecryptLabsRemoteCDM: for key_data in cached_keys_data: if "kid" in key_data and "key" in key_data: keys.append({"kid": key_data["kid"], "key": key_data["key"], "type": "CONTENT"}) - except Exception: + # non-dict entries in the API payload raise TypeError on membership/index + except TypeError: pass return keys - def _parse_keys_response(self, data: Dict[str, Any]) -> List[Dict[str, Any]]: + def parse_keys_response(self, data: Dict[str, Any]) -> List[Dict[str, Any]]: """Parse keys from decrypt response.""" keys = [] diff --git a/unshackle/core/cdm/loader.py b/unshackle/core/cdm/loader.py index be66c428..cf97672d 100644 --- a/unshackle/core/cdm/loader.py +++ b/unshackle/core/cdm/loader.py @@ -13,7 +13,7 @@ from typing import Any, Optional log = logging.getLogger("cdm") -def _stamp_remote(cdm: Any, drm: str) -> Any: +def stamp_remote(cdm: Any, drm: str) -> Any: """Record a remote CDM's DRM type and mark it as remote. The loader knows the DRM type from config, so it sets it here as the value @@ -49,12 +49,12 @@ def load_cdm( cdm_api = next(iter(x.copy() for x in config.remote_cdm if x["name"] == cdm_name), None) if cdm_api: - return _load_remote_cdm(cdm_api, cdm_name, service_name, vaults) + return load_remote_cdm(cdm_api, cdm_name, service_name, vaults) - return _load_local_cdm(cdm_name) + return load_local_cdm(cdm_name) -def _load_remote_cdm( +def load_remote_cdm( cdm_api: dict, cdm_name: str, service_name: str, @@ -80,7 +80,7 @@ def _load_remote_cdm( ) cdm: Any = DecryptLabsRemoteCDM(service_name=service_name, vaults=vaults, **cdm_api) - return _stamp_remote(cdm, "playready" if cdm.is_playready else "widevine") + return stamp_remote(cdm, "playready" if cdm.is_playready else "widevine") if cdm_type == "custom_api": from unshackle.core.cdm.custom_remote_cdm import CustomRemoteCDM @@ -88,13 +88,13 @@ def _load_remote_cdm( del cdm_api["name"] del cdm_api["type"] cdm = CustomRemoteCDM(service_name=service_name, vaults=vaults, **cdm_api) - return _stamp_remote(cdm, "playready" if cdm.is_playready else "widevine") + return stamp_remote(cdm, "playready" if cdm.is_playready else "widevine") device_type = cdm_api.get("Device Type", cdm_api.get("device_type", "")) if str(device_type).upper() == "PLAYREADY": from pyplayready.remote.remotecdm import RemoteCdm as PlayReadyRemoteCdm - return _stamp_remote( + return stamp_remote( PlayReadyRemoteCdm( security_level=cdm_api.get("Security Level", cdm_api.get("security_level", 3000)), host=cdm_api.get("Host", cdm_api.get("host")), @@ -106,7 +106,7 @@ def _load_remote_cdm( from pywidevine.remotecdm import RemoteCdm - return _stamp_remote( + return stamp_remote( RemoteCdm( device_type=cdm_api.get("Device Type", cdm_api.get("device_type", "")), system_id=cdm_api.get("System ID", cdm_api.get("system_id", "")), @@ -119,7 +119,7 @@ def _load_remote_cdm( ) -def _load_local_cdm(cdm_name: str) -> Any: +def load_local_cdm(cdm_name: str) -> Any: """Instantiate a local CDM from a .prd or .wvd file.""" from unshackle.core.config import config diff --git a/unshackle/core/cdm/monalisa/monalisa_cdm.py b/unshackle/core/cdm/monalisa/monalisa_cdm.py index ec9214e9..87842cfb 100644 --- a/unshackle/core/cdm/monalisa/monalisa_cdm.py +++ b/unshackle/core/cdm/monalisa/monalisa_cdm.py @@ -115,8 +115,8 @@ class MonaLisaCDM: memory_type = wasmtime.MemoryType(wasmtime.Limits(256, 256)) self.memory = wasmtime.Memory(self.store, memory_type) - self._write_i32(self.DYNAMICTOP_PTR, self.DYNAMIC_BASE) - imports = self._build_imports() + self.write_i32(self.DYNAMICTOP_PTR, self.DYNAMIC_BASE) + imports = self.build_imports() self.instance = wasmtime.Instance(self.store, self.module, imports) ex = self.instance.exports(self.store) @@ -140,7 +140,7 @@ class MonaLisaCDM: # Treat 0/negative/non-int-like values as allocation failure. try: ctx_int = int(ctx) - except Exception: + except (TypeError, ValueError): ctx_int = None if ctx_int is None or ctx_int <= 0: @@ -193,7 +193,7 @@ class MonaLisaCDM: else: license_b64 = license_data - ret = self._ccall( + ret = self.ccall( "monalisa_set_license", int, self.ctx, @@ -205,7 +205,7 @@ class MonaLisaCDM: if ret != 0: raise RuntimeError(f"License validation failed with code: {ret}") - key_bytes = self._extract_license_key_bytes() + key_bytes = self.extract_license_key_bytes() # Extract DCID from license to generate KID try: @@ -225,7 +225,7 @@ class MonaLisaCDM: # No DCID in the license: derive a deterministic per-license KID to avoid collisions. try: license_raw = base64.b64decode(license_b64) - except Exception: + except ValueError: license_raw = license_b64.encode("utf-8", errors="replace") license_hash = hashlib.sha256(license_raw).hexdigest() @@ -233,7 +233,7 @@ class MonaLisaCDM: return {"kid": kid_bytes.hex(), "key": key_bytes.hex(), "type": "CONTENT"} - def _extract_license_key_bytes(self) -> bytes: + def extract_license_key_bytes(self) -> bytes: """Extract the 16-byte decryption key from WASM memory.""" data_ptr = self.memory.data_ptr(self.store) data_len = self.memory.data_len(self.store) @@ -247,7 +247,7 @@ class MonaLisaCDM: return bytes(mem_ptr.contents[start:end]) - def _ccall(self, func_name: str, return_type: type, *args): + def ccall(self, func_name: str, return_type: type, *args): """Call a WASM function with automatic string conversion.""" stack = 0 converted_args = [] @@ -259,7 +259,7 @@ class MonaLisaCDM: stack = self.exports["stackSave"](self.store) max_length = (len(arg) << 2) + 1 ptr = self.exports["stackAlloc"](self.store, max_length) - self._string_to_utf8(arg, ptr, max_length) + self.string_to_utf8(arg, ptr, max_length) converted_args.append(ptr) else: converted_args.append(arg) @@ -280,7 +280,7 @@ class MonaLisaCDM: return bool(result) return result - def _write_i32(self, addr: int, value: int) -> None: + def write_i32(self, addr: int, value: int) -> None: """Write a 32-bit integer to WASM memory.""" if addr % 4 != 0: raise ValueError(f"Unaligned i32 write: addr={addr} (must be 4-byte aligned)") @@ -293,7 +293,7 @@ class MonaLisaCDM: mem_ptr = ctypes.cast(data, ctypes.POINTER(ctypes.c_int32)) mem_ptr[addr >> 2] = value - def _string_to_utf8(self, data: str, ptr: int, max_length: int) -> int: + def string_to_utf8(self, data: str, ptr: int, max_length: int) -> int: """Convert string to UTF-8 and write to WASM memory.""" encoded = data.encode("utf-8") write_length = min(len(encoded), max_length - 1) @@ -306,7 +306,7 @@ class MonaLisaCDM: mem_ptr[ptr + write_length] = 0 return write_length - def _write_ascii_to_memory(self, string: str, buffer: int, dont_add_null: int = 0) -> None: + def write_ascii_to_memory(self, string: str, buffer: int, dont_add_null: int = 0) -> None: """Write ASCII string to WASM memory.""" mem_data = self.memory.data_ptr(self.store) mem_ptr = ctypes.cast(mem_data, ctypes.POINTER(ctypes.c_ubyte)) @@ -318,7 +318,7 @@ class MonaLisaCDM: if dont_add_null == 0: mem_ptr[buffer + len(encoded)] = 0 - def _build_imports(self): + def build_imports(self): """Build the WASM import stubs required by the MonaLisa module.""" def sys_fcntl64(a, b, c): @@ -381,15 +381,15 @@ class MonaLisaCDM: buf_size = 0 for index, string in enumerate(self.ENV_STRINGS): ptr = environ_buf + buf_size - self._write_i32(environ_ptr + index * 4, ptr) - self._write_ascii_to_memory(string, ptr) + self.write_i32(environ_ptr + index * 4, ptr) + self.write_ascii_to_memory(string, ptr) buf_size += len(string) + 1 return 0 def environ_sizes_get(penviron_count, penviron_buf_size): - self._write_i32(penviron_count, len(self.ENV_STRINGS)) + self.write_i32(penviron_count, len(self.ENV_STRINGS)) buf_size = sum(len(s) + 1 for s in self.ENV_STRINGS) - self._write_i32(penviron_buf_size, buf_size) + self.write_i32(penviron_buf_size, buf_size) return 0 i32 = wasmtime.ValType.i32() diff --git a/unshackle/core/commands.py b/unshackle/core/commands.py index 24bb7d2b..459f6143 100644 --- a/unshackle/core/commands.py +++ b/unshackle/core/commands.py @@ -9,7 +9,7 @@ from unshackle.core.utilities import import_module_by_path log = logging.getLogger("commands") -_COMMANDS = sorted( +COMMANDS = sorted( (path for path in config.directories.commands.glob("*.py") if path.stem.lower() != "__init__"), key=lambda x: x.stem ) @@ -23,12 +23,12 @@ def load_command(path: Path) -> object: try: module = import_module_by_path(path) except Exception as e: - raise RuntimeError(f"{path.stem}: failed to import — {type(e).__name__}: {e} ({path})") from e + raise RuntimeError(f"{path.stem}: failed to import - {type(e).__name__}: {e} ({path})") from e try: return getattr(module, path.stem) except AttributeError as e: raise RuntimeError( - f"{path.stem}: no object named '{path.stem}' found in {path} — it must match the filename" + f"{path.stem}: no object named '{path.stem}' found in {path} - it must match the filename" ) from e @@ -49,7 +49,7 @@ def load_commands(paths: list[Path]) -> tuple[dict[str, object], list[str]]: return modules, errors -_MODULES, LOAD_ERRORS = load_commands(_COMMANDS) +MODULES, LOAD_ERRORS = load_commands(COMMANDS) def check_load_errors() -> None: @@ -65,19 +65,18 @@ class Commands(click.Group): def list_commands(self, ctx: click.Context) -> list[str]: """Returns a list of command names from the command filenames.""" check_load_errors() - return [x.stem.replace("_", "-") for x in _COMMANDS] + return [x.stem.replace("_", "-") for x in COMMANDS] def get_command(self, ctx: click.Context, name: str) -> Optional[click.Command]: """Load the command code and return the main click command function.""" check_load_errors() - module = _MODULES.get(name) or _MODULES.get(name.replace("-", "_")) + module = MODULES.get(name) or MODULES.get(name.replace("-", "_")) if not module: raise click.ClickException(f"Unable to find command by the name '{name}'") - if hasattr(module, "cli"): - return module.cli - - return module + cmd = module.cli if hasattr(module, "cli") else module + cmd.name = name + return cmd # Hide direct access to commands from quick import form, they shouldn't be accessed directly diff --git a/unshackle/core/config.py b/unshackle/core/config.py index adc7e9f4..3a6cea21 100644 --- a/unshackle/core/config.py +++ b/unshackle/core/config.py @@ -3,7 +3,7 @@ from __future__ import annotations import re import warnings from pathlib import Path -from typing import Any, Optional +from typing import Any, Optional, Union import yaml from appdirs import AppDirs @@ -110,17 +110,19 @@ class Config: self.decryption_map = {} self.decryption = decryption_cfg or "shaka" + self.theme: str = kwargs.get("theme") or "default" self.set_terminal_bg: bool = kwargs.get("set_terminal_bg", False) self.tag: str = kwargs.get("tag") or "" self.tag_group_name: bool = kwargs.get("tag_group_name", True) self.tag_imdb_tmdb: bool = kwargs.get("tag_imdb_tmdb", True) - self.imdb_api_enabled: bool = kwargs.get("imdb_api_enabled", False) self.omdb_api_key: str = kwargs.get("omdb_api_key") or "" self.tmdb_api_key: str = kwargs.get("tmdb_api_key") or "" self.tvdb_api_key: str = kwargs.get("tvdb_api_key") or "" self.tvdb_pin: str = kwargs.get("tvdb_pin") or "" self.tvdb_order: str = (kwargs.get("tvdb_order") or "").lower() - self.metadata_providers: list = kwargs.get("metadata_providers") or [] + self.metadata_providers: Union[list, dict] = kwargs.get("metadata_providers") or [] + self.anilist_title_language: str = (kwargs.get("anilist_title_language") or "english").lower() + self.disable_metadata: bool = kwargs.get("disable_metadata", False) self.simkl_client_id: str = kwargs.get("simkl_client_id") or "" self.decrypt_labs_api_key: str = kwargs.get("decrypt_labs_api_key") or "" self.ipinfo_api_key: str = kwargs.get("ipinfo_api_key") or "" @@ -149,7 +151,7 @@ class Config: ) if self.output_template: - self._validate_output_templates() + self.validate_output_templates() self.unicode_filenames: bool = kwargs.get("unicode_filenames", False) @@ -161,7 +163,7 @@ class Config: self.debug_keys: bool = kwargs.get("debug_keys", False) self.debug_requests: bool = kwargs.get("debug_requests", False) - def _validate_output_templates(self) -> None: + def validate_output_templates(self) -> None: """Validate output template configurations and warn about potential issues.""" if not self.output_template: return @@ -174,6 +176,7 @@ class Config: "season_episode", "episode_name", "part", + "absolute", "date", "quality", "resolution", diff --git a/unshackle/core/console.py b/unshackle/core/console.py index 52c0415f..9ba1ae8c 100644 --- a/unshackle/core/console.py +++ b/unshackle/core/console.py @@ -21,6 +21,7 @@ from rich.text import Text, TextType from rich.theme import Theme from unshackle.core.config import config +from unshackle.core.themes import DEFAULT_THEME, PALETTES, apply_help_theme, resolve_palette class ComfyLogRenderer(LogRender): @@ -317,25 +318,16 @@ class ComfyConsole(Console): live.start() -catppuccin_mocha = { - # Colors based on "CatppuccinMocha" from Gogh themes - "bg": "rgb(30,30,46)", - "text": "rgb(205,214,244)", - "text2": "rgb(162,169,193)", # slightly darker - "black": "rgb(69,71,90)", - "bright_black": "rgb(88,91,112)", - "red": "rgb(243,139,168)", - "green": "rgb(166,227,161)", - "yellow": "rgb(249,226,175)", - "blue": "rgb(137,180,250)", - "pink": "rgb(245,194,231)", - "cyan": "rgb(148,226,213)", - "gray": "rgb(166,173,200)", - "bright_gray": "rgb(186,194,222)", - "dark_gray": "rgb(54,54,84)", -} - -primary_scheme = catppuccin_mocha +primary_scheme = resolve_palette(config.theme) +if primary_scheme is None: + logging.getLogger("console").warning( + "Unknown theme %r, falling back to %s (available: %s)", + config.theme, + DEFAULT_THEME, + ", ".join(PALETTES), + ) + primary_scheme = dict(PALETTES[DEFAULT_THEME]) +apply_help_theme(primary_scheme) primary_scheme["none"] = primary_scheme["text"] primary_scheme["grey23"] = primary_scheme["black"] primary_scheme["magenta"] = primary_scheme["pink"] @@ -369,7 +361,7 @@ class SyncLive(Live): class GradientPulseBarColumn(BarColumn): - _LUT_SIZE = 32 + LUT_SIZE = 32 def __init__(self, *args: Any, pulse_period: float = 40.0, pulse_speed: float = 15.0, **kwargs: Any) -> None: super().__init__(*args, **kwargs) @@ -379,7 +371,7 @@ class GradientPulseBarColumn(BarColumn): hi = Color.parse(primary_scheme["pink"]).triplet assert lo and hi self._lut = [ - Style(color=Color.from_triplet(blend_rgb(lo, hi, i / (self._LUT_SIZE - 1)))) for i in range(self._LUT_SIZE) + Style(color=Color.from_triplet(blend_rgb(lo, hi, i / (self.LUT_SIZE - 1)))) for i in range(self.LUT_SIZE) ] def render(self, task: Task) -> RenderableType: # type: ignore[override] @@ -435,4 +427,20 @@ console = ComfyConsole( ) -__all__ = ("ComfyLogRenderer", "ComfyRichHandler", "ComfyConsole", "GradientPulseBarColumn", "SyncLive", "console") +def prompt_user(prompt: str) -> str: + """Ask the user for input on the shared console, themed and indented like the rest of the output.""" + indent = " " * 5 + body = Text(indent + prompt.rstrip("\n ").replace("\n", "\n" + indent), style="text") + body.append("\n" + indent + "> ", style="rule.text") + return console.input(body) + + +__all__ = ( + "ComfyLogRenderer", + "ComfyRichHandler", + "ComfyConsole", + "GradientPulseBarColumn", + "SyncLive", + "console", + "prompt_user", +) diff --git a/unshackle/core/downloaders/requests.py b/unshackle/core/downloaders/requests.py index 79defb71..54d3d7f1 100644 --- a/unshackle/core/downloaders/requests.py +++ b/unshackle/core/downloaders/requests.py @@ -101,8 +101,8 @@ class TokenBucket: DOWNLOAD_CANCELLED.wait(wait) -_speed_limiter: Optional[TokenBucket] = None -_speed_limit_locked = False +speed_limiter: Optional[TokenBucket] = None +speed_limit_locked = False SPEED_UNITS = { "": 1, @@ -152,14 +152,14 @@ def set_speed_limit(bytes_per_sec: Optional[float], lock: bool = False) -> None: A locked limit (serve's global_speed_limit) wins: later unlocked calls, like the per-job one in dl.result(), become no-ops for the process. """ - global _speed_limiter, _speed_limit_locked - if _speed_limit_locked and not lock: + global speed_limiter, speed_limit_locked + if speed_limit_locked and not lock: return - _speed_limit_locked = lock - _speed_limiter = TokenBucket(bytes_per_sec) if bytes_per_sec else None + speed_limit_locked = lock + speed_limiter = TokenBucket(bytes_per_sec) if bytes_per_sec else None -def _adaptive_chunk_size(content_length: int) -> int: +def adaptive_chunk_size(content_length: int) -> int: """Pick chunk size based on content length. Benchmarked sweet spot: 512KB-4MB.""" if content_length <= 0: return DEFAULT_CHUNK @@ -204,7 +204,7 @@ class AdaptiveWorkerController: self._cooldown = False self._ramping = True # slow-start: double per probe until the first plateau or error burst - def _prune(self, now: float) -> None: + def prune(self, now: float) -> None: cutoff = now - self.window while self._samples and self._samples[0][0] < cutoff: self._samples.popleft() @@ -213,14 +213,14 @@ class AdaptiveWorkerController: if self._first_sample is None: self._first_sample = now self._samples.append((now, n)) - self._prune(now) + self.prune(now) def record_error(self, now: float) -> None: self._errors += 1 def speed(self, now: float) -> float: """Rolling bytes/sec over the window.""" - self._prune(now) + self.prune(now) if not self._samples: return 0.0 span = now - self._samples[0][0] @@ -228,7 +228,7 @@ class AdaptiveWorkerController: return 0.0 return sum(n for _, n in self._samples) / span - def _warmed_up(self, now: float) -> bool: + def warmed_up(self, now: float) -> bool: # half a tick of samples gives enough of a baseline to probe against; waiting for # a full speed window would idle the ramp at the start of every download. # measured from the first-ever sample so pruning cannot starve the check @@ -251,7 +251,7 @@ class AdaptiveWorkerController: self._last_tick = now # hold until half a tick of throughput history has accumulated - if not self._warmed_up(now): + if not self.warmed_up(now): self._errors = 0 return self.target @@ -307,7 +307,7 @@ class AdaptiveWorkerController: return self.target -def _retry_sleep(exc: Exception, attempts: int) -> float: +def retry_sleep(exc: Exception, attempts: int) -> float: """Seconds to wait before retry number ``attempts``. download() owns segment retries (session-level retries are suppressed), so backoff @@ -344,19 +344,19 @@ def _retry_sleep(exc: Exception, attempts: int) -> float: return min(backoff + random.uniform(0, backoff * 0.1), MAX_BACKOFF) -def _is_requests_session(session: Any) -> bool: +def is_requests_session(session: Any) -> bool: """Check if the session is a standard requests.Session (supports resp.raw).""" return isinstance(session, Session) -def _is_rnet_session(session: Any) -> bool: +def is_rnet_session(session: Any) -> bool: """Check if the session is an RnetSession (uses resp.stream()).""" from unshackle.core.session import RnetSession return isinstance(session, RnetSession) -def _is_content_encoded(value: Optional[str]) -> bool: +def is_content_encoded(value: Optional[str]) -> bool: """ Check a Content-Encoding value for any coding other than absent or `identity`. @@ -368,10 +368,10 @@ def _is_content_encoded(value: Optional[str]) -> bool: return any(coding.strip() not in ("", "identity") for coding in (value or "").lower().split(",")) -def _probe_ranged(url: str, session: Any, **kwargs: Any) -> tuple[int, bool]: +def probe_ranged(url: str, session: Any, **kwargs: Any) -> tuple[int, bool]: headers = {**(kwargs.get("headers") or {}), "Range": "bytes=0-0"} rest = {k: v for k, v in kwargs.items() if k != "headers"} - if _is_rnet_session(session): + if is_rnet_session(session): rest.setdefault("read_timeout", READ_TIMEOUT) # let download() own segment retries, with no nested session-level retry loop rest.setdefault("max_retries", 0) @@ -384,7 +384,7 @@ def _probe_ranged(url: str, session: Any, **kwargs: Any) -> tuple[int, bool]: try: if resp.status_code != 206: return 0, False - if _is_content_encoded(resp.headers.get("Content-Encoding") or resp.headers.get("content-encoding")): + if is_content_encoded(resp.headers.get("Content-Encoding") or resp.headers.get("content-encoding")): return 0, False content_range = resp.headers.get("Content-Range") or resp.headers.get("content-range") or "" total = content_range.rsplit("/", 1)[-1].strip() @@ -396,7 +396,7 @@ def _probe_ranged(url: str, session: Any, **kwargs: Any) -> tuple[int, bool]: pass -def _has_range_header(item: dict[str, Any]) -> bool: +def has_range_header(item: dict[str, Any]) -> bool: """True when a URL item's per-request headers already carry a Range. Such an item is itself a byte-range slice of a larger resource (DASH SegmentBase, @@ -406,7 +406,7 @@ def _has_range_header(item: dict[str, Any]) -> bool: return any(k.lower() == "range" for k in (item.get("headers") or {})) -def _range_covers_full_body(headers: MutableMapping[str, Any], content_length: int) -> bool: +def range_covers_full_body(headers: MutableMapping[str, Any], content_length: int) -> bool: """True when a 200 body is exactly the bytes the item's Range slice asked for. A server may ignore Range and answer 200 with the whole resource (RFC 9110 @@ -422,7 +422,7 @@ def _range_covers_full_body(headers: MutableMapping[str, Any], content_length: i return start == 0 and content_length == end + 1 -def _force_unblock_stream(stream: Any) -> None: +def force_unblock_stream(stream: Any) -> None: """Best-effort: shut the stream's socket so a thread parked in a blocking read returns now. A superseded hedge loser can sit in a full-chunk ``raw.read`` on a slow connection; the @@ -437,7 +437,7 @@ def _force_unblock_stream(stream: Any) -> None: pass -def _split_ranges(size: int, max_parts: int, part_target: int) -> list[tuple[int, int]]: +def split_ranges(size: int, max_parts: int, part_target: int) -> list[tuple[int, int]]: """Split ``[0, size)`` into inclusive ``[start, end]`` byte ranges with no gaps or overlaps. At most ``max_parts`` parts of ~``part_target`` bytes each. Pure, with no I/O, so testable. @@ -449,7 +449,7 @@ def _split_ranges(size: int, max_parts: int, part_target: int) -> list[tuple[int ] -def _plan_tail_parts(size: int, spare: int) -> list[tuple[int, int]]: +def plan_tail_parts(size: int, spare: int) -> list[tuple[int, int]]: """Plan intra-segment range parts for a tail segment. Returns inclusive ``[start, end]`` byte ranges covering ``[0, size)`` with no gaps or @@ -459,10 +459,10 @@ def _plan_tail_parts(size: int, spare: int) -> list[tuple[int, int]]: """ if size < TAIL_BOOST_MIN_SEGMENT_SIZE or spare < 2: return [] - return _split_ranges(size, spare, TAIL_BOOST_PART_SIZE) + return split_ranges(size, spare, TAIL_BOOST_PART_SIZE) -def _tail_boost_engages(remaining: int, pending: int, target: int) -> bool: +def tail_boost_engages(remaining: int, pending: int, target: int) -> bool: """True when idle workers outnumber the remaining whole segments. That is the tail condition where running one worker per remaining segment would leave @@ -474,7 +474,7 @@ def _tail_boost_engages(remaining: int, pending: int, target: int) -> bool: return remaining > 0 and spare >= 2 and remaining <= spare -def _dispatch_parts( +def dispatch_parts( url: str, save_path: Path, session: Any, @@ -485,7 +485,7 @@ def _dispatch_parts( save_path.parent.mkdir(parents=True, exist_ok=True) control_file = save_path.with_name(f"{save_path.name}.!dev") - parts = _split_ranges(total_size, max_workers, RANGE_PARALLEL_PART_SIZE) + parts = split_ranges(total_size, max_workers, RANGE_PARALLEL_PART_SIZE) control_file.write_bytes(b"") with open(save_path, "wb") as f: @@ -496,7 +496,7 @@ def _dispatch_parts( # DOWNLOAD_CANCELLED (which would kill the sequential fallback and sibling tracks) abort = threading.Event() - def _worker(start: int, end: int) -> None: + def worker(start: int, end: int) -> None: for ev in download( url=url, save_path=save_path, @@ -509,7 +509,7 @@ def _dispatch_parts( events.put(ev) pool = ThreadPoolExecutor(max_workers=len(parts)) - futures = [pool.submit(_worker, s, e) for s, e in parts] + futures = [pool.submit(worker, s, e) for s, e in parts] pending = set(futures) yield {"total": total_size} @@ -637,7 +637,7 @@ def download( register: Optional callback ``(stream, active)`` invoked with active=True while a response is being read and active=False when that read ends. Lets the batch reach a superseded loser's socket to unblock its in-flight read at teardown (see - _force_unblock_stream) instead of waiting the slow read out. + force_unblock_stream) instead of waiting the slow read out. kwargs: Any extra keyword arguments to pass to the session.get() call. Use this for one-time request changes like a header, cookie, or proxy. For example, to request Byte-ranges use e.g., `headers={"Range": "bytes=0-128"}`. @@ -665,11 +665,11 @@ def download( resume_offset = tmp_file.stat().st_size _time = time.time - use_raw = _is_requests_session(session) + use_raw = is_requests_session(session) # item carries its own Range (DASH SegmentBase / HLS EXT-X-BYTERANGE slice): never # overwrite it with a resume Range (that would fetch the parent's tail, not the slice); # retries rewrite the whole slice in "wb" mode instead - item_range = _has_range_header(kwargs) + item_range = has_range_header(kwargs) attempts = 1 written = 0 @@ -685,7 +685,7 @@ def download( last_speed_refresh = _time() try: - use_rnet = _is_rnet_session(session) + use_rnet = is_rnet_session(session) request_kwargs = dict(kwargs) if use_rnet: @@ -697,7 +697,8 @@ def download( req_headers = dict(request_kwargs.get("headers", {}) or {}) # media bytes must arrive unrecoded: transparent CDN (de)compression breaks # Content-Length accounting and is meaningless on ranged requests - req_headers.setdefault("Accept-Encoding", "identity") + if not any(str(k).lower() == "accept-encoding" for k in req_headers): + req_headers["Accept-Encoding"] = "identity" if part_mode: req_headers["Range"] = f"bytes={part_offset + written}-{part_end}" elif resume_offset > 0 and not item_range: @@ -727,12 +728,12 @@ def download( if part_mode and stream.status_code != 206: raise IOError(f"expected 206 for ranged part, got {stream.status_code}") if use_rnet: - content_encoded = _is_content_encoded( + content_encoded = is_content_encoded( stream.headers.get("Content-Encoding") or stream.headers.get("content-encoding") ) content_length = 0 if content_encoded else (stream.content_length or 0) else: - content_encoded = _is_content_encoded(stream.headers.get("Content-Encoding")) + content_encoded = is_content_encoded(stream.headers.get("Content-Encoding")) try: content_length = int(stream.headers.get("Content-Length", "0")) if content_encoded: @@ -745,7 +746,7 @@ def download( # a 200 here); writing that as the segment would silently corrupt the merge, so # fail the attempt unless the slice spans the whole resource, in which case the # 200 body is byte-identical to the 206 and safe to keep - if not (stream.status_code == 200 and _range_covers_full_body(req_headers, content_length)): + if not (stream.status_code == 200 and range_covers_full_body(req_headers, content_length)): raise IOError(f"expected 206 for byte-range segment, got {stream.status_code}") if resumed and content_encoded: @@ -757,8 +758,8 @@ def download( resume_offset = 0 continue - limiter = _speed_limiter - chunk_size = _adaptive_chunk_size(content_length) + limiter = speed_limiter + chunk_size = adaptive_chunk_size(content_length) if limiter: chunk_size = min(chunk_size, max(8192, int(limiter.rate / 4))) total_size = (resume_offset + content_length) if resumed and content_length > 0 else content_length @@ -781,7 +782,7 @@ def download( # offset is derived from the tmp file's size, so preallocating to content_length # would make an interrupted write look fully downloaded and poison the resume # (unsatisfiable Range on retry, silent zero-padded corruption). part_mode still - # pre-truncates in _dispatch_parts because its workers seek to fixed offsets. + # pre-truncates in dispatch_parts because its workers seek to fixed offsets. with open(save_path if part_mode else tmp_file, file_mode, buffering=file_buffering) as f: if part_mode: f.seek(part_offset + written) @@ -894,7 +895,7 @@ def download( pass if not part_mode: resume_offset = tmp_file.stat().st_size if tmp_file.exists() else 0 - delay = _retry_sleep(exc, attempts) + delay = retry_sleep(exc, attempts) if abort is not None: # interruptible nap: backoff can reach MAX_BACKOFF, and teardown always # sets the batch abort, so a parked worker must wake and exit at once @@ -906,7 +907,7 @@ def download( attempts += 1 -def _build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]: +def build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]: """Picklable spec to rebuild ``session`` in a child process; None if not cheaply rebuildable. Live sessions (sockets, TLS state, threads) can't cross a process boundary, so each child @@ -915,14 +916,14 @@ def _build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]: """ if session is None: return {"kind": "none"} - if _is_requests_session(session): + if is_requests_session(session): return { "kind": "requests", "headers": dict(session.headers), "cookies": session.cookies, # RequestsCookieJar pickles cleanly "proxies": dict(session.proxies), } - if _is_rnet_session(session): + if is_rnet_session(session): name = session.impersonate_name if not name: return None @@ -936,8 +937,8 @@ def _build_session_spec(session: Optional[Any]) -> Optional[dict[str, Any]]: return None -def _rebuild_session(spec: dict[str, Any]) -> Optional[Any]: - """Reconstruct a session inside a child process from a spec built by ``_build_session_spec``.""" +def rebuild_session(spec: dict[str, Any]) -> Optional[Any]: + """Reconstruct a session inside a child process from a spec built by ``build_session_spec``.""" kind = spec["kind"] if kind == "requests": rs = Session() @@ -961,7 +962,7 @@ def _rebuild_session(spec: dict[str, Any]) -> Optional[Any]: return None # "none": child builds its own Session from the passed headers/cookies/proxy -def _mp_worker(queue: Any, kwargs: dict[str, Any]) -> None: +def mp_worker(queue: Any, kwargs: dict[str, Any]) -> None: """Child entry point (top-level so ``spawn`` can pickle it). Rebuilds the session, iterates ``requests()`` for its url chunk, and relays every event over @@ -980,7 +981,7 @@ def _mp_worker(queue: Any, kwargs: dict[str, Any]) -> None: if name in ("READ_TIMEOUT", "RETRY_WAIT", "HEDGE_MIN_WAIT"): setattr(sys.modules[__name__], name, float(value)) spec = kwargs.pop("_session_spec") - kwargs["session"] = _rebuild_session(spec) + kwargs["session"] = rebuild_session(spec) for event in requests(**kwargs): queue.put(event) except Exception: @@ -989,7 +990,7 @@ def _mp_worker(queue: Any, kwargs: dict[str, Any]) -> None: queue.put({"__mp_done__": True}) -def _download_multiprocess( +def download_multiprocess( urls: list[Any], output_dir: Path, filename: str, @@ -1056,7 +1057,7 @@ def _download_multiprocess( index_stride=processes, _session_spec=spec, ) - p = mp_ctx.Process(target=_mp_worker, args=(queue, child_kwargs), daemon=True) + p = mp_ctx.Process(target=mp_worker, args=(queue, child_kwargs), daemon=True) p.start() procs.append(p) @@ -1246,24 +1247,24 @@ def requests( # A spawned child re-imports this module and would start with no TokenBucket, so the # configured cap would silently stop applying; the cap wins over the fan-out, matching - # the _speed_limiter gates on the ranged-parallel paths. - speed_limiter = _speed_limiter - if processes > 1 and len(urls) >= MP_MIN_SEGMENTS and speed_limiter is not None: + # the speed_limiter gates on the ranged-parallel paths. + limiter = speed_limiter + if processes > 1 and len(urls) >= MP_MIN_SEGMENTS and limiter is not None: processes = 1 if debug_logger: debug_logger.log( level="DEBUG", operation="downloader_mp_fallback", message="Speed limit is set; spawned children cannot share its budget, using a single process", - context={"url_count": len(urls), "speed_limit": speed_limiter.rate}, + context={"url_count": len(urls), "speed_limit": limiter.rate}, ) # Process fan-out: split a large segment batch across spawned children. Single-URL and small # batches keep the in-process path (a lone file is already parallelized by ranged parts). if processes > 1 and len(urls) >= MP_MIN_SEGMENTS: - spec = _build_session_spec(session) + spec = build_session_spec(session) if spec is not None: - yield from _download_multiprocess( + yield from download_multiprocess( urls=cast("list[Any]", urls), # normalized to a list above output_dir=output_dir, filename=filename, @@ -1342,11 +1343,11 @@ def requests( url_item = urls[0] try: ranged_used = False - if max_workers > 1 and _speed_limiter is None and not _has_range_header(url_item): - total_size, supports_ranges = _probe_ranged(url_item["url"], session) + if max_workers > 1 and speed_limiter is None and not has_range_header(url_item): + total_size, supports_ranges = probe_ranged(url_item["url"], session) if supports_ranges and total_size >= RANGE_PARALLEL_MIN_SIZE: try: - yield from _dispatch_parts( + yield from dispatch_parts( session=session, total_size=total_size, max_workers=max_workers, @@ -1401,11 +1402,11 @@ def requests( batch_abort = threading.Event() # streams currently being read, so teardown can unblock a superseded loser parked in a - # slow read instead of the wait-gated shutdown draining it (see _force_unblock_stream) + # slow read instead of the wait-gated shutdown draining it (see force_unblock_stream) active_streams: set = set() active_lock = threading.Lock() - def _register_stream(stream: Any, active: bool) -> None: + def register_stream(stream: Any, active: bool) -> None: with active_lock: if active: active_streams.add(stream) @@ -1419,7 +1420,7 @@ def requests( # feed CDN error signals (429/timeout/reset) from each retrying segment into the controller on_retry_cb = (lambda _exc: controller.record_error(time.time())) if controller else None - def _download_worker(index: int, url_item: dict[str, Any], hedge: bool = False) -> None: + def download_worker(index: int, url_item: dict[str, Any], hedge: bool = False) -> None: item = dict(url_item) save_path = item.pop("save_path") if save_path.exists(): # finished in a previous run @@ -1449,7 +1450,7 @@ def requests( racing=(lambda: True) if hedge else (lambda: index in hedged), abort=batch_abort, on_retry=on_retry_cb, - register=_register_stream, + register=register_stream, **item, ): if "file_downloaded" in event: @@ -1474,7 +1475,7 @@ def requests( tail_boosted: set[int] = set() tail_skipped: set[int] = set() - def _submit(count: int, only: Optional[set[int]] = None) -> None: + def submit(count: int, only: Optional[set[int]] = None) -> None: # only= submits just those indices (used in the tail window to release # segments the boost declined while holding back boost candidates); with no # `only`, submits the leading `count` segments in FIFO order. @@ -1486,13 +1487,13 @@ def requests( break if only is not None and item[0] not in only: continue - pending.add(pool.submit(_download_worker, item[0], item[1])) + pending.add(pool.submit(download_worker, item[0], item[1])) picked.append(item) count -= 1 for item in picked: remaining.remove(item) - def _tail_part_worker( + def tail_part_worker( index: int, url: str, part_target: Path, @@ -1540,7 +1541,7 @@ def requests( event_queue.put(dict(file_downloaded=save_path, written=size)) event_queue.put(dict(advance=1)) - def _maybe_tail_boost(target: int) -> None: + def maybe_tail_boost(target: int) -> None: # Tail boost: near the end, split the few not-yet-started segments across idle # workers with intra-segment range parts so aggregate throughput doesn't collapse # to (few segments)x(per-connection speed). Only not-yet-started segments in @@ -1549,7 +1550,7 @@ def requests( # Engage on idle capacity so the boost fires despite `remaining` draining in strides # of the worker target; probe at most TAIL_BOOST_MAX_PER_CYCLE per cycle so a run of # blocking range probes can't stall the drain loop. - if DOWNLOAD_CANCELLED.is_set() or not _tail_boost_engages(len(remaining), len(pending), target): + if DOWNLOAD_CANCELLED.is_set() or not tail_boost_engages(len(remaining), len(pending), target): return # completed segments predict the tail's sizes: below the min every probe would # decline anyway, and at real-CDN RTT those sequential probes cost seconds on the @@ -1571,12 +1572,12 @@ def requests( if save_path.exists(): # finished in a previous run; let the normal fast-path handle it tail_skipped.add(index) continue - if _has_range_header(item): # byte-range slice; part-mode would clobber its Range + if has_range_header(item): # byte-range slice; part-mode would clobber its Range tail_skipped.add(index) continue req_kwargs = {k: v for k, v in item.items() if k != "url"} - size, supports_ranges = _probe_ranged(item["url"], session, **req_kwargs) - parts = _plan_tail_parts(size, spare) if supports_ranges else [] + size, supports_ranges = probe_ranged(item["url"], session, **req_kwargs) + parts = plan_tail_parts(size, spare) if supports_ranges else [] if not parts: tail_skipped.add(index) # no ranges / too small / no capacity -> normal path continue @@ -1599,7 +1600,7 @@ def requests( for start, end in parts: pending.add( pool.submit( - _tail_part_worker, + tail_part_worker, index, item["url"], part_target, @@ -1615,10 +1616,14 @@ def requests( ) boosted += 1 + # submitting every segment up front convoys on the pool's global lock and starves the + # event drain, so fixed mode meters submission through a window the loop tops up + queue_depth = max_workers * 2 + if controller: - _submit(controller.update(time.time())) + submit(controller.update(time.time())) else: - _submit(len(remaining)) + submit(queue_depth) pending_advance = 0 @@ -1655,16 +1660,18 @@ def requests( target = max_workers if controller: target = controller.update(now, len(pending) + len(remaining)) - _maybe_tail_boost(target) # spend spare workers on the tail before topping up + maybe_tail_boost(target) # spend spare workers on the tail before topping up if len(pending) < target: # In the final stride (unstarted segments <= one worker target), hold back # boost candidates so idle workers accumulate for a range-split; only # release segments the boost already declined (too small / no range) so # nothing stalls. Otherwise top up in FIFO order as usual. if remaining and len(remaining) <= target: - _submit(target - len(pending), only=tail_skipped) + submit(target - len(pending), only=tail_skipped) else: - _submit(target - len(pending)) + submit(target - len(pending)) + elif len(pending) < queue_depth: + submit(queue_depth - len(pending)) # Yield speed every 0.5s (throttled to avoid spamming Rich) if now - last_speed_report > 0.5: @@ -1684,7 +1691,7 @@ def requests( if ( len(pending) < target and seg_durations - and _speed_limiter is None + and speed_limiter is None and now - last_hedge_check > 0.5 and not DOWNLOAD_CANCELLED.is_set() ): @@ -1705,7 +1712,7 @@ def requests( ] for i in stuck[: target - len(pending)]: hedged.add(i) - pending.add(pool.submit(_download_worker, i, urls[i], True)) + pending.add(pool.submit(download_worker, i, urls[i], True)) # all segments claimed; superseded losers exit via their claimed() check if len(seg_done) == len(urls): @@ -1753,7 +1760,7 @@ def requests( with active_lock: losers = list(active_streams) for stream in losers: - _force_unblock_stream(stream) + force_unblock_stream(stream) pool.shutdown(wait=not DOWNLOAD_CANCELLED.is_set(), cancel_futures=True) # Drain remaining events diff --git a/unshackle/core/drm/clearkey.py b/unshackle/core/drm/clearkey.py index a1732769..0fca7110 100644 --- a/unshackle/core/drm/clearkey.py +++ b/unshackle/core/drm/clearkey.py @@ -50,7 +50,7 @@ class ClearKey: self.iv: bytes = iv self._warned_clear: bool = False - def _warn_clear(self, reason: str) -> None: + def warn_clear(self, reason: str) -> None: """Warn once per track that a segment was passed through undecrypted.""" if not self._warned_clear: log.warning("ClearKey: manifest signaled AES-128 encryption but %s; passing segment through.", reason) @@ -71,13 +71,13 @@ class ClearKey: and len(data) >= TS_PACKET_SIZE * TS_SYNC_CHECKS and all(data[i * TS_PACKET_SIZE] == SYNC_BYTE for i in range(TS_SYNC_CHECKS)) ): - self._warn_clear("the segment is clear MPEG-TS") + self.warn_clear("the segment is clear MPEG-TS") return # Valid AES-128-CBC ciphertext is always a 16-byte multiple; anything else is not # ciphertext (truncated/clear/SAMPLE-AES). Decrypting would raise, so pass it through. if len(data) % AES.block_size != 0: - self._warn_clear(f"the segment length ({len(data)} bytes) is not a multiple of {AES.block_size}") + self.warn_clear(f"the segment length ({len(data)} bytes) is not a multiple of {AES.block_size}") return log_event( diff --git a/unshackle/core/drm/monalisa.py b/unshackle/core/drm/monalisa.py index 284a19bf..fa4c2182 100644 --- a/unshackle/core/drm/monalisa.py +++ b/unshackle/core/drm/monalisa.py @@ -84,9 +84,9 @@ class MonaLisa: self.data: dict = kwargs or {} # Extract keys immediately - self._extract_keys() + self.extract_keys() - def _extract_keys(self) -> None: + def extract_keys(self) -> None: """Extract keys from the ticket using the MonaLisa CDM.""" # Import here to avoid circular import from unshackle.core.cdm.monalisa import MonaLisaCDM @@ -183,7 +183,7 @@ class MonaLisa: # Validate it looks like a content ID if content_id.startswith("H5DCID-"): return content_id - except Exception: + except (ValueError, TypeError): pass return None diff --git a/unshackle/core/drm/playready.py b/unshackle/core/drm/playready.py index 58685361..18434016 100644 --- a/unshackle/core/drm/playready.py +++ b/unshackle/core/drm/playready.py @@ -42,7 +42,7 @@ class PlayReady: raise TypeError(f"Expected pssh to be a {PSSH}, not {pssh!r}") if pssh_b64: - kids = self._extract_kids_from_pssh_b64(pssh_b64) + kids = self.extract_kids_from_pssh_b64(pssh_b64) else: kids = [] @@ -55,7 +55,7 @@ class PlayReady: kids.append(signed_key_id.value) else: kids.append(UUID(bytes_le=base64.b64decode(signed_key_id.value))) - except Exception: + except (ValueError, TypeError): continue if kid: @@ -79,7 +79,7 @@ class PlayReady: if pssh_b64: self.data.setdefault("pssh_b64", pssh_b64) - def _extract_kids_from_pssh_b64(self, pssh_b64: str) -> list[UUID]: + def extract_kids_from_pssh_b64(self, pssh_b64: str) -> list[UUID]: """Extract all KIDs from base64-encoded PSSH data.""" try: # PSSH XML comes from third-party manifests; defusedxml guards against entity expansion @@ -118,7 +118,7 @@ class PlayReady: kid_bytes = base64.b64decode(value + "==") kid_uuid = UUID(bytes_le=kid_bytes) kids.append(kid_uuid) - except Exception: + except ValueError: pass # v4.2/v4.3: DATA/PROTECTINFO/KIDS/KID @@ -131,7 +131,7 @@ class PlayReady: kid_uuid = UUID(bytes_le=kid_bytes) if kid_uuid not in kids: kids.append(kid_uuid) - except Exception: + except ValueError: pass # v4.1: DATA/PROTECTINFO/KID @@ -144,7 +144,7 @@ class PlayReady: kid_uuid = UUID(bytes_le=kid_bytes) if kid_uuid not in kids: kids.append(kid_uuid) - except Exception: + except ValueError: pass # v4.0: DATA/KID @@ -156,12 +156,14 @@ class PlayReady: kid_uuid = UUID(bytes_le=kid_bytes) if kid_uuid not in kids: kids.append(kid_uuid) - except Exception: + except ValueError: pass return kids - except Exception: + # covers bad base64 (binascii.Error), defusedxml rejections (both ValueError) and XML + # ParseError (SyntaxError) from arbitrary third-party PSSH data + except (ValueError, SyntaxError): pass return [] @@ -265,7 +267,7 @@ class PlayReady: def kids(self) -> list[UUID]: return self._kids - def _extract_keys_from_cdm(self, cdm: PlayReadyCdm, session_id: bytes) -> dict: + def extract_keys_from_cdm(self, cdm: PlayReadyCdm, session_id: bytes) -> dict: """Extract keys from CDM session with cross-library compatibility. Args: @@ -330,7 +332,7 @@ class PlayReady: if "" not in license_str: try: license_str = base64.b64decode(license_str + "===").decode() - except Exception: + except ValueError: pass cdm.parse_license(session_id, license_str) @@ -344,12 +346,12 @@ class PlayReady: except PlayReady.Exceptions.DeviceRevoked: raise except Exception as e: - revoked = self._detect_revocation(license_str) or self._detect_revocation(str(e)) + revoked = self.detect_revocation(license_str) or self.detect_revocation(str(e)) if revoked: raise PlayReady.Exceptions.DeviceRevoked(revoked) from e raise - keys = self._extract_keys_from_cdm(cdm, session_id) + keys = self.extract_keys_from_cdm(cdm, session_id) self.content_keys.update(keys) if keys: @@ -368,7 +370,7 @@ class PlayReady: raise PlayReady.Exceptions.EmptyLicense("No Content Keys were within the License") @staticmethod - def _detect_revocation(text: str) -> Optional[str]: + def detect_revocation(text: str) -> Optional[str]: """Return the decoded error string if a revocation HRESULT is present, else None. Reads the code from the raw SOAP body (0x8004C065) @@ -415,9 +417,9 @@ class PlayReady: decrypt_start = time.monotonic() if decrypter == "mp4decrypt": - self._decrypt_with_mp4decrypt(path) + self.decrypt_with_mp4decrypt(path) else: - self._decrypt_with_shaka_packager(path) + self.decrypt_with_shaka_packager(path) log_event( "drm_decrypt_complete", @@ -430,7 +432,7 @@ class PlayReady: output_size=path.stat().st_size if path.exists() else 0, ) - def _decrypt_with_mp4decrypt(self, path: Path) -> None: + def decrypt_with_mp4decrypt(self, path: Path) -> None: """Decrypt using mp4decrypt""" if not binaries.Mp4decrypt: raise EnvironmentError("mp4decrypt executable not found but is required.") @@ -483,7 +485,7 @@ class PlayReady: path.unlink() shutil.move(output_path, path) - def _decrypt_with_shaka_packager(self, path: Path) -> None: + def decrypt_with_shaka_packager(self, path: Path) -> None: """Decrypt using Shaka Packager (original method)""" if not binaries.ShakaPackager: raise EnvironmentError("Shaka Packager executable not found but is required.") diff --git a/unshackle/core/drm/widevine.py b/unshackle/core/drm/widevine.py index 2b036235..d44f4e49 100644 --- a/unshackle/core/drm/widevine.py +++ b/unshackle/core/drm/widevine.py @@ -331,9 +331,9 @@ class Widevine: decrypt_start = time.monotonic() if decrypter == "mp4decrypt": - self._decrypt_with_mp4decrypt(path) + self.decrypt_with_mp4decrypt(path) else: - self._decrypt_with_shaka_packager(path) + self.decrypt_with_shaka_packager(path) log_event( "drm_decrypt_complete", @@ -346,7 +346,7 @@ class Widevine: output_size=path.stat().st_size if path.exists() else 0, ) - def _decrypt_with_mp4decrypt(self, path: Path) -> None: + def decrypt_with_mp4decrypt(self, path: Path) -> None: """Decrypt using mp4decrypt""" if not binaries.Mp4decrypt: raise EnvironmentError("mp4decrypt executable not found but is required.") @@ -399,7 +399,7 @@ class Widevine: path.unlink() shutil.move(output_path, path) - def _decrypt_with_shaka_packager(self, path: Path) -> None: + def decrypt_with_shaka_packager(self, path: Path) -> None: """Decrypt using Shaka Packager (original method)""" if not binaries.ShakaPackager: raise EnvironmentError("Shaka Packager executable not found but is required.") diff --git a/unshackle/core/import_service.py b/unshackle/core/import_service.py index 3dedd1f8..adf26a18 100644 --- a/unshackle/core/import_service.py +++ b/unshackle/core/import_service.py @@ -16,7 +16,7 @@ from unshackle.core.constants import AnyTrack from unshackle.core.credential import Credential from unshackle.core.drm import drm_from_dict from unshackle.core.manifests import DASH, HLS, ISM -from unshackle.core.remote_service import RemoteService, _build_title, _match_track, _resolve_proxy +from unshackle.core.remote_service import RemoteService, build_title, match_track, resolve_proxy_arg from unshackle.core.titles import Episode, Movies, Series, Title_T, Titles_T, remap_titles from unshackle.core.tracks import Audio, Chapter, Chapters, Tracks, Video from unshackle.core.tracks.attachment import Attachment @@ -28,7 +28,7 @@ PARSERS = {"DASH": DASH, "HLS": HLS, "ISM": ISM} MANIFEST_DATA_KEYS = {"DASH": "dash", "ISM": "ism"} -def _fallback_language(language: Any) -> Optional[Any]: +def fallback_language(language: Any) -> Optional[Any]: """The title language if the parsers would accept it, else None so they raise. DASH drops an und/invalid fallback but ISM does not, and a truthy 'und' there silently @@ -40,7 +40,7 @@ def _fallback_language(language: Any) -> Optional[Any]: return language -def _resolve_import_manifest_data( +def resolve_import_manifest_data( tracks: Tracks, manifest_type: Optional[str], *, @@ -68,7 +68,7 @@ def _resolve_import_manifest_data( if not pending: return - fallback_lang = _fallback_language(language) + fallback_lang = fallback_language(language) for url in {str(track.url) for track in pending if track.url}: try: manifest = parser.from_url(url=url, session=session) @@ -90,7 +90,7 @@ def _resolve_import_manifest_data( for track in pending: if track.data.get(data_key) or str(track.url) != url: continue - matched = _match_track(track, local_tracks) + matched = match_track(track, local_tracks) if matched and matched.data.get(data_key): track.data.update(matched.data) @@ -160,7 +160,7 @@ class ImportService: return session try: - proxy = _resolve_proxy(proxy_query) + proxy = resolve_proxy_arg(proxy_query) except Exception as e: if explicit: raise click.ClickException(f"Failed to resolve proxy '{proxy_query}': {e}") @@ -184,7 +184,7 @@ class ImportService: if self.titles is not None: return self.titles titles_list = [ - _build_title(entry.get("meta", {}), self.service_tag, fallback_id=title_id) + build_title(entry.get("meta", {}), self.service_tag, fallback_id=title_id) for title_id, entry in self.titles_data.items() ] self.titles = ( @@ -231,7 +231,7 @@ class ImportService: f"The manifest URL may have expired since export. ({e})" ) try: - parsed = manifest.to_tracks(language=_fallback_language(title.language)) + parsed = manifest.to_tracks(language=fallback_language(title.language)) except ValueError as e: if "Language information could not be derived" in str(e): raise click.ClickException( @@ -275,7 +275,7 @@ class ImportService: track.drm = drm tracks.add(track, warn_only=True) - _resolve_import_manifest_data( + resolve_import_manifest_data( tracks, manifest_type, session=self.session, @@ -328,7 +328,7 @@ class ImportService: if drm_obj is None and keys: drm_type = (drm_dicts[0].get("system", "Widevine").lower()) if drm_dicts else "widevine" - drm_obj = RemoteService._create_drm_stub(drm_type, list(keys.keys())) + drm_obj = RemoteService.create_drm_stub(drm_type, list(keys.keys())) if drm_obj is None: return None @@ -354,7 +354,7 @@ class ImportService: for track in title.tracks: if not isinstance(track, (Video, Audio)) or not self.track_is_encrypted(track): continue - drm_obj = track.drm[0] if track.drm else RemoteService._create_drm_stub(system, kid_hexes) + drm_obj = track.drm[0] if track.drm else RemoteService.create_drm_stub(system, kid_hexes) for kid, key in pool.items(): drm_obj.content_keys[kid] = key track.drm = [drm_obj] diff --git a/unshackle/core/manifests/dash.py b/unshackle/core/manifests/dash.py index 9727d7ca..df9a903f 100644 --- a/unshackle/core/manifests/dash.py +++ b/unshackle/core/manifests/dash.py @@ -115,7 +115,7 @@ class DASH: if period_id := period.get("id"): filtered_period_ids.append(period_id) continue - if not DASH._is_content_period(period, []): + if not DASH.is_content_period(period, []): if period_id := period.get("id"): filtered_period_ids.append(period_id) continue @@ -126,8 +126,8 @@ class DASH: continue for rep in adaptation_set.findall("Representation"): - get = partial(self._get, adaptation_set=adaptation_set, representation=rep) - findall = partial(self._findall, adaptation_set=adaptation_set, representation=rep, both=True) + get = partial(self.get_attr, adaptation_set=adaptation_set, representation=rep) + findall = partial(self.find_elements, adaptation_set=adaptation_set, representation=rep, both=True) segment_base = rep.find("SegmentBase") codecs = get("codecs") @@ -346,33 +346,29 @@ class DASH: init_data: Optional[bytes] = None track_kid: Optional[UUID] = None - content_periods = [p for p in all_periods if DASH._is_content_period(p, filtered_period_ids)] + content_periods = [p for p in all_periods if DASH.is_content_period(p, filtered_period_ids)] period_count = len(content_periods) if period_count > 1: log.debug(f"Multi-period manifest detected with {period_count} content periods") + stored_period: Optional[Element] = track.data["dash"].get("period") + for period_idx, content_period in enumerate(content_periods): - # Find the matching representation in this period - matched_rep = None - matched_as = None - for as_ in content_period.findall("AdaptationSet"): - if DASH.is_trick_mode(as_): - continue - for rep in as_.findall("Representation"): - if rep.get("id") == rep_id: - matched_rep = rep - matched_as = as_ - break - if matched_rep is not None: - break + if content_period is stored_period: + matched_as, matched_rep = adaptation_set, representation + else: + matched_as, matched_rep = DASH.resolve_representation(content_period, rep_id, adaptation_set) or ( + None, + None, + ) if matched_rep is None or matched_as is None: period_id = content_period.get("id", period_idx) log.warning(f"Representation '{rep_id}' not found in period '{period_id}', skipping") continue - p_init, p_segments, p_timescale, p_durations, p_kid = DASH._get_period_segments( + p_init, p_segments, p_timescale, p_durations, p_kid = DASH.get_period_segments( period=content_period, adaptation_set=matched_as, representation=matched_rep, @@ -454,7 +450,7 @@ class DASH: # When every segment is a byte range of one parent resource and init is its [0, len) # prefix, the whole resource is itself a valid MP4, so one direct download to save_path # replaces the per-segment files, init write and merge pass. - collapse_single_url = DASH._collapsible_single_url( + collapse_single_url = DASH.collapsible_single_url( isinstance(track, Subtitle), segments, len(init_data) if init_data is not None else None, @@ -620,7 +616,7 @@ class DASH: progress(downloaded="Downloaded") @staticmethod - def _collapsible_single_url( + def collapsible_single_url( is_subtitle: bool, segments: list[tuple[str, Optional[str]]], init_len: Optional[int], @@ -663,7 +659,7 @@ class DASH: return True @staticmethod - def _is_content_period(period: Element, filtered_period_ids: list[str]) -> bool: + def is_content_period(period: Element, filtered_period_ids: list[str]) -> bool: """Check if a period is a valid content period (not an ad, not filtered, not trick mode).""" period_id = period.get("id") if period_id and period_id in filtered_period_ids: @@ -677,7 +673,7 @@ class DASH: return True @staticmethod - def _merge_segment_templates(adaptation_set: Element, representation: Element) -> Optional[Element]: + def merge_segment_templates(adaptation_set: Element, representation: Element) -> Optional[Element]: """ Build the effective SegmentTemplate for a Representation by cascading the AdaptationSet > Representation levels (ISO/IEC 23009-1 5.3.9.1). @@ -703,7 +699,7 @@ class DASH: return merged @staticmethod - def _get_period_segments( + def get_period_segments( period: Element, adaptation_set: Element, representation: Element, @@ -736,7 +732,7 @@ class DASH: period_duration = period.get("duration") or manifest.get("mediaPresentationDuration") init_data: Optional[bytes] = None - segment_template = DASH._merge_segment_templates(adaptation_set, representation) + segment_template = DASH.merge_segment_templates(adaptation_set, representation) segment_list = representation.find("SegmentList") if segment_list is None: @@ -891,8 +887,12 @@ class DASH: return init_data, segments, segment_timescale, segment_durations, track_kid + # Deprecated 5.5.0 shims for service repos still on the old underscored names; drop once they migrate. + _is_content_period = is_content_period + _get_period_segments = get_period_segments + @staticmethod - def _get(item: str, adaptation_set: Element, representation: Optional[Element] = None) -> Optional[Any]: + def get_attr(item: str, adaptation_set: Element, representation: Optional[Element] = None) -> Optional[Any]: """Helper to get a requested item from the Representation, otherwise from the AdaptationSet.""" adaptation_set_item = adaptation_set.get(item) if representation is None: @@ -905,7 +905,7 @@ class DASH: return adaptation_set_item @staticmethod - def _findall( + def find_elements( item: str, adaptation_set: Element, representation: Optional[Element] = None, both: bool = False ) -> list[Any]: """ @@ -1010,6 +1010,54 @@ class DASH: for prop in essential_props + supplemental_props ) + @staticmethod + def adaptation_set_key(adaptation_set: Element) -> tuple[Optional[str], ...]: + """Identity of an AdaptationSet, used to re-find its counterpart in another Period.""" + return tuple( + adaptation_set.get(attr) + for attr in ("contentType", "mimeType", "lang", "audioTrackId", "audioTrackSubtype", "id") + ) + + @staticmethod + def resolve_representation( + period: Element, rep_id: Optional[str], adaptation_set: Element + ) -> Optional[tuple[Element, Element]]: + """ + Find the (AdaptationSet, Representation) in another Period matching rep_id. + + Representation ids are only unique within an AdaptationSet, so the set matching + `adaptation_set`'s identity wins over an earlier sibling that reuses the same id + for different content (e.g. audio description vs dialog). + """ + as_key = DASH.adaptation_set_key(adaptation_set) + + def candidates(within: Element) -> list[tuple[Element, Element]]: + found = [] + for as_ in within.findall("AdaptationSet"): + if DASH.is_trick_mode(as_) or DASH.adaptation_set_key(as_) != as_key: + continue + for rep in as_.findall("Representation"): + if rep.get("id") == rep_id: + found.append((as_, rep)) + break + return found + + matches = candidates(period) + if matches: + parent = adaptation_set.getparent() + siblings = [as_ for as_, _ in candidates(parent)] if parent is not None else [] + ordinal = siblings.index(adaptation_set) if adaptation_set in siblings else 0 + return matches[min(ordinal, len(matches) - 1)] + + for as_ in period.findall("AdaptationSet"): + if DASH.is_trick_mode(as_): + continue + for rep in as_.findall("Representation"): + if rep.get("id") == rep_id: + return as_, rep + + return None + @staticmethod def is_descriptive(adaptation_set: Element) -> bool: """Check if contents of Adaptation Set is Descriptive.""" @@ -1050,7 +1098,7 @@ class DASH: return next( ( int(x.get("value")) - for x in DASH._findall("SupplementalProperty", adaptation_set, representation, both=True) + for x in DASH.find_elements("SupplementalProperty", adaptation_set, representation, both=True) if x.get("schemeIdUri") == "tag:dolby.com,2018:dash:EC3_ExtensionComplexityIndex:2018" ), None, @@ -1112,7 +1160,7 @@ class DASH: if kid_b64: try: kid = UUID(bytes=base64.b64decode(kid_b64)) - except Exception: + except ValueError: kid = None drm.append(PlayReady(pssh=pr_pssh, kid=kid, pssh_b64=pr_pssh_b64)) @@ -1128,7 +1176,7 @@ class DASH: except ValueError: try: kid = UUID(bytes=base64.b64decode(kid_attr)) - except Exception: + except ValueError: kid = None if not kid: @@ -1183,7 +1231,7 @@ class DASH: @staticmethod @lru_cache(maxsize=None) - def _field_format_pattern(field: str) -> re.Pattern: + def field_format_pattern(field: str) -> re.Pattern: # printf-style `$Field%fmt$` matcher, compiled once per field name return re.compile(rf"\${re.escape(field)}%([a-z0-9]+)\$", flags=re.I) @@ -1197,7 +1245,7 @@ class DASH: for field, value in kwargs.items(): url = url.replace(f"${field}$", str(value)) if has_fmt: - m = DASH._field_format_pattern(field).search(url) + m = DASH.field_format_pattern(field).search(url) if m: url = url.replace(m.group(), f"{value:{m.group(1)}}") return url diff --git a/unshackle/core/manifests/hls.py b/unshackle/core/manifests/hls.py index d5cb0950..fb973989 100644 --- a/unshackle/core/manifests/hls.py +++ b/unshackle/core/manifests/hls.py @@ -311,15 +311,16 @@ class HLS: if has_resolution and has_codec: continue try: - probe = HLS._probe_ts_info(video.url, self.session) + probe = HLS.probe_ts_info(video.url, self.session) if probe: width, height, codec = probe if not has_resolution: video.width, video.height = width, height if not has_codec: video.codec = codec - except Exception: - pass + # best-effort probe over network + ffprobe; the track stays usable without it + except Exception as e: + logging.getLogger("HLS").debug(f"TS probe failed for {video.url}: {e!r}") if self.url: tracks.manifest_url = self.url @@ -342,7 +343,7 @@ class HLS: return tracks @staticmethod - def _probe_ts_info( + def probe_ts_info( variant_url: str, session: Optional[Union[Session, RnetSession]] = None ) -> Optional[tuple[int, int, Video.Codec]]: """Probe the first TS segment of a variant playlist to extract resolution and codec.""" @@ -360,10 +361,10 @@ class HLS: res = session.get(seg_uri, headers={"Range": "bytes=0-8191"}) data = res.content - return HLS._parse_ts_video_info(data) + return HLS.parse_ts_video_info(data) @staticmethod - def _parse_ts_video_info(data: bytes) -> Optional[tuple[int, int, Video.Codec]]: + def parse_ts_video_info(data: bytes) -> Optional[tuple[int, int, Video.Codec]]: """Parse H.264/H.265 NAL units from TS segment data to extract resolution and codec.""" class _BitReader: @@ -1148,12 +1149,12 @@ class HLS: try: decoded = base64.b64decode(value).decode() - except Exception: + except ValueError: decoded = value try: items = json.loads(decoded) - except Exception: + except ValueError: continue for item in items if isinstance(items, list) else []: @@ -1224,8 +1225,9 @@ class HLS: map_res = session.get(url=map_uri, headers=headers) if map_res.ok: return track.get_key_id(map_res.content) - except Exception: - pass + # best-effort key-id probe over network + init parse; caller falls back without it + except Exception as e: + logging.getLogger("HLS").debug(f"Init map key-id probe failed for {map_uri}: {e!r}") return None @staticmethod @@ -1292,9 +1294,15 @@ class HLS: elif key.method == "ISO-23001-7": drm = Widevine(pssh=WV_PSSH.new(key_ids=[key.uri.split(",")[-1]], system_id=WV_PSSH.SystemId.Widevine)) elif key.keyformat and key.keyformat.lower() == WidevineCdm.urn: + extra_params = dict(key._extra_params) # noqa + # a v0 PSSH carries no KID, leaving the tag's own KEYID as the only source + kid = extra_params.pop("keyid", None) or extra_params.pop("kid", None) + if isinstance(kid, str): + kid = kid.removeprefix("0x").removeprefix("0X") drm = Widevine( pssh=WV_PSSH(key.uri.split(",")[-1]), - **key._extra_params, # noqa + kid=kid, + **extra_params, ) elif key.keyformat and key.keyformat.lower() in {f"urn:uuid:{PR_PSSH.SYSTEM_ID}", "com.microsoft.playready"}: drm = PlayReady( diff --git a/unshackle/core/manifests/ism.py b/unshackle/core/manifests/ism.py index 4cf82026..0423df16 100644 --- a/unshackle/core/manifests/ism.py +++ b/unshackle/core/manifests/ism.py @@ -3,6 +3,7 @@ from __future__ import annotations import base64 import hashlib import html +import logging import re import shutil import struct @@ -78,24 +79,28 @@ class ISM: return cls(load_xml(text), url) @staticmethod - def _get_drm(headers: list[Element]) -> list[DRM_T]: + def get_drm(headers: list[Element]) -> list[DRM_T]: drm: list[DRM_T] = [] for header in headers: - system_id = (header.get("SystemID") or header.get("SystemId") or "").lower() + # MS-SSTR manifests often write the GUID in registry format ({...}). + system_id = (header.get("SystemID") or header.get("SystemId") or "").strip().strip("{}").lower() data = "".join(header.itertext()).strip() if not data: continue if system_id == "edef8ba9-79d6-4ace-a3c8-27dcd51d21ed": try: pssh = PSSH(base64.b64decode(data)) - except Exception: + # a dropped header can leave the track with no DRM, so never skip silently + except Exception as e: + logging.getLogger("ISM").warning(f"Skipping unparseable Widevine ProtectionHeader: {e!r}") continue kid = next(iter(pssh.key_ids), None) drm.append(Widevine(pssh=pssh, kid=kid)) elif system_id == "9a04f079-9840-4286-ab92-e65be0885f95": try: pr_pssh = PR_PSSH(data) - except Exception: + except Exception as e: + logging.getLogger("ISM").warning(f"Skipping unparseable PlayReady ProtectionHeader: {e!r}") continue drm.append(PlayReady(pssh=pr_pssh, pssh_b64=data)) return drm @@ -122,7 +127,7 @@ class ISM: return ISM.get_video_range_and_fps(fourcc, codec_private_data)[0] @staticmethod - def _init_segment( + def init_segment( track: AnyTrack, session_drm: Optional[DRM_T], first_segment: Optional[bytes] = None ) -> Optional[bytes]: # Smooth fragments are moof+mdat only; rebuild the ftyp+moov init box from @@ -225,7 +230,7 @@ class ISM: tracks = Tracks() base_url = self.url duration = int(self.manifest.get("Duration") or 0) - drm = self._get_drm(self.manifest.xpath(".//ProtectionHeader")) + drm = self.get_drm(self.manifest.xpath(".//ProtectionHeader")) for stream_index in self.manifest.findall("StreamIndex"): content_type = stream_index.get("Type") @@ -272,7 +277,8 @@ class ISM: if not duration_frag: try: next_time = int(fragments[idx + 1].get("t")) - except (IndexError, AttributeError): + except (IndexError, TypeError): + # no next fragment, or the next omits t (get returns None) next_time = duration # floor division: float times would corrupt segment URLs; # any drift is reset by the next fragment's explicit t. @@ -524,7 +530,7 @@ class ISM: progress(downloaded="Merging", completed=0, total=len(segments_to_merge)) with open(save_path, "wb") as f: first_segment = segments_to_merge[0].read_bytes() if segments_to_merge else None - init_segment = ISM._init_segment(track, session_drm, first_segment) + init_segment = ISM.init_segment(track, session_drm, first_segment) if init_segment: f.write(init_segment) iv_size = (read_per_sample_iv_size(first_segment) if session_drm and first_segment else None) or 8 diff --git a/unshackle/core/music/display.py b/unshackle/core/music/display.py index 08a8e747..19ceac8c 100644 --- a/unshackle/core/music/display.py +++ b/unshackle/core/music/display.py @@ -14,6 +14,7 @@ from io import BytesIO from typing import Any, Optional # RenderableType is rich's union of "things that can be printed to a console". +from rich import box from rich.console import Group, RenderableType from rich.padding import Padding from rich.panel import Panel @@ -145,7 +146,7 @@ def render_track_panel(rows: list[TrackRow], total: int) -> Panel: detail.append(" | ") detail.append("Hi-Res", style="gold1") node.add(detail, guide_style="bright_black") - return Panel(tree, title="Available Tracks") + return Panel(tree, title="Available Tracks", box=box.SQUARE, border_style="bright_black") def render_album_header( diff --git a/unshackle/core/music/integrity.py b/unshackle/core/music/integrity.py index c1706406..d38d3421 100644 --- a/unshackle/core/music/integrity.py +++ b/unshackle/core/music/integrity.py @@ -56,7 +56,7 @@ def verify_music_audio( if not audio_track: raise MusicAudioIntegrityError(f"MediaInfo could not find an audio stream in: {path.name}") - duration = _duration_seconds(getattr(audio_track, "duration", None)) + duration = duration_seconds(getattr(audio_track, "duration", None)) expected_duration = _expected_duration(song, track) warnings: list[str] = [] if expected_duration and duration: @@ -69,15 +69,15 @@ def verify_music_audio( result = MusicAudioIntegrityResult( path=path, size=size, - codec=_first_text( + codec=first_text( getattr(audio_track, "format", None), getattr(audio_track, "commercial_name", None), getattr(audio_track, "codec_id", None), ), duration=duration, - bit_depth=_first_int(getattr(audio_track, "bit_depth", None)), - sample_rate=_first_int(getattr(audio_track, "sampling_rate", None)), - channels=_first_float( + bit_depth=first_int(getattr(audio_track, "bit_depth", None)), + sample_rate=first_int(getattr(audio_track, "sampling_rate", None)), + channels=first_float( getattr(audio_track, "channel_s", None), getattr(audio_track, "channel_s_original", None), getattr(track, "channels", None), @@ -115,13 +115,13 @@ def _expected_duration(song: Any, track: Any) -> Optional[float]: data = getattr(song, "data", None) if isinstance(data, dict): metadata = data.get("metadata") if isinstance(data.get("metadata"), dict) else {} - value = _first_float(data.get("duration"), metadata.get("duration")) + value = first_float(data.get("duration"), metadata.get("duration")) if value: return value track_data = getattr(track, "data", None) if isinstance(track_data, dict): metadata = track_data.get("metadata") if isinstance(track_data.get("metadata"), dict) else {} - value = _first_float(track_data.get("duration"), metadata.get("duration")) + value = first_float(track_data.get("duration"), metadata.get("duration")) if value: return value return None @@ -140,7 +140,7 @@ def _expected_size(track: Any) -> Optional[int]: data.get("metadata") if isinstance(data.get("metadata"), dict) else {}, ] for source in sources: - value = _first_int( + value = first_int( source.get("content_length"), source.get("contentLength"), source.get("file_size"), @@ -152,8 +152,8 @@ def _expected_size(track: Any) -> Optional[int]: return None -def _duration_seconds(value: Any) -> Optional[float]: - number = _first_float(value) +def duration_seconds(value: Any) -> Optional[float]: + number = first_float(value) if number is None: return None if number > 1000: @@ -161,7 +161,7 @@ def _duration_seconds(value: Any) -> Optional[float]: return number -def _first_text(*values: Any) -> str: +def first_text(*values: Any) -> str: for value in values: text = str(value or "").strip() if text: @@ -169,7 +169,7 @@ def _first_text(*values: Any) -> str: return "" -def _first_float(*values: Any) -> Optional[float]: +def first_float(*values: Any) -> Optional[float]: for value in values: if value in (None, "", [], {}): continue @@ -181,8 +181,8 @@ def _first_float(*values: Any) -> Optional[float]: return None -def _first_int(*values: Any) -> Optional[int]: - value = _first_float(*values) +def first_int(*values: Any) -> Optional[int]: + value = first_float(*values) return int(value) if value is not None else None diff --git a/unshackle/core/music/planner.py b/unshackle/core/music/planner.py index 2ac0bcde..0ae26320 100644 --- a/unshackle/core/music/planner.py +++ b/unshackle/core/music/planner.py @@ -25,7 +25,7 @@ class MusicPlanner: or (first_song.album_artist or first_song.artist if first_song else ""), album_artist=(first_song.album_artist if first_song else "") or "", year=getattr(music, "year", None) or (first_song.year if first_song else None), - released=self._first_text( + released=self.first_text( getattr(music, "released", None), getattr(music, "release_date", None), first_data.get("release_date"), @@ -37,15 +37,15 @@ class MusicPlanner: label=(first_song.label if first_song else "") or "", artwork_url=getattr(music, "artwork_url", None), total_tracks=getattr(music, "total_tracks", None) or len(music), - total_discs=getattr(music, "total_discs", None) or self._max_value(music, "disc"), - total_duration=getattr(music, "total_duration", None) or self._sum_duration(music), - quality_requested=self._quality_requested(music), + total_discs=getattr(music, "total_discs", None) or self.max_value(music, "disc"), + total_duration=getattr(music, "total_duration", None) or self.sum_duration(music), + quality_requested=self.quality_requested(music), ) selected_options: list[MusicTrackOption] = [] discs: dict[int, list[MusicSongPlan]] = defaultdict(list) for song in music: - options = self._get_options(song) + options = self.get_options(song) selected = options[0] if options else None if selected: selected_options.append(selected) @@ -60,33 +60,33 @@ class MusicPlanner: for disc_number in sorted(discs): plan.discs.append(MusicDiscPlan(disc_number=disc_number, songs=discs[disc_number])) - quality = self._quality_summary_from_options(selected_options) + quality = self.quality_summary_from_options(selected_options) if quality: plan.quality_requested = quality return plan - def _get_options(self, song: Song) -> list[MusicTrackOption]: + def get_options(self, song: Song) -> list[MusicTrackOption]: provider = getattr(self.service, "get_music_track_options", None) if callable(provider): options = provider(song) if options: return options - option = self._option_from_song(song) + option = self.option_from_song(song) return [option] if option else [] @staticmethod - def _option_from_song(song: Song) -> MusicTrackOption | None: + def option_from_song(song: Song) -> MusicTrackOption | None: data = song.data if isinstance(song.data, dict) else {} metadata = data.get("metadata") if isinstance(data.get("metadata"), dict) else {} - quality = MusicPlanner._first_text(data.get("quality"), metadata.get("quality"), metadata.get("quality_label")) - duration = MusicPlanner._first_number(data.get("duration"), metadata.get("duration")) + quality = MusicPlanner.first_text(data.get("quality"), metadata.get("quality"), metadata.get("quality_label")) + duration = MusicPlanner.first_number(data.get("duration"), metadata.get("duration")) if not quality and duration is None: return None - codec = MusicPlanner._codec_from_quality(quality) - bit_depth, sample_rate = MusicPlanner._quality_numbers(quality) - bitrate = MusicPlanner._bitrate_from_quality(quality) + codec = MusicPlanner.codec_from_quality(quality) + bit_depth, sample_rate = MusicPlanner.quality_numbers(quality) + bitrate = MusicPlanner.bitrate_from_quality(quality) hires = bool((bit_depth and bit_depth > 16) or (sample_rate and sample_rate > 48000)) lossless = codec in {"FLAC", "ALAC", "WAV", "AIFF"} or "lossless" in quality.lower() atmos = "atmos" in quality.lower() or bool(data.get("atmos") or metadata.get("atmos")) @@ -97,7 +97,7 @@ class MusicPlanner: bit_depth=bit_depth, sample_rate=sample_rate, bitrate=bitrate, - channels=MusicPlanner._first_number(data.get("channels"), metadata.get("channels")), + channels=MusicPlanner.first_number(data.get("channels"), metadata.get("channels")), lossless=lossless, hires=hires, atmos=atmos, @@ -106,7 +106,7 @@ class MusicPlanner: quality_label=quality, ) - def _quality_requested(self, music: Music) -> str: + def quality_requested(self, music: Music) -> str: service_quality = getattr(self.service, "quality", None) if service_quality: quality_map = { @@ -121,10 +121,10 @@ class MusicPlanner: if not first_song: return "" data = first_song.data if isinstance(first_song.data, dict) else {} - return self._first_text(data.get("quality")) + return self.first_text(data.get("quality")) @staticmethod - def _first_text(*values: Any) -> str: + def first_text(*values: Any) -> str: for value in values: if value in (None, ""): continue @@ -134,7 +134,7 @@ class MusicPlanner: return "" @staticmethod - def _first_number(*values: Any) -> float | None: + def first_number(*values: Any) -> float | None: for value in values: if value in (None, ""): continue @@ -145,7 +145,7 @@ class MusicPlanner: return None @staticmethod - def _codec_from_quality(value: str) -> str: + def codec_from_quality(value: str) -> str: lowered = value.lower() for codec in ("flac", "alac", "aac", "mp3", "wav", "aiff"): if codec in lowered: @@ -153,7 +153,7 @@ class MusicPlanner: return "" @staticmethod - def _quality_numbers(value: str) -> tuple[int | None, int | None]: + def quality_numbers(value: str) -> tuple[int | None, int | None]: bit_depth = None sample_rate = None bit_match = re.search(r"(?P\d+)\s*[- ]?bit", value.lower()) @@ -165,14 +165,14 @@ class MusicPlanner: return bit_depth, sample_rate @staticmethod - def _bitrate_from_quality(value: str) -> int | None: + def bitrate_from_quality(value: str) -> int | None: match = re.search(r"(?P\d+)\s*kb/s", value.lower()) if not match: return None return int(match.group("rate")) * 1000 @staticmethod - def _quality_summary_from_options(options: list[MusicTrackOption]) -> str: + def quality_summary_from_options(options: list[MusicTrackOption]) -> str: if not options: return "" @@ -194,19 +194,19 @@ class MusicPlanner: return "AAC" if "MP3" in codecs: return "MP3" - return MusicPlanner._first_text(*labels) + return MusicPlanner.first_text(*labels) @staticmethod - def _sum_duration(music: Music) -> int | None: + def sum_duration(music: Music) -> int | None: total = 0 for song in music: data = song.data if isinstance(song.data, dict) else {} - value = MusicPlanner._first_number(data.get("duration")) + value = MusicPlanner.first_number(data.get("duration")) total += int(value or 0) return total or None @staticmethod - def _max_value(music: Music, attr: str) -> int | None: + def max_value(music: Music, attr: str) -> int | None: values = [getattr(song, attr, 0) or 0 for song in music] return max(values, default=0) or None diff --git a/unshackle/core/music/renderer.py b/unshackle/core/music/renderer.py index 65674aa0..af9d9faf 100644 --- a/unshackle/core/music/renderer.py +++ b/unshackle/core/music/renderer.py @@ -3,6 +3,7 @@ from __future__ import annotations import re from typing import Any, Optional +from rich import box from rich.console import Group, RenderableType from rich.markup import escape from rich.panel import Panel @@ -46,9 +47,9 @@ class MusicRenderer: kind = self.display_kind(music.kind) explicit = any(bool(getattr(song, "explicit", None)) for song in music) total_tracks = getattr(music, "total_tracks", None) or len(music) - total_discs = getattr(music, "total_discs", None) or self._max_value(music, "disc") - released = self._format_release_date( - self._first_value( + total_discs = getattr(music, "total_discs", None) or self.max_value(music, "disc") + released = self.format_release_date( + self.first_value( getattr(music, "released", None), getattr(music, "release_date", None), data.get("release_date"), @@ -57,23 +58,23 @@ class MusicRenderer: metadata.get("released_at"), ) ) - length = self._format_total_duration(getattr(music, "total_duration", None) or self._sum_duration(music)) - quality = self._quality_summary( - self._first_text( + length = self.format_total_duration(getattr(music, "total_duration", None) or self.sum_duration(music)) + quality = self.quality_summary( + self.first_text( getattr(music, "quality", None), data.get("quality"), metadata.get("quality"), metadata.get("quality_label"), ), - lossless=self._as_bool(self._first_value(data.get("lossless"), metadata.get("lossless"))), - hires=self._as_bool(self._first_value(data.get("hires"), metadata.get("hires"))), + lossless=self.as_bool(self.first_value(data.get("lossless"), metadata.get("lossless"))), + hires=self.as_bool(self.first_value(data.get("hires"), metadata.get("hires"))), ) - grid = self._metadata_grid() + grid = self.metadata_grid() grid.add_row(Text("Title", style="bright_black"), Text(str(title))) grid.add_row(Text("Artist", style="bright_black"), Text(str(artist))) - grid.add_row(Text("Type", style="bright_black"), self._kind_text(kind, explicit=explicit)) + grid.add_row(Text("Type", style="bright_black"), self.kind_text(kind, explicit=explicit)) if released: grid.add_row(Text("Released", style="bright_black"), Text(released)) if year: @@ -102,8 +103,8 @@ class MusicRenderer: visible_songs = visible_songs[: self.COMPACT_TRACK_LIMIT] for song in visible_songs: - node = tree.add(self._song_line(song, music), guide_style="bright_black") - option = self._option_from_song(song) + node = tree.add(self.song_line(song, music), guide_style="bright_black") + option = self.option_from_song(song) if option: node.add(option, guide_style="bright_black") @@ -112,7 +113,7 @@ class MusicRenderer: suffix = "s" if hidden != 1 else "" tree.add(f"[bright_black]... {hidden} more track{suffix}[/]", guide_style="bright_black") - return Panel(tree, title="Available Tracks") + return Panel(tree, title="Available Tracks", box=box.SQUARE, border_style="bright_black") def render_plan_header(self, plan: MusicDownloadPlan) -> Optional[Table]: title = plan.title @@ -123,16 +124,16 @@ class MusicRenderer: for disc in plan.discs for song_plan in disc.songs ) - released = self._format_release_date(getattr(plan, "released", None) or getattr(plan, "release_date", None)) - length = self._format_total_duration(plan.total_duration) - quality = self._quality_summary(plan.quality_requested) + released = self.format_release_date(getattr(plan, "released", None) or getattr(plan, "release_date", None)) + length = self.format_total_duration(plan.total_duration) + quality = self.quality_summary(plan.quality_requested) - grid = self._metadata_grid() + grid = self.metadata_grid() if title: grid.add_row(Text("Title", style="bright_black"), Text(str(title))) if artist: grid.add_row(Text("Artist", style="bright_black"), Text(str(artist))) - grid.add_row(Text("Type", style="bright_black"), self._kind_text(kind, explicit=explicit)) + grid.add_row(Text("Type", style="bright_black"), self.kind_text(kind, explicit=explicit)) if released: grid.add_row(Text("Released", style="bright_black"), Text(released)) if plan.year: @@ -162,22 +163,22 @@ class MusicRenderer: visible_songs = visible_songs[: self.COMPACT_TRACK_LIMIT] for song_plan in visible_songs: - node = tree.add(self._song_line(song_plan.song, plan), guide_style="bright_black") + node = tree.add(self.song_line(song_plan.song, plan), guide_style="bright_black") if song_plan.skip_reason: node.add(f"[yellow]Skipped:[/] {escape(song_plan.skip_reason)}", guide_style="bright_black") continue for option in song_plan.options: - node.add(self._option_line(option), guide_style="bright_black") + node.add(self.option_line(option), guide_style="bright_black") hidden = total - len(visible_songs) if hidden > 0: suffix = "s" if hidden != 1 else "" tree.add(f"[bright_black]... {hidden} more track{suffix}[/]", guide_style="bright_black") - return Panel(tree, title="Available Tracks") + return Panel(tree, title="Available Tracks", box=box.SQUARE, border_style="bright_black") @staticmethod - def _metadata_grid() -> Table: + def metadata_grid() -> Table: grid = Table.grid(padding=(0, 2)) grid.add_column(style="orchid1", no_wrap=True) grid.add_column() @@ -204,7 +205,7 @@ class MusicRenderer: return labels[key] return text.replace("_", " ").replace("-", " ").title() - def _song_line(self, song: Song, music: Music | MusicDownloadPlan) -> Text: + def song_line(self, song: Song, music: Music | MusicDownloadPlan) -> Text: number = f"{song.disc}.{song.track:02}" if song.disc > 1 else f"{song.track:02}" line = Text() line.append(number, style="repr.number") @@ -216,13 +217,13 @@ class MusicRenderer: line.append(f" - {song.artist}", style="bright_black") return line - def _option_from_song(self, song: Song) -> Text | str: + def option_from_song(self, song: Song) -> Text | str: data = song.data if isinstance(song.data, dict) else {} metadata = data.get("metadata") if isinstance(data.get("metadata"), dict) else {} - quality = self._first_text(data.get("quality"), metadata.get("quality"), metadata.get("quality_label")) - duration = self._format_duration(self._first_value(data.get("duration"), metadata.get("duration"))) - reason = self._first_text( + quality = self.first_text(data.get("quality"), metadata.get("quality"), metadata.get("quality_label")) + duration = self.format_duration(self.first_value(data.get("duration"), metadata.get("duration"))) + reason = self.first_text( data.get("unavailable_reason"), data.get("skip_reason"), metadata.get("unavailable_reason"), @@ -234,9 +235,9 @@ class MusicRenderer: badges = [] if song.explicit: badges.append(("E", "bold bright_red")) - if self._as_bool(self._first_value(data.get("atmos"), metadata.get("atmos"))): + if self.as_bool(self.first_value(data.get("atmos"), metadata.get("atmos"))): badges.append(("Atmos", "magenta")) - if self._is_hires_quality(quality): + if self.is_hires_quality(quality): badges.append(("Hi-Res", "gold1")) details = [] @@ -246,43 +247,43 @@ class MusicRenderer: details.append(duration) if not details and not badges: return "" - return self._format_option_text(details, badges) + return self.format_option_text(details, badges) - def _option_line(self, option: MusicTrackOption) -> Text: + def option_line(self, option: MusicTrackOption) -> Text: parts = [] codec = str(option.codec or "").strip() if codec: parts.append(f"[{codec}]") if option.quality_label: - parts.extend(self._split_quality_label(option.quality_label, codec)) + parts.extend(self.split_quality_label(option.quality_label, codec)) elif option.bit_depth and option.sample_rate: - parts.append(f"{option.bit_depth}-bit/{self._format_sample_rate(option.sample_rate)}") + parts.append(f"{option.bit_depth}-bit/{self.format_sample_rate(option.sample_rate)}") elif option.bitrate: parts.append(f"{int(option.bitrate / 1000)} kb/s") if option.channels: - parts.append(self._format_channels(option.channels)) + parts.append(self.format_channels(option.channels)) if option.duration: - parts.append(self._format_duration(option.duration)) + parts.append(self.format_duration(option.duration)) badges = [] if option.explicit: badges.append(("E", "bold bright_red")) if option.atmos: badges.append(("Atmos", "magenta")) - if self._is_cd_option(option): + if self.is_cd_option(option): badges.append(("CD", "yellow1")) if option.hires: badges.append(("Hi-Res", "gold1")) - return self._format_option_text(parts, badges) + return self.format_option_text(parts, badges) @staticmethod - def _kind_text(kind: str, *, explicit: bool = False) -> Text: + def kind_text(kind: str, *, explicit: bool = False) -> Text: text = Text(str(kind)) if explicit: text.append(" Explicit", style="bold red") return text @staticmethod - def _format_option_text(parts: list[str], badges: list[tuple[str, str]]) -> Text: + def format_option_text(parts: list[str], badges: list[tuple[str, str]]) -> Text: text = Text(style="text2") first = True for part in parts: @@ -300,13 +301,13 @@ class MusicRenderer: return text @staticmethod - def _format_option_parts(parts: list[str]) -> str: + def format_option_parts(parts: list[str]) -> str: if not parts: return "" return " | ".join(escape(part) for part in parts if part) @staticmethod - def _split_quality_label(value: str, codec: str = "") -> list[str]: + def split_quality_label(value: str, codec: str = "") -> list[str]: text = str(value or "").strip() if not text: return [] @@ -315,7 +316,7 @@ class MusicRenderer: return [text] if text else [] @staticmethod - def _is_cd_option(option: MusicTrackOption) -> bool: + def is_cd_option(option: MusicTrackOption) -> bool: codec = str(option.codec or "").upper() if codec not in {"FLAC", "ALAC", "WAV", "AIFF"}: return False @@ -324,7 +325,7 @@ class MusicRenderer: return "16-bit/44.1" in str(option.quality_label or "").lower() @staticmethod - def _format_sample_rate(value: Any) -> str: + def format_sample_rate(value: Any) -> str: try: sample_rate = float(value) except (TypeError, ValueError): @@ -336,7 +337,7 @@ class MusicRenderer: return f"{sample_rate:g} kHz" @staticmethod - def _format_channels(value: Any) -> str: + def format_channels(value: Any) -> str: try: channels = float(value) except (TypeError, ValueError): @@ -350,7 +351,7 @@ class MusicRenderer: return f"{channels:g}" @staticmethod - def _first_text(*values: Any) -> str: + def first_text(*values: Any) -> str: for value in values: if value is None: continue @@ -360,7 +361,7 @@ class MusicRenderer: if nested: return str(nested).strip() elif isinstance(value, (list, tuple)): - text = MusicRenderer._first_text(*value) + text = MusicRenderer.first_text(*value) if text: return text else: @@ -370,14 +371,14 @@ class MusicRenderer: return "" @staticmethod - def _first_value(*values: Any) -> Any: + def first_value(*values: Any) -> Any: for value in values: if value is not None and value != "": return value return None @staticmethod - def _as_bool(value: Any) -> bool: + def as_bool(value: Any) -> bool: if isinstance(value, bool): return value if isinstance(value, str): @@ -385,7 +386,7 @@ class MusicRenderer: return bool(value) @staticmethod - def _format_duration(value: Any) -> str: + def format_duration(value: Any) -> str: if value in (None, ""): return "" try: @@ -399,7 +400,7 @@ class MusicRenderer: return f"{minutes}:{seconds:02}" @staticmethod - def _format_total_duration(value: Any) -> str: + def format_total_duration(value: Any) -> str: if value in (None, ""): return "" try: @@ -415,7 +416,7 @@ class MusicRenderer: return f"{minutes}m {seconds:02}s" @staticmethod - def _format_release_date(value: Any) -> str: + def format_release_date(value: Any) -> str: if value in (None, ""): return "" text = str(value).strip() @@ -449,7 +450,7 @@ class MusicRenderer: return f"{month_names[month - 1]} {day}, {year}" @classmethod - def _quality_summary(cls, value: Any, *, lossless: bool = False, hires: bool = False) -> str: + def quality_summary(cls, value: Any, *, lossless: bool = False, hires: bool = False) -> str: text = str(value or "").strip() lowered = text.lower() if not text: @@ -461,7 +462,7 @@ class MusicRenderer: if "atmos" in lowered: return "Dolby Atmos" if any(codec in lowered for codec in ("flac", "alac", "wav", "aiff")): - return "Hi-Res Lossless" if cls._is_hires_quality(text) else "Lossless" + return "Hi-Res Lossless" if cls.is_hires_quality(text) else "Lossless" if "lossless" in lowered: return "Hi-Res Lossless" if "hi-res" in lowered or hires else "Lossless" if "aac" in lowered: @@ -471,7 +472,7 @@ class MusicRenderer: return text @staticmethod - def _is_hires_quality(value: str) -> bool: + def is_hires_quality(value: str) -> bool: lowered = value.lower() bit_depth = None sample_rate = None @@ -487,12 +488,12 @@ class MusicRenderer: return bool((bit_depth and bit_depth > 16) or (sample_rate and sample_rate > 48)) @staticmethod - def _sum_duration(music: Music) -> int: + def sum_duration(music: Music) -> int: total = 0 for song in music: data = song.data if isinstance(song.data, dict) else {} metadata = data.get("metadata") if isinstance(data.get("metadata"), dict) else {} - value = MusicRenderer._first_value(data.get("duration"), metadata.get("duration")) + value = MusicRenderer.first_value(data.get("duration"), metadata.get("duration")) try: total += int(float(value or 0)) except (TypeError, ValueError): @@ -500,7 +501,7 @@ class MusicRenderer: return total @staticmethod - def _max_value(music: Music, attr: str) -> int: + def max_value(music: Music, attr: str) -> int: values = [getattr(song, attr, 0) or 0 for song in music] return max(values, default=0) diff --git a/unshackle/core/music/tagger.py b/unshackle/core/music/tagger.py index f756f15b..84b1e9e4 100644 --- a/unshackle/core/music/tagger.py +++ b/unshackle/core/music/tagger.py @@ -74,49 +74,49 @@ def write_music_metadata(path: Path, song: Any, *, session: Any = None, source_m if extension in {".m4a", ".mp4"} and MP4 is None: return MusicMetadataResult(skipped=True, reason="install mutagen to write MP4/M4A tags") - tags = _build_tag_values(song, source_md5=source_md5) - artwork_url = _first_text(getattr(song, "artwork_url", None), _metadata(song).get("artwork_url")) - cover_data, mime_type = _download_cover(session, artwork_url) + tags = build_tag_values(song, source_md5=source_md5) + artwork_url = first_text(getattr(song, "artwork_url", None), _metadata(song).get("artwork_url")) + cover_data, mime_type = download_cover(session, artwork_url) if extension == ".flac": - _write_flac_tags(path, tags, cover_data, mime_type) + write_flac_tags(path, tags, cover_data, mime_type) elif extension == ".mp3": - _write_mp3_tags(path, tags, cover_data, mime_type) + write_mp3_tags(path, tags, cover_data, mime_type) else: - _write_mp4_tags(path, tags, cover_data, mime_type) + write_mp4_tags(path, tags, cover_data, mime_type) return MusicMetadataResult(written=True, artwork_embedded=bool(cover_data)) -def _build_tag_values(song: Any, *, source_md5: str = "") -> dict[str, str]: +def build_tag_values(song: Any, *, source_md5: str = "") -> dict[str, str]: metadata = _metadata(song) - track_number = _string_tag(getattr(song, "track", None) or metadata.get("track_number")) - track_total = _string_tag(getattr(song, "total_tracks", None) or metadata.get("total_tracks")) - disc_number = _string_tag(getattr(song, "disc", None) or metadata.get("disc_number")) - disc_total = _string_tag(getattr(song, "total_discs", None) or metadata.get("total_discs")) - explicit = _as_bool(getattr(song, "explicit", None), metadata.get("explicit"), metadata.get("parental_warning")) + track_number = string_tag(getattr(song, "track", None) or metadata.get("track_number")) + track_total = string_tag(getattr(song, "total_tracks", None) or metadata.get("total_tracks")) + disc_number = string_tag(getattr(song, "disc", None) or metadata.get("disc_number")) + disc_total = string_tag(getattr(song, "total_discs", None) or metadata.get("total_discs")) + explicit = as_bool(getattr(song, "explicit", None), metadata.get("explicit"), metadata.get("parental_warning")) tags = { - "TITLE": _first_text(getattr(song, "name", None), metadata.get("title")), - "ARTIST": _first_text(getattr(song, "artist", None), metadata.get("artist"), metadata.get("performer")), - "ALBUM": _first_text(getattr(song, "album", None), metadata.get("album")), - "ALBUMARTIST": _first_text(getattr(song, "album_artist", None), metadata.get("album_artist")), + "TITLE": first_text(getattr(song, "name", None), metadata.get("title")), + "ARTIST": first_text(getattr(song, "artist", None), metadata.get("artist"), metadata.get("performer")), + "ALBUM": first_text(getattr(song, "album", None), metadata.get("album")), + "ALBUMARTIST": first_text(getattr(song, "album_artist", None), metadata.get("album_artist")), "TRACKNUMBER": f"{track_number}/{track_total}" if track_number and track_total else track_number, "TRACKTOTAL": track_total, "DISCNUMBER": f"{disc_number}/{disc_total}" if disc_number and disc_total else disc_number, "DISCTOTAL": disc_total, - "DATE": _string_tag(metadata.get("release_date") or metadata.get("year") or getattr(song, "year", None)), - "RELEASEDATE": _string_tag(metadata.get("release_date")), - "GENRE": _first_text(getattr(song, "genre", None), metadata.get("genre")), - "COMPOSER": _first_text(metadata.get("composer")), - "PERFORMER": _first_text(metadata.get("performer")), - "ISRC": _string_tag(getattr(song, "isrc", None) or metadata.get("isrc")), - "BARCODE": _string_tag(getattr(song, "upc", None) or metadata.get("upc") or metadata.get("barcode")), - "UPC": _string_tag(getattr(song, "upc", None) or metadata.get("upc") or metadata.get("barcode")), - "COPYRIGHT": _string_tag(getattr(song, "copyright", None) or metadata.get("copyright")), - "LABEL": _first_text(getattr(song, "label", None), metadata.get("label")), - "COMMENT": _first_text(metadata.get("comment"), metadata.get("quality")), - "SOURCE": _first_text(metadata.get("source"), metadata.get("service")), + "DATE": string_tag(metadata.get("release_date") or metadata.get("year") or getattr(song, "year", None)), + "RELEASEDATE": string_tag(metadata.get("release_date")), + "GENRE": first_text(getattr(song, "genre", None), metadata.get("genre")), + "COMPOSER": first_text(metadata.get("composer")), + "PERFORMER": first_text(metadata.get("performer")), + "ISRC": string_tag(getattr(song, "isrc", None) or metadata.get("isrc")), + "BARCODE": string_tag(getattr(song, "upc", None) or metadata.get("upc") or metadata.get("barcode")), + "UPC": string_tag(getattr(song, "upc", None) or metadata.get("upc") or metadata.get("barcode")), + "COPYRIGHT": string_tag(getattr(song, "copyright", None) or metadata.get("copyright")), + "LABEL": first_text(getattr(song, "label", None), metadata.get("label")), + "COMMENT": first_text(metadata.get("comment"), metadata.get("quality")), + "SOURCE": first_text(metadata.get("source"), metadata.get("service")), "ENCODEDBY": "Unshackle", "UNSHACKLE_SOURCE_MD5": source_md5, } @@ -126,22 +126,22 @@ def _build_tag_values(song: Any, *, source_md5: str = "") -> dict[str, str]: if config.tag and config.tag_group_name: tags["GROUP"] = config.tag - service = _first_text(metadata.get("service"), metadata.get("source")) + service = first_text(metadata.get("service"), metadata.get("source")) if service: prefix = service.upper().replace(" ", "_").replace("-", "_") if metadata.get("track_id"): - tags[f"{prefix}_TRACK_ID"] = _string_tag(metadata.get("track_id")) + tags[f"{prefix}_TRACK_ID"] = string_tag(metadata.get("track_id")) if metadata.get("album_id"): - tags[f"{prefix}_ALBUM_ID"] = _string_tag(metadata.get("album_id")) + tags[f"{prefix}_ALBUM_ID"] = string_tag(metadata.get("album_id")) if metadata.get("track_url"): - tags[f"{prefix}_TRACK_URL"] = _string_tag(metadata.get("track_url")) + tags[f"{prefix}_TRACK_URL"] = string_tag(metadata.get("track_url")) if metadata.get("album_url"): - tags[f"{prefix}_ALBUM_URL"] = _string_tag(metadata.get("album_url")) + tags[f"{prefix}_ALBUM_URL"] = string_tag(metadata.get("album_url")) return {key: value for key, value in tags.items() if value} -def _write_flac_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes], mime_type: str) -> None: +def write_flac_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes], mime_type: str) -> None: audio = FLAC(path) for key, value in tags.items(): audio[key] = [value] @@ -163,7 +163,7 @@ def _write_flac_tags(path: Path, tags: dict[str, str], cover_data: Optional[byte audio.save() -def _write_mp3_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes], mime_type: str) -> None: +def write_mp3_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes], mime_type: str) -> None: try: audio = MP3(path) except ID3NoHeaderError: @@ -203,7 +203,7 @@ def _write_mp3_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes audio.save() -def _write_mp4_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes], mime_type: str) -> None: +def write_mp4_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes], mime_type: str) -> None: audio = MP4(path) if tags.get("TITLE"): audio["\xa9nam"] = [tags["TITLE"]] @@ -224,10 +224,10 @@ def _write_mp4_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes if tags.get("COPYRIGHT"): audio["cprt"] = [tags["COPYRIGHT"]] - track_number, track_total = _split_number_pair(tags.get("TRACKNUMBER", "")) + track_number, track_total = split_number_pair(tags.get("TRACKNUMBER", "")) if track_number: audio["trkn"] = [(track_number, track_total or 0)] - disc_number, disc_total = _split_number_pair(tags.get("DISCNUMBER", "")) + disc_number, disc_total = split_number_pair(tags.get("DISCNUMBER", "")) if disc_number: audio["disk"] = [(disc_number, disc_total or 0)] @@ -254,7 +254,7 @@ def _write_mp4_tags(path: Path, tags: dict[str, str], cover_data: Optional[bytes audio.save() -def _download_cover(session: Any, artwork_url: str) -> tuple[Optional[bytes], str]: +def download_cover(session: Any, artwork_url: str) -> tuple[Optional[bytes], str]: if not session or not artwork_url: return None, "" try: @@ -264,7 +264,7 @@ def _download_cover(session: Any, artwork_url: str) -> tuple[Optional[bytes], st if not data: return None, "" content_type = str(response.headers.get("Content-Type") or "").split(";")[0].strip().lower() - return data, _mime_from_image(data, content_type or "image/jpeg") + return data, mime_from_image(data, content_type or "image/jpeg") except Exception as error: log.debug("Music cover download failed for %s: %s", artwork_url, error) return None, "" @@ -275,14 +275,14 @@ def _metadata(song: Any) -> dict[str, Any]: return data if isinstance(data, dict) else {} -def _first_text(*values: Any) -> str: +def first_text(*values: Any) -> str: for value in values: if isinstance(value, dict): - text = _first_text(value.get("name"), value.get("title"), value.get("display_name"), value.get("value")) + text = first_text(value.get("name"), value.get("title"), value.get("display_name"), value.get("value")) if text: return text elif isinstance(value, list): - parts = [_first_text(item) for item in value] + parts = [first_text(item) for item in value] text = ", ".join(part for part in parts if part) if text: return text @@ -293,7 +293,7 @@ def _first_text(*values: Any) -> str: return "" -def _string_tag(value: Any) -> str: +def string_tag(value: Any) -> str: if value in (None, "", [], {}): return "" if isinstance(value, bool): @@ -301,7 +301,7 @@ def _string_tag(value: Any) -> str: return str(value).strip() -def _as_bool(*values: Any) -> bool: +def as_bool(*values: Any) -> bool: for value in values: if value in (None, "", [], {}): continue @@ -317,7 +317,7 @@ def _as_bool(*values: Any) -> bool: return False -def _mime_from_image(data: bytes, fallback: str = "image/jpeg") -> str: +def mime_from_image(data: bytes, fallback: str = "image/jpeg") -> str: if data.startswith(b"\x89PNG\r\n\x1a\n"): return "image/png" if data.startswith(b"\xff\xd8\xff"): @@ -327,7 +327,7 @@ def _mime_from_image(data: bytes, fallback: str = "image/jpeg") -> str: return fallback -def _split_number_pair(value: str) -> tuple[int, int]: +def split_number_pair(value: str) -> tuple[int, int]: if not value: return 0, 0 first, _, second = str(value).partition("/") diff --git a/unshackle/core/providers/__init__.py b/unshackle/core/providers/__init__.py index 157196bb..9c4160cf 100644 --- a/unshackle/core/providers/__init__.py +++ b/unshackle/core/providers/__init__.py @@ -1,11 +1,12 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Optional +from typing import TYPE_CHECKING, Optional, Union import requests from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, fuzzy_match, log -from unshackle.core.providers.imdbapi import IMDBApiProvider +from unshackle.core.providers.anilist import AniListProvider, parse_anilist_ref +from unshackle.core.providers.imdb import IMDBProvider from unshackle.core.providers.omdb import OMDBProvider from unshackle.core.providers.simkl import SimklProvider from unshackle.core.providers.tmdb import TMDBProvider @@ -15,18 +16,33 @@ if TYPE_CHECKING: from unshackle.core.title_cacher import TitleCacher REGISTRY: dict[str, type[MetadataProvider]] = { - cls.NAME: cls for cls in (IMDBApiProvider, OMDBProvider, SimklProvider, TMDBProvider, TVDBProvider) + cls.NAME: cls for cls in (AniListProvider, IMDBProvider, OMDBProvider, SimklProvider, TMDBProvider, TVDBProvider) } -# used when `metadata_providers` is unset -DEFAULT_ORDER: tuple[str, ...] = ("imdbapi", "omdb", "simkl", "tmdb", "tvdb") +# legacy `metadata_providers` names, still accepted +ALIASES: dict[str, str] = {"imdbapi": "imdb"} + +# used when `metadata_providers` is unset; anilist answers for anime only, so it costs +# nothing at the end of the order +DEFAULT_ORDER: tuple[str, ...] = ("imdb", "omdb", "simkl", "tmdb", "tvdb", "anilist") -def provider_order() -> list[type[MetadataProvider]]: - """Provider classes in the order configured by `metadata_providers`.""" +def provider_order(kind: Optional[str] = None, anime: bool = False) -> list[type[MetadataProvider]]: + """Provider classes in the order configured by `metadata_providers`. + + `metadata_providers` is either a flat list applying to both kinds, or a mapping of + kind ("tv"/"movie") to its own list. A kind the mapping omits uses `DEFAULT_ORDER`. + An `anime` title puts anilist first; the rest of the order stays behind it to + fall back on. + """ from unshackle.core.config import config - names = [str(n).lower() for n in (config.metadata_providers or DEFAULT_ORDER)] + configured = config.metadata_providers + selected = (configured.get(kind) if kind else None) if isinstance(configured, dict) else configured + + names = [ALIASES.get(str(n).lower(), str(n).lower()) for n in (selected or DEFAULT_ORDER)] + if anime and AniListProvider.NAME in names: + names = [AniListProvider.NAME, *names] unknown = [n for n in names if n not in REGISTRY] if unknown: log.warning("Ignoring unknown metadata_providers entries: %s", ", ".join(unknown)) @@ -58,9 +74,18 @@ def search_metadata( cache_title_id: Optional[str] = None, cache_region: Optional[str] = None, cache_account_hash: Optional[str] = None, + anime: bool = False, ) -> Optional[MetadataResult]: """Search all available providers for metadata. Returns best match.""" - ordered = provider_order() + from unshackle.core.config import config + + # the one gate for `disable_metadata`: every automatic lookup reaches a provider through + # here, while a user-supplied ID goes straight to `resolve_by_ids` and stays allowed + if config.disable_metadata: + log.debug("Metadata lookups are disabled by config; not searching for %r", title) + return None + + ordered = provider_order(kind, anime) # Check cache first if title_cacher and cache_title_id: @@ -70,7 +95,7 @@ def search_metadata( continue cached = title_cacher.get_cached_provider(p.NAME, cache_title_id, kind, cache_region, cache_account_hash) if cached: - result = _cached_to_result(cached, p.NAME, kind) + result = cached_to_result(cached, p.NAME, kind) if result and result.title and fuzzy_match(result.title, title): log.debug("Using cached %s data for %r", p.NAME, title) return result @@ -95,7 +120,7 @@ def search_metadata( if result.external_ids.tmdb_id or result.external_ids.tvdb_id: cache_data = { **result.raw, - "_enriched_ids": _external_ids_to_dict(result.external_ids), + "_enriched_ids": external_ids_to_dict(result.external_ids), } title_cacher.cache_provider( p.NAME, cache_title_id, cache_data, kind, cache_region, cache_account_hash @@ -140,7 +165,7 @@ def get_title_by_id( title_cacher.cache_provider( "tmdb", cache_title_id, - {"detail": result.raw, "external_ids": _external_ids_to_dict(ext_ids)}, + {"detail": result.raw, "external_ids": external_ids_to_dict(ext_ids)}, kind, cache_region, cache_account_hash, @@ -185,7 +210,7 @@ def get_year_by_id( title_cacher.cache_provider( "tmdb", cache_title_id, - {"detail": result.raw, "external_ids": _external_ids_to_dict(ext_ids)}, + {"detail": result.raw, "external_ids": external_ids_to_dict(ext_ids)}, kind, cache_region, cache_account_hash, @@ -196,6 +221,29 @@ def get_year_by_id( return result.year +def get_language_by_id( + tmdb_id: int, + kind: str, + title_cacher: Optional[TitleCacher] = None, + cache_title_id: Optional[str] = None, + cache_region: Optional[str] = None, + cache_account_hash: Optional[str] = None, +) -> Optional[str]: + """Get original language by TMDB ID.""" + if title_cacher and cache_title_id: + cached = title_cacher.get_cached_provider("tmdb", cache_title_id, kind, cache_region, cache_account_hash) + language = ((cached or {}).get("detail") or {}).get("original_language") + if language: + log.debug("Using cached TMDB original language: %s", language) + return language + + tmdb = get_provider("tmdb") + if not tmdb: + return None + result = tmdb.get_by_id(tmdb_id, kind) + return result.original_language if result else None + + def fetch_external_ids( tmdb_id: int, kind: str, @@ -234,7 +282,7 @@ def fetch_external_ids( title_cacher.cache_provider( "tmdb", cache_title_id, - {"detail": detail, "external_ids": _external_ids_to_dict(ext)}, + {"detail": detail, "external_ids": external_ids_to_dict(ext)}, kind, cache_region, cache_account_hash, @@ -245,18 +293,98 @@ def fetch_external_ids( return ext +def resolve_by_ids( + tmdb_id: Optional[int] = None, + imdb_id: Optional[str] = None, + tvdb_id: Optional[int] = None, + anilist_id: Optional[Union[int, str]] = None, + *, + title: Optional[str] = None, + year: Optional[int] = None, + kind: str = "movie", + title_cacher: Optional[TitleCacher] = None, + cache_title_id: Optional[str] = None, + cache_region: Optional[str] = None, + cache_account_hash: Optional[str] = None, + anime: bool = False, +) -> Optional[MetadataResult]: + """Resolve metadata from user-supplied external IDs, falling back to search only without them. + + A supplied ID is authoritative. It is looked up directly through the providers that + consume its namespace, in `metadata_providers` order for `kind`, and it always survives + into `external_ids` whatever a provider answers. A fuzzy title search only runs when no + ID was supplied at all. + """ + supplied: dict[str, Union[int, str]] = {} + if tmdb_id is not None: + supplied["tmdb"] = tmdb_id + if imdb_id: + supplied["imdb"] = imdb_id + if tvdb_id is not None: + supplied["tvdb"] = tvdb_id + if anilist_id is not None: + supplied["anilist"] = anilist_id + + if not supplied: + if not title: + return None + return search_metadata(title, year, kind, title_cacher, cache_title_id, cache_region, cache_account_hash, anime) + + result: Optional[MetadataResult] = None + for cls in provider_order(kind, anime): + provider_id = supplied.get(cls.ID_KIND or "") + if provider_id is None: + continue + p = get_provider(cls.NAME) + if not p: + continue + try: + result = p.get_by_id(provider_id, kind) + except Exception as exc: + log.debug("%s lookup of %s failed: %s", cls.NAME, provider_id, exc) + continue + if result: + break + + if not result: + result = MetadataResult(title=title, year=year, kind=kind) + + ids = result.external_ids + if tmdb_id is not None: + ids.tmdb_id = tmdb_id + ids.tmdb_kind = kind + if imdb_id: + ids.imdb_id = imdb_id + if tvdb_id is not None: + ids.tvdb_id = tvdb_id + if anilist_id is not None and ids.anilist_id is None: + ref = parse_anilist_ref(anilist_id) + if ref and ref[0] == "id": + ids.anilist_id = ref[1] + + if ids.tmdb_id and not (ids.imdb_id and ids.tvdb_id): + ext = fetch_external_ids(ids.tmdb_id, kind, title_cacher, cache_title_id, cache_region, cache_account_hash) + if ext.imdb_id and not ids.imdb_id: + ids.imdb_id = ext.imdb_id + if ext.tvdb_id and not ids.tvdb_id: + ids.tvdb_id = ext.tvdb_id + enrich_ids(result) + + return result + + # -- Internal helpers -- # trust ranking for cross-validating enrichments; `metadata_providers` filters this # set but sets search order only, not the ranking -_ENRICHMENT_AUTHORITY: tuple[str, ...] = ("tmdb", "simkl", "tvdb") +ENRICHMENT_AUTHORITY: tuple[str, ...] = ("tmdb", "simkl", "tvdb") -def _enrichment_providers() -> list[str]: +def enrichment_providers(kind: Optional[str] = None) -> list[str]: """Names of configured providers that can resolve an IMDB ID, most trusted first.""" - configured = {cls.NAME for cls in provider_order() if hasattr(cls, "find_by_imdb_id")} - return [name for name in _ENRICHMENT_AUTHORITY if name in configured] + configured = {cls.NAME for cls in provider_order(kind) if hasattr(cls, "find_by_imdb_id")} + return [name for name in ENRICHMENT_AUTHORITY if name in configured] def enrich_ids(result: MetadataResult) -> None: @@ -275,7 +403,7 @@ def enrich_ids(result: MetadataResult) -> None: kind = result.kind or "movie" # Step 1: Collect enrichment results from all available providers - authority = {name: i for i, name in enumerate(_enrichment_providers())} + authority = {name: i for i, name in enumerate(enrichment_providers(kind))} enrichments: list[tuple[str, ExternalIds]] = [] for provider_name in authority: p = get_provider(provider_name) @@ -293,7 +421,7 @@ def enrich_ids(result: MetadataResult) -> None: return # Step 2: Cross-validate using tmdb_id as anchor — drop providers that disagree - validated = _validate_enrichments(enrichments, authority) + validated = validate_enrichments(enrichments, authority) # Step 3: Merge validated data (fill gaps only) for _provider_name, ext in validated: @@ -304,7 +432,7 @@ def enrich_ids(result: MetadataResult) -> None: ids.tvdb_id = ext.tvdb_id -def _validate_enrichments( +def validate_enrichments( enrichments: list[tuple[str, ExternalIds]], authority: dict[str, int], ) -> list[tuple[str, ExternalIds]]: @@ -356,7 +484,7 @@ def _validate_enrichments( return validated -def _external_ids_to_dict(ext: ExternalIds) -> dict: +def external_ids_to_dict(ext: ExternalIds) -> dict: """Convert ExternalIds to a dict for caching.""" result: dict = {} if ext.imdb_id: @@ -367,10 +495,12 @@ def _external_ids_to_dict(ext: ExternalIds) -> dict: result["tmdb_kind"] = ext.tmdb_kind if ext.tvdb_id: result["tvdb_id"] = ext.tvdb_id + if ext.anilist_id: + result["anilist_id"] = ext.anilist_id return result -def _cached_to_result(cached: dict, provider_name: str, kind: str) -> Optional[MetadataResult]: +def cached_to_result(cached: dict, provider_name: str, kind: str) -> Optional[MetadataResult]: """Convert a cached provider dict back to a MetadataResult.""" if provider_name == "tmdb": detail = cached.get("detail", {}) @@ -436,10 +566,10 @@ def _cached_to_result(cached: dict, provider_name: str, kind: str) -> Optional[M raw=cached, ) elif provider_name == "tvdb": - from unshackle.core.providers.tvdb import _ids_from_remote, _parse_int + from unshackle.core.providers.tvdb import ids_from_remote, parse_int - tvdb_id = _parse_int(cached.get("tvdb_id") or cached.get("id")) - ext = _ids_from_remote(cached.get("remote_ids") or cached.get("remoteIds"), tvdb_id) + tvdb_id = parse_int(cached.get("tvdb_id") or cached.get("id")) + ext = ids_from_remote(cached.get("remote_ids") or cached.get("remoteIds"), tvdb_id) # restore IDs that enrichment filled in beyond the raw remote_ids enriched = cached.get("_enriched_ids", {}) ext.imdb_id = ext.imdb_id or enriched.get("imdb_id") @@ -448,15 +578,19 @@ def _cached_to_result(cached: dict, provider_name: str, kind: str) -> Optional[M ext.tmdb_kind = kind return MetadataResult( title=cached.get("name"), - year=_parse_int(cached.get("year")), + year=parse_int(cached.get("year")), kind=kind, external_ids=ext, source="tvdb", raw=cached, ) - elif provider_name == "imdbapi": - title = cached.get("primaryTitle") or cached.get("originalTitle") - year = cached.get("startYear") + elif provider_name == "anilist": + return AniListProvider().to_result(cached) + elif provider_name == "imdb": + from unshackle.core.providers.imdb import primary_language + + title = (cached.get("titleText") or {}).get("text") or (cached.get("originalTitleText") or {}).get("text") + year = (cached.get("releaseYear") or {}).get("year") imdb_id = cached.get("id") # Restore enriched IDs that were saved alongside the raw data enriched = cached.get("_enriched_ids", {}) @@ -470,7 +604,8 @@ def _cached_to_result(cached: dict, provider_name: str, kind: str) -> Optional[M tmdb_kind=enriched.get("tmdb_kind"), tvdb_id=enriched.get("tvdb_id"), ), - source="imdbapi", + original_language=primary_language(cached), + source="imdb", raw=cached, ) return None @@ -490,5 +625,6 @@ __all__ = [ "get_title_by_id", "get_year_by_id", "provider_order", + "resolve_by_ids", "search_metadata", ] diff --git a/unshackle/core/providers/_base.py b/unshackle/core/providers/_base.py index 62227869..d2a29689 100644 --- a/unshackle/core/providers/_base.py +++ b/unshackle/core/providers/_base.py @@ -26,6 +26,7 @@ class ExternalIds: tmdb_id: Optional[int] = None tmdb_kind: Optional[str] = None # "movie" or "tv" tvdb_id: Optional[int] = None + anilist_id: Optional[int] = None @dataclass @@ -36,7 +37,8 @@ class MetadataResult: year: Optional[int] = None kind: Optional[str] = None # "movie" or "tv" external_ids: ExternalIds = field(default_factory=ExternalIds) - source: str = "" # provider name, e.g. "tmdb", "simkl", "imdbapi" + original_language: Optional[str] = None # alpha-2 or alpha-3, whichever the provider speaks + source: str = "" # provider name, e.g. "tmdb", "simkl", "imdb" raw: Optional[dict] = None # original API response for caching @@ -45,6 +47,7 @@ class MetadataProvider(metaclass=ABCMeta): NAME: str = "" REQUIRES_KEY: bool = True + ID_KIND: Optional[str] = None def __init__(self) -> None: self.log = logging.getLogger(f"METADATA.{self.NAME.upper()}") @@ -83,15 +86,15 @@ class MetadataProvider(metaclass=ABCMeta): """Fetch external IDs for a title by this provider's native ID.""" -def _clean(s: str) -> str: +def clean(s: str) -> str: return STRIP_RE.sub("", s).lower() -def _strip_year(s: str) -> str: +def strip_year(s: str) -> str: return YEAR_RE.sub("", s).strip() def fuzzy_match(a: str, b: str, threshold: float = 0.8) -> bool: """Return True if ``a`` and ``b`` are a close match.""" - ratio = SequenceMatcher(None, _clean(a), _clean(b)).ratio() + ratio = SequenceMatcher(None, clean(a), clean(b)).ratio() return ratio >= threshold diff --git a/unshackle/core/providers/anilist.py b/unshackle/core/providers/anilist.py new file mode 100644 index 00000000..8b7d3f13 --- /dev/null +++ b/unshackle/core/providers/anilist.py @@ -0,0 +1,152 @@ +from __future__ import annotations + +from difflib import SequenceMatcher +from typing import Optional, Union + +import requests + +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, clean + +GRAPHQL_URL = "https://graphql.anilist.co" + +MEDIA_FIELDS = "id idMal title { romaji english native } format startDate { year } countryOfOrigin" + +SEARCH_QUERY = ( + "query UnshackleAniListSearch($search: String) { Page(perPage: 10) " + "{ media(search: $search, type: ANIME, sort: SEARCH_MATCH) { %s } } }" % MEDIA_FIELDS +) +MEDIA_QUERY = "query UnshackleAniListMedia($id: Int) { Media(id: $id, type: ANIME) { %s } }" % MEDIA_FIELDS +MAL_QUERY = "query UnshackleAniListMal($idMal: Int) { Media(idMal: $idMal, type: ANIME) { %s } }" % MEDIA_FIELDS + +TITLE_LANGUAGES: tuple[str, ...] = ("english", "romaji", "native") + +# AniList records where a title was produced, not what it is spoken in; these are the +# production countries whose anime output is reliably in the country's own language +COUNTRY_TO_LANGUAGE: dict[str, str] = {"JP": "ja", "KR": "ko", "CN": "zh", "TW": "zh", "HK": "zh"} + + +def parse_anilist_ref(value: Union[int, str]) -> Optional[tuple[str, int]]: + """Split an AniList reference into its namespace and number. + + A bare number is an AniList ID; ``mal:12345`` is a MyAnimeList ID, which AniList + resolves natively. Returns None for anything else. + """ + text = str(value).strip().lower() + namespace = "id" + if ":" in text: + prefix, _, text = text.partition(":") + if prefix != "mal": + return None + namespace = "mal" + # isdecimal, not isdigit: superscripts pass isdigit but crash int() + if not text.isdecimal(): + return None + number = int(text) + return (namespace, number) if number > 0 else None + + +class AniListProvider(MetadataProvider): + """Anime metadata provider using AniList's public GraphQL API (no API key).""" + + NAME = "anilist" + REQUIRES_KEY = False + ID_KIND = "anilist" + + def is_available(self) -> bool: + return True + + def graphql(self, query: str, variables: dict) -> Optional[dict]: + try: + r = self.session.post(GRAPHQL_URL, json={"query": query, "variables": variables}, timeout=30) + body = r.json() + except (requests.RequestException, ValueError) as exc: + self.log.debug("AniList GraphQL request failed: %s", exc) + return None + + if body.get("errors"): + # a miss is a 404 with an error body, not an empty result + self.log.debug("AniList GraphQL errors: %s", [e.get("message") for e in body["errors"]][:3]) + return None + return body.get("data") + + def title_variant(self, titles: dict) -> Optional[str]: + from unshackle.core.config import config + + configured = (config.anilist_title_language or "english").lower() + for variant in [configured, *(v for v in TITLE_LANGUAGES if v != configured)]: + value = titles.get(variant) + if value: + return str(value) + return None + + def to_result(self, node: dict) -> Optional[MetadataResult]: + title = self.title_variant(node.get("title") or {}) + if not title: + return None + return MetadataResult( + title=title, + year=(node.get("startDate") or {}).get("year"), + kind="movie" if node.get("format") == "MOVIE" else "tv", + external_ids=ExternalIds(anilist_id=node.get("id")), + original_language=COUNTRY_TO_LANGUAGE.get(node.get("countryOfOrigin") or ""), + source="anilist", + raw=node, + ) + + def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: + self.log.debug("Searching AniList for %r (%s, %s)", title, kind, year) + + data = self.graphql(SEARCH_QUERY, {"search": title}) + nodes = ((data or {}).get("Page") or {}).get("media") or [] + if not nodes: + self.log.debug("AniList returned no results for %r", title) + return None + + # soft filters with fallback: other providers search kind-scoped endpoints, here + # one ANIME search covers both kinds, so a TV hit must not shadow a movie search + matching = [n for n in nodes if (n.get("format") == "MOVIE") == (kind == "movie")] + nodes = matching or nodes + if year: + dated = [n for n in nodes if (n.get("startDate") or {}).get("year") == year] + nodes = dated or nodes + + best = max(nodes, key=lambda n: match_ratio(n, title)) + result = self.to_result(best) + if result: + self.log.debug("AniList -> %s (ID %s)", result.title, result.external_ids.anilist_id) + return result + + def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: + """Fetch metadata by AniList ID, or by MyAnimeList ID when given as 'mal:12345'.""" + ref = parse_anilist_ref(provider_id) + if not ref: + self.log.debug("Not an AniList reference: %r", provider_id) + return None + + namespace, number = ref + self.log.debug("Fetching AniList media %s:%d", namespace, number) + if namespace == "mal": + data = self.graphql(MAL_QUERY, {"idMal": number}) + else: + data = self.graphql(MEDIA_QUERY, {"id": number}) + + node = (data or {}).get("Media") + if not node: + return None + return self.to_result(node) + + def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: + """Return external IDs. AniList knows no western IDs, so only its own.""" + ref = parse_anilist_ref(provider_id) + if not ref: + return ExternalIds() + namespace, number = ref + if namespace == "id": + return ExternalIds(anilist_id=number) + result = self.get_by_id(provider_id, kind) + return result.external_ids if result else ExternalIds() + + +def match_ratio(node: dict, title: str) -> float: + names = (node.get("title") or {}).values() + return max((SequenceMatcher(None, clean(title), clean(n)).ratio() for n in names if n), default=0.0) diff --git a/unshackle/core/providers/imdb.py b/unshackle/core/providers/imdb.py new file mode 100644 index 00000000..5b42f440 --- /dev/null +++ b/unshackle/core/providers/imdb.py @@ -0,0 +1,156 @@ +from __future__ import annotations + +import hashlib +import json +from difflib import SequenceMatcher +from typing import Optional, Union + +import requests + +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, clean, fuzzy_match + +GRAPHQL_URL = "https://caching.graphql.imdb.com/" + +GRAPHQL_HEADERS = { + "Accept": "application/graphql+json, application/json", + "Content-Type": "application/json", + "Origin": "https://www.imdb.com", + "X-Imdb-Client-Name": "imdb-web-next-localized", + "X-Imdb-User-Country": "US", +} + +KIND_TO_TYPES: dict[str, list[str]] = { + "movie": ["movie", "tvMovie"], + "tv": ["tvSeries", "tvMiniSeries"], +} + +TITLE_FIELDS = ( + "id titleText { text } originalTitleText { text } releaseYear { year } " + "spokenLanguages { spokenLanguages { id text } }" +) + + +def escape_graphql(value: str) -> str: + return value.replace("\\", "\\\\").replace('"', '\\"') + + +def primary_language(data: dict) -> Optional[str]: + """IMDb orders spokenLanguages most prominent first. Either level can come back null.""" + spoken = data.get("spokenLanguages") or {} + langs = spoken.get("spokenLanguages") or [] + return langs[0].get("id") if langs else None + + +def title_to_result(node: dict, kind: str) -> Optional[MetadataResult]: + title = (node.get("titleText") or {}).get("text") or (node.get("originalTitleText") or {}).get("text") + if not title: + return None + return MetadataResult( + title=title, + year=(node.get("releaseYear") or {}).get("year"), + kind=kind, + external_ids=ExternalIds(imdb_id=node.get("id")), + original_language=primary_language(node), + source="imdb", + raw=node, + ) + + +class IMDBProvider(MetadataProvider): + """IMDb metadata provider using IMDb's own GraphQL edge cache (no API key).""" + + NAME = "imdb" + REQUIRES_KEY = False + ID_KIND = "imdb" + + def is_available(self) -> bool: + return True + + def graphql(self, operation: str, query: str) -> Optional[dict]: + """Run a persisted query, registering it with one POST the first time it is seen.""" + extensions = {"persistedQuery": {"version": 1, "sha256Hash": hashlib.sha256(query.encode()).hexdigest()}} + # IMDb's GraphQL gateway decodes "+" literally, so these JSON params must carry no spaces + params = { + "operationName": operation, + "variables": "{}", + "extensions": json.dumps(extensions, separators=(",", ":")), + } + + try: + r = self.session.get(GRAPHQL_URL, params=params, headers=GRAPHQL_HEADERS, timeout=30) + body = r.json() + if persisted_query_missing(body): + self.log.debug("Registering persisted query %s", operation) + payload = {"operationName": operation, "variables": {}, "query": query, "extensions": extensions} + r = self.session.post(GRAPHQL_URL, json=payload, headers=GRAPHQL_HEADERS, timeout=30) + body = r.json() + except (requests.RequestException, ValueError) as exc: + self.log.debug("IMDb GraphQL %s failed: %s", operation, exc) + return None + + if body.get("errors"): + self.log.debug("IMDb GraphQL %s errors: %s", operation, [e.get("message") for e in body["errors"]][:3]) + return None + return body.get("data") + + def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: + self.log.debug("Searching IMDb for %r (%s, %s)", title, kind, year) + + constraints = [f'titleTextConstraint: {{searchTerm: "{escape_graphql(title)}"}}'] + if year: + constraints.append( + f'releaseDateConstraint: {{releaseDateRange: {{start: "{year}-01-01", end: "{year}-12-31"}}}}' + ) + types = KIND_TO_TYPES.get(kind) + if types: + constraints.append(f"titleTypeConstraint: {{anyTitleTypeIds: {json.dumps(types)}}}") + + query = ( + "query UnshackleTitleSearch { advancedTitleSearch(first: 5, constraints: {%s}) " + "{ edges { node { title { %s } } } } }" % (", ".join(constraints), TITLE_FIELDS) + ) + + data = self.graphql("UnshackleTitleSearch", query) + edges = ((data or {}).get("advancedTitleSearch") or {}).get("edges") or [] + nodes = [edge["node"]["title"] for edge in edges if (edge.get("node") or {}).get("title")] + if not nodes: + self.log.debug("IMDb returned no results for %r", title) + return None + + best = max(nodes, key=lambda n: match_ratio(n, title)) + result = title_to_result(best, kind) + if not result or not result.title or not fuzzy_match(result.title, title): + self.log.debug("IMDb title mismatch: searched %r, got %r", title, result.title if result else None) + return None + + self.log.debug("IMDb -> %s (ID %s)", result.title, result.external_ids.imdb_id) + return result + + def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: + """Fetch metadata by IMDB ID (e.g. 'tt1375666').""" + imdb_id = str(provider_id) + self.log.debug("Fetching IMDb title %s", imdb_id) + + query = 'query UnshackleTitle { title(id: "%s") { %s } }' % (escape_graphql(imdb_id), TITLE_FIELDS) + data = self.graphql("UnshackleTitle", query) + node = (data or {}).get("title") + if not node: + return None + return title_to_result(node, kind) + + def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: + """Return external IDs. For IMDB, the provider_id IS the IMDB ID.""" + return ExternalIds(imdb_id=str(provider_id)) + + +def match_ratio(node: dict, title: str) -> float: + names = [(node.get("titleText") or {}).get("text"), (node.get("originalTitleText") or {}).get("text")] + return max((SequenceMatcher(None, clean(title), clean(n)).ratio() for n in names if n), default=0.0) + + +def persisted_query_missing(body: dict) -> bool: + for error in body.get("errors") or []: + code = (error.get("extensions") or {}).get("code") + if code == "PERSISTED_QUERY_NOT_FOUND" or error.get("message") == "PersistedQueryNotFound": + return True + return False diff --git a/unshackle/core/providers/imdbapi.py b/unshackle/core/providers/imdbapi.py deleted file mode 100644 index b57bbd95..00000000 --- a/unshackle/core/providers/imdbapi.py +++ /dev/null @@ -1,124 +0,0 @@ -from __future__ import annotations - -from difflib import SequenceMatcher -from typing import Optional, Union - -import requests - -from unshackle.core.config import config -from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, _clean, fuzzy_match - -# Mapping from our kind ("movie"/"tv") to IMDxAPI title types -KIND_TO_TYPES: dict[str, list[str]] = { - "movie": ["movie"], - "tv": ["tvSeries", "tvMiniSeries"], -} - - -class IMDBApiProvider(MetadataProvider): - """IMDb metadata provider using IMDxAPI (api.tiffara.com, free, no API key).""" - - NAME = "imdbapi" - REQUIRES_KEY = False - BASE_URL = "https://api.tiffara.com" - - def is_available(self) -> bool: - return config.imdb_api_enabled - - def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: - self.log.debug("Searching IMDBApi for %r (%s, %s)", title, kind, year) - - try: - params: dict[str, str | int] = {"query": title, "limit": 20} - r = self.session.get( - f"{self.BASE_URL}/search/titles", - params=params, - timeout=30, - ) - r.raise_for_status() - data = r.json() - except (requests.RequestException, ValueError) as exc: - self.log.debug("IMDBApi search failed: %s", exc) - return None - - results = data.get("titles") or data.get("results") or [] - if not results: - self.log.debug("IMDBApi returned no results for %r", title) - return None - - # Filter by type if possible - type_filter = KIND_TO_TYPES.get(kind, []) - filtered = [r for r in results if r.get("type") in type_filter] if type_filter else results - candidates = filtered if filtered else results - - # Find best fuzzy match, optionally filtered by year - best_match: Optional[dict] = None - best_ratio = 0.0 - - for candidate in candidates: - primary = candidate.get("primaryTitle") or "" - original = candidate.get("originalTitle") or "" - - for name in [primary, original]: - if not name: - continue - ratio = SequenceMatcher(None, _clean(title), _clean(name)).ratio() - if ratio > best_ratio: - # If year provided, prefer matches within 1 year - candidate_year = candidate.get("startYear") - if year and candidate_year and abs(year - candidate_year) > 1: - continue - best_ratio = ratio - best_match = candidate - - if not best_match: - self.log.debug("No matching result found in IMDBApi for %r", title) - return None - - result_title = best_match.get("primaryTitle") or best_match.get("originalTitle") - if not result_title or not fuzzy_match(result_title, title): - self.log.debug("IMDBApi title mismatch: searched %r, got %r", title, result_title) - return None - - imdb_id = best_match.get("id") - result_year = best_match.get("startYear") - - self.log.debug("IMDBApi -> %s (ID %s)", result_title, imdb_id) - - return MetadataResult( - title=result_title, - year=result_year, - kind=kind, - external_ids=ExternalIds(imdb_id=imdb_id), - source="imdbapi", - raw=best_match, - ) - - def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: - """Fetch metadata by IMDB ID (e.g. 'tt1375666').""" - imdb_id = str(provider_id) - self.log.debug("Fetching IMDBApi title %s", imdb_id) - - try: - r = self.session.get(f"{self.BASE_URL}/titles/{imdb_id}", timeout=30) - r.raise_for_status() - data = r.json() - except (requests.RequestException, ValueError) as exc: - self.log.debug("IMDBApi get_by_id failed: %s", exc) - return None - - title = data.get("primaryTitle") or data.get("originalTitle") - result_year = data.get("startYear") - - return MetadataResult( - title=title, - year=result_year, - kind=kind, - external_ids=ExternalIds(imdb_id=data.get("id")), - source="imdbapi", - raw=data, - ) - - def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: - """Return external IDs. For IMDB, the provider_id IS the IMDB ID.""" - return ExternalIds(imdb_id=str(provider_id)) diff --git a/unshackle/core/providers/omdb.py b/unshackle/core/providers/omdb.py index 8f450f84..083bfe93 100644 --- a/unshackle/core/providers/omdb.py +++ b/unshackle/core/providers/omdb.py @@ -4,9 +4,10 @@ from difflib import SequenceMatcher from typing import Optional, Union import requests +from langcodes import Language from unshackle.core.config import config -from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, _clean, fuzzy_match +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, clean, fuzzy_match # Mapping from our kind ("movie"/"tv") to OMDb title types KIND_TO_TYPE: dict[str, str] = { @@ -15,7 +16,18 @@ KIND_TO_TYPE: dict[str, str] = { } -def _parse_year(value: Optional[str]) -> Optional[int]: +def primary_language(data: dict) -> Optional[str]: + """OMDb gives English language names, most prominent first, e.g. 'Korean, English'.""" + name = (data.get("Language") or "").split(",")[0].strip() + if not name: + return None + try: + return str(Language.find(name)) + except LookupError: + return None + + +def parse_year(value: Optional[str]) -> Optional[int]: # OMDb years look like "2017", "2008–2013" or "2023–" if value and len(value) >= 4 and value[:4].isdigit(): return int(value[:4]) @@ -27,18 +39,19 @@ class OMDBProvider(MetadataProvider): NAME = "omdb" REQUIRES_KEY = True + ID_KIND = "imdb" BASE_URL = "https://www.omdbapi.com/" def is_available(self) -> bool: return bool(config.omdb_api_key) @property - def _api_key(self) -> str: + def api_key(self) -> str: return config.omdb_api_key - def _get(self, params: dict[str, str]) -> Optional[dict]: + def api_get(self, params: dict[str, str]) -> Optional[dict]: try: - r = self.session.get(self.BASE_URL, params={"apikey": self._api_key, **params}, timeout=30) + r = self.session.get(self.BASE_URL, params={"apikey": self.api_key, **params}, timeout=30) r.raise_for_status() data = r.json() except (requests.RequestException, ValueError) as exc: @@ -59,11 +72,11 @@ class OMDBProvider(MetadataProvider): if year is not None: params["y"] = str(year) - data = self._get(params) + data = self.api_get(params) if not data and year is not None: # OMDb's year filter is exact; retry without it for off-by-one metadata del params["y"] - data = self._get(params) + data = self.api_get(params) if not data: return None @@ -75,9 +88,9 @@ class OMDBProvider(MetadataProvider): name = candidate.get("Title") or "" if not name: continue - ratio = SequenceMatcher(None, _clean(title), _clean(name)).ratio() + ratio = SequenceMatcher(None, clean(title), clean(name)).ratio() if ratio > best_ratio: - candidate_year = _parse_year(candidate.get("Year")) + candidate_year = parse_year(candidate.get("Year")) if year and candidate_year and abs(year - candidate_year) > 1: continue best_ratio = ratio @@ -96,13 +109,14 @@ class OMDBProvider(MetadataProvider): self.log.debug("OMDb -> %s (ID %s)", result_title, imdb_id) # Fetch full detail so raw carries ratings, genre, rating cert, etc. - detail = self._get({"i": imdb_id}) if imdb_id else None + detail = self.api_get({"i": imdb_id}) if imdb_id else None return MetadataResult( title=result_title, - year=_parse_year(best_match.get("Year")), + year=parse_year(best_match.get("Year")), kind=kind, external_ids=ExternalIds(imdb_id=imdb_id), + original_language=primary_language(detail or best_match), source="omdb", raw=detail or best_match, ) @@ -112,15 +126,16 @@ class OMDBProvider(MetadataProvider): imdb_id = str(provider_id) self.log.debug("Fetching OMDb title %s", imdb_id) - data = self._get({"i": imdb_id}) + data = self.api_get({"i": imdb_id}) if not data: return None return MetadataResult( title=data.get("Title"), - year=_parse_year(data.get("Year")), + year=parse_year(data.get("Year")), kind=kind, external_ids=ExternalIds(imdb_id=data.get("imdbID")), + original_language=primary_language(data), source="omdb", raw=data, ) diff --git a/unshackle/core/providers/simkl.py b/unshackle/core/providers/simkl.py index 00bad342..c61c174f 100644 --- a/unshackle/core/providers/simkl.py +++ b/unshackle/core/providers/simkl.py @@ -47,7 +47,7 @@ class SimklProvider(MetadataProvider): self.log.debug("Simkl returned list (no matches) for %r", filename) return None - return self._parse_response(data, title, year, kind) + return self.parse_response(data, title, year, kind) def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: return None # SIMKL has no direct ID lookup used here @@ -115,7 +115,7 @@ class SimklProvider(MetadataProvider): tvdb_id=tvdb_id, ) - def _parse_response( + def parse_response( self, data: dict, search_title: str, search_year: Optional[int], kind: str ) -> Optional[MetadataResult]: """Parse a SIMKL response into a MetadataResult.""" diff --git a/unshackle/core/providers/tmdb.py b/unshackle/core/providers/tmdb.py index f50a3b16..60014161 100644 --- a/unshackle/core/providers/tmdb.py +++ b/unshackle/core/providers/tmdb.py @@ -6,7 +6,7 @@ from typing import Optional, Union import requests from unshackle.core.config import config -from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, _clean, _strip_year +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, clean, strip_year class TMDBProvider(MetadataProvider): @@ -14,20 +14,21 @@ class TMDBProvider(MetadataProvider): NAME = "tmdb" REQUIRES_KEY = True + ID_KIND = "tmdb" BASE_URL = "https://api.themoviedb.org/3" def is_available(self) -> bool: return bool(config.tmdb_api_key) @property - def _api_key(self) -> str: + def api_key(self) -> str: return config.tmdb_api_key def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: - search_title = _strip_year(title) + search_title = strip_year(title) self.log.debug("Searching TMDB for %r (%s, %s)", search_title, kind, year) - params: dict[str, str | int] = {"api_key": self._api_key, "query": search_title} + params: dict[str, str | int] = {"api_key": self.api_key, "query": search_title} if year is not None: params["year" if kind == "movie" else "first_air_date_year"] = year @@ -55,7 +56,7 @@ class TMDBProvider(MetadataProvider): candidates = [c for c in candidates if c] for candidate in candidates: - ratio = SequenceMatcher(None, _clean(search_title), _clean(candidate)).ratio() + ratio = SequenceMatcher(None, clean(search_title), clean(candidate)).ratio() if ratio > best_ratio: best_ratio = ratio best_id = result.get("id") @@ -72,8 +73,8 @@ class TMDBProvider(MetadataProvider): return None # Fetch full detail for caching - detail = self._fetch_detail(best_id, kind) - ext_raw = self._fetch_external_ids_raw(best_id, kind) + detail = self.fetch_detail(best_id, kind) + ext_raw = self.fetch_external_ids_raw(best_id, kind) date = (detail or {}).get("release_date") or (detail or {}).get("first_air_date") result_year = int(date[:4]) if date and len(date) >= 4 and date[:4].isdigit() else None @@ -90,12 +91,13 @@ class TMDBProvider(MetadataProvider): year=result_year, kind=kind, external_ids=ext, + original_language=(detail or {}).get("original_language"), source="tmdb", raw={"detail": detail or {}, "external_ids": ext_raw or {}}, ) def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: - detail = self._fetch_detail(int(provider_id), kind) + detail = self.fetch_detail(int(provider_id), kind) if not detail: return None @@ -108,12 +110,13 @@ class TMDBProvider(MetadataProvider): year=year, kind=kind, external_ids=ExternalIds(tmdb_id=int(provider_id), tmdb_kind=kind), + original_language=detail.get("original_language"), source="tmdb", raw=detail, ) def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: - raw = self._fetch_external_ids_raw(int(provider_id), kind) + raw = self.fetch_external_ids_raw(int(provider_id), kind) if not raw: return ExternalIds(tmdb_id=int(provider_id), tmdb_kind=kind) return ExternalIds( @@ -129,7 +132,7 @@ class TMDBProvider(MetadataProvider): try: r = self.session.get( f"{self.BASE_URL}/find/{imdb_id}", - params={"api_key": self._api_key, "external_source": "imdb_id"}, + params={"api_key": self.api_key, "external_source": "imdb_id"}, timeout=30, ) r.raise_for_status() @@ -163,7 +166,7 @@ class TMDBProvider(MetadataProvider): self.log.debug("TMDB find -> ID %s (%s) for IMDB %s", tmdb_id, kind, imdb_id) # Now fetch the full external IDs from TMDB to get TVDB etc. - ext_raw = self._fetch_external_ids_raw(tmdb_id, kind) + ext_raw = self.fetch_external_ids_raw(tmdb_id, kind) return ExternalIds( imdb_id=imdb_id, @@ -172,11 +175,11 @@ class TMDBProvider(MetadataProvider): tvdb_id=ext_raw.get("tvdb_id") if ext_raw else None, ) - def _fetch_detail(self, tmdb_id: int, kind: str) -> Optional[dict]: + def fetch_detail(self, tmdb_id: int, kind: str) -> Optional[dict]: try: r = self.session.get( f"{self.BASE_URL}/{kind}/{tmdb_id}", - params={"api_key": self._api_key}, + params={"api_key": self.api_key}, timeout=30, ) r.raise_for_status() @@ -185,11 +188,11 @@ class TMDBProvider(MetadataProvider): self.log.debug("Failed to fetch TMDB detail: %s", exc) return None - def _fetch_external_ids_raw(self, tmdb_id: int, kind: str) -> Optional[dict]: + def fetch_external_ids_raw(self, tmdb_id: int, kind: str) -> Optional[dict]: try: r = self.session.get( f"{self.BASE_URL}/{kind}/{tmdb_id}/external_ids", - params={"api_key": self._api_key}, + params={"api_key": self.api_key}, timeout=30, ) r.raise_for_status() diff --git a/unshackle/core/providers/tvdb.py b/unshackle/core/providers/tvdb.py index 8fa7f097..592eb0a9 100644 --- a/unshackle/core/providers/tvdb.py +++ b/unshackle/core/providers/tvdb.py @@ -7,7 +7,13 @@ from typing import Any, Optional, Union import requests from unshackle.core.config import config -from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, _clean, _strip_year +from unshackle.core.providers._base import ExternalIds, MetadataProvider, MetadataResult, clean, strip_year + + +def primary_language(data: dict) -> Optional[str]: + """Search results spell the key primary_language, the extended record originalLanguage.""" + return data.get("primary_language") or data.get("originalLanguage") or None + KIND_TO_TYPE: dict[str, str] = {"movie": "movie", "tv": "series"} KIND_TO_PATH: dict[str, str] = {"movie": "movies", "tv": "series"} @@ -20,14 +26,14 @@ AIRED_ORDER = "official" _token: Optional[str] = None -def _parse_int(value: Optional[str]) -> Optional[int]: +def parse_int(value: Optional[str]) -> Optional[int]: try: return int(value) # type: ignore[arg-type] except (TypeError, ValueError): return None -def _ids_from_remote(remote_ids: Optional[list], tvdb_id: Optional[int]) -> ExternalIds: +def ids_from_remote(remote_ids: Optional[list], tvdb_id: Optional[int]) -> ExternalIds: """Build ExternalIds from a TVDB remote_ids/remoteIds list.""" ext = ExternalIds(tvdb_id=tvdb_id) for entry in remote_ids or []: @@ -36,11 +42,11 @@ def _ids_from_remote(remote_ids: Optional[list], tvdb_id: Optional[int]) -> Exte if source == "imdb": ext.imdb_id = value elif source.startswith("themoviedb"): - ext.tmdb_id = _parse_int(value) + ext.tmdb_id = parse_int(value) return ext -def _pick_match( +def pick_match( results: list[dict], search_title: str, year: Optional[int] ) -> tuple[Optional[dict], Optional[str], float]: """Best title match among results, ignoring any whose year is more than a year out.""" @@ -48,13 +54,13 @@ def _pick_match( best_title: Optional[str] = None best_ratio = 0.0 for result in results: - result_year = _parse_int(result.get("year")) + result_year = parse_int(result.get("year")) if year and result_year and abs(result_year - year) > 1: continue for candidate in [result.get("name"), *(result.get("aliases") or [])]: if not candidate: continue - ratio = SequenceMatcher(None, _clean(search_title), _clean(candidate)).ratio() + ratio = SequenceMatcher(None, clean(search_title), clean(candidate)).ratio() if ratio > best_ratio: best, best_title, best_ratio = result, candidate, ratio return best, best_title, best_ratio @@ -65,6 +71,7 @@ class TVDBProvider(MetadataProvider): NAME = "tvdb" REQUIRES_KEY = True + ID_KIND = "tvdb" BASE_URL = "https://api4.thetvdb.com/v4" def __init__(self) -> None: @@ -74,7 +81,7 @@ class TVDBProvider(MetadataProvider): def is_available(self) -> bool: return bool(config.tvdb_api_key) - def _login(self) -> Optional[str]: + def login(self) -> Optional[str]: global _token payload: dict[str, str] = {"apikey": config.tvdb_api_key} if config.tvdb_pin: @@ -88,10 +95,10 @@ class TVDBProvider(MetadataProvider): _token = None return _token - def _get(self, path: str, params: Optional[dict] = None) -> Any: + def api_get(self, path: str, params: Optional[dict] = None) -> Any: """The `data` payload of a TVDB response: a dict for entities, a list for searches.""" global _token - token = _token or self._login() + token = _token or self.login() if not token: return None for _ in range(2): @@ -104,7 +111,7 @@ class TVDBProvider(MetadataProvider): ) if r.status_code == 401: _token = None - token = self._login() + token = self.login() if not token: return None continue @@ -116,7 +123,7 @@ class TVDBProvider(MetadataProvider): return None def search(self, title: str, year: Optional[int], kind: str) -> Optional[MetadataResult]: - search_title = _strip_year(title) + search_title = strip_year(title) self.log.debug("Searching TVDB for %r (%s, %s)", search_title, kind, year) params: dict[str, str | int] = {"query": search_title, "limit": 10} @@ -126,23 +133,23 @@ class TVDBProvider(MetadataProvider): if year is not None: params["year"] = year - results = self._get("/search", params) + results = self.api_get("/search", params) if not results and year is not None: del params["year"] - results = self._get("/search", params) + results = self.api_get("/search", params) if not results: self.log.debug("No TVDB results for %r", search_title) return None # year first, so an identically named remake cannot win on title alone - best, best_title, best_ratio = _pick_match(results, search_title, year) + best, best_title, best_ratio = pick_match(results, search_title, year) if best is None: - best, best_title, best_ratio = _pick_match(results, search_title, None) + best, best_title, best_ratio = pick_match(results, search_title, None) if best is None: best, best_title = results[0], results[0].get("name") - tvdb_id = _parse_int(best.get("tvdb_id")) - ext = _ids_from_remote(best.get("remote_ids"), tvdb_id) + tvdb_id = parse_int(best.get("tvdb_id")) + ext = ids_from_remote(best.get("remote_ids"), tvdb_id) if ext.tmdb_id: ext.tmdb_kind = kind @@ -150,42 +157,44 @@ class TVDBProvider(MetadataProvider): return MetadataResult( title=best_title or best.get("name"), - year=_parse_int(best.get("year")), + year=parse_int(best.get("year")), kind=kind, external_ids=ext, + original_language=primary_language(best), source="tvdb", raw=best, ) - def _fetch_extended(self, tvdb_id: int, kind: str) -> Optional[dict]: + def fetch_extended(self, tvdb_id: int, kind: str) -> Optional[dict]: path = KIND_TO_PATH.get(kind, "series") - return self._get(f"/{path}/{tvdb_id}/extended", {"short": "true"}) + return self.api_get(f"/{path}/{tvdb_id}/extended", {"short": "true"}) def get_by_id(self, provider_id: Union[int, str], kind: str) -> Optional[MetadataResult]: tvdb_id = int(provider_id) - detail = self._fetch_extended(tvdb_id, kind) + detail = self.fetch_extended(tvdb_id, kind) if not detail: return None - ext = _ids_from_remote(detail.get("remoteIds"), tvdb_id) + ext = ids_from_remote(detail.get("remoteIds"), tvdb_id) if ext.tmdb_id: ext.tmdb_kind = kind return MetadataResult( title=detail.get("name"), - year=_parse_int(detail.get("year")), + year=parse_int(detail.get("year")), kind=kind, external_ids=ext, + original_language=primary_language(detail), source="tvdb", raw=detail, ) def get_external_ids(self, provider_id: Union[int, str], kind: str) -> ExternalIds: tvdb_id = int(provider_id) - detail = self._fetch_extended(tvdb_id, kind) + detail = self.fetch_extended(tvdb_id, kind) if not detail: return ExternalIds(tvdb_id=tvdb_id) - ext = _ids_from_remote(detail.get("remoteIds"), tvdb_id) + ext = ids_from_remote(detail.get("remoteIds"), tvdb_id) if ext.tmdb_id: ext.tmdb_kind = kind return ext @@ -198,7 +207,7 @@ class TVDBProvider(MetadataProvider): episodes: list[dict] = [] for page in range(20): # ponytail: hard page cap, 500 eps/page covers any real series - data = self._get(f"/series/{tvdb_id}/episodes/{order}", {"page": page}) + data = self.api_get(f"/series/{tvdb_id}/episodes/{order}", {"page": page}) if data is None and page: # a partial listing would renumber episodes wrongly self.log.warning("TVDB %s order listing for series %s failed at page %d", order, tvdb_id, page) @@ -223,7 +232,7 @@ class TVDBProvider(MetadataProvider): episodes = self.get_episodes(tvdb_id, order) if not episodes: continue - slots = {(_parse_int(e.get("seasonNumber")), _parse_int(e.get("number"))) for e in episodes} + slots = {(parse_int(e.get("seasonNumber")), parse_int(e.get("number"))) for e in episodes} score = sum(1 for key in keys if key in slots) self.log.debug("TVDB order %s matches %d/%d of the service's episodes", order, score, len(keys)) if score > best_score: @@ -255,8 +264,8 @@ class TVDBProvider(MetadataProvider): match = by_id.get(ep.get("id")) if not match: continue - src_season, src_number = _parse_int(ep.get("seasonNumber")), _parse_int(ep.get("number")) - dst_season, dst_number = _parse_int(match.get("seasonNumber")), _parse_int(match.get("number")) + src_season, src_number = parse_int(ep.get("seasonNumber")), parse_int(ep.get("number")) + dst_season, dst_number = parse_int(match.get("seasonNumber")), parse_int(match.get("number")) if src_season is None or src_number is None or dst_season is None or dst_number is None: continue mapping[(src_season, src_number)] = (dst_season, dst_number, match.get("name")) @@ -266,7 +275,7 @@ class TVDBProvider(MetadataProvider): def find_by_imdb_id(self, imdb_id: str, kind: str) -> Optional[ExternalIds]: """Resolve a TVDB ID from an IMDb ID via /search/remoteid.""" - results = self._get(f"/search/remoteid/{imdb_id}") + results = self.api_get(f"/search/remoteid/{imdb_id}") if not results: self.log.debug("No TVDB results for IMDB ID %s", imdb_id) return None @@ -281,6 +290,6 @@ class TVDBProvider(MetadataProvider): if not entity: return None - tvdb_id = _parse_int(entity.get("id")) + tvdb_id = parse_int(entity.get("id")) self.log.debug("TVDB find -> TVDB %s for IMDB %s", tvdb_id, imdb_id) return ExternalIds(imdb_id=imdb_id, tvdb_id=tvdb_id) diff --git a/unshackle/core/proxies/gluetun.py b/unshackle/core/proxies/gluetun.py index 68ea4a2e..59972186 100644 --- a/unshackle/core/proxies/gluetun.py +++ b/unshackle/core/proxies/gluetun.py @@ -17,17 +17,17 @@ from unshackle.core.utilities import get_country_code, get_country_name, get_deb from unshackle.core.utils.ip_info import get_ip_info # Global registry for cleanup on exit -_gluetun_instances: list["Gluetun"] = [] -_cleanup_lock = threading.Lock() -_cleanup_registered = False +gluetun_instances: list["Gluetun"] = [] +cleanup_lock = threading.Lock() +cleanup_registered = False -def _cleanup_all_gluetun_containers(): +def cleanup_all_gluetun_containers(): """Cleanup all Gluetun containers on exit.""" # Get instances without holding the lock during cleanup - with _cleanup_lock: - instances = list(_gluetun_instances) - _gluetun_instances.clear() + with cleanup_lock: + instances = list(gluetun_instances) + gluetun_instances.clear() # Cleanup each instance (no lock held, so no deadlock possible) for instance in instances: @@ -37,15 +37,15 @@ def _cleanup_all_gluetun_containers(): pass -def _register_cleanup(): +def register_cleanup(): """Register cleanup handlers (only once).""" - global _cleanup_registered - with _cleanup_lock: - if not _cleanup_registered: + global cleanup_registered + with cleanup_lock: + if not cleanup_registered: # Only use atexit for cleanup - don't override signal handlers # This allows Ctrl+C to work normally while still cleaning up on exit - atexit.register(_cleanup_all_gluetun_containers) - _cleanup_registered = True + atexit.register(cleanup_all_gluetun_containers) + cleanup_registered = True class Gluetun(Proxy): @@ -215,12 +215,12 @@ class Gluetun(Proxy): # Validate provider configurations for provider_name, config in self.providers.items(): - self._validate_provider_config(provider_name, config) + self.validate_provider_config(provider_name, config) # Register this instance for cleanup on exit - _register_cleanup() - with _cleanup_lock: - _gluetun_instances.append(self) + register_cleanup() + with cleanup_lock: + gluetun_instances.append(self) # Log initialization log_event( @@ -271,7 +271,7 @@ class Gluetun(Proxy): # This handles multiple concurrent Unshackle sessions if query_key in self.active_containers: container = self.active_containers[query_key] - if self._is_container_running(container["container_name"]): + if self.is_container_running(container["container_name"]): log_event( "gluetun_container_reuse", level="DEBUG", @@ -284,11 +284,11 @@ class Gluetun(Proxy): ) # Re-verify if needed if self.verify_ip: - self._verify_container(query_key) - return self._build_proxy_uri(container["port"]) + self.verify_container(query_key) + return self.build_proxy_uri(container["port"]) else: # Not in memory, but might exist in Docker (from another session) - existing_info = self._get_existing_container_info(container_name) + existing_info = self.get_existing_container_info(container_name) if existing_info: # Container exists in Docker, reuse it self.active_containers[query_key] = existing_info @@ -304,8 +304,8 @@ class Gluetun(Proxy): ) # Re-verify if needed if self.verify_ip: - self._verify_container(query_key) - return self._build_proxy_uri(existing_info["port"]) + self.verify_container(query_key) + return self.build_proxy_uri(existing_info["port"]) # Get provider configuration provider_config = self.providers[provider_name] @@ -329,7 +329,7 @@ class Gluetun(Proxy): server_num = specific_server_match.group(2) # Build hostname based on provider - hostname = self._build_server_hostname(provider_name, country_code, server_num) + hostname = self.build_server_hostname(provider_name, country_code, server_num) country = country_code # Set country for verification # If not explicitly mapped and not a specific server, try to use query as country code @@ -349,14 +349,14 @@ class Gluetun(Proxy): ) # Remove any stopped container with the same name - self._remove_stopped_container(container_name) + self.remove_stopped_container(container_name) # Find available port - port = self._get_available_port() + port = self.get_available_port() # Create container (name already set above) try: - self._create_container( + self.create_container( container_name=container_name, port=port, provider_name=provider_name, @@ -378,9 +378,9 @@ class Gluetun(Proxy): } # Wait for container to be ready (60s timeout for VPN connection) - if not self._wait_for_container(container_name, timeout=60): + if not self.wait_for_container(container_name, timeout=60): # Get container logs for better error message - logs = self._get_container_logs(container_name, tail=30) + logs = self.get_container_logs(container_name, tail=30) error_msg = f"Gluetun container '{container_name}' failed to start" if hasattr(self, "_last_wait_error") and self._last_wait_error: error_msg += f": {self._last_wait_error}" @@ -392,13 +392,13 @@ class Gluetun(Proxy): # Verify IP and region if enabled if self.verify_ip: - self._verify_container(query_key) + self.verify_container(query_key) - return self._build_proxy_uri(port) + return self.build_proxy_uri(port) except Exception as e: # Cleanup on failure - self._remove_container(container_name) + self.remove_container(container_name) if query_key in self.active_containers: del self.active_containers[query_key] raise RuntimeError(f"Failed to create Gluetun container: {e}") @@ -420,7 +420,7 @@ class Gluetun(Proxy): for query_key, container_info in list(self.active_containers.items()): container_name = container_info["container_name"] - self._remove_container(container_name) + self.remove_container(container_name) log_event( "gluetun_container_removed", @@ -474,7 +474,7 @@ class Gluetun(Proxy): "org": container.get("ip_org"), } - def _validate_provider_config(self, provider_name: str, config: dict): + def validate_provider_config(self, provider_name: str, config: dict): """Validate a provider's configuration.""" vpn_type = config.get("vpn_type", "wireguard").lower() credentials = config.get("credentials", {}) @@ -517,16 +517,16 @@ class Gluetun(Proxy): f"Provider '{provider_name}': OpenVPN requires 'username' and 'password' in credentials" ) - def _get_available_port(self) -> int: + def get_available_port(self) -> int: """Find an available port starting from base_port (thread-safe).""" with self._port_lock: used_ports = {info["port"] for info in self.active_containers.values()} port = self.base_port - while port in used_ports or self._is_port_in_use(port): + while port in used_ports or self.is_port_in_use(port): port += 1 return port - def _is_port_in_use(self, port: int) -> bool: + def is_port_in_use(self, port: int) -> bool: """Check if a port is in use on the system or by any Docker container.""" import socket @@ -554,7 +554,7 @@ class Gluetun(Proxy): except (subprocess.TimeoutExpired, FileNotFoundError): return False - def _build_server_hostname(self, provider_name: str, country_code: str, server_num: str) -> str: + def build_server_hostname(self, provider_name: str, country_code: str, server_num: str) -> str: """ Build a server hostname for specific server selection. @@ -585,7 +585,7 @@ class Gluetun(Proxy): # Generic format: country_code + server_num return f"{country_lower}{server_num}" - def _ensure_image_available(self, image: str = "qmcgaw/gluetun:latest") -> bool: + def ensure_image_available(self, image: str = "qmcgaw/gluetun:latest") -> bool: """ Ensure the Gluetun Docker image is available locally. @@ -637,7 +637,7 @@ class Gluetun(Proxy): except subprocess.TimeoutExpired: raise RuntimeError(f"Timed out pulling Docker image '{image}'") - def _create_container( + def create_container( self, container_name: str, port: int, @@ -666,7 +666,7 @@ class Gluetun(Proxy): # Ensure the Gluetun image is available (pulls if needed) gluetun_image = "qmcgaw/gluetun:latest" - if not self._ensure_image_available(gluetun_image): + if not self.ensure_image_available(gluetun_image): log_event( "gluetun_image_pull_failed", level="ERROR", @@ -880,7 +880,7 @@ class Gluetun(Proxy): except Exception: pass - def _is_container_running(self, container_name: str) -> bool: + def is_container_running(self, container_name: str) -> bool: """Check if a Docker container is running.""" try: result = subprocess.run( @@ -906,7 +906,7 @@ class Gluetun(Proxy): except (subprocess.TimeoutExpired, FileNotFoundError): return False - def _get_existing_container_info(self, container_name: str) -> Optional[dict]: + def get_existing_container_info(self, container_name: str) -> Optional[dict]: """ Check if a container exists in Docker and get its info. @@ -921,7 +921,7 @@ class Gluetun(Proxy): """ try: # Check if container is running - if not self._is_container_running(container_name): + if not self.is_container_running(container_name): return None # Get container port mapping @@ -964,7 +964,7 @@ class Gluetun(Proxy): if specific_server_match: country_code = specific_server_match.group(1).upper() server_num = specific_server_match.group(2) - hostname = self._build_server_hostname(provider_name, country_code, server_num) + hostname = self.build_server_hostname(provider_name, country_code, server_num) country = country_code # Otherwise check config @@ -989,7 +989,7 @@ class Gluetun(Proxy): except (subprocess.TimeoutExpired, FileNotFoundError, ValueError): return None - def _wait_for_container(self, container_name: str, timeout: int = 60) -> bool: + def wait_for_container(self, container_name: str, timeout: int = 60) -> bool: """ Wait for Gluetun container to be ready by checking logs for proxy readiness. @@ -1015,9 +1015,9 @@ class Gluetun(Proxy): while time.time() - start_time < timeout: try: # First check if container is still running - if not self._is_container_running(container_name): + if not self.is_container_running(container_name): # Container may have exited - check if it crashed - exit_info = self._get_container_exit_info(container_name) + exit_info = self.get_container_exit_info(container_name) if exit_info: last_error = f"Container exited with code {exit_info.get('exit_code', 'unknown')}" time.sleep(1) @@ -1108,7 +1108,7 @@ class Gluetun(Proxy): ) return False - def _get_container_exit_info(self, container_name: str) -> Optional[dict]: + def get_container_exit_info(self, container_name: str) -> Optional[dict]: """Get exit information for a stopped container.""" try: result = subprocess.run( @@ -1129,7 +1129,7 @@ class Gluetun(Proxy): except (subprocess.TimeoutExpired, FileNotFoundError, ValueError): return None - def _get_container_logs(self, container_name: str, tail: int = 50) -> str: + def get_container_logs(self, container_name: str, tail: int = 50) -> str: """Get recent logs from a container for error reporting.""" try: result = subprocess.run( @@ -1144,7 +1144,7 @@ class Gluetun(Proxy): except (subprocess.TimeoutExpired, FileNotFoundError): return "" - def _verify_container(self, query_key: str, max_retries: int = 3): + def verify_container(self, query_key: str, max_retries: int = 3): """ Verify container's VPN IP and region using ipinfo.io lookup. @@ -1165,7 +1165,7 @@ class Gluetun(Proxy): return container = self.active_containers[query_key] - proxy_url = self._build_proxy_uri(container["port"]) + proxy_url = self.build_proxy_uri(container["port"]) expected_country = container.get("country", "").upper() log_event( @@ -1299,7 +1299,7 @@ class Gluetun(Proxy): f"after {max_retries} attempts. Last error: {last_error}" ) - def _remove_stopped_container(self, container_name: str) -> bool: + def remove_stopped_container(self, container_name: str) -> bool: """ Remove a stopped container with the given name if it exists. @@ -1349,7 +1349,7 @@ class Gluetun(Proxy): except (subprocess.TimeoutExpired, FileNotFoundError): return False - def _remove_container(self, container_name: str): + def remove_container(self, container_name: str): """Stop and remove a Docker container.""" try: if self.auto_cleanup: @@ -1386,7 +1386,7 @@ class Gluetun(Proxy): except subprocess.TimeoutExpired: pass - def _build_proxy_uri(self, port: int) -> str: + def build_proxy_uri(self, port: int) -> str: """Build HTTP proxy URI.""" if self.auth_user and self.auth_password: return f"http://{self.auth_user}:{self.auth_password}@localhost:{port}" diff --git a/unshackle/core/proxies/proton.py b/unshackle/core/proxies/proton.py index 72388b57..c5e9d11a 100644 --- a/unshackle/core/proxies/proton.py +++ b/unshackle/core/proxies/proton.py @@ -11,10 +11,9 @@ from pathlib import Path from typing import Optional import requests -from rich.text import Text from unshackle.core.config import config -from unshackle.core.console import console +from unshackle.core.console import prompt_user from unshackle.core.proxies.proxy import Proxy log = logging.getLogger("proxies.proton") @@ -326,13 +325,12 @@ class ProtonVPN(Proxy): log.error("Proton: unexpected TV login response") return False - indent = " " * 5 prompt = ( "Proton: open https://account.proton.me/vpn/tv/code\n" f"Enter code {user_code}, then press Enter to continue..." ) try: - console.input(Text(indent + prompt.replace("\n", "\n" + indent), style="text")) + prompt_user(prompt) except EOFError: return False diff --git a/unshackle/core/remote_service.py b/unshackle/core/remote_service.py index 996d4010..450b2ac2 100644 --- a/unshackle/core/remote_service.py +++ b/unshackle/core/remote_service.py @@ -22,7 +22,7 @@ from rich.padding import Padding from rich.rule import Rule from unshackle.core.config import config -from unshackle.core.console import console +from unshackle.core.console import console, prompt_user from unshackle.core.constants import AnyTrack from unshackle.core.credential import Credential from unshackle.core.titles import Title_T, Titles_T, remap_titles @@ -37,6 +37,8 @@ log = logging.getLogger("remote_service") SENSITIVE_DATA_KEYS = ("credential", "credentials", "password", "token", "api_key") +DEFAULT_AUTH_HEADERS = ["X-Secret-Key", "X-Api-Key"] + def redact_secrets(text: str, data: Optional[Dict[str, Any]] = None) -> str: """Mask URL userinfo and any request-payload secrets before the text is logged.""" @@ -47,9 +49,11 @@ def redact_secrets(text: str, data: Optional[Dict[str, Any]] = None) -> str: class RemoteClient: """HTTP client for the unshackle serve API.""" - def __init__(self, server_url: str, api_key: str) -> None: + def __init__(self, server_url: str, api_key: str, auth_headers: Optional[list[str]] = None) -> None: self.server_url = server_url.rstrip("/") self.api_key = api_key + self.auth_headers = list(auth_headers) if auth_headers else list(DEFAULT_AUTH_HEADERS) + self._auth_header_index = 0 self._session: Optional[requests.Session] = None @property @@ -60,20 +64,35 @@ class RemoteClient: self._session = requests.Session() self._session.headers["User-Agent"] = f"unshackle/{__version__}" if self.api_key: - self._session.headers["X-Secret-Key"] = self.api_key + self._session.headers[self.auth_headers[self._auth_header_index]] = self.api_key return self._session - def _request(self, method: str, endpoint: str, data: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + def next_auth_header(self) -> bool: + """Move the api key onto the next candidate header. False once they are exhausted.""" + if not self.api_key or self._auth_header_index + 1 >= len(self.auth_headers): + return False + session = self.session + session.headers.pop(self.auth_headers[self._auth_header_index], None) + self._auth_header_index += 1 + session.headers[self.auth_headers[self._auth_header_index]] = self.api_key + return True + + def request(self, method: str, endpoint: str, data: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: url = f"{self.server_url}{endpoint}" - try: - resp = getattr(self.session, method)(url, json=data, timeout=120 if method == "post" else 30) - except requests.ConnectionError: - server_url = safe_display_url(self.server_url) - log.error(f"Could not connect to remote server at {server_url}. Is it running? (unshackle serve)") - raise SystemExit(1) - except requests.Timeout: - log.error(f"Request to remote server timed out: {endpoint}") - raise SystemExit(1) + while True: + try: + resp = getattr(self.session, method)(url, json=data, timeout=120 if method == "post" else 30) + except requests.ConnectionError: + server_url = safe_display_url(self.server_url) + log.error(f"Could not connect to remote server at {server_url}. Is it running? (unshackle serve)") + raise SystemExit(1) + except requests.Timeout: + log.error(f"Request to remote server timed out: {endpoint}") + raise SystemExit(1) + # servers differ on which header carries the key, so retry the rest before giving up + if resp.status_code == 401 and self.next_auth_header(): + continue + break result = resp.json() if resp.status_code >= 400: error_msg = redact_secrets(str(result.get("message", resp.text)), data) @@ -83,16 +102,16 @@ class RemoteClient: return result def post(self, endpoint: str, data: Dict[str, Any]) -> Dict[str, Any]: - return self._request("post", endpoint, data) + return self.request("post", endpoint, data) def get(self, endpoint: str) -> Dict[str, Any]: - return self._request("get", endpoint) + return self.request("get", endpoint) def delete(self, endpoint: str) -> Dict[str, Any]: - return self._request("delete", endpoint) + return self.request("delete", endpoint) -def _enum_get(enum_cls: type[Enum], name: Optional[str], default: Any = None) -> Any: +def enum_get(enum_cls: type[Enum], name: Optional[str], default: Any = None) -> Any: """Safely get an enum value by name.""" if not name: return default @@ -102,13 +121,13 @@ def _enum_get(enum_cls: type[Enum], name: Optional[str], default: Any = None) -> return default -def _deserialize_video(data: Dict[str, Any]) -> Video: +def deserialize_video(data: Dict[str, Any]) -> Video: v = Video( url=data.get("url") or "https://placeholder", language=Language.get(data.get("language") or "und"), - descriptor=_enum_get(Track.Descriptor, data.get("descriptor"), Track.Descriptor.URL), - codec=_enum_get(Video.Codec, data.get("codec")), - range_=_enum_get(Video.Range, data.get("range"), Video.Range.SDR), + descriptor=enum_get(Track.Descriptor, data.get("descriptor"), Track.Descriptor.URL), + codec=enum_get(Video.Codec, data.get("codec")), + range_=enum_get(Video.Range, data.get("range"), Video.Range.SDR), bitrate=data["bitrate"] * 1000 if data.get("bitrate") else 0, width=data.get("width") or 0, height=data.get("height") or 0, @@ -118,12 +137,12 @@ def _deserialize_video(data: Dict[str, Any]) -> Video: return v -def _deserialize_audio(data: Dict[str, Any]) -> Audio: +def deserialize_audio(data: Dict[str, Any]) -> Audio: a = Audio( url=data.get("url") or "https://placeholder", language=Language.get(data.get("language") or "und"), - descriptor=_enum_get(Track.Descriptor, data.get("descriptor"), Track.Descriptor.URL), - codec=_enum_get(Audio.Codec, data.get("codec")), + descriptor=enum_get(Track.Descriptor, data.get("descriptor"), Track.Descriptor.URL), + codec=enum_get(Audio.Codec, data.get("codec")), bitrate=data["bitrate"] * 1000 if data.get("bitrate") else 0, channels=data.get("channels"), joc=1 if data.get("atmos") else 0, @@ -133,12 +152,12 @@ def _deserialize_audio(data: Dict[str, Any]) -> Audio: return a -def _deserialize_subtitle(data: Dict[str, Any]) -> Subtitle: +def deserialize_subtitle(data: Dict[str, Any]) -> Subtitle: return Subtitle( url=data.get("url") or "https://placeholder", language=Language.get(data.get("language") or "und"), - descriptor=_enum_get(Track.Descriptor, data.get("descriptor"), Track.Descriptor.URL), - codec=_enum_get(Subtitle.Codec, data.get("codec")), + descriptor=enum_get(Track.Descriptor, data.get("descriptor"), Track.Descriptor.URL), + codec=enum_get(Subtitle.Codec, data.get("codec")), cc=data.get("cc", False), sdh=data.get("sdh", False), forced=data.get("forced", False), @@ -146,7 +165,7 @@ def _deserialize_subtitle(data: Dict[str, Any]) -> Subtitle: ) -def _reconstruct_drm(drm_list: Optional[list]) -> list: +def reconstruct_drm(drm_list: Optional[list]) -> list: """Reconstruct DRM objects from serialized API data.""" if not drm_list: return [] @@ -173,22 +192,24 @@ def _reconstruct_drm(drm_list: Optional[list]) -> list: pr_pssh = PlayReadyPSSH(b64.b64decode(pssh_str)) result.append(PlayReady(pssh=pr_pssh, pssh_b64=pssh_str)) - except Exception: + # a dropped entry can leave the track with no DRM, so never skip silently + except Exception as e: + log.warning(f"Skipping unparseable {drm_type} DRM entry from server: {e!r}") continue return result -def _build_tracks(data: Dict[str, Any]) -> Tracks: +def build_tracks(data: Dict[str, Any]) -> Tracks: tracks = Tracks() - tracks.videos = [_deserialize_video(v) for v in data.get("video", [])] - tracks.audio = [_deserialize_audio(a) for a in data.get("audio", [])] - tracks.subtitles = [_deserialize_subtitle(s) for s in data.get("subtitles", [])] + tracks.videos = [deserialize_video(v) for v in data.get("video", [])] + tracks.audio = [deserialize_audio(a) for a in data.get("audio", [])] + tracks.subtitles = [deserialize_subtitle(s) for s in data.get("subtitles", [])] for track_data, track_obj in [ *zip(data.get("video", []), tracks.videos), *zip(data.get("audio", []), tracks.audio), ]: - drm_objs = _reconstruct_drm(track_data.get("drm")) + drm_objs = reconstruct_drm(track_data.get("drm")) if drm_objs: track_obj.drm = drm_objs tracks.attachments = [ @@ -198,7 +219,7 @@ def _build_tracks(data: Dict[str, Any]) -> Tracks: return tracks -def _resolve_manifest_data(tracks: Tracks, manifests: list) -> None: +def resolve_manifest_data(tracks: Tracks, manifests: list) -> None: """Re-parse serialized manifests and populate track.data for downloading. The server serializes DASH and ISM manifest XML as zlib-compressed base64. @@ -250,7 +271,7 @@ def _resolve_manifest_data(tracks: Tracks, manifests: list) -> None: for remote_track in all_tracks: if remote_track.data.get(m_type): continue - matched = _match_track(remote_track, local_all) + matched = match_track(remote_track, local_all) if matched and matched.data.get(m_type): remote_track.data.update(matched.data) remote_track.descriptor = matched.descriptor @@ -261,7 +282,7 @@ def _resolve_manifest_data(tracks: Tracks, manifests: list) -> None: log_m.warning("Failed to re-parse %s manifest from %s: %s", m_type, m_url, e) -def _match_track(remote_track: Track, local_tracks: list) -> Optional[Track]: +def match_track(remote_track: Track, local_tracks: list) -> Optional[Track]: """Match a remote track to a locally-parsed track by ID or attributes.""" remote_id = str(remote_track.id) for lt in local_tracks: @@ -285,11 +306,12 @@ def _match_track(remote_track: Track, local_tracks: list) -> Optional[Track]: return None -def _build_title(info: Dict[str, Any], service_tag: str, fallback_id: str) -> Union[Episode, Movie]: +def build_title(info: Dict[str, Any], service_tag: str, fallback_id: str) -> Union[Episode, Movie]: svc_class = type(service_tag, (), {}) lang = Language.get(info["language"]) if info.get("language") else None + title: Union[Episode, Movie] if info.get("type") == "episode": - return Episode( + title = Episode( id_=info.get("id", fallback_id), service=svc_class, title=info.get("series_title", "Unknown"), @@ -300,18 +322,49 @@ def _build_title(info: Dict[str, Any], service_tag: str, fallback_id: str) -> Un language=lang, air_date=info.get("air_date"), part=info.get("part"), + absolute=info.get("absolute"), ) - return Movie( - id_=info.get("id", fallback_id), - service=svc_class, - name=info.get("name", "Unknown"), - year=info.get("year"), - language=lang, - ) + if "daily" in info: + title.daily = info["daily"] + else: + title = Movie( + id_=info.get("id", fallback_id), + service=svc_class, + name=info.get("name", "Unknown"), + year=info.get("year"), + language=lang, + ) + # the synthetic service class carries no ANIME/DAILY, so the flags only survive per title + if "anime" in info: + title.anime = info["anime"] + return title + + +def resolve_auth_headers(svc: dict, server_name: str) -> list[str]: + """Configured header names first, then the defaults as fallbacks.""" + auth_headers = svc.get("auth_headers") + if auth_headers is None: + return list(DEFAULT_AUTH_HEADERS) + if ( + not isinstance(auth_headers, list) + or not auth_headers + or not all(isinstance(h, str) and h.strip() for h in auth_headers) + ): + raise click.ClickException( + f"Remote service '{server_name}': 'auth_headers' must be a list of header names, e.g.\n\n" + ' auth_headers: ["Authorization"]' + ) + resolved = [h.strip() for h in auth_headers] + seen = {h.lower() for h in resolved} + resolved.extend(h for h in DEFAULT_AUTH_HEADERS if h.lower() not in seen) + return resolved def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]: - """Resolve server URL, API key, and per-service config from remote_services.""" + """Resolve server URL, API key, and per-service config from remote_services. + + The per-service config carries the ``_auth_headers`` candidate list for RemoteClient. + """ remote_services = config.remote_services if not remote_services: raise click.ClickException( @@ -329,6 +382,7 @@ def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]: raise click.ClickException(f"Remote service '{server_name}' not found. Available: {available}") services = svc.get("services", {}) services["_server_cdm"] = svc.get("server_cdm", False) + services["_auth_headers"] = resolve_auth_headers(svc, server_name) return svc["url"], svc.get("api_key", ""), services if len(remote_services) == 1: @@ -336,13 +390,14 @@ def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]: log.info(f"Using remote service: {name}") services = svc.get("services", {}) services["_server_cdm"] = svc.get("server_cdm", False) + services["_auth_headers"] = resolve_auth_headers(svc, name) return svc["url"], svc.get("api_key", ""), services available = ", ".join(remote_services.keys()) raise click.ClickException(f"Multiple remote services configured. Use --server to select one: {available}") -def _load_credentials_for_transport(service_tag: str, profile: Optional[str]) -> Optional[Dict[str, str]]: +def load_credentials_for_transport(service_tag: str, profile: Optional[str]) -> Optional[Dict[str, str]]: from unshackle.commands.dl import dl credential = dl.get_credentials(service_tag, profile) @@ -354,7 +409,7 @@ def _load_credentials_for_transport(service_tag: str, profile: Optional[str]) -> return None -def _load_cookies_for_transport(service_tag: str, profile: Optional[str]) -> Optional[str]: +def load_cookies_for_transport(service_tag: str, profile: Optional[str]) -> Optional[str]: import zlib from unshackle.commands.dl import dl @@ -365,7 +420,7 @@ def _load_cookies_for_transport(service_tag: str, profile: Optional[str]) -> Opt return None -def _resolve_proxy(proxy_arg: Optional[str]) -> Optional[str]: +def resolve_proxy_arg(proxy_arg: Optional[str]) -> Optional[str]: if not proxy_arg: return None @@ -388,6 +443,8 @@ class RemoteService: ALIASES: tuple[str, ...] = () GEOFENCE: tuple[str, ...] = () NO_SUBTITLES: bool = False + ANIME: bool = False + DAILY: bool = False def __init__( self, @@ -404,7 +461,7 @@ class RemoteService: self.service_tag = service_tag self.title_id = title_id - self.client = RemoteClient(server_url, api_key) + self.client = RemoteClient(server_url, api_key, services_config.get("_auth_headers")) self.ctx = ctx self._service_params = service_params or {} self.log = logging.getLogger(service_tag) @@ -431,9 +488,9 @@ class RemoteService: svc_config = services_config.get(service_tag, {}) self._server_cdm = services_config.get("_server_cdm", False) - self._apply_service_config(svc_config) + self.apply_service_config(svc_config) - def _apply_service_config(self, svc_config: dict) -> None: + def apply_service_config(self, svc_config: dict) -> None: if not svc_config: return config_maps = { @@ -477,16 +534,16 @@ class RemoteService: create_data: Dict[str, Any] = {"service": self.service_tag, "title_id": self.title_id} - credentials = _load_credentials_for_transport(self.service_tag, profile) + credentials = load_credentials_for_transport(self.service_tag, profile) if credentials: create_data["credentials"] = credentials - cookies_text = _load_cookies_for_transport(self.service_tag, profile) + cookies_text = load_cookies_for_transport(self.service_tag, profile) if cookies_text: create_data["cookies"] = cookies_text if not no_proxy and proxy: - resolved_proxy = _resolve_proxy(proxy) + resolved_proxy = resolve_proxy_arg(proxy) if resolved_proxy: create_data["proxy"] = resolved_proxy @@ -527,7 +584,7 @@ class RemoteService: create_data["cdm_type"] = "playready" if is_playready_cdm(cdm) else "widevine" - cache_data = self._load_cache_files() + cache_data = self.load_cache_files() if cache_data: create_data["cache"] = cache_data @@ -536,9 +593,9 @@ class RemoteService: status = result.get("status", "authenticated") if status == "authenticating": - self._poll_auth_completion() + self.poll_auth_completion() - def _poll_auth_completion(self, poll_interval: float = 2.0, timeout: float = 600.0) -> None: + def poll_auth_completion(self, poll_interval: float = 2.0, timeout: float = 600.0) -> None: """Poll the server until authentication completes, handling interactive prompts. When the server needs user input (OTP, device code, PIN), it returns @@ -561,7 +618,11 @@ class RemoteService: if status == "pending_input": prompt = resp.get("prompt", "Enter input: ") - user_response = click.prompt(prompt.rstrip("\n "), default="", show_default=False) + try: + user_response = prompt_user(prompt) + except EOFError: + log.error("Remote auth needs interactive input but stdin is closed") + raise SystemExit(1) from None self.client.post( f"/api/session/{self._session_id}/prompt", {"response": user_response}, @@ -577,7 +638,7 @@ class RemoteService: if self._titles is not None: return self._titles result = self.client.get(f"/api/session/{self._session_id}/titles") - titles_list = [_build_title(t, self.service_tag, self.title_id) for t in result.get("titles", [])] + titles_list = [build_title(t, self.service_tag, self.title_id) for t in result.get("titles", [])] self._titles = ( Series(titles_list) if titles_list and isinstance(titles_list[0], Episode) else Movies(titles_list) ) @@ -597,7 +658,7 @@ class RemoteService: if title_id in self._tracks_by_title: return self._tracks_by_title[title_id] result = self.client.post(f"/api/session/{self._session_id}/tracks", {"title_id": title_id}) - tracks = _build_tracks(result) + tracks = build_tracks(result) for k, v in result.get("session_headers", {}).items(): if k.lower() not in ("host", "content-length", "content-type"): @@ -605,8 +666,11 @@ class RemoteService: for k, v in result.get("session_cookies", {}).items(): self._session.cookies.set(k, v) - _resolve_manifest_data(tracks, result.get("manifests", [])) + resolve_manifest_data(tracks, result.get("manifests", [])) + if self._server_cdm and not result.get("server_cdm", True): + self._server_cdm = False + self.log.warning("Server CDM licensing is not enabled for this key, using the local CDM") self._server_cdm_type = result.get("server_cdm_type", "widevine") self._tracks_by_title[title_id] = tracks @@ -655,7 +719,7 @@ class RemoteService: continue kid_list = list(track_keys.keys()) - drm_obj = self._create_drm_stub(server_drm_type, kid_list) + drm_obj = self.create_drm_stub(server_drm_type, kid_list) for kid_hex, key_hex in track_keys.items(): drm_obj.content_keys[UUID(hex=kid_hex)] = key_hex track.drm = [drm_obj] @@ -669,7 +733,7 @@ class RemoteService: self.log.warning("Failed to resolve server CDM keys: %s", e) @staticmethod - def _create_drm_stub(drm_type: str, kid_hexes: list[str]) -> Any: + def create_drm_stub(drm_type: str, kid_hexes: list[str]) -> Any: """Create a DRM object stub matching the type the server actually used. For server_cdm mode, this is only used for display — keys are already @@ -721,12 +785,12 @@ class RemoteService: return Chapters([Chapter(ch["timestamp"], ch.get("name")) for ch in raw]) def get_widevine_license(self, *, challenge: bytes, title: Title_T, track: AnyTrack) -> Optional[Union[bytes, str]]: - return self._proxy_license(challenge, track, "widevine") + return self.proxy_license(challenge, track, "widevine") def get_playready_license( self, *, challenge: bytes, title: Title_T, track: AnyTrack ) -> Optional[Union[bytes, str]]: - return self._proxy_license(challenge, track, "playready") + return self.proxy_license(challenge, track, "playready") def get_clearkey_license( self, *, challenge: bytes, title: Title_T, track: AnyTrack @@ -751,10 +815,12 @@ class RemoteService: }, ) return base64.b64decode(resp["license"]) - except Exception: + # a missing certificate is legal (services may not use one); log and continue without + except Exception as e: + self.log.debug(f"Service certificate fetch failed: {e!r}") return None - def _proxy_license(self, challenge: Union[bytes, str], track: AnyTrack, drm_type: str) -> bytes: + def proxy_license(self, challenge: Union[bytes, str], track: AnyTrack, drm_type: str) -> bytes: if isinstance(challenge, str): challenge = challenge.encode("utf-8") @@ -824,12 +890,12 @@ class RemoteService: if self._session_id: try: result = self.client.delete(f"/api/session/{self._session_id}") - self._save_returned_cache(result.get("cache", {})) + self.save_returned_cache(result.get("cache", {})) except Exception as e: self.log.warning(f"Failed to clean up remote session: {e}") self._session_id = None - def _save_returned_cache(self, cache_data: Dict[str, str]) -> None: + def save_returned_cache(self, cache_data: Dict[str, str]) -> None: """Save cache files returned by the server to the local cache directory. The server returns updated cache files (e.g. refreshed tokens) on @@ -858,7 +924,7 @@ class RemoteService: self.log.info(f"Saved {len(cache_data)} cache file(s) from server") - def _load_cache_files(self) -> Dict[str, str]: + def load_cache_files(self) -> Dict[str, str]: import zlib cache_dir = config.directories.cache / self.service_tag diff --git a/unshackle/core/service.py b/unshackle/core/service.py index f8964e8c..8578d594 100644 --- a/unshackle/core/service.py +++ b/unshackle/core/service.py @@ -16,11 +16,10 @@ import requests from requests.adapters import HTTPAdapter, Retry from rich.padding import Padding from rich.rule import Rule -from rich.text import Text from unshackle.core.cacher import Cacher from unshackle.core.config import config -from unshackle.core.console import console +from unshackle.core.console import console, prompt_user from unshackle.core.constants import AnyTrack from unshackle.core.credential import Credential from unshackle.core.drm import DRM_T @@ -132,7 +131,7 @@ def sanitize_proxy_for_log(uri: Optional[str]) -> Optional[str]: netloc = f"REDACTED@{netloc.split('@', 1)[1]}" return urlunparse(parsed._replace(netloc=netloc)) - except Exception: + except ValueError: if "@" in uri: return f"REDACTED@{uri.split('@', 1)[1]}" return uri @@ -144,6 +143,8 @@ class Service(metaclass=ABCMeta): # Abstract class variables ALIASES: tuple[str, ...] = () # list of aliases for the service; alternatives to the service tag. GEOFENCE: tuple[str, ...] = () # list of ip regions required to use the service. empty list == no specific region. + ANIME: bool = False # service catalogue is anime; metadata lookups prefer AniList. Title.anime overrides per title. + DAILY: bool = False # catalog is daily/date-based. episodes are named by air date. Title.daily overrides per title. # vault namespace override; when set, key vault read/write uses this tag instead of the service's own. VAULT_TAG: Optional[str] = None # Auth methods the service accepts ("cookies"/"credentials"); when None the REST /services @@ -289,7 +290,7 @@ class Service(metaclass=ABCMeta): self.log.debug(f"Failed to get cached IP info: {e}") self.current_region = None - def _get_tracks_for_variants( + def get_tracks_for_variants( self, title: Title_T, fetch_fn: Callable[..., Tracks], @@ -361,6 +362,9 @@ class Service(metaclass=ABCMeta): return all_tracks + # Deprecated 5.5.0 shim for service repos still on the old underscored name; drop once they migrate. + _get_tracks_for_variants = get_tracks_for_variants + # Optional Abstract functions # The following functions may be implemented by the Service. # Otherwise, the base service code (if any) of the function will be executed on call. @@ -432,9 +436,7 @@ class Service(metaclass=ABCMeta): """ if self._input_bridge is not None: return self._input_bridge.request_input(prompt) - indent = " " * 5 - padded = indent + prompt.replace("\n", "\n" + indent) - return console.input(Text(padded, style="text")) + return prompt_user(prompt) def search(self) -> Generator[SearchResult, None, None]: """ diff --git a/unshackle/core/service_repo.py b/unshackle/core/service_repo.py index 92649bfb..1d708b05 100644 --- a/unshackle/core/service_repo.py +++ b/unshackle/core/service_repo.py @@ -54,10 +54,10 @@ def parse_spec(spec: str) -> tuple[str, Optional[str], str]: url, branch = spec.rsplit("@", 1) if SHORTHAND_RE.match(url) and not url.startswith(("http", "ssh", "git@")): url = f"https://github.com/{url}" - return url, branch, _slug(url) + return url, branch, slug(url) -def _slug(url: str) -> str: +def slug(url: str) -> str: """Filesystem-safe clone dir name from a repo URL, host included so same owner/repo on different hosts don't collide, e.g. github.com__owner__repo.""" cleaned = re.sub(r"^[a-z]+://", "", url).split("@")[-1].replace(":", "/") @@ -91,15 +91,15 @@ def resolve_service_repo(spec: str, *, ttl: int = DEFAULT_TTL, force: bool = Fal stamp = stamp_for(dest) if not dest.exists(): - return _clone(url, branch, dest, stamp) + return clone(url, branch, dest, stamp) if force: - _force_sync(dest, stamp) - elif _is_stale(stamp, ttl): - _pull(dest, stamp) + force_sync(dest, stamp) + elif is_stale(stamp, ttl): + pull(dest, stamp) return dest # ponytail: git IS the cache layer — no bespoke fetch/integrity code -def _clone(url: str, branch: Optional[str], dest: Path, stamp: Path) -> Optional[Path]: +def clone(url: str, branch: Optional[str], dest: Path, stamp: Path) -> Optional[Path]: if not binaries.Git: log.error("git not found on PATH — cannot fetch service repo %s", url) return None @@ -111,26 +111,26 @@ def _clone(url: str, branch: Optional[str], dest: Path, stamp: Path) -> Optional try: subprocess.run(args, check=True, capture_output=True) # nosec B603 except subprocess.CalledProcessError as e: - log.error("failed to clone service repo %s: %s", url, _stderr(e)) + log.error("failed to clone service repo %s: %s", url, stderr_text(e)) shutil.rmtree(dest, ignore_errors=True) # don't leave a half-cloned dir that sticks on retries return None - _write_stamp(stamp) + write_stamp(stamp) log.info("cloned service repo %s", url) return dest -def _pull(dest: Path, stamp: Path) -> None: +def pull(dest: Path, stamp: Path) -> None: if not binaries.Git: - _write_stamp(stamp) + write_stamp(stamp) return # never clobber local edits — refuse to refresh a dirty clone and let the caller exit cleanly - if _is_dirty(dest): + if is_dirty(dest): raise DirtyServiceRepo(dest) try: subprocess.run([str(binaries.Git), "-C", str(dest), "pull", "--ff-only"], check=True, capture_output=True) # nosec B603 except subprocess.CalledProcessError as e: - log.warning("could not update service repo at %s, using existing copy: %s", dest, _stderr(e)) - _write_stamp(stamp) # stamp regardless so a flaky remote can't trigger a fetch every run (force overrides) + log.warning("could not update service repo at %s, using existing copy: %s", dest, stderr_text(e)) + write_stamp(stamp) # stamp regardless so a flaky remote can't trigger a fetch every run (force overrides) def refresh_repo(spec: str) -> tuple[Optional[Path], list[str]]: @@ -143,23 +143,23 @@ def refresh_repo(spec: str) -> tuple[Optional[Path], list[str]]: dest = repos_base() / dest_name stamp = stamp_for(dest) if not dest.exists(): - return _clone(url, branch, dest, stamp), ["cloned (new)"] + return clone(url, branch, dest, stamp), ["cloned (new)"] if not binaries.Git: return dest, [] - old = _head(dest) - discarded = _local_dirty_services(dest) # uncommitted edits the force reset is about to wipe - _force_sync(dest, stamp) + old = head(dest) + discarded = local_dirty_services(dest) # uncommitted edits the force reset is about to wipe + force_sync(dest, stamp) lines = [f"! {tag} (local changes discarded)" for tag in discarded] - lines += _changed_services(dest, old, _head(dest)) # upstream changes pulled in + lines += changed_services(dest, old, head(dest)) # upstream changes pulled in return dest, lines -def _head(dest: Path) -> Optional[str]: +def head(dest: Path) -> Optional[str]: r = subprocess.run([str(binaries.Git), "-C", str(dest), "rev-parse", "HEAD"], capture_output=True) # nosec B603 return r.stdout.decode(errors="ignore").strip() if r.returncode == 0 else None -def _local_dirty_services(dest: Path) -> list[str]: +def local_dirty_services(dest: Path) -> list[str]: """Top-level service dirs with uncommitted edits to tracked files (lost on a force reset).""" r = subprocess.run([str(binaries.Git), "-C", str(dest), "diff", "--name-only", "HEAD"], capture_output=True) # nosec B603 if r.returncode != 0: @@ -167,7 +167,7 @@ def _local_dirty_services(dest: Path) -> list[str]: return sorted({line.split("/")[0] for line in r.stdout.decode(errors="ignore").splitlines() if line}) -def _changed_services(dest: Path, old: Optional[str], new: Optional[str]) -> list[str]: +def changed_services(dest: Path, old: Optional[str], new: Optional[str]) -> list[str]: """Summarise a HEAD old->new diff grouped by top-level service dir, plus a line shortstat.""" if not old or not new or old == new: return [] @@ -192,21 +192,21 @@ def _changed_services(dest: Path, old: Optional[str], new: Optional[str]) -> lis return lines -def _force_sync(dest: Path, stamp: Path) -> None: +def force_sync(dest: Path, stamp: Path) -> None: """Hard-reset the clone to its upstream, discarding local changes (manual refresh only).""" if not binaries.Git: - _write_stamp(stamp) + write_stamp(stamp) return git = str(binaries.Git) try: subprocess.run([git, "-C", str(dest), "fetch", "--depth", "1", "origin"], check=True, capture_output=True) # nosec B603 subprocess.run([git, "-C", str(dest), "reset", "--hard", "@{u}"], check=True, capture_output=True) # nosec B603 except subprocess.CalledProcessError as e: - log.warning("could not force-refresh service repo at %s, using existing copy: %s", dest, _stderr(e)) - _write_stamp(stamp) + log.warning("could not force-refresh service repo at %s, using existing copy: %s", dest, stderr_text(e)) + write_stamp(stamp) -def _is_dirty(dest: Path) -> bool: +def is_dirty(dest: Path) -> bool: """True if the clone has uncommitted edits to tracked files or local commits not on its upstream. Untracked files are intentionally ignored: a `git pull --ff-only` only touches tracked files, so @@ -230,14 +230,14 @@ def _is_dirty(dest: Path) -> bool: return False -def _is_stale(stamp: Path, ttl: int) -> bool: +def is_stale(stamp: Path, ttl: int) -> bool: try: return (time.time() - float(stamp.read_text().strip())) >= ttl except (OSError, ValueError): return True -def _write_stamp(stamp: Path) -> None: +def write_stamp(stamp: Path) -> None: try: stamp.parent.mkdir(parents=True, exist_ok=True) stamp.write_text(str(time.time())) @@ -245,5 +245,5 @@ def _write_stamp(stamp: Path) -> None: pass -def _stderr(e: subprocess.CalledProcessError) -> str: +def stderr_text(e: subprocess.CalledProcessError) -> str: return (e.stderr or b"").decode(errors="ignore").strip() diff --git a/unshackle/core/services.py b/unshackle/core/services.py index aa3ca24d..97195aa3 100644 --- a/unshackle/core/services.py +++ b/unshackle/core/services.py @@ -1,5 +1,6 @@ from __future__ import annotations +import inspect import logging import re from pathlib import Path @@ -14,39 +15,37 @@ from unshackle.core.utils.redact import redact_path log = logging.getLogger("services") -_raw = config.directories.services -if not isinstance(_raw, list): - _raw = [_raw] -# resolve in config order — priority IS list order: the first source to define a tag is the source, +raw = config.directories.services +if not isinstance(raw, list): + raw = [raw] +# resolve in config order - priority IS list order: the first source to define a tag is the source, # later sources (local or repo) with the same tag are shadowed. List local last to make it a fallback. -_service_dirs: list[Path] = [] +service_dirs: list[Path] = [] DIRTY_REPOS: list[str] = [] -for _entry in _raw: +for _entry in raw: if isinstance(_entry, str) and is_repo_spec(_entry): try: _resolved = resolve_service_repo(_entry) except DirtyServiceRepo as e: - # local edits in the clone — record it and let check_load_errors() exit cleanly + # local edits in the clone - record it and let check_load_errors() exit cleanly DIRTY_REPOS.append(redact_path(str(e.path))) _resolved = None if _resolved: - _service_dirs.append(_resolved) + service_dirs.append(_resolved) else: - _service_dirs.append(Path(_entry)) + service_dirs.append(Path(_entry)) # dedupe by tag honoring list order (first wins); track shadows for the load summary -_seen: dict[str, Path] = {} +seen: dict[str, Path] = {} SHADOWED: list[str] = [] -for _dir in _service_dirs: +for _dir in service_dirs: for _path in _dir.glob("*/__init__.py"): tag = _path.parent.stem - if tag in _seen: - SHADOWED.append( - f"{tag}: using {redact_path(str(_seen[tag]))}, ignoring duplicate {redact_path(str(_path))}" - ) + if tag in seen: + SHADOWED.append(f"{tag}: using {redact_path(str(seen[tag]))}, ignoring duplicate {redact_path(str(_path))}") else: - _seen[tag] = _path -_SERVICES = sorted(_seen.values(), key=lambda x: x.parent.stem) + seen[tag] = _path +SERVICES = sorted(seen.values(), key=lambda x: x.parent.stem) def load_service(path: Path) -> object: @@ -59,12 +58,12 @@ def load_service(path: Path) -> object: try: module = import_module_by_path(path) except Exception as e: - raise RuntimeError(f"{tag}: failed to import — {type(e).__name__}: {e} ({path})") from e + raise RuntimeError(f"{tag}: failed to import - {type(e).__name__}: {e} ({path})") from e try: return getattr(module, tag) except AttributeError as e: raise RuntimeError( - f"{tag}: no class named '{tag}' found in {path} — the class name must match the directory name" + f"{tag}: no class named '{tag}' found in {path} - the class name must match the directory name" ) from e @@ -86,12 +85,12 @@ def load_services(paths: list[Path]) -> tuple[dict[str, object], list[str]]: return modules, errors -_MODULES, LOAD_ERRORS = load_services(_SERVICES) +MODULES, LOAD_ERRORS = load_services(SERVICES) -_ALIASES = {tag: getattr(module, "ALIASES", ()) for tag, module in _MODULES.items()} +ALIASES = {tag: getattr(module, "ALIASES", ()) for tag, module in MODULES.items()} -_SUMMARY_LOGGED = False +SUMMARY_LOGGED = False def check_load_errors() -> None: @@ -101,18 +100,18 @@ def check_load_errors() -> None: is rendered once by Click, without a traceback and without cascading through every command that imports this module. Duplicate services (a tag also served by an earlier, higher-priority source, which is ignored) are summarised here; the full - list — naming the path used and the duplicate ignored — is debug-only. + list - naming the path used and the duplicate ignored - is debug-only. """ if DIRTY_REPOS: joined = "\n".join(f" - {p}" for p in DIRTY_REPOS) raise click.ClickException( - "Service repo has local changes — refusing to refresh so your edits are not lost.\n" + "Service repo has local changes - refusing to refresh so your edits are not lost.\n" "Commit and push them to the upstream repo (or revert them), then retry:\n" + joined ) - global _SUMMARY_LOGGED - if not _SUMMARY_LOGGED: - _SUMMARY_LOGGED = True - summary = f"Loaded {len(_MODULES)} services" + global SUMMARY_LOGGED + if not SUMMARY_LOGGED: + SUMMARY_LOGGED = True + summary = f"Loaded {len(MODULES)} services" if SHADOWED: summary += f" ({len(SHADOWED)} duplicate(s) ignored)" log.info(summary) @@ -126,7 +125,7 @@ def check_load_errors() -> None: class Services(click.Group): """Lazy-loaded command group of project services.""" - _remote_services_cache: list[dict] | None = None + remote_services_cache: list[dict] | None = None # Click-specific methods @@ -155,7 +154,7 @@ class Services(click.Group): """ remote = ctx.params.get("remote") or (ctx.parent and ctx.parent.params.get("remote")) if remote: - remote_services = Services._fetch_remote_services(ctx) + remote_services = Services.fetch_remote_services(ctx) if remote_services is not None: return [s["tag"] for s in remote_services] tags = Services.get_tags() @@ -174,11 +173,11 @@ class Services(click.Group): import_file = ctx.params.get("import_file") or (ctx.parent and ctx.parent.params.get("import_file")) if import_file: - return Services._make_import_command(tag, ctx) + return Services.make_import_command(tag, ctx) remote = ctx.params.get("remote") or (ctx.parent and ctx.parent.params.get("remote")) if remote: - return Services._make_remote_command(tag, ctx) + return Services.make_remote_command(tag, ctx) try: service = Services.load(tag) @@ -191,35 +190,58 @@ class Services(click.Group): raise click.ClickException(f"{e}. Available Services: {', '.join(available_services)}") if hasattr(service, "cli"): - return service.cli + cli = service.cli + cli.name = tag + doc = service.__doc__ + if doc and doc.strip() and cli.help in (None, "", doc): + cli.help = Services.docstring_help(doc) + return cli raise click.ClickException(f"Service '{tag}' has no 'cli' method configured.") @staticmethod - def _fetch_remote_services(ctx: click.Context) -> list[dict] | None: + def docstring_help(doc: str) -> str: + """Format a service docstring for Click help, one \\b per paragraph so Click keeps the layout. + + The first paragraph stays unprefixed: it is the summary line, and a \\b there + renders as a stray blank line and empties click's derived short help. + """ + doc = inspect.cleandoc(doc) + if "\b" in doc: + return doc + first, *rest = doc.split("\n\n") + return "\n\n".join([first] + [f"\b\n{p}" for p in rest]) + + @staticmethod + def fetch_remote_services(ctx: click.Context) -> list[dict] | None: """Fetch the service list from the remote server (cached per process).""" - if Services._remote_services_cache is not None: - return Services._remote_services_cache + if Services.remote_services_cache is not None: + return Services.remote_services_cache try: from unshackle.core.remote_service import RemoteClient, resolve_server server_name = ctx.params.get("server") - server_url, api_key, _ = resolve_server(server_name) - client = RemoteClient(server_url, api_key) + server_url, api_key, services_config = resolve_server(server_name) + client = RemoteClient(server_url, api_key, services_config.get("_auth_headers")) result = client.get("/api/services") - Services._remote_services_cache = result.get("services", []) - return Services._remote_services_cache + Services.remote_services_cache = result.get("services", []) + return Services.remote_services_cache + except click.ClickException: + raise except Exception: return None @staticmethod - def _make_remote_command(tag: str, ctx: click.Context) -> click.Command: + def make_remote_command(tag: str, ctx: click.Context) -> click.Command: """Create a Click command for a remote service with server-provided options.""" - svc_info = Services._fetch_remote_service_info(tag, ctx) + svc_info = Services.fetch_remote_service_info(tag, ctx) short_help = svc_info.get("url") if svc_info else None + help_text = svc_info.get("help") if svc_info else None + if help_text: + help_text = Services.docstring_help(help_text) cli_params = svc_info.get("cli_params") if svc_info else None - @click.command(name=tag, short_help=short_help) + @click.command(name=tag, short_help=short_help, help=help_text) @click.argument("title", type=str) @click.pass_context def remote_cli(ctx: click.Context, title: str, **kwargs: object) -> object: @@ -248,7 +270,7 @@ class Services(click.Group): return remote_cli @staticmethod - def _make_import_command(tag: str, ctx: click.Context) -> click.Command: + def make_import_command(tag: str, ctx: click.Context) -> click.Command: """Create a synthetic command that yields an ImportService from an export JSON. Mirrors how remote services are wired so dl.py's result() runs unchanged. @@ -266,16 +288,16 @@ class Services(click.Group): return import_cli @staticmethod - def _fetch_remote_service_info(tag: str, ctx: click.Context) -> dict | None: + def fetch_remote_service_info(tag: str, ctx: click.Context) -> dict | None: """Fetch service info for a specific service from the remote server.""" try: - services = Services._fetch_remote_services(ctx) + services = Services.fetch_remote_services(ctx) if services: for svc in services: if svc.get("tag") == tag: return svc - except Exception: - pass + except Exception as e: + log.debug(f"Remote service info lookup failed for {tag}: {e!r}") return None # Methods intended to be used anywhere @@ -283,14 +305,14 @@ class Services(click.Group): @staticmethod def get_tags() -> list[str]: """Returns a list of service tags from all available Services.""" - return [x.parent.stem for x in _SERVICES] + return [x.parent.stem for x in SERVICES] @staticmethod def get_path(name: str) -> Path: """Get the directory path of a command.""" tag = Services.get_tag(name) - for service in _SERVICES: + for service in SERVICES: if service.parent.stem == tag: return service.parent raise KeyError(f"There is no Service added by the Tag '{name}'") @@ -305,9 +327,9 @@ class Services(click.Group): original_value = value value = value.lower() - for path in _SERVICES: + for path in SERVICES: tag = path.parent.stem - if value in (tag.lower(), *_ALIASES.get(tag, [])): + if value in (tag.lower(), *ALIASES.get(tag, [])): return tag return original_value @@ -315,7 +337,7 @@ class Services(click.Group): @staticmethod def load(tag: str) -> Service: """Load a Service module by Service tag.""" - module = _MODULES.get(tag) + module = MODULES.get(tag) if module: return module diff --git a/unshackle/core/session.py b/unshackle/core/session.py index edb584fa..74cb2f26 100644 --- a/unshackle/core/session.py +++ b/unshackle/core/session.py @@ -45,7 +45,7 @@ READ_TIMEOUT = 30 DEFAULT_IMPERSONATE = rnet.Impersonate.Chrome131 -def _resolve_impersonate(browser: str) -> rnet.Impersonate: +def resolve_impersonate(browser: str) -> rnet.Impersonate: """Resolve a browser string to an rnet.Impersonate preset. Accepts exact rnet preset names (e.g. "Chrome131", "OkHttp4_12", "Edge101"). @@ -62,7 +62,7 @@ def _resolve_impersonate(browser: str) -> rnet.Impersonate: # Map string method names to rnet.Method enum -_METHOD_MAP: dict[str, rnet.Method] = { +METHOD_MAP: dict[str, rnet.Method] = { "GET": rnet.Method.GET, "POST": rnet.Method.POST, "PUT": rnet.Method.PUT, @@ -85,12 +85,12 @@ class RnetResponseHeaders(MutableMapping): def __init__(self, header_map: Any) -> None: self._map = header_map - def _decode(self, val: Any) -> str: + def decode(self, val: Any) -> str: return val.decode("utf-8", errors="replace") if isinstance(val, (bytes, bytearray)) else str(val) def __getitem__(self, key: str) -> str: val = self._map[key] - return self._decode(val) + return self.decode(val) def __setitem__(self, key: str, value: str) -> None: raise TypeError("Response headers are read-only") @@ -106,7 +106,7 @@ class RnetResponseHeaders(MutableMapping): def __iter__(self) -> Iterator[str]: seen: set[str] = set() for k, _ in self._map.items(): - dk = self._decode(k) + dk = self.decode(k) if dk not in seen: seen.add(dk) yield dk @@ -118,10 +118,10 @@ class RnetResponseHeaders(MutableMapping): val = self._map.get(key) if val is None: return default - return self._decode(val) + return self.decode(val) def items(self) -> list[tuple[str, str]]: - return [(self._decode(k), self._decode(v)) for k, v in self._map.items()] + return [(self.decode(k), self.decode(v)) for k, v in self._map.items()] # --------------------------------------------------------------------------- @@ -256,14 +256,14 @@ class RnetSessionHeaders(CaseInsensitiveDict): self._client = client super().__init__() - def _sync(self) -> None: + def sync(self) -> None: """Push current headers to the rnet client.""" if self._client is not None and hasattr(self, "_store") and self._store: self._client.update(headers={k: v for k, v in self.items()}) def __setitem__(self, key: str, value: str) -> None: super().__setitem__(key, value) - self._sync() + self.sync() def update(self, __m: Any = None, **kwargs: Any) -> None: if __m: @@ -275,7 +275,7 @@ class RnetSessionHeaders(CaseInsensitiveDict): super().__setitem__(k, v) for k, v in kwargs.items(): super().__setitem__(k, v) - self._sync() + self.sync() def pop(self, key: str, *args: Any) -> Any: result = super().pop(key, *args) @@ -301,7 +301,7 @@ class RnetCookieAdapter(MutableMapping): self._flat: dict[str, str] = {} self._original_cookies: list[Any] = [] - def _set_cookie_on_client(self, url: str, name: str, value: str) -> None: + def set_cookie_on_client(self, url: str, name: str, value: str) -> None: """Set a cookie on the rnet client, or buffer locally if the client is not yet created.""" if self._client is not None: try: @@ -309,7 +309,7 @@ class RnetCookieAdapter(MutableMapping): except Exception: pass - def _flush_to_client(self) -> None: + def flush_to_client(self) -> None: """Push all buffered cookies to the rnet client once it is created.""" if self._client is None: return @@ -344,20 +344,20 @@ class RnetCookieAdapter(MutableMapping): self._cookies.setdefault(domain, {})[name] = value self._original_cookies.append(cookie) url = f"https://{domain.lstrip('.')}" if domain else "https://localhost" - self._set_cookie_on_client(url, name, value) + self.set_cookie_on_client(url, name, value) elif isinstance(other, dict): for name, value in other.items(): self._flat[name] = value - self._set_cookie_on_client("https://localhost", name, str(value)) + self.set_cookie_on_client("https://localhost", name, str(value)) self._flat.update(other) elif hasattr(other, "items"): for name, value in other.items(): self._flat[name] = str(value) - self._set_cookie_on_client("https://localhost", name, str(value)) + self.set_cookie_on_client("https://localhost", name, str(value)) for name, value in kwargs.items(): self._flat[name] = value - self._set_cookie_on_client("https://localhost", name, value) + self.set_cookie_on_client("https://localhost", name, value) def get( self, name: str, default: Optional[str] = None, domain: Optional[str] = None, path: Optional[str] = None @@ -370,7 +370,7 @@ class RnetCookieAdapter(MutableMapping): self._flat[name] = value self._cookies.setdefault(domain, {})[name] = value url = f"https://{domain.lstrip('.')}" - self._set_cookie_on_client(url, name, value) + self.set_cookie_on_client(url, name, value) def __getitem__(self, name: str) -> str: return self._flat[name] @@ -456,7 +456,7 @@ class RnetProxyDict(dict): super().__init__() self._session = session - def _sync(self) -> None: + def sync(self) -> None: proxy = self.get("all") or self.get("https") or self.get("http") proxies = [rnet.Proxy.all(proxy)] if proxy else [] self._session._client_kwargs["proxies"] = proxies or None @@ -465,11 +465,11 @@ class RnetProxyDict(dict): def update(self, __m: Any = None, **kwargs: Any) -> None: super().update(__m or {}, **kwargs) - self._sync() + self.sync() def __setitem__(self, key: str, value: str) -> None: super().__setitem__(key, value) - self._sync() + self.sync() # --------------------------------------------------------------------------- @@ -582,17 +582,17 @@ class RnetSession: name = str(preset).rsplit(".", 1)[-1] return name or None - def _ensure_client(self) -> rnet.BlockingClient: + def ensure_client(self) -> rnet.BlockingClient: """Lazily create the rnet client on first use, flushing any buffered state.""" if self._client is None: self._client = rnet.BlockingClient(**self._client_kwargs) self.headers._client = self._client - self.headers._sync() + self.headers.sync() self.cookies._client = self._client - self.cookies._flush_to_client() + self.cookies.flush_to_client() return self._client - def _build_url(self, url: str, params: Optional[Any] = None) -> str: + def build_url(self, url: str, params: Optional[Any] = None) -> str: """Encode params into the URL (rnet ignores the params kwarg). Accepts the same shapes as requests: a mapping, a sequence of pairs, or a @@ -631,7 +631,7 @@ class RnetSession: return min(sleep_time, self.max_backoff) def request(self, method: str, url: str, **kwargs: Any) -> RnetResponse: - client = self._ensure_client() + client = self.ensure_client() method_upper = method.upper() if isinstance(method, str) else str(method).upper() # Per-request override of the session retry budget (rnet.request() would reject the kwarg). @@ -641,7 +641,7 @@ class RnetSession: max_retries = self.max_retries # Build URL with params - url = self._build_url(url, kwargs.pop("params", None)) + url = self.build_url(url, kwargs.pop("params", None)) # Default allow_redirects=True kwargs.setdefault("allow_redirects", True) @@ -664,7 +664,7 @@ class RnetSession: kwargs["body"] = data # Resolve method enum - rnet_method = _METHOD_MAP.get(method_upper) + rnet_method = METHOD_MAP.get(method_upper) if rnet_method is None: raise ValueError(f"Unsupported HTTP method: {method}") @@ -813,7 +813,7 @@ def session( if browser is None: browser = config.network.get("browser", "Chrome131") - impersonate = _resolve_impersonate(browser) + impersonate = resolve_impersonate(browser) session_kwargs: dict[str, Any] = {"impersonate": impersonate} # optional rnet client knobs, see docs/NETWORK_CONFIG.md diff --git a/unshackle/core/themes.py b/unshackle/core/themes.py new file mode 100644 index 00000000..56f760f8 --- /dev/null +++ b/unshackle/core/themes.py @@ -0,0 +1,231 @@ +""" +Named CLI color themes. + +Each palette maps the same role keys to colors: `text`/`text2` for body copy, `pink` for the +brand accent, the six ANSI hues, and the gray ramp used for rules and dim details. The active +palette drives both the rich Console theme (banner, logs, progress) and the click help screens, +so a single `theme` config value restyles the whole CLI. +""" + +from __future__ import annotations + +from rich_click import RichHelpConfiguration + +DEFAULT_THEME = "catppuccin-mocha" + +PALETTES: dict[str, dict[str, str]] = { + # Colors based on "CatppuccinMocha" from Gogh themes + "catppuccin-mocha": { + "bg": "rgb(30,30,46)", + "text": "rgb(205,214,244)", + "text2": "rgb(162,169,193)", # slightly darker + "black": "rgb(69,71,90)", + "bright_black": "rgb(88,91,112)", + "red": "rgb(243,139,168)", + "green": "rgb(166,227,161)", + "yellow": "rgb(249,226,175)", + "blue": "rgb(137,180,250)", + "pink": "rgb(245,194,231)", + "cyan": "rgb(148,226,213)", + "gray": "rgb(166,173,200)", + "bright_gray": "rgb(186,194,222)", + "dark_gray": "rgb(54,54,84)", + }, + # draculatheme.com spec; purple stands in for blue, comment for the dim grays + "dracula": { + "bg": "#282a36", + "text": "#f8f8f2", + "text2": "#b8bfd8", + "black": "#21222c", + "bright_black": "#6272a4", + "red": "#ff5555", + "green": "#50fa7b", + "yellow": "#f1fa8c", + "blue": "#bd93f9", + "pink": "#ff79c6", + "cyan": "#8be9fd", + "gray": "#6272a4", + "bright_gray": "#e9e9f4", + "dark_gray": "#44475a", + }, + # nordtheme.com: polar night / snow storm / frost / aurora + "nord": { + "bg": "#2e3440", + "text": "#d8dee9", + "text2": "#9aa4b8", + "black": "#3b4252", + "bright_black": "#4c566a", + "red": "#bf616a", + "green": "#a3be8c", + "yellow": "#ebcb8b", + "blue": "#81a1c1", + "pink": "#b48ead", + "cyan": "#88c0d0", + "gray": "#616e88", + "bright_gray": "#e5e9f0", + "dark_gray": "#434c5e", + }, + # github.com/morhetz/gruvbox dark, bright variants + "gruvbox": { + "bg": "#282828", + "text": "#ebdbb2", + "text2": "#bdae93", + "black": "#504945", + "bright_black": "#665c54", + "red": "#fb4934", + "green": "#b8bb26", + "yellow": "#fabd2f", + "blue": "#83a598", + "pink": "#d3869b", + "cyan": "#8ec07c", + "gray": "#928374", + "bright_gray": "#d5c4a1", + "dark_gray": "#3c3836", + }, + # Atom One Dark syntax palette + "one-dark": { + "bg": "#282c34", + "text": "#abb2bf", + "text2": "#818896", + "black": "#3b4048", + "bright_black": "#5c6370", + "red": "#e06c75", + "green": "#98c379", + "yellow": "#e5c07b", + "blue": "#61afef", + "pink": "#c678dd", + "cyan": "#56b6c2", + "gray": "#5c6370", + "bright_gray": "#c8ccd4", + "dark_gray": "#3e4451", + }, + # github.com/enkia/tokyo-night-vscode-theme, night variant + "tokyo-night": { + "bg": "#1a1b26", + "text": "#c0caf5", + "text2": "#a9b1d6", + "black": "#15161e", + "bright_black": "#414868", + "red": "#f7768e", + "green": "#9ece6a", + "yellow": "#e0af68", + "blue": "#7aa2f7", + "pink": "#bb9af7", + "cyan": "#7dcfff", + "gray": "#565f89", + "bright_gray": "#a9b1d6", + "dark_gray": "#292e42", + }, + # classic Monokai; purple stands in for pink, 66d9ef doubles as blue + "monokai": { + "bg": "#272822", + "text": "#f8f8f2", + "text2": "#cfcfc2", + "black": "#2e2e2e", + "bright_black": "#75715e", + "red": "#f92672", + "green": "#a6e22e", + "yellow": "#e6db74", + "blue": "#66d9ef", + "pink": "#ae81ff", + "cyan": "#a1efe4", + "gray": "#a59f85", + "bright_gray": "#cfcfc2", + "dark_gray": "#3e3d32", + }, + # flexoki.com dark, 400-level accents + "flexoki": { + "bg": "#100f0f", + "text": "#cecdc3", + "text2": "#878580", + "black": "#282726", + "bright_black": "#403e3c", + "red": "#d14d41", + "green": "#879a39", + "yellow": "#d0a215", + "blue": "#4385be", + "pink": "#ce5d97", + "cyan": "#3aa99f", + "gray": "#6f6e69", + "bright_gray": "#b7b5ac", + "dark_gray": "#343331", + }, + # ethanschoonover.com/solarized dark; magenta stands in for pink + "solarized-dark": { + "bg": "#002b36", + "text": "#839496", + "text2": "#586e75", + "black": "#073642", + "bright_black": "#586e75", + "red": "#dc322f", + "green": "#859900", + "yellow": "#b58900", + "blue": "#268bd2", + "pink": "#d33682", + "cyan": "#2aa198", + "gray": "#657b83", + "bright_gray": "#93a1a1", + "dark_gray": "#073642", + }, +} + +ALIASES = { + "default": DEFAULT_THEME, + "catppuccin": DEFAULT_THEME, + "mocha": DEFAULT_THEME, + "onedark": "one-dark", + "tokyonight": "tokyo-night", + "solarized": "solarized-dark", +} + + +def resolve_palette(name: object) -> dict[str, str] | None: + """Look up a palette by theme name or alias, case-insensitively. + + Takes any YAML scalar since config values are unvalidated; non-strings never match. + Returns a copy: console.py extends the active palette in place, and that must + not pollute the master definitions. + """ + key = str(name or "").strip().lower().replace("_", "-") + palette = PALETTES.get(ALIASES.get(key, key)) + return dict(palette) if palette else None + + +def build_help_config(palette: dict[str, str]) -> RichHelpConfiguration: + """Build the rich-click help styling for a palette.""" + return RichHelpConfiguration( + style_option=palette["text"], + style_switch=palette["green"], + style_argument=palette["cyan"], + style_command=palette["pink"], + style_metavar=palette["yellow"], + style_metavar_separator=f"dim {palette['gray']}", + style_usage=palette["yellow"], + style_usage_command=palette["text"], + style_helptext_first_line=palette["text"], + style_helptext=palette["text2"], + style_option_help=palette["text"], + style_command_help=palette["text"], + style_option_default=f"dim {palette['gray']}", + style_option_envvar=f"dim {palette['yellow']}", + style_required_short=palette["red"], + style_required_long=f"dim {palette['red']}", + style_errors_suggestion=palette["text2"], + style_aborted=palette["red"], + style_options_panel_box=None, + style_options_panel_border=palette["pink"], + style_options_panel_title_style=palette["pink"], + style_commands_panel_box=None, + style_commands_panel_border=palette["pink"], + style_commands_panel_title_style=palette["pink"], + style_errors_panel_box="SQUARE", + style_errors_panel_border=palette["red"], + style_options_table_box="HORIZONTALS", + style_options_table_show_lines=True, + style_options_table_border_style=palette["dark_gray"], + ) + + +def apply_help_theme(palette: dict[str, str]) -> None: + """Point the patched click help renderer at this palette (read at render time).""" + build_help_config(palette).dump_to_globals() diff --git a/unshackle/core/title_cacher.py b/unshackle/core/title_cacher.py index da386c3f..3726518b 100644 --- a/unshackle/core/title_cacher.py +++ b/unshackle/core/title_cacher.py @@ -28,7 +28,7 @@ class TitleCacher: self.stats = {"hits": 0, "misses": 0, "fallbacks": 0} self.no_cache = False - def _generate_cache_key( + def generate_cache_key( self, title_id: str, region: Optional[str] = None, account_hash: Optional[str] = None ) -> str: """ @@ -92,7 +92,7 @@ class TitleCacher: return fetch_function() # Generate cache key - cache_key = self._generate_cache_key(title_id, region, account_hash) + cache_key = self.generate_cache_key(title_id, region, account_hash) # If reset_cache flag is set, clear the cache entry if reset_cache: @@ -193,7 +193,7 @@ class TitleCacher: if not config.title_cache_enabled or self.no_cache: return None - cache_key = self._generate_cache_key(title_id, region, account_hash) + cache_key = self.generate_cache_key(title_id, region, account_hash) cache = self.cacher.get(cache_key, version=1) if not cache or not cache.data: @@ -246,7 +246,7 @@ class TitleCacher: if not config.title_cache_enabled or self.no_cache: return - cache_key = self._generate_cache_key(title_id, region, account_hash) + cache_key = self.generate_cache_key(title_id, region, account_hash) cache = self.cacher.get(cache_key, version=1) if not cache or not cache.data: diff --git a/unshackle/core/titles/episode.py b/unshackle/core/titles/episode.py index 4b1d5d7a..fc3bec1d 100644 --- a/unshackle/core/titles/episode.py +++ b/unshackle/core/titles/episode.py @@ -16,8 +16,10 @@ from unshackle.core.utils.template_formatter import TemplateFormatter, detect_sp class Episode(Title): - # class-level default so Episodes restored from an older title cache read as part-less + # class-level defaults so Episodes restored from an older title cache read as part-less + # and without an absolute number part: Optional[int] = None + absolute: Optional[int] = None def __init__( self, @@ -33,6 +35,7 @@ class Episode(Title): description: Optional[str] = None, air_date: Optional[Union[date, str]] = None, part: Optional[Union[int, str]] = None, + absolute: Optional[Union[int, str]] = None, ) -> None: super().__init__(id_, service, language, data) @@ -65,6 +68,15 @@ class Episode(Title): if part <= 0: raise ValueError(f"Episode part cannot be {part}") + if absolute is not None: + if isinstance(absolute, str) and absolute.isdigit(): + absolute = int(absolute) + # bool is an int subclass; True would render as "001" in filenames + elif isinstance(absolute, bool) or not isinstance(absolute, int): + raise TypeError(f"Expected absolute to be an int, not {absolute!r}") + if absolute <= 0: + raise ValueError(f"Episode absolute cannot be {absolute}") + if name is not None and not isinstance(name, str): raise TypeError(f"Expected name to be a str, not {name!r}") @@ -96,6 +108,9 @@ class Episode(Title): except ValueError: pass + if isinstance(air_date, date) and air_date.year < 1970: + raise ValueError(f"Episode air date cannot be {air_date}") + self.title = title self.season = season self.number = number @@ -104,31 +119,35 @@ class Episode(Title): self.description = description self.air_date = air_date self.part = part + self.absolute = absolute def matches_wanted(self, wanted: Collection[str]) -> bool: """Whether a parsed ``-w`` key set selects this episode. A part-ful episode answers to both its base key and its part key, so ``-w s1e1`` - takes every part. ``!`` keys are part-qualified exclusions resolved here. + takes every part. A dated episode also answers to its ISO air date. + ``!`` keys are part-qualified exclusions resolved here. """ base = f"{self.season}x{self.number}" keys = (base,) if self.part is None else (base, f"{base}.{self.part}") + if isinstance(self.air_date, date): + keys = (*keys, self.air_date.isoformat()) if any(f"!{k}" in wanted for k in keys): return False return any(k in wanted for k in keys) - def _part_suffix(self) -> str: + def part_suffix(self) -> str: """``.Part.2`` / `` Part 2`` in the series template's own separator style.""" if self.part is None: return "" sep = config.get_template_separator("series") if config.output_template.get("series") else "." return f"{sep}Part{sep}{self.part}" - def _build_template_context( + def build_template_context( self, media_info: MediaInfo, show_service: bool = True, include_part: bool = True ) -> dict: """Build template context dictionary from MediaInfo.""" - context = self._build_base_template_context(media_info, show_service) + context = self.build_base_template_context(media_info, show_service) context["title"] = self.title.replace("$", "S") context["year"] = self.year or "" context["season"] = f"S{self.season:02}" @@ -137,9 +156,10 @@ class Episode(Title): context["episode_name"] = self.name or "" context["date"] = "" context["part"] = self.part if include_part and self.part is not None else "" + context["absolute"] = f"{self.absolute:03}" if self.absolute is not None else "" if self.air_date: # daily/sports: air date replaces SxxExx - disp = self._air_date_display() + disp = self.air_date_display() context["season"] = disp context["episode"] = "" context["season_episode"] = disp @@ -148,25 +168,25 @@ class Episode(Title): if include_part and self.part is not None: # folded into the identity tokens, not a token of its own: a standalone {part} is # absent from every shipped template, so two parts would render the same filename - suffix = self._part_suffix() + suffix = self.part_suffix() context["episode"] = f"{context['episode']}{suffix}" context["season_episode"] = f"{context['season_episode']}{suffix}" return context - def _air_date_display(self) -> str: + def air_date_display(self) -> str: """Render air_date using the series template's own separator (dots or spaces).""" if isinstance(self.air_date, date): sep = config.get_template_separator("series") if config.output_template.get("series") else "." return f"{self.air_date.year:04}{sep}{self.air_date.month:02}{sep}{self.air_date.day:02}" return str(self.air_date) - def _folder_season(self) -> str: + def folder_season(self) -> str: """Season folder label: air year for dated content, else SxxExx-style season.""" if isinstance(self.air_date, date): return f"{self.air_date.year:04}" return f"S{self.season:02}" - def _part_label(self) -> str: + def part_label(self) -> str: """``.2`` selection-syntax suffix for console output, empty when part-less.""" return f".{self.part}" if self.part is not None else "" @@ -177,8 +197,8 @@ class Episode(Title): return "{title}{year} {date}{part} {name}".format( title=self.title, year=f" {self.year}" if self.year else "", - date=self._air_date_display(), - part=self._part_label(), + date=self.air_date_display(), + part=self.part_label(), name=self.name or "", ).strip() # the console shows the -w selection syntax (S01E01.2), not the filename form @@ -187,7 +207,7 @@ class Episode(Title): year=f" {self.year}" if self.year else "", season=self.season, number=self.number, - part=self._part_label(), + part=self.part_label(), name=self.name or "", ).strip() @@ -196,8 +216,8 @@ class Episode(Title): template = config.get_folder_template("series") if template: # all parts of an episode land in the same season folder - context = self._build_template_context(media_info, show_service, include_part=False) - context["season"] = self._folder_season() + context = self.build_template_context(media_info, show_service, include_part=False) + context["season"] = self.folder_season() context["year"] = self.year or "" # folders keep the year spacer = detect_spacer(template) # one style for the whole path segments = [ @@ -220,8 +240,8 @@ class Episode(Title): derived_template = re.sub(r"^[\.\s]+|[\.\s]+$", "", derived_template) formatter = TemplateFormatter(derived_template) - context = self._build_template_context(media_info, show_service, include_part=False) - context["season"] = self._folder_season() + context = self.build_template_context(media_info, show_service, include_part=False) + context["season"] = self.folder_season() context["year"] = self.year or "" # folders keep the year folder_name = formatter.format(context) @@ -233,11 +253,11 @@ class Episode(Title): name = f"{self.title}" if self.year: name += f" {self.year}" - name += f" {self._folder_season()}" + name += f" {self.folder_season()}" return sanitize_filename(name, " ") formatter = TemplateFormatter(config.output_template["series"]) - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) return formatter.format(context) @@ -269,10 +289,10 @@ class Series(SortedKeyList, ABC): for episode in self: if episode.season == season: label = ( - episode._air_date_display() + episode.air_date_display() if episode.air_date else str(episode.number).zfill(len(str(episodes))) - ) + episode._part_label() + ) + episode.part_label() if episode.name: season_tree.add(f"[bold]{label}.[/] [bright_black]{episode.name}") elif episode.air_date: diff --git a/unshackle/core/titles/movie.py b/unshackle/core/titles/movie.py index f02a6223..6d6508c5 100644 --- a/unshackle/core/titles/movie.py +++ b/unshackle/core/titles/movie.py @@ -46,9 +46,9 @@ class Movie(Title): self.year = year self.description = description - def _build_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: + def build_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: """Build template context dictionary from MediaInfo.""" - context = self._build_base_template_context(media_info, show_service) + context = self.build_base_template_context(media_info, show_service) context["title"] = self.name.replace("$", "S") context["year"] = self.year or "" return context @@ -62,7 +62,7 @@ class Movie(Title): if folder: template = config.get_folder_template("movies") if template: - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) spacer = detect_spacer(template) # one style for the whole path segments = [ TemplateFormatter(seg, spacer).format(context) @@ -76,7 +76,7 @@ class Movie(Title): return sanitize_filename(name, " ") formatter = TemplateFormatter(config.output_template["movies"]) - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) return formatter.format(context) diff --git a/unshackle/core/titles/music.py b/unshackle/core/titles/music.py index 41aba6ff..a6b0d1b1 100644 --- a/unshackle/core/titles/music.py +++ b/unshackle/core/titles/music.py @@ -144,9 +144,9 @@ class Song(Title): artist=self.artist, album=self.album, year=self.year, track=self.track, name=self.name ).strip() - def _build_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: + def build_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: """Build template context dictionary from MediaInfo.""" - context = self._build_base_template_context(media_info, show_service) + context = self.build_base_template_context(media_info, show_service) context["title"] = self.name.replace("$", "S") context["year"] = self.year or "" context["track_number"] = f"{self.track:02}" @@ -170,7 +170,7 @@ class Song(Title): # legacy "songs" folder template, then to "{artist} - {album} ({year})". template = config.get_folder_template("albums") or config.get_folder_template("songs") if template: - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) spacer = detect_spacer(template) # one style for the whole path segments = [ TemplateFormatter(seg, spacer).format(context) @@ -185,7 +185,7 @@ class Song(Title): template = config.output_template.get("songs") or "{track_number}. {title}" formatter = TemplateFormatter(template) - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) return formatter.format(context) diff --git a/unshackle/core/titles/song.py b/unshackle/core/titles/song.py index af114c93..8a7766b9 100644 --- a/unshackle/core/titles/song.py +++ b/unshackle/core/titles/song.py @@ -82,9 +82,9 @@ class Song(Title): artist=self.artist, album=self.album, year=self.year, track=self.track, name=self.name ).strip() - def _build_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: + def build_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: """Build template context dictionary from MediaInfo.""" - context = self._build_base_template_context(media_info, show_service) + context = self.build_base_template_context(media_info, show_service) context["title"] = self.name.replace("$", "S") context["year"] = self.year or "" context["track_number"] = f"{self.track:02}" @@ -97,7 +97,7 @@ class Song(Title): if folder: template = config.get_folder_template("songs") if template: - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) spacer = detect_spacer(template) # one style for the whole path segments = [ TemplateFormatter(seg, spacer).format(context) @@ -111,7 +111,7 @@ class Song(Title): return sanitize_filename(name, " ") formatter = TemplateFormatter(config.output_template["songs"]) - context = self._build_template_context(media_info, show_service) + context = self.build_template_context(media_info, show_service) return formatter.format(context) diff --git a/unshackle/core/titles/title.py b/unshackle/core/titles/title.py index 91224e3a..fe351806 100644 --- a/unshackle/core/titles/title.py +++ b/unshackle/core/titles/title.py @@ -47,13 +47,15 @@ class Title: self.service = service self.language = language self.data = data + self.anime: Optional[bool] = None + self.daily: Optional[bool] = None self.tracks = Tracks() def __eq__(self, other: Title) -> bool: return self.id == other.id - def _build_base_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: + def build_base_template_context(self, media_info: MediaInfo, show_service: bool = True) -> dict: """Build base template context dictionary from MediaInfo. Extracts video, audio, HDR, HFR, and multi-language information shared @@ -130,7 +132,8 @@ class Title: track_height = primary_video_track.height if abs(resolution - track_height) <= 10 or track_height in (2160, 1440, 1080, 720, 480): resolution = track_height - except Exception: + # garbage DAR strings or a missing height fall back to the plain min(width, height) + except (ValueError, TypeError, ZeroDivisionError): pass scan_suffix = "i" if str(getattr(primary_video_track, "scan_type", "")).lower() == "interlaced" else "p" diff --git a/unshackle/core/tracks/hybrid.py b/unshackle/core/tracks/hybrid.py index 1f95d403..538ade44 100644 --- a/unshackle/core/tracks/hybrid.py +++ b/unshackle/core/tracks/hybrid.py @@ -435,7 +435,7 @@ class Hybrid: max_fall = int(float(stripped.split("MaxFALL:")[1].split("nits")[0].strip().rstrip(","))) if any(v is None for v in (max_cll, max_fall, max_mdl, min_mdl)): - base_max_mdl, base_min_mdl, base_cll, base_fall = self._probe_hdr_metadata() + base_max_mdl, base_min_mdl, base_cll, base_fall = self.probe_hdr_metadata() if max_cll is None: max_cll = base_cll if max_fall is None: @@ -536,7 +536,7 @@ class Hybrid: success=True, ) - def extract_hdr10plus(self, _video): + def extract_hdr10plus(self, video): """Extract HDR10+ metadata from the video stream""" if os.path.isfile(config.directories.temp / self.hdr10plus_file): return @@ -584,7 +584,7 @@ class Hybrid: success=True, ) - def _probe_hdr_metadata(self): + def probe_hdr_metadata(self): """Extract mastering display and content light level metadata from the HDR10 stream via ffprobe. Returns (max_mdl, min_mdl, max_cll, max_fall) in dovi_tool level6 units: @@ -652,7 +652,7 @@ class Hybrid: with console.status("Converting HDR10+ metadata to Dolby Vision...", spinner="dots"): # Extract actual HDR metadata from the source stream - max_mdl, min_mdl, max_cll, max_fall = self._probe_hdr_metadata() + max_mdl, min_mdl, max_cll, max_fall = self.probe_hdr_metadata() max_mdl, min_mdl, max_cll, max_fall = self.sanitize_l6(max_mdl, min_mdl, max_cll, max_fall) # First create the extra metadata JSON for dovi_tool diff --git a/unshackle/core/tracks/subtitle.py b/unshackle/core/tracks/subtitle.py index ca935cc8..75fa87d4 100644 --- a/unshackle/core/tracks/subtitle.py +++ b/unshackle/core/tracks/subtitle.py @@ -97,10 +97,10 @@ class Subtitle(Track): raise ValueError(f"The Content Profile '{profile}' is not a supported Subtitle Codec") # WebVTT sanitization patterns (compiled once for performance) - _CUE_ID_PATTERN = re.compile(r"^[A-Za-z]+\d+$") - _TIMING_START_PATTERN = re.compile(r"^\d+:\d+[:\.]") - _TIMING_LINE_PATTERN = re.compile(r"^((?:\d+:)?\d+:\d+[.,]\d+)\s*-->\s*((?:\d+:)?\d+:\d+[.,]\d+)(.*)$") - _LINE_POS_PATTERN = re.compile(r"line:(\d+(?:\.\d+)?%?)") + CUE_ID_PATTERN = re.compile(r"^[A-Za-z]+\d+$") + TIMING_START_PATTERN = re.compile(r"^\d+:\d+[:\.]") + TIMING_LINE_PATTERN = re.compile(r"^((?:\d+:)?\d+:\d+[.,]\d+)\s*-->\s*((?:\d+:)?\d+:\d+[.,]\d+)(.*)$") + LINE_POS_PATTERN = re.compile(r"line:(\d+(?:\.\d+)?%?)") def __init__( self, @@ -302,6 +302,9 @@ class Subtitle(Track): Subtitle.filter_unwanted_cues(caption_set) subtitle_text = pycaption.WebVTTWriter().write(caption_set) self.path.write_text(subtitle_text, encoding="utf8") + except pycaption.exceptions.CaptionReadNoCaptions: + # some renditions carry headers but no cues; write them out rather than fail the download + self.path.write_text(text, encoding="utf8") except pycaption.exceptions.CaptionReadSyntaxError: # If first attempt fails, try more aggressive sanitization text = Subtitle.sanitize_webvtt(text) @@ -355,12 +358,12 @@ class Subtitle(Track): for i, line in enumerate(lines): line = line.strip() - if Subtitle._CUE_ID_PATTERN.match(line): + if Subtitle.CUE_ID_PATTERN.match(line): # Look ahead to see if next non-empty line is a timing line j = i + 1 while j < len(lines) and not lines[j].strip(): j += 1 - if j < len(lines) and ("-->" in lines[j] or Subtitle._TIMING_START_PATTERN.match(lines[j].strip())): + if j < len(lines) and ("-->" in lines[j] or Subtitle.TIMING_START_PATTERN.match(lines[j].strip())): return True return False @@ -391,12 +394,12 @@ class Subtitle(Track): line = lines[i].strip() # Check if this line is a cue identifier followed by a timing line - if Subtitle._CUE_ID_PATTERN.match(line): + if Subtitle.CUE_ID_PATTERN.match(line): # Look ahead to see if next non-empty line is a timing line j = i + 1 while j < len(lines) and not lines[j].strip(): j += 1 - if j < len(lines) and ("-->" in lines[j] or Subtitle._TIMING_START_PATTERN.match(lines[j].strip())): + if j < len(lines) and ("-->" in lines[j] or Subtitle.TIMING_START_PATTERN.match(lines[j].strip())): # This is a cue identifier, skip it i += 1 continue @@ -407,7 +410,7 @@ class Subtitle(Track): return "\n".join(sanitized_lines) @staticmethod - def _parse_vtt_time(t: str) -> int: + def parse_vtt_time(t: str) -> int: """Parse WebVTT timestamp to milliseconds. Returns 0 for malformed input.""" try: t = t.replace(",", ".") @@ -443,10 +446,10 @@ class Subtitle(Track): """ timings = [] for line in text.split("\n"): - match = Subtitle._TIMING_LINE_PATTERN.match(line) + match = Subtitle.TIMING_LINE_PATTERN.match(line) if match: start_str, end_str = match.group(1), match.group(2) - timings.append((Subtitle._parse_vtt_time(start_str), Subtitle._parse_vtt_time(end_str))) + timings.append((Subtitle.parse_vtt_time(start_str), Subtitle.parse_vtt_time(end_str))) # Check for overlapping cues (within 50ms start, same end) for i in range(len(timings) - 1): @@ -493,11 +496,11 @@ class Subtitle(Track): i += 1 continue - match = Subtitle._TIMING_LINE_PATTERN.match(line) + match = Subtitle.TIMING_LINE_PATTERN.match(line) if match: start_str, end_str, settings = match.groups() line_pos = 100.0 # Default to bottom - line_match = Subtitle._LINE_POS_PATTERN.search(settings) + line_match = Subtitle.LINE_POS_PATTERN.search(settings) if line_match: pos_str = line_match.group(1).rstrip("%") line_pos = float(pos_str) @@ -510,8 +513,8 @@ class Subtitle(Track): cues.append( { - "start_ms": Subtitle._parse_vtt_time(start_str), - "end_ms": Subtitle._parse_vtt_time(end_str), + "start_ms": Subtitle.parse_vtt_time(start_str), + "end_ms": Subtitle.parse_vtt_time(end_str), "start_str": start_str, "end_str": end_str, "line_pos": line_pos, @@ -1049,8 +1052,8 @@ class Subtitle(Track): if status is True: stripped.save(self.path) return - except Exception: - pass # Fall through to other methods + except Exception as e: # Fall through to other methods + logging.getLogger("Subtitle").debug(f"subby SDH strip failed, falling back: {e!r}") conversion_method = config.subtitle.get("conversion_method", "auto") use_subtitleedit = sdh_method == "subtitleedit" or ( @@ -1102,8 +1105,9 @@ class Subtitle(Track): sub.save() except (IOError, OSError) as e: if "is not valid subtitle file" in str(e): - self.log.warning(f"Failed to strip SDH from {self.path.name}: {e}") - self.log.warning("Continuing without SDH stripping for this subtitle") + log = logging.getLogger("Subtitle") + log.warning(f"Failed to strip SDH from {self.path.name}: {e}") + log.warning("Continuing without SDH stripping for this subtitle") else: raise diff --git a/unshackle/core/tracks/track.py b/unshackle/core/tracks/track.py index 83b2c660..1493e63a 100644 --- a/unshackle/core/tracks/track.py +++ b/unshackle/core/tracks/track.py @@ -58,7 +58,7 @@ def direct_session(session: Union[Session, "RnetSession"]) -> Session: return new -def _read_top_level_box(path: Path, box_type: bytes) -> Optional[bytes]: +def read_top_level_box(path: Path, box_type: bytes) -> Optional[bytes]: # seek through top-level box headers; only the wanted box's bytes are read into RAM file_size = path.stat().st_size with path.open("rb") as f: @@ -94,8 +94,8 @@ def has_encrypted_sample_entry(path: Path) -> bool: scoping avoids chance byte collisions in mdat. Any read error -> False. """ try: - moov = _read_top_level_box(path, b"moov") - except Exception: + moov = read_top_level_box(path, b"moov") + except OSError: return False if not moov: return False @@ -110,7 +110,7 @@ def has_encrypted_sample_entry(path: Path) -> bool: return False -def _senc_protects_samples(buf: bytes, body: int, end: int) -> bool: +def senc_protects_samples(buf: bytes, body: int, end: int) -> bool: """True if a senc/PIFF-uuid box describes protected samples. An empty senc (sample_count 0) sits on a genuinely clear fragment, so treating it as @@ -124,7 +124,7 @@ def _senc_protects_samples(buf: bytes, body: int, end: int) -> bool: return int.from_bytes(buf[pos : pos + 4], "big") > 0 -def _moof_still_encrypted(moof: bytes) -> bool: +def moof_still_encrypted(moof: bytes) -> bool: # senc only: a decrypter detaches just the atom it consumed, so a PIFF uuid can # outlive a good decrypt. ISM arrives as senc via piff_senc_to_cenc. # structural walk: a 4CC byte scan collides with trun payload @@ -135,7 +135,7 @@ def _moof_still_encrypted(moof: bytes) -> bool: if box_type != b"traf": continue for child, _usertype, child_body, child_end in iter_boxes(moof, traf_body, traf_end): - if child == b"senc" and _senc_protects_samples(moof, child_body, child_end): + if child == b"senc" and senc_protects_samples(moof, child_body, child_end): return True return False @@ -194,7 +194,7 @@ def assert_fragments_decrypted(path: Path) -> None: if header[4:8] == b"moof": total += 1 f.seek(start) - if _moof_still_encrypted(f.read(size)): + if moof_still_encrypted(f.read(size)): surviving += 1 if first_offset is None: first_offset = start @@ -926,7 +926,7 @@ class Track: original_path = self.path output_path = original_path.with_stem(f"{original_path.stem}_repack") - def _ffmpeg(extra_args: list[str] = None, bsf: Optional[str] = None): + def ffmpeg(extra_args: list[str] = None, bsf: Optional[str] = None): args = [ binaries.FFMPEG, "-nostdin", @@ -983,18 +983,18 @@ class Track: bsf_applied = False try: - _ffmpeg(bsf=bsf_v) + ffmpeg(bsf=bsf_v) bsf_applied = bsf_v is not None except subprocess.CalledProcessError as e: if b"Malformed AAC bitstream detected" in e.stderr: # e.g., TruTV's dodgy encodes - _ffmpeg(["-y", "-bsf:a", "aac_adtstoasc"], bsf=bsf_v) + ffmpeg(["-y", "-bsf:a", "aac_adtstoasc"], bsf=bsf_v) bsf_applied = bsf_v is not None elif bsf_v is not None: # Repack is mandatory, the VUI bitstream filter is best-effort: retry # without it so the remux still succeeds; caller falls back to normalize_vui. output_path.unlink(missing_ok=True) - _ffmpeg() + ffmpeg() else: raise diff --git a/unshackle/core/tracks/tracks.py b/unshackle/core/tracks/tracks.py index faa5ce26..2a09761f 100644 --- a/unshackle/core/tracks/tracks.py +++ b/unshackle/core/tracks/tracks.py @@ -523,6 +523,8 @@ class Tracks: lang_name = video_track.language.display_name() for video in self.videos: + if video.data.get("audio_language"): + continue video.needs_repack = True video.data["audio_language"] = lang_code video.data["audio_language_name"] = lang_name diff --git a/unshackle/core/tracks/video.py b/unshackle/core/tracks/video.py index 6d2aecd9..0c297d97 100644 --- a/unshackle/core/tracks/video.py +++ b/unshackle/core/tracks/video.py @@ -178,15 +178,15 @@ class Video(Track): # The codebase is currently agnostic to either, so a manual conversion to 6 is done. transfer = 6 - def _safe(enum_cls, value): + def safe(enum_cls, value): try: return enum_cls(value) except ValueError: return enum_cls(2) # Unspecified for unknown/private-use codes - primaries = _safe(Primaries, primaries) - transfer = _safe(Transfer, transfer) - matrix = _safe(Matrix, matrix) + primaries = safe(Primaries, primaries) + transfer = safe(Transfer, transfer) + matrix = safe(Matrix, matrix) # primaries and matrix does not strictly correlate to a range @@ -540,7 +540,7 @@ class Video(Track): out_path = Path(out_path) - def _run_ccextractor() -> bool: + def run_ccextractor() -> bool: cc_start = time.monotonic() try: subprocess.run( @@ -575,9 +575,9 @@ class Video(Track): # Try on the original file first (preserves container-level CC data like c608 boxes), # then fall back to repacked file (ccextractor can fail on some container formats). - if not _run_ccextractor(): + if not run_ccextractor(): self.repackage() - _run_ccextractor() + run_ccextractor() if out_path.exists(): cc_track = Subtitle( diff --git a/unshackle/core/update_checker.py b/unshackle/core/update_checker.py index 6d919083..d18a50ea 100644 --- a/unshackle/core/update_checker.py +++ b/unshackle/core/update_checker.py @@ -67,7 +67,7 @@ class UpdateChecker: return version_string.lstrip("v") @staticmethod - def _is_valid_version(version: str) -> bool: + def is_valid_version(version: str) -> bool: """ Validate version string format. @@ -93,7 +93,7 @@ class UpdateChecker: return False @classmethod - def _fetch_latest_version(cls) -> Optional[str]: + def fetch_latest_version(cls) -> Optional[str]: """ Fetch the latest version from GitHub API. @@ -109,13 +109,13 @@ class UpdateChecker: data = response.json() latest_version = cls.parse_version(data.get("tag_name", "")) - return latest_version if cls._is_valid_version(latest_version) else None + return latest_version if cls.is_valid_version(latest_version) else None except Exception: return None @classmethod - def _should_check_for_updates(cls, check_interval: int = DEFAULT_CHECK_INTERVAL) -> bool: + def should_check_for_updates(cls, check_interval: int = DEFAULT_CHECK_INTERVAL) -> bool: """ Check if enough time has passed since the last update check. @@ -136,7 +136,7 @@ class UpdateChecker: return (current_time - last_check) >= check_interval @classmethod - def _update_cache(cls, latest_version: Optional[str] = None, current_version: Optional[str] = None) -> None: + def update_cache(cls, latest_version: Optional[str] = None, current_version: Optional[str] = None) -> None: """ Update the cache file with the current timestamp and version info. @@ -158,11 +158,12 @@ class UpdateChecker: with open(cache_file, "w") as f: json.dump(cache_data, f, indent=2) - except (OSError, json.JSONEncodeError): + # json.dump signals encode failures as TypeError/ValueError, not a json-specific class + except (OSError, TypeError, ValueError): pass @staticmethod - def _compare_versions(current: str, latest: str) -> bool: + def compare_versions(current: str, latest: str) -> bool: """ Simple semantic version comparison. @@ -173,7 +174,7 @@ class UpdateChecker: Returns: True if latest > current, False otherwise """ - if not UpdateChecker._is_valid_version(current) or not UpdateChecker._is_valid_version(latest): + if not UpdateChecker.is_valid_version(current) or not UpdateChecker.is_valid_version(latest): return False try: @@ -205,13 +206,13 @@ class UpdateChecker: Returns: The latest version string if an update is available, None otherwise """ - if not cls._is_valid_version(current_version): + if not cls.is_valid_version(current_version): return None try: - latest_version = await asyncio.to_thread(cls._fetch_latest_version) + latest_version = await asyncio.to_thread(cls.fetch_latest_version) - if latest_version and cls._compare_versions(current_version, latest_version): + if latest_version and cls.compare_versions(current_version, latest_version): return latest_version except Exception: @@ -220,7 +221,7 @@ class UpdateChecker: return None @classmethod - def _get_cached_update_info(cls, current_version: str) -> Optional[str]: + def get_cached_update_info(cls, current_version: str) -> Optional[str]: """ Check if there's a cached update available for the current version. @@ -239,7 +240,7 @@ class UpdateChecker: cached_latest = cache_data.get("latest_version") if cached_current == current_version and cached_latest: - if cls._compare_versions(current_version, cached_latest): + if cls.compare_versions(current_version, cached_latest): return cached_latest return None @@ -256,7 +257,7 @@ class UpdateChecker: Returns: The latest version string if an update is available, None otherwise """ - if not cls._is_valid_version(current_version): + if not cls.is_valid_version(current_version): return None if check_interval is None: @@ -264,12 +265,12 @@ class UpdateChecker: check_interval = config.update_check_interval * 60 * 60 - if not cls._should_check_for_updates(check_interval): - return cls._get_cached_update_info(current_version) + if not cls.should_check_for_updates(check_interval): + return cls.get_cached_update_info(current_version) - latest_version = cls._fetch_latest_version() - cls._update_cache(latest_version, current_version) - if latest_version and cls._compare_versions(current_version, latest_version): + latest_version = cls.fetch_latest_version() + cls.update_cache(latest_version, current_version) + if latest_version and cls.compare_versions(current_version, latest_version): return latest_version return None diff --git a/unshackle/core/utilities.py b/unshackle/core/utilities.py index 285a0520..0870b8a7 100644 --- a/unshackle/core/utilities.py +++ b/unshackle/core/utilities.py @@ -164,6 +164,49 @@ def is_exact_match(language: Union[str, Language], languages: Sequence[Union[str return closest_match(language, list(map(str, languages)))[1] <= LANGUAGE_EXACT_DISTANCE +def partition_exclusions(tokens: Optional[Sequence[str]]) -> tuple[list[str], list[str]]: + """ + Split selection tokens into wanted values and values excluded with a leading '-'. + + Both sides keep the order they were written in and drop later repeats. A bare '-' + carries no value and is skipped. + + Example: + >>> partition_exclusions(["all", "-es"]) + (['all'], ['es']) + """ + includes: list[str] = [] + excludes: list[str] = [] + for raw in tokens or []: + token = str(raw).strip() + if not token or token == "-": + continue + target = includes + if token.startswith("-"): + token = token[1:].strip() + if not token: + continue + target = excludes + if token not in target: + target.append(token) + return includes, excludes + + +def excluded_language_tags( + excludes: Sequence[str], + available: Sequence[Union[str, Language, None]], + exact: bool = False, +) -> set[str]: + """ + The language tags out of `available` that the exclusion tokens remove. + + Matches through matching_languages so an exclusion drops exactly the tracks the same + token would select, including its exact-mode string preference. Untagged tracks are + never excluded. + """ + return {tag for token in excludes for tag in matching_languages(token, available, exact)} + + def keep_forced_subtitle( forced: bool, language: Union[str, Language], @@ -179,6 +222,19 @@ def keep_forced_subtitle( return match_func(language, forced_s_lang) +def embedded_audio_langs(videos: Sequence[Any], keep_videos: bool) -> list[str]: + """ + Return the audio languages carried inside video tracks rather than beside them. + + A muxed stream keeps its audio in the video track, which a service declares by setting + ``data["audio_language"]``. That audio is only available while the video is kept, so + dropping the video (``--audio-only``, ``--no-video``) drops the language with it. + """ + if not keep_videos: + return [] + return [video.data["audio_language"] for video in videos if video.data.get("audio_language")] + + def find_missing_langs( requested: Sequence[str], available: Sequence[Union[str, Language, None]], @@ -467,12 +523,12 @@ def try_ensure_utf8(data: bytes) -> bytes: if data[:2] == b"\x1f\x8b": try: data = gzip.decompress(data) - except Exception: + except (OSError, EOFError, zlib.error): pass elif data[:1] == b"\x78" and len(data) > 1 and data[1:2] in (b"\x01", b"\x5e", b"\x9c", b"\xda"): try: data = zlib.decompress(data) - except Exception: + except zlib.error: pass try: @@ -1109,12 +1165,12 @@ class DebugLogger: # Global debug logger instance -_debug_logger: Optional[DebugLogger] = None +debug_logger: Optional[DebugLogger] = None def get_debug_logger() -> Optional[DebugLogger]: """Get the global debug logger instance.""" - return _debug_logger + return debug_logger def log_event(operation: str, *, level: str = "DEBUG", message: str = "", **kwargs: Any) -> None: @@ -1124,7 +1180,7 @@ def log_event(operation: str, *, level: str = "DEBUG", message: str = "", **kwar ``if dl := get_debug_logger(): dl.log(...)`` guard boilerplate. To add logging to a new feature, call ``log_event("my_feature_event", message="...", context={...})``. """ - dl = _debug_logger + dl = debug_logger if dl: dl.log(level=level, operation=operation, message=message, **kwargs) @@ -1142,7 +1198,7 @@ def timed_operation(operation: str, *, level: str = "DEBUG", message: str = "", with timed_operation("mux", context={"output": str(path)}): run_mkvmerge(...) """ - dl = _debug_logger + dl = debug_logger if not dl: yield return @@ -1179,18 +1235,18 @@ def init_debug_logger(log_path: Optional[Path] = None, enabled: bool = False, lo enabled: Whether debug logging is enabled log_keys: Whether to log decryption keys (for debugging key issues) """ - global _debug_logger - if _debug_logger: - _debug_logger.close() - _debug_logger = DebugLogger(log_path=log_path, enabled=enabled, log_keys=log_keys) + global debug_logger + if debug_logger: + debug_logger.close() + debug_logger = DebugLogger(log_path=log_path, enabled=enabled, log_keys=log_keys) def close_debug_logger(): """Close the global debug logger.""" - global _debug_logger - if _debug_logger: - _debug_logger.close() - _debug_logger = None + global debug_logger + if debug_logger: + debug_logger.close() + debug_logger = None __all__ = ( diff --git a/unshackle/core/utils/animeapi.py b/unshackle/core/utils/animeapi.py deleted file mode 100644 index e16df7ff..00000000 --- a/unshackle/core/utils/animeapi.py +++ /dev/null @@ -1,92 +0,0 @@ -from __future__ import annotations - -import logging -from typing import Optional - -from unshackle.core.providers._base import ExternalIds - -log = logging.getLogger("ANIMEAPI") - -PLATFORM_MAP: dict[str, str] = { - "mal": "myanimelist", - "anilist": "anilist", - "kitsu": "kitsu", - "tmdb": "themoviedb", - "trakt": "trakt", - "tvdb": "thetvdb", -} - - -def resolve_animeapi(value: str) -> tuple[Optional[str], ExternalIds]: - """Resolve an anime database ID via AnimeAPI to a title and external IDs. - - Accepts formats like 'mal:12345', 'anilist:98765', or just '12345' (defaults to MAL). - Returns (anime_title, ExternalIds) with any TMDB/IMDB/TVDB IDs found. - """ - import animeapi - - platform_str, id_str = _parse_animeapi_value(value) - - platform_enum = _get_platform(platform_str) - if platform_enum is None: - log.warning("Unknown AnimeAPI platform: %s (supported: %s)", platform_str, ", ".join(PLATFORM_MAP)) - return None, ExternalIds() - - log.info("Resolving AnimeAPI %s:%s", platform_str, id_str) - - try: - with animeapi.AnimeAPI() as api: - relation = api.get_anime_relations(id_str, platform_enum) - except Exception as exc: - log.warning("AnimeAPI lookup failed for %s:%s: %s", platform_str, id_str, exc) - return None, ExternalIds() - - title = getattr(relation, "title", None) - - tmdb_id = getattr(relation, "themoviedb", None) - tmdb_type = getattr(relation, "themoviedb_type", None) - imdb_id = getattr(relation, "imdb", None) - tvdb_id = getattr(relation, "thetvdb", None) - - tmdb_kind: Optional[str] = None - if tmdb_type is not None: - tmdb_kind = tmdb_type.value if hasattr(tmdb_type, "value") else str(tmdb_type).lower() - if tmdb_kind not in ("movie", "tv"): - tmdb_kind = "tv" - - external_ids = ExternalIds( - tmdb_id=int(tmdb_id) if tmdb_id is not None else None, - tmdb_kind=tmdb_kind, - imdb_id=str(imdb_id) if imdb_id is not None else None, - tvdb_id=int(tvdb_id) if tvdb_id is not None else None, - ) - - log.info( - "AnimeAPI resolved: title=%r, tmdb=%s, imdb=%s, tvdb=%s", - title, - external_ids.tmdb_id, - external_ids.imdb_id, - external_ids.tvdb_id, - ) - - return title, external_ids - - -def _parse_animeapi_value(value: str) -> tuple[str, str]: - """Parse 'platform:id' format. Defaults to 'mal' if no prefix.""" - if ":" in value: - platform, _, id_str = value.partition(":") - return platform.lower().strip(), id_str.strip() - return "mal", value.strip() - - -def _get_platform(platform_str: str) -> object | None: - """Map a platform string to an animeapi.Platform enum value.""" - import animeapi - - canonical = PLATFORM_MAP.get(platform_str) - if canonical is None: - return None - - platform_name = canonical.upper() - return getattr(animeapi.Platform, platform_name, None) diff --git a/unshackle/core/utils/bitrate.py b/unshackle/core/utils/bitrate.py index 63ff44cb..013cfe89 100644 --- a/unshackle/core/utils/bitrate.py +++ b/unshackle/core/utils/bitrate.py @@ -261,7 +261,7 @@ def uniform_segments( """ Build Segments giving each an equal share of the total duration. - Used for DASH: ``DASH._get_period_segments`` returns timeline *start times* + Used for DASH: ``DASH.get_period_segments`` returns timeline *start times* rather than per-segment durations, so they cannot be trusted. Segment lengths are near-uniform in practice, so the track duration (from ``mediaPresentationDuration``) split evenly is both correct and timeline-safe. @@ -278,28 +278,26 @@ def extract_dash(track: "Track", session: Union[Session, RnetSession]) -> list[S data = track.data["dash"] manifest = data["manifest"] - rep_id = data.get("representation_id") or data["representation"].get("id") + adaptation_set = data["adaptation_set"] + representation = data["representation"] + rep_id = data.get("representation_id") or representation.get("id") filtered_period_ids = data.get("filtered_period_ids", []) + stored_period = data.get("period") track_url = track.url if isinstance(track.url, str) else track.url[0] - content_periods = [p for p in manifest.findall("Period") if DASH._is_content_period(p, filtered_period_ids)] + content_periods = [p for p in manifest.findall("Period") if DASH.is_content_period(p, filtered_period_ids)] raw_segments: list[tuple[str, Optional[str]]] = [] for period in content_periods: - matched_rep = matched_as = None - for as_ in period.findall("AdaptationSet"): - if DASH.is_trick_mode(as_): - continue - for rep in as_.findall("Representation"): - if rep.get("id") == rep_id: - matched_rep, matched_as = rep, as_ - break - if matched_rep is not None: - break + # rep ids are only unique within an AdaptationSet, so resolve like DASH.download_track + if period is stored_period: + matched_as, matched_rep = adaptation_set, representation + else: + matched_as, matched_rep = DASH.resolve_representation(period, rep_id, adaptation_set) or (None, None) if matched_rep is None or matched_as is None: continue - _, period_segments, _, _, _ = DASH._get_period_segments( + _, period_segments, _, _, _ = DASH.get_period_segments( period=period, adaptation_set=matched_as, representation=matched_rep, diff --git a/unshackle/core/utils/click_types.py b/unshackle/core/utils/click_types.py index 9f0f4334..2d846ddf 100644 --- a/unshackle/core/utils/click_types.py +++ b/unshackle/core/utils/click_types.py @@ -1,4 +1,5 @@ import re +from datetime import date, timedelta from typing import Any, Optional, Union import click @@ -176,6 +177,23 @@ class SeasonRange(click.ParamType): MAX_EPISODE = 999 MIN_PART = 1 MAX_PART = 99 + MAX_DATE_SPAN = 1000 + + DATE_TOKEN = re.compile(r"^(?P\d{4}-\d{2}-\d{2})(:(?P\d{4}-\d{2}-\d{2}))?$") + + def parse_date_token(self, token: str, match: re.Match) -> list[str]: + """Expand an ISO date token or ':'-separated date range into ISO day keys.""" + try: + left = date.fromisoformat(match.group("left")) + right = date.fromisoformat(match.group("right")) if match.group("right") else left + except ValueError: + self.fail(f"Invalid date, must be a real YYYY-MM-DD date: {token}") + if left > right: + self.fail(f"Invalid range, left side date cannot be later than right side date: {token}") + span = (right - left).days + 1 + if span > self.MAX_DATE_SPAN: + self.fail(f"Invalid range, a date range cannot span more than {self.MAX_DATE_SPAN} days: {token}") + return [(left + timedelta(days=i)).isoformat() for i in range(span)] def parse_tokens(self, *tokens: str) -> list[str]: """ @@ -186,6 +204,9 @@ class SeasonRange(click.ParamType): is not knowable here. A part-qualified exclusion cannot be removed from the computed keys (they are base keys), so it becomes a '!' key resolved at match time. + Dated content is addressed by ISO air date. A date range uses ':' only, because + a date's own '-' separators are not a range separator. + Supports exclusioning by putting a `-` before the token. Example: @@ -200,6 +221,10 @@ class SeasonRange(click.ParamType): ["1x1.1", "1x1.2", "1x1.3"] >>> sr.parse_tokens("S01E01", "-S01E01.2") ["1x1", "!1x1.2"] + >>> sr.parse_tokens("2026-08-11") + ["2026-08-11"] + >>> sr.parse_tokens("2026-08-01:2026-08-03", "-2026-08-02") + ["2026-08-01", "2026-08-03"] """ if len(tokens) == 0: return [] @@ -209,6 +234,11 @@ class SeasonRange(click.ParamType): exclude = token.startswith("-") if exclude: token = token[1:] + # dates carry their own '-' separators, so they must be read before the range split + date_match = self.DATE_TOKEN.match(token) + if date_match: + (computed if not exclude else exclusions).extend(self.parse_date_token(token, date_match)) + continue parsed = [ re.match(r"^S(?P\d+)(E(?P\d+)(\.(?P\d+))?)?$", x, re.IGNORECASE) for x in re.split(r"[:-]", token) diff --git a/unshackle/core/utils/dovi.py b/unshackle/core/utils/dovi.py index 805120f7..d9f31738 100644 --- a/unshackle/core/utils/dovi.py +++ b/unshackle/core/utils/dovi.py @@ -21,7 +21,7 @@ from unshackle.core import binaries from unshackle.core.utils.subprocess import log_tool_run, run_step -def _require_dovi_tool() -> str: +def require_dovi_tool() -> str: if not binaries.DoviTool: raise EnvironmentError("dovi_tool executable was not found but is required.") return str(binaries.DoviTool) @@ -36,7 +36,7 @@ def extract_rpu( label: str = "dovi_tool extract-rpu", ) -> bytes: """Extract DV RPU NALs from a raw HEVC stream. `mode=None` skips the -m flag (untouched).""" - tool = _require_dovi_tool() + tool = require_dovi_tool() args: list = [tool] if mode is not None: args += ["-m", str(mode)] @@ -53,7 +53,7 @@ def inject_rpu( label: str = "dovi_tool inject-rpu", ) -> bytes: """Inject a DV RPU back into a raw HEVC stream, producing DV-signaled output.""" - tool = _require_dovi_tool() + tool = require_dovi_tool() return run_step( [tool, "inject-rpu", "-i", source, "--rpu-in", rpu, "-o", output], status=status, @@ -71,7 +71,7 @@ def editor( label: str = "dovi_tool editor", ) -> bytes: """Apply a JSON edit spec to an RPU file.""" - tool = _require_dovi_tool() + tool = require_dovi_tool() return run_step( [tool, "editor", "-i", source, "-j", json_spec, "-o", output], status=status, @@ -82,7 +82,7 @@ def editor( def info_summary(rpu: Path) -> str: """Return the textual summary (`dovi_tool info -i ... -s`) for an RPU file.""" - tool = _require_dovi_tool() + tool = require_dovi_tool() info_start = time.monotonic() p = subprocess.run( [tool, "info", "-i", str(rpu), "-s"], capture_output=True, text=True, encoding="utf-8", errors="replace" @@ -107,7 +107,7 @@ def generate_from_hdr10plus( label: str = "dovi_tool generate", ) -> bytes: """Build a DV RPU from extracted HDR10+ metadata + an extra JSON descriptor.""" - tool = _require_dovi_tool() + tool = require_dovi_tool() return run_step( [tool, "generate", "-j", extra_json, "--hdr10plus-json", hdr10plus_json, "-o", output], status=status, diff --git a/unshackle/core/utils/language_tags.py b/unshackle/core/utils/language_tags.py index abb62ce8..a8b0ccd1 100644 --- a/unshackle/core/utils/language_tags.py +++ b/unshackle/core/utils/language_tags.py @@ -36,14 +36,14 @@ def evaluate_language_tag( log.warning("Language tag rule missing 'tag' field, skipping: %s", rule) continue - if _rule_matches(rule, audio_languages, subtitle_languages): + if rule_matches(rule, audio_languages, subtitle_languages): log.debug("Language tag rule matched: %s -> %s", rule, tag) return str(tag) return "" -def _rule_matches( +def rule_matches( rule: dict[str, Any], audio_languages: Sequence[Language], subtitle_languages: Sequence[Language], diff --git a/unshackle/core/utils/redact.py b/unshackle/core/utils/redact.py index e4c1f5b8..3350e1bb 100644 --- a/unshackle/core/utils/redact.py +++ b/unshackle/core/utils/redact.py @@ -41,7 +41,7 @@ def safe_display_url(url: str) -> str: return urlunsplit((parts.scheme, netloc, parts.path, "", "")) -def _path_bases() -> list[tuple[str, str]]: +def path_bases() -> list[tuple[str, str]]: """Local base directories to strip from logged paths, longest-first. Anchors everything left of the unshackle install/root (and the venv / home dir) so an @@ -52,14 +52,14 @@ def _path_bases() -> list[tuple[str, str]]: try: # redact.py lives at /unshackle/core/utils/redact.py -> parents[3] == project/install root candidates.append((str(Path(__file__).resolve().parents[3]), "")) - except Exception: + except (OSError, IndexError): pass for raw in (getattr(sys, "prefix", ""), getattr(sys, "base_prefix", "")): if raw: candidates.append((str(Path(raw)), "")) try: candidates.append((str(Path.home()), "~")) - except Exception: + except (OSError, RuntimeError): pass bases: list[tuple[str, str]] = [] @@ -74,10 +74,10 @@ def _path_bases() -> list[tuple[str, str]]: return bases -_PATH_BASES = _path_bases() +PATH_BASES = path_bases() -def _path_redaction_enabled() -> bool: +def path_redaction_enabled() -> bool: """Path-prefix redaction is on unless ``config.redact_paths`` is explicitly False.""" try: from unshackle.core.config import config # lazy: config doesn't import redact, so no cycle @@ -93,9 +93,9 @@ def redact_path(text: Optional[str]) -> Optional[str]: Idempotent and cheap; only touches strings that actually contain a known base dir, so URLs and relative paths pass through unchanged. Disabled by ``config.redact_paths: false``. """ - if not isinstance(text, str) or not text or not _path_redaction_enabled(): + if not isinstance(text, str) or not text or not path_redaction_enabled(): return text - for base, token in _PATH_BASES: + for base, token in PATH_BASES: if base in text: text = text.replace(base, token) return text @@ -104,16 +104,16 @@ def redact_path(text: Optional[str]) -> Optional[str]: # any http(s) URL embedded in free text (content/manifest/segment/api locations) URL_RE = re.compile(r"https?://[^\s\"'<>\\]+", re.IGNORECASE) # a plausible file extension to preserve (e.g. .mpd, .m3u8, .mp4, .m4s, .vtt) -_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,5}$") +EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,5}$") -def _collapse_url(match: "re.Match[str]") -> str: +def collapse_url(match: "re.Match[str]") -> str: url = match.group(0) try: suffix = Path(urlsplit(url).path).suffix except Exception: suffix = "" - if not _EXT_RE.match(suffix): + if not EXT_RE.match(suffix): suffix = "" return f"redacted{suffix}" @@ -127,7 +127,7 @@ def redact_url(text: Optional[str]) -> Optional[str]: """ if not isinstance(text, str) or not text: return text - return URL_RE.sub(_collapse_url, text) + return URL_RE.sub(collapse_url, text) def redact_all(text: Optional[str]) -> Optional[str]: diff --git a/unshackle/core/utils/tag_rules.py b/unshackle/core/utils/tag_rules.py index 7e902b1f..53636087 100644 --- a/unshackle/core/utils/tag_rules.py +++ b/unshackle/core/utils/tag_rules.py @@ -10,7 +10,7 @@ from typing import Any, Callable, Optional log = logging.getLogger(__name__) -_OPS: dict[str, Callable[[Any, Any], bool]] = { +OPS: dict[str, Callable[[Any, Any], bool]] = { ">=": operator.ge, "<=": operator.le, "!=": operator.ne, @@ -19,8 +19,8 @@ _OPS: dict[str, Callable[[Any, Any], bool]] = { "<": operator.lt, "=": operator.eq, } -_ORDERING = (">=", "<=", ">", "<") -_NUMBER = re.compile(r"\d+(?:\.\d+)?") +ORDERING = (">=", "<=", ">", "<") +NUMBER = re.compile(r"\d+(?:\.\d+)?") def evaluate_tag_rules(rules: Any, context: dict[str, Any]) -> Optional[str]: @@ -56,7 +56,7 @@ def evaluate_tag_rules(rules: Any, context: dict[str, Any]) -> Optional[str]: if not when or not tag: log.warning("Tag rule needs both 'when' conditions and a 'tag', skipping: %s", rule) continue - if isinstance(tag, bool) or any(_has_bool(value) for value in when.values()): + if isinstance(tag, bool) or any(has_bool(value) for value in when.values()): log.warning("Tag rule has a YAML boolean, quote the value to use it as text, skipping: %s", rule) continue if not isinstance(tag, (str, int, float)): @@ -68,35 +68,35 @@ def evaluate_tag_rules(rules: Any, context: dict[str, Any]) -> Optional[str]: log.warning("Tag rule has unknown 'when' key(s) %s, skipping: %s", ", ".join(sorted(unknown)), rule) continue - if all(_matches(expected, context[key]) for key, expected in when.items()): + if all(matches(expected, context[key]) for key, expected in when.items()): log.debug("Tag rule matched: %s -> %s", rule, tag) return str(tag) return None -def _has_bool(expected: Any) -> bool: +def has_bool(expected: Any) -> bool: """Check for a YAML boolean, which is an unquoted word the user meant as text.""" values = expected if isinstance(expected, list) else [expected] return any(isinstance(x, bool) for x in values) -def _matches(expected: Any, actual: Any) -> bool: +def matches(expected: Any, actual: Any) -> bool: """Check one condition, where a list matches on any entry.""" values = expected if isinstance(expected, list) else [expected] - return any(_matches_one(x, actual) for x in values) + return any(matches_one(x, actual) for x in values) -def _matches_one(expected: Any, actual: Any) -> bool: +def matches_one(expected: Any, actual: Any) -> bool: """Check a single value, which may carry a leading comparison operator.""" text = "" if expected is None else str(expected) - for op in sorted(_OPS, key=len, reverse=True): + for op in sorted(OPS, key=len, reverse=True): if text.startswith(op): - return _compare(op, text[len(op) :].strip(), actual) + return compare(op, text[len(op) :].strip(), actual) return str(actual or "").lower() == text.lower() -def _compare(op: str, operand: str, actual: Any) -> bool: +def compare(op: str, operand: str, actual: Any) -> bool: """Compare against an operand, numerically when the operand is a number.""" if not operand: log.warning("Tag rule condition '%s' has no value to compare against, condition failed", op) @@ -105,12 +105,12 @@ def _compare(op: str, operand: str, actual: Any) -> bool: try: wanted = float(operand) except ValueError: - if op in _ORDERING: + if op in ORDERING: log.warning("Tag rule condition '%s%s' needs a numeric operand, condition failed", op, operand) return False - return _OPS[op](str(actual or "").lower(), operand.lower()) + return OPS[op](str(actual or "").lower(), operand.lower()) - found = _NUMBER.search(str(actual or "")) + found = NUMBER.search(str(actual or "")) if not found: return op == "!=" - return _OPS[op](float(found.group()), wanted) + return OPS[op](float(found.group()), wanted) diff --git a/unshackle/core/utils/tags.py b/unshackle/core/utils/tags.py index 1b92e2d5..79b206a1 100644 --- a/unshackle/core/utils/tags.py +++ b/unshackle/core/utils/tags.py @@ -5,20 +5,16 @@ import subprocess import tempfile import time from pathlib import Path -from typing import Optional +from typing import Optional, Union from xml.sax.saxutils import escape from unshackle.core import binaries from unshackle.core.config import config from unshackle.core.providers import ( ExternalIds, - MetadataResult, - enrich_ids, - fetch_external_ids, fuzzy_match, get_available_providers, - get_provider, - search_metadata, + resolve_by_ids, ) from unshackle.core.titles.episode import Episode from unshackle.core.titles.movie import Movie @@ -68,7 +64,7 @@ def apply_tags(path: Path, tags: dict[str, str]) -> None: tmp_path.unlink(missing_ok=True) -def _build_tags_from_ids(ids: ExternalIds, kind: str) -> dict[str, str]: +def build_tags_from_ids(ids: ExternalIds, kind: str) -> dict[str, str]: """Build standard MKV tags from external IDs.""" tags: dict[str, str] = {} if ids.imdb_id: @@ -78,6 +74,8 @@ def _build_tags_from_ids(ids: ExternalIds, kind: str) -> dict[str, str]: if ids.tvdb_id: prefix = "movies" if kind == "movie" else "series" tags["TVDB2"] = f"{prefix}/{ids.tvdb_id}" + if ids.anilist_id: + tags["ANILIST"] = str(ids.anilist_id) return tags @@ -87,6 +85,8 @@ def tag_file( tmdb_id: Optional[int] = None, imdb_id: Optional[str] = None, tvdb_id: Optional[int] = None, + anilist_id: Optional[Union[int, str]] = None, + anime: bool = False, ) -> None: log.debug("Tagging file %s with title %r", path, title) custom_tags: dict[str, str] = {} @@ -118,56 +118,18 @@ def tag_file( if config.tag_imdb_tmdb: try: - providers = get_available_providers() - if not providers and tvdb_id is None: + has_ids = any(value is not None for value in (tmdb_id, imdb_id, tvdb_id, anilist_id)) + if not get_available_providers() and not has_ids: log.debug("No metadata providers available; skipping tag lookup") apply_tags(path, custom_tags) return - result: Optional[MetadataResult] = None - - # Direct ID lookup path - if imdb_id: - imdbapi = get_provider("imdbapi") - if imdbapi: - result = imdbapi.get_by_id(imdb_id, kind) - if not result: - # IMDxAPI is off or had no answer; the ID alone still resolves the others - result = MetadataResult(title=name, year=year, kind=kind) - result.external_ids.imdb_id = imdb_id - enrich_ids(result) - elif tmdb_id is not None: - tmdb = get_provider("tmdb") - if tmdb: - result = tmdb.get_by_id(tmdb_id, kind) - if result: - ext = tmdb.get_external_ids(tmdb_id, kind) - result.external_ids = ext - enrich_ids(result) - else: - # Search across providers in priority order - result = search_metadata(name, year, kind) - - # If we got a TMDB ID from search but not the other IDs, fetch them - if ( - result - and result.external_ids.tmdb_id - and not (result.external_ids.imdb_id and result.external_ids.tvdb_id) - ): - ext = fetch_external_ids(result.external_ids.tmdb_id, kind) - if ext.imdb_id and not result.external_ids.imdb_id: - result.external_ids.imdb_id = ext.imdb_id - if ext.tvdb_id and not result.external_ids.tvdb_id: - result.external_ids.tvdb_id = ext.tvdb_id - enrich_ids(result) - - if tvdb_id is not None: - if result is None: - result = MetadataResult(kind=kind) - result.external_ids.tvdb_id = tvdb_id + result = resolve_by_ids( + tmdb_id, imdb_id, tvdb_id, anilist_id, title=name, year=year, kind=kind, anime=anime + ) if result and result.external_ids: - standard_tags = _build_tags_from_ids(result.external_ids, kind) + standard_tags = build_tags_from_ids(result.external_ids, kind) except Exception as e: log.warning("Metadata lookup failed, applying custom tags only: %s", e) diff --git a/unshackle/core/utils/template_formatter.py b/unshackle/core/utils/template_formatter.py index 47c777a6..a7c9be34 100644 --- a/unshackle/core/utils/template_formatter.py +++ b/unshackle/core/utils/template_formatter.py @@ -38,9 +38,9 @@ class TemplateFormatter: """ self.template = template self.spacer = spacer or detect_spacer(template) - self.variables = self._extract_variables() + self.variables = self.extract_variables() - def _extract_variables(self) -> list[str]: + def extract_variables(self) -> list[str]: """Extract all variables from the template.""" pattern = r"\{([^}]+)\}" matches = re.findall(pattern, self.template) @@ -82,7 +82,7 @@ class TemplateFormatter: # Remove the placeholder and consume the adjacent separator on one side # e.g. "{disc?}-{track}" → "{track}" when disc is empty # e.g. "{title}.{edition?}.{quality}" → "{title}.{quality}" when edition is empty - def _remove_conditional(m: re.Match) -> str: + def remove_conditional(m: re.Match) -> str: s = m.group(0) has_left = s[0] in ".- " has_right = s[-1] in ".- " @@ -94,7 +94,7 @@ class TemplateFormatter: result = re.sub( rf"[\.\s\-]?{re.escape(placeholder)}[\.\s\-]?", - _remove_conditional, + remove_conditional, result, count=1, ) diff --git a/unshackle/core/utils/webvtt.py b/unshackle/core/utils/webvtt.py index 09ff17a6..5e84acc3 100644 --- a/unshackle/core/utils/webvtt.py +++ b/unshackle/core/utils/webvtt.py @@ -2,10 +2,10 @@ import re from typing import Optional # Timing line: optional hours, verbatim settings tail. -_TIMING = re.compile(r"^((?:\d+:)?\d{2}:\d{2}\.\d{3})[ \t]+-->[ \t]+((?:\d+:)?\d{2}:\d{2}\.\d{3})(.*)$") +TIMING = re.compile(r"^((?:\d+:)?\d{2}:\d{2}\.\d{3})[ \t]+-->[ \t]+((?:\d+:)?\d{2}:\d{2}\.\d{3})(.*)$") -def _timestamp_ms(ts: str) -> int: +def timestamp_ms(ts: str) -> int: hms, ms = ts.rsplit(".", 1) parts = [int(p) for p in hms.split(":")] while len(parts) < 3: @@ -52,7 +52,7 @@ def merge_segmented_webvtt(vtt_raw: str, segment_durations: Optional[list[int]] if header not in headers: # segments repeat the same block headers.append(header) continue - timing = _TIMING.match(lines[timing_i].strip()) + timing = TIMING.match(lines[timing_i].strip()) if not timing: continue start, end, settings = timing.groups() @@ -61,15 +61,15 @@ def merge_segmented_webvtt(vtt_raw: str, segment_durations: Optional[list[int]] continue normalized = "\n".join(line.strip() for line in payload) - if prev is not None and _timestamp_ms(start) - prev_end_ms <= 1 and normalized == prev[4]: - end_ms = _timestamp_ms(end) + if prev is not None and timestamp_ms(start) - prev_end_ms <= 1 and normalized == prev[4]: + end_ms = timestamp_ms(end) if end_ms > prev_end_ms: # splice: extend only, never shorten the kept cue prev[1] = end prev_end_ms = end_ms continue prev = [start, end, settings, "\n".join(payload), normalized] - prev_end_ms = _timestamp_ms(end) + prev_end_ms = timestamp_ms(end) cues.append(prev) blocks = headers + [f"{c[0]} --> {c[1]}{c[2]}\n{c[3]}" for c in cues] diff --git a/unshackle/core/vaults.py b/unshackle/core/vaults.py index 538eec3e..936bd29a 100644 --- a/unshackle/core/vaults.py +++ b/unshackle/core/vaults.py @@ -10,11 +10,11 @@ from unshackle.core.vault import Vault log = logging.getLogger(__name__) -_VAULTS = sorted( +VAULTS = sorted( (path for path in config.directories.vaults.glob("*.py") if path.stem.lower() != "__init__"), key=lambda x: x.stem ) -_MODULES = {path.stem: getattr(import_module_by_path(path), path.stem) for path in _VAULTS} +MODULES = {path.stem: getattr(import_module_by_path(path), path.stem) for path in VAULTS} class Vaults: @@ -32,7 +32,7 @@ class Vaults: def load(self, type_: str, **kwargs: Any) -> bool: """Load a Vault into the vaults list. Returns True if successful, False otherwise.""" - module = _MODULES.get(type_) + module = MODULES.get(type_) if not module: raise ValueError(f"Unable to find vault command by the name '{type_}'.") # only pass the global default to vaults that declare a timeout param (per-vault config still wins) @@ -42,12 +42,14 @@ class Vaults: vault = module(**kwargs) self.vaults.append(vault) return True - except Exception: + # one bad vault config must not stop the others from loading + except Exception as e: + log.debug(f"Vault '{type_}' failed to load: {e!r}") return False def load_critical(self, type_: str, **kwargs: Any) -> None: """Load a critical Vault that must succeed or raise an exception.""" - module = _MODULES.get(type_) + module = MODULES.get(type_) if not module: raise ValueError(f"Unable to find vault command by the name '{type_}'.") vault = module(**kwargs) diff --git a/unshackle/services/EXAMPLE/__init__.py b/unshackle/services/EXAMPLE/__init__.py index 703608f3..e283c5e4 100644 --- a/unshackle/services/EXAMPLE/__init__.py +++ b/unshackle/services/EXAMPLE/__init__.py @@ -55,7 +55,7 @@ class EXAMPLE(Service): search SearchResult generator get_titles Movies / Series / Album (music) + data passthrough get_tracks DASH variant fan-out (default) + HLS/ISM alternates - _fetch_dash_manifest range override, HDR10+ flip, DV-composite, Atmos, + fetch_dash_manifest range override, HDR10+ flip, DV-composite, Atmos, descriptive audio, channel fixups, cover-art attachment, VUI normalize, bitrate-window awareness get_chapters Chapters() with named + unnamed markers @@ -72,6 +72,9 @@ class EXAMPLE(Service): TITLE_RE = r"^(?:https?://(?:www\.)?domain\.com/details/)?(?P[^/?#]+)" # NO_SUBTITLES: service-level idiom telling the pipeline subs are handled in-band. NO_SUBTITLES = False + # ANIME: this catalogue is anime, so metadata lookups ask AniList first. Set it False (the + # default) on a mixed catalogue and flag the anime titles individually in get_titles(). + ANIME = True # VAULT_TAG: store/read keys under a different vault namespace than this service's tag. # Lets sibling services share one key vault. Omit to use the service's own tag. VAULT_TAG = "DIFFERENT_NAME" @@ -285,19 +288,24 @@ class EXAMPLE(Service): data=ep, # daily/sports: name by date instead of SxxExx (date or ISO string) # air_date=ep.get("airDate"), + # feeds the {absolute} naming token, --enrich fills it in from TVDB when unset + absolute=ep.get("absoluteNumber"), ) ) + # Per-title override of the ANIME class attr, for a mixed catalogue. Leave it + # None (the default) and the title inherits the service's ANIME value. + episodes[-1].anime = metadata.get("category") == "anime" return Series(episodes) # DEFAULT (shown live): this service needs a SEPARATE manifest per codec/range, - # so we fan out with the base helper `_get_tracks_for_variants`. It walks every + # so we fan out with the base helper `get_tracks_for_variants`. It walks every # codec x range in the TrackRequest, handles HYBRID (HDR10 + DV merge), and - # when --best-available is set - skips combos the service can't deliver. def get_tracks(self, title: Title_T) -> Tracks: - def _fetch_variant(title: Title_T, codec: Optional[Video.Codec], range_: Video.Range) -> Tracks: + def fetch_variant(title: Title_T, codec: Optional[Video.Codec], range_: Video.Range) -> Tracks: vcodec_str = "H265" if codec == Video.Codec.HEVC else "H264" self.log.info(f" + Fetching {vcodec_str} {range_.name} manifest") - tracks = self._fetch_dash_manifest(title, vcodec=vcodec_str, range_=range_) + tracks = self.fetch_dash_manifest(title, vcodec=vcodec_str, range_=range_) # Guard: if we asked for HDR/DV but the manifest came back SDR-only, raise # so the helper can fall back (best-available) or fail loudly otherwise. @@ -306,7 +314,7 @@ class EXAMPLE(Service): raise ValueError(f"{range_.name} requested but unavailable") return tracks - return self._get_tracks_for_variants(title, _fetch_variant) + return self.get_tracks_for_variants(title, fetch_variant) # ── ALTERNATE A - HLS (one master playlist returns every codec/range) ─────── # When a service exposes a single master playlist, you do NOT need the variant @@ -324,7 +332,7 @@ class EXAMPLE(Service): # from unshackle.core.manifests import ISM # return ISM.from_url(url=ism_url, session=self.session).to_tracks(title.language) - def _fetch_dash_manifest( + def fetch_dash_manifest( self, title: Title_T, vcodec: str = "H264", range_: Video.Range = Video.Range.SDR ) -> Tracks: video_format = self.VIDEO_RANGE_MAP.get(range_, "sdr") diff --git a/unshackle/utils/base58.py b/unshackle/utils/base58.py index eae39fb0..e63f665b 100644 --- a/unshackle/utils/base58.py +++ b/unshackle/utils/base58.py @@ -64,7 +64,7 @@ def b58encode(v: Union[str, bytes], alphabet: bytes = BITCOIN_ALPHABET) -> bytes @lru_cache() -def _get_base58_decode_map(alphabet: bytes, autofix: bool) -> Mapping[int, int]: +def get_base58_decode_map(alphabet: bytes, autofix: bool) -> Mapping[int, int]: invmap = {char: index for index, char in enumerate(alphabet)} if autofix: @@ -86,7 +86,7 @@ def b58decode_int(v: Union[str, bytes], alphabet: bytes = BITCOIN_ALPHABET, *, a v = v.rstrip() v = scrub_input(v) - map = _get_base58_decode_map(alphabet, autofix=autofix) + map = get_base58_decode_map(alphabet, autofix=autofix) decimal = 0 base = len(alphabet) diff --git a/unshackle/utils/base62.py b/unshackle/utils/base62.py index 433b9c14..e74ddf3a 100644 --- a/unshackle/utils/base62.py +++ b/unshackle/utils/base62.py @@ -38,7 +38,7 @@ def encodebytes(barray, charset=CHARSET_DEFAULT): :rtype: str """ - _check_type(barray, bytes) + check_type(barray, bytes) # Count the number of leading zeros. leading_zeros_count = 0 @@ -68,11 +68,11 @@ def decode(encoded, charset=CHARSET_DEFAULT): :type encoded: str :rtype: int """ - _check_type(encoded, str) + check_type(encoded, str) length, i, v = len(encoded), 0, 0 for x in encoded: - v += _value(x, charset=charset) * (BASE ** (length - (i + 1))) + v += value(x, charset=charset) * (BASE ** (length - (i + 1))) i += 1 return v @@ -88,7 +88,7 @@ def decodebytes(encoded, charset=CHARSET_DEFAULT): leading_null_bytes = b"" while encoded.startswith("0") and len(encoded) >= 2: - leading_null_bytes += b"\x00" * _value(encoded[1], charset) + leading_null_bytes += b"\x00" * value(encoded[1], charset) encoded = encoded[2:] decoded = decode(encoded, charset=charset) @@ -101,7 +101,7 @@ def decodebytes(encoded, charset=CHARSET_DEFAULT): return leading_null_bytes + bytes(buf) -def _value(ch, charset): +def value(ch, charset): """Decodes an individual digit of a base62 encoded string.""" try: @@ -110,7 +110,7 @@ def _value(ch, charset): raise ValueError("base62: Invalid character (%s)" % ch) -def _check_type(value, expected_type): +def check_type(value, expected_type): """Checks if the input is in an appropriate type.""" if not isinstance(value, expected_type): diff --git a/unshackle/vaults/MySQL.py b/unshackle/vaults/MySQL.py index 8f6ef185..7dca4749 100644 --- a/unshackle/vaults/MySQL.py +++ b/unshackle/vaults/MySQL.py @@ -253,10 +253,10 @@ class ConnectionFactory: self._con = con self._store = threading.local() - def _create_connection(self) -> pymysql.Connection: + def create_connection(self) -> pymysql.Connection: return pymysql.connect(**self._con) def get(self) -> pymysql.Connection: if not hasattr(self._store, "conn"): - self._store.conn = self._create_connection() + self._store.conn = self.create_connection() return self._store.conn diff --git a/unshackle/vaults/SQLite.py b/unshackle/vaults/SQLite.py index 25362962..0407bcc8 100644 --- a/unshackle/vaults/SQLite.py +++ b/unshackle/vaults/SQLite.py @@ -207,7 +207,7 @@ class ConnectionFactory: self._path = path self._store = threading.local() - def _create_connection(self) -> Connection: + def create_connection(self) -> Connection: conn = sqlite3.connect(self._path, timeout=30.0) conn.execute("PRAGMA journal_mode=WAL") conn.execute("PRAGMA synchronous=NORMAL") @@ -216,5 +216,5 @@ class ConnectionFactory: def get(self) -> Connection: if not hasattr(self._store, "conn"): - self._store.conn = self._create_connection() + self._store.conn = self.create_connection() return self._store.conn diff --git a/uv.lock b/uv.lock index d7674e55..619998b1 100644 --- a/uv.lock +++ b/uv.lock @@ -158,20 +158,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, ] -[[package]] -name = "animeapi-py" -version = "3.8.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "aiohttp" }, - { name = "dacite" }, - { name = "requests" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/28/a2/f868bba3b6399e2c953f8796fa90c8de91f797ad0fe19c7e2c536cd0092f/animeapi_py-3.8.1.tar.gz", hash = "sha256:9ed10f8207f1ccd7949ee3728285f4aced9a5ec57f71d211594ca5dc356fdea5", size = 25663, upload-time = "2026-02-25T15:29:17.879Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ba/98/6775d71cf7d88d554e8394f5ce5cda90041c99fdf1b2b60af02001e8c790/animeapi_py-3.8.1-py3-none-any.whl", hash = "sha256:c29f6e633d17bb613f459aa6514c0baab7ae325881f8a109eb6e4b3be5c22827", size = 26983, upload-time = "2026-02-25T15:29:16.685Z" }, -] - [[package]] name = "appdirs" version = "1.4.4" @@ -541,15 +527,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a7/ec/bb273b7208c606890dc36540fe667d06ce840a6f62f9fae7e658fcdc90fb/cssutils-2.11.1-py3-none-any.whl", hash = "sha256:a67bfdfdff4f3867fab43698ec4897c1a828eca5973f4073321b3bccaf1199b1", size = 385747, upload-time = "2024-06-04T15:51:37.499Z" }, ] -[[package]] -name = "dacite" -version = "1.9.2" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/55/a0/7ca79796e799a3e782045d29bf052b5cde7439a2bbb17f15ff44f7aacc63/dacite-1.9.2.tar.gz", hash = "sha256:6ccc3b299727c7aa17582f0021f6ae14d5de47c7227932c47fec4cdfefd26f09", size = 22420, upload-time = "2025-02-05T09:27:29.757Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/94/35/386550fd60316d1e37eccdda609b074113298f23cef5bddb2049823fe666/dacite-1.9.2-py3-none-any.whl", hash = "sha256:053f7c3f5128ca2e9aceb66892b1a3c8936d02c686e707bee96e19deef4bc4a0", size = 16600, upload-time = "2025-02-05T09:27:24.345Z" }, -] - [[package]] name = "defusedxml" version = "0.7.1" @@ -1851,6 +1828,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/14/25/b208c5683343959b670dc001595f2f3737e051da617f66c31f7c4fa93abc/rich-14.3.3-py3-none-any.whl", hash = "sha256:793431c1f8619afa7d3b52b2cdec859562b950ea0d4b6b505397612db8d5362d", size = 310458, upload-time = "2026-02-19T17:23:13.732Z" }, ] +[[package]] +name = "rich-click" +version = "1.9.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "rich" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f7/ea/21e4867ea0ef881ffd4c0550fc21a061435e50d6324bcd034396633cbc18/rich_click-1.9.8.tar.gz", hash = "sha256:4008f921da88b5d91646c134ec881c1500e5a6b3f093e90e8f29400e09608371", size = 75363, upload-time = "2026-05-28T19:54:59.144Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/97/a87901aef6b7e7e4a34c6dd6cc17dca8594a592ef9d9dd765fca2b7facf7/rich_click-1.9.8-py3-none-any.whl", hash = "sha256:12873865396e6927835d4eabb1cc3996edcd65b7ac9b2391a29eca4f335a2f93", size = 72189, upload-time = "2026-05-28T19:54:57.867Z" }, +] + [[package]] name = "rlaphoenix-m3u8" version = "3.4.0" @@ -2143,12 +2134,11 @@ wheels = [ [[package]] name = "unshackle" -version = "5.3.0" +version = "5.5.0" source = { editable = "." } dependencies = [ { name = "aiohttp" }, { name = "aiohttp-swagger3" }, - { name = "animeapi-py" }, { name = "appdirs" }, { name = "bandit" }, { name = "brotli" }, @@ -2181,6 +2171,7 @@ dependencies = [ { name = "pyyaml" }, { name = "requests", extra = ["socks"] }, { name = "rich" }, + { name = "rich-click" }, { name = "rlaphoenix-m3u8" }, { name = "rnet" }, { name = "ruamel-yaml" }, @@ -2216,7 +2207,6 @@ test = [ requires-dist = [ { name = "aiohttp", specifier = ">=3.14.0,<4" }, { name = "aiohttp-swagger3", specifier = ">=0.9.0,<1" }, - { name = "animeapi-py", specifier = ">=0.6.0" }, { name = "appdirs", specifier = ">=1.4.4,<2" }, { name = "bandit", specifier = ">=1.9.4" }, { name = "brotli", specifier = ">=1.1.0,<2" }, @@ -2249,6 +2239,7 @@ requires-dist = [ { name = "pyyaml", specifier = ">=6.0.1,<7" }, { name = "requests", extras = ["socks"], specifier = ">=2.32.5,<3" }, { name = "rich", specifier = ">=13.7.1,<15" }, + { name = "rich-click", specifier = ">=1.9.8" }, { name = "rlaphoenix-m3u8", specifier = ">=3.4.0,<4" }, { name = "rnet", specifier = ">=2.4.2" }, { name = "ruamel-yaml", specifier = ">=0.18.6,<0.19" },