Compare commits

...
6 Commits
Author SHA1 Message Date
imSp4rky 83eef01fad chore(release): bump version to 6.0.0 2026-10-13 16:32:19 -06:00
imSp4rky e7bb825def perf(manifests): decode manifests without charset detection
Parse an MPD from bytes and read every M3U8 playlist as UTF-8 unless the server sent a charset. Fixes mojibake on text/* responses.
2026-10-13 16:11:13 -06:00
imSp4rky 4fbaf2a61a feat(api): forward exact_lang to the service context
The remote-dl client sends exact_lang in dl_params, and REST download jobs set it in the service ctx, so a service can skip the manifests of close-match languages.
2026-10-13 15:07:01 -06:00
imSp4rky 3e64a7d684 fix(api): follow the cdm config key order for the server CDM 2026-10-13 13:40:38 -06:00
imSp4rky 2ba442f50e feat(dl): load the DRM system a cdm entry lists first
The first of the widevine / playready config keys in a cdm entry sets which device loads first and which DRM system licenses a track that offers the two. An entry that lists playready first changes from Widevine to PlayReady.
2026-10-13 12:44:11 -06:00
imSp4rky af4e6d5d47 fix(hls): read the key id from the init segment when the PSSH has none
An EXT-X-KEY whose Widevine PSSH holds a content id and no key id raised KIDNotFound. Covers download, licence-only, --all-drm, session keys, import and the server CDM path. Fixes #166.
2026-10-13 12:14:34 -06:00
32 changed files with 729 additions and 117 deletions
+261
View File
@@ -6,6 +6,267 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
This changelog is automatically generated using [git-cliff](https://git-cliff.org).
## [6.0.0] - 2026-10-13
### Features
- *binary*: Add custom binary registration for services
- *cdm*: Send license_type and privacy_mode with the request to the server
- *click*: Change the minimum time for slow
- *downloader*: Adaptive worker scaling, tail boost, session parity, resume fixes
- *downloader*: Fan large segment batches across worker processes
- *downloader*: Resume failed downloads from completed segments across runs
- *dl*: Run user post-download scripts with metadata variables
- *dl*: Add --require-audio/--require-video and --warn-only alias
- *drm*: Let a track pin the DRM system it licenses with
- *dl*: Add --no-attachments to skip cover art and subtitle fonts
- *dl*: Add --cdm to pin one CDM device for a run
- *post-scripts*: Add wait option to block until the script exits
- *post-scripts*: Show a spinner while waiting for a script
- *services*: Add services_repo_force and compact refresh output
- *serve*: Hot-reload service repos on a timer without a restart
- *serve*: Gate server_cdm per service and let clients follow it
- *kv*: Show a progress bar while adding keys to a vault
- *serve*: [**breaking**] Gate the server's own proxies behind server_proxy
- *vaults*: Batch HTTP json vault writes with InsertKeys
- *remote*: Forward HLS segment filters to remote clients
- *serve*: Add --quiet mode and developer dashboard API
- *serve*: [**breaking**] Add server-held accounts for remote sessions
- *serve*: Forward per-session service logs to remote-dl clients
- *proxy*: Mask proxy credentials in console and API output
- *remote*: Spoof a type-only CDM so server_cdm clients need no local .wvd/.prd
- *serve*: Add operator dashboard endpoints and per-key rate limits
- *audio*: Support DTS:X Profile 2 with MP4 muxing
- *dl*: Add --no-postscript to skip post-scripts for one run
- *api*: Report the live title and the client's command line to the dashboard
- *vaults*: Flag and skip wrong content keys from poisoned vaults
- *remote*: Verify server-CDM vault keys on the client and report bad pairs
- *dl*: Add --proxy-download for a separate download proxy
- *vaults*: Report bad keys to the vault that served them
- *service*: Add GEOBLOCK negative geofence
- *dash*: Add merge_segments to append segments during the download
- *manifests*: Extend merge_segments rolling merge to HLS and ISM
- *export*: Write mediaexport files and read legacy exports through the package
- *proxies*: Add Control D proxy provider
- *export*: Record each track's KIDs
- *import*: Build tracks from direct-file export rows
- *export*: Name export files after their titles and tracks
- *export*: Separate export file name parts with dots
- *remote*: Keep slow remote session requests alive with a heartbeat
- *remote*: Cap server CDM licensing by height per API key and service
- *remote*: List only configured services on the dashboard
- *serve*: Relay service prompts from download jobs to the REST client
- *remote*: Let a remote client license with its own device identity
- *dl*: Add --latest-episodes N to download the N newest episodes
- *setup*: Add unshackle setup command and cross-platform install scripts
- *remote*: License with the client's device when the user picked one
- *dl*: Add --all-drm to license Widevine and PlayReady per track
- *dl*: Add --audio-description-only and --forced-subs-only
- *serve*: Log CDM calls for the dashboard
- *serve*: Reload unshackle.yaml on SIGHUP without a restart
- *serve*: Add --cdm-only for a CDM backend with no REST API
- *service*: Add wait_for_approval and return the cache for a login with no prompt
- *serve*: Relay approval messages to jobs and proxy remote downloads
- *search*: Add ANONYMOUS_SEARCH to skip the login for a service
- *dl*: Add --sample and --include-sample
- *dl*: Load the DRM system a cdm entry lists first
- *api*: Forward exact_lang to the service context
### Bug Fixes
- *downloader*: Release hedge loser file handles before merge sweep
- *downloader*: Validate 206 on byte-range segments and force identity encoding
- *downloader*: Stop leaked racers holding segment handles after a failed batch
- *dash*: License ClearKeyCENC before decrypt
- *subtitle*: Convert ASS to SRT in the SDH-strip fallback
- *api*: Keep sub_format "original" working through the API
- *webvtt*: Don't shorten cues when splicing boundary duplicates
- *api*: Reject backslash traversal in safe_cache_key
- *downloader*: Honor Retry-After with cancel-safe backoff, add fault-injection bench and tests
- *downloader*: Keep speed-limited batches in a single process
- *downloader*: Fail closed on rnet content codings outside the old allowlist
- *downloader*: Accept a 200 whose body is exactly the requested byte-range slice
- *downloader*: Report rolling speed on the multiprocess progress bar
- *downloaders*: Submit segments in a window so progress can drain
- *api*: Pass adaptive_workers and download_processes to dl.result
- *downloader*: Validate Content-Range before placing 206 bytes
- *downloader*: Harden ranged reads, teardown and multiprocess sessions
- *titles*: Drop folder {year?} when it repeats the dated season label
- *cli*: Work around rich-click's patched click classes
- *vaults*: Look tables up by the name the vault stores
- *hls*: Keep ad-filtered segments out of DRM key state
- *music*: Remux songs into the container of their codec
- *api*: Survive Windows errno 13 on the job progress file
- *api*: Guard two more Windows errno 13 paths in the serve layer
- *dl*: Accept a clamped 206 and report audio-description-only titles
- *services*: Match service aliases without case
- *downloader*: Keep retrying while a cut connection makes progress
- *downloader*: Bound requests after a cut and stop retry nesting
- *subtitle*: Keep WebVTT-only cue markup out of converted subtitles
- *proxies*: Resolve windscribevpn hostname prefixes and uk alias
- *mux*: Give each parallel mux its own chapters temp file
- *remote*: Carry the full title and track state across the remote API
- *HTTP*: Simplify key retrieval condition in get_key method
- *tracks*: Match resolutions per range and codec
- *remote*: Resolve service names against the remote server
- *remote*: Send only the active profile's cache files to the server
- *utils*: Preserve _ssl_context when copying SSLCiphers adapter
- *util*: Remove :
- *cli*: Show banner when dl -q is used
- *serve*: License multi-key PlayReady through server CDM instead of vault subset
- *remote*: Raise GET timeout to 120s so large catalogs load
- *cdm*: Scope remote CDM licence state to its session
- *remote*: Pass fallback language to ISM re-parse
- *ism*: Signal Dolby Vision and Atmos from Smooth manifests
- *dash*: Number SegmentTimeline segments from startNumber
- *remote*: Match re-parsed manifest tracks by ID before attributes
- *remote*: Send per-track MPD fragments for service-built AdaptationSets
- *dl*: Repair damaged cookie files without rewriting them
- *kv*: Keep copy log rows single-spaced
- *dl*: Fall back to local vaults when the server CDM returns no key
- *cli*: Stop service tags from renaming CLI options
- *dl*: Log vault writes under their own title
- *dl*: Specify 'subtitle tracks' in error log message
- *remote*: Keep option choices and optional title in remote -h
- *audio*: Accept immersive channel layouts like 5.1.4
- *serve*: Resolve DRM keys for manifests with no ContentProtection PSSH
- *dl*: Reject local proxies when --remote targets another machine
- *dl*: End a cancelled track by raising instead of returning short
- *services*: Name the build a compiled service is missing
- *serve*: Lock shared session state and throttle progress writes
- *downloader*: Correct truncation, ranged sizing, and session carry
- *dl*: Cache server-CDM keys to the local vaults
- *drm*: License every PlayReady header until all KIDs are covered
- *dash*: Treat a missing FFprobe as no PSSH instead of crashing
- *api*: License every PlayReady header in the server CDM path
- *api*: Reach every PlayReady header on all server CDM routes
- *api*: Let the server CDM see service-supplied keys and HLS track DRM
- *api*: Tell the service that the client sent a cookie jar
- *api*: Stop serve leaking server credentials, paths, and cross-tenant data
- *api*: Keep error URLs readable and mask host paths in dashboard logs
- *api*: Reject a NUL byte in output_dir before resolving it
- *manifests*: Flag DASH audio using the standard description role
- *api*: Serialize the job progress replace to stop Windows errno 13
- *api*: Decode challenge bytes to UTF-8 for PlayReady license request
- *proxy*: Check the proxy exit IP before any service request
- *cookies*: Find Firefox profiles in the XDG config directory
- *naming*: Correct language tag rule matching and add state conditions
- *config*: Keep no and yes as language tags
- *import*: Inject only the imported title's keys into each track
- *main*: Remove unwanted -
- *serve*: Defer service hot reload while a remote session uses it
- *remote*: Return session cache for device-code and cache-only logins
- *remote*: Honor proxy client cdm_type for DRM system selection
- *drm*: Log missing Widevine/PlayReady PSSH at debug level
- *cli*: Make the click param types the one source of codec spellings
- *vaults*: Verify vault content keys per KID from the fragment map
- *vaults*: Harden the KID window check and clear a stale flag on licence
- *remote*: Forward nested service cache files to and from the server
- *remote*: Stop warning about missing keys for tracks that carry no DRM
- *remote*: Keep the server's own CDM when the tier licenses with server_cdm
- *post-scripts*: Expose {part} and all title-owned variables on failure
- *serve*: Apply staged service reloads when sessions end, with a max staging age
- *remote*: Let the server's CDM mapping decide the DRM system under server_cdm
- *remote*: Forward dl track selection and profile to remote sessions
- *dl*: Skip empty vault pushes and count cached keys once
- *serve*: Stop repeating server vault pushes and retry failed ones
- *export*: Import side-loaded subtitles and keep conflicting keys apart
- *export*: Strip session headers from exports and normalise merged keys
- *import*: Read export files through mediaexport
- *export*: Skip bad keys and failed writes without stopping the download
- *import*: Report refused titles and fail when none is usable
- *drm*: Give the zero-KID fallback only the track's own key
- *import*: Give each track only the keys it declares
- *import*: Stop clearly when no export or vault key exists
- *export*: Name the file from the first write
- *import*: Say an import uses the export's keys, not a server CDM
- *import*: Give no title keys to a track that is clear
- *import*: Read an HLS export that names no title language or master key
- *remote*: Report a missing title list and a refused track URI clearly
- *tracks*: Keep the Dolby Vision record when remuxing video with ffmpeg
- *proxies*: Match surfshark's current server api
- *proxies*: Skip a proxy provider that fails to load
- *proxies*: Return None for unservable queries and repair live providers
- *proxies*: Link to the expressvpn device login with the code prefilled
- *cdm*: Skip the server header probe for pywidevine and pyplayready remote CDMs
- *api*: Stop dashboard event and log reads racing worker threads
- *api*: Keep the worker's error on failed jobs, gate tracebacks behind --debug-api
- *sanitize*: Keep nonspacing marks on letters in unicode filenames
- *sanitize*: Normalise filenames to NFC and keep combining marks
- *console*: Write UTF-8 to redirected streams on Windows
- *dl*: Keep the --enrich original language after get_tracks
- *attachment*: Keep custom fonts after mux cleanup
- *dl*: Keep the --enrich title and year after get_tracks
- *docs*: Hide the release version in the header repo widget
- *dl*: Read the vaults first under --all-drm
- *hybrid*: Read the container file for a base that had the DV fixup
- *cdm*: Send a PlayReady licence without an XML declaration as XML
- *import*: Carry the HLS segment filter in an export
- *cdm*: Simplify cached keys retrieval in CustomRemoteCDM
- *hls*: Read the key id from the init segment when the PSSH has none
- *api*: Follow the cdm config key order for the server CDM
### Documentation
- *session*: Correct stale pool-size comment
- Strip non-essential comments, fold key facts into docstrings
- *services*: Add MUSIC_EXAMPLE reference service
- Apply simplified technical english across docs and docstrings.
- Phase 2 of the prose style across docs and docstrings
- Widen supported shaka-packager range to 2.6.1 or >= 3.10.0
- *dev*: Correct get_chapters guidance on markers at timestamp 0
- *example*: Show drm_preference pinning for dual-PSSH manifests
- *post-scripts*: Use post-script as the one term for hooks
- *api*: Describe the dashboard session payload in the OpenAPI yaml
- *cli*: Correct --merge-video scope in cli reference
- *export*: Describe the export file name and its messages
### Performance Improvements
- *downloader*: Slow-start ramp for the adaptive worker controller
- *downloader*: Use per-arrival reads only while a segment is hedge-raced
- *dash*: Collapse single-URL byte-range segments into one direct download
- *downloader*: Stride multiprocess chunk assignment across children
- *api*: Skip progress re-parse when the file is unchanged
- *api*: Append job history instead of rewriting the file
- *downloader*: Unblock hedge-loser reads at teardown
- *dash*: Speed up segment collection and URL templating
- *dl*: Parallelise mux, licensing, and vault lookups; size the shared pool
- *dl*: Overlap hybrid builds with muxing, tag in the mux thread, preload DRM in parallel
- *hybrid*: Reuse DV-side files across resolutions, trim the RPU chain, fix L6 fallback
- *dl*: Push vault key writes to a background thread
- *drm*: Decrypt fMP4 segments during download with mp4decrypt
- *kv*: Push whole services to SQL vaults in one call
- *kv*: Push keys to vaults concurrently, log one summary per title
- *sample*: Download only the subtitle segments around the window
- *manifests*: Decode manifests without charset detection
### Changes
- *monalisa*: Remove ML-Worker and improve MonaLisa DRM
- *downloader*: Extract shared byte-range split helper
- *console*: Route interactive prompts through a themed helper
- *core*: [**breaking**] Drop leading underscores from internal definitions
- *tests*: Drop leading underscores from test helper functions
- *music*: [**breaking**] Make Song an ordinary Title on the generic path
- *console*: Style env listings like the rich-click help screens
- *drm*: Drop the hardcoded service branch in the license flow
- *proxies*: Share proxy dispatch between dl, search and resolve_proxy
### Maintenance
- Restore memory/ to .gitignore from dev
- *scripts*: Expand bench suite for cross-machine runs
- *scripts*: Harden bench suite report capture
- *scripts*: Add attempt-1-only tail-slow fault to downloader bench
- Bump dev version to 5.5.0
- Mark install.sh executable
### Builds
- *deps*: Pull mediaexport from github in the package metadata
- *deps*: Lock mediaexport at fa9357e
## [5.4.0] - 2026-08-15
### Features
+2 -2
View File
@@ -316,7 +316,7 @@ Show the video, audio, and subtitle tracks for a title. For series, you can scop
| `part` | int/string | no | Part index of a split episode (combined with `season` and `episode`). |
| `profile`, `proxy`, `no_proxy`, `cdm_type` | - | no | As on `list-titles`. |
| `credentials`, `cookies`, `cache` | - | no | As on `list-titles`. |
| `dl_params` | object | no | The `dl` track selection that the service reads from `ctx.parent.params`: `lang`, `v_lang`, `a_lang`, `acodec` and `forced_subs`, as on [`POST /api/session/create`](#post-apisessioncreate). Some services select the manifests they fetch from these values. |
| `dl_params` | object | no | The `dl` track selection that the service reads from `ctx.parent.params`: `lang`, `v_lang`, `a_lang`, `acodec`, `forced_subs` and `exact_lang`, as on [`POST /api/session/create`](#post-apisessioncreate). Some services select the manifests they fetch from these values. |
When you give both `season` and `episode`, the server combines them into a `"{season}x{episode}"` selector. `part` narrows that to `"{season}x{episode}.{part}"`, one part of a [split episode](../creating-a-service.md#split-episodes). Without `part` you get every part of the episode. The server reads `part` only when you give both `season` and `episode`, and `wanted` takes precedence over all three. To scope a part by range, put it in `wanted` instead (`"S01E01.2"`).
@@ -958,7 +958,7 @@ the newlines must go out at once.
Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files, keyed by the file path relative to the service cache directory with `/` separators and no `.json` suffix, so a nested key such as `session_web/<sha1>` lands in a subdirectory), `client_region`, `proxy_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, `dl_params`, `client`, and any service CLI options. `proxy_region` is the country code the client resolved `proxy` from; the server matches it against its own accounts. `client` is a freeform object the dashboard shows as sent (the CLI puts `version`, `code_hash`, `platform` and a redacted `argv` in it); the server ignores it above 4096 bytes of JSON.
`dl_params` is an object with the `dl` track selection: `lang`, `v_lang` and `a_lang` (arrays of language strings), `acodec` (an array of audio codec names or `dl -a` aliases such as `ddp`) and `forced_subs` (boolean). The service reads these values from `ctx.parent.params`. The server uses the `dl` default for each absent or malformed value, and drops an unknown codec name.
`dl_params` is an object with the `dl` track selection: `lang`, `v_lang` and `a_lang` (arrays of language strings), `acodec` (an array of audio codec names or `dl -a` aliases such as `ddp`), and `forced_subs` and `exact_lang` (booleans). The service reads these values from `ctx.parent.params`. The server uses the `dl` default for each absent or malformed value, and drops an unknown codec name.
Service CLI options also travel in a nested `service_params` object, which wins over a flat key with the same name. `profile` at the top level always means the credentials profile, never a service's own `--profile` option.
+2 -2
View File
@@ -432,11 +432,11 @@ object the dashboard shows as sent), and the track-selection hints `range_`, `vc
`dl_params` holds the `dl` track selection: `lang`, `v_lang` and `a_lang`
(arrays of language strings), `acodec` (an array of audio codec names, such as
`EC3`), and `forced_subs` (boolean). The server gives these values to the
`EC3`), and `forced_subs` and `exact_lang` (booleans). The server gives these values to the
service in `ctx.parent.params`, never as service options, so a service option
with the same name, such as an own `--lang`, does not get them. The server uses
the `dl` default for each absent or malformed value: `lang` is `["orig"]`,
`v_lang`, `a_lang` and `acodec` are `[]`, and `forced_subs` is `false`. The
`v_lang`, `a_lang` and `acodec` are `[]`, and `forced_subs` and `exact_lang` are `false`. The
server drops an unknown codec name. An empty array is a valid value and stays
empty.
+3
View File
@@ -495,6 +495,9 @@ unshackle dl --skip-dl --all-drm --export EXAMPLE 81234567
- The `cdm` mapping must name a device for each system. A single device name, or `--cdm`,
stops the run with an error that names the missing system.
- The order of the `widevine` and `playready` config keys in the `cdm` entry sets the order
of the two systems, as in a run without the flag. Each track licenses first with the
system of the first config key.
- A Key Vault is read first, as in a run without the flag. A DRM system sends a challenge
only for a KID that no Key Vault and no earlier track of the run gave a content key
for. When the first system returns all the KIDs of the second, the second sends no
+5
View File
@@ -45,6 +45,11 @@ are `widevine` / `playready`. unshackle matches the quality keys first, but only
height is known. If the value is still a dict after that, unshackle uses
`widevine`/`playready` when present, or else the credential profile name, then `default`.
When an entry holds `widevine` and `playready`, their order sets which device unshackle loads
first. A track that offers the two DRM systems licenses with that system. A track that offers
only the other system uses the other device. A `widevine` / `playready` dict in a quality key
does not set the order. The server CDM of a remote session follows the same order.
A quality key can also hold a dict of `widevine` / `playready` devices, and so can the
top-level `default` entry:
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "unshackle"
version = "5.5.0"
version = "6.0.0"
description = "Modular Movie, TV, and Music Archival Software."
authors = [{ name = "unshackle team" }]
requires-python = ">=3.11,<3.15"
+40 -1
View File
@@ -17,7 +17,7 @@ from pywidevine.pssh import PSSH
from rich.table import Table
from unshackle.commands.dl import dl
from unshackle.core.api.handlers import validate_download_parameters
from unshackle.core.api.handlers import load_full_cdm, resolve_server_cdm, validate_download_parameters
from unshackle.core.cdm.detect import cdm_type_stub, is_widevine_cdm
from unshackle.core.drm import PlayReady, Widevine
from unshackle.core.manifests import HLS
@@ -286,3 +286,42 @@ def test_the_init_segment_adds_the_drm_system_the_track_does_not_have(monkeypatc
Track.add_init_drm(track, b"init", None)
assert track.drm == [wv, pr]
@pytest.mark.parametrize(
("entry", "expected"),
[
({"playready": "pr_device", "widevine": "wv_device"}, "pr_device"),
({"widevine": "wv_device", "playready": "pr_device"}, "wv_device"),
],
)
def test_the_first_drm_system_of_the_config_entry_loads_first(
cmd: dl, monkeypatch: pytest.MonkeyPatch, entry: dict, expected: str
) -> None:
monkeypatch.setattr("unshackle.commands.dl.config.cdm", {"EXAMPLE": entry})
monkeypatch.setattr("unshackle.core.cdm.load_cdm", lambda name, **_: name)
monkeypatch.setattr("unshackle.core.config.config.cdm", {"EXAMPLE": entry})
cmd.vaults = None
assert cmd.get_cdm("EXAMPLE") == expected
assert load_full_cdm("EXAMPLE", None) == expected
assert cmd.get_cdm("EXAMPLE", drm="widevine") == "wv_device"
@pytest.mark.parametrize(
("entry", "expected"),
[
({"playready": "pr_device", "widevine": "wv_device"}, "playready"),
({"widevine": "wv_device", "playready": "pr_device"}, "widevine"),
],
)
def test_the_server_cdm_stub_is_the_first_drm_system_of_the_config_entry(
monkeypatch: pytest.MonkeyPatch, entry: dict, expected: str
) -> None:
monkeypatch.setattr("unshackle.core.config.config.cdm", {"EXAMPLE": entry})
monkeypatch.setattr(
"unshackle.core.api.handlers.detect_cdm_type",
lambda name, _: "playready" if name == "pr_device" else "widevine",
)
assert is_widevine_cdm(resolve_server_cdm("EXAMPLE", None, None)) is (expected == "widevine")
+132
View File
@@ -0,0 +1,132 @@
"""An HLS Widevine key whose PSSH holds a content id and no Key ID takes the Key ID of its init segment."""
from __future__ import annotations
from pathlib import Path
from types import SimpleNamespace
from typing import Any
from uuid import UUID
import m3u8
from pywidevine.license_protocol_pb2 import WidevinePsshData
from pywidevine.pssh import PSSH
from unshackle.core.constants import DOWNLOAD_LICENCE_ONLY
from unshackle.core.drm import PlayReady, Widevine
from unshackle.core.import_service import hls_playlist_kids
from unshackle.core.manifests.hls import HLS
from unshackle.core.tracks import Video
from unshackle.core.tracks.track import DownloadContext, Track
INIT_KID = UUID("11111111-1111-1111-1111-111111111111")
PSSH_KID = UUID("22222222-2222-2222-2222-222222222222")
TAG_KID = UUID("33333333-3333-3333-3333-333333333333")
ZERO_KID = UUID(int=0)
def no_kid_pssh(content_id: bytes = b"content-id-only") -> str:
return PSSH.new(
system_id=PSSH.SystemId.Widevine,
init_data=WidevinePsshData(content_id=content_id).SerializeToString(),
).dumps()
NO_KID_PSSH = no_kid_pssh()
KEY_LINE = (
'#EXT-X-KEY:METHOD=SAMPLE-AES-CTR,URI="data:text/plain;base64,{pssh}",'
'KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed",KEYFORMATVERSIONS="1"'
)
def keyed_segment(pssh: str, extra: str = "", init: str = "init.mp4") -> str:
return f'#EXT-X-MAP:URI="{init}"\n{KEY_LINE.format(pssh=pssh)}{extra}\n#EXTINF:6.0,\n{init}.m4s\n'
def playlist(*segments: str) -> str:
body = "#EXT-X-DISCONTINUITY\n".join(segments)
return f"#EXTM3U\n#EXT-X-VERSION:7\n#EXT-X-TARGETDURATION:6\n{body}#EXT-X-ENDLIST\n"
def key_of(pssh: str, extra: str = "") -> m3u8.Key:
return m3u8.loads(playlist(keyed_segment(pssh, extra))).keys[0]
def licence_only(text: str, tmp_path: Path) -> list[tuple[list[UUID], Any]]:
"""Run a licence-only download of the playlist and return the Key IDs of each licence call."""
media = tmp_path / "media.m3u8"
media.write_text(text, encoding="utf-8")
track = Video(
url="https://example.test/media.m3u8",
language="en",
descriptor=Track.Descriptor.HLS,
codec=Video.Codec.AVC,
)
track.from_file = media
licensed: list[tuple[list[UUID], Any]] = []
ctx = DownloadContext(
save_path=tmp_path / "video.mp4",
save_dir=tmp_path / "video_segments",
progress=lambda **_: None,
license_widevine=lambda drm, track_kid=None: licensed.append((drm.kids, track_kid)),
cdm=None,
)
DOWNLOAD_LICENCE_ONLY.set()
try:
HLS.download_track(track=track, ctx=ctx)
finally:
DOWNLOAD_LICENCE_ONLY.clear()
return licensed
def test_download_track_licenses_with_the_init_segment_kid(tmp_path: Path, monkeypatch: Any) -> None:
# the probe itself needs the network and FFprobe; the order of the calls is under test
monkeypatch.setattr(HLS, "get_track_kid_from_init", staticmethod(lambda *args, **kwargs: INIT_KID))
assert licence_only(playlist(keyed_segment(NO_KID_PSSH)), tmp_path) == [([INIT_KID], INIT_KID)]
def test_licence_only_reads_each_later_key_from_its_own_init_segment(tmp_path: Path, monkeypatch: Any) -> None:
def probe(master: Any, track: Any, session: Any, init_section: Any = None) -> UUID:
return PSSH_KID if init_section is not None and init_section.uri == "init2.mp4" else INIT_KID
monkeypatch.setattr(HLS, "get_track_kid_from_init", staticmethod(probe))
text = playlist(keyed_segment(no_kid_pssh(b"first")), keyed_segment(no_kid_pssh(b"later"), init="init2.mp4"))
assert licence_only(text, tmp_path) == [([INIT_KID], INIT_KID), ([PSSH_KID], PSSH_KID)]
def test_get_drm_keeps_a_pssh_that_has_its_own_kid() -> None:
pssh = PSSH.new(key_ids=[PSSH_KID], system_id=PSSH.SystemId.Widevine).dumps()
drm = HLS.get_drm(key_of(pssh), kid=INIT_KID)
assert drm.kids == [PSSH_KID]
assert drm.pssh.dumps() == pssh
def test_get_drm_prefers_the_tag_keyid_over_the_init_segment_kid() -> None:
drm = HLS.get_drm(key_of(NO_KID_PSSH, f",KEYID=0x{TAG_KID.hex}"), kid=INIT_KID)
assert drm.kids == [TAG_KID]
def test_get_drm_replaces_a_placeholder_pssh_kid_and_ignores_a_placeholder_init_kid() -> None:
placeholder = PSSH.new(key_ids=[ZERO_KID], system_id=PSSH.SystemId.Widevine).dumps()
assert HLS.get_drm(key_of(placeholder), kid=INIT_KID).kids == [INIT_KID]
assert HLS.get_drm(key_of(placeholder), kid=ZERO_KID).kids == [ZERO_KID]
def test_session_key_without_a_kid_does_not_stop_the_parse() -> None:
with_kid = PSSH.new(key_ids=[PSSH_KID], system_id=PSSH.SystemId.Widevine).dumps()
drm = HLS.get_all_drm([key_of(NO_KID_PSSH), key_of(with_kid)])
assert [d.kids for d in drm] == [[PSSH_KID]]
def test_alternate_drm_builds_the_other_system_with_the_init_segment_kid() -> None:
chosen = object.__new__(PlayReady)
alternate = HLS.alternate_drm([key_of(NO_KID_PSSH)], chosen, kid=INIT_KID)
assert [type(d) for d in alternate] == [Widevine] and alternate[0].kids == [INIT_KID]
def test_import_reads_the_kid_of_a_playlist_whose_key_has_none() -> None:
def get(url: str, headers: Any = None) -> Any:
return SimpleNamespace(text=playlist(keyed_segment(NO_KID_PSSH)), ok=True, content=b"init")
track = SimpleNamespace(id="vid", url="https://example.test/media.m3u8", get_key_id=lambda init: INIT_KID)
assert hls_playlist_kids(track, SimpleNamespace(get=get)) == {INIT_KID}
+66 -3
View File
@@ -1,6 +1,7 @@
"""Unit tests for M3U8 response decoding. Reading `.text` on a response with no encoding
makes requests run chardet over the whole body, so both playlist fetch paths set utf-8
first. A charset the server did send must survive."""
"""Unit tests for manifest response decoding. Reading `.text` on a response with no encoding
makes requests run chardet over the whole body, so the playlist fetch paths set utf-8
first. A charset the server did send must survive. An MPD is parsed from bytes, so the
XML declaration gives the encoding."""
from __future__ import annotations
@@ -9,7 +10,9 @@ from typing import Any, Optional
import pytest
import requests
from unshackle.core.manifests.dash import DASH
from unshackle.core.manifests.hls import HLS
from unshackle.core.session import RnetResponse, utf8_text
from unshackle.core.tracks import Video
pytestmark = pytest.mark.unit
@@ -24,6 +27,8 @@ def make_response(body: bytes, url: str, encoding: Optional[str]) -> requests.Re
res._content = body
res.url = url
res.encoding = encoding
if encoding:
res.headers["Content-Type"] = f"application/vnd.apple.mpegurl; charset={encoding}"
return res
@@ -74,3 +79,61 @@ def test_load_drm_from_playlist_keeps_a_charset_the_server_sent() -> None:
make_track(res).load_drm_from_playlist()
assert res.encoding == "iso-8859-1"
MPD_LATIN1 = '<?xml version="1.0" encoding="ISO-8859-1"?><MPD><Title>café</Title></MPD>'.encode("iso-8859-1")
class StubRnetResponse:
"""The part of an rnet response that `RnetResponse` reads."""
ok = True
status_code = 200
encoding = None
def __init__(self, body: bytes, url: str) -> None:
self.body = body
self.url = url
def bytes(self) -> bytes:
return self.body
def test_dash_from_url_follows_the_xml_declaration() -> None:
res = make_response(MPD_LATIN1, "https://example.com/manifest.mpd", None)
dash = DASH.from_url("https://example.com/manifest.mpd", FakeSession(res))
assert dash.manifest.findtext("Title") == "café"
def test_dash_from_url_reads_an_rnet_response() -> None:
res = RnetResponse(StubRnetResponse(MPD_LATIN1, "https://example.com/manifest.mpd"))
dash = DASH.from_url("https://example.com/manifest.mpd", FakeSession(res)) # type: ignore[arg-type]
assert dash.manifest.findtext("Title") == "café"
def test_utf8_text_reads_an_rnet_response() -> None:
assert utf8_text(RnetResponse(StubRnetResponse(MEDIA, "https://example.com/video/720.m3u8"))) == MEDIA.decode()
def test_utf8_text_ignores_the_requests_default_for_a_text_type() -> None:
res = make_response('#EXTM3U\n#EXT-X-MEDIA:TYPE=AUDIO,NAME="中文"\n'.encode(), "https://example.com/a.m3u8", None)
res.headers["Content-Type"] = "text/plain"
res.encoding = "ISO-8859-1"
assert "中文" in utf8_text(res)
def test_dash_from_url_uses_the_http_charset_without_an_xml_declaration() -> None:
res = make_response("<MPD><Title>中文</Title></MPD>".encode("gbk"), "https://example.com/manifest.mpd", "gbk")
dash = DASH.from_url("https://example.com/manifest.mpd", FakeSession(res))
assert dash.manifest.findtext("Title") == "中文"
def test_dash_from_url_rejects_an_empty_body() -> None:
res = make_response(b"", "https://example.com/manifest.mpd", None)
with pytest.raises(ValueError, match="must be provided"):
DASH.from_url("https://example.com/manifest.mpd", FakeSession(res))
+32
View File
@@ -21,6 +21,7 @@ from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import pad
from pywidevine.pssh import PSSH
from tests.core.test_hls_kid_from_init import KEY_LINE, no_kid_pssh
from tests.core.test_ism_init import piff_fragment
from unshackle.core.config import config
from unshackle.core.constants import DOWNLOAD_CANCELLED
@@ -264,3 +265,34 @@ def test_hls_aes128_keeps_post_merge(server, tmp_path, monkeypatch):
plain, rolling = download_both_ways(HLS, make, tmp_path, monkeypatch, False)
assert rolling == plain == INIT + b"".join(PARTS)
def test_hls_rekey_takes_the_kid_from_the_init_segment_of_the_new_key(server, tmp_path, monkeypatch):
"""Each key carries a PSSH with a content id and no Key ID, so only the init segment names the KID."""
later_kid = UUID(hex="22222222222222222222222222222222")
later_init = bytes(range(100, 200))
server.routes["/init2.mp4"] = later_init
no_decrypt(monkeypatch, hls_module)
monkeypatch.setattr(config, "merge_segments", False)
def key_line(content_id: bytes) -> str:
return KEY_LINE.format(pssh=no_kid_pssh(content_id))
track = hls_track(server, False)
lines = ["#EXTM3U", "#EXT-X-VERSION:7", "#EXT-X-TARGETDURATION:4", '#EXT-X-MAP:URI="init.mp4"', key_line(b"a")]
for i in range(len(PARTS)):
if i == 2:
lines += ["#EXT-X-DISCONTINUITY", '#EXT-X-MAP:URI="init2.mp4"', key_line(b"b")]
lines += ["#EXTINF:4.0,", f"part{i}.m4s"]
server.routes["/media.m3u8"] = "\n".join(lines + ["#EXT-X-ENDLIST"]).encode()
# the real probe needs FFprobe and a tenc box; the KID follows the init bytes the parser hands over
monkeypatch.setattr(
type(track), "get_key_id", lambda self, init_data=None, **kw: {later_init: later_kid}.get(init_data, KID)
)
licensed: list[UUID] = []
ctx = make_ctx(tmp_path, "rekey.mp4")
ctx.license_widevine = lambda drm, track_kid=None: licensed.append(track_kid)
HLS.download_track(track, ctx)
assert licensed == [KID, later_kid]
+12 -4
View File
@@ -192,10 +192,17 @@ class _CapturedCtx(Exception):
@pytest.mark.parametrize(
("params", "expected"),
[
({}, (["orig"], [], [], [], False)),
({}, (["orig"], [], [], [], False, False)),
(
{"lang": ["es-419"], "v_lang": ["en"], "a_lang": ["ja"], "acodec": "AAC,EC3", "forced_subs": True},
(["es-419"], ["en"], ["ja"], ["AAC", "EC3"], True),
{
"lang": ["es-419"],
"v_lang": ["en"],
"a_lang": ["ja"],
"acodec": "AAC,EC3",
"forced_subs": True,
"exact_lang": True,
},
(["es-419"], ["en"], ["ja"], ["AAC", "EC3"], True, True),
),
],
)
@@ -224,7 +231,8 @@ def test_perform_download_puts_the_lang_selection_in_the_service_ctx(
ctx = exc.value.args[0]
p = ctx.params
assert (p["lang"], p["v_lang"], p["a_lang"], [c.name for c in p["acodec"]], p["forced_subs"]) == expected
selection = (p["lang"], p["v_lang"], p["a_lang"], [c.name for c in p["acodec"]], p["forced_subs"], p["exact_lang"])
assert selection == expected
def test_perform_download_passes_latest_episodes_to_dl_result(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -13,6 +13,7 @@ from uuid import UUID
import pytest
from pywidevine.pssh import PSSH as WvPSSH
from tests.core.test_hls_kid_from_init import NO_KID_PSSH
from unshackle.core.api import handlers
from unshackle.core.drm import Widevine
from unshackle.core.tracks import Track
@@ -22,13 +23,14 @@ pytestmark = pytest.mark.unit
KID = UUID("161219ec3df64e0eadbb3827c0ccc46d")
def _media_playlist() -> str:
pssh = WvPSSH.new(key_ids=[KID], system_id=WvPSSH.SystemId.Widevine).dumps()
def _media_playlist(pssh: str | None = None) -> str:
pssh = pssh or WvPSSH.new(key_ids=[KID], system_id=WvPSSH.SystemId.Widevine).dumps()
return "\n".join(
[
"#EXTM3U",
"#EXT-X-VERSION:6",
"#EXT-X-TARGETDURATION:6",
'#EXT-X-MAP:URI="init.mp4"',
f'#EXT-X-KEY:METHOD=SAMPLE-AES,URI="data:text/plain;base64,{pssh}",'
'KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed",KEYFORMATVERSIONS="1"',
"#EXTINF:6.0,",
@@ -49,3 +51,21 @@ def test_hls_descriptor_alone_reaches_the_media_playlist_keys():
assert track.drm and isinstance(track.drm[0], Widevine), "media playlist DRM not read"
assert track.drm[0].kids == [KID]
assert base64.b64decode(handlers.extract_pssh_from_track(track, "widevine"))
def test_key_without_a_kid_takes_it_from_the_init_segment():
"""A PSSH that holds a content id and no Key ID is DRM, and the server must not call the track clear."""
response = SimpleNamespace(text=_media_playlist(NO_KID_PSSH), ok=True, content=b"init")
session = SimpleNamespace(get=lambda url, headers=None: response)
track = SimpleNamespace(
id="vid",
url="https://cdn.example.invalid/media.m3u8",
descriptor=Track.Descriptor.HLS,
data={},
drm=None,
get_key_id=lambda init: KID,
)
handlers.ensure_track_drm(track, session)
assert track.drm and track.drm[0].kids == [KID]
@@ -616,6 +616,7 @@ def test_dl_selection_travels_nested_and_a_service_option_cannot_override_it() -
"a_lang": [],
"acodec": [Audio.Codec.EC3],
"forced_subs": True,
"exact_lang": True,
}
data = _sent_create_data(parent_params, {"lang": 3, "a_lang": "x"})
assert data["dl_params"] == {
@@ -624,6 +625,7 @@ def test_dl_selection_travels_nested_and_a_service_option_cannot_override_it() -
"a_lang": [],
"acodec": ["EC3"],
"forced_subs": True,
"exact_lang": True,
}
assert data["lang"] == 3
assert data["service_params"] == {"lang": 3, "a_lang": "x"}
+11 -6
View File
@@ -162,6 +162,9 @@ def _create_with_real_ctx(monkeypatch, data):
return svc
DL_DEFAULTS = {"lang": ["orig"], "v_lang": [], "a_lang": [], "acodec": [], "forced_subs": False, "exact_lang": False}
def test_nested_dl_params_reach_the_parent_ctx(monkeypatch):
from unshackle.core.tracks import Audio
@@ -172,6 +175,7 @@ def test_nested_dl_params_reach_the_parent_ctx(monkeypatch):
"a_lang": ["es-419"],
"acodec": ["EC3", "NOPE", "AAC"],
"forced_subs": True,
"exact_lang": True,
}
}
svc = _create_with_real_ctx(monkeypatch, data)
@@ -180,12 +184,13 @@ def test_nested_dl_params_reach_the_parent_ctx(monkeypatch):
assert svc.parent_params["a_lang"] == ["es-419"]
assert svc.parent_params["acodec"] == [Audio.Codec.EC3, Audio.Codec.AAC]
assert svc.parent_params["forced_subs"] is True
assert svc.parent_params["exact_lang"] is True
def test_absent_dl_params_fall_back_to_the_dl_defaults(monkeypatch):
svc = _create_with_real_ctx(monkeypatch, {})
selection = {k: svc.parent_params[k] for k in ("lang", "v_lang", "a_lang", "acodec", "forced_subs")}
assert selection == {"lang": ["orig"], "v_lang": [], "a_lang": [], "acodec": [], "forced_subs": False}
selection = {k: svc.parent_params[k] for k in (*DL_DEFAULTS,)}
assert selection == DL_DEFAULTS
def test_an_empty_lang_list_is_kept(monkeypatch):
@@ -204,14 +209,14 @@ def test_a_service_lang_option_never_gets_the_dl_lang(monkeypatch):
"dl_params",
[
"en",
{"lang": "en", "v_lang": "ja", "a_lang": {"x": 1}, "acodec": "EC3", "forced_subs": "yes"},
{"lang": [1], "v_lang": [None], "a_lang": ["en", None], "acodec": [3], "forced_subs": 1},
{"lang": "en", "v_lang": "ja", "a_lang": {"x": 1}, "acodec": "EC3", "forced_subs": "yes", "exact_lang": "yes"},
{"lang": [1], "v_lang": [None], "a_lang": ["en", None], "acodec": [3], "forced_subs": 1, "exact_lang": 1},
],
)
def test_malformed_dl_params_are_ignored(monkeypatch, dl_params):
svc = _create_with_real_ctx(monkeypatch, {"dl_params": dl_params})
selection = {k: svc.parent_params[k] for k in ("lang", "v_lang", "a_lang", "acodec", "forced_subs")}
assert selection == {"lang": ["orig"], "v_lang": [], "a_lang": [], "acodec": [], "forced_subs": False}
selection = {k: svc.parent_params[k] for k in (*DL_DEFAULTS,)}
assert selection == DL_DEFAULTS
def test_profile_reaches_the_parent_ctx_params(monkeypatch):
+1 -1
View File
@@ -268,7 +268,7 @@ def test_season_context_blanks_every_episode_identity_field():
def test_every_documented_variable_is_one_build_context_produces():
"""The variable tables in docs/reference/configuration/post-scripts.md must not drift from the code."""
from unshackle.core.titles.song import Song
from unshackle.core.titles.music import Song
doc = Path(__file__).resolve().parents[1] / "docs" / "reference" / "configuration" / "post-scripts.md"
documented = {
+3 -1
View File
@@ -5580,6 +5580,7 @@ class dl:
"""
Get CDM for a specified service (either Local or Remote CDM).
Now supports quality-based selection when the caller gives a quality.
With no ``drm``, an entry gives the DRM system that it lists first.
Raises a ValueError if there is a problem getting a CDM.
"""
# A per-request override (set by the REST API per job) takes precedence over the
@@ -5660,7 +5661,8 @@ class dl:
"pr": "playready",
"playready": "playready",
}.get(drm.lower())
cdm_name = lower_keys.get(drm_key or "widevine") or lower_keys.get("playready")
first = next(key for key in lower_keys if key in ("widevine", "playready"))
cdm_name = lower_keys.get(drm_key or first) or lower_keys.get("playready")
else:
cdm_name = cdm_name.get(profile) or cdm_name.get("default") or ci_get(config.cdm, "default")
if not cdm_name:
+1 -1
View File
@@ -7,7 +7,7 @@ from rich_click.patch import patch as _patch_click_help
_patch_click_help()
__version__ = "5.5.0"
__version__ = "6.0.0"
PKG = Path(__file__).parent.parent
# Framework code only. Services are user-swappable, so they are not part of the identity.
+1
View File
@@ -500,6 +500,7 @@ def perform_download(
"v_lang": params.get("v_lang", []),
"a_lang": params.get("a_lang", []),
"forced_subs": params.get("forced_subs", False),
"exact_lang": params.get("exact_lang", False),
}
# Hand-built context: record parameter sources so service dl overrides
# apply to defaults but never clobber client-sent values.
+11 -9
View File
@@ -31,6 +31,7 @@ from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_C
from unshackle.core.providers.anilist import parse_anilist_ref
from unshackle.core.proxies.resolve import initialize_proxy_providers, resolve_proxy
from unshackle.core.services import Services
from unshackle.core.session import utf8_text
from unshackle.core.titles import Episode, Movie, Song, Title_T
from unshackle.core.tracks import Audio, Subtitle, Tracks, Video
from unshackle.core.utilities import declared_kwargs, kind_only_conflict
@@ -156,7 +157,8 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]
drm_key = {"wv": "widevine", "widevine": "widevine", "pr": "playready", "playready": "playready"}.get(
cdm_type.lower()
)
cdm_name = lower_keys.get(drm_key or "widevine") or lower_keys.get("playready")
first = next(key for key in lower_keys if key in ("widevine", "playready"))
cdm_name = lower_keys.get(drm_key or first) or lower_keys.get("playready")
else:
cdm_name = cdm_name.get(profile) or cdm_name.get("default") or ci_get(app_config.cdm, "default")
@@ -3276,6 +3278,7 @@ def forwarded_dl_params(data: Dict[str, Any]) -> Dict[str, Any]:
except click.BadParameter:
log.warning(f"Ignoring unknown dl_params.acodec value: {sanitize_log(name)}")
params["forced_subs"] = raw.get("forced_subs") is True
params["exact_lang"] = raw.get("exact_lang") is True
return params
@@ -3913,7 +3916,8 @@ def resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[
Checks the server's own CDM config (``config.cdm[service]``) to
determine the CDM type without loading the full CDM object. This
makes sure that when you use ``server_cdm: true``, the server's CDM
determines device selection (e.g. PlayReady vs Widevine).
determines device selection (e.g. PlayReady vs Widevine). An entry with
``widevine`` and ``playready`` gives the DRM system that it lists first.
Falls back to a lightweight stub from *cdm_type* only if the configuration
has no server CDM for the service.
@@ -3925,7 +3929,8 @@ def resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[
if isinstance(cdm_name, dict):
lower_keys = {k.lower(): v for k, v in cdm_name.items()}
if {"widevine", "playready"} & lower_keys.keys():
cdm_name = lower_keys.get("playready") or lower_keys.get("widevine")
systems = [lower_keys[key] for key in lower_keys if key in ("widevine", "playready")]
cdm_name = next(filter(None, systems), None)
else:
cdm_name = cdm_name.get("default") or next(iter(cdm_name.values()), None)
@@ -4323,11 +4328,11 @@ def ensure_track_drm(track: Any, session: Any = None, init_data: Optional[bytes]
from unshackle.core.manifests import HLS
http = getattr(track, "session", None) or session
playlist = m3u8.loads(http.get(track.url).text, uri=track.url) if http else m3u8.load(track.url)
playlist = m3u8.loads(utf8_text(http.get(track.url)), uri=track.url) if http else m3u8.load(track.url)
keys = [k for k in (playlist.keys or []) + (playlist.session_keys or []) if k is not None]
for key in keys:
try:
drm = HLS.get_drm(key)
drm = HLS.get_drm_or_probe(key, playlist, track, probe_session=http)
if isinstance(drm, (Widevine, PlayReady)):
track.drm = [drm]
return
@@ -4803,14 +4808,11 @@ async def session_segment_filter_handler(
def unwanted_segments() -> list[str]:
import m3u8
import requests
response = session.service_instance.session.get(track.url)
if not getattr(response, "ok", True):
raise ValueError(f"playlist request failed: {response.status_code}")
if isinstance(response, requests.Response):
response.encoding = response.encoding or "utf-8"
playlist = m3u8.loads(response.text, uri=str(track.url))
playlist = m3u8.loads(utf8_text(response), uri=str(track.url))
return [segment.absolute_uri for segment in playlist.segments if segment_filter(segment)]
try:
+4 -1
View File
@@ -726,7 +726,7 @@ async def list_tracks(request: web.Request) -> web.Response:
dl_params:
type: object
description: |
The dl track selection (lang, v_lang, a_lang, acodec, forced_subs), in the same
The dl track selection (lang, v_lang, a_lang, acodec, forced_subs, exact_lang), in the same
form as on `/api/session/create`. The service reads it from `ctx.parent.params`.
credentials:
type: object
@@ -1831,6 +1831,9 @@ async def session_create(request: web.Request) -> web.Response:
forced_subs:
type: boolean
default: false
exact_lang:
type: boolean
default: false
server_cdm:
type: boolean
description: |
+2 -1
View File
@@ -25,6 +25,7 @@ from unshackle.core.config import config
from unshackle.core.console import console
from unshackle.core.constants import AnyTrack
from unshackle.core.drm import key_args as drm_key_args
from unshackle.core.session import utf8_text
from unshackle.core.utilities import get_boxes, log_event
from unshackle.core.utils.subprocess import ffprobe
@@ -180,7 +181,7 @@ class PlayReady:
if track.descriptor == track.Descriptor.HLS:
m3u_url = track.url
master = m3u8.loads(session.get(m3u_url).text, uri=m3u_url)
master = m3u8.loads(utf8_text(session.get(m3u_url)), uri=m3u_url)
pssh_boxes.extend(
Box.parse(base64.b64decode(x.uri.split(",")[-1]))
for x in (master.session_keys or master.keys)
+2 -1
View File
@@ -22,6 +22,7 @@ from unshackle.core.config import config
from unshackle.core.console import console
from unshackle.core.constants import AnyTrack
from unshackle.core.drm import key_args as drm_key_args
from unshackle.core.session import utf8_text
from unshackle.core.utilities import get_boxes, log_event
from unshackle.core.utils.subprocess import ffprobe
@@ -94,7 +95,7 @@ class Widevine:
if track.descriptor == track.Descriptor.HLS:
m3u_url = track.url
master = m3u8.loads(session.get(m3u_url).text, uri=m3u_url)
master = m3u8.loads(utf8_text(session.get(m3u_url)), uri=m3u_url)
pssh_boxes.extend(
Box.parse(base64.b64decode(x.uri.split(",")[-1]))
for x in (master.session_keys or master.keys)
+5 -3
View File
@@ -25,6 +25,7 @@ from unshackle.core.remote_service import (
resolve_proxy_arg,
segment_uri_filter,
)
from unshackle.core.session import utf8_text
from unshackle.core.titles import Episode, Movies, Series, Title_T, Titles_T, remap_titles
from unshackle.core.tracks import Audio, Chapter, Chapters, Subtitle, Tracks, Video
from unshackle.core.tracks.attachment import Attachment
@@ -67,9 +68,10 @@ def hls_playlist_kids(track: Any, session: Optional[requests.Session] = None) ->
key from its URI, so such a track takes no exported content key. An empty set means that
the track is encrypted but names no KID that unshackle can read.
"""
http = session or requests.Session()
try:
text = (session or requests.Session()).get(str(track.url)).text
keys = [k for k in m3u8.loads(text, uri=str(track.url)).keys if k is not None and k.method != "NONE"]
playlist = m3u8.loads(utf8_text(http.get(str(track.url))), uri=str(track.url))
keys = [k for k in playlist.keys if k is not None and k.method != "NONE"]
except Exception as e:
log.warning(f"Cannot read the media playlist of {track.id} for its KIDs: {e!r}")
return None
@@ -78,7 +80,7 @@ def hls_playlist_kids(track: Any, session: Optional[requests.Session] = None) ->
if key.method == "AES-128":
continue
try:
kids.update(real_kids(HLS.get_drm(key)))
kids.update(real_kids(HLS.get_drm_or_probe(key, playlist, track, probe_session=http)))
except Exception as e:
log.debug(f"Skipping HLS key {key.keyformat or key.method} of {track.id}: {e!r}")
if kids or any(k.method != "AES-128" for k in keys):
+9 -8
View File
@@ -19,7 +19,7 @@ from zlib import crc32
import requests
from langcodes import Language, tag_is_valid
from lxml.etree import Element, ElementTree
from lxml.etree import Element, ElementTree, XMLSyntaxError
from pyplayready.system.pssh import PSSH as PR_PSSH
from pywidevine.cdm import Cdm as WidevineCdm
from pywidevine.pssh import PSSH
@@ -118,6 +118,7 @@ class DASH:
@classmethod
def from_url(cls, url: str, session: Optional[Union[Session, RnetSession]] = None, **args: Any) -> DASH:
"""Fetch and parse an MPD. The XML declaration gives the encoding, then the HTTP charset."""
if not url:
raise requests.URLRequired("DASH manifest URL must be provided for relative path computations.")
if not isinstance(url, str):
@@ -138,11 +139,15 @@ class DASH:
log_event(
"manifest_dash_fetch",
level="DEBUG",
message=f"Fetched DASH manifest ({len(res.text)} bytes)",
context={"url": safe_display_url(url), "size": len(res.text)},
message=f"Fetched DASH manifest ({len(res.content)} bytes)",
context={"url": safe_display_url(url), "size": len(res.content)},
)
return DASH.from_text(res.text, url)
try:
return cls(load_xml(res.content), url)
except XMLSyntaxError:
# no declaration and not UTF-8: only the HTTP charset can name the encoding
return cls.from_text(res.text, url)
@classmethod
def from_text(cls, text: str, url: str) -> DASH:
@@ -1083,10 +1088,6 @@ class DASH:
return init_data, segments, segment_timescale, segment_durations, track_kid
# Deprecated 5.5.0 shims for service repos still on the old underscored names; drop once they migrate.
_is_content_period = is_content_period
_get_period_segments = get_period_segments
@staticmethod
def get_attr(item: str, adaptation_set: Element, representation: Optional[Element] = None) -> Optional[Any]:
"""Helper to get a requested item from the Representation, otherwise from the AdaptationSet."""
+77 -51
View File
@@ -37,7 +37,7 @@ from unshackle.core.drm.segment_decrypt import SegmentDecrypter, can_use
from unshackle.core.drm.verify import decrypt_track
from unshackle.core.events import events
from unshackle.core.manifests.dash import RollingMerge
from unshackle.core.session import RnetResponse, RnetSession
from unshackle.core.session import RnetResponse, RnetSession, utf8_text
from unshackle.core.tracks import Audio, DownloadContext, Subtitle, Tracks, Video, resume
from unshackle.core.tracks.track import DRM_PREFERENCE_TYPES, assert_fragments_decrypted
from unshackle.core.utilities import get_extension, is_close_match, log_event, try_ensure_utf8
@@ -81,17 +81,11 @@ class HLS:
res = session.get(url, **args)
if isinstance(res, requests.Response):
if not res.ok:
raise requests.ConnectionError("Failed to request the M3U(8) document.", response=res)
res.encoding = res.encoding or "utf-8"
content = res.text
elif isinstance(res, RnetResponse):
if not res.ok:
raise requests.ConnectionError("Failed to request the M3U(8) document.", response=res)
content = res.text
else:
if not isinstance(res, (requests.Response, RnetResponse)):
raise TypeError(f"Expected response to be a requests.Response or rnet.Response, not {type(res)}")
if not res.ok:
raise requests.ConnectionError("Failed to request the M3U(8) document.", response=res)
content = utf8_text(res)
master = m3u8.loads(content, uri=url)
@@ -397,9 +391,7 @@ class HLS:
session = Session()
res = session.get(variant_url)
if isinstance(res, requests.Response):
res.encoding = res.encoding or "utf-8"
variant = m3u8.loads(res.text, uri=variant_url)
variant = m3u8.loads(utf8_text(res), uri=variant_url)
if not variant.segments:
return None
@@ -659,17 +651,13 @@ class HLS:
master = m3u8.load(str(track.from_file))
else:
response = session.get(track.url)
if isinstance(response, requests.Response) or isinstance(response, RnetResponse):
if not response.ok:
log.error(f"Failed to request the invariant M3U8 playlist: {response.status_code}")
sys.exit(1)
if isinstance(response, requests.Response):
response.encoding = response.encoding or "utf-8"
playlist_text = response.text
else:
if not isinstance(response, (requests.Response, RnetResponse)):
raise TypeError(f"Expected response to be a requests.Response or rnet.Response, not {type(response)}")
if not response.ok:
log.error(f"Failed to request the invariant M3U8 playlist: {response.status_code}")
sys.exit(1)
master = m3u8.loads(playlist_text, uri=track.url)
master = m3u8.loads(utf8_text(response), uri=track.url)
if not master.segments:
log.error("Track's HLS playlist has no segments, expecting an invariant M3U8 playlist.")
@@ -706,9 +694,14 @@ class HLS:
media_playlist_key = HLS.get_supported_key(cdm_media_keys) if cdm_media_keys else None
if media_playlist_key:
media_drm = HLS.get_drm(media_playlist_key, session)
init_kid = (
None
if media_playlist_key.method == "AES-128"
else HLS.get_track_kid_from_init(master, track, session)
)
media_drm = HLS.get_drm(media_playlist_key, session, **({"kid": init_kid} if init_kid else {}))
if isinstance(media_drm, (Widevine, PlayReady)):
track_kid = HLS.get_track_kid_from_init(master, track, session) or media_drm.kid
track_kid = init_kid or media_drm.kid
# Preserve pre-existing keys (e.g. from server_cdm)
if track.drm:
for existing_drm in track.drm:
@@ -716,7 +709,7 @@ class HLS:
media_drm.content_keys.update(existing_drm.content_keys)
track.drm = [media_drm]
if DOWNLOAD_ALL_DRM.is_set():
track.drm += HLS.alternate_drm(media_keys, media_drm, session)
track.drm += HLS.alternate_drm(media_keys, media_drm, session, init_kid)
try:
if not license_widevine:
raise ValueError("license_widevine func must be supplied to use DRM")
@@ -772,7 +765,7 @@ class HLS:
if key is None or key.method == "AES-128" or key in licensed_keys:
continue
licensed_keys.append(key)
later_drm = HLS.get_drm(key, session)
later_drm = HLS.get_drm_or_probe(key, master, track, session, segment.init_section)
if isinstance(later_drm, (Widevine, PlayReady)) and license_widevine:
license_widevine(later_drm, track_kid=later_drm.kid)
progress(downloaded="[yellow]SKIPPED")
@@ -1152,22 +1145,18 @@ class HLS:
if key is None:
encryption_data = None
elif not encryption_data or encryption_data[0] != key:
drm = HLS.get_drm(key, session)
if isinstance(drm, (Widevine, PlayReady)):
try:
if map_data:
track_kid = track.get_key_id(map_data[1])
else:
track_kid = None
if not track_kid:
track_kid = drm.kid
try:
init_kid = track.get_key_id(map_data[1]) if map_data and key.method != "AES-128" else None
drm = HLS.get_drm(key, session, **({"kid": init_kid} if init_kid else {}))
if isinstance(drm, (Widevine, PlayReady)):
track_kid = init_kid or drm.kid
progress(downloaded="LICENSING")
license_widevine(drm, track_kid=track_kid)
progress(downloaded="[yellow]LICENSED")
except Exception: # noqa
DOWNLOAD_CANCELLED.set() # skip pending track downloads
progress(downloaded="[red]FAILED")
raise
except Exception: # noqa
DOWNLOAD_CANCELLED.set() # skip pending track downloads
progress(downloaded="[red]FAILED")
raise
if (
encryption_data
and isinstance(drm, (Widevine, PlayReady))
@@ -1384,8 +1373,7 @@ class HLS:
if not value and data.uri:
if not session:
session = Session()
res = session.get(urljoin(manifest.base_uri or "", data.uri))
value = res.text
value = utf8_text(session.get(urljoin(manifest.base_uri or "", data.uri)))
if not value:
continue
@@ -1462,6 +1450,7 @@ class HLS:
keys: list[Union[m3u8.model.SessionKey, m3u8.model.Key]],
chosen: DRM_T,
session: Optional[Union[Session, RnetSession]] = None,
kid: Optional[UUID] = None,
) -> list[DRM_T]:
"""Build the Widevine or PlayReady DRM that the EXT-X-KEY entries offer next to ``chosen``.
@@ -1472,7 +1461,7 @@ class HLS:
if not other_keys:
return []
try:
return [HLS.get_drm(other_keys[0], session)]
return [HLS.get_drm(other_keys[0], session, **({"kid": kid} if kid else {}))]
except Exception as e:
logging.getLogger("HLS").warning(f"Could not read the {system} EXT-X-KEY entry: {e}")
return []
@@ -1482,14 +1471,15 @@ class HLS:
master: M3U8,
track: AnyTrack,
session: Union[Session, RnetSession],
init_section: Optional[m3u8.model.InitializationSection] = None,
) -> Optional[UUID]:
"""
Extract the track's Key ID from its init segment (EXT-X-MAP).
Extract the track's Key ID from an init segment (EXT-X-MAP), by default the first in the playlist.
Returns None if no init segment exists or KID extraction fails.
The caller should fall back to drm.kid from the PSSH if this returns None.
"""
map_section = next((seg.init_section for seg in master.segments if seg.init_section), None)
map_section = init_section or next((seg.init_section for seg in master.segments if seg.init_section), None)
if not map_section:
return None
@@ -1503,6 +1493,10 @@ class HLS:
map_res = session.get(url=map_uri, headers=headers)
if map_res.ok:
return track.get_key_id(map_res.content)
logging.getLogger("HLS").warning(
f"Could not read the Key ID from the init segment {safe_display_url(map_uri)}: "
f"HTTP {map_res.status_code}"
)
# best-effort key-id probe over network + init parse; caller falls back without it
except Exception as e:
logging.getLogger("HLS").debug(f"Init map key-id probe failed for {map_uri}: {e!r}")
@@ -1550,6 +1544,7 @@ class HLS:
def get_drm(
key: Union[m3u8.model.SessionKey, m3u8.model.Key],
session: Optional[Union[Session, RnetSession]] = None,
kid: Optional[UUID] = None,
) -> DRM_T:
"""
Convert HLS EXT-X-KEY data to an initialized DRM object.
@@ -1558,6 +1553,8 @@ class HLS:
key: The m3u8 `EXT-X-KEY` object.
session: Optional HTTP session used to request AES-128 URIs.
Useful to set headers, proxies, cookies, and so forth.
kid: The Key ID from the init segment. Widevine uses it only when the tag
and the PSSH both carry no real Key ID.
Raises a NotImplementedError when unshackle cannot use the DRM system.
"""
@@ -1573,13 +1570,16 @@ class HLS:
drm = Widevine(pssh=WV_PSSH.new(key_ids=[key.uri.split(",")[-1]], system_id=WV_PSSH.SystemId.Widevine))
elif key.keyformat and key.keyformat.lower() == WidevineCdm.urn:
extra_params = dict(key._extra_params) # noqa
# a v0 PSSH carries no KID, leaving the tag's own KEYID as the only source
kid = extra_params.pop("keyid", None) or extra_params.pop("kid", None)
if isinstance(kid, str):
kid = kid.removeprefix("0x").removeprefix("0X")
pssh = WV_PSSH(key.uri.split(",")[-1])
if kid in Widevine.PLACEHOLDER_KIDS or any(k not in Widevine.PLACEHOLDER_KIDS for k in pssh.key_ids):
kid = None
# a v0 PSSH can carry no KID: the tag's own KEYID is the first source, the init segment the last
tag_kid = extra_params.pop("keyid", None) or extra_params.pop("kid", None)
if isinstance(tag_kid, str):
tag_kid = tag_kid.removeprefix("0x").removeprefix("0X")
drm = Widevine(
pssh=WV_PSSH(key.uri.split(",")[-1]),
kid=kid,
pssh=pssh,
kid=tag_kid or kid,
**extra_params,
)
elif key.keyformat and key.keyformat.lower() in {f"urn:uuid:{PR_PSSH.SYSTEM_ID}", "com.microsoft.playready"}:
@@ -1592,6 +1592,28 @@ class HLS:
return drm
@staticmethod
def get_drm_or_probe(
key: Union[m3u8.model.SessionKey, m3u8.model.Key],
master: M3U8,
track: AnyTrack,
session: Optional[Union[Session, RnetSession]] = None,
init_section: Optional[m3u8.model.InitializationSection] = None,
probe_session: Any = None,
) -> DRM_T:
"""
Convert HLS EXT-X-KEY data to a DRM object, with the Key ID of the init segment when the key carries none.
The init segment request uses `probe_session` when the caller has no HTTP session that `get_drm` accepts.
"""
try:
return HLS.get_drm(key, session)
except Widevine.Exceptions.KIDNotFound:
kid = HLS.get_track_kid_from_init(master, track, probe_session or session or Session(), init_section)
if not kid:
raise
return HLS.get_drm(key, session, kid=kid)
@staticmethod
def get_all_drm(
keys: list[Union[m3u8.model.SessionKey, m3u8.model.Key]], proxy: Optional[str] = None
@@ -1617,6 +1639,10 @@ class HLS:
drm_objects.append(drm)
except NotImplementedError:
unsupported_keys.append(key)
except Widevine.Exceptions.KIDNotFound:
logging.getLogger("HLS").warning(
"Skipped an EXT-X-SESSION-KEY entry that carries no Key ID. The media playlist must supply the key."
)
if not drm_objects and unsupported_keys:
logging.debug(
+3 -2
View File
@@ -1073,8 +1073,9 @@ class RemoteService:
}
if params.get("acodec") is not None:
dl_params["acodec"] = [c.name for c in params["acodec"]]
if "forced_subs" in params:
dl_params["forced_subs"] = bool(params["forced_subs"])
for key in ("forced_subs", "exact_lang"):
if key in params:
dl_params[key] = bool(params[key])
if dl_params:
create_data["dl_params"] = dl_params
-3
View File
@@ -406,9 +406,6 @@ class Service(metaclass=ABCMeta):
return all_tracks
# Deprecated 5.5.0 shim for service repos still on the old underscored name; drop once they migrate.
_get_tracks_for_variants = get_tracks_for_variants
@staticmethod
def get_session() -> requests.Session:
"""
+13 -1
View File
@@ -15,7 +15,7 @@ from typing import Any, Optional
from urllib.parse import urlencode, urlparse, urlunparse
import rnet
from requests import HTTPError, Request
from requests import HTTPError, Request, Response
from requests.structures import CaseInsensitiveDict
from unshackle.core.config import config
@@ -227,6 +227,18 @@ class RnetResponse:
pass
def utf8_text(res: Response | RnetResponse) -> str:
"""Return the response text, decoded as UTF-8 unless the server sent a charset.
With no charset, requests assumes ISO-8859-1 for a `text/*` type and runs charset detection
on the full body for any other type. Use this only for a format that is UTF-8 by
specification, such as M3U8.
"""
if isinstance(res, Response) and "charset" not in res.headers.get("content-type", "").lower():
res.encoding = "utf-8"
return res.text
class RnetSessionHeaders(CaseInsensitiveDict):
"""Dict-like headers that write to the rnet client through update()."""
-5
View File
@@ -1,5 +0,0 @@
"""Deprecated. Import from unshackle.core.titles.music instead."""
from unshackle.core.titles.music import Album, Song # noqa: F401
__all__ = ("Song", "Album")
+3 -5
View File
@@ -20,7 +20,7 @@ import click
from construct import ConstructError
from langcodes import Language
from pymp4.parser import Box
from requests import Response, Session
from requests import Session
from requests.adapters import HTTPAdapter, Retry
from unshackle.core import binaries
@@ -31,7 +31,7 @@ from unshackle.core.downloaders import requests
from unshackle.core.drm import DRM_T, ClearKeyCENC, PlayReady, Widevine
from unshackle.core.drm.verify import decrypt_track
from unshackle.core.events import events
from unshackle.core.session import RnetSession
from unshackle.core.session import RnetSession, utf8_text
from unshackle.core.tracks import resume
from unshackle.core.utilities import get_boxes, log_event, try_ensure_utf8
from unshackle.core.utils.subprocess import ffprobe
@@ -1141,9 +1141,7 @@ class Track:
session = getattr(self, "session", None) or Session()
response = session.get(self.url)
if isinstance(response, Response):
response.encoding = response.encoding or "utf-8"
playlist = m3u8.loads(response.text, self.url)
playlist = m3u8.loads(utf8_text(response), self.url)
drm_list = []
+2 -2
View File
@@ -13,7 +13,7 @@ from urllib.parse import urljoin
from requests import Session
from unshackle.core.binaries import FFProbe
from unshackle.core.session import RnetSession
from unshackle.core.session import RnetSession, utf8_text
from unshackle.core.utils.subprocess import log_tool_run
if TYPE_CHECKING:
@@ -320,7 +320,7 @@ def extract_hls(track: "Track", session: Union[Session, RnetSession]) -> list[Se
playlist_url = track.url if isinstance(track.url, str) else track.url[0]
res = session.get(playlist_url, timeout=PROBE_TIMEOUT)
playlist = m3u8.loads(res.text, uri=playlist_url)
playlist = m3u8.loads(utf8_text(res), uri=playlist_url)
out: list[Segment] = []
for segment in playlist.segments:
Generated
+1 -1
View File
@@ -2139,7 +2139,7 @@ wheels = [
[[package]]
name = "unshackle"
version = "5.5.0"
version = "6.0.0"
source = { editable = "." }
dependencies = [
{ name = "aiohttp" },