Implements Apple HLS SAMPLE-AES decryption for MPEG-TS elementary streams, which neither Shaka Packager (rejects stream type 0xDB) nor mp4decrypt (ISO-BMFF only) can handle. Covers H.264 (1:9 pattern from offset 32 with EPB strip/reinsert), AAC and AC-3, then remuxes to a clean TS with the PMT stream types patched.
The FairPlay->PlayReady bridge synthesized headers and routed licensing but the HLS download loop still rejected the skd EXT-X-KEY. Teach get_supported_key and get_drm the com.apple.streamingkeydelivery keyformat, and reuse a service-provided FairPlay session DRM for skd segments (its KID encoding is service-specific, e.g. base64).
FairPlay HLS (SAMPLE-AES/cbcs) ships a content KID in its skd:// key but no PlayReady/Widevine PSSH. Synthesize a PlayReady header from that KID so a PlayReady CDM can license and decrypt it whenever the backend is multi-DRM.
- FairPlay DRM (PlayReady subclass) + FairPlay.from_kid; fairplay_kid_from_skd extracts the KID from GUID / ?KID= / 32-hex skd forms (core/drm/fairplay.py)
- build_pr_header_from_kid synthesizes a WRMHEADER/PRO (cbcs->AESCBC v4.3, cenc->AESCTR v4.0); PlayReady.from_track falls back to a tenc KID when no PSSH
- Service.get_fairplay_license hook (defaults to get_playready_license)
- dl.py routes FairPlay tracks to get_fairplay_license through the PlayReady CDM
- EXAMPLE service + config.yaml document the bridge end to end
- tests for the synthesized header and skd KID extraction
Group selected videos by (resolution, range, codec) and mux each group into one MKV; only language is collapsed, so ranges/codecs/resolutions stay in separate files. Adds --merge-video flag + muxing.merge_video config (global and per-service), docs, and tests.
* Add native Music core workflow
* feat(music): add shared music core helpers, album folder template, and UX cleanup
Consolidate duplicated music-service logic into core and extend the native music workflow.
- add core/music/extract.py: shared stateless helpers (first_text, first_number, year/duration/name formatting, classify_release_kind, dedupe_track_options, build_music_from_songs) so services stop carrying their own copies
- add core/music/display.py: shared rich rendering (render_track_panel, render_album_header, render_artwork_preview) with TrackRow / MusicHeaderInfo data holders
- export the new helpers from core/music/__init__.py
- add dedicated `albums` folder template kind (output_template.folder.albums) resolving albums -> songs -> "{artist} - {album} ({year})"; whitelist music template variables (album_artist, track_total, disc_total, release_type, genre, explicit, isrc, upc, label)
- fix song filename crash: config.get_output_template(...) did not exist; use config.output_template.get("songs") with a sane default
- strip emojis from music output (renderer, dl.py music branch) to match unshackle UX; remove dead MusicSongPlan import and music_icon logic
- document the albums folder key in unshackle-example.yaml
- add tests for extract, display, and the folder template
---------
Co-authored-by: MrMovies-Dev <MrMovies-Dev@users.noreply.github.com>
Smooth Streaming manifests carry no range attributes, so every ISM video track was labelled SDR even when the stream is HDR10/HLG/DV, breaking range-based selection (-r HDR10 / -r DV) for ISM services.
- ism_init: walk the full HEVC SPS (incl. scaling-list and st_ref_pic_set skippers) to read the VUI colour triple (colour_primaries, transfer_characteristics, matrix_coeffs); expose parse_codec_private_data_colour() keyed by FourCC. No unshackle imports added.
- ism: new ISM.get_video_range() maps the CICP triple via Video.Range.from_cicp (PQ -> HDR10, HLG -> HLG, BT.709/absent -> SDR); DVHE/DVH1 FourCCs map straight to DV since DV bitstreams signal Unspecified (2,2,2) in the VUI. to_tracks() now sets range_ on every video track. Soft-fails to SDR on malformed data.
- ism: accept RnetSession in download_track() so TLS-impersonated sessions pass the type check.
- tests: real PQ/HLG/BT.709 (x265-minted) and Dolby Vision (live-manifest, DoViProfile=stn, out-of-order SPS,PPS,VPS NALs) CodecPrivateData samples; byte-level VUI assertions in test_ism_init and manifest->Range characterization in new test_ism_range.
ISM (Smooth Streaming) tracks raw-concatenate moof+mdat fragments with no ftyp/moov, so shaka-packager/mp4decrypt fail with PARSER_FAILURE (exit 2) on decrypt. The init box was previously built by n_m3u8dl_re, removed in the downloader consolidation.
Add ism_init.py, a dependency-free byte-level MP4 init-segment synthesizer that rebuilds ftyp+moov from the manifest CodecPrivateData, ported from yt-dlp's write_piff_header and N_m3u8DL-RE's MSSMoovProcessor with full codec parity:
- AVC (H264/AVC1/DAVC), with SPS/PPS picked by NAL type rather than position and NALUnitLengthField honored
- HEVC (HVC1/HEV1) with chroma format and bit depths parsed from the de-emulated SPS via exp-Golomb so 10-bit/HDR signals correctly, and profile/tier/level lifted from the SPS PTL
- Dolby Vision (DVHE/DVH1) as hvcC with a dvh1 sample entry
- AAC (AACL/AACH) with the AudioSpecificConfig synthesized from SamplingRate/Channels when the manifest omits CodecPrivateData
- EC-3 with a real dec3 box extracted from the WAVEFORMATEXTENSIBLE CodecPrivateData (Dolby GUID located by search, not fixed offset)
- TTML subtitles as stpp/sthd/subt, wired for fragmented-TTML tracks
CENC wrapping (encv/enca + sinf/tenc with default_KID) covers encrypted tracks: the per-sample IV size is derived from the fragment senc/saiz (PIFF override flag, payload arithmetic, saiz fallback) instead of assuming 8, and the constant-IV tenc form is supported. Read the track_ID from the first fragment's tfhd so the moov matches and the muxer does not drop samples.
Wire ISM.download_track to prepend the synthesized init before merging; unsupported codecs soft-fail to raw concatenation with a warning. Harden against real-world inputs: 2-letter/uppercase manifest language tags normalize to ISO-639-2 (und fallback), >65535 Hz sample rates no longer overflow the 16.16 field, truncated tfhd returns None, struct.error joins the soft-fail handler, and the emulation-prevention scan no longer over-strips consecutive escapes.
Add regression tests (37) covering box structure, every supported FourCC, 10-bit SPS parsing, ASC synthesis, dec3 extraction, IV-size derivation and the crash fixes. Validated structurally per codec with ffmpeg-minted fragments: shaka-packager parses synth-init+fragments with exit 0 and ffprobe reports the expected codec, including a live run against a public Smooth Streaming server.
unshackle's DASH parser only recognised Widevine and PlayReady ContentProtection, so services using W3C EME ClearKey had to fake a Widevine object and monkey-patch get_content_keys. Add a first-class ClearKeyCENC DRM type so services just implement a license callback.
- ClearKeyCENC (core/drm/clearkey_cenc.py): KID-based, no CDM/PSSH; builds the W3C JSON license request (unpadded base64url), parses the JWK Set response (dict/str/bytes), falls back to POSTing the manifest Laurl when the service returns None, decrypts via the same shaka/ mp4decrypt CENC path as Widevine
- DASH.get_drm emits ClearKeyCENC for scheme e2719d58-...; KID from own or sibling mp4protection cenc:default_KID, Laurl across dashif/legacy/ bare namespaces
- track.download dispatches prepare_drm for ClearKeyCENC; dl.prepare_drm gains a clearkey branch (cache/vault lookup, license-failure tolerated when content_keys pre-populated, vault push, export)
- Service.get_clearkey_license base callback (default None -> Laurl); drm_from_dict reconstructs ClearKeyCENC for export/import round-trip
- EXAMPLE service + config demo the callback
- Tests: tests/core/test_clearkey_cenc.py and an export round-trip case
- Docs: DRM_CONFIG.md ClearKey section
Add version-bump.yml: daily git-cliff semver bump on dev (version files, uv.lock, prepended CHANGELOG entry, tag). Switch release.yml gate to compare against latest GitHub release instead of latest tag, since every bump is now tagged. Allow .github/ through the hidden-dir gitignore rule.
Instrument the full download pipeline in the structured JSONL debug logger and make adding logging to new features a one-liner.
Coverage:
- DRM license request/response, content keys (incl. remote-CDM seam) and decrypt timing across Widevine/PlayReady/ClearKey
- DASH/HLS/ISM manifest fetch + parse milestones (HLS.to_tracks also covers the m3u8_parser path used by iTunes/ATV-style services)
- Per-backend vault get/add via the Vaults manager, track selection, subtitle conversion, repackage, normalize_vui, and full mkvmerge mux (command, duration, output size, warnings)
- All external tooling (ffmpeg, ffprobe, mkvmerge, mkvpropedit, dovi_tool, SubtitleEdit, ccextractor) via a unified `tool_run` op, centralised in run_step/ffprobe + log_tool_run
DX:
- Add log_event() / timed_operation() primitives (no-op when disabled); retrofit ~91 guard/timing blocks onto them
- Fix message= collision in log_drm_operation/log_vault_query/log_service_call that raised TypeError on the live decrypt path
Redaction (redact_all = redact_text -> redact_url -> redact_path):
- Collapse content/CDN/api URLs to `redacted[.ext]`
- Strip local path prefixes (install root -> <unshackle>, venv -> <venv>, home -> ~)
- Apply to every logged string so shared logs leak no URLs, paths or usernames
- Drop per-request service_call logging (manifest parse is the request seam)
services.<TAG>.dl values only applied when the key was also set in the global dl: section (equality check against config.dl missed Click's declared defaults). Gate on Click's ParameterSource instead: CLI/env > service dl > global dl > defaults, converted via each option's own type.
- record parameter sources on serve's hand-built context so client values are never clobbered
- accept range/list as natural keys for range_/list_
- harden QualityList (YAML int) and SlowDelayRange (YAML bool) converts
Replace the class-level Track.download monkeypatch with a per-job progress sink threaded through dl.result(). The API now reports a single aggregate signal instead of each track's bouncing 0-100%:
- bitrate-weighted completion so video/audio dominate subtitles
- completed_tracks/total_tracks counts and active_tracks labels (e.g. "video 2160p DV", "audio en-US 5.1")
- downloads fill 0-90%; repackaging (when needed) and a "muxing" stage carry it to 100% so post-download work is no longer frozen at 100%
- monotonic throughout (handles the download->decrypt callable reuse)
Also:
- accept "HDR10P" as the canonical API range value ("HDR10+" still works)
- declare AUTH_METHODS opt-in on the Service base
- raise typed APIError (WORKER_ERROR/DOWNLOAD_ERROR) from the worker path
- move the progress helpers to unshackle/core/api/progress.py
* feat(api): live download phase, granular progress, swallowed-failure detection, per-request CDM
- Tee Track.download progress so the job reports real percentage (not just 5/100%) and a
human-readable phase ('downloading video 1080p') via the new job.phase field.
- Detect a swallowed download-worker failure (dl.result() prints 'Download Failed' but
exits 0) and raise, so the job is marked failed instead of completed-with-no-output.
- Per-request CDM override (dl_instance.cdm_override; get_cdm prefers it) so a job can use
a specific CDM device without mutating shared config.
* feat(api): expose service capability flags + auth methods in /services
needs_auth / has_search / has_drm derived from which Service hooks are overridden, and
auth_methods inferred from what the service's authenticate() body references (cookies /
credentials), so clients can show only the relevant auth options per service.
* feat(api): per-request credential injection for downloads
Accept a 'credential' ('user:pass') job parameter and feed it into the credentials map that
dl.get_credentials() reads, so a client can authenticate a download without persisting
anything to disk. (Kept on the deployment branch; the PR branch uses the client-sent path.)
* feat(api): gate per-request CDM override behind serve.cdm_overrides
A per-request `cdm` selects a server-side device, so honour it only when allow-listed.
`serve.cdm_overrides` opts in: a list permits those device names, or `true` permits any
(single trusted client). Unset/false rejects every override with 403, so an arbitrary device
can't be selected by default.
* fix(api): redact credentials and proxy userinfo in serialized job parameters
Job parameters can carry a raw user:pass credential and a proxy URL with embedded userinfo;
mask them wherever parameters are serialized for an API response so secrets don't leak via
the job-detail endpoint. Also read skipped-subtitle / download-failure state from the dl
instance instead of scraping stdout, and drop the dead n_m3u8dl percent branch.
* feat(api): prefer explicit AUTH_METHODS class var over source inference in /services
Inferring auth methods from authenticate() source mostly returns both options because services
call super().authenticate(cookies, credential). Prefer an explicit AUTH_METHODS class var when a
service declares one, falling back to inference.
* style: use plain hyphens instead of em-dashes in comments
* feat(api): gate per-job credentials, isolate caches, scrub error fields
Address review feedback on #113:
- Gate per-request credential/credentials behind serve.allow_job_credentials (default off,
403 when not opted in), mirroring the existing serve.cdm_overrides CDM gate.
- Isolate the token cache per credential: when a per-job credential is set, namespace
config.directories.cache by its hash in the worker, so two clients on the same service
with different credentials can't share a cached session.
- Scrub the credential, its password half, and proxy userinfo out of error_message,
error_details, error_traceback and worker_stderr before they leave via the job-detail API.
- Remove the unused _execute_download_sync in-process path (would have leaked one job's
credential into the shared global config).
- Document serve.cdm_overrides and serve.allow_job_credentials in the example config.
- Add tests for both gates (403 default, allowlist pass) and the parameter/error redaction.
* fix(dl): flag download_failed when result() swallows a worker failure
dl.result() catches a download-worker exception, reports it, and returns
without re-raising so the CLI still exits cleanly. An embedding caller (the
API worker) had no way to tell the title actually failed and would report
the job completed with no output. Expose a download_failed flag, set in the
swallow path, that the worker reads after result() returns.
* feat(api): surface skipped subtitles and pass skip_subtitle_errors
Thread skip_subtitle_errors from the job into dl() so the API can opt into
non-fatal subtitle handling, and report which subtitles were skipped: store
them on the job (dl.SkippedSubtitle dicts) and include them in job details
so a client can tell the user which weren't available.
---------
Co-authored-by: Avi Cohen <avraham.coh770@gmail.com>
Replace the hardcoded conversion if/elif in Subtitle.convert with a capability-matrix backend registry (subtitle_convert.py): each backend declares the source->target pairs it supports plus a rank, and run_conversion tries them in order as a real fallback chain. conversion_method pins a backend but still falls back (pin-then-fallback).
- Detect the cross-platform SubtitleEdit 5+ CLI (seconv) and use its --flag syntax for convert, SDH stripping, and reverse-RTL
- Protect styled ASS/SSA from automatic SRT downconversion; honor an explicit --sub-format / sidecar_format
- Read segmented fVTT (wvtt) and fTTML (stpp) directly from fragmented MP4
- Improve ASS/SSA font detection: inline \fn overrides, Format-located Fontname column, @-prefix strip, case-insensitive de-dup; covers SSA too
- Update SUBTITLE_CONFIG.md, example yaml, README; add regression tests and a backend benchmark script
Lets sibling services read/write one key-vault namespace instead of being tied to their own tag. New Service.VAULT_TAG class var, resolved via Services.get_vault_tag() in both the dl and serve paths. AMZN_WEB opts into the AMZN namespace; EXAMPLE documents usage.
-r HYBRID alone muxed the HDR10/HDR10+ base layer as a standalone output because only the ingredient DV track was flagged hybrid_base_only. The inverse was also broken: HDR10/HDR10+/DV tracks never entered the standalone-deliverable pool, so -r HYBRID,HDR10P only delivered the standalone HDR10+ by accident of the first bug.
- Add Tracks.partition_hybrid_videos: ingredient ranges (HDR10/HDR10+/DV) enter the deliverable pool only when their range is explicitly requested alongside HYBRID; replaces the duplicated filter in dl.py.
- Add Tracks.flag_hybrid_ingredients: any track in the hybrid selection but not in the deliverable selection is flagged hybrid_base_only; replaces and generalises the DV-only dv_is_deliverable special case.
Addresses review feedback on the failed-subtitle handling:
- Fully annotate download_tracks_in_passes and the download_track closure (MyPy strict).
- Replace list[str] skipped_subtitles with a documented SkippedSubtitle TypedDict
(id + language + title) so a client can report which subtitle of which title was
skipped; pin the shape with a contract test.
- Clear DOWNLOAD_CANCELLED in a finally so no failed track leaves it set for later code.
- Document why the subtitle pass must stay sequential (a concurrent pass would silently
drop in-flight subtitles via the cancel event).
- Warn only when a title skipped a subtitle and produced no video/audio/subtitle (was a
loose len(title.tracks) check that ignored chapters/attachments).
- Narrow the over-broad remove() except to ValueError with a debug log.
- Add tests: final-clear on the fatal path, all-subs-skipped keeps video/audio, and
duplicate-language subtitles distinguished by id.
ast.Num/node.n deprecated since Python 3.8; NodeVisitor falls back to visit_Num with a DeprecationWarning per dispatch. Under -W error this surfaced as a misleading fps ValueError in Video.init. Replace with visit_Constant, reject non-numeric constants, and pin parse results for int/fraction/float inputs in a regression test.
MultiCommand is deprecated since Click 8.3 and removed in 9.0; its functionality was folded into Group, which supports the same list_commands/get_command lazy-loading overrides. Unblocks a future Click 9 bump and silences the import-time DeprecationWarning that broke test collection under -W error::DeprecationWarning.
A single failing subtitle track previously aborted the whole download. Add an opt-in
--skip-subtitle-errors flag: when set, a Subtitle failure is logged and the track dropped from
the mux while the video/audio still complete (Video/Audio failures stay fatal; default
behaviour is unchanged).
Done at the right layer to avoid the shared-event race: a failed track sets the process-global
DOWNLOAD_CANCELLED event, which makes other in-flight tracks early-return without raising — so a
skipped subtitle could otherwise silently truncate the video/audio that still got muxed. The
download is split into two passes (download_tracks_in_passes): the fatal tracks download
concurrently first, then the skippable subtitles run in a separate sequential pass once nothing
else is in flight, with the event reset before each and at the start of every title. Skipped
languages are recorded on the dl instance (skipped_subtitles) for callers to surface.
Adds tests for the cancel-event interaction (a failing subtitle no longer truncates the
video/audio), the good/bad subtitle mix, the flag-off fatal path, and the per-title reset.
The download path only wired up media-playlist keys when they were Widevine/PlayReady; an
AES-128 ClearKey was built but never assigned to session_drm, so segments were merged still
encrypted and the output was unplayable. Handle ClearKey from the media playlist (no license
needed), and when EXT-X-KEY has no explicit IV use each segment's media sequence number as
the IV (RFC 8216 5.2).
Drop the manually-set Proxy-Authorization header in the Service base class. It was malformed (base64 of user:pass with no "Basic" scheme) and redundant: both rnet (Proxy.all) and requests authenticate from the credentials embedded in the proxy URL. The broken header was tunnelled harmlessly on HTTPS (CONNECT) but handed to the proxy on plaintext-http forward requests, causing HTTP 407 (e.g. http MPD/segment URLs behind an authenticated geofence proxy).
Also make RnetSession._build_url accept the same params shapes as requests (mapping, sequence of pairs, or a pre-built str/bytes query). urlencode() previously raised TypeError on a string params value.
The legacy aria2c, curl-impersonate, and N_m3u8DL-RE downloader backends were replaced by the unified in-process requests/rnet downloader. This strips the leftover references those removals missed.
Make the DV-ingredient marker a declared Video attribute (sibling of needs_duration_fix) instead of a runtime-stamped attribute read via getattr, so the flag is explicit and type-checked.
Extract mux_video_standalone() to replace the duplicated per-track mux logic shared by the hybrid and normal-mode paths, removing the copy-paste and the getattr probes. Behavior unchanged.
When -r included HYBRID, the dl pipeline only muxed the hybrid output and dropped every other downloaded range. DV was also treated purely as a hybrid ingredient pool, so an explicitly requested DV range never produced a standalone deliverable - only the lowest DV (the RPU ingredient) was selected.
- Select the best DV per resolution as a standalone deliverable when DV is in -r, while still using the lowest DV as the hybrid ingredient (dv_is_deliverable).
- Flag the ingredient-only DV (hybrid_base_only) so it is downloaded for the hybrid build but skipped during standalone muxing.
- Mux every requested range standalone after hybrid processing; build hybrids from deepcopied ingredients so the originals stay muxable.
- Add Tracks.merge_video_selections to de-dup the ingredient/deliverable overlap so a shared DV track is not downloaded or muxed twice.
Cache only country/country_code (drop full IP/org/asn), bump CACHE_KEY and auto-purge stale cache versions. Dedup the three provider parsers into one normalize(). Use a plain retry-free requests session for lookups instead of the TLS-fingerprinted session, carrying only the proxy over, so a 429 returns directly and hands over to the next provider reliably.
A SegmentTemplate can sit at both the AdaptationSet and Representation levels, with shared attrs like @timescale only on the outer node. The parser picked one or the other, dropping the outer node when an inner existed - so @timescale defaulted to 1 and only the first segment was emitted.
Merge instead: Representation node as base, inheriting any attribute or SegmentTimeline it omits from the AdaptationSet node (per ISO/IEC 23009-1).
Export URL-backed attachments per title and rebuild them on import so they ride along into the muxed output. Local font attachments are skipped (not portable).
