Commit Graph

618 Commits

Author SHA1 Message Date
imSp4rky
e6613e8ed8 Merge branch 'dev' of https://github.com/unshackle-dl/unshackle into dev 2026-06-07 16:00:43 -06:00
imSp4rky
ce0d9d8355 feat(vault): add VAULT_TAG to share key vault across services
Lets sibling services read/write one key-vault namespace instead of being tied to their own tag. New Service.VAULT_TAG class var, resolved via Services.get_vault_tag() in both the dl and serve paths. AMZN_WEB opts into the AMZN namespace; EXAMPLE documents usage.
2026-06-07 15:59:59 -06:00
sp4rk.y
27c8fcdae1 Merge pull request #116 from avi-dev-user/subtitle-nonfatal
fix(dl): make a failed subtitle non-fatal under --skip-subtitle-errors
2026-06-07 12:56:48 -06:00
imSp4rky
4a543c59e6 fix(dl): mux hybrid ingredients standalone only when range explicitly requested
-r HYBRID alone muxed the HDR10/HDR10+ base layer as a standalone output because only the ingredient DV track was flagged hybrid_base_only. The inverse was also broken: HDR10/HDR10+/DV tracks never entered the standalone-deliverable pool, so -r HYBRID,HDR10P only delivered the standalone HDR10+ by accident of the first bug.

- Add Tracks.partition_hybrid_videos: ingredient ranges (HDR10/HDR10+/DV) enter the deliverable pool only when their range is explicitly requested alongside HYBRID; replaces the duplicated filter in dl.py.
- Add Tracks.flag_hybrid_ingredients: any track in the hybrid selection but not in the deliverable selection is flagged hybrid_base_only; replaces and generalises the DV-only dv_is_deliverable special case.
2026-06-06 16:01:54 -06:00
Avi Cohen
4fdffc9435 refactor(dl): type the two-pass download helper + pin skipped_subtitles shape
Addresses review feedback on the failed-subtitle handling:
- Fully annotate download_tracks_in_passes and the download_track closure (MyPy strict).
- Replace list[str] skipped_subtitles with a documented SkippedSubtitle TypedDict
  (id + language + title) so a client can report which subtitle of which title was
  skipped; pin the shape with a contract test.
- Clear DOWNLOAD_CANCELLED in a finally so no failed track leaves it set for later code.
- Document why the subtitle pass must stay sequential (a concurrent pass would silently
  drop in-flight subtitles via the cancel event).
- Warn only when a title skipped a subtitle and produced no video/audio/subtitle (was a
  loose len(title.tracks) check that ignored chapters/attachments).
- Narrow the over-broad remove() except to ValueError with a debug log.
- Add tests: final-clear on the fatal path, all-subs-skipped keeps video/audio, and
  duplicate-language subtitles distinguished by id.
2026-06-06 09:56:29 +03:00
sp4rk.y
51fbc33979 Merge pull request #115 from avi-dev-user/clearkey-aes128
fix(hls): decrypt AES-128 (ClearKey) media-playlist keys + per-segment sequence IV
2026-06-05 16:26:09 -06:00
imSp4rky
c03ff01c32 fix(core): replace deprecated ast.Num visitor in FPS parser
ast.Num/node.n deprecated since Python 3.8; NodeVisitor falls back to visit_Num with a DeprecationWarning per dispatch. Under -W error this surfaced as a misleading fps ValueError in Video.init. Replace with visit_Constant, reject non-numeric constants, and pin parse results for int/fraction/float inputs in a regression test.
2026-06-05 15:23:27 -06:00
imSp4rky
23cc8c9ec9 refactor(cli): replace deprecated click.MultiCommand with click.Group
MultiCommand is deprecated since Click 8.3 and removed in 9.0; its functionality was folded into Group, which supports the same list_commands/get_command lazy-loading overrides. Unblocks a future Click 9 bump and silences the import-time DeprecationWarning that broke test collection under -W error::DeprecationWarning.
2026-06-05 15:23:27 -06:00
sp4rk.y
6ae0fc2c81 Merge pull request #114 from unshackle-dl/dependabot/uv/aiohttp-3.14.0
chore(deps): bump aiohttp from 3.13.5 to 3.14.0
2026-06-05 15:06:12 -06:00
Avi Cohen
73d2cbccf8 style: use plain hyphens instead of em-dashes in comments 2026-06-05 18:25:14 +03:00
Avi Cohen
6a9b9238ba style: use plain hyphens instead of em-dashes in comments 2026-06-05 18:25:14 +03:00
Avi Cohen
41fef5d33a fix(dl): make a failed subtitle non-fatal under --skip-subtitle-errors
A single failing subtitle track previously aborted the whole download. Add an opt-in
--skip-subtitle-errors flag: when set, a Subtitle failure is logged and the track dropped from
the mux while the video/audio still complete (Video/Audio failures stay fatal; default
behaviour is unchanged).

Done at the right layer to avoid the shared-event race: a failed track sets the process-global
DOWNLOAD_CANCELLED event, which makes other in-flight tracks early-return without raising — so a
skipped subtitle could otherwise silently truncate the video/audio that still got muxed. The
download is split into two passes (download_tracks_in_passes): the fatal tracks download
concurrently first, then the skippable subtitles run in a separate sequential pass once nothing
else is in flight, with the event reset before each and at the start of every title. Skipped
languages are recorded on the dl instance (skipped_subtitles) for callers to surface.

Adds tests for the cancel-event interaction (a failing subtitle no longer truncates the
video/audio), the good/bad subtitle mix, the flag-off fatal path, and the per-title reset.
2026-06-04 08:06:23 +03:00
Avi Cohen
9590c54c33 fix(hls): decrypt AES-128 (ClearKey) media-playlist keys + per-segment sequence IV
The download path only wired up media-playlist keys when they were Widevine/PlayReady; an
AES-128 ClearKey was built but never assigned to session_drm, so segments were merged still
encrypted and the output was unplayable. Handle ClearKey from the media playlist (no license
needed), and when EXT-X-KEY has no explicit IV use each segment's media sequence number as
the IV (RFC 8216 5.2).
2026-06-04 07:09:25 +03:00
dependabot[bot]
a79cf2b5ee chore(deps): bump aiohttp from 3.13.5 to 3.14.0
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-04 00:21:47 +00:00
imSp4rky
3dcc584b2f chore(changelog): update changelog and cleanup readme 5.1.0 2026-06-03 11:11:23 -06:00
imSp4rky
538deac63e fix(session): proxy auth on http targets and string params support
Drop the manually-set Proxy-Authorization header in the Service base class. It was malformed (base64 of user:pass with no "Basic" scheme) and redundant: both rnet (Proxy.all) and requests authenticate from the credentials embedded in the proxy URL. The broken header was tunnelled harmlessly on HTTPS (CONNECT) but handed to the proxy on plaintext-http forward requests, causing HTTP 407 (e.g. http MPD/segment URLs behind an authenticated geofence proxy).

Also make RnetSession._build_url accept the same params shapes as requests (mapping, sequence of pairs, or a pre-built str/bytes query). urlencode() previously raised TypeError on a string params value.
2026-05-30 18:13:55 +00:00
imSp4rky
0561623825 docs: simplify README with demo and requirements 2026-05-29 19:01:57 -06:00
imSp4rky
4ce0dd5bb9 chore(changelog): update version date and add new features and fixes for 5.1.0 2026-05-29 18:27:31 -06:00
imSp4rky
191c4f47e0 refactor: remove dead aria2c and n_m3u8dl_re downloader code
The legacy aria2c, curl-impersonate, and N_m3u8DL-RE downloader backends were replaced by the unified in-process requests/rnet downloader. This strips the leftover references those removals missed.
2026-05-29 18:25:56 -06:00
imSp4rky
accfe9cee1 refactor(dl): declare hybrid_base_only flag and extract standalone mux helper
Make the DV-ingredient marker a declared Video attribute (sibling of needs_duration_fix) instead of a runtime-stamped attribute read via getattr, so the flag is explicit and type-checked.

Extract mux_video_standalone() to replace the duplicated per-track mux logic shared by the hybrid and normal-mode paths, removing the copy-paste and the getattr probes. Behavior unchanged.
2026-05-29 17:13:29 -06:00
imSp4rky
5899c1eec8 refactor(example): showcase full unshackle feature surface 2026-05-29 12:54:37 -06:00
imSp4rky
6bc601db39 fix(dl): mux all requested ranges and select highest DV alongside hybrid
When -r included HYBRID, the dl pipeline only muxed the hybrid output and dropped every other downloaded range. DV was also treated purely as a hybrid ingredient pool, so an explicitly requested DV range never produced a standalone deliverable - only the lowest DV (the RPU ingredient) was selected.

- Select the best DV per resolution as a standalone deliverable when DV is in -r, while still using the lowest DV as the hybrid ingredient (dv_is_deliverable).
- Flag the ingredient-only DV (hybrid_base_only) so it is downloaded for the hybrid build but skipped during standalone muxing.
- Mux every requested range standalone after hybrid processing; build hybrids from deepcopied ingredients so the originals stay muxable.
- Add Tracks.merge_video_selections to de-dup the ingredient/deliverable overlap so a shared DV track is not downloaded or muxed twice.
2026-05-28 21:25:30 -06:00
imSp4rky
fb8dc0bd9d refactor(ip_info): simplify lookup and trim cache
Cache only country/country_code (drop full IP/org/asn), bump CACHE_KEY and auto-purge stale cache versions. Dedup the three provider parsers into one normalize(). Use a plain retry-free requests session for lookups instead of the TLS-fingerprinted session, carrying only the proxy over, so a 429 returns directly and hands over to the next provider reliably.
2026-05-27 22:48:39 -06:00
imSp4rky
40104be738 fix(dash): inherit SegmentTemplate attributes across AdaptationSet/Representation
A SegmentTemplate can sit at both the AdaptationSet and Representation levels, with shared attrs like @timescale only on the outer node. The parser picked one or the other, dropping the outer node when an inner existed - so @timescale defaulted to 1 and only the first segment was emitted.

Merge instead: Representation node as base, inheriting any attribute or SegmentTimeline it omits from the AdaptationSet node (per ISO/IEC 23009-1).
2026-05-27 12:12:42 -06:00
imSp4rky
96fd971af0 feat(import): include cover-art attachments in --export/--import
Export URL-backed attachments per title and rebuild them on import so they ride along into the muxed output. Local font attachments are skipped (not portable).
2026-05-26 17:37:44 -06:00
imSp4rky
1cb0e4b766 feat(naming): per-service title_map remapping (#106)
Add services.<TAG>.title_map exact-match dict to rewrite service-provided titles before naming/output. Shared remap_titles helper applied on local dl, import, and client-side dl --remote (server stays raw so clients can override names for services they don't have installed locally).
2026-05-26 14:08:52 -06:00
imSp4rky
f4544b4a70 fix(hls): resolve per-rendition KID in no-EXT-X-KEY fallback
Manifests that signal DRM only via EXT-X-SESSION-DATA share one session_drm across all renditions, carrying just the manifest-level KID. The fallback licensing path licensed without the track's real KID, so --vaults-only only looked up the shared KID and could not find the rendition's key even when the vault held it (--cdm-only masked this since the license returns all keys). Resolve the KID from the init segment's tenc box and pass it through, matching the EXT-X-KEY path.
2026-05-26 14:04:06 -06:00
imSp4rky
f1febe7e43 chore(release): bump version to 5.1.0 2026-05-26 10:54:14 -06:00
imSp4rky
b3a6db915b fix(cli): report broken command/service loads once and cleanly
Loaders raised at import time, and since Python does not cache a failed import, every command importing services re-ran and re-reported the same errors. Build the registry once, collect failures into LOAD_ERRORS, and surface them via a single click.ClickException at the list/get chokepoints so the message renders once without a traceback or cascade.
2026-05-25 19:50:11 -06:00
imSp4rky
5e3ffdeaa1 fix(hybrid): correct static L6 source and reset stale L5 active area
level_6() read MaxCLL/MaxFALL from the dovi_tool L1 line (dynamic per-shot content-light peak) and baked it into the static L6 field, so displays tone-map to a phantom peak and HDR brightness breaks. Read MaxCLL/MaxFALL from the static L6 block instead (mastering display parse unchanged), falling back to the HDR10 base stream only when the RPU has no L6 block. Add sanitize_l6() to clamp MaxCLL to the mastering-display peak and MaxFALL to MaxCLL (0 = "unknown" preserved); also applied to the HDR10+ -> DV path.

level_5() early-returned when cropdetect found no bars on the base, leaving whatever L5 offsets the DV source carried. When DV and HDR10 geometry differ (e.g. a letterboxed DV injected into an already-cropped base) the stale offsets rode through and signalled phantom bars. Always write the detected active area, including zeros, so stale source L5 resets to the full base frame.
2026-05-25 17:04:50 -06:00
imSp4rky
7017bd0460 feat(import): reconstruct downloads from an --export sidecar
Add a shareable --export schema and a new `unshackle import` command that rebuilds tracks from the export, injects the licensed content keys, and runs the normal download/decrypt/mux pipeline with no re-authentication or licensing.
2026-05-24 21:17:31 -06:00
imSp4rky
13f924f825 feat(dl): add real bitrate probing (-rvb/-rab)
Add --real-video-bitrate/-rvb and --real-audio-bitrate/-rab to measure a track's true bitrate from actual media size instead of the manifest's declared value, which is often inflated. Useful for accurate track listings and --vbitrate/--vbitrate-range selection.
Single-file tracks are measured exactly; segmented HLS/DASH/ISM are sampled. Only the top renditions per quality tier are probed, in parallel, so it stays fast. Without the flags, behaviour is unchanged.
2026-05-24 17:30:40 -06:00
imSp4rky
7654e91ebc feat(dl): gate s_lang/a_lang miss behind --best-available
Missing requested subtitle and audio languages now warn and continue when --best-available is set instead of hard-exiting. Without the flag, missing languages still produce an error and exit, matching the prior strict behavior. Audio missing-lang detection is now symmetric with subtitles.

- add find_missing_langs helper in core/utilities for reuse between s_lang and a_lang paths (skips all/best/orig sentinels)
- refactor dl.py s_lang/a_lang checks to share the helper
- add tests/lang_selection covering match primitives, helper output, and tricky langcodes corners (zh-Hans/zh-Hant/zh-CN/zh-TW/zh-HK, cmn/yue, fil/tl/tgl)
- clean up unused-var ruff F841 in tests/remote/unit/
2026-05-22 13:52:35 -06:00
imSp4rky
b0ae88812c fix(vaults): enable WAL on SQLite vault to fix concurrent locks 2026-05-21 23:00:39 +00:00
imSp4rky
746b573711 test(remote): add unit + e2e suite for remote-services subsystem
Covers RemoteClient/RemoteService, REST routes, handlers, SessionStore, InputBridge, DownloadQueueManager, errors, compression, and serve CLI. E2e tier opts in via --live and can auto-spawn its own serve.
2026-05-21 10:45:25 -06:00
imSp4rky
9c905ef7a3 fix(api): load real CDM for service init in list/session/download flows 2026-05-21 10:31:52 -06:00
imSp4rky
c19a5ebdd2 fix(api): propagate Click default=None through service instantiation
Manual kwarg binding dropped None defaults, so services with required positional args like `type` or `profile` raised TypeError via the API while local CLI worked. Replace it with `ctx.invoke(service.cli, ...)` so Click handles defaults, and factor the duplicated setup into load_service_yaml / build_parent_ctx / instantiate_service.

Also resolve a real CDM (load_full_cdm) for search/list_titles/list_tracks; the lightweight stub lacks device_type/security_levelthat services read in __init__.
2026-05-21 08:49:16 -06:00
imSp4rky
d2380cff97 feat(dl): live countdown for --slow delay 2026-05-18 13:33:46 -06:00
imSp4rky
684e56eb97 feat(tracks): configurable audio codec priority for tie-breaking
Adds optional `audio.codec_priority` list in unshackle.yaml to define preferred audio codec order when tracks share the same bitrate and language. Listed codecs rank in the order given; unlisted codecs retain bitrate-based order and fall after the listed group (soft priority - nothing dropped). Atmos and descriptive rules still apply last.
2026-05-18 11:08:55 -06:00
imSp4rky
900ad1fde1 fix(titles): normalize odd resolutions in filename quality token
Non-16:9 aspect ratios with widths like 1918x1080 or 1620x720 were producing filenames like 911p. Snap width to the nearest standard (3840/2560/1920/1280) within 50px and snap the resulting resolution back to the track's actual height when within 10px or already a standard step.
2026-05-17 19:06:53 -06:00
imSp4rky
64da561534 feat(vaults): tolerate vault failures during key get/add
Wrap vault get_key/add_key/add_keys calls in broad exception handlers so a single failing vault (network, auth, driver error) no longer aborts the operation - other vaults are still consulted/written. Failure cause is logged at WARNING so issues remain debuggable.

Inspired by unshackle-dl/unshackle#104 by @CodeName393.

Co-authored-by: CodeName393 <62503817+CodeName393@users.noreply.github.com>
2026-05-17 12:18:32 -06:00
sp4rk.y
13dcd7aa1a Merge pull request #109 from JohnVeness/seasons
episode.py: season/seasons pluralization
2026-05-17 12:03:54 -06:00
imSp4rky
61fe16e8d7 feat(api): sync /api/download with dl CLI flags and add serve.* defaults
- Wire --no-proxy-download through download_manager + handlers + swagger
- Add tag/proxy/tmdb_id/animeapi_id/enrich/worst to DEFAULT_DOWNLOAD_PARAMS
- Normalize `slow` (bool/"MIN-MAX" string/list) to tuple before invoking dl.result
- Overlay any /api/download flag declared under `serve:` in unshackle.yaml as a default (downloads, workers, best_available, etc.); request body still wins
- Quiet successful worker stderr from `warning` to `debug` (kept under job.worker_stderr for ?full=true)
- Include HYBRID in range validator
- Document new flags + overlay layering + max_concurrent_downloads / download_job_retention_hours
2026-05-17 11:54:02 -06:00
John Veness
e5a287bc14 episode.py: season/seasons pluralization
If num_seasons = 0, output "0 seasons" (not sure if this would ever occur). If num_seasons = 1, output "1 season". If num_seasons >=2 output "X seasons".
2026-05-17 15:06:55 +01:00
imSp4rky
34a6e2d8e2 fix(service): render request_input prompt via rich console 2026-05-16 16:43:47 -06:00
imSp4rky
f26f9bcbe2 feat(video): normalize SPS VUI to match manifest-derived range
Some services ship HDR10/HLG bitstreams whose SPS VUI still carries BT.709 colour primaries/transfer/matrix, causing mediainfo and downstream players to mis-classify the file. The manifest-derived `Video.range` is the source of truth; rewrite the VUI with ffmpeg h264_metadata/hevc_metadata BSF after repackage and before mux. Skips SDR, DV, and HYBRID; no-op when the VUI is already correct.
2026-05-16 13:55:05 -06:00
imSp4rky
ead88fe066 feat(hls): detect DV-composite tracks and restore signaling post-mux
Parse HLS SUPPLEMENTAL-CODECS to identify tracks that ship Dolby Vision RPU NALs in a stream whose primary codec is plain hvc1 (e.g. fMP4 ladders signalled as dvh1.08.x only via SUPPLEMENTAL-CODECS). Tag such tracks with the new `dv_compatible_bitstream` flag on Video.

Add DVFixup helper that runs `dovi_tool extract-rpu | inject-rpu` on flagged tracks before mux so the muxed MKV is recognised as Dolby Vision. Soft-fails to the source bitstream if dovi_tool is missing or any step errors.

Range stays whatever VIDEO-RANGE signalled. HDR10+ presence is a bitstream feature, not a codec-string feature, so services that know their encoder embeds HDR10+ SEI must override Video.range themselves.
2026-05-16 13:52:57 -06:00
imSp4rky
b4d422459c refactor(hybrid): extract dovi_tool and run_step helpers
Centralise dovi_tool subcommand invocations behind a thin wrapper module so Hybrid no longer re-implements argv construction, status spinners, and stderr handling per call site. Adds a generic `run_step` helper for subprocess steps that must produce a non-empty output file.
2026-05-16 13:50:55 -06:00
imSp4rky
cda8120b6d fix(title): use original-language audio for filename metadata
When a non-original audio language is the default (via muxing.default_language or sort order), the filename audio codec/channel fields still reflect the title's original-language track instead of whichever track appears first in the muxed MKV.
2026-05-16 11:56:04 -06:00
imSp4rky
c0dc1eb91f docs(output): document muxing.default_language override 2026-05-15 08:22:45 -06:00