Commit Graph
68 Commits
Author SHA1 Message Date
imSp4rky d7e08e979b fix(downloader): surface failed downloads instead of returning silently 2026-08-07 11:31:11 -06:00
imSp4rky d448ee8ff9 fix(downloader): decode content-encoded bodies instead of writing raw
Reading .raw skips urllib3's decoding, so brotli-served subtitles landed compressed and converted to empty files. Also drops a partial rather than Range-resuming an encoded body, since a Range indexes the encoded bytes.
2026-08-05 17:56:33 -06:00
imSp4rky 02fd7b9387 fix(subtitles): strip X-TIMESTAMP-MAP so WebVTT cues are not offset on conversion 2026-08-05 11:07:38 -06:00
imSp4rky 041ba5293b feat(titles): add optional Episode.part, selected with -w s1e1.2
Opt-in and additive: an episode without a part behaves exactly as before.
2026-08-04 17:38:30 -06:00
imSp4rky 3e0d7b5f75 feat(dl): add tag_rules to conditionally swap the group tag 2026-08-04 09:42:42 -06:00
imSp4rky a5f5e7d14d refactor(tracks): honour the best token and share one match rule
sort_subtitles resolved only the all token, while the video and audio sorts
resolved all and best. by_language now asks matching_languages rather than
repeating the RFC 4647 preference, with no behaviour change.
2026-08-03 10:29:09 -06:00
imSp4rky a6997f6bca feat(tracks): add language_priority to sort preferred languages first
subtitle.language_priority and audio.language_priority prepend to the CLI
languages, so the named languages sort first and nothing is filtered.
--exact-lang now applies to the sorts too, through matching_languages, which
follows the same RFC 4647 rule as Tracks.by_language and ignores invalid tags.
2026-08-03 10:15:05 -06:00
imSp4rky 4e1ccf3230 feat(dl): add -fsl/--forced-s-lang to filter forced subtitles by language 2026-08-03 09:27:40 -06:00
imSp4rky 330e274da6 feat(temp): give each download task its own temp directory
Each dl run works in temp/task_<id>/, removed on every exit path. Orphans from hard-killed processes are swept via lock liveness, and env clear temp skips directories whose task is still running.
2026-08-02 13:28:23 -06:00
imSp4rky b8824b819d merge: subtitle.group_by track option 2026-08-02 10:50:20 -06:00
imSp4rky 4ac34069ec feat(providers): add TheTVDB v4 as a metadata provider
TVDB covers TV far better than the existing sources, so add it as a
provider and let users decide where it sits in the lookup chain.

- new TVDB v4 provider: search, get_by_id, get_external_ids and
  find_by_imdb_id, with a process-level token cache and 401 re-auth
- metadata_providers config both orders and filters the providers used,
  so TVDB can be primary rather than a TMDB fallback. Cross-validation
  keeps its own fixed trust ranking, unchanged by search order
- --tvdb-order renumbers episodes into a TVDB season order. A service
  does not always use aired order, so the source order is detected
  rather than assumed, and the renumber is refused when an order that
  omits episodes would give two of them the same filename
- --tvdb sets the series ID directly, skipping the lookup
- the Matroska TVDB2 tag now populates on every lookup path, including
  --imdb, which previously produced no ID tags when IMDxAPI was off

Search matches on year as well as title so an identically named remake
cannot win on title alone.
2026-08-02 10:48:32 -06:00
imSp4rky 8baf064750 feat(tracks): add subtitle.group_by to keep a language with its variants
sort_subtitles applied the language sort before the type sort. Python sorts
are stable, so the last pass wins and type became the major key: every forced
track, then every normal, then every SDH, each block ordered by language. In a
title with many languages this put a track far from its own SDH version, for
example Finnish at track 5 and Finnish SDH at track 42.

Add a group_by parameter that selects the major key. "type" is the default and
keeps the current order byte for byte. "language" swaps the two passes so each
language stays next to its own variants. Read it from the new subtitle.group_by
config key.

Also correct the docs: type_priority was described as sorting the variants
within each language, which was never true. It ranks the types across all
languages.
2026-08-02 10:45:28 -06:00
imSp4rky bf85c948b4 fix(dl): honour --no-proxy-download for attachments 2026-07-31 13:22:18 -06:00
imSp4rky 734df87bad feat(dl): add --speed-limit and serve global_speed_limit
Shared token bucket across all download threads; decimal suffixes (5M = 5.0 MB/s) or plain bytes/sec
2026-07-31 12:48:46 -06:00
imSp4rky e9a40a3af3 perf: speed up the code hash with sha1 and a plain file reader 2026-07-30 17:11:18 -06:00
imSp4rky c14277e6dc feat: identify the running code by source hash 2026-07-30 16:38:11 -06:00
imSp4rky 72b8ab7027 feat: show git commit hash alongside the version 2026-07-30 13:44:07 -06:00
imSp4rky 42d7fd8303 fix(import): scope the merge-back and detect ISM protection
Only merge back tracks from another manifest, so a drifted id no longer duplicates a rung or wipes the main manifest's own subtitles. track_is_encrypted now recognises an ISM ProtectionHeader.
2026-07-29 10:48:34 -06:00
imSp4rky a24c9b92da fix(tracks): ignore leftover PIFF uuid when verifying decryption
A decrypter detaches only the atom it consumed, so content shipping both boxes keeps the uuid through a good decrypt and failed on mp4decrypt.
2026-07-27 13:11:13 -06:00
imSp4rky 8bacb247ee fix(lang): surface orig in errors and flag the original audio track over the API
Error messages named the resolved language for a user who only ever defaulted to orig. Audio tracks now carry is_original, resolved through Tracks.by_language so it agrees with the downloader.
2026-07-26 18:22:46 -06:00
imSp4rky b74b64825e fix(templates): use one separator style across a whole folder path
Detect the dot/space style from the full folder template instead of per segment, so `{artist}/{album} ({year})` no longer yields `Some.Artist/Some Album (2025)`. Also drops empty `[]` groups like the existing `()` handling and removes redundant cleanup passes.
2026-07-26 09:48:58 -06:00
imSp4rky d60d8abf53 fix(ism): repair synthesized init and verify fragments decrypted
UNITY_MATRIX emitted 11 int32s instead of 9, oversizing mvhd/tkhd so pymp4 desynced and fTTML subtitles yielded no cues. Also rewrites PIFF senc to CENC so shaka can decrypt Smooth, labels HEV1 as hev1, and raises when a fragment survives decryption still carrying a sample-encryption box.
2026-07-25 15:55:31 -06:00
imSp4rky b7029c343e fix(ism): emit two stsd entries so sample_description_index 2 resolves
Smooth fragments set tfhd sample_description_index=2 against a server init carrying a second, identical sample entry. A lone entry left that dangling: mp4decrypt skipped every such fragment silently and demuxers stopped after the first, yielding a file that looked fine but was almost entirely encrypted.
2026-07-25 12:59:55 -06:00
imSp4rky 8cf1700684 fix(import): stop losing side-loaded subtitles and blaming the manifest URL
Only fall back to Title.language when the manifest labels nothing, and reject und as firmly as a missing tag. Exported direct-URL subtitles now replace the re-parsed set, and fetch failures stay distinct from parse failures.
2026-07-25 11:32:17 -06:00
imSp4rky ae9ffc7731 feat(drm): auto-repack tracks with a residual encrypted sample entry
Detect an encv/enca sample entry surviving decryption (moov-scoped 4CC scan) and force a repack so mkvmerge isn't first-FourCC-locked to a generic encrypted codec; warn if one persists after repacking.

Closes #142
2026-07-23 09:33:14 -06:00
imSp4rky 95a754fa6c fix: harden ClearKey/CENC decryption correctness 2026-07-19 10:49:45 -06:00
imSp4rky 2c8a98f337 feat: original-language aware DUAL/MULTi tagging and new DUBBED variable
DUAL now requires the original language plus one dub (strict default);
dual_multi_mode: count restores the old pure-count behavior.
2026-07-18 11:02:02 -06:00
imSp4rky e31e60a418 style: run ruff format over the codebase 2026-07-15 14:49:27 -06:00
imSp4rky 59a7c76b2f feat(ism): harden Smooth parsing and derive fps/codec from spec fields
Regex-based UrlPattern substitution (handles {Bitrate}/{start_time} spellings), AudioTag/FourCC codec fallback, Width/Height fallbacks, integer fragment times, live-manifest and missing-language guards. Extract fps from the HEVC SPS VUI (HEVC-only; AVC timing untrusted).
2026-07-14 21:24:14 -06:00
imSp4rky 2ccae1a0ee feat(cdm): stamp remote CDM DRM type at load for reliable detection 2026-07-14 17:32:00 -06:00
imSp4rky b8ec9e2498 fix(tracks): keep regional language variants distinct in exact selection
by_language now applies RFC 4647 Lookup in exact mode, preferring the string-equal tag so --exact-lang separates base/regional pairs (ar/ar-EG, en/en-US) instead of collapsing them, and dedupes by id to prevent the duplicate-track-id crash. Adds a language-selection regression test.
2026-07-14 09:17:22 -06:00
imSp4rky a450184ab8 test(remote): isolate job history writes from the real api_history.jsonl 2026-07-12 10:30:30 -06:00
imSp4rky d7a2bb6b8e fix(subs): stop segmented WebVTT merge from stripping formatting
Merge cues at the string level instead of round-tripping through
pycaption, which dropped inline tags and cue settings. STYLE and
REGION blocks are kept, duplicate boundary cues splice correctly,
and a bad timing line now skips the cue instead of crashing.

Also rank pysubs2 above SubtitleEdit for ASS to TTML/VTT since
SubtitleEdit flattens inline styling there. Documented the measured
backend fidelity table and added regression tests for the merge.
2026-07-12 09:39:39 -06:00
imSp4rky 6434fc2081 fix(dl): make audio/video track selection consistent and crash-safe
Share one best-audio selector across the episode and music paths so best/all works for songs, descriptive audio is capped to one track, and codec_priority is honored. Guard --channels and 16:9 canvas math against tracks missing channel/width metadata. Extend --best-available to video languages and full resolution misses in both normal and hybrid mode, and count only deliverable ranges when reporting missing resolutions. Rank video by resolution before bitrate, and trim group audio selections to one track per song with a warning.

BREAKING CHANGE: a missing resolution with -r HYBRID now errors without --best-available, matching non-hybrid behavior.
2026-07-11 18:41:41 -06:00
imSp4rky 4ebe41fafd feat(tracks): add subtitle.type_priority config for variant ordering
Optional ranking of forced/normal/sdh per language, mirroring audio.codec_priority; first track gets the default flag, so this also picks the default subtitle. Unset = existing behavior.
2026-07-07 17:36:58 -06:00
imSp4rky 5cd50544a8 fix(remote): harden serve API access control and input handling
Bind sessions and download jobs to the API key (not source IP), reject path-traversal cache keys, bound decompression of peer-supplied blobs, gate the CDM device before the vault fallback, and compare API keys in constant time.
2026-07-06 23:54:05 -06:00
imSp4rky 52e04002e4 test(remote): add RemoteService method-contract tests 2026-07-03 21:26:47 -06:00
imSp4rky 59f3cb4fbe feat(titles): add air_date for date-based episode naming
Optional Episode.air_date names daily/sports content by date instead of SxxExx (no template change). Presentation-only; `{year}` dropped only when it duplicates the date's year. Round-trips via export/import.
2026-06-30 15:45:57 -06:00
imSp4rky 5d4a27b8ca style: apply pre-commit auto-fixes (isort, whitespace, eof) 2026-06-25 11:06:55 -06:00
imSp4rky 66638ead6b feat(services): load service plugins from git repos
Allow directories.services entries to be git repo specs (URL/owner-repo/ @branch) alongside local paths: clone + TTL-pull as cache, list-order priority with a duplicate-ignored summary, a refresh-services command that force-syncs and reports the per-service diff, and a dirty-clone guard that protects local edits on the automated path. Adds binaries.Git and a redact_paths config toggle.
2026-06-24 08:59:03 -06:00
imSp4rky 6294566823 fix(config): match per-service cdm keys case-insensitively
cdm lookups used the raw service key while decryption upper-cases its keys, so a case-mismatched cdm entry silently used the default. Add a shared ci_get helper, route cdm/decryption through resolvers, drop a redundant decryption mutation and dead downloader writes, and add a regression bench.
2026-06-22 16:50:59 -06:00
imSp4rky 3f5b5e0fe1 refactor(api): drop dead error codes, lock error contract harness
Remove 4 unused APIErrorCode members; strengthen test_errors.py to pin the full (status, code, retryable) contract before the PRD 05 collapse.
2026-06-21 14:13:44 -06:00
imSp4rky d4b0c81c63 refactor(api): collapse route registration into single ROUTES table
Derive both setup_routes and setup_swagger from one ROUTES table, and move the repeated trailing APIError->response block into an @api_handler decorator.
2026-06-21 13:47:23 -06:00
imSp4rky fb90cd87b0 refactor(api): dedup handler serializers, validation, and service setup
Collapse repeated serializer/validation/CDM/list-setup logic in handlers.py behind small helpers; output behaviour unchanged.
2026-06-21 13:23:22 -06:00
imSp4rky 8a9a1e1bdc refactor(api): remove dead code paths in serve/remote layer
Delete code paths that never executed in the serve/API/remote stack:

- BridgeCancelledError: raised but never caught; replace raises with RuntimeError and drop the class.
- InputBridge.error setter and _error field: written at the handler layer, never read (session.auth_error is the surfaced channel).
- cancel_event param and _check_cancel inner fn on _perform_download: the only caller always passed None (the live job.cancel_event is untouched).
- result_with_progress monkeypatch: its status pings were redundant with the file-polled progress_sink, which is now the sole path.
- session param on _resolve_manifest_data: unused in the body.

Reconcile SessionStore's hardcoded 600s auth-state TTL into a shared AUTH_INPUT_TIMEOUT constant referenced by both the InputBridge input timeout and the store, removing the magic number without changing the longer grace window.

Update input_bridge/session_store tests for the new channels and add an assertion that auth-state entries expire at the single rule.
2026-06-21 13:07:45 -06:00
imSp4rky a2f670b140 feat: support Python 3.14 and widen range to 3.11-3.14
Move the supported interpreter range to >=3.11,<3.15, dropping the EOL 3.10 floor and unblocking installs on 3.13/3.14.

- pyproject: requires-python ">=3.10,<3.13" -> ">=3.11,<3.15"
- update_checker: replace deprecated asyncio.get_event_loop() with asyncio.to_thread() (warns on 3.12+, slated for removal)
- test_compression: simplify _run() to plain asyncio.run()
- README: bump documented Python range to 3.11 - 3.14
- uv.lock: regenerate for the new interpreter floor
2026-06-20 15:25:42 -06:00
imSp4rky ba69bc7d61 feat(dl): add --merge-video to merge video language variants
Group selected videos by (resolution, range, codec) and mux each group into one MKV; only language is collapsed, so ranges/codecs/resolutions stay in separate files. Adds --merge-video flag + muxing.merge_video config (global and per-service), docs, and tests.
2026-06-17 13:31:21 -06:00
sp4rk.yandMrMovies-Dev 78a6a97fcf feat(music): native music core - shared helpers, album folder template, display cleanup (#125)
* Add native Music core workflow

* feat(music): add shared music core helpers, album folder template, and UX cleanup

Consolidate duplicated music-service logic into core and extend the native music workflow.

- add core/music/extract.py: shared stateless helpers (first_text, first_number, year/duration/name formatting, classify_release_kind, dedupe_track_options, build_music_from_songs) so services stop carrying their own copies
- add core/music/display.py: shared rich rendering (render_track_panel, render_album_header, render_artwork_preview) with TrackRow / MusicHeaderInfo data holders
- export the new helpers from core/music/__init__.py
- add dedicated `albums` folder template kind (output_template.folder.albums) resolving albums -> songs -> "{artist} - {album} ({year})"; whitelist music template variables (album_artist, track_total, disc_total, release_type, genre, explicit, isrc, upc, label)
- fix song filename crash: config.get_output_template(...) did not exist; use config.output_template.get("songs") with a sane default
- strip emojis from music output (renderer, dl.py music branch) to match unshackle UX; remove dead MusicSongPlan import and music_icon logic
- document the albums folder key in unshackle-example.yaml
- add tests for extract, display, and the folder template

---------

Co-authored-by: MrMovies-Dev <MrMovies-Dev@users.noreply.github.com>
2026-06-15 13:34:34 -06:00
imSp4rky e207116d30 fix(ism): derive video colour range from CodecPrivateData SPS VUI
Smooth Streaming manifests carry no range attributes, so every ISM video track was labelled SDR even when the stream is HDR10/HLG/DV, breaking range-based selection (-r HDR10 / -r DV) for ISM services.

  - ism_init: walk the full HEVC SPS (incl. scaling-list and st_ref_pic_set skippers) to read the VUI colour triple (colour_primaries, transfer_characteristics, matrix_coeffs); expose parse_codec_private_data_colour() keyed by FourCC. No unshackle imports added.
  - ism: new ISM.get_video_range() maps the CICP triple via Video.Range.from_cicp (PQ -> HDR10, HLG -> HLG, BT.709/absent -> SDR); DVHE/DVH1 FourCCs map straight to DV since DV bitstreams signal Unspecified (2,2,2) in the VUI. to_tracks() now sets range_ on every video track. Soft-fails to SDR on malformed data.
  - ism: accept RnetSession in download_track() so TLS-impersonated sessions pass the type check.
  - tests: real PQ/HLG/BT.709 (x265-minted) and Dolby Vision (live-manifest, DoViProfile=stn, out-of-order SPS,PPS,VPS NALs) CodecPrivateData samples; byte-level VUI assertions in test_ism_init and manifest->Range characterization in new test_ism_range.
2026-06-11 18:28:35 -06:00
imSp4rky 39034f2bb5 fix(ism): rebuild moov init segment for Smooth Streaming decrypt
ISM (Smooth Streaming) tracks raw-concatenate moof+mdat fragments with no ftyp/moov, so shaka-packager/mp4decrypt fail with PARSER_FAILURE (exit 2) on decrypt. The init box was previously built by n_m3u8dl_re, removed in the downloader consolidation.

Add ism_init.py, a dependency-free byte-level MP4 init-segment synthesizer that rebuilds ftyp+moov from the manifest CodecPrivateData, ported from yt-dlp's write_piff_header and N_m3u8DL-RE's MSSMoovProcessor with full codec parity:

- AVC (H264/AVC1/DAVC), with SPS/PPS picked by NAL type rather than position and NALUnitLengthField honored
- HEVC (HVC1/HEV1) with chroma format and bit depths parsed from the de-emulated SPS via exp-Golomb so 10-bit/HDR signals correctly, and profile/tier/level lifted from the SPS PTL
- Dolby Vision (DVHE/DVH1) as hvcC with a dvh1 sample entry
- AAC (AACL/AACH) with the AudioSpecificConfig synthesized from SamplingRate/Channels when the manifest omits CodecPrivateData
- EC-3 with a real dec3 box extracted from the WAVEFORMATEXTENSIBLE CodecPrivateData (Dolby GUID located by search, not fixed offset)
- TTML subtitles as stpp/sthd/subt, wired for fragmented-TTML tracks

CENC wrapping (encv/enca + sinf/tenc with default_KID) covers encrypted tracks: the per-sample IV size is derived from the fragment senc/saiz (PIFF override flag, payload arithmetic, saiz fallback) instead of assuming 8, and the constant-IV tenc form is supported. Read the track_ID from the first fragment's tfhd so the moov matches and the muxer does not drop samples.

Wire ISM.download_track to prepend the synthesized init before merging; unsupported codecs soft-fail to raw concatenation with a warning. Harden against real-world inputs: 2-letter/uppercase manifest language tags normalize to ISO-639-2 (und fallback), >65535 Hz sample rates no longer overflow the 16.16 field, truncated tfhd returns None, struct.error joins the soft-fail handler, and the emulation-prevention scan no longer over-strips consecutive escapes.

Add regression tests (37) covering box structure, every supported FourCC, 10-bit SPS parsing, ASC synthesis, dec3 extraction, IV-size derivation and the crash fixes. Validated structurally per codec with ffmpeg-minted fragments: shaka-packager parses synth-init+fragments with exit 0 and ffprobe reports the expected codec, including a live run against a public Smooth Streaming server.
2026-06-11 13:41:58 -06:00