Commit Graph
676 Commits
Author SHA1 Message Date
imSp4rky f69ebb4fba feat(api): report per-episode and per-track progress on download jobs
Adds title, current_title, incremental output_files, and track_progress to job status so clients can render live multi-episode progress.
2026-07-01 18:46:41 -06:00
imSp4rky f004fa7e42 fix(api): use PBKDF2 instead of SHA256 for cache-namespace hashing
Silences CodeQL py/weak-sensitive-data-hashing on the X-Secret-Key and per-job credential hashes; fixed salt keeps namespaces deterministic.
2026-06-30 21:08:29 -06:00
imSp4rky d4a58267d4 chore(release): bump version to 5.3.0 5.3.0 2026-06-30 20:51:17 -06:00
imSp4rky 96da1ac0b0 docs: fix inaccuracies in service-creation guide 2026-06-30 16:04:12 -06:00
imSp4rky 59f3cb4fbe feat(titles): add air_date for date-based episode naming
Optional Episode.air_date names daily/sports content by date instead of SxxExx (no template change). Presentation-only; `{year}` dropped only when it duplicates the date's year. Round-trips via export/import.
2026-06-30 15:45:57 -06:00
imSp4rky 987de239ee docs: add "Creating a Service" guide 2026-06-30 15:45:57 -06:00
imSp4rky 140c2a4b2f docs(readme): link Docs to docs.unshackle.dev, remove Wiki 2026-06-30 10:54:49 -06:00
imSp4rky 2f92ffe1a2 ci(docs): serve site from docs.unshackle.dev custom domain 2026-06-30 10:40:43 -06:00
imSp4rky ecbcdff314 docs: add MkDocs Material documentation site 2026-06-30 10:28:48 -06:00
imSp4rky 03bcfc0c45 perf(hls): rename single decrypted range instead of re-copying
Short-circuit merge() to os.replace() when joining one already-merged file with no init prefix; saves a full track-size copy, falls back to streamed copy across devices.
2026-06-29 09:55:08 -06:00
imSp4rky efb560f887 fix(hls): stream segment merges to avoid OOM on large tracks
Replace whole-file read_bytes() in merge()/discontinuity merge with shutil.copyfileobj (1MB buffer); fixes MemoryError on multi-GB tracks.
2026-06-29 09:31:38 -06:00
imSp4rky 39f5bc2a9e feat(proxies): add ProtonVPN TV login, drop cookie refresh
Self-sustaining TV-code auth; cookies are now access-only and re-exported on expiry.
2026-06-28 18:42:11 -06:00
imSp4rky f9ab349e9c perf(tracks): skip redundant DV/VUI bitstream passes
Expose DV via a plain elementary-stream demux instead of a dovi_tool RPU round-trip, and skip normalize_vui when the VUI already matches the target range. Both avoid a full bitstream pass; stale-VUI streams are still rewritten.
2026-06-28 15:40:19 -06:00
imSp4rky b6fe0ae4b8 fix(dl): apply per-service decryption tool override
Resolve config.decryption_map into config.decryption before decrypt so services honor mp4decrypt instead of defaulting to shaka.
2026-06-28 14:48:16 -06:00
CodeName393 a0feb51d5a fix(console): pause active live contexts during terminal input (#127)
When a service calls `request_input` (e.g., during OTP verification in the `authenticate` phase), the terminal's active `Live` or `status` spinner contexts (handled by `rich`) would continuously refresh and redraw in the background. This caused the input prompt to disappear or become hidden immediately after rendering, although keystrokes were still registered.

This commit resolves the issue by overriding `Console.input` in `ComfyConsole` to:
1. Temporarily pause (`stop()`) all active `Live` displays before requesting input.
2. Read the user input safely without interference from the refresh loop.
3. Resume (`start()`) all previously active displays in the reverse order afterwards.
2026-06-26 10:01:18 -06:00
sp4rk.y 2ed243aae4 fix(output): treat / and \ as folder separators on any OS
Split folder templates on both separators and format each segment separately so sanitize_filename never rewrites the divider; validate folder.* templates per-segment.
2026-06-25 16:03:27 -06:00
sp4rk.y 4539f63106 fix(env): only resolve Path items in directory lists
Raw string repo specs in directories.services crashed env info with AttributeError; strings are now passed through unchanged.
2026-06-25 14:00:48 -06:00
imSp4rky 5d4a27b8ca style: apply pre-commit auto-fixes (isort, whitespace, eof) 2026-06-25 11:06:55 -06:00
imSp4rky 06e0593da3 fix(drm): don't switch to a mismatched CDM type during licensing
Guard the CDM switch on the fetched CDM's actual DRM type and raise a clear error on mismatch instead of sending a mismatched challenge.
2026-06-25 11:05:18 -06:00
imSp4rky e96e9be143 feat(output): support nested directories in folder templates
Treat `/` in folder templates as a path separator (sanitize per segment) and let movies use a folder when a `movies` template is set.
2026-06-25 10:40:49 -06:00
imSp4rky 14c96371b9 feat(proxies): add Proton VPN proxy provider
Cookie-based ProtonVPN provider with country/city/server-number selection
2026-06-24 22:11:51 -06:00
imSp4rky 66638ead6b feat(services): load service plugins from git repos
Allow directories.services entries to be git repo specs (URL/owner-repo/ @branch) alongside local paths: clone + TTL-pull as cache, list-order priority with a duplicate-ignored summary, a refresh-services command that force-syncs and reports the per-service diff, and a dirty-clone guard that protects local edits on the automated path. Adds binaries.Git and a redact_paths config toggle.
2026-06-24 08:59:03 -06:00
imSp4rky 6294566823 fix(config): match per-service cdm keys case-insensitively
cdm lookups used the raw service key while decryption upper-cases its keys, so a case-mismatched cdm entry silently used the default. Add a shared ci_get helper, route cdm/decryption through resolvers, drop a redundant decryption mutation and dead downloader writes, and add a regression bench.
2026-06-22 16:50:59 -06:00
imSp4rky ce69e8a028 docs(config): make docs/ canonical, slim unshackle-example.yaml to a starter 2026-06-21 21:23:03 -06:00
imSp4rky 605c0e65cb feat(docs): add per-vault network timeout configuration for remote vaults 2026-06-21 20:57:17 -06:00
imSp4rky ecf0bf41ab feat(vaults): add configurable network timeout for remote vaults
Default 10s on API/HTTP vault requests, overridable per-vault (`timeout:`) or globally (`vault_timeout`); previously they could hang indefinitely on an unreachable host.
2026-06-21 20:39:37 -06:00
imSp4rky 3f5b5e0fe1 refactor(api): drop dead error codes, lock error contract harness
Remove 4 unused APIErrorCode members; strengthen test_errors.py to pin the full (status, code, retryable) contract before the PRD 05 collapse.
2026-06-21 14:13:44 -06:00
imSp4rky 2f86d782bf fix(api): annotate routes.py to pass strict mypy
Type cors_middleware and the service_data dict; annotations only, no behaviour change.
2026-06-21 13:58:23 -06:00
imSp4rky d4b0c81c63 refactor(api): collapse route registration into single ROUTES table
Derive both setup_routes and setup_swagger from one ROUTES table, and move the repeated trailing APIError->response block into an @api_handler decorator.
2026-06-21 13:47:23 -06:00
imSp4rky 9cb20693e0 refactor(api): use Event.wait and a shared to_enum helper
Replace InputBridge's poll loop with Event.wait(timeout) and collapse the three download_manager enum maps into one to_enum() helper. No behaviour change.
2026-06-21 13:31:08 -06:00
imSp4rky fb90cd87b0 refactor(api): dedup handler serializers, validation, and service setup
Collapse repeated serializer/validation/CDM/list-setup logic in handlers.py behind small helpers; output behaviour unchanged.
2026-06-21 13:23:22 -06:00
imSp4rky 8a9a1e1bdc refactor(api): remove dead code paths in serve/remote layer
Delete code paths that never executed in the serve/API/remote stack:

- BridgeCancelledError: raised but never caught; replace raises with RuntimeError and drop the class.
- InputBridge.error setter and _error field: written at the handler layer, never read (session.auth_error is the surfaced channel).
- cancel_event param and _check_cancel inner fn on _perform_download: the only caller always passed None (the live job.cancel_event is untouched).
- result_with_progress monkeypatch: its status pings were redundant with the file-polled progress_sink, which is now the sole path.
- session param on _resolve_manifest_data: unused in the body.

Reconcile SessionStore's hardcoded 600s auth-state TTL into a shared AUTH_INPUT_TIMEOUT constant referenced by both the InputBridge input timeout and the store, removing the magic number without changing the longer grace window.

Update input_bridge/session_store tests for the new channels and add an assertion that auth-state entries expire at the single rule.
2026-06-21 13:07:45 -06:00
AviDev 40eafce17b feat(api): expose active-track segment counts and transfer speed in job progress (#124)
The job progress already reports a weighted percentage, phase and track counts.
The per-track tee in build_job_progress_callables already tracks each track's
segment counts and sees the downloader's speed string, so also surface the
currently-downloading track's segments_done / segments_total and speed in the
emitted progress (and on the job + its to_dict). Lets a client show
"197/663 segments, 3.2 MB/s" instead of only a percentage.
2026-06-21 12:40:16 -06:00
AviDev 733cb02391 feat(api): expose title description, date and cover_url in serialize_title (#119)
Surface optional display metadata a service can provide: the synopsis
(title.description) and a release/air date or poster image URL stashed in
title.data ("date" / "cover_url"). Purely additive, all fields default to
null when the service does not set them, so existing clients are unaffected.
2026-06-21 12:38:27 -06:00
sk8ordi3 5b724b48f8 feat: add automatic Firefox cookie and localstorage extraction (#112)
* feat: add automatic Firefox cookie and localstorage extraction

* address security and consistency findings from PR review
2026-06-21 12:26:17 -06:00
imSp4rky a2f670b140 feat: support Python 3.14 and widen range to 3.11-3.14
Move the supported interpreter range to >=3.11,<3.15, dropping the EOL 3.10 floor and unblocking installs on 3.13/3.14.

- pyproject: requires-python ">=3.10,<3.13" -> ">=3.11,<3.15"
- update_checker: replace deprecated asyncio.get_event_loop() with asyncio.to_thread() (warns on 3.12+, slated for removal)
- test_compression: simplify _run() to plain asyncio.run()
- README: bump documented Python range to 3.11 - 3.14
- uv.lock: regenerate for the new interpreter floor
2026-06-20 15:25:42 -06:00
a9c677c349 feat: Native ExpressVPN HTTPS Proxy Provider (#126)
* feat: add native ExpressVPN HTTPS proxy provider

Introduces ExpressVPN as a first-class native proxy provider for unshackle.

This provider implements the full browser extension proxy authentication flow — from OAuth PKCE bootstrap through subscription validation to authenticated HTTPS proxy endpoints — without requiring any external tools or manual token management.

Supports smart location selection, city-level targeting with intelligent abbreviation matching, and pinned server selection. All tokens are automatically cached and refreshed.

No new dependencies required.

* fix: address PR #126 review feedback

- __repr__ no longer triggers network/OAuth calls; reads cached locations only
- _match_city uses tiered priority buckets (abbrev > exact > prefix > substring)
- Token cache writes race-free: os.open + os.fchmod(0o600) on Unix, fallback on Windows
- Cookie/cache paths resolve under config.directories, not CWD
- _get_connection_token parses response JSON once
- Drop unused PKCE state parameter (never validated)
- Add inline doc for _is_jwt_expired no-exp behavior
- Fix 3 lines exceeding 120-col project style

---------

Co-authored-by: MrMovies-Dev <MrMovies-Dev@users.noreply.github.com>
Co-authored-by: sp4rk.y <george+slight@slight.me>
2026-06-17 16:51:33 -06:00
imSp4rky ba69bc7d61 feat(dl): add --merge-video to merge video language variants
Group selected videos by (resolution, range, codec) and mux each group into one MKV; only language is collapsed, so ranges/codecs/resolutions stay in separate files. Adds --merge-video flag + muxing.merge_video config (global and per-service), docs, and tests.
2026-06-17 13:31:21 -06:00
imSp4rky 2e1ffebec0 docs: rename to match other docs. 2026-06-15 20:55:27 -06:00
imSp4rky 1b27b10746 chore(release): bump version to 5.2.0 5.2.0 2026-06-15 20:52:22 -06:00
imSp4rky b29ad6bc0d ci: remove release automation workflows 2026-06-15 20:46:56 -06:00
sp4rk.yandMrMovies-Dev 78a6a97fcf feat(music): native music core - shared helpers, album folder template, display cleanup (#125)
* Add native Music core workflow

* feat(music): add shared music core helpers, album folder template, and UX cleanup

Consolidate duplicated music-service logic into core and extend the native music workflow.

- add core/music/extract.py: shared stateless helpers (first_text, first_number, year/duration/name formatting, classify_release_kind, dedupe_track_options, build_music_from_songs) so services stop carrying their own copies
- add core/music/display.py: shared rich rendering (render_track_panel, render_album_header, render_artwork_preview) with TrackRow / MusicHeaderInfo data holders
- export the new helpers from core/music/__init__.py
- add dedicated `albums` folder template kind (output_template.folder.albums) resolving albums -> songs -> "{artist} - {album} ({year})"; whitelist music template variables (album_artist, track_total, disc_total, release_type, genre, explicit, isrc, upc, label)
- fix song filename crash: config.get_output_template(...) did not exist; use config.output_template.get("songs") with a sane default
- strip emojis from music output (renderer, dl.py music branch) to match unshackle UX; remove dead MusicSongPlan import and music_icon logic
- document the albums folder key in unshackle-example.yaml
- add tests for extract, display, and the folder template

---------

Co-authored-by: MrMovies-Dev <MrMovies-Dev@users.noreply.github.com>
2026-06-15 13:34:34 -06:00
imSp4rky 680f5059b5 fix(subtitle): drive SubtitleEdit 4.x with legacy /convert syntax
Registry refactor (2923292) emitted seconv-only --flags; 4.x opened the GUI per subtitle. Pick syntax per binary; prefer seconv for 5.x.
2026-06-11 19:09:06 -06:00
imSp4rky e207116d30 fix(ism): derive video colour range from CodecPrivateData SPS VUI
Smooth Streaming manifests carry no range attributes, so every ISM video track was labelled SDR even when the stream is HDR10/HLG/DV, breaking range-based selection (-r HDR10 / -r DV) for ISM services.

  - ism_init: walk the full HEVC SPS (incl. scaling-list and st_ref_pic_set skippers) to read the VUI colour triple (colour_primaries, transfer_characteristics, matrix_coeffs); expose parse_codec_private_data_colour() keyed by FourCC. No unshackle imports added.
  - ism: new ISM.get_video_range() maps the CICP triple via Video.Range.from_cicp (PQ -> HDR10, HLG -> HLG, BT.709/absent -> SDR); DVHE/DVH1 FourCCs map straight to DV since DV bitstreams signal Unspecified (2,2,2) in the VUI. to_tracks() now sets range_ on every video track. Soft-fails to SDR on malformed data.
  - ism: accept RnetSession in download_track() so TLS-impersonated sessions pass the type check.
  - tests: real PQ/HLG/BT.709 (x265-minted) and Dolby Vision (live-manifest, DoViProfile=stn, out-of-order SPS,PPS,VPS NALs) CodecPrivateData samples; byte-level VUI assertions in test_ism_init and manifest->Range characterization in new test_ism_range.
2026-06-11 18:28:35 -06:00
imSp4rky 39034f2bb5 fix(ism): rebuild moov init segment for Smooth Streaming decrypt
ISM (Smooth Streaming) tracks raw-concatenate moof+mdat fragments with no ftyp/moov, so shaka-packager/mp4decrypt fail with PARSER_FAILURE (exit 2) on decrypt. The init box was previously built by n_m3u8dl_re, removed in the downloader consolidation.

Add ism_init.py, a dependency-free byte-level MP4 init-segment synthesizer that rebuilds ftyp+moov from the manifest CodecPrivateData, ported from yt-dlp's write_piff_header and N_m3u8DL-RE's MSSMoovProcessor with full codec parity:

- AVC (H264/AVC1/DAVC), with SPS/PPS picked by NAL type rather than position and NALUnitLengthField honored
- HEVC (HVC1/HEV1) with chroma format and bit depths parsed from the de-emulated SPS via exp-Golomb so 10-bit/HDR signals correctly, and profile/tier/level lifted from the SPS PTL
- Dolby Vision (DVHE/DVH1) as hvcC with a dvh1 sample entry
- AAC (AACL/AACH) with the AudioSpecificConfig synthesized from SamplingRate/Channels when the manifest omits CodecPrivateData
- EC-3 with a real dec3 box extracted from the WAVEFORMATEXTENSIBLE CodecPrivateData (Dolby GUID located by search, not fixed offset)
- TTML subtitles as stpp/sthd/subt, wired for fragmented-TTML tracks

CENC wrapping (encv/enca + sinf/tenc with default_KID) covers encrypted tracks: the per-sample IV size is derived from the fragment senc/saiz (PIFF override flag, payload arithmetic, saiz fallback) instead of assuming 8, and the constant-IV tenc form is supported. Read the track_ID from the first fragment's tfhd so the moov matches and the muxer does not drop samples.

Wire ISM.download_track to prepend the synthesized init before merging; unsupported codecs soft-fail to raw concatenation with a warning. Harden against real-world inputs: 2-letter/uppercase manifest language tags normalize to ISO-639-2 (und fallback), >65535 Hz sample rates no longer overflow the 16.16 field, truncated tfhd returns None, struct.error joins the soft-fail handler, and the emulation-prevention scan no longer over-strips consecutive escapes.

Add regression tests (37) covering box structure, every supported FourCC, 10-bit SPS parsing, ASC synthesis, dec3 extraction, IV-size derivation and the crash fixes. Validated structurally per codec with ffmpeg-minted fragments: shaka-packager parses synth-init+fragments with exit 0 and ffprobe reports the expected codec, including a live run against a public Smooth Streaming server.
2026-06-11 13:41:58 -06:00
imSp4rky 466bf610cc feat(drm): add native DASH ClearKey (org.w3.clearkey) support
unshackle's DASH parser only recognised Widevine and PlayReady ContentProtection, so services using W3C EME ClearKey had to fake a Widevine object and monkey-patch get_content_keys. Add a first-class ClearKeyCENC DRM type so services just implement a license callback.

- ClearKeyCENC (core/drm/clearkey_cenc.py): KID-based, no CDM/PSSH; builds the W3C JSON license request (unpadded base64url), parses the JWK Set response (dict/str/bytes), falls back to POSTing the manifest Laurl when the service returns None, decrypts via the same shaka/ mp4decrypt CENC path as Widevine
- DASH.get_drm emits ClearKeyCENC for scheme e2719d58-...; KID from own or sibling mp4protection cenc:default_KID, Laurl across dashif/legacy/ bare namespaces
- track.download dispatches prepare_drm for ClearKeyCENC; dl.prepare_drm gains a clearkey branch (cache/vault lookup, license-failure tolerated when content_keys pre-populated, vault push, export)
- Service.get_clearkey_license base callback (default None -> Laurl); drm_from_dict reconstructs ClearKeyCENC for export/import round-trip
- EXAMPLE service + config demo the callback
- Tests: tests/core/test_clearkey_cenc.py and an export round-trip case
- Docs: DRM_CONFIG.md ClearKey section
2026-06-11 12:26:57 -06:00
imSp4rky dd1633e603 test(export): pin DRM-free export round-trip through ImportService 2026-06-11 10:28:57 -06:00
imSp4rky 57a5d4269a fix(dl): export DRM-free, ClearKey, MonaLisa and server-CDM tracks
write_export now tolerates drm=None; every downloaded track is written to the --export sidecar, not just Widevine/PlayReady-licensed ones.
2026-06-11 09:36:58 -06:00
imSp4rky 0ebe33869b ci(release): add daily automated version bump and changelog workflow
Add version-bump.yml: daily git-cliff semver bump on dev (version files, uv.lock, prepended CHANGELOG entry, tag). Switch release.yml gate to compare against latest GitHub release instead of latest tag, since every bump is now tagged. Allow .github/ through the hidden-dir gitignore rule.
2026-06-10 18:48:37 +00:00
imSp4rky 5337639035 feat(logging): add debug_requests flag and aggregate subtitle-conversion logs 2026-06-09 22:26:35 -06:00