Commit Graph

571 Commits

Author SHA1 Message Date
imSp4rky
0ebf9278d1 fix(deps): bump aiohttp and pygments to resolve 11 security vulnerabilities 2026-04-02 11:44:54 -06:00
imSp4rky
bb0a800ab6 docs(api): update --export from string path to boolean flag
Update API docs, Swagger schema, handlers, and example config to reflect --export as a boolean flag that auto-generates export files in the configurable exports directory.
2026-04-02 11:27:39 -06:00
imSp4rky
fabc96ba1b feat(dl): change --export flag with manifest URL, subtitles, and track info
changed --export flag to export decryption keys, manifest URLs, subtitle URLs, and track info to a JSON file in the configurable exports directory. Manifest URLs are captured from DASH, ISM, and HLS parsers and propagated through the Tracks system via a new manifest_url attribute. Deduplicate Widevine/PlayReady export logic into a shared _write_export helper with dedicated EXPORT_LOCK. Add Tracks.filter() method that preserves metadata when filtering tracks by predicate.
2026-04-02 10:29:22 -06:00
imSp4rky
655e4197c3 fix(session): native rnet proxy support and cookie compat layer
Proxies now use rnet's native `proxies` parameter (`List[rnet.Proxy]`) with in-place `client.update()` for live proxy changes. Client is created lazily on first request, allowing headers, cookies, and proxies to be configured freely before any connection is established.

Cookie adapter supports RequestsCookieJar-compatible methods (jar, get_dict, clear) for seamless interop with cookie persistence.
2026-04-01 17:08:25 -06:00
imSp4rky
3aaca77c48 feat(serve): add service allowlist for global and per-user access control
Allow server operators to restrict which services are exposed via the API using serve.services (global) and serve.users.<key>.services (per-user).
Effective access is the intersection of both when both are set. Unlisted services return the same error as non-existent ones to prevent enumeration.
2026-03-31 23:06:07 -06:00
imSp4rky
8a714d6455 fix(template): detect folder spacer from template separators, not raw string
The previous heuristic checked the raw template string for dots, which could match dots inside variable names or title content, causing
Plex-friendly folder names to incorrectly use dots as spacers. Now strips template variables first and checks only the separators between them to determine user intent.
2026-03-31 09:13:44 -06:00
imSp4rky
5e801580a3 fix(hybrid): read actual HDR metadata for HDR10+ to DV conversion
The L6 metadata in convert_hdr10plus_to_dv was hardcoded to 1000 nits max mastering display luminance. Now probes the source stream via ffprobe to extract the real mastering display and content light level values, preserving accurate luminance for sources mastered above 1000 nits.
2026-03-30 17:02:06 -06:00
imSp4rky
47b3390bd0 feat(dl): allow --slow to accept custom delay range
--slow now supports an optional MIN-MAX range (e.g., --slow 20-40) for custom delay between title downloads. Bare --slow retains the original 60-120s default. Minimum delay enforced at 20 seconds.
2026-03-30 16:25:29 -06:00
imSp4rky
c7fd2a904c Merge branch 'dev' of https://github.com/unshackle-dl/unshackle into dev 2026-03-29 23:46:28 -06:00
imSp4rky
13ebdddaf4 Merge branch 'main' into dev 2026-03-29 23:45:53 -06:00
imSp4rky
d37a1a514f chore(gitignore): ignore binary files in unshackle/binaries/ 2026-03-30 05:27:37 +00:00
imSp4rky
ccc494be06 Merge branch 'feat/unified-downloader' into dev
# Conflicts:
#	unshackle/core/manifests/hls.py
2026-03-30 05:24:09 +00:00
imSp4rky
d3594ca67c fix(remote): forward track selection params to server and improve error display
Forward range, vcodec, quality, and best_available from client to server so services fetch the correct manifests (e.g. HDR10 instead of SDR).
Exit with error when no video tracks match the requested range filter.
Use Rich logger for remote API errors instead of plain click.ClickException.
2026-03-30 05:20:32 +00:00
imSp4rky
1a6f2c5b7e fix(serve): allow remote-only mode without output_template and fix CORS/auth for Cloudflare
Move output_template validation from config init to dl command so serve, search, and other non-download commands work without it configured. Fix CORS header to use X-Secret-Key (matching actual auth header). Exempt /api/health from auth for Cloudflare tunnel health checks.
2026-03-30 04:36:23 +00:00
imSp4rky
3d5e46a2e3 feat(hls): probe TS segments for resolution and codec when master playlist lacks RESOLUTION/CODECS tags
Some HLS services serve master playlists without RESOLUTION or CODECS attributes, leaving video tracks with no width/height/codec info which causes crashes in by_resolutions() quality selection.

After parsing tracks in to_tracks(), any video track missing resolution or codec is now probed by fetching the first 8KB of its first TS segment and parsing the H.264/H.265 SPS NAL unit to extract the actual width, height, and codec. This approach mirrors how cat-catch/hls.js determines resolution from the media data rather than relying on playlist metadata.
2026-03-29 15:55:53 -06:00
Andy
5a3ac81ff9 feat(session): translate requests 'data' kwarg to rnet equivalents for compatibility 2026-03-26 16:36:35 -06:00
Andy
e323f6f3b3 feat(template): add configurable folder naming via output_template.folder (#94)
Adds an optional `folder` key under `output_template` to customize output folder names using the same template variables as file naming.
2026-03-25 21:42:47 -06:00
Andy
10cca7d0ea fix(sanitize): restore parentheses stripping in filename sanitization (#93)
Commit 6ce7b6c accidentally removed () from the unsafe-characters regex
2026-03-25 19:46:00 -06:00
Andy
7358619a40 fix(deps): bump PyJWT minimum to 2.12.0 for CVE-2026-32597
PyJWT <= 2.11.0 accepts unknown `crit` header extensions in violation of RFC 7515 §4.1.11. Bump lower bound to 2.12.0 which includes the fix.
2026-03-25 15:16:21 -06:00
Andy
b524585d78 fix(drm): add zero-KID fallback for mp4decrypt and clear HLS track.drm after download
mp4decrypt silently copies files unchanged when the tenc box default KID is all zeros, since none of the real KID:KEY pairs match. Add zero-KID fallback entries to both Widevine and PlayReady mp4decrypt methods, matching what Shaka Packager already does.

Also clear track.drm after HLS download when decryption was performed, preventing unnecessary double-decryption. DASH and URL descriptors already did this.
2026-03-25 15:06:41 -06:00
Andy
fe1ccd085c Revert "fix(drm): add track ID fallback for mp4decrypt CBCS zero-KID content"
This reverts commit 23466cae8b.
2026-03-25 14:39:08 -06:00
Andy
23466cae8b fix(drm): add track ID fallback for mp4decrypt CBCS zero-KID content
Some CBCS-encrypted content has an all-zeros default_KID in the tenc box while the real KID is only in the PSSH boxes. mp4decrypt matches keys against the tenc KID, so it silently skips decryption when the provided KID doesn't match. This adds a track ID-based key fallback when a zero KID is detected, matching the existing shaka-packager zero-KID fallback behavior.
2026-03-25 14:36:26 -06:00
Andy
c930abc6fd fix(subtitle): decompress gzip/zlib responses for subtitle downloads
The requests downloader used decode_content=False on raw socket reads, which skipped HTTP content-encoding decompression. Subtitle files served with Content-Encoding: gzip were saved as raw compressed bytes, then mangled by try_ensure_utf8 falling back to CP1252 decoding.

Remove decode_content=False from the raw read path — the speed gain comes from raw.read() itself, not from skipping decompression. Also add gzip/zlib magic byte detection in try_ensure_utf8 as a safety net for any edge cases where compressed data reaches encoding detection.
2026-03-24 17:44:23 -06:00
Andy
99be88dc08 feat(session): replace curl_cffi with rnet for TLS-fingerprinted HTTP
Replace CurlSession (curl_cffi) with RnetSession powered by rnet (Rust/BoringSSL). Benchmarks show 3.5x faster segmented downloads (1.06 GB/s vs 304 MB/s) and 16% faster single-file downloads with near-zero TLS fingerprinting overhead.

- Add RnetSession wrapper with requests-compatible API (headers, cookies, proxies, retry logic, prepared requests)
- Add RnetResponse wrapper normalizing rnet quirks (status_code as int, text as property, bytes-to-str headers, iter_content re-chunking)
- Replace CurlSession isinstance checks across manifests, tracks, DRM
- Update downloader with rnet native streaming path and byte-based progress tracking for accurate Rich progress bars
- Add speed display column to Rich progress bar (DASH/HLS/URL prefix)
- Add rnet dependency, services use exact preset names (e.g. OkHttp4_12)
2026-03-24 10:08:17 -06:00
Andy
6840944738 perf(downloader): optimize hot loop and threading efficiency
Replace list.pop(0) with deque.popleft() for O(1) speed tracker eviction, skip urllib3 decode chain with decode_content=False on raw reads, use running total instead of sum() for progress reporting, add explicit stream.close() on CurlSession path, replace busy-poll loop with concurrent.futures.wait(FIRST_COMPLETED), skip ThreadPoolExecutor for single-URL downloads, and DRY up duplicated raw/iter_content progress logic into a unified chunk iterator.
2026-03-23 18:17:12 -06:00
Andy
006d080416 feat(downloader): optimize download throughput with Queue-based threading and raw reads
Fix critical bug where ThreadPoolExecutor was not actually parallelizing downloads (generator functions returned instantly, I/O ran on main thread).

Performance improvements:
Queue-based event dispatch: workers consume generators in threads, push events to a thread-safe Queue for truly parallel segment downloads
Raw socket reads (resp.raw.read) for requests.Session — 30-35% faster than iter_content, with iter_content fallback for CurlSession
File pre-allocation via truncate when Content-Length is known
Hot loop caching: time.time, f.write, stream.raw.read cached as locals
HTTPAdapter connection pooling mounted on passed sessions for reuse
2026-03-23 17:20:26 -06:00
Andy
732709d3a9 feat(remote): interactive auth handshake, server CDM, cache round-trip, and serve remote-only mode
Add InputBridge for interactive client-server authentication (OTP, device codes, PINs) with async auth via asyncio.to_thread, prompt polling endpoints, and cancellation support.
Server CDM mode detects CDM type from config.cdm per-service, resolves keys server-side, and returns DRM type to client for correct display. Cache files round-trip between client and server on session create/delete. Vault loading fixed for server-side key caching.
HLS/ISM/DASH DRM extraction in license handler. Serve --remote-only mode exposes only session endpoints. Clean connection error handling for unreachable servers.
2026-03-22 22:44:36 -06:00
Andy
1ad226fbcf feat(remote): server vault lookups, service CDM mapping, key display, and service param forwarding 2026-03-20 21:13:56 -06:00
Andy
51d6921eaf fix(drm): include external KID in PSSH when it differs from existing KIDs
When the DASH manifest provides a cenc:default_KID that differs from the PSSH's embedded KIDs, the external KID must be added to the PSSH so the license server returns keys for both. Previously, the PSSH was only modified when all existing KIDs were placeholders, causing a CEKNotFound error when the track's actual encryption KID wasn't in the license response.
2026-03-20 12:48:49 -06:00
Andy
dc197af29e feat(dash): refactor segment extraction and add content period validation 2026-03-20 12:47:49 -06:00
Andy
561a609040 fix(audio): support 'xheaac' profile 2026-03-20 10:34:49 -06:00
Andy
a21c32df5d feat(dl): add --vbitrate-range and --abitrate-range options for bitrate range selection
Allow users to specify a bitrate range (e.g., --abitrate-range 300-400) to filter tracks within that range, with downstream selection picking the highest per language. Mutually exclusive with the existing --vbitrate/--abitrate exact match options.
2026-03-20 10:34:26 -06:00
Andy
2f721266f0 Merge branch 'main' into dev 2026-03-19 20:36:55 -06:00
Andy
faaaf08bd5 fix(drm): add zero-KID fallback for mp4decrypt and clear HLS track.drm after download
mp4decrypt silently copies files unchanged when the tenc box default KID is all zeros, since none of the real KID:KEY pairs match. Add zero-KID fallback entries to both Widevine and PlayReady mp4decrypt methods, matching what Shaka Packager already does.

Also clear track.drm after HLS download when decryption was performed, preventing unnecessary double-decryption. DASH and URL descriptors already did this.
2026-03-19 18:43:43 -06:00
Andy
c323db9481 feat(downloader): consolidate into unified requests-based downloader
Replace 4 separate downloaders (requests, curl_impersonate, aria2c, n_m3u8dl_re) with a single optimized requests downloader with adaptive chunk sizing and session passthrough for TLS fingerprinting support.

- Adaptive chunk sizing (512KB-4MB) based on content length, up from fixed 1KB
- Buffered writes (1MB buffer) for improved I/O throughput
- Session passthrough: accepts both requests.Session and CurlSession
- Per-call speed tracking with rolling window (fixes cross-track speed bleed)
- Worker count default capped at 16
- Removed all downloader.__name__ special-casing from manifest parsers
- Removed aria2c/curl_impersonate/n_m3u8dl_re downloader modules
- Deprecated downloader config key in unshackle.yaml
2026-03-19 18:13:43 -06:00
Andy
4c55f7af5b docs: update API and configuration documentation with example service tags 2026-03-19 12:55:39 -06:00
Andy
e9dbe3f0ac Merge branch 'dev' of https://github.com/unshackle-dl/unshackle into dev 2026-03-19 12:38:40 -06:00
Andy
b3b67b0c96 feat(session): add IP validation for session access and enhance session management 2026-03-19 12:38:33 -06:00
Sp5rky
313efe5b03 Rename remote-services-flow.md to REMOTE-SERVICES-FLOW.md 2026-03-19 12:13:45 -06:00
Andy
9d21d8a246 feat(remote): fetch service list and CLI options from server for --remote help
When using --remote -h, fetch available services from the server's /api/services endpoint instead of listing local services.
2026-03-19 00:06:36 -06:00
Andy
61ee5cb10a feat(remote): add zlib compression for API payloads and gzip transport
Compress manifest XML, cache files, and cookies with zlib before base64 encoding, reducing /tracks payload by 83-95%. Add configurable compression_level in serve config (default: 1, fast).

Enable aiohttp gzip middleware for transport-level compression on all JSON responses. Set User-Agent to unshackle/<version> for remote client requests. Add remote-aware status messages for auth, titles, and tracks.
2026-03-18 23:47:06 -06:00
Andy
b420e2bd08 refactor(remote): deduplicate CDM loading, proxy resolution, and license handling 2026-03-18 22:54:11 -06:00
Andy
b1a5f8f37f feat(remote): add server-CDM mode, manifest transfer, and region-aware proxy
Server-side:
- Add server_cdm mode: server handles full CDM licensing using its own devices, returns KID:KEY pairs instead of raw license bytes
- Support batch license resolution for multiple tracks in one request
- Extract DRM from manifest ContentProtection when track.drm is empty
- Serialize DASH/ISM manifest XML as base64 in /tracks response
- Include session cookies/headers and server_cdm_type in /tracks response
- Detect server CDM type from actual track DRM + configured devices
- Check server region against client_region to skip unnecessary proxy
- Support decrypt_labs and custom_api remote CDMs for both WV and PR

Client-side:
- Re-parse DASH/ISM manifests locally from base64 to populate track.data
- Match remote tracks to re-parsed tracks by ID with attribute fallback
- Copy DRM objects from re-parsed manifests to remote tracks
- Pre-fetch keys via resolve_server_keys() before downloads start
- Fallback per-track licensing via _proxy_license during download
- Apply session cookies/headers from server for CDN access
- Apply downloader/decryption config directly for remote services
- Preserve pre-injected content_keys during DASH DRM override
- Skip redundant CDM calls when all KIDs already have keys

Docs:
- Add comprehensive remote-services-flow.md with Mermaid diagrams covering proxy mode, server-CDM mode, manifest transfer, and config
2026-03-18 19:30:59 -06:00
Andy
7fafdf024c fix(drm): preserve original PSSH for content_id-based Widevine manifests 2026-03-18 11:06:04 -06:00
Andy
96911cb626 Merge branch 'dev' of https://github.com/unshackle-dl/unshackle into dev 2026-03-17 16:54:35 -06:00
Sp5rky
e994c483c5 Merge pull request #92 from unshackle-dl/feat/remote-services
feat(dl): add --remote flag for remote server downloads
2026-03-17 16:53:59 -06:00
Andy
f86d4e7937 docs(dl): add comprehensive list of available dl keys and their descriptions 2026-03-17 09:27:52 -06:00
Andy
d4bc095f96 fix: update actions/checkout to v5 in release workflow 2026-03-17 09:16:46 -06:00
Andy
79e8184474 ci: enable manual triggering of release workflow 4.0.0 2026-03-17 09:10:50 -06:00
Andy
178eed9236 ci: add GitHub Actions release workflow for major/minor versions 2026-03-17 09:08:20 -06:00