Commit Graph
743 Commits
Author SHA1 Message Date
imSp4rky 036368b3c3 fix(api): keep sub_format "original" working through the API 2026-07-16 23:22:13 -06:00
imSp4rky 15a368f7e9 perf(api): skip progress re-parse when the file is unchanged 2026-07-16 23:22:12 -06:00
imSp4rky 0b93de94a2 fix(subtitle): convert ASS to SRT in the SDH-strip fallback 2026-07-16 23:22:12 -06:00
imSp4rky e50c005106 fix(dash): license ClearKeyCENC before decrypt 2026-07-16 23:22:11 -06:00
imSp4rky 2086be5ea6 chore(scripts): add attempt-1-only tail-slow fault to downloader bench 2026-07-16 23:22:04 -06:00
imSp4rky 8fecb49605 chore(scripts): harden bench suite report capture
Speedtest failures print an exit-code note instead of an empty section,
the report file writes progressively again (utf8 rewrite at the end),
and pytest-asyncio's py3.14 deprecation warning is filtered.
2026-07-16 18:07:55 -06:00
imSp4rky b2363e11c7 chore(scripts): expand bench suite for cross-machine runs
Adds machine info to the report header, multiprocess and strided-split
runs, hedge/slow-tail cases, the rnet path, and the leak regression
tests; ps1 reports save as utf8 and expected failures no longer render
as NativeCommandError blocks.
2026-07-16 18:01:13 -06:00
imSp4rky a11fbcd328 perf(downloader): stride multiprocess chunk assignment across children
Positionally clustered slow segments no longer pile into one child:
16.4s -> 8.4s at procs=4 with a 4-segment slow tail, uniform load
unchanged.
2026-07-16 17:44:07 -06:00
imSp4rky 7408162805 test(downloader): leak regression tests and bench sweep-safety hardening
Cross-platform WinError 32 regression tests; the bench now drains
leaked workers after a failed run and relays --fast-timeouts to
--procs children.
2026-07-16 17:44:06 -06:00
imSp4rky b841388028 fix(downloader): stop leaked racers holding segment handles after a failed batch
A leaked racer could retry and reopen its .!dev once dl.py cleared the
global cancel flag for the next track (WinError 32 at cleanup). A
batch-local abort event set at teardown makes its exit unconditional.
Also join terminated mp children and detect one that dies silently.
2026-07-16 17:43:55 -06:00
imSp4rky bdef22f335 test(scripts): size-verify senv-brand MP4s that ffprobe can't parse 2026-07-16 12:59:27 -06:00
imSp4rky 5de6fe9014 Merge branch 'dev' into feat/downloader-speed
# Conflicts:
#	unshackle/core/downloaders/requests.py
#	unshackle/core/manifests/dash.py
2026-07-16 12:47:59 -06:00
imSp4rky 7ca7a6066e chore: restore memory/ to .gitignore from dev 2026-07-16 12:44:26 -06:00
imSp4rky 7eee34014e perf(dash): collapse single-URL byte-range segments into one direct download
Also throttle hedge-median recompute to 0.5s, cache replace_fields format
patterns and short-circuit when no $ token, and stream HLS merge with
copyfileobj. Adds test_dash_collapse and live A/B tooling.
2026-07-16 12:42:16 -06:00
imSp4rky 094c598ef5 perf: little improvements here and there 2026-07-16 11:02:34 -06:00
imSp4rky 78d80eb68a test(scripts): add downloader benchmark and live manifest test tooling
bench_downloader.py sweeps the unified downloader over synthetic
loopback segments (latency/rate shaping, fault injection, hedge and
sweep-safety checks) or real URLs, with --procs and --no-read1 A/B
flags. live_manifest_test.py drives the full DASH/HLS/ISM pipeline
against 11 public streams on 8 CDNs; bench-urls/ holds reproducible
real-CDN URL lists.
2026-07-16 10:27:14 -06:00
imSp4rky e06cbe3dd8 perf(downloader): use per-arrival reads only while a segment is hedge-raced
Unconditional read1 for every segmented download paid a per-TLS-record
Python iteration on all segments and capped single-process throughput
at ~160 MB/s (253 MB/s with blocking full-chunk reads, measured on a
fast CDN). A segment can only be claimed once a hedge racer exists for
it, so fine-grained claimed() checks are needed only during an actual
race: download() now takes a racing predicate and workers pass the
hedged-set membership check.
2026-07-16 10:27:14 -06:00
imSp4rky f0e0c6d39e feat(downloader): fan large segment batches across worker processes
One CPython interpreter caps sustained segment throughput at ~160 MB/s;
spawned children each run the existing thread-pool path on a contiguous
chunk of the batch, reaching NIC line rate on fast CDNs. Opt-in via
--download-processes (default 1), engaged only for 24+ segment batches.
Sessions are rebuilt per child from a picklable spec; RnetSession falls
back to a single process when it has no impersonate preset.
2026-07-16 10:26:44 -06:00
imSp4rky 2adf8bad87 fix(downloader): validate 206 on byte-range segments and force identity encoding
Segments carrying their own Range (DASH SegmentBase, HLS EXT-X-BYTERANGE) now
fail the attempt when a server ignores the Range and returns 200, instead of
writing the whole parent resource as the segment and corrupting the merge. The
resume path no longer clobbers such a slice's Range, and retries rewrite the
slice whole. All segment GETs request Accept-Encoding: identity and the rnet
path mirrors the gzip guard, so transparent CDN (de)compression can't break the
Content-Length truncation check. Adds tests pinning these plus the adaptive
start-at-cap wiring and the small-segment tail-probe gate.
2026-07-16 00:00:26 -06:00
imSp4rky 4cf7a651a0 style: apply ruff format to branch files predating the format sweep 2026-07-16 00:00:19 -06:00
imSp4rky 686d996346 ci: give the build job its own uv cache key
stops the 'unable to reserve cache' annotation when lint and build
race to save the same key
2026-07-15 15:00:49 -06:00
imSp4rky 46cf72062d ci: pin setup-uv and dependency-review to full versions
neither publishes a moving major tag anymore, so v8/v5 don't resolve
2026-07-15 14:56:37 -06:00
imSp4rky 0a6f1cbe07 ci: bump actions to node 24 versions and serialize docs deploys
checkout v7, setup-uv v8, setup-python v6, dependency-review v5; a
concurrency group queues docs deploys so parallel runs can't race the push.
2026-07-15 14:54:05 -06:00
imSp4rky e31e60a418 style: run ruff format over the codebase 2026-07-15 14:49:27 -06:00
imSp4rky d853ef9600 ci: add multi-os test, lint, and wheel build workflows
Also fixes the sdist leaving wheels empty, pins subby to its git
source in wheel metadata, and fixes docstrings that broke mkdocs --strict.
2026-07-15 14:49:19 -06:00
imSp4rky 59a7c76b2f feat(ism): harden Smooth parsing and derive fps/codec from spec fields
Regex-based UrlPattern substitution (handles {Bitrate}/{start_time} spellings), AudioTag/FourCC codec fallback, Width/Height fallbacks, integer fragment times, live-manifest and missing-language guards. Extract fps from the HEVC SPS VUI (HEVC-only; AVC timing untrusted).
2026-07-14 21:24:14 -06:00
imSp4rky 2ccae1a0ee feat(cdm): stamp remote CDM DRM type at load for reliable detection 2026-07-14 17:32:00 -06:00
imSp4rky c6ecdb605c fix(services): register modules in sys.modules so repo-cloned services can self-import and remove old benchmark that should have been deleted 2026-07-14 13:46:29 -06:00
imSp4rky b8ec9e2498 fix(tracks): keep regional language variants distinct in exact selection
by_language now applies RFC 4647 Lookup in exact mode, preferring the string-equal tag so --exact-lang separates base/regional pairs (ar/ar-EG, en/en-US) instead of collapsing them, and dedupes by id to prevent the duplicate-track-id crash. Adds a language-selection regression test.
2026-07-14 09:17:22 -06:00
imSp4rky 1cf9e5b487 docs: correct DRM, muxing, and proxy references (+ case-insensitive proxy prefix fix) 2026-07-13 09:09:46 -06:00
imSp4rky 2e14e30f2f build(mypy): enable explicit_package_bases 2026-07-12 20:26:05 -06:00
imSp4rky b32b301dcc fix(proxies): show ExpressVPN country/location counts at startup 2026-07-12 20:25:56 -06:00
imSp4rky a450184ab8 test(remote): isolate job history writes from the real api_history.jsonl 2026-07-12 10:30:30 -06:00
imSp4rky bd679d2571 docs(changelog): regenerate with git-cliff 2026-07-12 09:52:35 -06:00
imSp4rky d7a2bb6b8e fix(subs): stop segmented WebVTT merge from stripping formatting
Merge cues at the string level instead of round-tripping through
pycaption, which dropped inline tags and cue settings. STYLE and
REGION blocks are kept, duplicate boundary cues splice correctly,
and a bad timing line now skips the cue instead of crashing.

Also rank pysubs2 above SubtitleEdit for ASS to TTML/VTT since
SubtitleEdit flattens inline styling there. Documented the measured
backend fidelity table and added regression tests for the merge.
2026-07-12 09:39:39 -06:00
imSp4rky 6ef24661c8 style(dl): restore isort imports and drop dead pass 2026-07-12 08:55:01 -06:00
sp4rk.y 3326e9b72c Merge pull request #140 from CodeName393/prevent-missing-tracks-error-when-excluding-track-types
fix(dl): prevent missing tracks error when excluding track types
2026-07-12 08:50:28 -06:00
imSp4rky 6b168c774e fix(gitignore): add memory directory to .gitignore 2026-07-12 07:35:40 -06:00
imSp4rky 8769f678b1 fix(tags): point IMDbApi provider at api.tiffara.com (IMDxAPI)
Upstream migrated the free IMDxAPI from imdbapi.dev to api.tiffara.com.
Endpoints and JSON schema are unchanged, so only BASE_URL and the
user-facing docs/docstrings needed updating.
2026-07-12 07:34:47 -06:00
CodeName393 ccb2a19e7a fix(dl): prevent missing tracks error when excluding track types
- Define keep_* flags early in the track selection loop to accurately trace which track types (video, audio, subs, chapters) should be kept.
- Guard specific track filtering and validation blocks in dl.py using keep_* variables.
- Prevent fatal 'not found in video/audio/subtitle tracks' errors when using skip options like --no-video (-nv) or --no-subs (-ns).
- Simplify and clean up the trailing track cleanup logic to rely on the pre-determined keep_* flags.
2026-07-12 19:07:27 +09:00
imSp4rky f42af571a7 build(deps): cap pycaption below 2.2.27
2.2.27's WebVTT reader rewrite removes _decode and adds strict header
validation, breaking WebVTTReaderExt and HLS X-TIMESTAMP-MAP segments.
2026-07-11 18:47:35 -06:00
imSp4rky 6434fc2081 fix(dl): make audio/video track selection consistent and crash-safe
Share one best-audio selector across the episode and music paths so best/all works for songs, descriptive audio is capped to one track, and codec_priority is honored. Guard --channels and 16:9 canvas math against tracks missing channel/width metadata. Extend --best-available to video languages and full resolution misses in both normal and hybrid mode, and count only deliverable ranges when reporting missing resolutions. Rank video by resolution before bitrate, and trim group audio selections to one track per song with a warning.

BREAKING CHANGE: a missing resolution with -r HYBRID now errors without --best-available, matching non-hybrid behavior.
2026-07-11 18:41:41 -06:00
imSp4rky 575f453592 fix(dl): select only best descriptive audio track per language 2026-07-11 17:10:26 -06:00
imSp4rky c47aad6fa1 feat(tags): add OMDb metadata provider with omdb_api_key config 2026-07-11 11:46:58 -06:00
imSp4rky e7b716d79c fix(dl): restore real progress bars where progression is trackable 2026-07-11 10:51:39 -06:00
imSp4rky 20a4e6e2eb feat(tags): add imdb_api_enabled toggle, off by default 2026-07-11 09:42:52 -06:00
imSp4rky a519af1af2 fix(downloader): release hedge loser file handles before merge sweep
superseded racers now exit on the next network arrival and the pool
waits for them on clean completion, so Windows can delete stray .!dev
files at merge time (WinError 32)
2026-07-10 13:16:51 -06:00
imSp4rky 459ed1a38e fix(downloader): release hedge loser file handles before merge sweep
superseded racers now exit on the next network arrival and the pool
waits for them on clean completion, so Windows can delete stray .!dev
files at merge time (WinError 32)
2026-07-10 13:16:30 -06:00
imSp4rky 457801d8d2 feat(dl): animate merge/decrypt/mux stages with gradient pulse bars 2026-07-10 11:28:41 -06:00
imSp4rky bc8ebdfc67 perf(downloader): slow-start ramp for the adaptive worker controller
Warm up after half a tick and double the worker target while probes pay off, dropping to AIMD +2 on first plateau/error burst. 33% faster in shaped benchmarks; backoff semantics unchanged.
2026-07-09 23:17:00 -06:00