Commit Graph
36 Commits
Author SHA1 Message Date
imSp4rky bfda31ce4a fix(proxy): keep service traffic on --proxy and warn about requests that bypass it
--proxy now overrides HTTP(S)_PROXY and the Windows system proxy while keeping the env CA bundle. Core fallback sessions use the run's proxy, and a warn-only audit hook names service code that sends a request or opens a raw socket outside the proxy; a session with proxy_exempt = True opts out.
2026-10-16 16:12:42 -06:00
imSp4rky 89f433963a fix(session): log "Retrying" only when a retry follows
A per-call max_retries=0 logged "Retrying... (1/0)" before raising, and the last of the default retries logged "(6/5)".
2026-10-15 14:48:20 -06:00
imSp4rky e7bb825def perf(manifests): decode manifests without charset detection
Parse an MPD from bytes and read every M3U8 playlist as UTF-8 unless the server sent a charset. Fixes mojibake on text/* responses.
2026-10-13 16:11:13 -06:00
imSp4rky 22cc739e83 fix(downloader): correct truncation, ranged sizing, and session carry
Check a sequential download against the complete length whenever one is known, and settle the unknown-total case on the 416. An empty body of unknown length, a 206 with no complete length, and a 206 starting past the resume offset all passed the size checks before and were renamed into place. Four restart paths spent no attempt, so a host that always answered the same way cycled forever.
2026-09-06 20:03:08 -06:00
imSp4rky 66971230c3 docs: apply simplified technical english across docs and docstrings. 2026-08-19 16:31:06 -06:00
imSp4rky 29d0828c33 style: clean up prose 2026-08-19 13:45:58 -06:00
imSp4rky 511f96d8db docs: strip non-essential comments, fold key facts into docstrings 2026-08-16 23:32:02 -06:00
imSp4rky dd9dafef91 fix(downloader): harden ranged reads, teardown and multiprocess sessions
Validates the 206 range end, bounds the DASH collapse, and preserves cookie domains and pool size in spawned children.
2026-08-16 17:07:28 -06:00
imSp4rky ee9d3edfe3 Merge branch 'dev' into feat/downloader-speed
# Conflicts:
#	unshackle/core/api/download_manager.py
#	unshackle/core/downloaders/requests.py
#	unshackle/core/manifests/dash.py
#	unshackle/core/session.py
#	unshackle/core/utils/webvtt.py
2026-08-16 12:43:01 -06:00
imSp4rky 21d5f1c5e1 refactor(core)!: drop leading underscores from internal definitions
Renames 314 defs and 41 module/class-level constants. Keeps 11 names underscored where the stripped form collides in scope, plus 5.5.0 shims for the three symbols external service repos call.
2026-08-15 22:07:41 -06:00
imSp4rky 9466db37a6 style: tighten comments added on this branch 2026-08-07 11:43:13 -06:00
imSp4rky d91159fb37 docs(session): correct stale pool-size comment
pool_max_idle_per_host no longer matches the downloader worker cap;
POOL_MAX_SIZE was deliberately raised above it so hedge racers and
tail-boost parts do not park threads under pool_block=True. Point the
comment at the constant so the value is not reverted to the worker cap.
2026-08-07 11:36:31 -06:00
imSp4rky cfae8f8882 Merge branch 'dev' into feat/downloader-speed
Catch the branch up with dev (45 commits) so testers can run the perf work
against current dev.

Conflict resolutions:
- downloaders/requests.py: keep the branch's racing-gated read1, adaptive
  controller speed reporting and byte-range slice guards; take dev's
  content-encoding checks (raw reads are skipped on an encoded body) and its
  rolling SpeedWindow for the no-controller speed fallback. The single-URL
  ranged-parallel gate now requires no speed limit and no caller Range.
- service.py / session.py: keep the branch's TimeoutHTTPAdapter and retry
  config, raise POOL_MAX_SIZE to 64 so dev's pool-starvation fix survives
  (pool_block=True stalls threads on a pool smaller than the in-flight count).
- docs: dev split configuration.md into per-area pages, so the branch's rnet
  network option rows moved to reference/configuration/network.md.
2026-08-07 10:45:47 -06:00
imSp4rky b54aadea7a fix(downloader): honor Retry-After with cancel-safe backoff, add fault-injection bench and tests 2026-07-18 20:21:30 -06:00
imSp4rky f0e0c6d39e feat(downloader): fan large segment batches across worker processes
One CPython interpreter caps sustained segment throughput at ~160 MB/s;
spawned children each run the existing thread-pool path on a contiguous
chunk of the batch, reaching NIC line rate on fast CDNs. Opt-in via
--download-processes (default 1), engaged only for 24+ segment batches.
Sessions are rebuilt per child from a picklable spec; RnetSession falls
back to a single process when it has no impersonate preset.
2026-07-16 10:26:44 -06:00
imSp4rky 2f8492ebaf feat(downloader): adaptive worker scaling, tail boost, session parity, resume fixes
Opt-in --adaptive-workers (AIMD scaling + tail range-split), request/rnet session parity, and fixes for resume poisoning, truncated-segment acceptance, 416 handling, and byte-range-slice corruption. Tests + docs.
2026-07-06 11:25:29 -06:00
CodeName393 774c1a4555 feat(core): add debug logging for rnet requests (#135)
- Implement request logging in RnetSession when debug_requests is enabled.
- Log requests with the actual HTTP version and size in urllib3 style.
2026-07-05 11:06:28 -06:00
imSp4rky ef2e880c7f perf(downloader): cut per-segment overhead, bound stalled connections
Segment tmp-file+rename scheme, request timeouts, hedged tail requests, streamed ISM/DASH merges, rnet client knobs, curl_impersonate -> network.
2026-07-02 09:20:03 -06:00
imSp4rky 538deac63e fix(session): proxy auth on http targets and string params support
Drop the manually-set Proxy-Authorization header in the Service base class. It was malformed (base64 of user:pass with no "Basic" scheme) and redundant: both rnet (Proxy.all) and requests authenticate from the credentials embedded in the proxy URL. The broken header was tunnelled harmlessly on HTTPS (CONNECT) but handed to the proxy on plaintext-http forward requests, causing HTTP 407 (e.g. http MPD/segment URLs behind an authenticated geofence proxy).

Also make RnetSession._build_url accept the same params shapes as requests (mapping, sequence of pairs, or a pre-built str/bytes query). urlencode() previously raised TypeError on a string params value.
2026-05-30 18:13:55 +00:00
CodeName393 bddb305c5d feat(session): Optimize header handling in session requests
Removed redundant conversion of headers to dict for requests.
2026-04-25 14:53:07 +09:00
CodeName393 9dbdf9804f Fix(session): header handling in session request method
Ensure headers are converted to a dictionary if provided.
2026-04-24 23:54:24 +09:00
imSp4rky 50d2b127ec style: apply ruff linting fixes across codebase 2026-04-08 22:09:19 -06:00
imSp4rky 655e4197c3 fix(session): native rnet proxy support and cookie compat layer
Proxies now use rnet's native `proxies` parameter (`List[rnet.Proxy]`) with in-place `client.update()` for live proxy changes. Client is created lazily on first request, allowing headers, cookies, and proxies to be configured freely before any connection is established.

Cookie adapter supports RequestsCookieJar-compatible methods (jar, get_dict, clear) for seamless interop with cookie persistence.
2026-04-01 17:08:25 -06:00
Andy 5a3ac81ff9 feat(session): translate requests 'data' kwarg to rnet equivalents for compatibility 2026-03-26 16:36:35 -06:00
Andy 99be88dc08 feat(session): replace curl_cffi with rnet for TLS-fingerprinted HTTP
Replace CurlSession (curl_cffi) with RnetSession powered by rnet (Rust/BoringSSL). Benchmarks show 3.5x faster segmented downloads (1.06 GB/s vs 304 MB/s) and 16% faster single-file downloads with near-zero TLS fingerprinting overhead.

- Add RnetSession wrapper with requests-compatible API (headers, cookies, proxies, retry logic, prepared requests)
- Add RnetResponse wrapper normalizing rnet quirks (status_code as int, text as property, bytes-to-str headers, iter_content re-chunking)
- Replace CurlSession isinstance checks across manifests, tracks, DRM
- Update downloader with rnet native streaming path and byte-based progress tracking for accurate Rich progress bars
- Add speed display column to Rich progress bar (DASH/HLS/URL prefix)
- Add rnet dependency, services use exact preset names (e.g. OkHttp4_12)
2026-03-24 10:08:17 -06:00
Andy e02aa66843 feat(dl): add --worst flag and SHIELD OkHttp fingerprint preset
Add --worst CLI flag to select the lowest bitrate video track within a specified resolution (e.g. --worst -q 720). Requires -q/--quality.
Add shield_okhttp TLS fingerprint preset for NVIDIA SHIELD Android TV with OkHttp 4.11 JA3 signature.
2026-03-11 13:59:07 -06:00
Andy 86a7a95b6f docs: update and correct configuration documentation 2026-02-24 23:50:33 -07:00
Andy 983fd18d53 fix(session): reduce default max_retries from 10 to 5 for CurlSession
fix(service): reduce max_retries from 15 to 5 for HTTPAdapter
2026-02-23 12:59:07 -07:00
Andy 55db8da125 refactor: remove unnecessary underscore prefixes from function names 2025-11-08 22:53:47 +00:00
Andy f979e94235 fix(session): remove padding extension from OkHttp JA3 fingerprints
Remove extension 21 (TLS padding) from okhttp4 and okhttp5 JA3 strings to resolve SSL/TLS handshake failures.
2025-11-03 05:32:57 +00:00
Andy f1fe940708 fix(session): update OkHttp fingerprint presets 2025-11-03 03:16:54 +00:00
Andy 565b0e0ea7 feat(session): add custom fingerprint and preset support
Add support for custom TLS/HTTP fingerprints to session() function, enabling services to impersonate Android/OkHttp clients instead of just browsers.
2025-11-03 01:15:49 +00:00
Andy 3571c5eb3c style: apply ruff formatting fixes 2025-10-23 18:11:30 +00:00
Andy 5384b775a4 refactor(session): modernize type annotations to PEP 604 syntax 2025-10-20 21:09:19 +00:00
stabbedbybrick 1409f93de5 feat: add retry handler to curl_cffi Session 2025-10-20 18:28:12 +02:00
Andy 35efdbff6d feat: add curl_cffi session support with browser impersonation
Add new session utility with curl_cffi support for anti-bot protection
Update all manifest parsers (DASH, HLS, ISM, M3U8) to accept curl_cffi sessions
Add browser impersonation support (Chrome, Firefox, Safari)
Fix cookie handling compatibility between requests and curl_cffi
Suppress HTTPS proxy warnings for better UX
Maintain full backward compatibility with requests.Session
2025-09-25 06:27:14 +00:00