kenzuya/unshackle
1
0
Fork 2
mirror of https://github.com/unshackle-dl/unshackle.git synced 2026-11-14 13:05:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(downloader): fan large segment batches across worker processes

Browse Source 
One CPython interpreter caps sustained segment throughput at ~160 MB/s;
spawned children each run the existing thread-pool path on a contiguous
chunk of the batch, reaching NIC line rate on fast CDNs. Opt-in via
--download-processes (default 1), engaged only for 24+ segment batches.
Sessions are rebuilt per child from a picklable spec; RnetSession falls
back to a single process when it has no impersonate preset.
This commit is contained in:
imSp4rky
2026-07-16 10:26:44 -06:00
parent 2adf8bad87
commit f0e0c6d39e
10 changed files with 277 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide/cli-reference.md
+1
View File
@@ -172,6 +172,7 @@ Keep only certain track types, or skip certain track types. Attachments are alwa
|---|---|---|
| `--workers` | downloader default | Per-track download threads. |
| `--adaptive-workers` | off | Scale per-track segment workers dynamically (up to the `--workers` cap) based on measured CDN throughput and errors. |
| `--download-processes` | `1` | Split large segment batches (24+) across this many download processes to exceed the single-process throughput cap on multi-gigabit connections. |
| `--downloads` | `1` | Number of tracks downloaded concurrently. |
| `--no-cache` | off | Bypass the title cache. |
| `--reset-cache` | off | Clear the title cache. |
docs/guide/downloading.md
+1
View File
@@ -463,6 +463,7 @@ Two related flags:
| --- | --- |
| `--workers N` | Threads used per track for segment downloads. Default depends on the downloader. |
| `--adaptive-workers` | Opt-in: start with a moderate per-track worker count and ramp it up or back off based on measured CDN throughput and errors, capped at `--workers`. Off by default (fixed worker count). |
| `--download-processes N` | Opt-in: split a large segment batch across `N` download processes, each with its own worker pool. A single process tops out around 1.3 Gb/s (Python interpreter limit); `2` reaches 2.5GbE line rate on fast CDNs. Only engages for batches of 24+ segments. Default `1`. |
| `--downloads N` | Number of tracks downloaded concurrently. Default `1`. |
| `--slow [MIN-MAX]` | Add a delay between titles to look more like a real device. `--slow` alone means 60-120s; `--slow 20-40` sets a custom range. Minimum 20s. |
| `--no-cache` | Bypass the title cache for this download. |