kenzuya/unshackle
1
0
Fork 2
mirror of https://github.com/unshackle-dl/unshackle.git synced 2026-11-12 12:05:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(downloader): fan large segment batches across worker processes

Browse Source 
One CPython interpreter caps sustained segment throughput at ~160 MB/s;
spawned children each run the existing thread-pool path on a contiguous
chunk of the batch, reaching NIC line rate on fast CDNs. Opt-in via
--download-processes (default 1), engaged only for 24+ segment batches.
Sessions are rebuilt per child from a picklable spec; RnetSession falls
back to a single process when it has no impersonate preset.
This commit is contained in:
imSp4rky
2026-07-16 10:26:44 -06:00
parent 2adf8bad87
commit f0e0c6d39e
10 changed files with 277 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide/cli-reference.md
+1
View File
@@ -172,6 +172,7 @@ Keep only certain track types, or skip certain track types. Attachments are alwa
|---|---|---|
| `--workers` | downloader default | Per-track download threads. |
| `--adaptive-workers` | off | Scale per-track segment workers dynamically (up to the `--workers` cap) based on measured CDN throughput and errors. |
| `--download-processes` | `1` | Split large segment batches (24+) across this many download processes to exceed the single-process throughput cap on multi-gigabit connections. |
| `--downloads` | `1` | Number of tracks downloaded concurrently. |
| `--no-cache` | off | Bypass the title cache. |
| `--reset-cache` | off | Clear the title cache. |