Compare commits

...
Author SHA1 Message Date
imSp4rky 83514d12fe fix(core): cut user-facing errors and warnings to one line 2026-10-23 20:23:05 -06:00
imSp4rky dfc99783b5 fix(downloader): report spent segment retries in one line without the URL
The session message names the host, not the signed URL, and a segment that fails every attempt prints one line with no traceback. API job errors come from the exception.
2026-10-23 19:47:12 -06:00
imSp4rky d8663b0dad fix(tracks): flag every encrypted sample entry type after decrypt 2026-10-23 17:14:12 -06:00
imSp4rky 9f054269db fix(vaults): copy no vault key from a clear track
kid_windows returns an empty map for an MP4 with no encrypted sample entry or protecting senc, so the key check judges, flags and copies nothing for it.
2026-10-23 16:59:40 -06:00
imSp4rky 3345422790 fix(drm): load a bare PlayReady WRMHEADER from DASH, HLS and ISM manifests
PlayReady.from_pssh_b64 wraps a bare UTF-16LE WRMHEADER in a PlayReady Object and keeps that as pssh_b64. DASH and ISM log and skip a PlayReady entry they cannot parse.
2026-10-23 15:55:01 -06:00
imSp4rky cc4bf19706 fix(vaults): stop false bad-key flags from a failed decode check
Judge a vault KID only on a file whose KID map holds it, tell remote vaults of a bad pair only once another key replaces it, and let the server CDM unflag a pair it returns; the server never flags a CDM-licensed pair. Log FFmpeg stderr on a failed check and name the missing CDM when none is set.
2026-10-23 15:31:26 -06:00
imSp4rky eeaf16cd83 fix(remote): keep each session cookie's domain, path and secure flag
The client set server cookies with no domain and sent all of them to every host, so a CDN refused segment requests with 400 header too large. rnet sessions now send cookies too.
2026-10-23 09:42:01 -06:00
imSp4rky af69e8aad4 feat(proxy): try bare --proxy providers in proxy_providers key order
Basic stays first and Control D last; the rest follow their config key order. Geofence and proxy_map lookups go through pick_proxy and skip a provider that fails.
2026-10-23 08:20:39 -06:00
imSp4rky 71f293161e fix(cfg): show only the duplicate-key warning, once 2026-10-22 20:43:40 -06:00
imSp4rky 3e9c26bd46 refactor(config): log the duplicate-key warning without the python warning prefix 2026-10-22 20:16:35 -06:00
imSp4rky 9dd97bcd25 feat(config): warn with the lines of a duplicate key in any config load 2026-10-22 20:12:59 -06:00
imSp4rky f83a594c91 fix(cfg): give the lines of each duplicate config key instead of crashing 2026-10-22 19:45:46 -06:00
72 changed files with 1415 additions and 710 deletions
+1 -1
View File
@@ -42,7 +42,7 @@ This value becomes the one and only API key that the server accepts. The server
If you start `unshackle serve` with no `api_secret` configured, and you have *not* passed `--no-key`, the command fails immediately with:
```
API secret key is not configured. Please add 'api_secret' to the 'serve' section in your config.
No 'api_secret' in the 'serve' section of your config
```
Choose a long, unpredictable value. The API key is the only thing standing between the internet and your CDM and download endpoints.
+6 -3
View File
@@ -1096,6 +1096,7 @@ Get tracks and chapters for a specific title within the remote session. Body: `{
"track_manifests": [ { "track_id": "s-1", "type": "dash", "url": "https://...", "data": "<base64 zlib xml>" } ],
"session_headers": { "User-Agent": "..." },
"session_cookies": { "sid": "..." },
"session_cookie_jar": [ { "name": "sid", "value": "...", "domain": ".example.com", "path": "/", "secure": true } ],
"server_cdm_type": "widevine"
}
```
@@ -1115,11 +1116,13 @@ Get the download URL, DRM info, headers, cookies, and raw track data for the sel
```json
{
"tracks": {
"v-1": { "descriptor": "DASH", "url": "https://...", "drm": [ ... ], "headers": { ... }, "cookies": { ... }, "data": { ... } }
"v-1": { "descriptor": "DASH", "url": "https://...", "drm": [ ... ], "headers": { ... }, "cookies": { ... }, "cookie_jar": [ ... ], "data": { ... } }
}
}
```
`cookies` gives names and values only. `cookie_jar` gives the same cookies with their domain, path and `secure` flag, in the `session_cookie_jar` form of the tracks response.
| Status | Error code | Meaning |
| --- | --- | --- |
| `200` | - | Segment info returned. |
@@ -1215,7 +1218,7 @@ A slow answer arrives as a [slow response](#slow-responses).
### `POST /api/session/{session_id}/keys/bad`
Report a content key from `vault_keys` that did not decrypt the track. The server flags the pair in its local vaults and reports it to the vault that served it, and the next licence for that KID reaches the CDM. The client never learns the server vault names.
Report a content key from `vault_keys` that did not decrypt the track. The server flags the pair in its local vaults and reports it to the vault that served it, and the next licence for that KID reaches the CDM. When the CDM returns the same pair, the server removes the flag. The client never learns the server vault names.
```json
{ "kid": "<kid_hex>", "key": "<key_hex>" }
@@ -1223,7 +1226,7 @@ Report a content key from `vault_keys` that did not decrypt the track. The serve
| Status | Error code | Meaning |
| --- | --- | --- |
| `200` | - | The pair is flagged. |
| `200` | - | `{"flagged": true}`: the pair is flagged. `{"flagged": false}`: the CDM licensed the pair, so it is not flagged. |
| `400` | `INVALID_INPUT` | This remote session was not served that pair. |
| `404` | `SESSION_NOT_FOUND` | Unknown remote session. |
+11 -7
View File
@@ -214,8 +214,10 @@ hood the client walks a remote session through its lifecycle.
downloading. A service can build an AdaptationSet of its own that the served
manifest does not contain. Each such track gets a one-AdaptationSet MPD in
`track_manifests`, and the client re-parses that instead. Any
`session_headers` / `session_cookies` the server used are merged into the
client's local HTTP session.
`session_headers` and cookies the server used are merged into the
client's local HTTP session. Each cookie in `session_cookie_jar` keeps its
domain, path and `secure` flag, so it goes only to its own hosts. `session_cookies` carries
the same cookies as names and values only, for older clients.
The tracks response repeats the title. A service can rewrite title fields inside
`get_tracks`, so the client takes this copy in place of the one from the title
@@ -301,9 +303,11 @@ There are two ways DRM keys get resolved, chosen by the client's `server_cdm` fl
decodes, and a wrong pair goes back to `POST /api/session/{id}/keys/bad`. The
server flags the pair in every local vault it holds, reports it to the vault
that served it, and the next licence for that KID skips the pair and reaches
the CDM. A server with no local SQLite vault cannot store the flag. The
server accepts a report only for a pair it served to that remote session. The
client never learns the server vault names.
the CDM. When the CDM returns the same pair, the server removes the flag,
because then the client's check was wrong. A server with no local SQLite
vault cannot store the flag. The server accepts a report only for a pair it
served to that remote session, and does not flag a pair its CDM licensed.
The client never learns the server vault names.
- A content key in the client's own vaults goes first, even when the server
already answered. The client proves it, and a server key waits as the next
candidate.
@@ -653,7 +657,7 @@ remote session API and a good template for any client.
- **Flow.** `authenticate()` → `create` (+ poll `prompt` every 2s up to a 600s
deadline, answering `pending_input` prompts. With stdin closed the client logs an
error and raises `SystemExit(1)` instead of posting an empty answer) → `get_titles()` → `get_tracks()`
(merging returned `session_headers`/`session_cookies`, re-parsing `manifests`) →
(merging returned `session_headers`/`session_cookie_jar`, re-parsing `manifests`) →
get the licence in proxy mode or `server_cdm` mode → `close()` (`DELETE`, saving any returned
`cache`).
- **Server resolution.** `resolve_server()` reads `config.remote_services.<name>`
@@ -664,7 +668,7 @@ remote session API and a good template for any client.
## Troubleshooting
!!! example "\"Could not connect to remote server ... Is it running?\""
!!! example "\"Could not connect to remote server at ...\""
The client could not reach the URL. Make sure that `unshackle serve` operates on
the server, the `url`/port in `remote_services` are correct (default port `8786`),
and any firewall or reverse proxy allows the connection.
+1 -1
View File
@@ -339,7 +339,7 @@ unshackle cfg [KEY] [VALUE] [OPTIONS]
| `--list` | List all set configuration values. |
!!! warning "Comments are stripped"
Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error.
Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error. If a config key is set twice in the same block, every command warns with the line numbers of the two copies and uses the later copy. `cfg` stops without a change.
!!! example
```shell
+20 -23
View File
@@ -125,14 +125,18 @@ forwarder for it. See [Control D](#control-d).
### Provider load order
When you do a download without `--no-proxy`, unshackle instantiates every configured
proxy provider once, in this fixed order. It uses that same order for bare-query
resolution:
proxy provider once. It uses that same order for bare-query resolution:
**Basic → ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola → Control D**
1. Basic is always first, so your own proxies win.
2. The other proxy providers follow in the order of their keys under `proxy_providers` in
`unshackle.yaml`. Proxy providers that load without a config key, such as Hola, come
after the listed ones in this default order:
**ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola**
3. Control D is always last.
So if you have both Basic and NordVPN configured and use `--proxy us`, a `us` entry in
your Basic config wins, because unshackle tries Basic first. To skip ahead to a specific
proxy provider, prefix the query (`--proxy nordvpn:us`).
So if you list `windscribevpn` above `nordvpn` and use `--proxy us`, a `us` entry in your
Basic config answers first, then Windscribe. To skip ahead to a specific proxy provider,
prefix the query (`--proxy nordvpn:us`).
If a proxy provider cannot load, for example because its server list cannot be fetched
or its config is not valid, unshackle logs a warning and continues with the other proxy
@@ -555,8 +559,7 @@ in unshackle itself.
!!! warning "Docker is required"
Gluetun needs a working [Docker](https://docs.docker.com/get-docker/) installation on
your `PATH`. If Docker is not found, the proxy provider fails to load with an error linking
to the install docs. The container also needs the `NET_ADMIN` capability and access
your `PATH`. If Docker is not found, the proxy provider fails to load. The container also needs the `NET_ADMIN` capability and access
to `/dev/net/tun` to bring up the tunnel. unshackle requests both automatically.
### Query format
@@ -639,6 +642,7 @@ proxy_providers:
- `surfshark`, `mullvad`, and `ivpn` also require `addresses`.
- `windscribe` additionally requires **both** `addresses` and `preshared_key`
(`preshared_key` may be an empty string, but the config key must be present).
Get both from the [Windscribe WireGuard config generator](https://windscribe.com/getconfig/wireguard).
- NordVPN and Proton VPN need only `private_key`.
=== "OpenVPN"
@@ -1025,21 +1029,14 @@ The returned proxy is HTTPS on **port 443**.
## Developer notes
!!! note "Two resolution paths (developers)"
unshackle has **two** proxy-resolution code paths that are intentionally not
identical:
- `unshackle/commands/dl.py` inlines its own logic and loads **all nine** providers,
including Windscribe and Gluetun.
- `unshackle/core/proxies/resolve.py::initialize_proxy_providers()` is the shared
resolver used by the REST API handlers and the remote-service client. It **omits**
Windscribe and Gluetun.
Both paths accept the same query grammar (a direct URI, a `provider:query`, a bare
country query tried against providers in order, or a `controld://` resolver), but the
set of available providers differs. Keep this in mind when you add a proxy provider:
an entry in `dl.py` does not make it available over the API, and the opposite is also
true.
!!! note "One resolution path (developers)"
`dl`, `search`, the REST API handlers and the remote-service client all load proxy
providers through `load_proxy_providers()` in `unshackle/core/proxies/resolve.py`.
The REST API and the remote-service client call it through
`initialize_proxy_providers()`, which leaves out Gluetun because Gluetun starts a
local Docker container. All paths accept the same query grammar: a direct URI, a
`provider:query`, a bare query tried against providers in order, or a `controld://`
resolver.
The remote-service client resolves `--proxy` with its own
`resolve_remote_proxy_arg()`. It changes a `controld:` query to a `controld://`
+5
View File
@@ -258,6 +258,11 @@ Each command reports how many files it removed and how much space it freed, then
**Inspect:** set `debug_requests: true` and operate unshackle with `-d` to show the underlying `urllib3` / `rnet` / `httpx` logs. In the JSON debug log, filter for `service_call` operations to see which request failed. If you use a proxy, also read [Proxies & VPN](proxies-and-vpn.md).
!!! example "A segment download fails after every retry"
**Symptom:** the download stops with a line like `MaxRetriesError: GET <host> failed: RequestError (stream closed because of a broken pipe), gave up after 5 failed attempts over 30s`.
**Inspect:** the connection to the CDN host dropped on every attempt. This is almost always the network, the VPN, or the proxy, not the service. Run the download again: unshackle keeps the segments it already has. Over HTTP/2, all segments share one connection, so one drop fails every segment in flight. Set `network.http1_only: true` (see [Network & proxy](../reference/configuration/network.md)) to spread them over separate connections. Operate unshackle with `-d` to see each retry as a `segment_retry` entry in the JSON debug log.
!!! example "The wrong titles/metadata keep coming back"
**Symptom:** a service returns outdated titles even after the source changed.
+2 -2
View File
@@ -34,11 +34,11 @@ unshackle pushes keys from every source, not only a local CDM. A run against a r
### Wrong keys from a poisoned vault
A public vault can return the right KID with a wrong content key. The decrypters accept a wrong content key and write noise. unshackle therefore trusts a content key from a vault only after FFmpeg decodes the decrypted track with strict error detection. For a fragmented MP4, unshackle reads from the encrypted file which fragments each KID encrypts, and FFmpeg decodes up to three windows of each KID: from its first, middle and last fragment on through the next fragments of that KID, for up to 4 seconds. A window never runs into a fragment of another KID or a clear fragment. A clear lead or a lead under another KID therefore cannot hide a wrong content key, and only the KID that fails is flagged. A KID that is not in this map, and every KID of any other file, gets one check of the whole file. FFmpeg decodes the first seconds and the last seconds. A failure there flags each of those KIDs. When a mapped KID fails, the KIDs that are not in the map get no verdict from that pass; the retry with the replaced key judges them. Content keys from a licence skip this check. A file this FFmpeg build cannot judge counts as a pass, never as a wrong content key: a codec it has no decoder for, or a header it cannot read. A pass is not proof. A wrong content key for HEVC video can decode without an error for some seconds, most often at a low resolution, and that content key passes the check. Without FFmpeg there is no check at all.
A public vault can return the right KID with a wrong content key. The decrypters accept a wrong content key and write noise. unshackle therefore trusts a content key from a vault only after FFmpeg decodes the decrypted track with strict error detection. For a fragmented MP4, unshackle reads from the encrypted file which fragments each KID encrypts, and FFmpeg decodes up to three windows of each KID: from its first, middle and last fragment on through the next fragments of that KID, for up to 4 seconds. A window never runs into a fragment of another KID or a clear fragment. A clear lead or a lead under another KID therefore cannot hide a wrong content key, and only the KID that fails is flagged. When every KID in the map has a content key, the map is complete, and a KID that is not in it does not encrypt this file. This file does not judge that KID; the file of its own track judges it. A file with no encrypted sample entry and no encrypted fragment, for example a clear audio track, judges no KID, so it neither flags nor copies a content key. A KID that is not in an incomplete map, for example one with an all-zero KID, and every KID of any other file, gets one check of the whole file. FFmpeg decodes the first seconds and the last seconds. A failure there flags each of those KIDs. When a mapped KID fails, the KIDs that are not in the map get no verdict from that pass; the retry with the replaced key judges them. Content keys from a licence skip this check. A file this FFmpeg build cannot judge counts as a pass, never as a wrong content key: a codec it has no decoder for, or a header it cannot read. A pass is not proof. A wrong content key for HEVC video can decode without an error for some seconds, most often at a low resolution, and that content key passes the check. Without FFmpeg there is no check at all.
When the decode fails, unshackle:
1. Flags the KID:KEY in the `bad_keys` table of every local SQLite vault, with the service and the name of the vault it came from. The pair also goes out of the service table, except in a vault marked `no_push`, where only the flag is written. A flagged pair cannot go back into a local vault from another vault, a later run, or `kv copy`. A licence that returns the pair clears the flag and stores it. An `API` vault that served the pair also gets a report; `HTTP` and `MySQL` vaults have no report route. A pair from a local vault goes to no remote vault, even when a remote vault holds the same pair.
1. Flags the KID:KEY in the `bad_keys` table of every local SQLite vault, with the service and the name of the vault it came from. The pair also goes out of the service table, except in a vault marked `no_push`, where only the flag is written. A flagged pair cannot go back into a local vault from another vault, a later run, or `kv copy`. A licence that returns the pair clears the flag and stores it. An `API` vault that served the pair gets a report only when a different content key for that KID is accepted. When the CDM returns the same content key, that vault gets no report. `HTTP` and `MySQL` vaults have no report route. A pair from a local vault goes to no remote vault, even when a remote vault holds the same pair.
2. Restores the ciphertext from a hard link it made before decryption, then asks for the content key again. Every vault lookup skips a flagged pair, so the request reaches the remaining vaults and then the CDM. A remote answer that arrived after the winning one is reused instead of queried again.
3. Removes the flag when the CDM returns the same content key, because then the decode check was wrong, not the content key. A content key from another vault is checked in its turn, and flagged as well if it also fails.
+5 -2
View File
@@ -61,8 +61,11 @@ headers:
- **Type:** `dict` &nbsp;·&nbsp; **Default:** `{}`
Proxy/VPN provider configuration. Each sub-key names a proxy provider, and unshackle passes its
block straight to that provider's constructor. See [Proxies & VPN](../../guide/proxies-and-vpn.md)
for the full proxy provider guide. Recognised proxy providers and their exit ports:
block straight to that provider's constructor. A bare `--proxy` query tries Basic first, then
the other proxy providers in the order of their sub-keys, and Control D last (see
[Provider load order](../../guide/proxies-and-vpn.md#provider-load-order)). See
[Proxies & VPN](../../guide/proxies-and-vpn.md) for the full proxy provider guide. Recognised proxy
providers and their exit ports:
| Provider | Config key | Credentials | Proxy scheme/port |
|----------|-----------|-------------|-------------------|
+60
View File
@@ -0,0 +1,60 @@
from __future__ import annotations
import logging
from pathlib import Path
import pytest
from click.testing import CliRunner
from unshackle.commands import cfg as cfg_module
@pytest.fixture
def config_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
path = tmp_path / "unshackle.yaml"
monkeypatch.setattr(cfg_module, "get_config_path", lambda: path)
return path
def test_a_duplicate_key_logs_its_lines_once_and_names_no_key_or_value(
config_file: Path, caplog: pytest.LogCaptureFixture
) -> None:
config_file.write_text(
"theme: rainbow\ncredentials:\n EXAMPLE: first-secret\n OTHER: other\n EXAMPLE: second-secret\n"
"serve:\n users:\n key-secret: {tier: a}\n key-secret: {tier: b}\n",
encoding="utf-8",
)
with caplog.at_level(logging.WARNING, logger="config"):
result = CliRunner().invoke(cfg_module.cfg, ["theme"])
CliRunner().invoke(cfg_module.cfg, ["theme"])
assert isinstance(result.exception, SystemExit), result.exception
assert result.exit_code == 1
assert caplog.messages == [
f"{config_file}: duplicate key on lines 3 and 5, lines 8 and 9. Unshackle uses the later one."
]
assert "Error" not in result.output and "lines" not in result.output
for output in (result.output, caplog.text):
assert "secret" not in output and "EXAMPLE" not in output
def test_a_duplicate_key_blocks_a_write(config_file: Path) -> None:
original = "theme: rainbow\ncredentials:\n EXAMPLE: a\n EXAMPLE: b\n"
config_file.write_text(original, encoding="utf-8")
result = CliRunner().invoke(cfg_module.cfg, ["theme", "plain"])
assert result.exit_code == 1
assert config_file.read_text(encoding="utf-8") == original
def test_a_valid_config_still_reads_a_key(config_file: Path) -> None:
config_file.write_text("theme: rainbow\n", encoding="utf-8")
result = CliRunner().invoke(cfg_module.cfg, ["theme"])
assert result.exit_code == 0, result.output
assert "rainbow" in result.output
def test_a_duplicate_only_ruamel_sees_still_gives_its_line(config_file: Path, caplog: pytest.LogCaptureFixture) -> None:
config_file.write_text("a: 1\n1: x\n0x1: y\n", encoding="utf-8")
with caplog.at_level(logging.WARNING):
result = CliRunner().invoke(cfg_module.cfg, ["a"])
assert result.exit_code == 1
assert caplog.messages == [f"{config_file}: duplicate key on line 3. Unshackle uses the later one."]
+53
View File
@@ -0,0 +1,53 @@
from __future__ import annotations
import logging
from pathlib import Path
import pytest
import yaml
from ruamel.yaml import YAML
from unshackle.core.config import Config, duplicate_key_lines, load_yaml
ALIASES = "base: &b\n x: 1\n x: 2\n self: *b\nother: *b\nlist:\n - y: 1\n y: 2\n"
@pytest.mark.parametrize("compose", [yaml.compose, YAML().compose], ids=["pyyaml", "ruamel"])
def test_duplicate_key_lines_follows_aliases_without_looping(compose) -> None:
assert duplicate_key_lines(compose(ALIASES)) == [(2, 3), (7, 8)]
def test_load_yaml_warns_once_with_lines_and_keeps_the_later_copy(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
path = tmp_path / "unshackle.yaml"
path.write_text("credentials:\n EXAMPLE: first-secret\n EXAMPLE: second-secret\n", encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="config"):
data = load_yaml(path)
load_yaml(path)
assert data == {"credentials": {"EXAMPLE": "second-secret"}}
assert caplog.messages == [f"{path}: duplicate key on lines 2 and 3. Unshackle uses the later one."]
assert "EXAMPLE" not in caplog.text and "secret" not in caplog.text
def test_config_from_yaml_warns_on_a_duplicate_key(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
path = tmp_path / "unshackle.yaml"
path.write_text("tag: first\ntag: second\n", encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="config"):
assert Config.from_yaml(path).tag == "second"
assert "duplicate key on lines 1 and 2" in caplog.text
@pytest.mark.parametrize(
"text",
["a: 1\nb: [1, 2]\n", "", "a: &a\n x: 1\nb: &b\n y: 2\nc:\n <<: *a\n <<: *b\n"],
ids=["valid", "empty", "two-merges"],
)
def test_load_yaml_matches_safe_load_without_a_warning(
tmp_path: Path, text: str, caplog: pytest.LogCaptureFixture
) -> None:
path = tmp_path / "config.yaml"
path.write_text(text, encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="config"):
assert load_yaml(path) == yaml.safe_load(text)
assert not caplog.records
+8 -27
View File
@@ -652,19 +652,14 @@ def video(
def test_hybrid_fails_with_neither_a_base_nor_dv() -> None:
assert hybrid_gate_failure([video("sdr", Video.Range.SDR)], True) == (
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but neither is available",
"Available ranges: SDR",
)
assert hybrid_gate_failure([], True) == (
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but neither is available",
"Available ranges: none",
"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: SDR"
)
assert hybrid_gate_failure([], True) == "HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: none"
def test_hybrid_fails_with_dv_only() -> None:
assert hybrid_gate_failure([video("dv", Video.Range.DV)], True) == (
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only DV is available",
"Available ranges: DV",
"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: DV"
)
@@ -681,27 +676,16 @@ def test_hybrid_goes_on_with_a_base_and_dv_whatever_the_flag() -> None:
def test_hybrid_fails_up_front_when_no_base_is_hevc() -> None:
"""dovi_tool injects into HEVC, so an all-AV1 title fails before it downloads anything."""
videos = [video("av1", Video.Range.HDR10P, codec=Video.Codec.AV1), video("dv", Video.Range.DV)]
assert hybrid_gate_failure(videos, False) == (
"HYBRID mode requires an HEVC HDR10/HDR10+ base, but none is available",
"Available HDR10/HDR10+ codecs: AV1",
)
assert hybrid_gate_failure(videos, False) == "HYBRID needs an HEVC HDR10/HDR10+ base, available codecs: AV1"
assert hybrid_gate_failure(videos + [video("h", Video.Range.HDR10, codec=None)], False) is None
@pytest.mark.parametrize(
"ranges, to_dv, detail",
[
(
[Video.Range.HDR10P],
False,
"Available ranges: HDR10P. Add --generate-dv to generate Dolby Vision from HDR10+.",
),
([Video.Range.HDR10], False, "Available ranges: HDR10"),
(
[Video.Range.HDR10],
True,
"Available ranges: HDR10. --generate-dv needs an HEVC HDR10+ track, and the service returned none.",
),
([Video.Range.HDR10P], False, "HDR10P; add --generate-dv to make DV from HDR10+"),
([Video.Range.HDR10], False, "HDR10"),
([Video.Range.HDR10], True, "HDR10; --generate-dv found no HEVC HDR10+ track"),
],
)
def test_hybrid_fails_without_dv_and_hints_at_the_flag_only_when_it_is_off(
@@ -709,10 +693,7 @@ def test_hybrid_fails_without_dv_and_hints_at_the_flag_only_when_it_is_off(
) -> None:
"""A service that fetches HYBRID by hand may never return HDR10+; the hint must not blame the user then."""
videos = [video(r.name, r) for r in ranges]
assert hybrid_gate_failure(videos, to_dv) == (
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only HDR10 is available",
detail,
)
assert hybrid_gate_failure(videos, to_dv) == f"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: {detail}"
@pytest.mark.parametrize("to_dv", [False, True])
+1 -1
View File
@@ -150,4 +150,4 @@ def test_a_download_stopped_by_a_click_exception_reports_only_its_message() -> N
lines = dl.failure_lines(error)
assert lines == [":x: Download Failed...", f" {error.format_message()}"]
assert "unexpected error" in dl.failure_lines(ValueError("boom"))[-1]
assert dl.failure_lines(ValueError("boom")) == [":x: Download Failed...", " ValueError: boom"]
+61 -4
View File
@@ -3,6 +3,8 @@ from pathlib import Path
from typing import Optional
from uuid import UUID
import pytest
from unshackle.core.drm.verify import PIFF_TENC, PIFF_TFXD, KidMap, kid_windows, pick_windows
A, B, C = UUID(int=0xA), UUID(int=0xB), UUID(int=0xC)
@@ -44,15 +46,28 @@ def sbgp(*runs: tuple[int, int]) -> bytes:
return full(b"sbgp", b"seig" + struct.pack(">I", len(runs)) + b"".join(struct.pack(">II", *run) for run in runs))
def movie(*sample_entries: bytes, handler: bytes = b"vide", stbl: bytes = b"") -> bytes:
def trak(*sample_entries: bytes, track: int = 1, handler: bytes = b"vide", stbl: bytes = b"") -> bytes:
stsd = full(b"stsd", struct.pack(">I", len(sample_entries)) + b"".join(sample_entries))
hdlr = full(b"hdlr", bytes(4) + handler + bytes(12))
trak = box(
return box(
b"trak",
full(b"tkhd", bytes(8) + struct.pack(">I", 1)),
full(b"tkhd", bytes(8) + struct.pack(">I", track)),
box(b"mdia", full(b"mdhd", bytes(8) + struct.pack(">I", 1000)), hdlr, box(b"minf", box(b"stbl", stsd, stbl))),
)
return box(b"ftyp", b"iso6") + box(b"moov", trak, box(b"mvex", full(b"trex", struct.pack(">III", 1, 1, 0))))
def movie(*sample_entries: bytes, handler: bytes = b"vide", stbl: bytes = b"", others: bytes = b"") -> bytes:
moov = box(
b"moov",
trak(*sample_entries, handler=handler, stbl=stbl),
others,
box(b"mvex", full(b"trex", struct.pack(">III", 1, 1, 0))),
)
return box(b"ftyp", b"iso6") + moov
def clear_audio(others: bytes = b"") -> bytes:
return movie(box(b"mp4a", bytes(28)), handler=b"soun", others=others)
def moof(
@@ -178,3 +193,45 @@ def test_not_a_fragmented_mp4(tmp_path: Path) -> None:
path = tmp_path / "video.ts"
path.write_bytes(b"\x47" * 188 * 4)
assert kid_windows(path) is None
def test_clear_mp4_maps_no_kid(tmp_path: Path) -> None:
path = tmp_path / "clear.m4a"
path.write_bytes(clear_audio() + moof(1, 0))
assert kid_windows(path) == KidMap({}, False)
def test_encrypted_entry_without_a_tenc_is_not_clear(tmp_path: Path) -> None:
path = tmp_path / "enc.mp4"
path.write_bytes(movie(box(b"encv", bytes(78), box(b"sinf", box(b"frma", b"avc1")))) + moof(1, 0))
assert kid_windows(path) is None
@pytest.mark.parametrize("kind", [b"enct", b"encs", b"drmi"])
def test_other_encrypted_entry_kinds_are_not_clear(tmp_path: Path, kind: bytes) -> None:
path = tmp_path / "enc.mp4"
path.write_bytes(movie(box(kind, bytes(16)), handler=b"text") + moof(1, 0))
assert kid_windows(path) is None
def test_a_clear_entry_with_a_protecting_senc_is_not_clear(tmp_path: Path) -> None:
path = tmp_path / "enc.m4a"
senc = full(b"senc", struct.pack(">I", 1) + bytes(8))
path.write_bytes(clear_audio() + moof(1, 0, senc))
assert kid_windows(path) is None
path.write_bytes(clear_audio() + moof(1, 0, full(b"senc", bytes(4))))
assert kid_windows(path) == KidMap({}, False)
def test_only_the_tracks_with_fragments_decide_clear(tmp_path: Path) -> None:
path = tmp_path / "clear.m4a"
path.write_bytes(clear_audio(others=trak(encv(A), track=2)) + moof(1, 0))
assert kid_windows(path) == KidMap({}, False)
path.write_bytes(clear_audio(others=trak(encv(A), track=2)))
assert kid_windows(path) is None
def test_a_clear_lead_beside_an_encrypted_entry_is_not_clear(tmp_path: Path) -> None:
path = tmp_path / "lead.mp4"
path.write_bytes(movie(box(b"avc1", bytes(78)), encv(A)) + moof(1, 0))
assert kid_windows(path) is None
+2 -2
View File
@@ -115,10 +115,10 @@ def test_other_audio_still_muxes_with_mkvmerge(tmp_path: Path, monkeypatch: pyte
tracks.mux("Title", delete=False, output_path=tmp_path / "out.muxed.mkv")
def test_a_missing_mp4box_names_the_install(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
def test_a_missing_mp4box_names_the_binary(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(binaries, "MP4Box", None, raising=False)
tracks = build_tracks(tmp_path, b"dtsx")
with pytest.raises(RuntimeError, match="gpac"):
with pytest.raises(RuntimeError, match="MP4Box"):
tracks.mux("Title", delete=False, output_path=tmp_path / "out.muxed.mkv")
+112
View File
@@ -0,0 +1,112 @@
"""A bare UTF-16LE WRMHEADER in place of a PSSH box or PlayReady Object loads as PlayReady DRM."""
from __future__ import annotations
import base64
import logging
from uuid import UUID
import m3u8
import pytest
from lxml import etree
from pyplayready.misc.exceptions import InvalidPssh
from pyplayready.system.pssh import PSSH
from pywidevine.pssh import PSSH as WV_PSSH
from unshackle.core.drm import PlayReady, Widevine, drm_from_dict
from unshackle.core.manifests.dash import DASH
from unshackle.core.manifests.hls import HLS
from unshackle.core.manifests.ism import ISM
KID = UUID("00112233-4455-6677-8899-aabbccddeeff")
KID_B64 = base64.b64encode(KID.bytes_le).decode()
PR_URN = "urn:uuid:9a04f079-9840-4286-ab92-e65be0885f95"
WV_URN = "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed"
def bare_header(version: str) -> str:
"""Base64 of a bare WRMHEADER, with the KID where the given header version puts it."""
if version == "4.0.0.0":
data = f"<PROTECTINFO><KEYLEN>16</KEYLEN><ALGID>AESCTR</ALGID></PROTECTINFO><KID>{KID_B64}</KID>"
else:
data = f'<PROTECTINFO><KIDS><KID ALGID="AESCTR" VALUE="{KID_B64}"></KID></KIDS></PROTECTINFO>'
xml = (
f'<WRMHEADER xmlns="http://schemas.microsoft.com/DRM/2007/03/PlayReadyHeader" version="{version}">'
f"<DATA>{data}<LA_URL>https://example.invalid/pr</LA_URL></DATA></WRMHEADER>"
)
return base64.b64encode(xml.encode("utf-16-le")).decode()
def pr_protection(child: str, value: str) -> etree._Element:
return etree.fromstring(
f'<ContentProtection xmlns:cenc="urn:mpeg:cenc:2013" xmlns:mspr="urn:microsoft:playready" '
f'schemeIdUri="{PR_URN}"><{child}>{value}</{child}></ContentProtection>'
)
@pytest.mark.parametrize("version", ["4.0.0.0", "4.1.0.0", "4.2.0.0", "4.3.0.0"])
@pytest.mark.parametrize("child", ["mspr:pro", "cenc:pssh"])
def test_dash_loads_a_bare_header(child: str, version: str) -> None:
drm = DASH.get_drm([pr_protection(child, bare_header(version))])
assert len(drm) == 1
assert isinstance(drm[0], PlayReady)
assert KID in drm[0].kids
assert drm[0].pssh_b64
PSSH(drm[0].pssh_b64)
assert KID in drm_from_dict(drm[0].to_dict()).kids
def test_wrapped_header_bytes_are_unchanged() -> None:
header = bare_header("4.0.0.0")
drm = PlayReady.from_pssh_b64(header)
assert drm.pssh.wrm_headers[0].dumps().encode("utf-16-le") == base64.b64decode(header)
def test_dash_skips_an_unparseable_playready_entry_and_keeps_widevine(caplog: pytest.LogCaptureFixture) -> None:
widevine = etree.fromstring(
f'<ContentProtection xmlns:cenc="urn:mpeg:cenc:2013" schemeIdUri="{WV_URN}">'
f"<cenc:pssh>{WV_PSSH.new(system_id=WV_PSSH.SystemId.Widevine, key_ids=[KID]).dumps()}</cenc:pssh>"
"</ContentProtection>"
)
garbage = pr_protection("mspr:pro", base64.b64encode(b"\x00\x00not a header").decode())
with caplog.at_level(logging.WARNING, logger="DASH"):
drm = DASH.get_drm([widevine, garbage])
assert [type(d) for d in drm] == [Widevine]
assert "Skipping unparseable PlayReady ContentProtection" in caplog.text
@pytest.mark.parametrize(
"data",
[b"\x00\x00not a header", "<WRMHEADER/>".encode("utf-8")],
ids=["garbage", "utf-8-header"],
)
def test_other_bad_data_keeps_the_original_error(data: bytes) -> None:
with pytest.raises(InvalidPssh):
PlayReady.from_pssh_b64(base64.b64encode(data).decode())
def test_hls_loads_a_bare_header() -> None:
key = m3u8.loads(
"#EXTM3U\n"
f'#EXT-X-KEY:METHOD=SAMPLE-AES-CTR,URI="data:text/plain;charset=UTF-16;base64,{bare_header("4.0.0.0")}",'
'KEYFORMAT="com.microsoft.playready",KEYFORMATVERSIONS="1"\n'
"#EXTINF:4.0,\nseg0.mp4\n"
).keys[0]
drm = HLS.get_drm(key)
assert isinstance(drm, PlayReady)
assert KID in drm.kids
def test_ism_loads_a_bare_header() -> None:
header = etree.fromstring(f'<ProtectionHeader SystemID="{PR_URN[9:]}">{bare_header("4.0.0.0")}</ProtectionHeader>')
drm = ISM.get_drm([header])
assert len(drm) == 1
assert KID in drm[0].kids
+11
View File
@@ -63,6 +63,17 @@ def test_raise_errors_raises_the_load_failure() -> None:
load_proxy_providers(OTHERS, raise_errors=True)
def test_bare_query_order_follows_config_key_order_with_controld_last(monkeypatch: pytest.MonkeyPatch) -> None:
from unshackle.core.proxies import ControlD
order = {"controld": {"token": "t"}, "surfsharkvpn": CREDENTIALS, "nordvpn": CREDENTIALS, "basic": {"us": "x"}}
monkeypatch.setattr(config, "proxy_providers", order)
monkeypatch.setattr(ControlD, "get_locations", lambda self: [])
loaded = load_proxy_providers(("expressvpn", "protonvpn", "hola"), quiet=True)
names = [getattr(p, "name", p.__class__.__name__) for p in loaded]
assert names == ["Basic", "SurfsharkVPN", "NordVPN", "ControlD"]
def test_rest_path_loads_windscribe_but_not_gluetun(monkeypatch: pytest.MonkeyPatch) -> None:
from unshackle.core.proxies import Gluetun, WindscribeVPN
from unshackle.core.proxies.resolve import initialize_proxy_providers
+8
View File
@@ -27,3 +27,11 @@ def test_retry_log_counts_only_real_retries(rate_limited, caplog, max_retries):
retries = [r.getMessage() for r in caplog.records if "Retrying" in r.getMessage()]
assert retries == [f"429 Too Many Requests(/x). Retrying... ({n}/{max_retries})" for n in range(1, max_retries + 1)]
@pytest.mark.parametrize(("max_retries", "attempts"), [(0, ""), (2, " after 3 attempts")])
def test_spent_budget_names_the_attempts_and_host_not_the_url(rate_limited, max_retries, attempts):
with pytest.raises(MaxRetriesError) as raised:
rate_limited.get("https://example.invalid/x?hmac=secret", max_retries=max_retries)
assert str(raised.value) == f"GET example.invalid failed{attempts}: 429 Too Many Requests"
+147 -12
View File
@@ -287,6 +287,26 @@ def test_cdm_returning_the_flagged_key_clears_the_flag(tmp_path: Path, monkeypat
assert not local.is_bad_key(KID, BAD)
def test_a_licence_key_that_replaces_a_bad_local_pair_is_stored(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The local vault held the bad pair when the licence key arrived, so it skipped the KID; store it now."""
local = SQLite("local", tmp_path / "local.db")
local.add_key("SVC", KID, BAD)
vaults = Vaults("SVC")
vaults.vaults = [local]
cmd = make_cmd(monkeypatch, vaults)
drm = FakeDRM({KID: vaults.get_key(KID)[0]})
path = tmp_path / "video.mp4"
path.write_bytes(b"cipher")
assert local.add_keys("SVC", {KID: GOOD}) == 0
cmd.decrypt_verified(drm, path, lambda drm, track_kid: drm.content_keys.update({KID: GOOD}), KID)
cmd.wait_vault_writes()
assert local.is_bad_key(KID, BAD)
assert local.get_key(KID, "SVC") == GOOD
def test_stale_backup_from_a_killed_run_is_not_restored(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
poisoned = Remote("poisoned", tmp_path / "poisoned.db")
poisoned.add_key("SVC", KID, BAD)
@@ -426,7 +446,7 @@ def test_only_the_kid_whose_window_fails_is_flagged(tmp_path: Path, monkeypatch:
def test_a_kid_absent_from_the_map_is_judged_by_the_fallback(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A KID the map does not know (a zero-KID tenc, a licence KID the file does not carry)
"""A map that holds an all-zero tenc KID is not complete, so a KID it does not know
shares the whole-file verdict, and a wrong key there is flagged."""
lead = UUID(int=7)
local = SQLite("local", tmp_path / "local.db")
@@ -446,7 +466,9 @@ def test_a_kid_absent_from_the_map_is_judged_by_the_fallback(tmp_path: Path, mon
return drm.decrypted_with[-1][lead if start is not None else KID] == GOOD
monkeypatch.setattr(
dl_module.verify, "kid_windows", lambda path: dl_module.verify.KidMap({lead: [(4.0, 6.0)]}, True)
dl_module.verify,
"kid_windows",
lambda path: dl_module.verify.KidMap({lead: [(4.0, 6.0)], UUID(int=0): [(8.0, 10.0)]}, True),
)
monkeypatch.setattr(dl_module, "ffmpeg_decodes", decodes)
path = tmp_path / "video.mp4"
@@ -501,11 +523,93 @@ def test_a_wrong_mapped_key_does_not_flag_a_kid_the_file_does_not_carry(
assert drm.content_keys == {audio: GOOD, KID: GOOD}
assert local.is_bad_key(KID, BAD)
assert not local.is_bad_key(audio, GOOD)
assert local.get_key(audio, "SVC") == GOOD # judged on the retry and copied
assert local.get_key(audio, "SVC") is None
def test_a_kid_the_file_does_not_carry_is_not_judged_by_it(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The CDM licensed the video KID and a vault served the audio KID. An FFmpeg that fails
every decode of the video file must not flag the audio KID, nor copy it to another vault."""
audio = UUID(int=7)
local = SQLite("local", tmp_path / "local.db")
remote = Remote("remote", tmp_path / "remote.db")
remote.add_key("SVC", audio, GOOD)
vaults = Vaults("SVC")
vaults.vaults = [local, remote]
cmd = make_cmd(monkeypatch, vaults)
drm = FakeDRM({audio: vaults.get_key(audio)[0], KID: GOOD})
monkeypatch.setattr(
dl_module.verify, "kid_windows", lambda path: dl_module.verify.KidMap({KID: [(4.0, 6.0)]}, True)
)
monkeypatch.setattr(dl_module, "ffmpeg_decodes", lambda path, start=None, seconds=3, video=None: False)
path = tmp_path / "video.mp4"
path.write_bytes(b"cipher")
licences: list[UUID] = []
cmd.decrypt_verified(drm, path, lambda drm, track_kid: licences.append(track_kid), KID)
cmd.wait_vault_writes()
assert drm.content_keys == {audio: GOOD, KID: GOOD}
assert not licences
assert not local.is_bad_key(audio, GOOD)
assert local.get_key(audio, "SVC") is None
@pytest.mark.parametrize(
("served", "decodes", "told"),
[
(GOOD, False, False), # broken FFmpeg, the CDM returns the same key
(BAD, True, True), # the CDM returns another key
],
)
def test_a_remote_vault_is_told_only_when_another_key_replaces_its_pair(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, served: str, decodes: bool, told: bool
) -> None:
remote = Remote("remote", tmp_path / "remote.db")
remote.add_key("SVC", KID, served)
vaults = Vaults("SVC")
vaults.vaults = [SQLite("local", tmp_path / "local.db"), remote]
cmd = make_cmd(monkeypatch, vaults)
if not decodes:
monkeypatch.setattr(dl_module, "ffmpeg_decodes", lambda path, start=None, seconds=3, video=None: False)
drm = FakeDRM({KID: vaults.get_key(KID)[0]})
path = tmp_path / "video.mp4"
path.write_bytes(b"cipher")
def licence(drm: FakeDRM, track_kid: UUID) -> None:
assert not remote.is_bad_key(KID, served)
drm.content_keys[KID] = GOOD
cmd.decrypt_verified(drm, path, licence, KID)
assert drm.content_keys == {KID: GOOD}
assert remote.is_bad_key(KID, served) is told
def test_the_server_vault_is_told_at_once(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Over --remote the report is what makes the server skip the pair on the next licence."""
reports: list[tuple[UUID, str]] = []
server = SimpleNamespace(
name="server_cdm",
local=False,
report_now=True,
flag_bad_key=lambda service, kid, key, source: reports.append((kid, key)),
)
vaults = Vaults("SVC")
vaults.vaults = [SQLite("local", tmp_path / "local.db")]
vaults.sources[KID] = (BAD, server)
cmd = make_cmd(monkeypatch, vaults)
drm = FakeDRM({KID: BAD})
path = tmp_path / "video.mp4"
path.write_bytes(b"cipher")
def licence(drm: FakeDRM, track_kid: UUID) -> None:
assert reports == [(KID, BAD)]
drm.content_keys[KID] = GOOD
cmd.decrypt_verified(drm, path, licence, KID)
assert drm.content_keys == {KID: GOOD}
def test_only_checked_kids_are_copied_to_other_vaults(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Both the mapped KID and the absent one are proven before they reach another vault."""
"""Only the mapped KID is proven and copied; the KID this file does not carry waits for its own file."""
lead = UUID(int=7)
local = SQLite("local", tmp_path / "local.db")
remote = Remote("remote", tmp_path / "remote.db")
@@ -530,9 +634,35 @@ def test_only_checked_kids_are_copied_to_other_vaults(tmp_path: Path, monkeypatc
cmd.decrypt_verified(drm, path, None, KID)
cmd.wait_vault_writes()
assert starts.count(None) == 1 # the absent KID went through the fallback before its copy
assert None not in starts
assert local.get_key(lead, "SVC") == GOOD
assert local.get_key(KID, "SVC") == GOOD
assert local.get_key(KID, "SVC") is None
def test_a_clear_file_judges_and_copies_no_vault_key(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
local = SQLite("local", tmp_path / "local.db")
remote = Remote("remote", tmp_path / "remote.db")
remote.add_key("SVC", KID, BAD)
vaults = Vaults("SVC")
vaults.vaults = [local, remote]
cmd = make_cmd(monkeypatch, vaults)
drm = FakeDRM({KID: vaults.get_key(KID)[0]})
checks: list = []
def decodes(path: Path, start: object = None, seconds: float = 3, video: object = None) -> bool:
checks.append(start)
return True
monkeypatch.setattr(dl_module.verify, "kid_windows", lambda path: dl_module.verify.KidMap({}, False))
monkeypatch.setattr(dl_module, "ffmpeg_decodes", decodes)
path = tmp_path / "audio.mp4"
path.write_bytes(b"clear")
cmd.decrypt_verified(drm, path, None, KID)
cmd.wait_vault_writes()
assert checks == []
assert local.get_key(KID, "SVC") is None
assert not local.is_bad_key(KID, BAD)
def test_flagged_pair_warns_once_per_vault(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
@@ -602,6 +732,8 @@ def test_api_vault_reports_the_pair_upstream(monkeypatch: pytest.MonkeyPatch, ca
vaults.vaults = [api]
vaults.sources[KID] = (BAD, api)
vaults.flag_bad_key(KID, BAD)
assert not calls
vaults.report_replaced({KID: GOOD})
vaults.unflag_bad_key(KID, BAD)
assert calls == [
@@ -609,28 +741,31 @@ def test_api_vault_reports_the_pair_upstream(monkeypatch: pytest.MonkeyPatch, ca
("DELETE", f"https://vault.example/bad-keys/{KID.hex}/{BAD}", None),
]
def flag_and_replace(source: object) -> None:
vaults.sources[KID] = (BAD, source)
vaults.flag_bad_key(KID, BAD)
vaults.report_replaced({KID: GOOD})
calls.clear()
vaults.sources[KID] = (BAD, SimpleNamespace(name="other"))
vaults.flag_bad_key(KID, BAD)
flag_and_replace(SimpleNamespace(name="other", local=False, report_now=False))
assert calls == []
vaults.sources[KID] = (BAD, api)
monkeypatch.setattr(api.session, "post", lambda url, **kw: (_ for _ in ()).throw(ConnectionError("down")))
caplog.clear()
with caplog.at_level(logging.DEBUG, logger="unshackle.core.vaults"):
vaults.flag_bad_key(KID, BAD)
flag_and_replace(api)
assert [r.levelname for r in caplog.records] == ["DEBUG"]
gone = SimpleNamespace(status_code=404, raise_for_status=lambda: (_ for _ in ()).throw(RuntimeError("404")))
monkeypatch.setattr(api.session, "post", lambda url, **kw: gone)
caplog.clear()
vaults.flag_bad_key(KID, BAD)
flag_and_replace(api)
assert not caplog.records
denied = SimpleNamespace(status_code=200, raise_for_status=lambda: None, json=lambda: {"code": 1, "message": "no"})
monkeypatch.setattr(api.session, "post", lambda url, **kw: denied)
monkeypatch.setattr(api.session, "delete", lambda url, **kw: denied)
vaults.flag_bad_key(KID, BAD)
flag_and_replace(api)
vaults.unflag_bad_key(KID, BAD)
assert not caplog.records
+3
View File
@@ -39,6 +39,9 @@ def fake_server_vaults(pushes: list, vault_list: list[FakeVault]) -> Any:
def __len__(self) -> int:
return len(vault_list)
def clear_licensed_flags(self, content_keys: dict) -> list:
return []
def add_keys(self, key_map: dict) -> int:
pushes.append(key_map)
return sum(vault.ok for vault in vault_list)
@@ -0,0 +1,50 @@
"""A segment that fails every attempt reports the attempts and the cause, without the signed URL."""
import sys
import pytest
from unshackle.commands.dl import dl
from unshackle.core.downloaders.requests import download, retries_exhausted
from unshackle.core.session import MaxRetriesError, RnetSession, failure_reason
downloader = sys.modules[download.__module__]
SIGNED_URL = "https://cdn.example.invalid/dvt2=exp=1~hmac=deadbeef/seg/40_000.mp4a"
RNET_BROKEN_PIPE = (
f'is_request error: wreq::Error {{ kind: Request, url: "{SIGNED_URL}", source: hyper::Error(Io, '
'Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) }'
)
class RequestError(Exception):
"""Stands in for rnet.RequestError, which carries the URL in its text."""
def test_failure_reason_keeps_the_rnet_error_text_and_drops_the_url():
reason = failure_reason(MaxRetriesError("m", cause=RequestError(RNET_BROKEN_PIPE)))
assert reason == "RequestError (stream closed because of a broken pipe)"
def test_spent_retries_report_one_line_without_the_url(monkeypatch, tmp_path):
session = RnetSession()
calls = []
def get(url, **kwargs):
calls.append(kwargs.get("max_retries"))
raise MaxRetriesError("GET cdn.example.invalid failed: x", cause=RequestError(RNET_BROKEN_PIPE))
monkeypatch.setattr(session, "get", get)
monkeypatch.setattr(downloader, "retry_sleep", lambda exc, attempts: 0)
with pytest.raises(MaxRetriesError) as raised:
for _ in download(url=SIGNED_URL, save_path=tmp_path / "seg", session=session):
pass
assert calls == [0] * downloader.MAX_ATTEMPTS
assert retries_exhausted(raised.value)
assert not retries_exhausted(ValueError("boom"))
header, line = dl.failure_lines(raised.value)
assert f"gave up after {downloader.MAX_ATTEMPTS} failed attempts" in line
assert "hmac" not in line
+1 -1
View File
@@ -389,7 +389,7 @@ def test_a_lent_device_session_keeps_its_auth_headers_and_track_secrets() -> Non
headers, cookies = handlers.client_session_auth(session)
assert headers == {"User-Agent": "ua"}
assert cookies == {"account": "client"}
assert handlers.cookie_values(cookies) == {"account": "client"}
assert handlers.scrub_track_data({"token": "server", "id": 1}, session) == {"id": 1}
+1 -3
View File
@@ -115,9 +115,7 @@ def test_build_error_response_debug_mode_includes_traceback() -> None:
(Exception("Not available in your region"), APIErrorCode.GEOFENCE),
(Exception("Service is geoblocked in GB"), APIErrorCode.GEOFENCE),
(
Exception(
"Service is not available in your region (GB). Pass --proxy with a proxy outside the blocked regions."
),
Exception("Service is not available in your region (GB)"),
APIErrorCode.GEOFENCE,
),
(Exception("Title not found"), APIErrorCode.NOT_FOUND),
@@ -43,11 +43,12 @@ def test_client_marks_vault_keys_and_forgets_cdm_keys():
assert svc.server_vault_keys == {}
def test_report_bad_posts_only_the_pair():
def test_server_vault_flag_posts_only_the_pair():
svc = remote_service()
posted: list = []
svc.client = SimpleNamespace(post_optional=lambda ep, data: posted.append((ep, data)) or {})
svc.server_vault.report_bad(KID, "v")
assert svc.server_vault.report_now
svc.server_vault.flag_bad_key("SVC", KID, "v", "server_cdm")
assert posted == [("/api/session/sess/keys/bad", {"kid": KID.hex, "key": "v"})]
@@ -77,18 +78,27 @@ def session(monkeypatch):
return entry, vault
async def test_server_flags_a_served_pair_with_its_own_vault_name(session):
async def test_server_flags_a_served_pair_with_its_own_vault_name(session, monkeypatch):
entry, vault = session
monkeypatch.setattr(handlers, "CACHED_PAIRS", {("SimpleNamespace", KID.hex, "v")})
resp = await handlers.session_bad_key_handler({"kid": KID.hex, "key": "v"}, "sess")
assert resp.status == 200
assert vault.flagged == [("SVC", KID, "v", "sqlite")]
assert KID.hex not in entry.served_keys
assert not handlers.CACHED_PAIRS
with pytest.raises(APIError):
await handlers.session_bad_key_handler({"kid": CDM_KID.hex, "key": "nope"}, "sess")
assert len(vault.flagged) == 1
async def test_server_does_not_flag_a_pair_its_cdm_licensed(session):
entry, vault = session
resp = await handlers.session_bad_key_handler({"kid": CDM_KID.hex, "key": "c"}, "sess")
assert json.loads(resp.body) == {"flagged": False}
assert vault.flagged == []
class _WV:
def __init__(self, b64, kids):
self._pssh = SimpleNamespace(dumps=lambda: b64)
@@ -218,3 +228,20 @@ def test_reconstruct_drm_keeps_the_default_kid_of_a_pssh_without_kids():
(drm,) = reconstruct_drm([{"type": "widevine", "pssh": content_id_only.dumps(), "kids": [str(KID)]}])
assert real_kids(drm) == [KID]
def test_a_cdm_licence_unflags_a_pair_a_client_reported(monkeypatch, tmp_path):
"""The CDM returned the reported pair, so the client's decode check was wrong."""
from unshackle.core.vaults import Vaults
from unshackle.vaults.SQLite import SQLite
local = SQLite("local", tmp_path / "local.db")
vaults = Vaults("SVC")
vaults.vaults = [local]
vaults.flag_bad_key(KID, "ab" * 16)
monkeypatch.setattr(handlers, "load_server_vaults", lambda service: vaults)
monkeypatch.setattr(handlers, "CACHED_PAIRS", set())
handlers.cache_to_vaults({KID.hex: "ab" * 16}, "SVC")
assert not local.is_bad_key(KID, "ab" * 16)
assert local.get_key(KID, "SVC") == "ab" * 16
@@ -42,6 +42,9 @@ class FakeVaults:
def is_flagged(self, kid, key):
return False
def clear_licensed_flags(self, content_keys):
return []
def add_keys(self, kid_keys):
self.added.update(kid_keys)
self.pushes.append(dict(kid_keys))
@@ -227,3 +230,21 @@ def test_cdm_only_trusts_server_vault_keys():
assert vaults.sources == {}
assert dl.LICENSE_KEY_CACHE == {CACHED: "k_cached"}
@pytest.mark.parametrize("system", ["widevine", "playready"])
def test_no_cdm_and_no_server_cdm_names_the_missing_cdm(system):
"""The server does not license this service for the API key, and the client has no CDM for it."""
import click
from unshackle.core.drm import PlayReady, Widevine
cmd = make_cmd(FakeVaults())
cmd._remote_service = SimpleNamespace(_server_cdm=False)
cmd.cdm = None
cmd.cdm_for = lambda system, quality=None: None
drm_class = PlayReady if system == "playready" else Widevine
drm = drm_class.__new__(drm_class)
drm.content_keys = {}
with pytest.raises(click.ClickException, match="No CDM configured for TEST"):
prepare(cmd, drm, lambda **kwargs: None, NEW)
+120 -3
View File
@@ -1,13 +1,15 @@
from types import SimpleNamespace
import pytest
import requests
from unshackle.core.api.errors import APIError
from unshackle.core.api.handlers import client_session_auth, resolve_handler_proxy, scrub_track_data
from unshackle.core.remote_service import apply_session_cookies
def _session(client_auth):
cookie = SimpleNamespace(name="sid", value="secret")
cookie = SimpleNamespace(name="sid", value="secret", domain=".example.com", path="/")
svc = SimpleNamespace(headers={"Authorization": "Bearer x", "User-Agent": "ua"}, cookies=[cookie])
return SimpleNamespace(service_instance=SimpleNamespace(session=svc), client_auth=client_auth)
@@ -15,14 +17,129 @@ def _session(client_auth):
def test_client_login_keeps_auth():
headers, cookies = client_session_auth(_session(True))
assert headers == {"Authorization": "Bearer x", "User-Agent": "ua"}
assert cookies == {"sid": "secret"}
assert cookies == [{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/", "secure": False}]
assert scrub_track_data({"token": "t", "hls": {}}, _session(True)) == {"token": "t", "hls": {}}
def test_client_cookies_keep_their_domain():
session = requests.Session()
_, cookies = client_session_auth(_session(True))
apply_session_cookies(session, {"session_cookie_jar": cookies})
assert session.cookies.get("sid", domain=".example.com") == "secret"
cdn = session.prepare_request(requests.Request("GET", "https://cdn.other.net/seg.mp4"))
assert "Cookie" not in cdn.headers
own = session.prepare_request(requests.Request("GET", "https://www.example.com/api"))
assert own.headers["Cookie"] == "sid=secret"
def test_client_takes_old_server_cookies_without_domain():
session = requests.Session()
apply_session_cookies(session, {"session_cookies": {"sid": "secret"}})
assert session.cookies.get("sid") == "secret"
def test_client_keeps_secure_cookies_off_http():
session = requests.Session()
jar = [{"name": "sid", "value": "s", "domain": ".example.com", "path": "/", "secure": True}]
apply_session_cookies(session, {"session_cookie_jar": jar})
plain = session.prepare_request(requests.Request("GET", "http://cdn.example.com/seg.mp4"))
assert "Cookie" not in plain.headers
tls = session.prepare_request(requests.Request("GET", "https://cdn.example.com/seg.mp4"))
assert tls.headers["Cookie"] == "sid=s"
def test_client_skips_malformed_server_cookies():
session = requests.Session()
jar = [{"value": "v"}, "str", {"name": "n", "value": 1}, {"name": "ok", "value": "v", "domain": ".example.com"}]
apply_session_cookies(session, {"session_cookie_jar": jar})
assert [c.name for c in session.cookies] == ["ok"]
session = requests.Session()
apply_session_cookies(session, {"session_cookie_jar": 5, "session_cookies": ["x"]})
assert not session.cookies
def test_rnet_session_sends_client_and_set_cookies():
from unshackle.core.session import RnetSession
svc = RnetSession()
svc.cookies.set("sid", "stale", domain=".example.com")
svc.cookies.set("cdn", "c", domain="cdn.other.net")
svc.export_origin_cookies = lambda: {"https://www.example.com": "sid=live; cf=1"}
_, cookies = client_session_auth(SimpleNamespace(service_instance=SimpleNamespace(session=svc), client_auth=True))
assert {(c["name"], c["value"], c["domain"]) for c in cookies} == {
("sid", "live", "www.example.com"),
("cf", "1", "www.example.com"),
("cdn", "c", "cdn.other.net"),
}
def test_tracks_response_carries_the_cookie_jar():
import asyncio
import json
from unshackle.core.api import session_store as store_mod
from unshackle.core.api.handlers import session_tracks_handler
from unshackle.core.tracks import Tracks
cookie = SimpleNamespace(name="sid", value="secret", domain=".example.com", path="/", secure=True)
class StubService:
session = SimpleNamespace(headers={}, cookies=[cookie])
def get_tracks(self, title):
return Tracks()
def get_chapters(self, title):
raise NotImplementedError
async def run():
store = store_mod.get_session_store()
entry = await store.create(
service_tag="EXAMPLE", service_instance=StubService(), session_id="jar", owner_key=None
)
entry.title_map = {"t1": "Example Title"}
entry.client_auth = True
try:
return json.loads((await session_tracks_handler({"title_id": "t1"}, "jar")).body)
finally:
await store.delete("jar")
body = asyncio.run(run())
assert body["session_cookies"] == {"sid": "secret"}
assert body["session_cookie_jar"] == [
{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/", "secure": True}
]
def test_remote_get_tracks_scopes_server_cookies():
import logging
from unshackle.core.remote_service import RemoteService
jar = [{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/"}]
service = object.__new__(RemoteService)
service.__dict__.update(
_tracks_by_title={},
_chapters_by_title={},
_sample_decrypt_tracks=set(),
_server_cdm=None,
_session=requests.Session(),
_session_id="s",
service_tag="EXAMPLE",
log=logging.getLogger("test"),
client=SimpleNamespace(post=lambda path, body: {"session_cookie_jar": jar}),
drain_server_logs=lambda: None,
)
service.get_tracks(SimpleNamespace(id="t1"))
cdn = service._session.prepare_request(requests.Request("GET", "https://cdn.other.net/seg.mp4"))
assert "Cookie" not in cdn.headers
assert service._session.cookies.get("sid", domain=".example.com") == "secret"
def test_server_login_strips_auth():
headers, cookies = client_session_auth(_session(False))
assert headers == {"User-Agent": "ua"}
assert cookies == {}
assert cookies == []
assert scrub_track_data({"token": "t", "hls": {}}, _session(False)) == {"hls": {}}
+9 -1
View File
@@ -1,5 +1,5 @@
"""Self-check for ``has_encrypted_sample_entry``: a finalized MP4 whose moov still carries
an encv/enca sample entry means the decrypt tool skipped/failed the track (frma was never
an encrypted (enc*/drm*) sample entry means the decrypt tool skipped/failed the track (frma was never
applied), so the detector must flag it before the file reaches the muxer. Detection is
scoped to moov, so stray 4CC byte runs in mdat must never trip (or mask) it.
@@ -11,6 +11,8 @@ from __future__ import annotations
import struct
from pathlib import Path
import pytest
from unshackle.core.tracks.track import has_encrypted_sample_entry
@@ -80,6 +82,12 @@ def test_text_containing_enca_in_moov_not_detected(tmp_path: Path) -> None:
assert has_encrypted_sample_entry(f) is False
@pytest.mark.parametrize("kind", [b"encs", b"enct", b"drmi"])
def test_other_encrypted_entry_kinds_detected(tmp_path: Path, kind: bytes) -> None:
f = write(tmp_path, "enc.mp4", FTYP, moov(stsd([sample_entry(kind, box(b"sinf", box(b"frma", b"stpp")))])))
assert has_encrypted_sample_entry(f) is True
def test_missing_or_bogus_file(tmp_path: Path) -> None:
assert has_encrypted_sample_entry(tmp_path / "nope.mp4") is False
bogus = write(tmp_path, "bogus.bin", b"\x00\x01\x02\x03not-an-mp4")
+12 -2
View File
@@ -4,8 +4,9 @@ import sys
import click
from ruamel.yaml import YAML
from ruamel.yaml.constructor import DuplicateKeyError
from unshackle.core.config import config, get_config_path
from unshackle.core.config import config, duplicate_key_lines, get_config_path, load_yaml
from unshackle.core.constants import context_settings
@@ -41,7 +42,16 @@ def cfg(ctx: click.Context, key: str, value: str, unset: bool, list_: bool) -> N
config_path = get_config_path() or config.directories.user_configs / config.filenames.root_config
if config_path.exists():
data = yaml.load(config_path)
try:
data = yaml.load(config_path)
except DuplicateKeyError as e:
if duplicate_key_lines(yaml.compose(config_path)):
load_yaml(config_path)
else:
log.warning(
f"{config_path}: duplicate key on line {e.problem_mark.line + 1}. Unshackle uses the later one."
)
sys.exit(1)
if not data:
log.warning("No config file was found or it has no data, yet")
+107 -140
View File
@@ -27,7 +27,6 @@ from uuid import UUID
import click
import mediaexport
import yaml
from click.core import ParameterSource
from langcodes import Language, tag_is_valid
from pymediainfo import MediaInfo
@@ -43,7 +42,7 @@ from rich.tree import Tree
from unshackle.core import __version__, binaries, providers
from unshackle.core.cdm import DecryptLabsRemoteCDM
from unshackle.core.cdm.detect import cdm_type_stub, is_playready_cdm, is_remote_cdm, is_widevine_cdm
from unshackle.core.config import cdm_entry_names_device, config, resolve_cdm_name, resolve_decryption
from unshackle.core.config import cdm_entry_names_device, config, load_yaml, resolve_cdm_name, resolve_decryption
from unshackle.core.console import GradientPulseBarColumn, SyncLive, console, listing_panel
from unshackle.core.constants import (
DEFAULT_SAMPLE,
@@ -54,7 +53,13 @@ from unshackle.core.constants import (
context_settings,
)
from unshackle.core.credential import Credential
from unshackle.core.downloaders import default_max_workers, format_speed, parse_speed_limit, set_speed_limit
from unshackle.core.downloaders import (
default_max_workers,
format_speed,
parse_speed_limit,
retries_exhausted,
set_speed_limit,
)
from unshackle.core.drm import DRM_T, ClearKeyCENC, MonaLisa, PlayReady, Widevine, cbcs, own_kids, real_kids, verify
from unshackle.core.events import events
from unshackle.core.export_name import ExportNamer, move_export, plural, season_episodes, title_label
@@ -176,10 +181,7 @@ def validate_metadata_ids(
(flag, value) for flag, value in (("--tmdb", tmdb_id), ("--imdb", imdb_id), ("--tvdb", tvdb_id)) if value
]
if len(supplied) > 1:
raise click.UsageError(
f"{', '.join(flag for flag, _ in supplied)} cannot be used together. "
"Give one ID and unshackle resolves the others from it."
)
raise click.UsageError(f"{', '.join(flag for flag, _ in supplied)} cannot be used together; give one ID")
if not supplied:
return
@@ -192,9 +194,7 @@ def validate_metadata_ids(
if consumers:
keys = sorted({ID_PROVIDER_KEYS[cls.NAME] for cls in consumers if cls.NAME in ID_PROVIDER_KEYS})
raise click.UsageError(
f"{flag} needs the {consumers[0].NAME} provider. Set {' or '.join(keys)} in your config."
)
raise click.UsageError(f"{flag} needs the {consumers[0].NAME} provider: set {' or '.join(keys)} in your config")
names = sorted({cls.NAME for cls in providers.REGISTRY.values() if cls.ID_KIND == id_kind})
raise click.UsageError(
f"{flag} needs the {' or '.join(names)} provider, which your metadata_providers config leaves out."
@@ -236,39 +236,28 @@ def select_hybrid_bases(videos: Sequence[Video]) -> list[Video]:
return list(bases.values())
def hybrid_gate_failure(videos: Sequence[Video], generate_dv: bool) -> Optional[tuple[str, str]]:
"""The (message, detail) that stops HYBRID for these selected videos, or None when it can go on."""
def hybrid_gate_failure(videos: Sequence[Video], generate_dv: bool) -> Optional[str]:
"""The message that stops HYBRID for these selected videos, or None when it can go on."""
if any(v.native_hybrid for v in videos):
return None
hdr_tracks = [v for v in videos if v.range in (Video.Range.HDR10, Video.Range.HDR10P)]
base_tracks = [v for v in hdr_tracks if v.codec in (Video.Codec.HEVC, None)]
dv_tracks = [v for v in videos if v.range == Video.Range.DV]
has_hdr10p = any(v.range == Video.Range.HDR10P for v in base_tracks)
available_ranges = sorted(set(v.range.name for v in videos))
ranges = ", ".join(sorted(set(v.range.name for v in videos))) or "none"
missing = f"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: {ranges}"
if not hdr_tracks and not dv_tracks:
return (
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but neither is available",
f"Available ranges: {', '.join(available_ranges) if available_ranges else 'none'}",
)
if not hdr_tracks:
return (
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only DV is available",
f"Available ranges: {', '.join(available_ranges)}",
)
return missing
if not base_tracks:
codecs = sorted(set(v.codec.name for v in hdr_tracks if v.codec))
return (
"HYBRID mode requires an HEVC HDR10/HDR10+ base, but none is available",
f"Available HDR10/HDR10+ codecs: {', '.join(codecs)}",
)
return f"HYBRID needs an HEVC HDR10/HDR10+ base, available codecs: {', '.join(codecs)}"
if not dv_tracks and not (generate_dv and has_hdr10p):
msg_detail = f"Available ranges: {', '.join(available_ranges)}"
if generate_dv:
msg_detail += ". --generate-dv needs an HEVC HDR10+ track, and the service returned none."
elif has_hdr10p:
msg_detail += ". Add --generate-dv to generate Dolby Vision from HDR10+."
return "HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only HDR10 is available", msg_detail
return f"{missing}; --generate-dv found no HEVC HDR10+ track"
if has_hdr10p:
return f"{missing}; add --generate-dv to make DV from HDR10+"
return missing
return None
@@ -486,10 +475,7 @@ def apply_service_dl_overrides(ctx: click.Context, service_dl_config: dict[str,
ctx.params[name] = param.type_cast_value(ctx, value)
log.debug(f"Applied service dl override '{name}': {ctx.params[name]}")
except Exception as e:
log.warning(
f"Failed to apply service dl override '{name}'={value!r}: {e}. "
f"Check that the value is valid for this parameter."
)
log.warning(f"Failed to apply service dl override '{name}'={value!r}: {e}")
def download_tracks_in_passes(
@@ -1278,16 +1264,12 @@ class dl:
# report which weren't available without parsing the console output. See SkippedSubtitle.
self.skipped_subtitles: list[SkippedSubtitle] = []
# result() catches a download-worker failure, reports it, and returns rather than
# re-raising (so the CLI still exits cleanly). Flag it so an embedding caller (the API
# re-raising (so the CLI still exits cleanly). Keep it so an embedding caller (the API
# worker) can tell the title did not complete instead of reading it as a success.
self.download_failed: bool = False
self.download_error: Optional[BaseException] = None
if not config.output_template:
raise click.ClickException(
"No 'output_template' configured in your unshackle.yaml.\n"
"Please add an 'output_template' section with movies, series, and songs/music templates.\n"
"See unshackle-example.yaml for examples."
)
raise click.ClickException("No 'output_template' in your unshackle.yaml")
self.service = Services.get_tag(ctx.invoked_subcommand)
self.vault_service = Services.get_vault_tag(self.service)
@@ -1330,7 +1312,7 @@ class dl:
)
if self.daily and not self.enrich:
self.log.warning("--daily fills air dates from TVDB only with --enrich. A service can set them itself.")
self.log.warning("--daily takes TVDB air dates only with --enrich")
# Initialize debug logger with service name if debug logging is enabled
if config.debug or logging.root.level == logging.DEBUG:
@@ -1454,7 +1436,7 @@ class dl:
try:
service_config_path = Services.get_path(self.service) / config.filenames.config
if service_config_path.exists():
self.service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
self.service_config = load_yaml(service_config_path)
self.log.info("Service Config loaded")
# log key names only: the full config carries service certificates,
# device fingerprints and endpoints that bloat the log and may be sensitive
@@ -1795,7 +1777,7 @@ class dl:
("-fsl/--forced-s-lang", fsl_excl),
):
if any(token.lower() == "all" for token in excludes):
self.log.error(f"{flag}: 'all' cannot be excluded. Name the languages to exclude instead.")
self.log.error(f"{flag}: 'all' cannot be excluded")
sys.exit(1)
# exclusion-only use falls back to the flag's own default; -vl/-al have none, they cascade to -l
if lang_excl and not lang:
@@ -1905,7 +1887,7 @@ class dl:
if vbitrate_min > vbitrate_max:
vbitrate_min, vbitrate_max = vbitrate_max, vbitrate_min
except (ValueError, IndexError):
self.log.error("Invalid --vbitrate-range format. Use 'MIN-MAX' (e.g., '6000-7000').")
self.log.error("Invalid --vbitrate-range: expected MIN-MAX, such as 6000-7000")
sys.exit(1)
abitrate_min, abitrate_max = None, None
@@ -1921,7 +1903,7 @@ class dl:
if abitrate_min > abitrate_max:
abitrate_min, abitrate_max = abitrate_max, abitrate_min
except (ValueError, IndexError):
self.log.error("Invalid --abitrate-range format. Use 'MIN-MAX' (e.g., '128-256').")
self.log.error("Invalid --abitrate-range: expected MIN-MAX, such as 128-256")
sys.exit(1)
if not acodec:
@@ -1945,15 +1927,13 @@ class dl:
from unshackle.core.binaries import DoviTool
if not DoviTool:
self.log.error("Unable to run hybrid mode: dovi_tool not detected")
self.log.error("Please install dovi_tool from https://github.com/quietvoid/dovi_tool")
self.log.error("Unable to run hybrid mode: dovi_tool not found")
sys.exit(1)
if generate_dv:
from unshackle.core.binaries import HDR10PlusTool
if not HDR10PlusTool:
self.log.error("Unable to run --generate-dv: hdr10plus_tool not detected")
self.log.error("Please install hdr10plus_tool from https://github.com/quietvoid/hdr10plus_tool")
self.log.error("Unable to run --generate-dv: hdr10plus_tool not found")
sys.exit(1)
if cdm_only is None:
@@ -2072,13 +2052,13 @@ class dl:
if enrich_lang:
self.log.info(f"Original language from metadata: {enrich_lang}")
if not (enrich_title or enrich_year or enrich_lang):
self.log.warning("--enrich found no metadata. Is the provider's API key configured?")
self.log.warning("--enrich found no metadata")
elif missing := [
name
for name, value in (("title", enrich_title), ("year", enrich_year), ("language", enrich_lang))
if not value
]:
self.log.warning(f"--enrich source gave no {', '.join(missing)}. Those fields are unchanged.")
self.log.warning(f"--enrich source gave no {', '.join(missing)}, kept the service's values")
if isinstance(titles, (Series, Movies)):
for t in titles:
@@ -2863,12 +2843,10 @@ class dl:
self.log.info("No Dolby Vision track: Dolby Vision will be generated from HDR10+")
if gate_failure:
msg, msg_detail = gate_failure
other_ranges = [r for r in range_ if r != Video.Range.HYBRID]
if best_available and other_ranges:
self.log.warning(msg)
self.log.warning(
f"Continuing with remaining range(s): {', '.join(r.name for r in other_ranges)}"
f"{gate_failure}; continuing with {', '.join(r.name for r in other_ranges)}"
)
range_ = other_ranges
fallback_pool = pre_hybrid_videos
@@ -2897,8 +2875,7 @@ class dl:
fallback_selected.append(match)
title.tracks.videos = fallback_selected
else:
self.log.error(msg)
self.log.error(msg_detail)
self.log.error(gate_failure)
sys.exit(1)
fsl = resolve_sort_langs(forced_s_lang, title.language, exact=exact_lang)
@@ -3193,17 +3170,14 @@ class dl:
system_cdms = [self.cdm_for(system.lower(), height) for height in heights]
if not all(system_cdm and matches(system_cdm) for system_cdm in system_cdms):
hint = (
"--cdm selects one device, so remove it."
"drop --cdm, which selects one device"
if self.cdm_override
else f"Set cdm.{self.service}.{system.lower()} in the configuration."
else f"set cdm.{self.service}.{system.lower()}"
)
raise click.ClickException(f"--all-drm needs a {system} CDM for {self.service}. {hint}")
raise click.ClickException(f"--all-drm needs a {system} CDM for {self.service}: {hint}")
if is_remote_cdm(system_cdms[0]) and system not in self.all_drm_warned:
self.all_drm_warned.add(system)
self.log.warning(
f"The {system} CDM is remote. It can answer from its own key cache "
"and send no licence request."
)
self.log.warning(f"The {system} CDM is remote and can answer with cached content keys")
if hasattr(service, "resolve_server_keys"):
service.resolve_server_keys(title)
@@ -3310,11 +3284,11 @@ class dl:
self.log_export_summary()
return
except Exception as e: # noqa
# Reported and swallowed (no re-raise) so the CLI exits cleanly; flag it so the
# Reported and swallowed (no re-raise) so the CLI exits cleanly; keep it so the
# API worker sees the title failed rather than completing with no output.
self.download_failed = True
self.download_error = e
console.print(Padding(Group(*self.failure_lines(e)), (1, 5)))
if not isinstance(e, click.ClickException):
if not isinstance(e, click.ClickException) and not retries_exhausted(e):
console.print_exception()
if self.debug_logger:
@@ -3505,10 +3479,7 @@ class dl:
has_repacked = True
events.emit(events.Types.TRACK_REPACKED, track=track)
if has_encrypted_sample_entry(track.path):
self.log.warning(
f"Track {track.id} still has an encrypted sample entry after repacking, so "
"decryption likely failed and the muxed output may be unplayable."
)
self.log.warning(f"Track {track.id} still has an encrypted sample entry after repack")
if has_repacked:
self.log.info("Repacked one or more tracks with FFMPEG")
@@ -4208,9 +4179,7 @@ class dl:
collisions = [slot for slot, count in Counter(renumbered).items() if count > 1]
if collisions:
self.log.error(
"TVDB's %s order does not cover %d of this series' episodes, and renumbering the rest "
"would give %d season/episode slot(s) two episodes each. Keeping the %s numbering "
"%s uses. Pick an order that covers the whole series.",
"TVDB's %s order misses %d episode(s) and puts two in %d slot(s), keeping the %s numbering of %s",
self.tvdb_order,
unmapped,
len(collisions),
@@ -4406,7 +4375,7 @@ class dl:
keys = len(doc.key_pool())
self.log.info(f"Saved the export to {namer.path}: {plural(len(doc.titles), 'title')}, {plural(keys, 'key')}")
if keys:
self.log.warning("The export holds content keys. Keep it private.")
self.log.warning("The export holds content keys: keep it private")
def export_entry(self, title: Title_T) -> mediaexport.Entry:
"""A mediaexport entry for ``title``: manifests, chapters, a small track list, no keys yet."""
@@ -4480,18 +4449,19 @@ class dl:
@staticmethod
def failure_lines(e: BaseException) -> list[str]:
"""Return the lines that report a failed download.
"""Return the header and the one line that report a failed download.
A ClickException is a deliberate stop with a message for the user, so it gets no
"unexpected error" line, and the caller prints no traceback for it.
A ClickException is a deliberate stop with a message for the user, so it is reported
without its type, and the caller prints no traceback for it. Neither does a request
that failed on every retry: its line ends with its retry notes.
"""
if isinstance(e, click.ClickException):
return [":x: Download Failed...", f" {e.format_message()}"]
lines = [":x: Download Failed...", f" {type(e).__name__}: {e}"]
if (returncode := getattr(e, "returncode", None)) is not None:
lines.append(f" Binary call failed, Process exit code: {returncode}")
lines.append(" An unexpected error occurred in one of the download workers.")
return lines
line = e.format_message()
else:
line = f"{type(e).__name__}: {e}"
if retries_exhausted(e):
line += f", {'; '.join(e.__notes__)}"
return [":x: Download Failed...", f" {line}"]
def decrypt_verified(
self,
@@ -4508,13 +4478,15 @@ class dl:
then copied to the other vaults, once FFmpeg can decode the result. Without FFmpeg
neither happens. For a fragmented MP4, ``verify.kid_windows`` reads from the ciphertext
which fragments each KID encrypts, and FFmpeg decodes windows inside each KID's
fragments, so only a KID that fails is flagged. A KID that is not in the map, and every KID
of any other file, gets a start and end check of the whole file, and a failure there flags
each of those KIDs. On failure the pair is flagged in the local vaults and reported
to the vault that served it, the ciphertext is restored, and the licence runs again. Every vault skips a
flagged pair, so each pass burns at most one vault; when the vaults run out the CDM
answers, and a CDM key is not checked. A CDM that returns the flagged key clears the
flag and stores the content key again, because then the decode check was wrong.
fragments, so only a KID that fails is flagged. A complete map does not judge a KID it does
not hold, so a clear file judges and copies no KID. A KID that is not in an incomplete map,
and every KID of any other file, gets a start and end check of the whole file, and a
failure there flags each of those KIDs. On failure the pair is flagged in the local
vaults, the ciphertext is restored, and the licence runs again; ``Vaults`` tells the
vault that served it. Every vault skips a flagged pair, so each pass burns at most one
vault; when the vaults run out the CDM answers, and a CDM key is not checked. A CDM
that returns the flagged key clears the flag and stores the content key again, because
then the decode check was wrong.
With ``decrypt=False`` the downloader already decrypted the segments in place, so a
failure can only flag and raise; the next run skips the flagged pair.
@@ -4575,6 +4547,13 @@ class dl:
def window_decodes(start: float, end: float) -> bool:
return ffmpeg_decodes(path, start=start, seconds=end - start if end > start else 0.5, video=video)
def judged_kids() -> dict[UUID, Vault]:
"""The vault KIDs this file can judge. A map whose KIDs all have keys skips the KIDs it lacks."""
kids = vault_kids()
if kid_map is not None and all(kid in keys for kid in windows):
return {kid: vault for kid, vault in kids.items() if kid in windows}
return kids
def failed_kids(kids: dict[UUID, Vault]) -> set[UUID]:
"""The vault KIDs whose content key did not decode.
@@ -4605,20 +4584,21 @@ class dl:
done += 1
if decrypt:
drm.decrypt(path)
kids = vault_kids()
kids = judged_kids()
stale = flagged_kids()
failed: set[UUID] = set()
if not stale:
if not kids or not binaries.FFMPEG or not path.exists():
return
failed = failed_kids(kids)
checked = bool(kids and binaries.FFMPEG and path.exists())
failed = failed_kids(kids) if checked else set()
if not failed:
self.flush_vault_writes(
[
partial(self.vaults.add_key, kid, keys[kid], excluding=vault)
for kid, vault in kids.items()
]
)
if checked:
self.flush_vault_writes(
[
partial(self.vaults.add_key, kid, keys[kid], excluding=vault)
for kid, vault in kids.items()
]
)
self.vaults.report_replaced(keys)
return
bad = {kid: drop(kid) for kid in {*failed, *stale}}
for kid, key in bad.items():
@@ -4626,21 +4606,23 @@ class dl:
continue
self.log.warning(f"{key} from {kids[kid].name} was bad, trying other vaults")
self.vaults.flag_bad_key(kid, key)
if report := getattr(kids[kid], "report_bad", None):
report(kid, key)
if not backup.exists() or not licence:
raise ValueError("The content key from the vault did not decrypt the track; run again")
path.unlink()
os.link(backup, path)
relicense(set(bad))
for kid, key in bad.items():
if keys.get(kid) != key:
new = keys.get(kid)
if new is None:
continue
source = self.vaults.sources.get(kid)
if source is None:
self.vaults.unflag_bad_key(kid, key)
self.flush_vault_writes([partial(self.vaults.add_key, kid, key)])
elif kid in kids and source[1] is kids[kid]:
if new == key:
self.vaults.unflag_bad_key(kid, key)
self.flush_vault_writes([partial(self.vaults.add_key, kid, key)])
elif not self.vaults.held_locally(kid, new):
self.flush_vault_writes([partial(self.vaults.add_key, kid, new)])
elif new == key and kid in kids and source[1] is kids[kid]:
raise ValueError(f"{key} from {kids[kid].name} was bad and no other source has the key")
raise ValueError("No vault or CDM produced a content key that decrypts the track")
finally:
@@ -4729,9 +4711,7 @@ class dl:
flag came from a decode check that was wrong, and a local vault refuses a flagged pair."""
if not content_keys or not self.vaults:
return
for kid, key in content_keys.items():
if self.vaults.is_flagged(kid, key):
self.vaults.unflag_bad_key(kid, key)
self.vaults.clear_licensed_flags(content_keys)
successful_caches = self.vaults.add_keys(content_keys)
kids, caches = self.vault_cache_tally or (set(), successful_caches)
self.vault_cache_tally = (kids | content_keys.keys(), min(caches, successful_caches))
@@ -4834,8 +4814,7 @@ class dl:
if not others and other_type.__name__ not in self.all_drm_warned:
self.all_drm_warned.add(other_type.__name__)
self.log.warning(
f"A track does not offer {other_type.__name__}, so it licenses with "
f"{type(drm).__name__} only. This warning shows one time: {track}"
f"Track {track} has no {other_type.__name__} DRM, licensing with {type(drm).__name__} only"
)
targets = [drm, *others[:1]]
@@ -5035,34 +5014,22 @@ class dl:
drm.content_keys.update(local_drm.content_keys)
return
licence = partial(licence, drm_system="playready" if isinstance(drm, PlayReady) else "widevine")
drm_system = "playready" if isinstance(drm, PlayReady) else "widevine"
licence = partial(licence, drm_system=drm_system)
if not server_cdm:
if isinstance(drm, Widevine):
if not is_widevine_cdm(self.cdm):
widevine_cdm = self.cdm_for("widevine", track_quality)
if widevine_cdm and is_widevine_cdm(widevine_cdm):
if track_quality:
self.log.info(f"Switching to Widevine CDM for Widevine {track_quality}p content")
else:
self.log.info("Switching to Widevine CDM for Widevine content")
self.cdm = widevine_cdm
track_cdm = widevine_cdm
else:
raise ValueError(f"Title needs a Widevine CDM but {self.service} is configured with PlayReady.")
elif isinstance(drm, PlayReady):
if not is_playready_cdm(self.cdm):
playready_cdm = self.cdm_for("playready", track_quality)
if playready_cdm and is_playready_cdm(playready_cdm):
if track_quality:
self.log.info(f"Switching to PlayReady CDM for PlayReady {track_quality}p content")
else:
self.log.info("Switching to PlayReady CDM for PlayReady content")
self.cdm = playready_cdm
track_cdm = playready_cdm
else:
raise ValueError(f"Title needs a PlayReady CDM but {self.service} is configured with Widevine.")
if not server_cdm and isinstance(drm, (Widevine, PlayReady)):
matches = is_playready_cdm if drm_system == "playready" else is_widevine_cdm
if not matches(self.cdm):
name, other = ("PlayReady", "Widevine") if drm_system == "playready" else ("Widevine", "PlayReady")
system_cdm = self.cdm_for(drm_system, track_quality)
if system_cdm and matches(system_cdm):
self.log.info(f"Switching to the {name} CDM")
self.cdm = system_cdm
track_cdm = system_cdm
elif self.cdm is None:
raise click.ClickException(f"No CDM configured for {self.service} and no default CDM")
else:
raise ValueError(f"Title needs a {name} CDM but {self.service} is configured with {other}.")
if isinstance(drm, Widevine):
if self.debug_logger:
+6 -6
View File
@@ -63,9 +63,9 @@ def new(
elif len(paths) == 2:
group_key, group_certificate = paths
if not group_key.is_file():
raise click.UsageError("group_key: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("group_key: Not a path to a file, or it does not exist.", ctx)
if not group_certificate.is_file():
raise click.UsageError("group_certificate: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("group_certificate: Not a path to a file, or it does not exist.", ctx)
device_dir = None
else:
raise click.UsageError(
@@ -73,9 +73,9 @@ def new(
ctx,
)
if encryption_key and not encryption_key.is_file():
raise click.UsageError("encryption_key: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("encryption_key: Not a path to a file, or it does not exist.", ctx)
if signing_key and not signing_key.is_file():
raise click.UsageError("signing_key: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("signing_key: Not a path to a file, or it does not exist.", ctx)
log = logging.getLogger("prd")
@@ -161,7 +161,7 @@ def reprovision_device(
) -> None:
"""Reprovision a PlayReady Device (.prd) file."""
if not prd_path.is_file():
raise click.UsageError("prd_path: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("prd_path: Not a path to a file, or it does not exist.", ctx)
log = logging.getLogger("prd")
log.info("Reprovisioning Playready Device (.prd) file, %s", prd_path.name)
@@ -234,7 +234,7 @@ def test(
"""Do a test of a PlayReady device on the Microsoft demo server."""
if not device.is_file():
raise click.UsageError("device: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("device: Not a path to a file, or it does not exist.", ctx)
log = logging.getLogger("prd")
+3 -6
View File
@@ -6,13 +6,12 @@ import sys
from typing import Any, Optional
import click
import yaml
from rich.padding import Padding
from rich.rule import Rule
from rich.tree import Tree
from unshackle.commands.dl import dl
from unshackle.core.config import config
from unshackle.core.config import config, load_yaml
from unshackle.core.console import console
from unshackle.core.constants import context_settings
from unshackle.core.proxies.proxy import Proxy
@@ -91,7 +90,7 @@ def search(
with console.status("Loading Service Config...", spinner="dots"):
service_config_path = Services.get_path(service) / config.filenames.config
if service_config_path.exists():
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
service_config = load_yaml(service_config_path)
log.info("Service Config loaded")
merge_dict(ci_get(config.services, service), service_config)
@@ -165,6 +164,4 @@ def result(service: Service, profile: Optional[str] = None, remote: bool = False
if search_results.children:
console.print(Padding(search_results, (0, 5)))
else:
console.print(
Padding("[bold text]No matches[/]\n[bright_black]Please check spelling and search again....[/]", (0, 5))
)
console.print(Padding("[bold text]No matches[/]", (0, 5)))
+3 -5
View File
@@ -135,9 +135,7 @@ def _prepare_serve(serve: dict, no_key: bool) -> Optional[str]:
"""
api_secret = None if no_key else serve.get("api_secret")
if not no_key and not api_secret:
raise ValueError(
"API secret key is not configured. Please add 'api_secret' to the 'serve' section in your config."
)
raise ValueError("No 'api_secret' in the 'serve' section of your config")
for key, directory, pattern in (
("devices", config.directories.wvds, "*.wvd"),
@@ -213,8 +211,8 @@ def _build_cdm_config(
user_config["playready_devices"] = []
if serve_playready:
log.warning(
f'User "{key_id(user_key)}" has no "playready_devices" configured; PlayReady access disabled '
f"for this user. Available PlayReady devices: {prd_device_names}"
f'User "{key_id(user_key)}" has no "playready_devices", PlayReady is off for this user '
f"(available: {prd_device_names})"
)
serve_config["devices"] = wvd_devices
serve_config["playready_devices"] = prd_devices
+1 -2
View File
@@ -38,8 +38,7 @@ def tunnel_test(server: Optional[str]) -> None:
# one exit can answer over IPv4 here and IPv6 there, so a matching country also passes
if remote.get("ip") != local["ip"] and remote.get("country") != local.get("country"):
raise click.ClickException(
f"Step 3 failed: the server's requests exit at {remote.get('ip')} ({country}), "
f"not where this machine exits. Report this with the IP addresses."
f"Step 3 failed: the server's requests exit at {remote.get('ip')} ({country}), not at this machine's exit"
)
log.info(f"Step 3: the server's requests exit at {remote['ip']} ({country}), {remote.get('ms')} ms for one lookup")
log.info("The tunnel works. Add --local-proxy to a --remote command.")
+5 -5
View File
@@ -223,15 +223,15 @@ def new(
# TODO: Remove need for name, create name based on Client IDs ClientInfo values
name = unidecode(name.strip().lower().replace(" ", "_"))
except UnidecodeError as e:
raise click.UsageError(f"name: Failed to sanitize name, {e}", ctx)
raise click.UsageError(f"name: Failed to sanitise name, {e}", ctx)
if not name:
raise click.UsageError("name: Empty after sanitizing, please make sure the name is valid.", ctx)
raise click.UsageError("name: Empty after sanitising", ctx)
if not private_key.is_file():
raise click.UsageError("private_key: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("private_key: Not a path to a file, or it does not exist.", ctx)
if not client_id.is_file():
raise click.UsageError("client_id: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("client_id: Not a path to a file, or it does not exist.", ctx)
if file_hashes and not file_hashes.is_file():
raise click.UsageError("file_hashes: Not a path to a file, or it doesn't exist.", ctx)
raise click.UsageError("file_hashes: Not a path to a file, or it does not exist.", ctx)
device = Device(
type_=DeviceTypes[type_.upper()],
+12 -16
View File
@@ -20,7 +20,7 @@ from typing import Any, Callable, Dict, List, Optional
from unshackle.core.api.events import bus, publish_service_event
from unshackle.core.api.sanitize import sanitize_log
from unshackle.core.utils.click_types import AUDIO_CODEC_LIST, QUALITY_LIST, VIDEO_CODEC_LIST
from unshackle.core.utils.redact import REDACTED, URL_USERINFO_RE, redact_path, redact_text
from unshackle.core.utils.redact import REDACTED, URL_USERINFO_RE, redact_all, redact_path, redact_text
log = logging.getLogger("download_manager")
@@ -371,11 +371,10 @@ def perform_download(
# Import dl.py components lazily to avoid circular deps during module import
import click
import yaml
from unshackle.commands.dl import dl
from unshackle.core.api.errors import APIError, APIErrorCode
from unshackle.core.config import config
from unshackle.core.config import config, load_yaml
from unshackle.core.services import Services
from unshackle.core.tracks import Video
from unshackle.core.utils.click_types import ContextData
@@ -467,7 +466,7 @@ def perform_download(
remote = params.get("remote")
service_config_path = None if remote else Services.get_path(service) / config.filenames.config
if service_config_path and service_config_path.exists():
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
service_config = load_yaml(service_config_path)
else:
service_config = {}
merge_dict(config.services.get(service), service_config)
@@ -678,26 +677,23 @@ def perform_download(
except SystemExit as exc:
if exc.code != 0:
stdout_str = stdout_capture.getvalue()
stderr_str = stderr_capture.getvalue()
log.error(f"Download exited with code {exc.code}")
log.error(f"Stdout: {stdout_str}")
log.error(f"Stderr: {stderr_str}")
log.debug("Stdout: %s", stdout_capture.getvalue())
log.debug("Stderr: %s", stderr_capture.getvalue())
raise APIError(APIErrorCode.DOWNLOAD_ERROR, f"Download failed with exit code {exc.code}")
except Exception as exc: # noqa: BLE001 - propagate to caller
stdout_str = stdout_capture.getvalue()
stderr_str = stderr_capture.getvalue()
log.error(f"Download execution failed: {exc}")
log.error(f"Stdout: {stdout_str}")
log.error(f"Stderr: {stderr_str}")
log.debug("Stdout: %s", stdout_capture.getvalue())
log.debug("Stderr: %s", stderr_capture.getvalue())
raise
# dl.result() catches a download-worker exception, reports it, but returns normally (exit 0).
# It sets download_failed in that case, so the job isn't reported as completed with no output.
if getattr(dl_instance, "download_failed", False):
detail = (stdout_capture.getvalue() + stderr_capture.getvalue())[-200:].strip()
raise APIError(APIErrorCode.WORKER_ERROR, "download worker failed: " + (detail or "see logs"))
# It keeps the error in download_error, so the job isn't reported as completed with no output.
if (error := getattr(dl_instance, "download_error", None)) is not None:
# segment URLs carry signed CDN tokens (hmac, session IDs) that redact_text does not match
detail = redact_all(dl_instance.failure_lines(error)[1].strip())
raise APIError(APIErrorCode.WORKER_ERROR, f"download worker failed: {detail}")
if progress_callback:
skipped_subs = getattr(dl_instance, "skipped_subtitles", None)
+59 -56
View File
@@ -28,13 +28,13 @@ from unshackle.core.api.session_log import SessionLogBuffer, SessionLogMirror, c
from unshackle.core.api.session_store import SessionStore
from unshackle.core.cacher import Cacher
from unshackle.core.cdm.detect import cdm_type_stub
from unshackle.core.config import config
from unshackle.core.config import config, load_yaml
from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_CODEC_MAP
from unshackle.core.drm import cbcs
from unshackle.core.providers.anilist import parse_anilist_ref
from unshackle.core.proxies.resolve import initialize_proxy_providers, resolve_proxy
from unshackle.core.services import Services
from unshackle.core.session import utf8_text
from unshackle.core.session import RnetSession, utf8_text
from unshackle.core.titles import Episode, Movie, Song, Title_T
from unshackle.core.titles.music import SONG_FIELDS
from unshackle.core.tracks import Audio, Subtitle, Tracks, Video
@@ -190,13 +190,11 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]
def load_service_yaml(normalized_service: str) -> dict:
"""Load a service's config.yaml and merge it with the global override block."""
import yaml
from unshackle.core.utils.collections import merge_dict
service_config_path = Services.get_path(normalized_service) / config.filenames.config
if service_config_path.exists():
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8")) or {}
service_config = load_yaml(service_config_path) or {}
else:
service_config = {}
merge_dict(config.services.get(normalized_service), service_config)
@@ -273,8 +271,7 @@ def instantiate_service(
for k in cli_param_names & transport_keys & set(data):
if k not in ("service", "title_id", "proxy", "no_proxy") and data.get(k) is not None:
log.warning(
f"Ignoring flat '{sanitize_log(k)}' as a service option (it names a transport field); "
"update the client to send service options under 'service_params'"
f"Ignoring flat '{sanitize_log(k)}' as a service option, send it under 'service_params'"
)
return parent_ctx.invoke(service_module.cli, title=title, **extras)
@@ -1057,8 +1054,7 @@ def choose_session_cdm(
if data.get("cdm_type"):
return False, cap
raise capped_error(
f"The server CDM licenses {service} up to {cap}p for this API key. "
f"Set up a local CDM to license {wanted}p with your own device, or pass -q {cap}.",
f"The server CDM licenses {service} up to {cap}p for this API key, {wanted}p needs a local CDM",
cap,
)
@@ -1208,8 +1204,7 @@ def next_server_profile(service: str, region: Optional[str]) -> Optional[str]:
regions = ", ".join(covered.get("regions") or []) or "none"
raise APIError(
APIErrorCode.INVALID_INPUT,
f"No server account for {service} in region '{region or 'unknown'}'. "
f"Its accounts cover: {regions}. Pass --proxy with one of those country codes.",
f"No server account for {service} in region '{region or 'unknown'}', accounts cover: {regions}",
details={"service": service, "region": region, "regions": covered.get("regions") or []},
)
key = f"{service}:{','.join(p or '' for p in pool)}"
@@ -2162,7 +2157,7 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
"""
for banned in ("postscript", "post_script", "post_scripts"):
if data.get(banned):
return f"'{banned}' is not accepted over the API. Define post_scripts in unshackle.yaml."
return f"'{banned}' is not accepted over the API, only from unshackle.yaml"
# The worker reads cookies/{service}/{profile}.txt, so a profile must not be able to name a path.
try:
@@ -2269,10 +2264,7 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
supplied_ids = [name for name in ("tmdb_id", "imdb_id", "tvdb_id") if data.get(name)]
if len(supplied_ids) > 1:
return (
f"Cannot use multiple external IDs: {', '.join(supplied_ids)}. "
"Give one ID and unshackle resolves the others from it."
)
return f"Cannot use multiple external IDs: {', '.join(supplied_ids)}; give one ID"
exclusive_flags = []
if data.get("video_only"):
@@ -2342,7 +2334,7 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request
if not server_cdm_allowed(request, params.get("service")):
raise APIError(
APIErrorCode.FORBIDDEN,
"Download jobs license with the server CDM, which is not enabled for this key on this service.",
"Download jobs license with the server CDM, which is not enabled for this API key on this service",
)
cap = server_cdm_max_height(request, params.get("service"))
if cap is not None:
@@ -2352,27 +2344,16 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request
"""The job's own value for this parameter, or the ``serve:`` dl default that the job falls back to."""
return params[key] if key in params else serve_defaults.get(key)
capped = f"Download jobs license with the server CDM, up to {cap}p for this API key"
wanted = requested_height(effective("quality"))
if wanted is None or wanted > cap:
raise capped_error(
f"Download jobs license with the server CDM, which covers up to {cap}p for this API key. "
f"Pass a quality of {cap} or lower.",
cap,
)
raise capped_error(f"{capped}: pass a quality", cap)
if effective("best_available"):
raise capped_error(
f"Download jobs license with the server CDM, which covers up to {cap}p for this API key. "
"Turn off best_available, because it can fall back to a track above that height.",
cap,
)
raise capped_error(f"{capped}: drop best_available", cap)
ranges = effective("range") or []
ranges = [ranges] if isinstance(ranges, str) else ranges
if any(str(getattr(r, "value", r)).upper() == "HYBRID" for r in ranges):
raise capped_error(
f"Download jobs license with the server CDM, which covers up to {cap}p for this API key. "
"Remove HYBRID from range, because it always keeps the lowest DV track, whatever the quality.",
cap,
)
raise capped_error(f"{capped}: drop HYBRID", cap)
if params.get("proxy_download") is not None and not isinstance(params["proxy_download"], str):
raise APIError(APIErrorCode.INVALID_INPUT, "proxy_download must be a string.")
@@ -4070,8 +4051,8 @@ async def session_titles_handler(session_id: str, request: Optional[web.Request]
)
def client_session_auth(session: Any) -> tuple[Dict[str, str], Dict[str, str]]:
"""The service session headers and cookies a remote client may receive.
def client_session_auth(session: Any) -> tuple[Dict[str, str], list[Dict[str, Any]]]:
"""The service session headers and cookies (name, value, domain, path, secure) a remote client may receive.
A client whose own login succeeded gets them back. For a server account the cookie jar
stays on the server and the server drops the auth-bearing headers, so the client cannot
@@ -4080,18 +4061,31 @@ def client_session_auth(session: Any) -> tuple[Dict[str, str], Dict[str, str]]:
"""
svc_session = session.service_instance.session
headers = dict(svc_session.headers) if hasattr(svc_session, "headers") and svc_session.headers else {}
cookies: Dict[str, str] = {}
if hasattr(svc_session, "cookies"):
for cookie in svc_session.cookies:
if hasattr(cookie, "name") and hasattr(cookie, "value"):
cookies[cookie.name] = cookie.value
cookies: list[Dict[str, Any]] = []
jar = svc_session.scoped_cookies() if isinstance(svc_session, RnetSession) else getattr(svc_session, "cookies", [])
for cookie in jar:
if hasattr(cookie, "name") and hasattr(cookie, "value"):
cookies.append(
{
"name": cookie.name,
"value": cookie.value,
"domain": getattr(cookie, "domain", None) or "",
"path": getattr(cookie, "path", None) or "/",
"secure": bool(getattr(cookie, "secure", False)),
}
)
if not client_owns_login(session):
headers = {k: v for k, v in headers.items() if not CONFIG_SECRET_KEY_RE.search(str(k))}
if not session.client_auth:
cookies = {}
cookies = []
return headers, cookies
def cookie_values(cookies: list[Dict[str, Any]]) -> Dict[str, str]:
"""The ``{name: value}`` form of :func:`client_session_auth` cookies, for clients that predate the jar."""
return {c["name"]: c["value"] for c in cookies}
def client_owns_login(session: Any) -> bool:
"""Whether the login state of ``session`` is the client's to take back.
@@ -4234,7 +4228,8 @@ async def session_tracks_handler(
"manifests": manifests,
"track_manifests": track_manifests,
"session_headers": session_headers,
"session_cookies": session_cookies,
"session_cookies": cookie_values(session_cookies),
"session_cookie_jar": session_cookies,
"server_cdm": session.server_cdm,
"server_cdm_type": server_cdm_type,
"sample_decrypt_tracks": [
@@ -4271,6 +4266,7 @@ async def session_segments_handler(
try:
result: Dict[str, Any] = {}
headers, cookies = client_session_auth(session)
for track_id in track_ids:
track = session.tracks.get(track_id)
@@ -4287,10 +4283,11 @@ async def session_segments_handler(
"descriptor": descriptor_name,
"url": str(track.url) if track.url else None,
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
"headers": headers,
"cookies": cookie_values(cookies),
"cookie_jar": cookies,
}
track_info["headers"], track_info["cookies"] = client_session_auth(session)
# Include manifest-specific data for segment resolution. Round-trip through
# JSON so any non-serializable value becomes its str() (default=str).
if hasattr(track, "data") and track.data:
@@ -4557,7 +4554,7 @@ def resolve_handler_proxy(
):
raise APIError(
APIErrorCode.FORBIDDEN,
"A server-account service does not accept a client proxy URI. Pass a country code or nothing.",
"A server-account service takes a country code or no proxy, not a proxy URI",
details={"service": normalized_service},
)
@@ -4587,8 +4584,7 @@ def resolve_handler_proxy(
)
raise APIError(
APIErrorCode.INVALID_PROXY,
"This server does not resolve country codes into proxies. "
"Pass --proxy with a full proxy URI (http://, https:// or socks5://).",
"This server does not resolve country codes into proxies, pass a full proxy URI",
details={"service": normalized_service},
)
@@ -4614,8 +4610,7 @@ def resolve_handler_proxy(
if client_region.lower() in {x.lower() for x in geoblock}:
raise APIError(
APIErrorCode.GEOFENCE,
f"Service is not available in your region ({client_region.upper()}). "
"Pass --proxy with a proxy outside the blocked regions.",
f"Service is not available in your region ({client_region.upper()})",
details={"service": normalized_service},
)
@@ -4624,8 +4619,7 @@ def resolve_handler_proxy(
if server_region and not in_client_region:
raise APIError(
APIErrorCode.INVALID_PROXY,
"This server is in a different region from yours. Pass --proxy with a proxy "
"in the region you want, or pass --no-proxy to use the server's own connection.",
"This server is in a different region from yours, pass --proxy or --no-proxy",
details={"service": normalized_service},
)
elif in_client_region:
@@ -4914,11 +4908,12 @@ CACHED_PAIRS: set[tuple[str, str, str]] = set()
def cache_to_vaults(keys: Dict[str, str], service_name: str) -> None:
"""Cache newly obtained keys to server vaults.
"""Cache the content keys of a CDM licence to the server vaults.
A PlayReady licence always runs, and a Widevine licence runs when one PSSH KID has no content
key in a vault, so the same content keys come back on every request for a title. This process
does not send a pair again after every vault accepted it.
does not send a pair again after every vault accepted it. A pair a client reported as bad
leaves that set, and is unflagged when the CDM returns it.
"""
from uuid import UUID
@@ -4931,6 +4926,8 @@ def cache_to_vaults(keys: Dict[str, str], service_name: str) -> None:
return
key_map = {UUID(hex=kid): key for kid, key in new.items()}
for kid in vaults.clear_licensed_flags(key_map):
log.info(f"Unflagged {kid.hex}:{key_map[kid]}, the CDM returned it")
cached = vaults.add_keys(key_map)
if cached == len([vault for vault in vaults.vaults if not vault.no_push]):
CACHED_PAIRS.update((service_name, kid, key) for kid, key in new.items())
@@ -5053,7 +5050,7 @@ def handle_single_server_cdm(
if sources is not None:
sources.update(vault_hit[1])
return vault_hit[0]
raise refusal or APIError(APIErrorCode.NO_CONTENT, "The server vault holds no key for this track")
raise refusal or APIError(APIErrorCode.NO_CONTENT, "The server vault holds no content key for this track")
api_key = request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous"
user_config = serve_user_config(api_key)
@@ -5333,7 +5330,7 @@ def license_response(
):
raise APIError(
APIErrorCode.FORBIDDEN,
"Server CDM licensing is not enabled for this key on this service. Use a local CDM (proxy mode).",
"Server CDM licensing is not enabled for this API key on this service",
)
if mode == "server_cdm" and track_ids:
@@ -5603,7 +5600,8 @@ async def session_bad_key_handler(
The server never sees a segment, so the client is the only side that can test a content key.
Only a pair this remote session served can be flagged, which keeps a client from
poisoning the bad-key table for content keys it never received.
poisoning the bad-key table for content keys it never received. A pair the CDM licensed is
not flagged, because the CDM is the authority.
"""
from uuid import UUID
@@ -5616,8 +5614,12 @@ async def session_bad_key_handler(
if not key or served_key.lower() != key:
log.warning(f"Session {sanitize_log(session_id[:12])} reported a bad content key it was never served: {kid}")
raise APIError(APIErrorCode.INVALID_INPUT, "This session was not served that KID:KEY pair")
if source == "cdm":
log.warning(f"Session {sanitize_log(session_id[:12])} reported a CDM key as bad, not flagged: {kid}")
return web.json_response({"flagged": False})
vaults = load_server_vaults(session.service_instance.__class__.__name__)
service_name = session.service_instance.__class__.__name__
vaults = load_server_vaults(service_name)
def flag() -> None:
for vault in vaults.vaults:
@@ -5630,6 +5632,7 @@ async def session_bad_key_handler(
await asyncio.to_thread(flag)
session.served_keys.pop(kid, None)
CACHED_PAIRS.discard((service_name, kid, served_key))
log.warning(
f"Client proved {kid}:{served_key} from vault {sanitize_log(source)} wrong, flagged in the server vaults"
)
+1 -4
View File
@@ -202,10 +202,7 @@ class InputBridge:
self._status = value
NO_USER_MESSAGE = (
"This login needs the user, and this request cannot reach the user. "
"Start a download job or a remote session to complete the authentication"
)
NO_USER_MESSAGE = "This login needs the user, and only a download job or a remote session can reach the user"
class NoUserBridge(InputBridge):
+1 -1
View File
@@ -2421,7 +2421,7 @@ async def session_bad_key(request: web.Request) -> web.Response:
description: Content key as hex
responses:
'200':
description: The pair is flagged
description: "flagged true when the pair is flagged; false when the CDM licensed it"
'400':
description: The remote session was not served that pair
'404':
+2 -6
View File
@@ -24,10 +24,7 @@ from unshackle.core.tracks import Track
log = logging.getLogger("api.session")
DRAIN_WARNING = (
"This server restarts for an update when the open downloads finish. "
"This download continues. Start new downloads after the restart."
)
DRAIN_WARNING = "This server restarts for an update after the open downloads finish; this download continues"
@dataclass
@@ -359,8 +356,7 @@ def refuse_while_draining() -> None:
if get_session_store().draining:
raise APIError(
APIErrorCode.SERVER_DRAINING,
"This server restarts for an update and does not start new sessions or downloads now. "
"Try again in a few minutes.",
"This server restarts for an update and starts no new sessions or downloads",
retryable=True,
)
+2 -6
View File
@@ -67,18 +67,14 @@ class Cacher:
checksum = data["crc32"]
calculated = zlib.crc32(jsonpickle.dumps(payload).encode("utf8"))
if calculated != checksum:
raise ValueError(
f"The checksum of the Cache payload mismatched. Checksum: {checksum} !== Calculated: {calculated}"
)
raise ValueError(f"Cache payload checksum mismatch: {checksum} != calculated {calculated}")
cache.data = data["data"]
# The pickled `expiration` is local wall-clock and only right in the zone that wrote it.
expires_at = data.get("expires_at")
cache.expiration = datetime.fromtimestamp(expires_at) if expires_at is not None else data["expiration"]
cache.version = data["version"]
if cache.version != version:
raise ValueError(
f"The version of your {self.service_tag} {key} cache is outdated. Please delete: {cache.path}"
)
raise ValueError(f"The {self.service_tag} {key} cache is outdated, delete {cache.path}")
return cache
def set(self, data: Any, expiration: Optional[EXP_T] = None) -> Any:
@@ -536,7 +536,7 @@ class DecryptLabsRemoteCDM:
session["decrypt_labs_session_id"] = data["session_id"]
return challenge
error_msg = f"Unexpected API response format. message_type={message_type}, available_fields={list(data.keys())}"
error_msg = f"Unexpected API response format: message_type={message_type}, available_fields={list(data.keys())}"
if data.get("message"):
error_msg = f"API response: {data['message']} - {error_msg}"
if "details" in data:
+53 -5
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import logging
import re
import warnings
from pathlib import Path
@@ -143,7 +144,7 @@ class Config:
downloader_cfg = kwargs.get("downloader")
if downloader_cfg and downloader_cfg != "requests":
warnings.warn(
f"downloader '{downloader_cfg}' is deprecated. The unified requests downloader is now used.",
f"downloader '{downloader_cfg}' is deprecated, using the requests downloader",
DeprecationWarning,
stacklevel=2,
)
@@ -214,9 +215,7 @@ class Config:
if kwargs.get("scene_naming") is not None:
raise SystemExit(
"ERROR: The 'scene_naming' option has been removed.\n"
"Please configure 'output_template' in your unshackle.yaml instead.\n"
"See unshackle-example.yaml for examples."
"ERROR: The 'scene_naming' option was removed, use 'output_template' in your unshackle.yaml"
)
if self.output_template:
@@ -374,7 +373,56 @@ class Config:
raise FileNotFoundError(f"Config file path ({path}) was not found")
if not path.is_file():
raise FileNotFoundError(f"Config file path ({path}) is not to a file.")
return cls(**restore_bool_languages(yaml.safe_load(path.read_text(encoding="utf8")) or {}))
return cls(**restore_bool_languages(load_yaml(path) or {}))
def duplicate_key_lines(root: Any) -> list[tuple[int, int]]:
"""
Return (first line, repeat line), counted from 1, for each repeated scalar key in a PyYAML or ruamel node tree.
Keys compare as written, so two spellings that load as the same key (yes and true, 1 and 0x1) are not found.
"""
pairs, stack, visited = [], [root], set()
while stack:
node = stack.pop()
if node in visited:
continue
visited.add(node)
if node.id == "mapping":
first: dict[tuple[str, str], int] = {}
for key_node, value_node in node.value:
if key_node.id == "scalar" and key_node.tag != "tag:yaml.org,2002:merge":
key = (key_node.tag, key_node.value)
line = key_node.start_mark.line + 1
if key in first:
pairs.append((first[key], line))
else:
first[key] = line
stack += [key_node, value_node]
elif node.id == "sequence":
stack += node.value
return sorted(pairs)
_warned: set[str] = set()
def load_yaml(path: Path) -> Any:
"""yaml.safe_load a file, and warn with the lines of each duplicate key, which PyYAML takes silently."""
loader = yaml.SafeLoader(path.read_text(encoding="utf8"))
try:
node = loader.get_single_node()
if node is None:
return None
if pairs := duplicate_key_lines(node):
where = ", ".join(f"lines {first} and {repeat}" for first, repeat in pairs)
message = f"{path}: duplicate key on {where}. Unshackle uses the later one."
if message not in _warned:
_warned.add(message)
logging.getLogger("config").warning(message)
return loader.construct_document(node)
finally:
loader.dispose()
# noinspection PyProtectedMember
+1 -1
View File
@@ -75,7 +75,7 @@ class Credential:
credential = re.fullmatch(r"^([^:]+?):([^:]+?)(?::(.+))?$", text)
if credential:
return cls(*credential.groups())
raise ValueError("No credentials found in text string. Expecting the format `username:password`")
raise ValueError("No credentials in text string, expected `username:password`")
@classmethod
def load(cls, path: Path) -> Credential:
+16 -2
View File
@@ -1,3 +1,17 @@
from .requests import default_max_workers, format_speed, parse_speed_limit, requests, set_speed_limit
from .requests import (
default_max_workers,
format_speed,
parse_speed_limit,
requests,
retries_exhausted,
set_speed_limit,
)
__all__ = ("requests", "default_max_workers", "format_speed", "parse_speed_limit", "set_speed_limit")
__all__ = (
"requests",
"default_max_workers",
"format_speed",
"parse_speed_limit",
"retries_exhausted",
"set_speed_limit",
)
+39 -18
View File
@@ -27,11 +27,12 @@ from requests.adapters import HTTPAdapter, Retry
from rich import filesize
from unshackle.core.constants import DOWNLOAD_CANCELLED, DownloadCancelled
from unshackle.core.utilities import get_debug_logger, get_extension
from unshackle.core.utilities import get_debug_logger, get_extension, log_event
from unshackle.core.utils import run_proxy
MAX_ATTEMPTS = 5
RETRY_WAIT = 2
GAVE_UP_NOTE = "gave up after"
# a cut that still delivered bytes is resumable and refreshes the attempt budget; bound it
# so a server that sends a little and closes, over and over, still terminates
MAX_RESUMES = 50
@@ -153,7 +154,7 @@ def parse_speed_limit(value: Union[str, int, float, None]) -> Optional[float]:
return None
match = re.fullmatch(r"(\d*\.?\d+)\s*([kmg]b?|b)?", text)
if not match:
raise ValueError(f"Invalid speed limit {value!r}: use e.g. 500k, 5M, 1.5G, plain bytes/sec, or 'off'.")
raise ValueError(f"Invalid speed limit {value!r}: expected 500k, 5M, 1.5G, bytes per second or 'off'")
return float(match.group(1)) * SPEED_UNITS[match.group(2) or ""]
@@ -324,15 +325,12 @@ class AdaptiveWorkerController:
def permanent_status(exc: BaseException) -> Optional[int]:
"""HTTP status of ``exc`` when it is a client error that no retry can fix.
Both session types raise requests' HTTPError; RnetSession wraps it in MaxRetriesError
as ``__cause__``, so unwrap one level like retry_sleep does. 408 and 429 do not count:
they are transient, and a fan-out that trips a per-client rate limit can still succeed
over one sequential connection.
408 and 429 do not count: they are transient, and a fan-out that trips a per-client rate
limit can still succeed over one sequential connection.
"""
response = getattr(exc, "response", None)
if response is None:
response = getattr(getattr(exc, "__cause__", None), "response", None)
status = getattr(response, "status_code", None)
from unshackle.core.session import failure_response
status = getattr(failure_response(exc), "status_code", None)
if isinstance(status, int) and 400 <= status < 500 and status not in (408, 429):
return status
return None
@@ -358,12 +356,9 @@ def retry_sleep(exc: Exception, attempts: int) -> float:
else exponential growth from RETRY_WAIT with jitter so concurrently rate-limited
segments do not retry in lockstep. Capped at ``session.MAX_BACKOFF`` either way.
"""
from unshackle.core.session import MAX_BACKOFF
from unshackle.core.session import MAX_BACKOFF, failure_response
response = getattr(exc, "response", None)
if response is None:
# RnetSession wraps forcelist failures in MaxRetriesError with the HTTPError as __cause__
response = getattr(getattr(exc, "__cause__", None), "response", None)
response = failure_response(exc)
if response is not None:
try:
retry_after = response.headers.get("Retry-After")
@@ -387,6 +382,11 @@ def retry_sleep(exc: Exception, attempts: int) -> float:
return min(backoff + random.uniform(0, backoff * 0.1), MAX_BACKOFF)
def retries_exhausted(exc: BaseException) -> bool:
"""Whether ``exc`` is the error download() re-raised after it spent every attempt."""
return any(str(note).startswith(GAVE_UP_NOTE) for note in getattr(exc, "__notes__", ()))
def is_requests_session(session: Any) -> bool:
"""Whether the HTTP session is a standard requests.Session (it has ``resp.raw``)."""
return isinstance(session, Session)
@@ -814,6 +814,9 @@ def download(
item_range = has_range_header(kwargs)
attempts = 1
# attempts resets on progress, so the give-up note counts failures on their own
failures = 0
started = _time()
written = 0
resumes = 0
request_limit: Optional[int] = None
@@ -1141,11 +1144,13 @@ def download(
# a superseded loser's error must not retry or kill the batch
if claimed is not None and claimed():
return
failures += 1
cancelled = DOWNLOAD_CANCELLED.is_set() or (abort is not None and abort.is_set())
if cancelled or attempts == MAX_ATTEMPTS:
# cancel/abort is not an error, but a genuine retry-exhaustion is, so surface it
# so the caller fails fast instead of hitting a missing segment at merge time
if not cancelled:
exc.add_note(f"{GAVE_UP_NOTE} {failures} failed attempts over {_time() - started:.0f}s")
raise
return
if not part_mode:
@@ -1169,6 +1174,13 @@ def download(
except Exception:
pass
delay = retry_sleep(exc, max(1, attempts))
from unshackle.core.session import failure_reason
log_event(
"segment_retry",
message=f"{failure_reason(exc)}, retry {max(1, attempts)}/{MAX_ATTEMPTS - 1} in {delay:.1f}s",
context={"file": save_path.name, "failures": failures, "resumes": resumes},
)
if abort is not None:
# interruptible nap: backoff can reach MAX_BACKOFF, and teardown always
# sets the batch abort, so a parked worker must wake and exit at once
@@ -1315,8 +1327,13 @@ def mp_worker(queue: Any, kwargs: dict[str, Any]) -> None:
for event in requests(**kwargs):
queue.put(event)
except Exception as exc:
summary = " ".join(f"{type(exc).__name__}: {exc}".split())
queue.put({"__mp_error__": f"{summary}\n{traceback.format_exc()}"})
queue.put(
{
"__mp_error__": " ".join(f"{type(exc).__name__}: {exc}".split()),
"__mp_traceback__": traceback.format_exc(),
"__mp_notes__": [str(note) for note in getattr(exc, "__notes__", ())],
}
)
finally:
queue.put({"__mp_done__": True})
@@ -1432,7 +1449,11 @@ def download_multiprocess(
if "__mp_error__" in event:
for p in procs:
p.terminate()
raise RuntimeError(f"segment download child failed: {event['__mp_error__']}")
log_event("segment_child_failed", level="ERROR", message=event.get("__mp_traceback__", ""))
child_error = RuntimeError(f"segment download child failed: {event['__mp_error__']}")
for note in event.get("__mp_notes__", ()):
child_error.add_note(note)
raise child_error
if "total" in event:
continue
if "downloaded" in event:
+21
View File
@@ -15,6 +15,7 @@ import m3u8
from construct import Container
from pymp4.parser import Box
from pyplayready.cdm import Cdm as PlayReadyCdm
from pyplayready.misc.exceptions import InvalidPssh
from pyplayready.system.pssh import PSSH
from pyplayready.system.wrmheader import WRMHeader
from requests import Session
@@ -170,6 +171,26 @@ class PlayReady:
return []
@classmethod
def from_pssh_b64(cls, pssh_b64: str, kid: Optional[UUID] = None) -> PlayReady:
"""Load a base64 PSSH box or PlayReady Object, or a bare UTF-16LE WRMHEADER that some packagers emit.
It wraps a bare WRMHEADER in a PlayReady Object and keeps that as `pssh_b64`, so remote CDMs and
the REST API can parse it again.
"""
try:
pssh = PSSH(pssh_b64)
except InvalidPssh:
try:
raw = base64.b64decode(pssh_b64)
except ValueError:
raw = b""
if not raw.startswith(b"<\x00"):
raise
pssh_b64 = cls.pro_b64_for(WRMHeader(raw))
pssh = PSSH(pssh_b64)
return cls(pssh=pssh, kid=kid, pssh_b64=pssh_b64)
@classmethod
def from_track(cls, track: AnyTrack, session: Optional[Session] = None) -> PlayReady:
if not session:
+47 -10
View File
@@ -9,7 +9,10 @@ DECRYPT_HOOK: Optional[Callable[..., None]] = None
CONTAINERS = {b"moov", b"trak", b"mdia", b"minf", b"stbl", b"mvex", b"moof", b"traf", b"sinf", b"schi"}
PIFF_TENC = bytes.fromhex("8974dbce7be74c5184f97148f9882554")
PIFF_TFXD = bytes.fromhex("6d1d9b0542d544e680e2141daff757b2")
PIFF_SENC = bytes.fromhex("a2394f525a9b4f14a2446c427c648df4")
TENC = {b"tenc", b"uuid" + PIFF_TENC}
ENCRYPTED_ENTRY = (b"enc", b"drm")
SENC = {b"senc", b"uuid" + PIFF_SENC}
# the fixed sample entry fields before the child boxes: visual; sound v0 (and ISO v1), QuickTime sound v1 and v2
ENTRY_FIELDS = {b"encv": (78,), b"enca": (28, 44, 64)}
@@ -123,9 +126,34 @@ def entry_kid(data: bytes, kind: bytes, entry: int, end: int) -> Optional[UUID]:
return None
def sample_entries(data: bytes, body: int, end: int) -> list[Optional[UUID]]:
"""Parse an ``stsd`` box. Returns the default KID of each sample entry, None for a clear one."""
return [entry_kid(data, kind, entry, entry_end) for kind, entry, entry_end in children(data, body + 8, end)]
def sample_entries(data: bytes, body: int, end: int) -> tuple[list[Optional[UUID]], bool]:
"""Parse an ``stsd`` box. Returns the default KID of each sample entry (None for a clear one), and
whether any entry is encrypted."""
found = list(children(data, body + 8, end))
kids = [entry_kid(data, kind, entry, entry_end) for kind, entry, entry_end in found]
return kids, any(kind[:3] in ENCRYPTED_ENTRY for kind, _, _ in found)
def has_encrypted_entry(moov: bytes) -> bool:
"""True if a moov box holds a sample entry of an encrypted type (``enc*`` or ``drm*``)."""
return any(
kind == b"stsd" and any(entry[:3] in ENCRYPTED_ENTRY for entry, _, _ in children(moov, body + 8, end))
for kind, body, end in walk(moov)
)
def senc_protects_samples(buf: bytes, body: int, end: int) -> bool:
"""True if a senc/PIFF-uuid box describes protected samples.
An empty senc (sample_count 0) sits on a genuinely clear fragment, so treating it as
a skipped one would raise on a healthy file. Field order follows
read_per_sample_iv_size.
"""
flags = int.from_bytes(buf[body + 1 : body + 4], "big")
pos = body + 4 + (20 if flags & 0x1 else 0) # PIFF override: AlgorithmID(3)+IV_size(1)+KID(16)
if pos + 4 > end:
return False
return int.from_bytes(buf[pos : pos + 4], "big") > 0
def read_top_boxes(f: BinaryIO) -> Iterator[tuple[bytes, Optional[bytes]]]:
@@ -164,8 +192,10 @@ def kid_windows(path: Path) -> Optional[KidMap]:
the result holds up to three (start, end) windows from ``pick_windows``, relative to
the first fragment of the track. A fragment ends where the next fragment of the track
starts. The last fragment ends after its sample durations, or after 3 seconds when
they are not known. Returns None when the file has no encrypted fragment this reader
can place in time, so the caller must use another check.
they are not known. Returns an empty map for a clear MP4: no track with fragments (or,
without fragments, no track) has an encrypted sample entry or a senc. Returns None when
the file has no encrypted fragment this reader can place in time, so the caller must
use another check; that includes an encrypted sample entry whose fragments are all clear.
"""
timescales: dict[int, int] = {}
handlers: dict[int, bytes] = {}
@@ -174,6 +204,7 @@ def kid_windows(path: Path) -> Optional[KidMap]:
moov_groups: dict[int, list[Seig]] = {}
fragments: list[Fragment] = []
clock: dict[int, int] = {}
protected: set[int] = set()
try:
with path.open("rb") as f:
for kind, box in read_top_boxes(f):
@@ -197,13 +228,15 @@ def kid_windows(path: Path) -> Optional[KidMap]:
# the mdia handler comes first; a QuickTime minf can hold a data handler after it
handlers.setdefault(track, box[body + 8 : body + 12])
elif kind_ == b"stsd":
entries[track] = sample_entries(box, body, box_end)
entries[track], encrypted = sample_entries(box, body, box_end)
if encrypted:
protected.add(track)
elif kind_ == b"sgpd":
groups = seig_entries(box, body)
if groups is not None:
moov_groups[track] = groups
elif kind == b"moof" and box:
fragments.extend(read_fragment(box, entries, defaults, moov_groups, clock))
fragments.extend(read_fragment(box, entries, defaults, moov_groups, clock, protected))
except (OSError, struct.error, IndexError, ValueError, MemoryError, OverflowError, RecursionError):
return None
@@ -230,7 +263,7 @@ def kid_windows(path: Path) -> Optional[KidMap]:
for kid in kids:
spans.setdefault(kid, []).append((start, end))
if not spans:
return None
return KidMap({}, False) if entries and protected.isdisjoint(tracks or entries) else None
return KidMap({kid: pick_windows(fragment_spans) for kid, fragment_spans in spans.items()}, video)
@@ -284,11 +317,13 @@ def read_fragment(
defaults: dict[int, tuple[int, int]],
moov_groups: dict[int, list[Seig]],
clock: dict[int, int],
protected: set[int],
) -> Iterator[Fragment]:
"""Yield (track, decode time, duration, KIDs that encrypt samples) for each track fragment in a moof.
A fragment without a tfdt or tfxd starts where the track's previous fragment ended, by
the running sum of sample durations in ``clock``.
the running sum of sample durations in ``clock``. A track with a senc that protects
samples goes into ``protected``.
"""
for kind, traf, traf_end in children(moof):
if kind != b"traf":
@@ -297,7 +332,7 @@ def read_fragment(
groups: list[Seig] = []
runs: Optional[list[tuple[int, int]]] = None
truns: list[int] = []
for kind_, body, _ in children(moof, traf, traf_end):
for kind_, body, box_end in children(moof, traf, traf_end):
if kind_ == b"tfhd":
flags = int.from_bytes(moof[body + 1 : body + 4], "big")
track = struct.unpack_from(">I", moof, body + 4)[0]
@@ -317,6 +352,8 @@ def read_fragment(
groups = seig_entries(moof, body) or groups
elif kind_ == b"sbgp":
runs = sbgp_runs(moof, body) or runs
elif kind_ in SENC and senc_protects_samples(moof, body, box_end):
protected.add(track)
if track not in entries:
continue
default_entry, trex_duration = defaults.get(track, (1, 0))
+16 -28
View File
@@ -17,7 +17,7 @@ from unshackle.core.config import config
from unshackle.core.constants import AnyTrack
from unshackle.core.credential import Credential
from unshackle.core.drm import drm_from_dict, real_kids
from unshackle.core.manifests import DASH, HLS, ISM
from unshackle.core.manifests import DASH, HLS, ISM, MissingLanguageError
from unshackle.core.remote_service import (
RemoteService,
build_title,
@@ -123,17 +123,15 @@ def resolve_import_manifest_data(
try:
manifest = parser.from_url(url=url, session=session)
parsed = manifest.to_tracks(language=fallback_lang)
except MissingLanguageError:
raise click.ClickException(
f"No language for '{title}': the {manifest_type} manifest has none and Title.language is unset"
)
except ValueError as e:
if "Language information could not be derived" in str(e):
raise click.ClickException(
f"No language for '{title}': the {manifest_type} manifest has none and the "
f"service did not set Title.language."
)
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}'. ({e})")
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}': {e}")
except Exception as e:
raise click.ClickException(
f"Failed to fetch the {manifest_type} manifest for '{title}'. "
f"The manifest URL may have expired since export. ({e})"
f"Failed to fetch the {manifest_type} manifest for '{title}', the URL may have expired: {e}"
)
local_tracks = [*parsed.videos, *parsed.audio, *parsed.subtitles]
@@ -303,7 +301,7 @@ class ImportService:
except Exception as e:
if explicit:
raise click.ClickException(f"Failed to resolve proxy '{proxy_query}': {e}")
log.warning(f"Could not auto-select a proxy for export region '{region}': {e}. Continuing without proxy.")
log.warning(f"No proxy for export region '{region}', continuing without one: {e}")
proxy = None
if proxy:
@@ -386,28 +384,21 @@ class ImportService:
manifest = parser.from_url(url=manifest_url, session=self.session)
except Exception as e:
raise click.ClickException(
f"Failed to fetch the {manifest_type} manifest for '{title}'. "
f"The manifest URL may have expired since export. ({e})"
f"Failed to fetch the {manifest_type} manifest for '{title}', the URL may have expired: {e}"
)
language = fallback_language(title.language)
if language is None and isinstance(manifest, HLS):
language = hls_audio_language(manifest)
if language is None:
raise click.ClickException(
f"No language for '{title}': the export gives no title language and the HLS "
"manifest gives no audio language."
)
raise click.ClickException(f"No language for '{title}': the export and the HLS manifest have none")
try:
parsed = manifest.to_tracks(language=language)
except ValueError as e:
if "Language information could not be derived" in str(e):
raise click.ClickException(
f"No language for '{title}': the {manifest_type} manifest has none and the "
f"service did not set Title.language."
)
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}'. ({e})")
except MissingLanguageError:
raise click.ClickException(
f"No language for '{title}': the {manifest_type} manifest has none and Title.language is unset"
)
except Exception as e:
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}'. ({e})")
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}': {e}")
for track in parsed:
tracks.add(track)
parsed_ids = {str(t.id) for t in tracks}
@@ -787,10 +778,7 @@ class ImportService:
}
)
subject = f"KID {', '.join(kids)}" if kids else f"track {track}"
return click.ClickException(
f"The export has no content key for {subject}, and no vault has it. An import does not "
"license through a CDM: export the title again with the tracks you need selected."
)
return click.ClickException(f"The export has no content key to decrypt {subject}, and no vault has it")
def decrypts_samples(self, track: AnyTrack) -> bool:
return False
+2 -2
View File
@@ -1,5 +1,5 @@
from .dash import DASH
from .dash import DASH, MissingLanguageError
from .hls import HLS
from .ism import ISM
__all__ = ("DASH", "HLS", "ISM")
__all__ = ("DASH", "HLS", "ISM", "MissingLanguageError")
+18 -9
View File
@@ -20,7 +20,6 @@ from zlib import crc32
import requests
from langcodes import Language, tag_is_valid
from lxml.etree import Element, ElementTree, XMLSyntaxError
from pyplayready.system.pssh import PSSH as PR_PSSH
from pywidevine.cdm import Cdm as WidevineCdm
from pywidevine.pssh import PSSH
from requests import Session
@@ -45,6 +44,19 @@ from unshackle.core.utils.redact import safe_display_url
from unshackle.core.utils.xml import load_xml
class MissingLanguageError(ValueError):
"""A manifest track has no language, and the fallback language is missing or invalid."""
def missing_language(where: str, fallback: Any) -> MissingLanguageError:
"""The MissingLanguageError for a track at ``where``, naming why the fallback did not apply."""
if fallback is None:
return MissingLanguageError(f"No language for {where}, and no fallback language was given")
if not tag_is_valid(str(fallback).strip()) or str(fallback).startswith("und"):
return MissingLanguageError(f"No language for {where}, and the fallback language {fallback} is invalid")
return MissingLanguageError(f"No language for {where}")
def append_segment(dst: BinaryIO, segment_file: Path, is_text_subtitle: bool) -> None:
"""Append one segment file to the output, with the text subtitle fixups when they apply."""
if is_text_subtitle:
@@ -287,12 +299,7 @@ class DASH:
track_lang = self.get_language(adaptation_set, rep, fallback=language)
if not track_lang:
msg = "Language information could not be derived from a Representation."
if language is None:
msg += " No fallback language was provided when calling DASH.to_tracks()."
elif not tag_is_valid((str(language) or "").strip()) or str(language).startswith("und"):
msg += f" The fallback language provided is also invalid: {language}"
raise ValueError(msg)
raise missing_language("a DASH Representation", language)
# for some reason it's incredibly common for services to not provide
# a good and actually unique track ID, sometimes because of the lang
@@ -1355,7 +1362,6 @@ class DASH:
)
if not pr_pssh_b64:
continue
pr_pssh = PR_PSSH(pr_pssh_b64)
kid_b64 = protection.findtext("kid")
kid = None
if kid_b64:
@@ -1364,7 +1370,10 @@ class DASH:
except ValueError:
kid = None
drm.append(PlayReady(pssh=pr_pssh, kid=kid, pssh_b64=pr_pssh_b64))
try:
drm.append(PlayReady.from_pssh_b64(pr_pssh_b64, kid=kid))
except Exception as e:
logging.getLogger("DASH").warning(f"Skipping unparseable PlayReady ContentProtection: {e!r}")
elif urn == ClearKeyCENC.urn:
# W3C EME ClearKey (org.w3.clearkey): match the scheme UUID alone,
+6 -16
View File
@@ -35,7 +35,7 @@ from unshackle.core.drm import DRM_T, ClearKey, MonaLisa, PlayReady, Widevine
from unshackle.core.drm.segment_decrypt import SegmentDecrypter, can_use
from unshackle.core.drm.verify import decrypt_track
from unshackle.core.events import events
from unshackle.core.manifests.dash import RollingMerge
from unshackle.core.manifests.dash import RollingMerge, missing_language
from unshackle.core.session import RnetResponse, RnetSession, utf8_text
from unshackle.core.tracks import Audio, DownloadContext, Subtitle, Tracks, Video, resume
from unshackle.core.tracks.track import DRM_PREFERENCE_TYPES, assert_fragments_decrypted
@@ -305,12 +305,7 @@ class HLS:
None,
)
if not track_lang:
msg = "Language information could not be derived for a media."
if language is None:
msg += " No fallback language was provided when calling HLS.to_tracks()."
elif not tag_is_valid((str(language) or "").strip()) or str(language).startswith("und"):
msg += f" The fallback language provided is also invalid: {language}"
raise ValueError(msg)
raise missing_language("an HLS media", language)
tracks.add(
track_type(
@@ -664,8 +659,8 @@ class HLS:
hls_data = track.data.setdefault("hls", {})
if hls_data.get("unwanted_segments") and not unwanted_segments:
log.warning(
f"None of the {len(hls_data['unwanted_segments'])} segments that the export drops is in the "
f"playlist of {track.id}. The download keeps every segment."
f"None of the {len(hls_data['unwanted_segments'])} segments the export drops is in the "
f"playlist of {track.id}; the download keeps every segment"
)
hls_data["unwanted_segments"] = [s.absolute_uri for s in unwanted_segments]
@@ -1578,10 +1573,7 @@ class HLS:
**extra_params,
)
elif key.keyformat and key.keyformat.lower() in {f"urn:uuid:{PR_PSSH.SYSTEM_ID}", "com.microsoft.playready"}:
drm = PlayReady(
pssh=PR_PSSH(key.uri.split(",")[-1]),
pssh_b64=key.uri.split(",")[-1],
)
drm = PlayReady.from_pssh_b64(key.uri.split(",")[-1])
else:
raise NotImplementedError(f"The key system is not supported: {key}")
@@ -1637,9 +1629,7 @@ class HLS:
except NotImplementedError:
unsupported_keys.append(key)
except Widevine.Exceptions.KIDNotFound:
logging.getLogger("HLS").warning(
"Skipped an EXT-X-SESSION-KEY entry that carries no Key ID. The media playlist must supply the key."
)
logging.getLogger("HLS").warning("Skipped an EXT-X-SESSION-KEY entry with no Key ID")
if not drm_objects and unsupported_keys:
logging.debug(
+3 -9
View File
@@ -15,7 +15,6 @@ from typing import Any, BinaryIO, Optional, Union
import requests
from langcodes import Language, tag_is_valid
from lxml.etree import Element
from pyplayready.system.pssh import PSSH as PR_PSSH
from pywidevine.pssh import PSSH
from requests import Session
@@ -24,7 +23,7 @@ from unshackle.core.constants import DOWNLOAD_CANCELLED, DOWNLOAD_LICENCE_ONLY,
from unshackle.core.drm import DRM_T, PlayReady, Widevine
from unshackle.core.drm.verify import decrypt_track
from unshackle.core.events import events
from unshackle.core.manifests.dash import RollingMerge, append_segment
from unshackle.core.manifests.dash import RollingMerge, append_segment, missing_language
from unshackle.core.manifests.ism_init import (
build_init_segment,
parse_codec_private_data_vui,
@@ -104,11 +103,9 @@ class ISM:
drm.append(Widevine(pssh=pssh, kid=kid))
elif system_id == "9a04f079-9840-4286-ab92-e65be0885f95":
try:
pr_pssh = PR_PSSH(data)
drm.append(PlayReady.from_pssh_b64(data))
except Exception as e:
logging.getLogger("ISM").warning(f"Skipping unparseable PlayReady ProtectionHeader: {e!r}")
continue
drm.append(PlayReady(pssh=pr_pssh, pssh_b64=data))
return drm
@staticmethod
@@ -254,10 +251,7 @@ class ISM:
track_lang = Language.get(lang)
if not track_lang and not language:
# Language is optional in MS-SSTR; video streams commonly omit it.
raise ValueError(
"Language information could not be derived from the manifest and no fallback "
"language was provided when calling ISM.to_tracks()."
)
raise missing_language("an ISM stream", language)
track_urls: list[str] = []
track_times: list[int] = []
+15 -29
View File
@@ -191,10 +191,7 @@ class Gluetun(Proxy):
verify_ip: Automatically make sure that the IP and region are correct after connection (default: True)
"""
if not binaries.Docker:
raise RuntimeError(
"Docker is not available. Please install Docker to use Gluetun proxy.\n"
"Visit: https://docs.docker.com/engine/install/"
)
raise RuntimeError("Docker not found, the Gluetun proxy needs it")
self.providers = providers or {}
self.base_port = base_port
@@ -247,14 +244,14 @@ class Gluetun(Proxy):
if len(parts) == 1:
return None
if len(parts) != 2:
raise ValueError(f"Invalid query format: '{query}'. Expected 'provider:region' (e.g., 'windscribe:us')")
raise ValueError(f"Invalid query '{query}': expected provider:region, such as windscribe:us")
provider_name = parts[0].lower()
region = parts[1].lower()
if provider_name not in self.providers:
available = ", ".join(self.providers.keys())
raise ValueError(f"VPN provider '{provider_name}' not configured. Available providers: {available}")
raise ValueError(f"VPN provider '{provider_name}' not configured, available: {available}")
query_key = f"{provider_name}:{region}"
container_name = f"{self.container_prefix}-{provider_name}-{region}"
@@ -320,15 +317,9 @@ class Gluetun(Proxy):
if re.match(r"^[a-z]{2}$", region):
country = get_country_name(region)
if not country:
raise ValueError(
f"Country code '{region}' not recognized. "
f"Configure it in server_countries or use a valid ISO 3166-1 alpha-2 code."
)
raise ValueError(f"Country code '{region}' not recognized")
else:
raise ValueError(
f"Region '{region}' not recognized for provider '{provider_name}'. "
f"Configure it in server_countries or server_cities, or use a 2-letter country code."
)
raise ValueError(f"Region '{region}' not recognized for provider '{provider_name}'")
self.remove_stopped_container(container_name)
@@ -453,7 +444,7 @@ class Gluetun(Proxy):
credentials = config.get("credentials", {})
if vpn_type not in ["wireguard", "openvpn"]:
raise ValueError(f"Provider '{provider_name}': Invalid vpn_type '{vpn_type}'. Use 'wireguard' or 'openvpn'")
raise ValueError(f"Provider '{provider_name}': vpn_type '{vpn_type}' is not wireguard or openvpn")
if vpn_type == "wireguard":
# private_key is always required for WireGuard
@@ -468,16 +459,11 @@ class Gluetun(Proxy):
# Windscribe requires preshared_key (can be empty string, but must be set)
if provider_lower == "windscribe":
if "preshared_key" not in credentials:
raise ValueError(
f"Provider '{provider_name}': Windscribe WireGuard requires 'preshared_key' in credentials "
"(can be empty string, but must be set). Get it from windscribe.com/getconfig/wireguard"
)
if "addresses" not in credentials:
raise ValueError(
f"Provider '{provider_name}': Windscribe WireGuard requires 'addresses' in credentials. "
"Get it from windscribe.com/getconfig/wireguard"
)
for key in ("preshared_key", "addresses"):
if key not in credentials:
raise ValueError(
f"Provider '{provider_name}': Windscribe WireGuard needs '{key}' in credentials"
)
# Providers that require addresses (but not preshared_key)
elif provider_lower in ["surfshark", "mullvad", "ivpn"]:
@@ -539,8 +525,8 @@ class Gluetun(Proxy):
country_lower = country_code.lower()
if provider_name == "surfshark":
raise ValueError(
"Surfshark has no numbered servers. Map the region to a server name such as "
f"'{country_lower}-dal.prod.surfshark.com' in server_hostnames instead."
f"Surfshark has no numbered servers: map the region in server_hostnames, "
f"such as '{country_lower}-dal.prod.surfshark.com'"
)
hostname_formats = {
"nordvpn": f"{country_lower}{server_num}.nordvpn.com",
@@ -646,7 +632,7 @@ class Gluetun(Proxy):
message=f"Failed to pull Docker image: {gluetun_image}",
success=False,
)
raise RuntimeError(f"Failed to ensure Gluetun Docker image '{gluetun_image}' is available")
raise RuntimeError(f"Failed to pull the Gluetun proxy Docker image '{gluetun_image}'")
vpn_type = provider_config.get("vpn_type", "wireguard").lower()
credentials = provider_config.get("credentials", {})
@@ -1234,7 +1220,7 @@ class Gluetun(Proxy):
)
raise RuntimeError(
f"Failed to verify VPN IP for {container['provider']}:{container['region']} "
f"after {max_retries} attempts. Last error: {last_error}"
f"after {max_retries} attempts: {last_error}"
)
def remove_stopped_container(self, container_name: str) -> bool:
+1 -1
View File
@@ -37,7 +37,7 @@ class Hola(Proxy):
return None
ban = "Transaction error: temporary ban detected."
banned = ConnectionError("Hola banned your IP temporarily from it's services. Try change your IP.")
banned = ConnectionError("Hola temporarily banned your IP")
try:
# hola-proxy logs to stderr from another thread, so a merged stream splits the Login/Password lines.
result = subprocess.run(
+4 -10
View File
@@ -19,14 +19,11 @@ class NordVPN(Proxy):
The Service Credentials can be found here: https://my.nordaccount.com/dashboard/nordvpn/
"""
if not username:
raise ValueError("No Username was provided to the NordVPN Proxy Service.")
raise ValueError("No username for the NordVPN proxy provider")
if not password:
raise ValueError("No Password was provided to the NordVPN Proxy Service.")
raise ValueError("No password for the NordVPN proxy provider")
if not re.match(r"^[a-z0-9]{48}$", username + password, re.IGNORECASE) or "@" in username:
raise ValueError(
"The Username and Password must be NordVPN Service Credentials, not your Login Credentials. "
"The Service Credentials can be found here: https://my.nordaccount.com/dashboard/nordvpn/"
)
raise ValueError("NordVPN needs its Service Credentials, not your login credentials")
if server_map is not None and not isinstance(server_map, dict):
raise TypeError(f"Expected server_map to be a dict mapping a region to a server ID, not '{server_map!r}'.")
@@ -101,10 +98,7 @@ class NordVPN(Proxy):
None,
)
if city_id is None:
raise ValueError(
f"No servers found in city '{city}' for country '{country['name']}'. "
"Try a different city or check the city name spelling."
)
raise ValueError(f"No servers found in city '{city}' for country '{country['name']}'")
if selector and is_ip(selector):
recommended_servers = self.get_all_servers(country["id"])
else:
+1 -2
View File
@@ -226,8 +226,7 @@ class ProtonVPN(Proxy):
self.load_session()
if not self.access_token:
log.error(
"Proton: no session available; export account.proton.me cookies to %s "
"or set proxy_providers.protonvpn.enable: true for the one-time TV login",
"Proton: no access token, export account.proton.me cookies to %s or set proxy_providers.protonvpn.enable",
self.cookie_path,
)
return None
+11 -4
View File
@@ -54,7 +54,7 @@ def find_provider(proxy_providers: List[Any], name: str, hint: str = "") -> Any:
def load_proxy_providers(exclude: tuple[str, ...] = (), raise_errors: bool = False, quiet: bool = False) -> List[Any]:
"""
Build the proxy providers set in config, in the order a bare query tries them.
Build the proxy providers set in config: Basic first, then config key order, Control D last.
Each proxy provider loads on its own. One that fails is logged and kept as Unavailable, so a bare
query skips it and a query that names it reports why. *raise_errors* raises the failure instead.
@@ -75,8 +75,15 @@ def load_proxy_providers(exclude: tuple[str, ...] = (), raise_errors: bool = Fal
)
settings = config.proxy_providers
# Control D is last: a bare region reaches it only when nothing else answers, since it changes the account.
classes = (Basic, ExpressVPN, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN, Gluetun, Hola, ControlD)
rank = {key: i for i, key in enumerate(settings)}
classes = [
Basic,
*sorted(
(ExpressVPN, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN, Gluetun, Hola),
key=lambda c: rank.get(c.__name__.lower(), len(rank)),
),
ControlD,
]
def build(cls: type) -> Any:
key = cls.__name__.lower()
@@ -120,7 +127,7 @@ def initialize_proxy_providers(raise_errors: bool = False, quiet: bool = False)
"""
proxy_providers = load_proxy_providers(("gluetun",), raise_errors, quiet)
if not quiet and not proxy_providers:
log.warning("No proxy providers were loaded. Check your proxy provider configuration in unshackle.yaml")
log.warning("No proxy providers were loaded")
return proxy_providers
+5 -11
View File
@@ -91,14 +91,11 @@ class SurfsharkVPN(Proxy):
The Service Credentials can be found here: https://my.surfshark.com/vpn/manual-setup/main/openvpn
"""
if not username:
raise ValueError("No Username was provided to the SurfsharkVPN Proxy Service.")
raise ValueError("No username for the SurfsharkVPN proxy provider")
if not password:
raise ValueError("No Password was provided to the SurfsharkVPN Proxy Service.")
raise ValueError("No password for the SurfsharkVPN proxy provider")
if not re.match(r"^[a-z0-9]{48}$", username + password, re.IGNORECASE) or "@" in username:
raise ValueError(
"The Username and Password must be SurfsharkVPN Service Credentials, not your Login Credentials. "
"The Service Credentials can be found here: https://my.surfshark.com/vpn/manual-setup/main/openvpn"
)
raise ValueError("SurfsharkVPN needs its Service Credentials, not your login credentials")
if server_map is not None and not isinstance(server_map, dict):
raise TypeError(
@@ -113,8 +110,7 @@ class SurfsharkVPN(Proxy):
if not server or server.isdigit():
# A warning, not an error: one stale entry must not disable the whole proxy provider.
log.warning(
"SurfsharkVPN: ignoring server_map entry '%s: %s'. Use a server name such as 'us-dal'. "
"SurfsharkVPN no longer uses numeric server IDs.",
"SurfsharkVPN: ignoring server_map entry '%s: %s', use a server name such as 'us-dal'",
region,
server,
)
@@ -240,9 +236,7 @@ class SurfsharkVPN(Proxy):
x for location, x in located if location.startswith(wanted)
]
if not servers:
raise ValueError(
f"No servers found in city '{city}'. Try a different city or check the city name spelling."
)
raise ValueError(f"No servers found in city '{city}'")
return servers
+3 -6
View File
@@ -21,9 +21,9 @@ class WindscribeVPN(Proxy):
code with a city ("ca:toronto"), and unshackle uses a pinned hostname before any server lookup.
"""
if not username:
raise ValueError("No Username was provided to the WindscribeVPN Proxy Service.")
raise ValueError("No username for the WindscribeVPN proxy provider")
if not password:
raise ValueError("No Password was provided to the WindscribeVPN Proxy Service.")
raise ValueError("No password for the WindscribeVPN proxy provider")
if server_map is not None and not isinstance(server_map, dict):
raise TypeError(f"Expected server_map to be a dict mapping a region to a hostname, not '{server_map!r}'.")
@@ -158,10 +158,7 @@ class WindscribeVPN(Proxy):
if hostnames:
return random.choice(hostnames)
elif city:
raise ValueError(
f"No servers found in city '{city}' for country code '{country_code}'. "
"Try a different city or check the city name spelling."
)
raise ValueError(f"No servers found in city '{city}' for country code '{country_code}'")
return None
+44 -42
View File
@@ -156,9 +156,7 @@ class RemoteClient:
if isinstance(e.args[0] if e.args else None, urllib3.exceptions.ReadTimeoutError):
raise self.timed_out(endpoint) from None
server_url = safe_display_url(self.server_url)
raise RemoteError(
f"Could not connect to remote server at {server_url}. Is it running? (unshackle serve)", "CONNECT"
) from None
raise RemoteError(f"Could not connect to remote server at {server_url}", "CONNECT") from None
except requests.Timeout:
raise self.timed_out(endpoint) from None
# servers differ on which header carries the key, so retry the rest before giving up
@@ -196,8 +194,7 @@ class RemoteClient:
return {}
if self.tunnel and self.tunnel.lost:
raise RemoteError(
"The tunnel to the server closed, so the server can no longer send its requests through this "
"machine. Run the command again.",
"The tunnel to the server closed",
"TUNNEL_CLOSED",
)
raise RemoteError(f"Server error [{error_code}]: {error_msg}", error_code)
@@ -205,8 +202,7 @@ class RemoteClient:
def timed_out(self, endpoint: str) -> RemoteError:
return RemoteError(
f"Request to remote server timed out after {self.timeout}s: {endpoint}. "
"Raise 'timeout' for this server in remote_services.",
f"Request to remote server timed out after {self.timeout}s: {endpoint}",
"TIMEOUT",
)
@@ -426,6 +422,27 @@ def apply_service_track_data(tracks: Tracks, data: Dict[str, Any]) -> None:
track.data.update(extra)
def apply_session_cookies(session: requests.Session, result: dict) -> None:
"""Copy the server's session cookies into ``session``, each scoped to its own domain.
An older server sends only ``session_cookies`` names and values, so those cookies go to every host.
It skips malformed entries.
"""
jar = result.get("session_cookie_jar")
if not isinstance(jar, list):
old = result.get("session_cookies")
jar = [{"name": k, "value": v} for k, v in old.items()] if isinstance(old, dict) else []
for c in jar:
if isinstance(c, dict) and isinstance(c.get("name"), str) and isinstance(c.get("value"), str):
session.cookies.set(
c["name"],
c["value"],
domain=str(c.get("domain") or ""),
path=str(c.get("path") or "/"),
secure=bool(c.get("secure")),
)
def resolve_track_manifests(tracks: Tracks, track_manifests: list) -> None:
"""Fill track.data from the per-track MPD fragments the server sent.
@@ -706,8 +723,7 @@ def build_title(info: Dict[str, Any], service_tag: str, fallback_id: str) -> Tit
)
else:
raise click.ClickException(
f"The remote server sent a title of type {title_type!r}, which this client does not support. "
"Update unshackle on both ends so they agree on the title types."
f"The remote server sent an unknown title type {title_type!r}; unshackle versions differ"
)
apply_title_fields(title, info)
return title
@@ -723,10 +739,7 @@ def resolve_auth_headers(svc: dict, server_name: str) -> list[str]:
or not auth_headers
or not all(isinstance(h, str) and h.strip() for h in auth_headers)
):
raise click.ClickException(
f"Remote service '{server_name}': 'auth_headers' must be a list of header names, e.g.\n\n"
' auth_headers: ["Authorization"]'
)
raise click.ClickException(f"Remote service '{server_name}': 'auth_headers' must be a list of header names")
resolved = [h.strip() for h in auth_headers]
seen = {h.lower() for h in resolved}
resolved.extend(h for h in DEFAULT_AUTH_HEADERS if h.lower() not in seen)
@@ -743,19 +756,13 @@ def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]:
"""
remote_services = config.remote_services
if not remote_services:
raise click.ClickException(
"No remote services configured. Add 'remote_services' to your unshackle.yaml:\n\n"
" remote_services:\n"
" my_server:\n"
' url: "https://server:8080"\n'
' api_key: "your-api-key"'
)
raise click.ClickException("No 'remote_services' in your unshackle.yaml")
if server_name:
svc = remote_services.get(server_name)
if not svc:
available = ", ".join(remote_services.keys())
raise click.ClickException(f"Remote service '{server_name}' not found. Available: {available}")
raise click.ClickException(f"Remote service '{server_name}' not found, available: {available}")
services = svc.get("services", {})
services["_server_cdm"] = svc.get("server_cdm")
services["_auth_headers"] = resolve_auth_headers(svc, server_name)
@@ -774,7 +781,7 @@ def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]:
return svc["url"], svc.get("api_key", ""), services
available = ", ".join(remote_services.keys())
raise click.ClickException(f"Multiple remote services configured. Use --server to select one: {available}")
raise click.ClickException(f"Multiple remote services configured, pick one with --server: {available}")
REMOTE_LIST_TTL = 300.0
@@ -937,7 +944,7 @@ def start_tunnel(client: RemoteClient) -> TunnelClient:
try:
tunnel.start()
except ConnectionError as e:
raise click.ClickException(f"--local-proxy needs a tunnel to the server. {e}")
raise click.ClickException(f"--local-proxy could not open a tunnel to the server: {e}")
client.tunnel = tunnel
return tunnel
@@ -1024,9 +1031,11 @@ class ServerVault(Vault):
``decrypt_verified`` needs a Vault to hold in ``Vaults.sources`` so a server-supplied
content key gets the same decode check as a local vault key. The client never learns
which server vault answered and can neither read nor write it, so every method returns
nothing and a bad verdict goes to the server, which flags the row itself.
nothing. A bad verdict goes to the server at once, so its next licence skips the pair.
"""
report_now = True
def __init__(self, service: "RemoteService"):
super().__init__("server_cdm", no_push=True)
self.service = service
@@ -1046,8 +1055,8 @@ class ServerVault(Vault):
def get_services(self) -> Iterator[str]:
return iter(())
def report_bad(self, kid: UUID, key: str) -> None:
self.service.report_bad_key(kid, key)
def flag_bad_key(self, service: str, kid: Union[UUID, str], key: str, source: str) -> None:
self.service.report_bad_key(UUID(str(kid)), key)
class RemoteService:
@@ -1217,10 +1226,7 @@ class RemoteService:
def require_reachable_proxy(self, proxy: Optional[str]) -> None:
"""Refuse a proxy on this machine when the server is elsewhere: the server cannot reach it."""
if proxy and is_loopback(proxy) and not is_loopback(self.client.server_url):
raise click.ClickException(
"That proxy is on your machine, so --remote cannot use it. "
"To send the server's requests through this machine's connection, use --local-proxy without --proxy."
)
raise click.ClickException("--remote cannot use a proxy on this machine; use --local-proxy without --proxy")
def tunnel(self) -> str:
"""The ``tunnel:<id>`` proxy to send the server for ``--local-proxy``, opening the tunnel the first time."""
@@ -1274,9 +1280,8 @@ class RemoteService:
self.log.info(f"Using a '{regions[0]}' proxy to match a server account ({', '.join(regions)})")
except click.ClickException:
self.log.warning(
f"The server's accounts work in {', '.join(regions)} and you are in {client_region}. "
"There is no local proxy for that region. Pass --proxy yourself if the server "
"rejects the remote session"
f"No local proxy for the server's account region ({', '.join(regions)}), "
f"you are in {client_region}"
)
elif not no_proxy and proxy:
resolved_proxy = resolve_remote_proxy_arg(server_proxy_arg(self.ctx.parent.params), providers)
@@ -1651,8 +1656,7 @@ class RemoteService:
for k, v in result.get("session_headers", {}).items():
if k.lower() not in ("host", "content-length", "content-type"):
self._session.headers[k] = v
for k, v in result.get("session_cookies", {}).items():
self._session.cookies.set(k, v)
apply_session_cookies(self._session, result)
resolve_track_manifests(tracks, result.get("track_manifests", []))
resolve_manifest_data(tracks, result.get("manifests", []))
@@ -1808,25 +1812,23 @@ class RemoteService:
else "no vault, neither the server's nor yours"
)
raise click.ClickException(
f"The key for {track} is in {vaults}. This session runs on the server's device, which "
"licenses nothing, so drop the service's server-identity option to license it with your own device."
f"The content key for {track} is in {vaults}; drop the service's server-identity option to license it locally"
)
cap = refusal.get("max_height")
if cap:
why = f"The server CDM licenses up to {cap}p for this key"
why = f"The server CDM licenses up to {cap}p for this API key"
elif getattr(self, "_server_vault", False):
why = "The server vault holds no key for this track"
why = "The server vault holds no content key for this track"
else:
why = "The server CDM refused this track"
cdm = self.local_cdm
if cdm is None:
raise click.ClickException(f"{why}, and {track} needs a local CDM to license it with your own device.")
raise click.ClickException(f"{why}, and {track} needs a local CDM to license it")
drm_class = "PlayReady" if is_playready_cdm(cdm) else "Widevine"
if track.drm and not any(d.__class__.__name__ == drm_class for d in track.drm):
height = getattr(track, "height", None)
raise click.ClickException(
f"{why}, and {track} carries no {drm_class} DRM for your local device to license. "
f"Pass -q {height or 'with that height'} so the session starts on your own device."
f"{why}, and {track} has no {drm_class} DRM for your local CDM; pass -q {height or 'with that height'}"
)
track.drm_preference = drm_class.lower()
self.client_licensed.add(str(track.id))
+18 -49
View File
@@ -25,7 +25,7 @@ from unshackle.core.constants import AnyTrack
from unshackle.core.credential import Credential
from unshackle.core.drm import DRM_T
from unshackle.core.proxies.basic import Basic
from unshackle.core.proxies.resolve import find_provider
from unshackle.core.proxies.resolve import pick_proxy
from unshackle.core.search_result import SearchResult
from unshackle.core.session import (
BACKOFF_FACTOR,
@@ -229,40 +229,16 @@ class Service(metaclass=ABCMeta):
self.log.info(
f"Found service-specific proxy mapping: {full_proxy_key} -> {sanitize_proxy_for_log(mapped_value)}"
)
if proxy_provider_name:
# Specific provider requested
try:
proxy_provider = find_provider(ctx.obj.proxy_providers, proxy_provider_name)
except ValueError:
proxy_provider = None
if proxy_provider:
mapped_proxy_uri = proxy_provider.get_proxy(mapped_value)
if mapped_proxy_uri:
proxy = mapped_proxy_uri
self.log.info(
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
)
else:
self.log.warning(
f"Failed to get proxy for mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
)
else:
self.log.warning(f"Proxy provider '{proxy_provider_name}' not found, using default proxy")
else:
for proxy_provider in ctx.obj.proxy_providers:
mapped_proxy_uri = proxy_provider.get_proxy(mapped_value)
if mapped_proxy_uri:
proxy = mapped_proxy_uri
self.log.info(
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
)
break
else:
self.log.warning(
f"No provider could resolve mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
)
try:
proxy_provider, proxy = pick_proxy(ctx.obj.proxy_providers, mapped_value, proxy_provider_name)
self.log.info(
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
)
except ValueError:
self.log.warning(
f"No provider could resolve mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
)
if not proxy:
# don't override the explicit proxy set by the user, even if they may be geoblocked
@@ -276,24 +252,18 @@ class Service(metaclass=ABCMeta):
self.log.info("Service is not Geoblocked in your region")
elif not (fence_targets := [x for x in self.GEOFENCE if self.is_region_allowed(x)]):
raise click.ClickException(
f"Service is not available in {whose} region ({current_region.upper()}). "
"Pass --proxy with a proxy outside the blocked regions."
f"Service is not available in {whose} region ({current_region.upper()})"
)
else:
requested_proxy = fence_targets[0] # first is likely main region
self.log.info(
f"Service is Geoblocked in your region, getting a Proxy to {requested_proxy}"
)
for proxy_provider in ctx.obj.proxy_providers:
proxy = proxy_provider.get_proxy(requested_proxy)
if proxy:
self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
break
if not proxy:
self.log.warning(
f"No proxy available for {requested_proxy}. "
f"Pass --proxy with a proxy in {requested_proxy}, or the request can fail."
)
try:
proxy_provider, proxy = pick_proxy(ctx.obj.proxy_providers, requested_proxy)
self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
except ValueError:
self.log.warning(f"No proxy available for {requested_proxy}")
except click.ClickException:
raise
except Exception as e:
@@ -316,8 +286,7 @@ class Service(metaclass=ABCMeta):
# Only GEOBLOCK applies here: an explicit --proxy outside GEOFENCE stays the user's call.
if exit_region and any(x.lower() == exit_region.lower() for x in self.GEOBLOCK):
raise click.ClickException(
f"Service is not available in {whose} proxy's region ({exit_region.upper()}). "
"Pass --proxy with a proxy outside the blocked regions."
f"Service is not available in {whose} proxy's region ({exit_region.upper()})"
)
else:
# No proxy, use cached IP info for title caching (non-critical)
+3 -7
View File
@@ -332,10 +332,7 @@ def check_load_errors() -> None:
"""
if DIRTY_REPOS:
joined = "\n".join(f" - {p}" for p in DIRTY_REPOS)
raise click.ClickException(
"Service repo has local changes - refusing to refresh so your edits are not lost.\n"
"Commit and push them to the upstream repo (or revert them), then retry:\n" + joined
)
raise click.ClickException("Service repo has local changes, not refreshing:\n" + joined)
global SUMMARY_LOGGED
if not SUMMARY_LOGGED:
SUMMARY_LOGGED = True
@@ -422,7 +419,7 @@ class Services(click.Group):
raise click.ClickException(
f"There are no Services added yet, therefore the '{name}' Service could not be found."
)
raise click.ClickException(f"{e}. Available Services: {', '.join(available_services)}")
raise click.ClickException(f"{e}, available: {', '.join(available_services)}")
if hasattr(service, "cli"):
cli = service.cli
@@ -518,8 +515,7 @@ class Services(click.Group):
if services:
available = ", ".join(sorted(svc["tag"] for svc in services))
raise click.ClickException(
f"The remote server does not offer a service named '{name}'. "
f"It offers these services: {available}"
f"The remote server does not offer a service named '{name}', available: {available}"
)
raise click.ClickException("The remote server offers no services to your API key.")
tag = svc_info["tag"]
+67 -13
View File
@@ -6,6 +6,7 @@ import http
import logging
import math
import random
import re
import sys
import time
from collections.abc import Iterator, MutableMapping
@@ -17,6 +18,7 @@ from urllib.parse import urlencode, urlparse, urlunparse
import rnet
from requests import HTTPError, Request, Response
from requests.cookies import create_cookie
from requests.structures import CaseInsensitiveDict
from unshackle.core.config import config
@@ -50,11 +52,7 @@ def resolve_impersonate(browser: str) -> rnet.Impersonate:
preset = getattr(rnet.Impersonate, browser, None)
if preset is not None:
return preset
raise ValueError(
f"Unknown impersonate preset: {browser!r}. "
f"Use exact rnet preset names like 'Chrome131', 'OkHttp4_12', 'Edge101'. "
f"See rnet.Impersonate for all available presets."
)
raise ValueError(f"Unknown impersonate preset {browser!r}, expected an rnet.Impersonate name such as 'Chrome131'")
METHOD_MAP: dict[str, rnet.Method] = {
@@ -530,6 +528,34 @@ class MaxRetriesError(Exception):
self.__cause__ = cause
# the innermost io error text in rnet's wreq::Error debug form, e.g. "stream closed because of a broken pipe"
RNET_ERROR_TEXT_RE = re.compile(r'error: "([^"]+)"')
def failure_response(exc: BaseException) -> Any:
"""The HTTP response of a status failure, or None.
RnetSession wraps a forcelist HTTPError in MaxRetriesError as ``__cause__``, so this unwraps one level.
"""
response = getattr(exc, "response", None)
return response if response is not None else getattr(exc.__cause__, "response", None)
def failure_reason(exc: BaseException) -> str:
"""Name a request failure without its URL, which can carry signed tokens.
Gives the HTTP status of a status failure, else the exception class and, for an rnet error,
its innermost error text.
"""
inner = exc.__cause__ or exc
response = failure_response(exc)
status = getattr(response, "status_code", None)
if isinstance(status, int):
return f"{status} {getattr(response, 'reason', '') or ''}".strip()
texts = RNET_ERROR_TEXT_RE.findall(str(inner))
return f"{type(inner).__name__} ({texts[-1]})" if texts else type(inner).__name__
class RnetSession:
"""TLS-fingerprinted HTTP session powered by rnet (Rust/BoringSSL).
@@ -664,14 +690,19 @@ class RnetSession:
self._client = client
@staticmethod
def set_cookie_header(client: rnet.BlockingClient, origin: str, header: str) -> None:
"""Set every cookie in a ``name=value; ...`` header string on one origin of *client*."""
def parse_cookie_header(header: str) -> Iterator[tuple[str, str]]:
"""The ``(name, value)`` pairs of a ``name=value; ...`` header string."""
for pair in header.split(";"):
name, separator, value = pair.partition("=")
if not separator:
continue
if separator:
yield name.strip(), value.strip()
@staticmethod
def set_cookie_header(client: rnet.BlockingClient, origin: str, header: str) -> None:
"""Set every cookie in a ``name=value; ...`` header string on one origin of *client*."""
for name, value in RnetSession.parse_cookie_header(header):
try:
client.set_cookie(origin, rnet.Cookie(name.strip(), value.strip()))
client.set_cookie(origin, rnet.Cookie(name, value))
except Exception:
pass
@@ -705,6 +736,26 @@ class RnetSession:
exported[origin] = raw.decode("utf-8", errors="replace")
return exported
def scoped_cookies(self) -> CookieJar:
"""Every cookie this HTTP session sends, each scoped to its domain.
The live client's cookies of each requested origin come first, then the cookies set
through :attr:`cookies` for any other host.
"""
jar = CookieJar()
hosts: dict[str, set[str]] = {}
for origin, header in self.export_origin_cookies().items():
host = urlparse(origin).hostname or ""
for name, value in self.parse_cookie_header(header):
jar.set_cookie(create_cookie(name, value, domain=host))
hosts.setdefault(name, set()).add(host)
for domain, values in self.cookies.get_dict_by_domain().items():
bare = domain.lstrip(".")
for name, value in values.items():
if not any(h == bare or h.endswith("." + bare) for h in hosts.get(name, ())):
jar.set_cookie(create_cookie(name, value, domain=domain))
return jar
def import_origin_cookies(self, exported: dict[str, str]) -> None:
"""Load cookie headers from :meth:`export_origin_cookies` into this HTTP session's client."""
if not exported:
@@ -842,12 +893,12 @@ class RnetSession:
if response.status_code not in self.status_forcelist:
return response
last_exception = HTTPError(f"Received status code: {response.status_code}", response=response)
failure = f"{response.status_code} {response.reason}"
failure = failure_reason(last_exception)
except self.catch_exceptions as e:
last_exception = e
response = None
failure = e.__class__.__name__
failure = failure_reason(e)
if attempt < max_retries:
self.log.warning(f"{failure}({urlparse(url).path}). Retrying... ({attempt + 1}/{max_retries})")
@@ -857,7 +908,10 @@ class RnetSession:
else:
break
raise MaxRetriesError(f"Max retries exceeded for {method} {url}", cause=last_exception)
attempts = f" after {max_retries + 1} attempts" if max_retries else ""
raise MaxRetriesError(
f"{method_upper} {parsed_origin.hostname} failed{attempts}: {failure}", cause=last_exception
)
def get(self, url: str, **kwargs: Any) -> RnetResponse:
return self.request("GET", url, **kwargs)
+3 -9
View File
@@ -386,10 +386,7 @@ class Hybrid:
message="Video has no container duration; RPU keeps the active area of the DV track",
context={"input": str(input_video)},
)
self.log.warning(
"Video has no container duration, skipping crop detection. "
"The RPU keeps the active area of the DV track."
)
self.log.warning("Video has no container duration, skipping crop detection")
return None
duration = float(duration_value)
@@ -665,7 +662,7 @@ class Hybrid:
return
if not HDR10PlusTool:
raise ValueError("HDR10Plus_tool not found. Please install it to use HDR10+ to DV conversion.")
raise ValueError("HDR10Plus_tool not found, HDR10+ to DV conversion needs it")
try:
run_step(
@@ -817,10 +814,7 @@ class Hybrid:
json.dump(generator_config, f)
if profile == "A":
self.log.warning(
"The HDR10+ metadata is Profile A, which is often less reliable than Profile B. "
"Check the Dolby Vision result before you keep it."
)
self.log.warning("The HDR10+ metadata is Profile A, less reliable than Profile B")
try:
dovi.generate(
+1 -2
View File
@@ -1128,8 +1128,7 @@ class Subtitle(Track):
except (IOError, OSError) as e:
if "is not valid subtitle file" in str(e):
log = logging.getLogger("Subtitle")
log.warning(f"Failed to strip SDH from {self.path.name}: {e}")
log.warning("Continuing without SDH stripping for this subtitle")
log.warning(f"Failed to strip SDH from {self.path.name}, keeping it as is: {e}")
else:
raise
+11 -39
View File
@@ -30,7 +30,7 @@ from unshackle.core.config import config
from unshackle.core.constants import DOWNLOAD_ALL_DRM, DOWNLOAD_CANCELLED, DOWNLOAD_LICENCE_ONLY, DownloadCancelled
from unshackle.core.downloaders import requests
from unshackle.core.drm import DRM_T, ClearKeyCENC, PlayReady, Widevine
from unshackle.core.drm.verify import decrypt_track
from unshackle.core.drm.verify import decrypt_track, has_encrypted_entry, senc_protects_samples
from unshackle.core.events import events
from unshackle.core.session import MAX_BACKOFF, RnetSession, utf8_text
from unshackle.core.temp import restore_segments, stash_segments
@@ -244,42 +244,18 @@ def has_dts_uhd_sample_entry(path: Path) -> bool:
def has_encrypted_sample_entry(path: Path) -> bool:
"""True if the MP4's moov still carries an encrypted sample entry (encv/enca).
"""True if the MP4's moov still carries an encrypted sample entry (``enc*`` or ``drm*``).
A faithful decrypt rewrites encv/enca back to the real codec 4CC through frma, so a
survivor means the decrypt tool skipped/failed the track. This is a 4CC scan
scoped to the moov box (where sample entries live), not a structural parse.
The scoping avoids chance byte collisions in mdat. Any read error -> False.
A faithful decrypt rewrites the entry back to the real codec 4CC through frma, so a
survivor means the decrypt tool skipped/failed the track. Only the moov is read, and its
stsd boxes are parsed, so byte runs in mdat or in metadata cannot match. Any read or
parse error -> False.
"""
try:
moov = read_top_level_box(path, b"moov")
except OSError:
return moov is not None and has_encrypted_entry(moov)
except (OSError, struct.error, RecursionError):
return False
if not moov:
return False
for tok in (b"encv", b"enca"):
i = moov.find(tok, 4)
while i != -1:
# require a plausible sample-entry box size right before the 4CC
size = int.from_bytes(moov[i - 4 : i], "big")
if 16 <= size <= len(moov) - (i - 4):
return True
i = moov.find(tok, i + 1)
return False
def senc_protects_samples(buf: bytes, body: int, end: int) -> bool:
"""True if a senc/PIFF-uuid box describes protected samples.
An empty senc (sample_count 0) sits on a genuinely clear fragment, so treating it as
a skipped one would raise on a healthy file. Field order follows
read_per_sample_iv_size.
"""
flags = int.from_bytes(buf[body + 1 : body + 4], "big")
pos = body + 4 + (20 if flags & 0x1 else 0) # PIFF override: AlgorithmID(3)+IV_size(1)+KID(16)
if pos + 4 > end:
return False
return int.from_bytes(buf[pos : pos + 4], "big") > 0
def moof_still_encrypted(moof: bytes) -> bool:
@@ -370,9 +346,7 @@ def assert_fragments_decrypted(path: Path) -> None:
first_offset=first_offset,
)
raise ValueError(
f"{path.name}: {surviving}/{total} fragment(s) still encrypted after decryption (first at "
f"byte {first_offset}). The decrypt tool skipped them silently, so check "
f"tfhd.sample_description_index against the stsd entry count in the init segment."
f"{path.name}: {surviving}/{total} fragment(s) still encrypted after decryption, first at byte {first_offset}"
)
@@ -553,9 +527,7 @@ class Track:
return slice(None)
low, high = (starts[0] + seconds * timescale for seconds in window)
if low > starts[-1] and len(starts) > 1:
raise click.ClickException(
"The title ends before the sample window starts. Use --sample START-END with an earlier window."
)
raise click.ClickException("The title ends before the sample window starts, use an earlier --sample window")
first = max(bisect_right(starts, low) - 1, 0)
self.sample_start = (starts[first] - starts[0]) / timescale
return slice(first, max(bisect_left(starts, high) + self.sample_tail, first + 1))
@@ -1174,7 +1146,7 @@ class Track:
drm_list.append(drm)
elif fmt in {f"urn:uuid:{PR_PSSH.SYSTEM_ID}", "com.microsoft.playready"}:
pssh_b64 = key.uri.split(",")[-1]
drm = PlayReady(pssh=PR_PSSH(pssh_b64), pssh_b64=pssh_b64)
drm = PlayReady.from_pssh_b64(pssh_b64)
drm_list.append(drm)
if drm_list:
+2 -13
View File
@@ -237,14 +237,7 @@ class Tracks:
for track in flatten(tracks):
if self.exists(by_id=track.id):
if not warn_only:
raise ValueError(
"One or more of the provided Tracks is a duplicate. "
"Track IDs must be unique but accurate using static values. The "
"value should stay the same no matter when you request the same "
"content. Use a value that has relation to the track content "
"itself and is static or permanent and not random/RNG data that "
"wont change each refresh or conflict in edge cases."
)
raise ValueError(f"Duplicate track ID {track.id}: track IDs must be unique and stable")
duplicates += 1
continue
@@ -573,11 +566,7 @@ class Tracks:
"""
mp4box = getattr(binaries, "MP4Box", None)
if not mp4box:
raise RuntimeError(
"MP4Box (GPAC) is required to mux DTS-UHD (DTS:X Profile 2) audio but was not found. "
"Install it with your package manager (e.g. `apt install gpac`), then check it with "
"`unshackle env check`."
)
raise RuntimeError("MP4Box (GPAC) not found, muxing DTS-UHD (DTS:X Profile 2) audio needs it")
progress = progress or partial(lambda **kwargs: None)
+2 -2
View File
@@ -495,7 +495,7 @@ class SlowDelayRange(click.ParamType):
match = re.match(r"^(\d+)-(\d+)$", str(value))
if not match:
self.fail(f"'{value}' is not a valid range. Use format: MIN-MAX (e.g., 20-40)", param, ctx)
self.fail(f"'{value}' is not a valid range, expected MIN-MAX such as 20-40", param, ctx)
low, high = int(match.group(1)), int(match.group(2))
if low < 5:
@@ -521,7 +521,7 @@ class SampleRange(click.ParamType):
match = re.match(r"^(\d+)-(\d+)$", str(value))
if not match or int(match.group(1)) >= int(match.group(2)):
self.fail(f"'{value}' is not a valid sample window. Use format: START-END (e.g., 60-120)", param, ctx)
self.fail(f"'{value}' is not a valid sample window, expected START-END such as 60-120", param, ctx)
return (int(match.group(1)), int(match.group(2)))
+1 -4
View File
@@ -288,10 +288,7 @@ def verify_proxy_exit(session: requests.Session) -> dict:
if len(errors) < len(build_providers()):
log.warning("No geolocation API returned the proxy exit IP, so the exit IP check does not occur")
return {}
raise ConnectionError(
f"Proxy check failed: no IP lookup got through the proxy ({type(errors[-1]).__name__}). "
"The proxy is down or refused the connection."
)
raise ConnectionError(f"Proxy check failed: no IP lookup got through the proxy ({type(errors[-1]).__name__})")
exit_ip = info.get("ip")
server_ip = (get_ip_info(cached=True) or {}).get("ip")
# The cached IP can be up to 24h old: a machine that was on a VPN when it cached its IP,
+1 -2
View File
@@ -229,8 +229,7 @@ def dispatch(event: str, mode: str, context: dict[str, str], overrides: Sequence
injected = _data_injected_flag(tokens, argv)
if injected is not None:
log.warning(
"Skipping post-script: a substituted value expands to the option-like token %r; "
"prefix the variable (--opt={var}) so service metadata cannot forge a flag",
"Skipping post-script: a substituted value expands to the option-like token %r",
redact_text(redact_path(injected)),
)
continue
+1
View File
@@ -131,6 +131,7 @@ def ffmpeg_decodes(path: Path, start: Optional[float] = None, seconds: float = 3
return True
if not keyframes and video is not False and any(m in err for m in FFMPEG_MISSING_REFERENCE):
return decode(window, keyframes=True)
log_tool_run("FFmpeg decode check", ffmpeg, ff.returncode, window=window, stderr=ff.stderr[-2000:])
return False
tail = ["-sseof", f"-{FFMPEG_CHECK_TAIL}", "-t", str(FFMPEG_CHECK_TAIL)]
+1
View File
@@ -51,6 +51,7 @@ def vault_session(timeout: float) -> Session:
class Vault(metaclass=ABCMeta):
# a vault on this machine; Vaults.get_key tries these first, before the network ones
local: bool = False
report_now: bool = False
def __init__(self, name: str, no_push: bool = False):
self.name = name
+34 -13
View File
@@ -30,6 +30,7 @@ class Vaults:
self.candidates: dict[Union[UUID, str], list[tuple[str, Vault]]] = {}
self.warned: set[tuple[str, Union[UUID, str], str]] = set()
self.unreachable: set[Vault] = set()
self.unreported: dict[tuple[Union[UUID, str], str], Vault] = {}
def __iter__(self) -> Iterator[Vault]:
return iter(self.vaults)
@@ -153,22 +154,42 @@ class Vaults:
return key if found else None
def flag_bad_key(self, kid: Union[UUID, str], key: str) -> None:
"""Flag a KID:KEY that failed decryption, naming the Vault it came from.
Every local vault records it. A remote vault is told only when it served the pair,
so it stops serving it too, and a vault that never held it is not bothered. A vault
that cannot take the flag is skipped with a debug line: the local flag already
protects this run, and a vault without the route is not a fault.
"""
"""Flag a KID:KEY in every local vault. A failed decode is no proof, so the remote vault that
served it hears of it from ``report_replaced``, or at once when it sets ``report_now``."""
self.flagged.add((kid, key))
source = self.sources.pop(kid, None)
name = source[1].name if source else "unknown"
for vault in self.live():
if not vault.local and (source is None or vault is not source[1]):
continue
try:
vault.flag_bad_key(self.service, kid, key, source[1].name if source else "unknown")
except Exception as e:
log.debug(f"Could not flag {key} as bad in Vault '{vault.name}': {e!r}")
if vault.local:
self.tell(vault, kid, key, name)
if source and not source[1].local:
if source[1].report_now:
self.tell(source[1], kid, key, name)
else:
self.unreported[kid, key] = source[1]
def report_replaced(self, content_keys: dict[Union[UUID, str], str]) -> None:
"""Tell each remote vault about its flagged pair once ``content_keys`` holds another key for the KID."""
live = self.live()
for (kid, key), vault in list(self.unreported.items()):
if content_keys.get(kid) not in (None, key):
del self.unreported[kid, key]
if vault in live:
self.tell(vault, kid, key, vault.name)
def tell(self, vault: Vault, kid: Union[UUID, str], key: str, source: str) -> None:
"""Flag one pair in one vault; a failure is only a debug line."""
try:
vault.flag_bad_key(self.service, kid, key, source)
except Exception as e:
log.debug(f"Could not flag {key} as bad in Vault '{vault.name}': {e!r}")
def clear_licensed_flags(self, content_keys: dict[UUID, str]) -> list[UUID]:
"""Unflag each pair a licence returned, since the decode check was wrong. Returns their KIDs."""
cleared = [kid for kid, key in content_keys.items() if self.is_flagged(kid, key)]
for kid in cleared:
self.unflag_bad_key(kid, content_keys[kid])
return cleared
def unflag_bad_key(self, kid: Union[UUID, str], key: str) -> None:
"""Remove a flag from every Vault."""