mirror of
https://github.com/unshackle-dl/unshackle.git
synced 2026-11-17 14:35:06 +00:00
Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
83514d12fe | ||
|
|
dfc99783b5 | ||
|
|
d8663b0dad | ||
|
|
9f054269db | ||
|
|
3345422790 | ||
|
|
cc4bf19706 | ||
|
|
eeaf16cd83 | ||
|
|
af69e8aad4 | ||
|
|
71f293161e | ||
|
|
3e9c26bd46 | ||
|
|
9dd97bcd25 | ||
|
|
f83a594c91 |
@@ -42,7 +42,7 @@ This value becomes the one and only API key that the server accepts. The server
|
||||
If you start `unshackle serve` with no `api_secret` configured, and you have *not* passed `--no-key`, the command fails immediately with:
|
||||
|
||||
```
|
||||
API secret key is not configured. Please add 'api_secret' to the 'serve' section in your config.
|
||||
No 'api_secret' in the 'serve' section of your config
|
||||
```
|
||||
|
||||
Choose a long, unpredictable value. The API key is the only thing standing between the internet and your CDM and download endpoints.
|
||||
|
||||
@@ -1096,6 +1096,7 @@ Get tracks and chapters for a specific title within the remote session. Body: `{
|
||||
"track_manifests": [ { "track_id": "s-1", "type": "dash", "url": "https://...", "data": "<base64 zlib xml>" } ],
|
||||
"session_headers": { "User-Agent": "..." },
|
||||
"session_cookies": { "sid": "..." },
|
||||
"session_cookie_jar": [ { "name": "sid", "value": "...", "domain": ".example.com", "path": "/", "secure": true } ],
|
||||
"server_cdm_type": "widevine"
|
||||
}
|
||||
```
|
||||
@@ -1115,11 +1116,13 @@ Get the download URL, DRM info, headers, cookies, and raw track data for the sel
|
||||
```json
|
||||
{
|
||||
"tracks": {
|
||||
"v-1": { "descriptor": "DASH", "url": "https://...", "drm": [ ... ], "headers": { ... }, "cookies": { ... }, "data": { ... } }
|
||||
"v-1": { "descriptor": "DASH", "url": "https://...", "drm": [ ... ], "headers": { ... }, "cookies": { ... }, "cookie_jar": [ ... ], "data": { ... } }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`cookies` gives names and values only. `cookie_jar` gives the same cookies with their domain, path and `secure` flag, in the `session_cookie_jar` form of the tracks response.
|
||||
|
||||
| Status | Error code | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `200` | - | Segment info returned. |
|
||||
@@ -1215,7 +1218,7 @@ A slow answer arrives as a [slow response](#slow-responses).
|
||||
|
||||
### `POST /api/session/{session_id}/keys/bad`
|
||||
|
||||
Report a content key from `vault_keys` that did not decrypt the track. The server flags the pair in its local vaults and reports it to the vault that served it, and the next licence for that KID reaches the CDM. The client never learns the server vault names.
|
||||
Report a content key from `vault_keys` that did not decrypt the track. The server flags the pair in its local vaults and reports it to the vault that served it, and the next licence for that KID reaches the CDM. When the CDM returns the same pair, the server removes the flag. The client never learns the server vault names.
|
||||
|
||||
```json
|
||||
{ "kid": "<kid_hex>", "key": "<key_hex>" }
|
||||
@@ -1223,7 +1226,7 @@ Report a content key from `vault_keys` that did not decrypt the track. The serve
|
||||
|
||||
| Status | Error code | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `200` | - | The pair is flagged. |
|
||||
| `200` | - | `{"flagged": true}`: the pair is flagged. `{"flagged": false}`: the CDM licensed the pair, so it is not flagged. |
|
||||
| `400` | `INVALID_INPUT` | This remote session was not served that pair. |
|
||||
| `404` | `SESSION_NOT_FOUND` | Unknown remote session. |
|
||||
|
||||
|
||||
@@ -214,8 +214,10 @@ hood the client walks a remote session through its lifecycle.
|
||||
downloading. A service can build an AdaptationSet of its own that the served
|
||||
manifest does not contain. Each such track gets a one-AdaptationSet MPD in
|
||||
`track_manifests`, and the client re-parses that instead. Any
|
||||
`session_headers` / `session_cookies` the server used are merged into the
|
||||
client's local HTTP session.
|
||||
`session_headers` and cookies the server used are merged into the
|
||||
client's local HTTP session. Each cookie in `session_cookie_jar` keeps its
|
||||
domain, path and `secure` flag, so it goes only to its own hosts. `session_cookies` carries
|
||||
the same cookies as names and values only, for older clients.
|
||||
|
||||
The tracks response repeats the title. A service can rewrite title fields inside
|
||||
`get_tracks`, so the client takes this copy in place of the one from the title
|
||||
@@ -301,9 +303,11 @@ There are two ways DRM keys get resolved, chosen by the client's `server_cdm` fl
|
||||
decodes, and a wrong pair goes back to `POST /api/session/{id}/keys/bad`. The
|
||||
server flags the pair in every local vault it holds, reports it to the vault
|
||||
that served it, and the next licence for that KID skips the pair and reaches
|
||||
the CDM. A server with no local SQLite vault cannot store the flag. The
|
||||
server accepts a report only for a pair it served to that remote session. The
|
||||
client never learns the server vault names.
|
||||
the CDM. When the CDM returns the same pair, the server removes the flag,
|
||||
because then the client's check was wrong. A server with no local SQLite
|
||||
vault cannot store the flag. The server accepts a report only for a pair it
|
||||
served to that remote session, and does not flag a pair its CDM licensed.
|
||||
The client never learns the server vault names.
|
||||
- A content key in the client's own vaults goes first, even when the server
|
||||
already answered. The client proves it, and a server key waits as the next
|
||||
candidate.
|
||||
@@ -653,7 +657,7 @@ remote session API and a good template for any client.
|
||||
- **Flow.** `authenticate()` → `create` (+ poll `prompt` every 2s up to a 600s
|
||||
deadline, answering `pending_input` prompts. With stdin closed the client logs an
|
||||
error and raises `SystemExit(1)` instead of posting an empty answer) → `get_titles()` → `get_tracks()`
|
||||
(merging returned `session_headers`/`session_cookies`, re-parsing `manifests`) →
|
||||
(merging returned `session_headers`/`session_cookie_jar`, re-parsing `manifests`) →
|
||||
get the licence in proxy mode or `server_cdm` mode → `close()` (`DELETE`, saving any returned
|
||||
`cache`).
|
||||
- **Server resolution.** `resolve_server()` reads `config.remote_services.<name>`
|
||||
@@ -664,7 +668,7 @@ remote session API and a good template for any client.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
!!! example "\"Could not connect to remote server ... Is it running?\""
|
||||
!!! example "\"Could not connect to remote server at ...\""
|
||||
The client could not reach the URL. Make sure that `unshackle serve` operates on
|
||||
the server, the `url`/port in `remote_services` are correct (default port `8786`),
|
||||
and any firewall or reverse proxy allows the connection.
|
||||
|
||||
@@ -339,7 +339,7 @@ unshackle cfg [KEY] [VALUE] [OPTIONS]
|
||||
| `--list` | List all set configuration values. |
|
||||
|
||||
!!! warning "Comments are stripped"
|
||||
Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error.
|
||||
Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error. If a config key is set twice in the same block, every command warns with the line numbers of the two copies and uses the later copy. `cfg` stops without a change.
|
||||
|
||||
!!! example
|
||||
```shell
|
||||
|
||||
@@ -125,14 +125,18 @@ forwarder for it. See [Control D](#control-d).
|
||||
### Provider load order
|
||||
|
||||
When you do a download without `--no-proxy`, unshackle instantiates every configured
|
||||
proxy provider once, in this fixed order. It uses that same order for bare-query
|
||||
resolution:
|
||||
proxy provider once. It uses that same order for bare-query resolution:
|
||||
|
||||
**Basic → ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola → Control D**
|
||||
1. Basic is always first, so your own proxies win.
|
||||
2. The other proxy providers follow in the order of their keys under `proxy_providers` in
|
||||
`unshackle.yaml`. Proxy providers that load without a config key, such as Hola, come
|
||||
after the listed ones in this default order:
|
||||
**ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola**
|
||||
3. Control D is always last.
|
||||
|
||||
So if you have both Basic and NordVPN configured and use `--proxy us`, a `us` entry in
|
||||
your Basic config wins, because unshackle tries Basic first. To skip ahead to a specific
|
||||
proxy provider, prefix the query (`--proxy nordvpn:us`).
|
||||
So if you list `windscribevpn` above `nordvpn` and use `--proxy us`, a `us` entry in your
|
||||
Basic config answers first, then Windscribe. To skip ahead to a specific proxy provider,
|
||||
prefix the query (`--proxy nordvpn:us`).
|
||||
|
||||
If a proxy provider cannot load, for example because its server list cannot be fetched
|
||||
or its config is not valid, unshackle logs a warning and continues with the other proxy
|
||||
@@ -555,8 +559,7 @@ in unshackle itself.
|
||||
|
||||
!!! warning "Docker is required"
|
||||
Gluetun needs a working [Docker](https://docs.docker.com/get-docker/) installation on
|
||||
your `PATH`. If Docker is not found, the proxy provider fails to load with an error linking
|
||||
to the install docs. The container also needs the `NET_ADMIN` capability and access
|
||||
your `PATH`. If Docker is not found, the proxy provider fails to load. The container also needs the `NET_ADMIN` capability and access
|
||||
to `/dev/net/tun` to bring up the tunnel. unshackle requests both automatically.
|
||||
|
||||
### Query format
|
||||
@@ -639,6 +642,7 @@ proxy_providers:
|
||||
- `surfshark`, `mullvad`, and `ivpn` also require `addresses`.
|
||||
- `windscribe` additionally requires **both** `addresses` and `preshared_key`
|
||||
(`preshared_key` may be an empty string, but the config key must be present).
|
||||
Get both from the [Windscribe WireGuard config generator](https://windscribe.com/getconfig/wireguard).
|
||||
- NordVPN and Proton VPN need only `private_key`.
|
||||
|
||||
=== "OpenVPN"
|
||||
@@ -1025,21 +1029,14 @@ The returned proxy is HTTPS on **port 443**.
|
||||
|
||||
## Developer notes
|
||||
|
||||
!!! note "Two resolution paths (developers)"
|
||||
unshackle has **two** proxy-resolution code paths that are intentionally not
|
||||
identical:
|
||||
|
||||
- `unshackle/commands/dl.py` inlines its own logic and loads **all nine** providers,
|
||||
including Windscribe and Gluetun.
|
||||
- `unshackle/core/proxies/resolve.py::initialize_proxy_providers()` is the shared
|
||||
resolver used by the REST API handlers and the remote-service client. It **omits**
|
||||
Windscribe and Gluetun.
|
||||
|
||||
Both paths accept the same query grammar (a direct URI, a `provider:query`, a bare
|
||||
country query tried against providers in order, or a `controld://` resolver), but the
|
||||
set of available providers differs. Keep this in mind when you add a proxy provider:
|
||||
an entry in `dl.py` does not make it available over the API, and the opposite is also
|
||||
true.
|
||||
!!! note "One resolution path (developers)"
|
||||
`dl`, `search`, the REST API handlers and the remote-service client all load proxy
|
||||
providers through `load_proxy_providers()` in `unshackle/core/proxies/resolve.py`.
|
||||
The REST API and the remote-service client call it through
|
||||
`initialize_proxy_providers()`, which leaves out Gluetun because Gluetun starts a
|
||||
local Docker container. All paths accept the same query grammar: a direct URI, a
|
||||
`provider:query`, a bare query tried against providers in order, or a `controld://`
|
||||
resolver.
|
||||
|
||||
The remote-service client resolves `--proxy` with its own
|
||||
`resolve_remote_proxy_arg()`. It changes a `controld:` query to a `controld://`
|
||||
|
||||
@@ -258,6 +258,11 @@ Each command reports how many files it removed and how much space it freed, then
|
||||
|
||||
**Inspect:** set `debug_requests: true` and operate unshackle with `-d` to show the underlying `urllib3` / `rnet` / `httpx` logs. In the JSON debug log, filter for `service_call` operations to see which request failed. If you use a proxy, also read [Proxies & VPN](proxies-and-vpn.md).
|
||||
|
||||
!!! example "A segment download fails after every retry"
|
||||
**Symptom:** the download stops with a line like `MaxRetriesError: GET <host> failed: RequestError (stream closed because of a broken pipe), gave up after 5 failed attempts over 30s`.
|
||||
|
||||
**Inspect:** the connection to the CDN host dropped on every attempt. This is almost always the network, the VPN, or the proxy, not the service. Run the download again: unshackle keeps the segments it already has. Over HTTP/2, all segments share one connection, so one drop fails every segment in flight. Set `network.http1_only: true` (see [Network & proxy](../reference/configuration/network.md)) to spread them over separate connections. Operate unshackle with `-d` to see each retry as a `segment_retry` entry in the JSON debug log.
|
||||
|
||||
!!! example "The wrong titles/metadata keep coming back"
|
||||
**Symptom:** a service returns outdated titles even after the source changed.
|
||||
|
||||
|
||||
@@ -34,11 +34,11 @@ unshackle pushes keys from every source, not only a local CDM. A run against a r
|
||||
|
||||
### Wrong keys from a poisoned vault
|
||||
|
||||
A public vault can return the right KID with a wrong content key. The decrypters accept a wrong content key and write noise. unshackle therefore trusts a content key from a vault only after FFmpeg decodes the decrypted track with strict error detection. For a fragmented MP4, unshackle reads from the encrypted file which fragments each KID encrypts, and FFmpeg decodes up to three windows of each KID: from its first, middle and last fragment on through the next fragments of that KID, for up to 4 seconds. A window never runs into a fragment of another KID or a clear fragment. A clear lead or a lead under another KID therefore cannot hide a wrong content key, and only the KID that fails is flagged. A KID that is not in this map, and every KID of any other file, gets one check of the whole file. FFmpeg decodes the first seconds and the last seconds. A failure there flags each of those KIDs. When a mapped KID fails, the KIDs that are not in the map get no verdict from that pass; the retry with the replaced key judges them. Content keys from a licence skip this check. A file this FFmpeg build cannot judge counts as a pass, never as a wrong content key: a codec it has no decoder for, or a header it cannot read. A pass is not proof. A wrong content key for HEVC video can decode without an error for some seconds, most often at a low resolution, and that content key passes the check. Without FFmpeg there is no check at all.
|
||||
A public vault can return the right KID with a wrong content key. The decrypters accept a wrong content key and write noise. unshackle therefore trusts a content key from a vault only after FFmpeg decodes the decrypted track with strict error detection. For a fragmented MP4, unshackle reads from the encrypted file which fragments each KID encrypts, and FFmpeg decodes up to three windows of each KID: from its first, middle and last fragment on through the next fragments of that KID, for up to 4 seconds. A window never runs into a fragment of another KID or a clear fragment. A clear lead or a lead under another KID therefore cannot hide a wrong content key, and only the KID that fails is flagged. When every KID in the map has a content key, the map is complete, and a KID that is not in it does not encrypt this file. This file does not judge that KID; the file of its own track judges it. A file with no encrypted sample entry and no encrypted fragment, for example a clear audio track, judges no KID, so it neither flags nor copies a content key. A KID that is not in an incomplete map, for example one with an all-zero KID, and every KID of any other file, gets one check of the whole file. FFmpeg decodes the first seconds and the last seconds. A failure there flags each of those KIDs. When a mapped KID fails, the KIDs that are not in the map get no verdict from that pass; the retry with the replaced key judges them. Content keys from a licence skip this check. A file this FFmpeg build cannot judge counts as a pass, never as a wrong content key: a codec it has no decoder for, or a header it cannot read. A pass is not proof. A wrong content key for HEVC video can decode without an error for some seconds, most often at a low resolution, and that content key passes the check. Without FFmpeg there is no check at all.
|
||||
|
||||
When the decode fails, unshackle:
|
||||
|
||||
1. Flags the KID:KEY in the `bad_keys` table of every local SQLite vault, with the service and the name of the vault it came from. The pair also goes out of the service table, except in a vault marked `no_push`, where only the flag is written. A flagged pair cannot go back into a local vault from another vault, a later run, or `kv copy`. A licence that returns the pair clears the flag and stores it. An `API` vault that served the pair also gets a report; `HTTP` and `MySQL` vaults have no report route. A pair from a local vault goes to no remote vault, even when a remote vault holds the same pair.
|
||||
1. Flags the KID:KEY in the `bad_keys` table of every local SQLite vault, with the service and the name of the vault it came from. The pair also goes out of the service table, except in a vault marked `no_push`, where only the flag is written. A flagged pair cannot go back into a local vault from another vault, a later run, or `kv copy`. A licence that returns the pair clears the flag and stores it. An `API` vault that served the pair gets a report only when a different content key for that KID is accepted. When the CDM returns the same content key, that vault gets no report. `HTTP` and `MySQL` vaults have no report route. A pair from a local vault goes to no remote vault, even when a remote vault holds the same pair.
|
||||
2. Restores the ciphertext from a hard link it made before decryption, then asks for the content key again. Every vault lookup skips a flagged pair, so the request reaches the remaining vaults and then the CDM. A remote answer that arrived after the winning one is reused instead of queried again.
|
||||
3. Removes the flag when the CDM returns the same content key, because then the decode check was wrong, not the content key. A content key from another vault is checked in its turn, and flagged as well if it also fails.
|
||||
|
||||
|
||||
@@ -61,8 +61,11 @@ headers:
|
||||
- **Type:** `dict` · **Default:** `{}`
|
||||
|
||||
Proxy/VPN provider configuration. Each sub-key names a proxy provider, and unshackle passes its
|
||||
block straight to that provider's constructor. See [Proxies & VPN](../../guide/proxies-and-vpn.md)
|
||||
for the full proxy provider guide. Recognised proxy providers and their exit ports:
|
||||
block straight to that provider's constructor. A bare `--proxy` query tries Basic first, then
|
||||
the other proxy providers in the order of their sub-keys, and Control D last (see
|
||||
[Provider load order](../../guide/proxies-and-vpn.md#provider-load-order)). See
|
||||
[Proxies & VPN](../../guide/proxies-and-vpn.md) for the full proxy provider guide. Recognised proxy
|
||||
providers and their exit ports:
|
||||
|
||||
| Provider | Config key | Credentials | Proxy scheme/port |
|
||||
|----------|-----------|-------------|-------------------|
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from unshackle.commands import cfg as cfg_module
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def config_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
||||
path = tmp_path / "unshackle.yaml"
|
||||
monkeypatch.setattr(cfg_module, "get_config_path", lambda: path)
|
||||
return path
|
||||
|
||||
|
||||
def test_a_duplicate_key_logs_its_lines_once_and_names_no_key_or_value(
|
||||
config_file: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
config_file.write_text(
|
||||
"theme: rainbow\ncredentials:\n EXAMPLE: first-secret\n OTHER: other\n EXAMPLE: second-secret\n"
|
||||
"serve:\n users:\n key-secret: {tier: a}\n key-secret: {tier: b}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
with caplog.at_level(logging.WARNING, logger="config"):
|
||||
result = CliRunner().invoke(cfg_module.cfg, ["theme"])
|
||||
CliRunner().invoke(cfg_module.cfg, ["theme"])
|
||||
assert isinstance(result.exception, SystemExit), result.exception
|
||||
assert result.exit_code == 1
|
||||
assert caplog.messages == [
|
||||
f"{config_file}: duplicate key on lines 3 and 5, lines 8 and 9. Unshackle uses the later one."
|
||||
]
|
||||
assert "Error" not in result.output and "lines" not in result.output
|
||||
for output in (result.output, caplog.text):
|
||||
assert "secret" not in output and "EXAMPLE" not in output
|
||||
|
||||
|
||||
def test_a_duplicate_key_blocks_a_write(config_file: Path) -> None:
|
||||
original = "theme: rainbow\ncredentials:\n EXAMPLE: a\n EXAMPLE: b\n"
|
||||
config_file.write_text(original, encoding="utf-8")
|
||||
result = CliRunner().invoke(cfg_module.cfg, ["theme", "plain"])
|
||||
assert result.exit_code == 1
|
||||
assert config_file.read_text(encoding="utf-8") == original
|
||||
|
||||
|
||||
def test_a_valid_config_still_reads_a_key(config_file: Path) -> None:
|
||||
config_file.write_text("theme: rainbow\n", encoding="utf-8")
|
||||
result = CliRunner().invoke(cfg_module.cfg, ["theme"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "rainbow" in result.output
|
||||
|
||||
|
||||
def test_a_duplicate_only_ruamel_sees_still_gives_its_line(config_file: Path, caplog: pytest.LogCaptureFixture) -> None:
|
||||
config_file.write_text("a: 1\n1: x\n0x1: y\n", encoding="utf-8")
|
||||
with caplog.at_level(logging.WARNING):
|
||||
result = CliRunner().invoke(cfg_module.cfg, ["a"])
|
||||
assert result.exit_code == 1
|
||||
assert caplog.messages == [f"{config_file}: duplicate key on line 3. Unshackle uses the later one."]
|
||||
@@ -0,0 +1,53 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
from ruamel.yaml import YAML
|
||||
|
||||
from unshackle.core.config import Config, duplicate_key_lines, load_yaml
|
||||
|
||||
ALIASES = "base: &b\n x: 1\n x: 2\n self: *b\nother: *b\nlist:\n - y: 1\n y: 2\n"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("compose", [yaml.compose, YAML().compose], ids=["pyyaml", "ruamel"])
|
||||
def test_duplicate_key_lines_follows_aliases_without_looping(compose) -> None:
|
||||
assert duplicate_key_lines(compose(ALIASES)) == [(2, 3), (7, 8)]
|
||||
|
||||
|
||||
def test_load_yaml_warns_once_with_lines_and_keeps_the_later_copy(
|
||||
tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
path = tmp_path / "unshackle.yaml"
|
||||
path.write_text("credentials:\n EXAMPLE: first-secret\n EXAMPLE: second-secret\n", encoding="utf-8")
|
||||
with caplog.at_level(logging.WARNING, logger="config"):
|
||||
data = load_yaml(path)
|
||||
load_yaml(path)
|
||||
assert data == {"credentials": {"EXAMPLE": "second-secret"}}
|
||||
assert caplog.messages == [f"{path}: duplicate key on lines 2 and 3. Unshackle uses the later one."]
|
||||
assert "EXAMPLE" not in caplog.text and "secret" not in caplog.text
|
||||
|
||||
|
||||
def test_config_from_yaml_warns_on_a_duplicate_key(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
|
||||
path = tmp_path / "unshackle.yaml"
|
||||
path.write_text("tag: first\ntag: second\n", encoding="utf-8")
|
||||
with caplog.at_level(logging.WARNING, logger="config"):
|
||||
assert Config.from_yaml(path).tag == "second"
|
||||
assert "duplicate key on lines 1 and 2" in caplog.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
["a: 1\nb: [1, 2]\n", "", "a: &a\n x: 1\nb: &b\n y: 2\nc:\n <<: *a\n <<: *b\n"],
|
||||
ids=["valid", "empty", "two-merges"],
|
||||
)
|
||||
def test_load_yaml_matches_safe_load_without_a_warning(
|
||||
tmp_path: Path, text: str, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
path = tmp_path / "config.yaml"
|
||||
path.write_text(text, encoding="utf-8")
|
||||
with caplog.at_level(logging.WARNING, logger="config"):
|
||||
assert load_yaml(path) == yaml.safe_load(text)
|
||||
assert not caplog.records
|
||||
@@ -652,19 +652,14 @@ def video(
|
||||
|
||||
def test_hybrid_fails_with_neither_a_base_nor_dv() -> None:
|
||||
assert hybrid_gate_failure([video("sdr", Video.Range.SDR)], True) == (
|
||||
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but neither is available",
|
||||
"Available ranges: SDR",
|
||||
)
|
||||
assert hybrid_gate_failure([], True) == (
|
||||
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but neither is available",
|
||||
"Available ranges: none",
|
||||
"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: SDR"
|
||||
)
|
||||
assert hybrid_gate_failure([], True) == "HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: none"
|
||||
|
||||
|
||||
def test_hybrid_fails_with_dv_only() -> None:
|
||||
assert hybrid_gate_failure([video("dv", Video.Range.DV)], True) == (
|
||||
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only DV is available",
|
||||
"Available ranges: DV",
|
||||
"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: DV"
|
||||
)
|
||||
|
||||
|
||||
@@ -681,27 +676,16 @@ def test_hybrid_goes_on_with_a_base_and_dv_whatever_the_flag() -> None:
|
||||
def test_hybrid_fails_up_front_when_no_base_is_hevc() -> None:
|
||||
"""dovi_tool injects into HEVC, so an all-AV1 title fails before it downloads anything."""
|
||||
videos = [video("av1", Video.Range.HDR10P, codec=Video.Codec.AV1), video("dv", Video.Range.DV)]
|
||||
assert hybrid_gate_failure(videos, False) == (
|
||||
"HYBRID mode requires an HEVC HDR10/HDR10+ base, but none is available",
|
||||
"Available HDR10/HDR10+ codecs: AV1",
|
||||
)
|
||||
assert hybrid_gate_failure(videos, False) == "HYBRID needs an HEVC HDR10/HDR10+ base, available codecs: AV1"
|
||||
assert hybrid_gate_failure(videos + [video("h", Video.Range.HDR10, codec=None)], False) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"ranges, to_dv, detail",
|
||||
[
|
||||
(
|
||||
[Video.Range.HDR10P],
|
||||
False,
|
||||
"Available ranges: HDR10P. Add --generate-dv to generate Dolby Vision from HDR10+.",
|
||||
),
|
||||
([Video.Range.HDR10], False, "Available ranges: HDR10"),
|
||||
(
|
||||
[Video.Range.HDR10],
|
||||
True,
|
||||
"Available ranges: HDR10. --generate-dv needs an HEVC HDR10+ track, and the service returned none.",
|
||||
),
|
||||
([Video.Range.HDR10P], False, "HDR10P; add --generate-dv to make DV from HDR10+"),
|
||||
([Video.Range.HDR10], False, "HDR10"),
|
||||
([Video.Range.HDR10], True, "HDR10; --generate-dv found no HEVC HDR10+ track"),
|
||||
],
|
||||
)
|
||||
def test_hybrid_fails_without_dv_and_hints_at_the_flag_only_when_it_is_off(
|
||||
@@ -709,10 +693,7 @@ def test_hybrid_fails_without_dv_and_hints_at_the_flag_only_when_it_is_off(
|
||||
) -> None:
|
||||
"""A service that fetches HYBRID by hand may never return HDR10+; the hint must not blame the user then."""
|
||||
videos = [video(r.name, r) for r in ranges]
|
||||
assert hybrid_gate_failure(videos, to_dv) == (
|
||||
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only HDR10 is available",
|
||||
detail,
|
||||
)
|
||||
assert hybrid_gate_failure(videos, to_dv) == f"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: {detail}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("to_dv", [False, True])
|
||||
|
||||
@@ -150,4 +150,4 @@ def test_a_download_stopped_by_a_click_exception_reports_only_its_message() -> N
|
||||
lines = dl.failure_lines(error)
|
||||
|
||||
assert lines == [":x: Download Failed...", f" {error.format_message()}"]
|
||||
assert "unexpected error" in dl.failure_lines(ValueError("boom"))[-1]
|
||||
assert dl.failure_lines(ValueError("boom")) == [":x: Download Failed...", " ValueError: boom"]
|
||||
|
||||
@@ -3,6 +3,8 @@ from pathlib import Path
|
||||
from typing import Optional
|
||||
from uuid import UUID
|
||||
|
||||
import pytest
|
||||
|
||||
from unshackle.core.drm.verify import PIFF_TENC, PIFF_TFXD, KidMap, kid_windows, pick_windows
|
||||
|
||||
A, B, C = UUID(int=0xA), UUID(int=0xB), UUID(int=0xC)
|
||||
@@ -44,15 +46,28 @@ def sbgp(*runs: tuple[int, int]) -> bytes:
|
||||
return full(b"sbgp", b"seig" + struct.pack(">I", len(runs)) + b"".join(struct.pack(">II", *run) for run in runs))
|
||||
|
||||
|
||||
def movie(*sample_entries: bytes, handler: bytes = b"vide", stbl: bytes = b"") -> bytes:
|
||||
def trak(*sample_entries: bytes, track: int = 1, handler: bytes = b"vide", stbl: bytes = b"") -> bytes:
|
||||
stsd = full(b"stsd", struct.pack(">I", len(sample_entries)) + b"".join(sample_entries))
|
||||
hdlr = full(b"hdlr", bytes(4) + handler + bytes(12))
|
||||
trak = box(
|
||||
return box(
|
||||
b"trak",
|
||||
full(b"tkhd", bytes(8) + struct.pack(">I", 1)),
|
||||
full(b"tkhd", bytes(8) + struct.pack(">I", track)),
|
||||
box(b"mdia", full(b"mdhd", bytes(8) + struct.pack(">I", 1000)), hdlr, box(b"minf", box(b"stbl", stsd, stbl))),
|
||||
)
|
||||
return box(b"ftyp", b"iso6") + box(b"moov", trak, box(b"mvex", full(b"trex", struct.pack(">III", 1, 1, 0))))
|
||||
|
||||
|
||||
def movie(*sample_entries: bytes, handler: bytes = b"vide", stbl: bytes = b"", others: bytes = b"") -> bytes:
|
||||
moov = box(
|
||||
b"moov",
|
||||
trak(*sample_entries, handler=handler, stbl=stbl),
|
||||
others,
|
||||
box(b"mvex", full(b"trex", struct.pack(">III", 1, 1, 0))),
|
||||
)
|
||||
return box(b"ftyp", b"iso6") + moov
|
||||
|
||||
|
||||
def clear_audio(others: bytes = b"") -> bytes:
|
||||
return movie(box(b"mp4a", bytes(28)), handler=b"soun", others=others)
|
||||
|
||||
|
||||
def moof(
|
||||
@@ -178,3 +193,45 @@ def test_not_a_fragmented_mp4(tmp_path: Path) -> None:
|
||||
path = tmp_path / "video.ts"
|
||||
path.write_bytes(b"\x47" * 188 * 4)
|
||||
assert kid_windows(path) is None
|
||||
|
||||
|
||||
def test_clear_mp4_maps_no_kid(tmp_path: Path) -> None:
|
||||
path = tmp_path / "clear.m4a"
|
||||
path.write_bytes(clear_audio() + moof(1, 0))
|
||||
assert kid_windows(path) == KidMap({}, False)
|
||||
|
||||
|
||||
def test_encrypted_entry_without_a_tenc_is_not_clear(tmp_path: Path) -> None:
|
||||
path = tmp_path / "enc.mp4"
|
||||
path.write_bytes(movie(box(b"encv", bytes(78), box(b"sinf", box(b"frma", b"avc1")))) + moof(1, 0))
|
||||
assert kid_windows(path) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", [b"enct", b"encs", b"drmi"])
|
||||
def test_other_encrypted_entry_kinds_are_not_clear(tmp_path: Path, kind: bytes) -> None:
|
||||
path = tmp_path / "enc.mp4"
|
||||
path.write_bytes(movie(box(kind, bytes(16)), handler=b"text") + moof(1, 0))
|
||||
assert kid_windows(path) is None
|
||||
|
||||
|
||||
def test_a_clear_entry_with_a_protecting_senc_is_not_clear(tmp_path: Path) -> None:
|
||||
path = tmp_path / "enc.m4a"
|
||||
senc = full(b"senc", struct.pack(">I", 1) + bytes(8))
|
||||
path.write_bytes(clear_audio() + moof(1, 0, senc))
|
||||
assert kid_windows(path) is None
|
||||
path.write_bytes(clear_audio() + moof(1, 0, full(b"senc", bytes(4))))
|
||||
assert kid_windows(path) == KidMap({}, False)
|
||||
|
||||
|
||||
def test_only_the_tracks_with_fragments_decide_clear(tmp_path: Path) -> None:
|
||||
path = tmp_path / "clear.m4a"
|
||||
path.write_bytes(clear_audio(others=trak(encv(A), track=2)) + moof(1, 0))
|
||||
assert kid_windows(path) == KidMap({}, False)
|
||||
path.write_bytes(clear_audio(others=trak(encv(A), track=2)))
|
||||
assert kid_windows(path) is None
|
||||
|
||||
|
||||
def test_a_clear_lead_beside_an_encrypted_entry_is_not_clear(tmp_path: Path) -> None:
|
||||
path = tmp_path / "lead.mp4"
|
||||
path.write_bytes(movie(box(b"avc1", bytes(78)), encv(A)) + moof(1, 0))
|
||||
assert kid_windows(path) is None
|
||||
|
||||
@@ -115,10 +115,10 @@ def test_other_audio_still_muxes_with_mkvmerge(tmp_path: Path, monkeypatch: pyte
|
||||
tracks.mux("Title", delete=False, output_path=tmp_path / "out.muxed.mkv")
|
||||
|
||||
|
||||
def test_a_missing_mp4box_names_the_install(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
def test_a_missing_mp4box_names_the_binary(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(binaries, "MP4Box", None, raising=False)
|
||||
tracks = build_tracks(tmp_path, b"dtsx")
|
||||
with pytest.raises(RuntimeError, match="gpac"):
|
||||
with pytest.raises(RuntimeError, match="MP4Box"):
|
||||
tracks.mux("Title", delete=False, output_path=tmp_path / "out.muxed.mkv")
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
"""A bare UTF-16LE WRMHEADER in place of a PSSH box or PlayReady Object loads as PlayReady DRM."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import logging
|
||||
from uuid import UUID
|
||||
|
||||
import m3u8
|
||||
import pytest
|
||||
from lxml import etree
|
||||
from pyplayready.misc.exceptions import InvalidPssh
|
||||
from pyplayready.system.pssh import PSSH
|
||||
from pywidevine.pssh import PSSH as WV_PSSH
|
||||
|
||||
from unshackle.core.drm import PlayReady, Widevine, drm_from_dict
|
||||
from unshackle.core.manifests.dash import DASH
|
||||
from unshackle.core.manifests.hls import HLS
|
||||
from unshackle.core.manifests.ism import ISM
|
||||
|
||||
KID = UUID("00112233-4455-6677-8899-aabbccddeeff")
|
||||
KID_B64 = base64.b64encode(KID.bytes_le).decode()
|
||||
PR_URN = "urn:uuid:9a04f079-9840-4286-ab92-e65be0885f95"
|
||||
WV_URN = "urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed"
|
||||
|
||||
|
||||
def bare_header(version: str) -> str:
|
||||
"""Base64 of a bare WRMHEADER, with the KID where the given header version puts it."""
|
||||
if version == "4.0.0.0":
|
||||
data = f"<PROTECTINFO><KEYLEN>16</KEYLEN><ALGID>AESCTR</ALGID></PROTECTINFO><KID>{KID_B64}</KID>"
|
||||
else:
|
||||
data = f'<PROTECTINFO><KIDS><KID ALGID="AESCTR" VALUE="{KID_B64}"></KID></KIDS></PROTECTINFO>'
|
||||
xml = (
|
||||
f'<WRMHEADER xmlns="http://schemas.microsoft.com/DRM/2007/03/PlayReadyHeader" version="{version}">'
|
||||
f"<DATA>{data}<LA_URL>https://example.invalid/pr</LA_URL></DATA></WRMHEADER>"
|
||||
)
|
||||
return base64.b64encode(xml.encode("utf-16-le")).decode()
|
||||
|
||||
|
||||
def pr_protection(child: str, value: str) -> etree._Element:
|
||||
return etree.fromstring(
|
||||
f'<ContentProtection xmlns:cenc="urn:mpeg:cenc:2013" xmlns:mspr="urn:microsoft:playready" '
|
||||
f'schemeIdUri="{PR_URN}"><{child}>{value}</{child}></ContentProtection>'
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version", ["4.0.0.0", "4.1.0.0", "4.2.0.0", "4.3.0.0"])
|
||||
@pytest.mark.parametrize("child", ["mspr:pro", "cenc:pssh"])
|
||||
def test_dash_loads_a_bare_header(child: str, version: str) -> None:
|
||||
drm = DASH.get_drm([pr_protection(child, bare_header(version))])
|
||||
|
||||
assert len(drm) == 1
|
||||
assert isinstance(drm[0], PlayReady)
|
||||
assert KID in drm[0].kids
|
||||
assert drm[0].pssh_b64
|
||||
PSSH(drm[0].pssh_b64)
|
||||
assert KID in drm_from_dict(drm[0].to_dict()).kids
|
||||
|
||||
|
||||
def test_wrapped_header_bytes_are_unchanged() -> None:
|
||||
header = bare_header("4.0.0.0")
|
||||
drm = PlayReady.from_pssh_b64(header)
|
||||
|
||||
assert drm.pssh.wrm_headers[0].dumps().encode("utf-16-le") == base64.b64decode(header)
|
||||
|
||||
|
||||
def test_dash_skips_an_unparseable_playready_entry_and_keeps_widevine(caplog: pytest.LogCaptureFixture) -> None:
|
||||
widevine = etree.fromstring(
|
||||
f'<ContentProtection xmlns:cenc="urn:mpeg:cenc:2013" schemeIdUri="{WV_URN}">'
|
||||
f"<cenc:pssh>{WV_PSSH.new(system_id=WV_PSSH.SystemId.Widevine, key_ids=[KID]).dumps()}</cenc:pssh>"
|
||||
"</ContentProtection>"
|
||||
)
|
||||
garbage = pr_protection("mspr:pro", base64.b64encode(b"\x00\x00not a header").decode())
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="DASH"):
|
||||
drm = DASH.get_drm([widevine, garbage])
|
||||
|
||||
assert [type(d) for d in drm] == [Widevine]
|
||||
assert "Skipping unparseable PlayReady ContentProtection" in caplog.text
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"data",
|
||||
[b"\x00\x00not a header", "<WRMHEADER/>".encode("utf-8")],
|
||||
ids=["garbage", "utf-8-header"],
|
||||
)
|
||||
def test_other_bad_data_keeps_the_original_error(data: bytes) -> None:
|
||||
with pytest.raises(InvalidPssh):
|
||||
PlayReady.from_pssh_b64(base64.b64encode(data).decode())
|
||||
|
||||
|
||||
def test_hls_loads_a_bare_header() -> None:
|
||||
key = m3u8.loads(
|
||||
"#EXTM3U\n"
|
||||
f'#EXT-X-KEY:METHOD=SAMPLE-AES-CTR,URI="data:text/plain;charset=UTF-16;base64,{bare_header("4.0.0.0")}",'
|
||||
'KEYFORMAT="com.microsoft.playready",KEYFORMATVERSIONS="1"\n'
|
||||
"#EXTINF:4.0,\nseg0.mp4\n"
|
||||
).keys[0]
|
||||
|
||||
drm = HLS.get_drm(key)
|
||||
|
||||
assert isinstance(drm, PlayReady)
|
||||
assert KID in drm.kids
|
||||
|
||||
|
||||
def test_ism_loads_a_bare_header() -> None:
|
||||
header = etree.fromstring(f'<ProtectionHeader SystemID="{PR_URN[9:]}">{bare_header("4.0.0.0")}</ProtectionHeader>')
|
||||
|
||||
drm = ISM.get_drm([header])
|
||||
|
||||
assert len(drm) == 1
|
||||
assert KID in drm[0].kids
|
||||
@@ -63,6 +63,17 @@ def test_raise_errors_raises_the_load_failure() -> None:
|
||||
load_proxy_providers(OTHERS, raise_errors=True)
|
||||
|
||||
|
||||
def test_bare_query_order_follows_config_key_order_with_controld_last(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from unshackle.core.proxies import ControlD
|
||||
|
||||
order = {"controld": {"token": "t"}, "surfsharkvpn": CREDENTIALS, "nordvpn": CREDENTIALS, "basic": {"us": "x"}}
|
||||
monkeypatch.setattr(config, "proxy_providers", order)
|
||||
monkeypatch.setattr(ControlD, "get_locations", lambda self: [])
|
||||
loaded = load_proxy_providers(("expressvpn", "protonvpn", "hola"), quiet=True)
|
||||
names = [getattr(p, "name", p.__class__.__name__) for p in loaded]
|
||||
assert names == ["Basic", "SurfsharkVPN", "NordVPN", "ControlD"]
|
||||
|
||||
|
||||
def test_rest_path_loads_windscribe_but_not_gluetun(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from unshackle.core.proxies import Gluetun, WindscribeVPN
|
||||
from unshackle.core.proxies.resolve import initialize_proxy_providers
|
||||
|
||||
@@ -27,3 +27,11 @@ def test_retry_log_counts_only_real_retries(rate_limited, caplog, max_retries):
|
||||
|
||||
retries = [r.getMessage() for r in caplog.records if "Retrying" in r.getMessage()]
|
||||
assert retries == [f"429 Too Many Requests(/x). Retrying... ({n}/{max_retries})" for n in range(1, max_retries + 1)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("max_retries", "attempts"), [(0, ""), (2, " after 3 attempts")])
|
||||
def test_spent_budget_names_the_attempts_and_host_not_the_url(rate_limited, max_retries, attempts):
|
||||
with pytest.raises(MaxRetriesError) as raised:
|
||||
rate_limited.get("https://example.invalid/x?hmac=secret", max_retries=max_retries)
|
||||
|
||||
assert str(raised.value) == f"GET example.invalid failed{attempts}: 429 Too Many Requests"
|
||||
|
||||
@@ -287,6 +287,26 @@ def test_cdm_returning_the_flagged_key_clears_the_flag(tmp_path: Path, monkeypat
|
||||
assert not local.is_bad_key(KID, BAD)
|
||||
|
||||
|
||||
def test_a_licence_key_that_replaces_a_bad_local_pair_is_stored(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The local vault held the bad pair when the licence key arrived, so it skipped the KID; store it now."""
|
||||
local = SQLite("local", tmp_path / "local.db")
|
||||
local.add_key("SVC", KID, BAD)
|
||||
vaults = Vaults("SVC")
|
||||
vaults.vaults = [local]
|
||||
cmd = make_cmd(monkeypatch, vaults)
|
||||
drm = FakeDRM({KID: vaults.get_key(KID)[0]})
|
||||
path = tmp_path / "video.mp4"
|
||||
path.write_bytes(b"cipher")
|
||||
assert local.add_keys("SVC", {KID: GOOD}) == 0
|
||||
|
||||
cmd.decrypt_verified(drm, path, lambda drm, track_kid: drm.content_keys.update({KID: GOOD}), KID)
|
||||
cmd.wait_vault_writes()
|
||||
assert local.is_bad_key(KID, BAD)
|
||||
assert local.get_key(KID, "SVC") == GOOD
|
||||
|
||||
|
||||
def test_stale_backup_from_a_killed_run_is_not_restored(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
poisoned = Remote("poisoned", tmp_path / "poisoned.db")
|
||||
poisoned.add_key("SVC", KID, BAD)
|
||||
@@ -426,7 +446,7 @@ def test_only_the_kid_whose_window_fails_is_flagged(tmp_path: Path, monkeypatch:
|
||||
|
||||
|
||||
def test_a_kid_absent_from_the_map_is_judged_by_the_fallback(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A KID the map does not know (a zero-KID tenc, a licence KID the file does not carry)
|
||||
"""A map that holds an all-zero tenc KID is not complete, so a KID it does not know
|
||||
shares the whole-file verdict, and a wrong key there is flagged."""
|
||||
lead = UUID(int=7)
|
||||
local = SQLite("local", tmp_path / "local.db")
|
||||
@@ -446,7 +466,9 @@ def test_a_kid_absent_from_the_map_is_judged_by_the_fallback(tmp_path: Path, mon
|
||||
return drm.decrypted_with[-1][lead if start is not None else KID] == GOOD
|
||||
|
||||
monkeypatch.setattr(
|
||||
dl_module.verify, "kid_windows", lambda path: dl_module.verify.KidMap({lead: [(4.0, 6.0)]}, True)
|
||||
dl_module.verify,
|
||||
"kid_windows",
|
||||
lambda path: dl_module.verify.KidMap({lead: [(4.0, 6.0)], UUID(int=0): [(8.0, 10.0)]}, True),
|
||||
)
|
||||
monkeypatch.setattr(dl_module, "ffmpeg_decodes", decodes)
|
||||
path = tmp_path / "video.mp4"
|
||||
@@ -501,11 +523,93 @@ def test_a_wrong_mapped_key_does_not_flag_a_kid_the_file_does_not_carry(
|
||||
assert drm.content_keys == {audio: GOOD, KID: GOOD}
|
||||
assert local.is_bad_key(KID, BAD)
|
||||
assert not local.is_bad_key(audio, GOOD)
|
||||
assert local.get_key(audio, "SVC") == GOOD # judged on the retry and copied
|
||||
assert local.get_key(audio, "SVC") is None
|
||||
|
||||
|
||||
def test_a_kid_the_file_does_not_carry_is_not_judged_by_it(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The CDM licensed the video KID and a vault served the audio KID. An FFmpeg that fails
|
||||
every decode of the video file must not flag the audio KID, nor copy it to another vault."""
|
||||
audio = UUID(int=7)
|
||||
local = SQLite("local", tmp_path / "local.db")
|
||||
remote = Remote("remote", tmp_path / "remote.db")
|
||||
remote.add_key("SVC", audio, GOOD)
|
||||
vaults = Vaults("SVC")
|
||||
vaults.vaults = [local, remote]
|
||||
cmd = make_cmd(monkeypatch, vaults)
|
||||
drm = FakeDRM({audio: vaults.get_key(audio)[0], KID: GOOD})
|
||||
monkeypatch.setattr(
|
||||
dl_module.verify, "kid_windows", lambda path: dl_module.verify.KidMap({KID: [(4.0, 6.0)]}, True)
|
||||
)
|
||||
monkeypatch.setattr(dl_module, "ffmpeg_decodes", lambda path, start=None, seconds=3, video=None: False)
|
||||
path = tmp_path / "video.mp4"
|
||||
path.write_bytes(b"cipher")
|
||||
licences: list[UUID] = []
|
||||
|
||||
cmd.decrypt_verified(drm, path, lambda drm, track_kid: licences.append(track_kid), KID)
|
||||
cmd.wait_vault_writes()
|
||||
assert drm.content_keys == {audio: GOOD, KID: GOOD}
|
||||
assert not licences
|
||||
assert not local.is_bad_key(audio, GOOD)
|
||||
assert local.get_key(audio, "SVC") is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("served", "decodes", "told"),
|
||||
[
|
||||
(GOOD, False, False), # broken FFmpeg, the CDM returns the same key
|
||||
(BAD, True, True), # the CDM returns another key
|
||||
],
|
||||
)
|
||||
def test_a_remote_vault_is_told_only_when_another_key_replaces_its_pair(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, served: str, decodes: bool, told: bool
|
||||
) -> None:
|
||||
remote = Remote("remote", tmp_path / "remote.db")
|
||||
remote.add_key("SVC", KID, served)
|
||||
vaults = Vaults("SVC")
|
||||
vaults.vaults = [SQLite("local", tmp_path / "local.db"), remote]
|
||||
cmd = make_cmd(monkeypatch, vaults)
|
||||
if not decodes:
|
||||
monkeypatch.setattr(dl_module, "ffmpeg_decodes", lambda path, start=None, seconds=3, video=None: False)
|
||||
drm = FakeDRM({KID: vaults.get_key(KID)[0]})
|
||||
path = tmp_path / "video.mp4"
|
||||
path.write_bytes(b"cipher")
|
||||
|
||||
def licence(drm: FakeDRM, track_kid: UUID) -> None:
|
||||
assert not remote.is_bad_key(KID, served)
|
||||
drm.content_keys[KID] = GOOD
|
||||
|
||||
cmd.decrypt_verified(drm, path, licence, KID)
|
||||
assert drm.content_keys == {KID: GOOD}
|
||||
assert remote.is_bad_key(KID, served) is told
|
||||
|
||||
|
||||
def test_the_server_vault_is_told_at_once(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Over --remote the report is what makes the server skip the pair on the next licence."""
|
||||
reports: list[tuple[UUID, str]] = []
|
||||
server = SimpleNamespace(
|
||||
name="server_cdm",
|
||||
local=False,
|
||||
report_now=True,
|
||||
flag_bad_key=lambda service, kid, key, source: reports.append((kid, key)),
|
||||
)
|
||||
vaults = Vaults("SVC")
|
||||
vaults.vaults = [SQLite("local", tmp_path / "local.db")]
|
||||
vaults.sources[KID] = (BAD, server)
|
||||
cmd = make_cmd(monkeypatch, vaults)
|
||||
drm = FakeDRM({KID: BAD})
|
||||
path = tmp_path / "video.mp4"
|
||||
path.write_bytes(b"cipher")
|
||||
|
||||
def licence(drm: FakeDRM, track_kid: UUID) -> None:
|
||||
assert reports == [(KID, BAD)]
|
||||
drm.content_keys[KID] = GOOD
|
||||
|
||||
cmd.decrypt_verified(drm, path, licence, KID)
|
||||
assert drm.content_keys == {KID: GOOD}
|
||||
|
||||
|
||||
def test_only_checked_kids_are_copied_to_other_vaults(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Both the mapped KID and the absent one are proven before they reach another vault."""
|
||||
"""Only the mapped KID is proven and copied; the KID this file does not carry waits for its own file."""
|
||||
lead = UUID(int=7)
|
||||
local = SQLite("local", tmp_path / "local.db")
|
||||
remote = Remote("remote", tmp_path / "remote.db")
|
||||
@@ -530,9 +634,35 @@ def test_only_checked_kids_are_copied_to_other_vaults(tmp_path: Path, monkeypatc
|
||||
|
||||
cmd.decrypt_verified(drm, path, None, KID)
|
||||
cmd.wait_vault_writes()
|
||||
assert starts.count(None) == 1 # the absent KID went through the fallback before its copy
|
||||
assert None not in starts
|
||||
assert local.get_key(lead, "SVC") == GOOD
|
||||
assert local.get_key(KID, "SVC") == GOOD
|
||||
assert local.get_key(KID, "SVC") is None
|
||||
|
||||
|
||||
def test_a_clear_file_judges_and_copies_no_vault_key(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
local = SQLite("local", tmp_path / "local.db")
|
||||
remote = Remote("remote", tmp_path / "remote.db")
|
||||
remote.add_key("SVC", KID, BAD)
|
||||
vaults = Vaults("SVC")
|
||||
vaults.vaults = [local, remote]
|
||||
cmd = make_cmd(monkeypatch, vaults)
|
||||
drm = FakeDRM({KID: vaults.get_key(KID)[0]})
|
||||
checks: list = []
|
||||
|
||||
def decodes(path: Path, start: object = None, seconds: float = 3, video: object = None) -> bool:
|
||||
checks.append(start)
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(dl_module.verify, "kid_windows", lambda path: dl_module.verify.KidMap({}, False))
|
||||
monkeypatch.setattr(dl_module, "ffmpeg_decodes", decodes)
|
||||
path = tmp_path / "audio.mp4"
|
||||
path.write_bytes(b"clear")
|
||||
|
||||
cmd.decrypt_verified(drm, path, None, KID)
|
||||
cmd.wait_vault_writes()
|
||||
assert checks == []
|
||||
assert local.get_key(KID, "SVC") is None
|
||||
assert not local.is_bad_key(KID, BAD)
|
||||
|
||||
|
||||
def test_flagged_pair_warns_once_per_vault(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
|
||||
@@ -602,6 +732,8 @@ def test_api_vault_reports_the_pair_upstream(monkeypatch: pytest.MonkeyPatch, ca
|
||||
vaults.vaults = [api]
|
||||
vaults.sources[KID] = (BAD, api)
|
||||
vaults.flag_bad_key(KID, BAD)
|
||||
assert not calls
|
||||
vaults.report_replaced({KID: GOOD})
|
||||
vaults.unflag_bad_key(KID, BAD)
|
||||
|
||||
assert calls == [
|
||||
@@ -609,28 +741,31 @@ def test_api_vault_reports_the_pair_upstream(monkeypatch: pytest.MonkeyPatch, ca
|
||||
("DELETE", f"https://vault.example/bad-keys/{KID.hex}/{BAD}", None),
|
||||
]
|
||||
|
||||
def flag_and_replace(source: object) -> None:
|
||||
vaults.sources[KID] = (BAD, source)
|
||||
vaults.flag_bad_key(KID, BAD)
|
||||
vaults.report_replaced({KID: GOOD})
|
||||
|
||||
calls.clear()
|
||||
vaults.sources[KID] = (BAD, SimpleNamespace(name="other"))
|
||||
vaults.flag_bad_key(KID, BAD)
|
||||
flag_and_replace(SimpleNamespace(name="other", local=False, report_now=False))
|
||||
assert calls == []
|
||||
vaults.sources[KID] = (BAD, api)
|
||||
|
||||
monkeypatch.setattr(api.session, "post", lambda url, **kw: (_ for _ in ()).throw(ConnectionError("down")))
|
||||
caplog.clear()
|
||||
with caplog.at_level(logging.DEBUG, logger="unshackle.core.vaults"):
|
||||
vaults.flag_bad_key(KID, BAD)
|
||||
flag_and_replace(api)
|
||||
assert [r.levelname for r in caplog.records] == ["DEBUG"]
|
||||
|
||||
gone = SimpleNamespace(status_code=404, raise_for_status=lambda: (_ for _ in ()).throw(RuntimeError("404")))
|
||||
monkeypatch.setattr(api.session, "post", lambda url, **kw: gone)
|
||||
caplog.clear()
|
||||
vaults.flag_bad_key(KID, BAD)
|
||||
flag_and_replace(api)
|
||||
assert not caplog.records
|
||||
|
||||
denied = SimpleNamespace(status_code=200, raise_for_status=lambda: None, json=lambda: {"code": 1, "message": "no"})
|
||||
monkeypatch.setattr(api.session, "post", lambda url, **kw: denied)
|
||||
monkeypatch.setattr(api.session, "delete", lambda url, **kw: denied)
|
||||
vaults.flag_bad_key(KID, BAD)
|
||||
flag_and_replace(api)
|
||||
vaults.unflag_bad_key(KID, BAD)
|
||||
assert not caplog.records
|
||||
|
||||
|
||||
@@ -39,6 +39,9 @@ def fake_server_vaults(pushes: list, vault_list: list[FakeVault]) -> Any:
|
||||
def __len__(self) -> int:
|
||||
return len(vault_list)
|
||||
|
||||
def clear_licensed_flags(self, content_keys: dict) -> list:
|
||||
return []
|
||||
|
||||
def add_keys(self, key_map: dict) -> int:
|
||||
pushes.append(key_map)
|
||||
return sum(vault.ok for vault in vault_list)
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
"""A segment that fails every attempt reports the attempts and the cause, without the signed URL."""
|
||||
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from unshackle.commands.dl import dl
|
||||
from unshackle.core.downloaders.requests import download, retries_exhausted
|
||||
from unshackle.core.session import MaxRetriesError, RnetSession, failure_reason
|
||||
|
||||
downloader = sys.modules[download.__module__]
|
||||
|
||||
SIGNED_URL = "https://cdn.example.invalid/dvt2=exp=1~hmac=deadbeef/seg/40_000.mp4a"
|
||||
RNET_BROKEN_PIPE = (
|
||||
f'is_request error: wreq::Error {{ kind: Request, url: "{SIGNED_URL}", source: hyper::Error(Io, '
|
||||
'Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" }) }'
|
||||
)
|
||||
|
||||
|
||||
class RequestError(Exception):
|
||||
"""Stands in for rnet.RequestError, which carries the URL in its text."""
|
||||
|
||||
|
||||
def test_failure_reason_keeps_the_rnet_error_text_and_drops_the_url():
|
||||
reason = failure_reason(MaxRetriesError("m", cause=RequestError(RNET_BROKEN_PIPE)))
|
||||
|
||||
assert reason == "RequestError (stream closed because of a broken pipe)"
|
||||
|
||||
|
||||
def test_spent_retries_report_one_line_without_the_url(monkeypatch, tmp_path):
|
||||
session = RnetSession()
|
||||
calls = []
|
||||
|
||||
def get(url, **kwargs):
|
||||
calls.append(kwargs.get("max_retries"))
|
||||
raise MaxRetriesError("GET cdn.example.invalid failed: x", cause=RequestError(RNET_BROKEN_PIPE))
|
||||
|
||||
monkeypatch.setattr(session, "get", get)
|
||||
monkeypatch.setattr(downloader, "retry_sleep", lambda exc, attempts: 0)
|
||||
|
||||
with pytest.raises(MaxRetriesError) as raised:
|
||||
for _ in download(url=SIGNED_URL, save_path=tmp_path / "seg", session=session):
|
||||
pass
|
||||
|
||||
assert calls == [0] * downloader.MAX_ATTEMPTS
|
||||
assert retries_exhausted(raised.value)
|
||||
assert not retries_exhausted(ValueError("boom"))
|
||||
header, line = dl.failure_lines(raised.value)
|
||||
assert f"gave up after {downloader.MAX_ATTEMPTS} failed attempts" in line
|
||||
assert "hmac" not in line
|
||||
@@ -389,7 +389,7 @@ def test_a_lent_device_session_keeps_its_auth_headers_and_track_secrets() -> Non
|
||||
headers, cookies = handlers.client_session_auth(session)
|
||||
|
||||
assert headers == {"User-Agent": "ua"}
|
||||
assert cookies == {"account": "client"}
|
||||
assert handlers.cookie_values(cookies) == {"account": "client"}
|
||||
assert handlers.scrub_track_data({"token": "server", "id": 1}, session) == {"id": 1}
|
||||
|
||||
|
||||
|
||||
@@ -115,9 +115,7 @@ def test_build_error_response_debug_mode_includes_traceback() -> None:
|
||||
(Exception("Not available in your region"), APIErrorCode.GEOFENCE),
|
||||
(Exception("Service is geoblocked in GB"), APIErrorCode.GEOFENCE),
|
||||
(
|
||||
Exception(
|
||||
"Service is not available in your region (GB). Pass --proxy with a proxy outside the blocked regions."
|
||||
),
|
||||
Exception("Service is not available in your region (GB)"),
|
||||
APIErrorCode.GEOFENCE,
|
||||
),
|
||||
(Exception("Title not found"), APIErrorCode.NOT_FOUND),
|
||||
|
||||
@@ -43,11 +43,12 @@ def test_client_marks_vault_keys_and_forgets_cdm_keys():
|
||||
assert svc.server_vault_keys == {}
|
||||
|
||||
|
||||
def test_report_bad_posts_only_the_pair():
|
||||
def test_server_vault_flag_posts_only_the_pair():
|
||||
svc = remote_service()
|
||||
posted: list = []
|
||||
svc.client = SimpleNamespace(post_optional=lambda ep, data: posted.append((ep, data)) or {})
|
||||
svc.server_vault.report_bad(KID, "v")
|
||||
assert svc.server_vault.report_now
|
||||
svc.server_vault.flag_bad_key("SVC", KID, "v", "server_cdm")
|
||||
assert posted == [("/api/session/sess/keys/bad", {"kid": KID.hex, "key": "v"})]
|
||||
|
||||
|
||||
@@ -77,18 +78,27 @@ def session(monkeypatch):
|
||||
return entry, vault
|
||||
|
||||
|
||||
async def test_server_flags_a_served_pair_with_its_own_vault_name(session):
|
||||
async def test_server_flags_a_served_pair_with_its_own_vault_name(session, monkeypatch):
|
||||
entry, vault = session
|
||||
monkeypatch.setattr(handlers, "CACHED_PAIRS", {("SimpleNamespace", KID.hex, "v")})
|
||||
resp = await handlers.session_bad_key_handler({"kid": KID.hex, "key": "v"}, "sess")
|
||||
assert resp.status == 200
|
||||
assert vault.flagged == [("SVC", KID, "v", "sqlite")]
|
||||
assert KID.hex not in entry.served_keys
|
||||
assert not handlers.CACHED_PAIRS
|
||||
|
||||
with pytest.raises(APIError):
|
||||
await handlers.session_bad_key_handler({"kid": CDM_KID.hex, "key": "nope"}, "sess")
|
||||
assert len(vault.flagged) == 1
|
||||
|
||||
|
||||
async def test_server_does_not_flag_a_pair_its_cdm_licensed(session):
|
||||
entry, vault = session
|
||||
resp = await handlers.session_bad_key_handler({"kid": CDM_KID.hex, "key": "c"}, "sess")
|
||||
assert json.loads(resp.body) == {"flagged": False}
|
||||
assert vault.flagged == []
|
||||
|
||||
|
||||
class _WV:
|
||||
def __init__(self, b64, kids):
|
||||
self._pssh = SimpleNamespace(dumps=lambda: b64)
|
||||
@@ -218,3 +228,20 @@ def test_reconstruct_drm_keeps_the_default_kid_of_a_pssh_without_kids():
|
||||
(drm,) = reconstruct_drm([{"type": "widevine", "pssh": content_id_only.dumps(), "kids": [str(KID)]}])
|
||||
|
||||
assert real_kids(drm) == [KID]
|
||||
|
||||
|
||||
def test_a_cdm_licence_unflags_a_pair_a_client_reported(monkeypatch, tmp_path):
|
||||
"""The CDM returned the reported pair, so the client's decode check was wrong."""
|
||||
from unshackle.core.vaults import Vaults
|
||||
from unshackle.vaults.SQLite import SQLite
|
||||
|
||||
local = SQLite("local", tmp_path / "local.db")
|
||||
vaults = Vaults("SVC")
|
||||
vaults.vaults = [local]
|
||||
vaults.flag_bad_key(KID, "ab" * 16)
|
||||
monkeypatch.setattr(handlers, "load_server_vaults", lambda service: vaults)
|
||||
monkeypatch.setattr(handlers, "CACHED_PAIRS", set())
|
||||
|
||||
handlers.cache_to_vaults({KID.hex: "ab" * 16}, "SVC")
|
||||
assert not local.is_bad_key(KID, "ab" * 16)
|
||||
assert local.get_key(KID, "SVC") == "ab" * 16
|
||||
|
||||
@@ -42,6 +42,9 @@ class FakeVaults:
|
||||
def is_flagged(self, kid, key):
|
||||
return False
|
||||
|
||||
def clear_licensed_flags(self, content_keys):
|
||||
return []
|
||||
|
||||
def add_keys(self, kid_keys):
|
||||
self.added.update(kid_keys)
|
||||
self.pushes.append(dict(kid_keys))
|
||||
@@ -227,3 +230,21 @@ def test_cdm_only_trusts_server_vault_keys():
|
||||
|
||||
assert vaults.sources == {}
|
||||
assert dl.LICENSE_KEY_CACHE == {CACHED: "k_cached"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("system", ["widevine", "playready"])
|
||||
def test_no_cdm_and_no_server_cdm_names_the_missing_cdm(system):
|
||||
"""The server does not license this service for the API key, and the client has no CDM for it."""
|
||||
import click
|
||||
|
||||
from unshackle.core.drm import PlayReady, Widevine
|
||||
|
||||
cmd = make_cmd(FakeVaults())
|
||||
cmd._remote_service = SimpleNamespace(_server_cdm=False)
|
||||
cmd.cdm = None
|
||||
cmd.cdm_for = lambda system, quality=None: None
|
||||
drm_class = PlayReady if system == "playready" else Widevine
|
||||
drm = drm_class.__new__(drm_class)
|
||||
drm.content_keys = {}
|
||||
with pytest.raises(click.ClickException, match="No CDM configured for TEST"):
|
||||
prepare(cmd, drm, lambda **kwargs: None, NEW)
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from unshackle.core.api.errors import APIError
|
||||
from unshackle.core.api.handlers import client_session_auth, resolve_handler_proxy, scrub_track_data
|
||||
from unshackle.core.remote_service import apply_session_cookies
|
||||
|
||||
|
||||
def _session(client_auth):
|
||||
cookie = SimpleNamespace(name="sid", value="secret")
|
||||
cookie = SimpleNamespace(name="sid", value="secret", domain=".example.com", path="/")
|
||||
svc = SimpleNamespace(headers={"Authorization": "Bearer x", "User-Agent": "ua"}, cookies=[cookie])
|
||||
return SimpleNamespace(service_instance=SimpleNamespace(session=svc), client_auth=client_auth)
|
||||
|
||||
@@ -15,14 +17,129 @@ def _session(client_auth):
|
||||
def test_client_login_keeps_auth():
|
||||
headers, cookies = client_session_auth(_session(True))
|
||||
assert headers == {"Authorization": "Bearer x", "User-Agent": "ua"}
|
||||
assert cookies == {"sid": "secret"}
|
||||
assert cookies == [{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/", "secure": False}]
|
||||
assert scrub_track_data({"token": "t", "hls": {}}, _session(True)) == {"token": "t", "hls": {}}
|
||||
|
||||
|
||||
def test_client_cookies_keep_their_domain():
|
||||
session = requests.Session()
|
||||
_, cookies = client_session_auth(_session(True))
|
||||
apply_session_cookies(session, {"session_cookie_jar": cookies})
|
||||
assert session.cookies.get("sid", domain=".example.com") == "secret"
|
||||
cdn = session.prepare_request(requests.Request("GET", "https://cdn.other.net/seg.mp4"))
|
||||
assert "Cookie" not in cdn.headers
|
||||
own = session.prepare_request(requests.Request("GET", "https://www.example.com/api"))
|
||||
assert own.headers["Cookie"] == "sid=secret"
|
||||
|
||||
|
||||
def test_client_takes_old_server_cookies_without_domain():
|
||||
session = requests.Session()
|
||||
apply_session_cookies(session, {"session_cookies": {"sid": "secret"}})
|
||||
assert session.cookies.get("sid") == "secret"
|
||||
|
||||
|
||||
def test_client_keeps_secure_cookies_off_http():
|
||||
session = requests.Session()
|
||||
jar = [{"name": "sid", "value": "s", "domain": ".example.com", "path": "/", "secure": True}]
|
||||
apply_session_cookies(session, {"session_cookie_jar": jar})
|
||||
plain = session.prepare_request(requests.Request("GET", "http://cdn.example.com/seg.mp4"))
|
||||
assert "Cookie" not in plain.headers
|
||||
tls = session.prepare_request(requests.Request("GET", "https://cdn.example.com/seg.mp4"))
|
||||
assert tls.headers["Cookie"] == "sid=s"
|
||||
|
||||
|
||||
def test_client_skips_malformed_server_cookies():
|
||||
session = requests.Session()
|
||||
jar = [{"value": "v"}, "str", {"name": "n", "value": 1}, {"name": "ok", "value": "v", "domain": ".example.com"}]
|
||||
apply_session_cookies(session, {"session_cookie_jar": jar})
|
||||
assert [c.name for c in session.cookies] == ["ok"]
|
||||
session = requests.Session()
|
||||
apply_session_cookies(session, {"session_cookie_jar": 5, "session_cookies": ["x"]})
|
||||
assert not session.cookies
|
||||
|
||||
|
||||
def test_rnet_session_sends_client_and_set_cookies():
|
||||
from unshackle.core.session import RnetSession
|
||||
|
||||
svc = RnetSession()
|
||||
svc.cookies.set("sid", "stale", domain=".example.com")
|
||||
svc.cookies.set("cdn", "c", domain="cdn.other.net")
|
||||
svc.export_origin_cookies = lambda: {"https://www.example.com": "sid=live; cf=1"}
|
||||
_, cookies = client_session_auth(SimpleNamespace(service_instance=SimpleNamespace(session=svc), client_auth=True))
|
||||
assert {(c["name"], c["value"], c["domain"]) for c in cookies} == {
|
||||
("sid", "live", "www.example.com"),
|
||||
("cf", "1", "www.example.com"),
|
||||
("cdn", "c", "cdn.other.net"),
|
||||
}
|
||||
|
||||
|
||||
def test_tracks_response_carries_the_cookie_jar():
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from unshackle.core.api import session_store as store_mod
|
||||
from unshackle.core.api.handlers import session_tracks_handler
|
||||
from unshackle.core.tracks import Tracks
|
||||
|
||||
cookie = SimpleNamespace(name="sid", value="secret", domain=".example.com", path="/", secure=True)
|
||||
|
||||
class StubService:
|
||||
session = SimpleNamespace(headers={}, cookies=[cookie])
|
||||
|
||||
def get_tracks(self, title):
|
||||
return Tracks()
|
||||
|
||||
def get_chapters(self, title):
|
||||
raise NotImplementedError
|
||||
|
||||
async def run():
|
||||
store = store_mod.get_session_store()
|
||||
entry = await store.create(
|
||||
service_tag="EXAMPLE", service_instance=StubService(), session_id="jar", owner_key=None
|
||||
)
|
||||
entry.title_map = {"t1": "Example Title"}
|
||||
entry.client_auth = True
|
||||
try:
|
||||
return json.loads((await session_tracks_handler({"title_id": "t1"}, "jar")).body)
|
||||
finally:
|
||||
await store.delete("jar")
|
||||
|
||||
body = asyncio.run(run())
|
||||
assert body["session_cookies"] == {"sid": "secret"}
|
||||
assert body["session_cookie_jar"] == [
|
||||
{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/", "secure": True}
|
||||
]
|
||||
|
||||
|
||||
def test_remote_get_tracks_scopes_server_cookies():
|
||||
import logging
|
||||
|
||||
from unshackle.core.remote_service import RemoteService
|
||||
|
||||
jar = [{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/"}]
|
||||
service = object.__new__(RemoteService)
|
||||
service.__dict__.update(
|
||||
_tracks_by_title={},
|
||||
_chapters_by_title={},
|
||||
_sample_decrypt_tracks=set(),
|
||||
_server_cdm=None,
|
||||
_session=requests.Session(),
|
||||
_session_id="s",
|
||||
service_tag="EXAMPLE",
|
||||
log=logging.getLogger("test"),
|
||||
client=SimpleNamespace(post=lambda path, body: {"session_cookie_jar": jar}),
|
||||
drain_server_logs=lambda: None,
|
||||
)
|
||||
service.get_tracks(SimpleNamespace(id="t1"))
|
||||
cdn = service._session.prepare_request(requests.Request("GET", "https://cdn.other.net/seg.mp4"))
|
||||
assert "Cookie" not in cdn.headers
|
||||
assert service._session.cookies.get("sid", domain=".example.com") == "secret"
|
||||
|
||||
|
||||
def test_server_login_strips_auth():
|
||||
headers, cookies = client_session_auth(_session(False))
|
||||
assert headers == {"User-Agent": "ua"}
|
||||
assert cookies == {}
|
||||
assert cookies == []
|
||||
assert scrub_track_data({"token": "t", "hls": {}}, _session(False)) == {"hls": {}}
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Self-check for ``has_encrypted_sample_entry``: a finalized MP4 whose moov still carries
|
||||
an encv/enca sample entry means the decrypt tool skipped/failed the track (frma was never
|
||||
an encrypted (enc*/drm*) sample entry means the decrypt tool skipped/failed the track (frma was never
|
||||
applied), so the detector must flag it before the file reaches the muxer. Detection is
|
||||
scoped to moov, so stray 4CC byte runs in mdat must never trip (or mask) it.
|
||||
|
||||
@@ -11,6 +11,8 @@ from __future__ import annotations
|
||||
import struct
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from unshackle.core.tracks.track import has_encrypted_sample_entry
|
||||
|
||||
|
||||
@@ -80,6 +82,12 @@ def test_text_containing_enca_in_moov_not_detected(tmp_path: Path) -> None:
|
||||
assert has_encrypted_sample_entry(f) is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("kind", [b"encs", b"enct", b"drmi"])
|
||||
def test_other_encrypted_entry_kinds_detected(tmp_path: Path, kind: bytes) -> None:
|
||||
f = write(tmp_path, "enc.mp4", FTYP, moov(stsd([sample_entry(kind, box(b"sinf", box(b"frma", b"stpp")))])))
|
||||
assert has_encrypted_sample_entry(f) is True
|
||||
|
||||
|
||||
def test_missing_or_bogus_file(tmp_path: Path) -> None:
|
||||
assert has_encrypted_sample_entry(tmp_path / "nope.mp4") is False
|
||||
bogus = write(tmp_path, "bogus.bin", b"\x00\x01\x02\x03not-an-mp4")
|
||||
|
||||
@@ -4,8 +4,9 @@ import sys
|
||||
|
||||
import click
|
||||
from ruamel.yaml import YAML
|
||||
from ruamel.yaml.constructor import DuplicateKeyError
|
||||
|
||||
from unshackle.core.config import config, get_config_path
|
||||
from unshackle.core.config import config, duplicate_key_lines, get_config_path, load_yaml
|
||||
from unshackle.core.constants import context_settings
|
||||
|
||||
|
||||
@@ -41,7 +42,16 @@ def cfg(ctx: click.Context, key: str, value: str, unset: bool, list_: bool) -> N
|
||||
|
||||
config_path = get_config_path() or config.directories.user_configs / config.filenames.root_config
|
||||
if config_path.exists():
|
||||
data = yaml.load(config_path)
|
||||
try:
|
||||
data = yaml.load(config_path)
|
||||
except DuplicateKeyError as e:
|
||||
if duplicate_key_lines(yaml.compose(config_path)):
|
||||
load_yaml(config_path)
|
||||
else:
|
||||
log.warning(
|
||||
f"{config_path}: duplicate key on line {e.problem_mark.line + 1}. Unshackle uses the later one."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
if not data:
|
||||
log.warning("No config file was found or it has no data, yet")
|
||||
|
||||
+107
-140
@@ -27,7 +27,6 @@ from uuid import UUID
|
||||
|
||||
import click
|
||||
import mediaexport
|
||||
import yaml
|
||||
from click.core import ParameterSource
|
||||
from langcodes import Language, tag_is_valid
|
||||
from pymediainfo import MediaInfo
|
||||
@@ -43,7 +42,7 @@ from rich.tree import Tree
|
||||
from unshackle.core import __version__, binaries, providers
|
||||
from unshackle.core.cdm import DecryptLabsRemoteCDM
|
||||
from unshackle.core.cdm.detect import cdm_type_stub, is_playready_cdm, is_remote_cdm, is_widevine_cdm
|
||||
from unshackle.core.config import cdm_entry_names_device, config, resolve_cdm_name, resolve_decryption
|
||||
from unshackle.core.config import cdm_entry_names_device, config, load_yaml, resolve_cdm_name, resolve_decryption
|
||||
from unshackle.core.console import GradientPulseBarColumn, SyncLive, console, listing_panel
|
||||
from unshackle.core.constants import (
|
||||
DEFAULT_SAMPLE,
|
||||
@@ -54,7 +53,13 @@ from unshackle.core.constants import (
|
||||
context_settings,
|
||||
)
|
||||
from unshackle.core.credential import Credential
|
||||
from unshackle.core.downloaders import default_max_workers, format_speed, parse_speed_limit, set_speed_limit
|
||||
from unshackle.core.downloaders import (
|
||||
default_max_workers,
|
||||
format_speed,
|
||||
parse_speed_limit,
|
||||
retries_exhausted,
|
||||
set_speed_limit,
|
||||
)
|
||||
from unshackle.core.drm import DRM_T, ClearKeyCENC, MonaLisa, PlayReady, Widevine, cbcs, own_kids, real_kids, verify
|
||||
from unshackle.core.events import events
|
||||
from unshackle.core.export_name import ExportNamer, move_export, plural, season_episodes, title_label
|
||||
@@ -176,10 +181,7 @@ def validate_metadata_ids(
|
||||
(flag, value) for flag, value in (("--tmdb", tmdb_id), ("--imdb", imdb_id), ("--tvdb", tvdb_id)) if value
|
||||
]
|
||||
if len(supplied) > 1:
|
||||
raise click.UsageError(
|
||||
f"{', '.join(flag for flag, _ in supplied)} cannot be used together. "
|
||||
"Give one ID and unshackle resolves the others from it."
|
||||
)
|
||||
raise click.UsageError(f"{', '.join(flag for flag, _ in supplied)} cannot be used together; give one ID")
|
||||
if not supplied:
|
||||
return
|
||||
|
||||
@@ -192,9 +194,7 @@ def validate_metadata_ids(
|
||||
|
||||
if consumers:
|
||||
keys = sorted({ID_PROVIDER_KEYS[cls.NAME] for cls in consumers if cls.NAME in ID_PROVIDER_KEYS})
|
||||
raise click.UsageError(
|
||||
f"{flag} needs the {consumers[0].NAME} provider. Set {' or '.join(keys)} in your config."
|
||||
)
|
||||
raise click.UsageError(f"{flag} needs the {consumers[0].NAME} provider: set {' or '.join(keys)} in your config")
|
||||
names = sorted({cls.NAME for cls in providers.REGISTRY.values() if cls.ID_KIND == id_kind})
|
||||
raise click.UsageError(
|
||||
f"{flag} needs the {' or '.join(names)} provider, which your metadata_providers config leaves out."
|
||||
@@ -236,39 +236,28 @@ def select_hybrid_bases(videos: Sequence[Video]) -> list[Video]:
|
||||
return list(bases.values())
|
||||
|
||||
|
||||
def hybrid_gate_failure(videos: Sequence[Video], generate_dv: bool) -> Optional[tuple[str, str]]:
|
||||
"""The (message, detail) that stops HYBRID for these selected videos, or None when it can go on."""
|
||||
def hybrid_gate_failure(videos: Sequence[Video], generate_dv: bool) -> Optional[str]:
|
||||
"""The message that stops HYBRID for these selected videos, or None when it can go on."""
|
||||
if any(v.native_hybrid for v in videos):
|
||||
return None
|
||||
hdr_tracks = [v for v in videos if v.range in (Video.Range.HDR10, Video.Range.HDR10P)]
|
||||
base_tracks = [v for v in hdr_tracks if v.codec in (Video.Codec.HEVC, None)]
|
||||
dv_tracks = [v for v in videos if v.range == Video.Range.DV]
|
||||
has_hdr10p = any(v.range == Video.Range.HDR10P for v in base_tracks)
|
||||
available_ranges = sorted(set(v.range.name for v in videos))
|
||||
ranges = ", ".join(sorted(set(v.range.name for v in videos))) or "none"
|
||||
missing = f"HYBRID needs HDR10/HDR10+ and DV tracks, available ranges: {ranges}"
|
||||
|
||||
if not hdr_tracks and not dv_tracks:
|
||||
return (
|
||||
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but neither is available",
|
||||
f"Available ranges: {', '.join(available_ranges) if available_ranges else 'none'}",
|
||||
)
|
||||
if not hdr_tracks:
|
||||
return (
|
||||
"HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only DV is available",
|
||||
f"Available ranges: {', '.join(available_ranges)}",
|
||||
)
|
||||
return missing
|
||||
if not base_tracks:
|
||||
codecs = sorted(set(v.codec.name for v in hdr_tracks if v.codec))
|
||||
return (
|
||||
"HYBRID mode requires an HEVC HDR10/HDR10+ base, but none is available",
|
||||
f"Available HDR10/HDR10+ codecs: {', '.join(codecs)}",
|
||||
)
|
||||
return f"HYBRID needs an HEVC HDR10/HDR10+ base, available codecs: {', '.join(codecs)}"
|
||||
if not dv_tracks and not (generate_dv and has_hdr10p):
|
||||
msg_detail = f"Available ranges: {', '.join(available_ranges)}"
|
||||
if generate_dv:
|
||||
msg_detail += ". --generate-dv needs an HEVC HDR10+ track, and the service returned none."
|
||||
elif has_hdr10p:
|
||||
msg_detail += ". Add --generate-dv to generate Dolby Vision from HDR10+."
|
||||
return "HYBRID mode requires both HDR10/HDR10+ and DV tracks, but only HDR10 is available", msg_detail
|
||||
return f"{missing}; --generate-dv found no HEVC HDR10+ track"
|
||||
if has_hdr10p:
|
||||
return f"{missing}; add --generate-dv to make DV from HDR10+"
|
||||
return missing
|
||||
return None
|
||||
|
||||
|
||||
@@ -486,10 +475,7 @@ def apply_service_dl_overrides(ctx: click.Context, service_dl_config: dict[str,
|
||||
ctx.params[name] = param.type_cast_value(ctx, value)
|
||||
log.debug(f"Applied service dl override '{name}': {ctx.params[name]}")
|
||||
except Exception as e:
|
||||
log.warning(
|
||||
f"Failed to apply service dl override '{name}'={value!r}: {e}. "
|
||||
f"Check that the value is valid for this parameter."
|
||||
)
|
||||
log.warning(f"Failed to apply service dl override '{name}'={value!r}: {e}")
|
||||
|
||||
|
||||
def download_tracks_in_passes(
|
||||
@@ -1278,16 +1264,12 @@ class dl:
|
||||
# report which weren't available without parsing the console output. See SkippedSubtitle.
|
||||
self.skipped_subtitles: list[SkippedSubtitle] = []
|
||||
# result() catches a download-worker failure, reports it, and returns rather than
|
||||
# re-raising (so the CLI still exits cleanly). Flag it so an embedding caller (the API
|
||||
# re-raising (so the CLI still exits cleanly). Keep it so an embedding caller (the API
|
||||
# worker) can tell the title did not complete instead of reading it as a success.
|
||||
self.download_failed: bool = False
|
||||
self.download_error: Optional[BaseException] = None
|
||||
|
||||
if not config.output_template:
|
||||
raise click.ClickException(
|
||||
"No 'output_template' configured in your unshackle.yaml.\n"
|
||||
"Please add an 'output_template' section with movies, series, and songs/music templates.\n"
|
||||
"See unshackle-example.yaml for examples."
|
||||
)
|
||||
raise click.ClickException("No 'output_template' in your unshackle.yaml")
|
||||
|
||||
self.service = Services.get_tag(ctx.invoked_subcommand)
|
||||
self.vault_service = Services.get_vault_tag(self.service)
|
||||
@@ -1330,7 +1312,7 @@ class dl:
|
||||
)
|
||||
|
||||
if self.daily and not self.enrich:
|
||||
self.log.warning("--daily fills air dates from TVDB only with --enrich. A service can set them itself.")
|
||||
self.log.warning("--daily takes TVDB air dates only with --enrich")
|
||||
|
||||
# Initialize debug logger with service name if debug logging is enabled
|
||||
if config.debug or logging.root.level == logging.DEBUG:
|
||||
@@ -1454,7 +1436,7 @@ class dl:
|
||||
try:
|
||||
service_config_path = Services.get_path(self.service) / config.filenames.config
|
||||
if service_config_path.exists():
|
||||
self.service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
|
||||
self.service_config = load_yaml(service_config_path)
|
||||
self.log.info("Service Config loaded")
|
||||
# log key names only: the full config carries service certificates,
|
||||
# device fingerprints and endpoints that bloat the log and may be sensitive
|
||||
@@ -1795,7 +1777,7 @@ class dl:
|
||||
("-fsl/--forced-s-lang", fsl_excl),
|
||||
):
|
||||
if any(token.lower() == "all" for token in excludes):
|
||||
self.log.error(f"{flag}: 'all' cannot be excluded. Name the languages to exclude instead.")
|
||||
self.log.error(f"{flag}: 'all' cannot be excluded")
|
||||
sys.exit(1)
|
||||
# exclusion-only use falls back to the flag's own default; -vl/-al have none, they cascade to -l
|
||||
if lang_excl and not lang:
|
||||
@@ -1905,7 +1887,7 @@ class dl:
|
||||
if vbitrate_min > vbitrate_max:
|
||||
vbitrate_min, vbitrate_max = vbitrate_max, vbitrate_min
|
||||
except (ValueError, IndexError):
|
||||
self.log.error("Invalid --vbitrate-range format. Use 'MIN-MAX' (e.g., '6000-7000').")
|
||||
self.log.error("Invalid --vbitrate-range: expected MIN-MAX, such as 6000-7000")
|
||||
sys.exit(1)
|
||||
|
||||
abitrate_min, abitrate_max = None, None
|
||||
@@ -1921,7 +1903,7 @@ class dl:
|
||||
if abitrate_min > abitrate_max:
|
||||
abitrate_min, abitrate_max = abitrate_max, abitrate_min
|
||||
except (ValueError, IndexError):
|
||||
self.log.error("Invalid --abitrate-range format. Use 'MIN-MAX' (e.g., '128-256').")
|
||||
self.log.error("Invalid --abitrate-range: expected MIN-MAX, such as 128-256")
|
||||
sys.exit(1)
|
||||
|
||||
if not acodec:
|
||||
@@ -1945,15 +1927,13 @@ class dl:
|
||||
from unshackle.core.binaries import DoviTool
|
||||
|
||||
if not DoviTool:
|
||||
self.log.error("Unable to run hybrid mode: dovi_tool not detected")
|
||||
self.log.error("Please install dovi_tool from https://github.com/quietvoid/dovi_tool")
|
||||
self.log.error("Unable to run hybrid mode: dovi_tool not found")
|
||||
sys.exit(1)
|
||||
if generate_dv:
|
||||
from unshackle.core.binaries import HDR10PlusTool
|
||||
|
||||
if not HDR10PlusTool:
|
||||
self.log.error("Unable to run --generate-dv: hdr10plus_tool not detected")
|
||||
self.log.error("Please install hdr10plus_tool from https://github.com/quietvoid/hdr10plus_tool")
|
||||
self.log.error("Unable to run --generate-dv: hdr10plus_tool not found")
|
||||
sys.exit(1)
|
||||
|
||||
if cdm_only is None:
|
||||
@@ -2072,13 +2052,13 @@ class dl:
|
||||
if enrich_lang:
|
||||
self.log.info(f"Original language from metadata: {enrich_lang}")
|
||||
if not (enrich_title or enrich_year or enrich_lang):
|
||||
self.log.warning("--enrich found no metadata. Is the provider's API key configured?")
|
||||
self.log.warning("--enrich found no metadata")
|
||||
elif missing := [
|
||||
name
|
||||
for name, value in (("title", enrich_title), ("year", enrich_year), ("language", enrich_lang))
|
||||
if not value
|
||||
]:
|
||||
self.log.warning(f"--enrich source gave no {', '.join(missing)}. Those fields are unchanged.")
|
||||
self.log.warning(f"--enrich source gave no {', '.join(missing)}, kept the service's values")
|
||||
|
||||
if isinstance(titles, (Series, Movies)):
|
||||
for t in titles:
|
||||
@@ -2863,12 +2843,10 @@ class dl:
|
||||
self.log.info("No Dolby Vision track: Dolby Vision will be generated from HDR10+")
|
||||
|
||||
if gate_failure:
|
||||
msg, msg_detail = gate_failure
|
||||
other_ranges = [r for r in range_ if r != Video.Range.HYBRID]
|
||||
if best_available and other_ranges:
|
||||
self.log.warning(msg)
|
||||
self.log.warning(
|
||||
f"Continuing with remaining range(s): {', '.join(r.name for r in other_ranges)}"
|
||||
f"{gate_failure}; continuing with {', '.join(r.name for r in other_ranges)}"
|
||||
)
|
||||
range_ = other_ranges
|
||||
fallback_pool = pre_hybrid_videos
|
||||
@@ -2897,8 +2875,7 @@ class dl:
|
||||
fallback_selected.append(match)
|
||||
title.tracks.videos = fallback_selected
|
||||
else:
|
||||
self.log.error(msg)
|
||||
self.log.error(msg_detail)
|
||||
self.log.error(gate_failure)
|
||||
sys.exit(1)
|
||||
|
||||
fsl = resolve_sort_langs(forced_s_lang, title.language, exact=exact_lang)
|
||||
@@ -3193,17 +3170,14 @@ class dl:
|
||||
system_cdms = [self.cdm_for(system.lower(), height) for height in heights]
|
||||
if not all(system_cdm and matches(system_cdm) for system_cdm in system_cdms):
|
||||
hint = (
|
||||
"--cdm selects one device, so remove it."
|
||||
"drop --cdm, which selects one device"
|
||||
if self.cdm_override
|
||||
else f"Set cdm.{self.service}.{system.lower()} in the configuration."
|
||||
else f"set cdm.{self.service}.{system.lower()}"
|
||||
)
|
||||
raise click.ClickException(f"--all-drm needs a {system} CDM for {self.service}. {hint}")
|
||||
raise click.ClickException(f"--all-drm needs a {system} CDM for {self.service}: {hint}")
|
||||
if is_remote_cdm(system_cdms[0]) and system not in self.all_drm_warned:
|
||||
self.all_drm_warned.add(system)
|
||||
self.log.warning(
|
||||
f"The {system} CDM is remote. It can answer from its own key cache "
|
||||
"and send no licence request."
|
||||
)
|
||||
self.log.warning(f"The {system} CDM is remote and can answer with cached content keys")
|
||||
|
||||
if hasattr(service, "resolve_server_keys"):
|
||||
service.resolve_server_keys(title)
|
||||
@@ -3310,11 +3284,11 @@ class dl:
|
||||
self.log_export_summary()
|
||||
return
|
||||
except Exception as e: # noqa
|
||||
# Reported and swallowed (no re-raise) so the CLI exits cleanly; flag it so the
|
||||
# Reported and swallowed (no re-raise) so the CLI exits cleanly; keep it so the
|
||||
# API worker sees the title failed rather than completing with no output.
|
||||
self.download_failed = True
|
||||
self.download_error = e
|
||||
console.print(Padding(Group(*self.failure_lines(e)), (1, 5)))
|
||||
if not isinstance(e, click.ClickException):
|
||||
if not isinstance(e, click.ClickException) and not retries_exhausted(e):
|
||||
console.print_exception()
|
||||
|
||||
if self.debug_logger:
|
||||
@@ -3505,10 +3479,7 @@ class dl:
|
||||
has_repacked = True
|
||||
events.emit(events.Types.TRACK_REPACKED, track=track)
|
||||
if has_encrypted_sample_entry(track.path):
|
||||
self.log.warning(
|
||||
f"Track {track.id} still has an encrypted sample entry after repacking, so "
|
||||
"decryption likely failed and the muxed output may be unplayable."
|
||||
)
|
||||
self.log.warning(f"Track {track.id} still has an encrypted sample entry after repack")
|
||||
if has_repacked:
|
||||
self.log.info("Repacked one or more tracks with FFMPEG")
|
||||
|
||||
@@ -4208,9 +4179,7 @@ class dl:
|
||||
collisions = [slot for slot, count in Counter(renumbered).items() if count > 1]
|
||||
if collisions:
|
||||
self.log.error(
|
||||
"TVDB's %s order does not cover %d of this series' episodes, and renumbering the rest "
|
||||
"would give %d season/episode slot(s) two episodes each. Keeping the %s numbering "
|
||||
"%s uses. Pick an order that covers the whole series.",
|
||||
"TVDB's %s order misses %d episode(s) and puts two in %d slot(s), keeping the %s numbering of %s",
|
||||
self.tvdb_order,
|
||||
unmapped,
|
||||
len(collisions),
|
||||
@@ -4406,7 +4375,7 @@ class dl:
|
||||
keys = len(doc.key_pool())
|
||||
self.log.info(f"Saved the export to {namer.path}: {plural(len(doc.titles), 'title')}, {plural(keys, 'key')}")
|
||||
if keys:
|
||||
self.log.warning("The export holds content keys. Keep it private.")
|
||||
self.log.warning("The export holds content keys: keep it private")
|
||||
|
||||
def export_entry(self, title: Title_T) -> mediaexport.Entry:
|
||||
"""A mediaexport entry for ``title``: manifests, chapters, a small track list, no keys yet."""
|
||||
@@ -4480,18 +4449,19 @@ class dl:
|
||||
|
||||
@staticmethod
|
||||
def failure_lines(e: BaseException) -> list[str]:
|
||||
"""Return the lines that report a failed download.
|
||||
"""Return the header and the one line that report a failed download.
|
||||
|
||||
A ClickException is a deliberate stop with a message for the user, so it gets no
|
||||
"unexpected error" line, and the caller prints no traceback for it.
|
||||
A ClickException is a deliberate stop with a message for the user, so it is reported
|
||||
without its type, and the caller prints no traceback for it. Neither does a request
|
||||
that failed on every retry: its line ends with its retry notes.
|
||||
"""
|
||||
if isinstance(e, click.ClickException):
|
||||
return [":x: Download Failed...", f" {e.format_message()}"]
|
||||
lines = [":x: Download Failed...", f" {type(e).__name__}: {e}"]
|
||||
if (returncode := getattr(e, "returncode", None)) is not None:
|
||||
lines.append(f" Binary call failed, Process exit code: {returncode}")
|
||||
lines.append(" An unexpected error occurred in one of the download workers.")
|
||||
return lines
|
||||
line = e.format_message()
|
||||
else:
|
||||
line = f"{type(e).__name__}: {e}"
|
||||
if retries_exhausted(e):
|
||||
line += f", {'; '.join(e.__notes__)}"
|
||||
return [":x: Download Failed...", f" {line}"]
|
||||
|
||||
def decrypt_verified(
|
||||
self,
|
||||
@@ -4508,13 +4478,15 @@ class dl:
|
||||
then copied to the other vaults, once FFmpeg can decode the result. Without FFmpeg
|
||||
neither happens. For a fragmented MP4, ``verify.kid_windows`` reads from the ciphertext
|
||||
which fragments each KID encrypts, and FFmpeg decodes windows inside each KID's
|
||||
fragments, so only a KID that fails is flagged. A KID that is not in the map, and every KID
|
||||
of any other file, gets a start and end check of the whole file, and a failure there flags
|
||||
each of those KIDs. On failure the pair is flagged in the local vaults and reported
|
||||
to the vault that served it, the ciphertext is restored, and the licence runs again. Every vault skips a
|
||||
flagged pair, so each pass burns at most one vault; when the vaults run out the CDM
|
||||
answers, and a CDM key is not checked. A CDM that returns the flagged key clears the
|
||||
flag and stores the content key again, because then the decode check was wrong.
|
||||
fragments, so only a KID that fails is flagged. A complete map does not judge a KID it does
|
||||
not hold, so a clear file judges and copies no KID. A KID that is not in an incomplete map,
|
||||
and every KID of any other file, gets a start and end check of the whole file, and a
|
||||
failure there flags each of those KIDs. On failure the pair is flagged in the local
|
||||
vaults, the ciphertext is restored, and the licence runs again; ``Vaults`` tells the
|
||||
vault that served it. Every vault skips a flagged pair, so each pass burns at most one
|
||||
vault; when the vaults run out the CDM answers, and a CDM key is not checked. A CDM
|
||||
that returns the flagged key clears the flag and stores the content key again, because
|
||||
then the decode check was wrong.
|
||||
|
||||
With ``decrypt=False`` the downloader already decrypted the segments in place, so a
|
||||
failure can only flag and raise; the next run skips the flagged pair.
|
||||
@@ -4575,6 +4547,13 @@ class dl:
|
||||
def window_decodes(start: float, end: float) -> bool:
|
||||
return ffmpeg_decodes(path, start=start, seconds=end - start if end > start else 0.5, video=video)
|
||||
|
||||
def judged_kids() -> dict[UUID, Vault]:
|
||||
"""The vault KIDs this file can judge. A map whose KIDs all have keys skips the KIDs it lacks."""
|
||||
kids = vault_kids()
|
||||
if kid_map is not None and all(kid in keys for kid in windows):
|
||||
return {kid: vault for kid, vault in kids.items() if kid in windows}
|
||||
return kids
|
||||
|
||||
def failed_kids(kids: dict[UUID, Vault]) -> set[UUID]:
|
||||
"""The vault KIDs whose content key did not decode.
|
||||
|
||||
@@ -4605,20 +4584,21 @@ class dl:
|
||||
done += 1
|
||||
if decrypt:
|
||||
drm.decrypt(path)
|
||||
kids = vault_kids()
|
||||
kids = judged_kids()
|
||||
stale = flagged_kids()
|
||||
failed: set[UUID] = set()
|
||||
if not stale:
|
||||
if not kids or not binaries.FFMPEG or not path.exists():
|
||||
return
|
||||
failed = failed_kids(kids)
|
||||
checked = bool(kids and binaries.FFMPEG and path.exists())
|
||||
failed = failed_kids(kids) if checked else set()
|
||||
if not failed:
|
||||
self.flush_vault_writes(
|
||||
[
|
||||
partial(self.vaults.add_key, kid, keys[kid], excluding=vault)
|
||||
for kid, vault in kids.items()
|
||||
]
|
||||
)
|
||||
if checked:
|
||||
self.flush_vault_writes(
|
||||
[
|
||||
partial(self.vaults.add_key, kid, keys[kid], excluding=vault)
|
||||
for kid, vault in kids.items()
|
||||
]
|
||||
)
|
||||
self.vaults.report_replaced(keys)
|
||||
return
|
||||
bad = {kid: drop(kid) for kid in {*failed, *stale}}
|
||||
for kid, key in bad.items():
|
||||
@@ -4626,21 +4606,23 @@ class dl:
|
||||
continue
|
||||
self.log.warning(f"{key} from {kids[kid].name} was bad, trying other vaults")
|
||||
self.vaults.flag_bad_key(kid, key)
|
||||
if report := getattr(kids[kid], "report_bad", None):
|
||||
report(kid, key)
|
||||
if not backup.exists() or not licence:
|
||||
raise ValueError("The content key from the vault did not decrypt the track; run again")
|
||||
path.unlink()
|
||||
os.link(backup, path)
|
||||
relicense(set(bad))
|
||||
for kid, key in bad.items():
|
||||
if keys.get(kid) != key:
|
||||
new = keys.get(kid)
|
||||
if new is None:
|
||||
continue
|
||||
source = self.vaults.sources.get(kid)
|
||||
if source is None:
|
||||
self.vaults.unflag_bad_key(kid, key)
|
||||
self.flush_vault_writes([partial(self.vaults.add_key, kid, key)])
|
||||
elif kid in kids and source[1] is kids[kid]:
|
||||
if new == key:
|
||||
self.vaults.unflag_bad_key(kid, key)
|
||||
self.flush_vault_writes([partial(self.vaults.add_key, kid, key)])
|
||||
elif not self.vaults.held_locally(kid, new):
|
||||
self.flush_vault_writes([partial(self.vaults.add_key, kid, new)])
|
||||
elif new == key and kid in kids and source[1] is kids[kid]:
|
||||
raise ValueError(f"{key} from {kids[kid].name} was bad and no other source has the key")
|
||||
raise ValueError("No vault or CDM produced a content key that decrypts the track")
|
||||
finally:
|
||||
@@ -4729,9 +4711,7 @@ class dl:
|
||||
flag came from a decode check that was wrong, and a local vault refuses a flagged pair."""
|
||||
if not content_keys or not self.vaults:
|
||||
return
|
||||
for kid, key in content_keys.items():
|
||||
if self.vaults.is_flagged(kid, key):
|
||||
self.vaults.unflag_bad_key(kid, key)
|
||||
self.vaults.clear_licensed_flags(content_keys)
|
||||
successful_caches = self.vaults.add_keys(content_keys)
|
||||
kids, caches = self.vault_cache_tally or (set(), successful_caches)
|
||||
self.vault_cache_tally = (kids | content_keys.keys(), min(caches, successful_caches))
|
||||
@@ -4834,8 +4814,7 @@ class dl:
|
||||
if not others and other_type.__name__ not in self.all_drm_warned:
|
||||
self.all_drm_warned.add(other_type.__name__)
|
||||
self.log.warning(
|
||||
f"A track does not offer {other_type.__name__}, so it licenses with "
|
||||
f"{type(drm).__name__} only. This warning shows one time: {track}"
|
||||
f"Track {track} has no {other_type.__name__} DRM, licensing with {type(drm).__name__} only"
|
||||
)
|
||||
|
||||
targets = [drm, *others[:1]]
|
||||
@@ -5035,34 +5014,22 @@ class dl:
|
||||
drm.content_keys.update(local_drm.content_keys)
|
||||
return
|
||||
|
||||
licence = partial(licence, drm_system="playready" if isinstance(drm, PlayReady) else "widevine")
|
||||
drm_system = "playready" if isinstance(drm, PlayReady) else "widevine"
|
||||
licence = partial(licence, drm_system=drm_system)
|
||||
|
||||
if not server_cdm:
|
||||
if isinstance(drm, Widevine):
|
||||
if not is_widevine_cdm(self.cdm):
|
||||
widevine_cdm = self.cdm_for("widevine", track_quality)
|
||||
if widevine_cdm and is_widevine_cdm(widevine_cdm):
|
||||
if track_quality:
|
||||
self.log.info(f"Switching to Widevine CDM for Widevine {track_quality}p content")
|
||||
else:
|
||||
self.log.info("Switching to Widevine CDM for Widevine content")
|
||||
self.cdm = widevine_cdm
|
||||
track_cdm = widevine_cdm
|
||||
else:
|
||||
raise ValueError(f"Title needs a Widevine CDM but {self.service} is configured with PlayReady.")
|
||||
|
||||
elif isinstance(drm, PlayReady):
|
||||
if not is_playready_cdm(self.cdm):
|
||||
playready_cdm = self.cdm_for("playready", track_quality)
|
||||
if playready_cdm and is_playready_cdm(playready_cdm):
|
||||
if track_quality:
|
||||
self.log.info(f"Switching to PlayReady CDM for PlayReady {track_quality}p content")
|
||||
else:
|
||||
self.log.info("Switching to PlayReady CDM for PlayReady content")
|
||||
self.cdm = playready_cdm
|
||||
track_cdm = playready_cdm
|
||||
else:
|
||||
raise ValueError(f"Title needs a PlayReady CDM but {self.service} is configured with Widevine.")
|
||||
if not server_cdm and isinstance(drm, (Widevine, PlayReady)):
|
||||
matches = is_playready_cdm if drm_system == "playready" else is_widevine_cdm
|
||||
if not matches(self.cdm):
|
||||
name, other = ("PlayReady", "Widevine") if drm_system == "playready" else ("Widevine", "PlayReady")
|
||||
system_cdm = self.cdm_for(drm_system, track_quality)
|
||||
if system_cdm and matches(system_cdm):
|
||||
self.log.info(f"Switching to the {name} CDM")
|
||||
self.cdm = system_cdm
|
||||
track_cdm = system_cdm
|
||||
elif self.cdm is None:
|
||||
raise click.ClickException(f"No CDM configured for {self.service} and no default CDM")
|
||||
else:
|
||||
raise ValueError(f"Title needs a {name} CDM but {self.service} is configured with {other}.")
|
||||
|
||||
if isinstance(drm, Widevine):
|
||||
if self.debug_logger:
|
||||
|
||||
@@ -63,9 +63,9 @@ def new(
|
||||
elif len(paths) == 2:
|
||||
group_key, group_certificate = paths
|
||||
if not group_key.is_file():
|
||||
raise click.UsageError("group_key: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("group_key: Not a path to a file, or it does not exist.", ctx)
|
||||
if not group_certificate.is_file():
|
||||
raise click.UsageError("group_certificate: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("group_certificate: Not a path to a file, or it does not exist.", ctx)
|
||||
device_dir = None
|
||||
else:
|
||||
raise click.UsageError(
|
||||
@@ -73,9 +73,9 @@ def new(
|
||||
ctx,
|
||||
)
|
||||
if encryption_key and not encryption_key.is_file():
|
||||
raise click.UsageError("encryption_key: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("encryption_key: Not a path to a file, or it does not exist.", ctx)
|
||||
if signing_key and not signing_key.is_file():
|
||||
raise click.UsageError("signing_key: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("signing_key: Not a path to a file, or it does not exist.", ctx)
|
||||
|
||||
log = logging.getLogger("prd")
|
||||
|
||||
@@ -161,7 +161,7 @@ def reprovision_device(
|
||||
) -> None:
|
||||
"""Reprovision a PlayReady Device (.prd) file."""
|
||||
if not prd_path.is_file():
|
||||
raise click.UsageError("prd_path: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("prd_path: Not a path to a file, or it does not exist.", ctx)
|
||||
|
||||
log = logging.getLogger("prd")
|
||||
log.info("Reprovisioning Playready Device (.prd) file, %s", prd_path.name)
|
||||
@@ -234,7 +234,7 @@ def test(
|
||||
"""Do a test of a PlayReady device on the Microsoft demo server."""
|
||||
|
||||
if not device.is_file():
|
||||
raise click.UsageError("device: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("device: Not a path to a file, or it does not exist.", ctx)
|
||||
|
||||
log = logging.getLogger("prd")
|
||||
|
||||
|
||||
@@ -6,13 +6,12 @@ import sys
|
||||
from typing import Any, Optional
|
||||
|
||||
import click
|
||||
import yaml
|
||||
from rich.padding import Padding
|
||||
from rich.rule import Rule
|
||||
from rich.tree import Tree
|
||||
|
||||
from unshackle.commands.dl import dl
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.config import config, load_yaml
|
||||
from unshackle.core.console import console
|
||||
from unshackle.core.constants import context_settings
|
||||
from unshackle.core.proxies.proxy import Proxy
|
||||
@@ -91,7 +90,7 @@ def search(
|
||||
with console.status("Loading Service Config...", spinner="dots"):
|
||||
service_config_path = Services.get_path(service) / config.filenames.config
|
||||
if service_config_path.exists():
|
||||
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
|
||||
service_config = load_yaml(service_config_path)
|
||||
log.info("Service Config loaded")
|
||||
merge_dict(ci_get(config.services, service), service_config)
|
||||
|
||||
@@ -165,6 +164,4 @@ def result(service: Service, profile: Optional[str] = None, remote: bool = False
|
||||
if search_results.children:
|
||||
console.print(Padding(search_results, (0, 5)))
|
||||
else:
|
||||
console.print(
|
||||
Padding("[bold text]No matches[/]\n[bright_black]Please check spelling and search again....[/]", (0, 5))
|
||||
)
|
||||
console.print(Padding("[bold text]No matches[/]", (0, 5)))
|
||||
|
||||
@@ -135,9 +135,7 @@ def _prepare_serve(serve: dict, no_key: bool) -> Optional[str]:
|
||||
"""
|
||||
api_secret = None if no_key else serve.get("api_secret")
|
||||
if not no_key and not api_secret:
|
||||
raise ValueError(
|
||||
"API secret key is not configured. Please add 'api_secret' to the 'serve' section in your config."
|
||||
)
|
||||
raise ValueError("No 'api_secret' in the 'serve' section of your config")
|
||||
|
||||
for key, directory, pattern in (
|
||||
("devices", config.directories.wvds, "*.wvd"),
|
||||
@@ -213,8 +211,8 @@ def _build_cdm_config(
|
||||
user_config["playready_devices"] = []
|
||||
if serve_playready:
|
||||
log.warning(
|
||||
f'User "{key_id(user_key)}" has no "playready_devices" configured; PlayReady access disabled '
|
||||
f"for this user. Available PlayReady devices: {prd_device_names}"
|
||||
f'User "{key_id(user_key)}" has no "playready_devices", PlayReady is off for this user '
|
||||
f"(available: {prd_device_names})"
|
||||
)
|
||||
serve_config["devices"] = wvd_devices
|
||||
serve_config["playready_devices"] = prd_devices
|
||||
|
||||
@@ -38,8 +38,7 @@ def tunnel_test(server: Optional[str]) -> None:
|
||||
# one exit can answer over IPv4 here and IPv6 there, so a matching country also passes
|
||||
if remote.get("ip") != local["ip"] and remote.get("country") != local.get("country"):
|
||||
raise click.ClickException(
|
||||
f"Step 3 failed: the server's requests exit at {remote.get('ip')} ({country}), "
|
||||
f"not where this machine exits. Report this with the IP addresses."
|
||||
f"Step 3 failed: the server's requests exit at {remote.get('ip')} ({country}), not at this machine's exit"
|
||||
)
|
||||
log.info(f"Step 3: the server's requests exit at {remote['ip']} ({country}), {remote.get('ms')} ms for one lookup")
|
||||
log.info("The tunnel works. Add --local-proxy to a --remote command.")
|
||||
|
||||
@@ -223,15 +223,15 @@ def new(
|
||||
# TODO: Remove need for name, create name based on Client IDs ClientInfo values
|
||||
name = unidecode(name.strip().lower().replace(" ", "_"))
|
||||
except UnidecodeError as e:
|
||||
raise click.UsageError(f"name: Failed to sanitize name, {e}", ctx)
|
||||
raise click.UsageError(f"name: Failed to sanitise name, {e}", ctx)
|
||||
if not name:
|
||||
raise click.UsageError("name: Empty after sanitizing, please make sure the name is valid.", ctx)
|
||||
raise click.UsageError("name: Empty after sanitising", ctx)
|
||||
if not private_key.is_file():
|
||||
raise click.UsageError("private_key: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("private_key: Not a path to a file, or it does not exist.", ctx)
|
||||
if not client_id.is_file():
|
||||
raise click.UsageError("client_id: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("client_id: Not a path to a file, or it does not exist.", ctx)
|
||||
if file_hashes and not file_hashes.is_file():
|
||||
raise click.UsageError("file_hashes: Not a path to a file, or it doesn't exist.", ctx)
|
||||
raise click.UsageError("file_hashes: Not a path to a file, or it does not exist.", ctx)
|
||||
|
||||
device = Device(
|
||||
type_=DeviceTypes[type_.upper()],
|
||||
|
||||
@@ -20,7 +20,7 @@ from typing import Any, Callable, Dict, List, Optional
|
||||
from unshackle.core.api.events import bus, publish_service_event
|
||||
from unshackle.core.api.sanitize import sanitize_log
|
||||
from unshackle.core.utils.click_types import AUDIO_CODEC_LIST, QUALITY_LIST, VIDEO_CODEC_LIST
|
||||
from unshackle.core.utils.redact import REDACTED, URL_USERINFO_RE, redact_path, redact_text
|
||||
from unshackle.core.utils.redact import REDACTED, URL_USERINFO_RE, redact_all, redact_path, redact_text
|
||||
|
||||
log = logging.getLogger("download_manager")
|
||||
|
||||
@@ -371,11 +371,10 @@ def perform_download(
|
||||
|
||||
# Import dl.py components lazily to avoid circular deps during module import
|
||||
import click
|
||||
import yaml
|
||||
|
||||
from unshackle.commands.dl import dl
|
||||
from unshackle.core.api.errors import APIError, APIErrorCode
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.config import config, load_yaml
|
||||
from unshackle.core.services import Services
|
||||
from unshackle.core.tracks import Video
|
||||
from unshackle.core.utils.click_types import ContextData
|
||||
@@ -467,7 +466,7 @@ def perform_download(
|
||||
remote = params.get("remote")
|
||||
service_config_path = None if remote else Services.get_path(service) / config.filenames.config
|
||||
if service_config_path and service_config_path.exists():
|
||||
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
|
||||
service_config = load_yaml(service_config_path)
|
||||
else:
|
||||
service_config = {}
|
||||
merge_dict(config.services.get(service), service_config)
|
||||
@@ -678,26 +677,23 @@ def perform_download(
|
||||
|
||||
except SystemExit as exc:
|
||||
if exc.code != 0:
|
||||
stdout_str = stdout_capture.getvalue()
|
||||
stderr_str = stderr_capture.getvalue()
|
||||
log.error(f"Download exited with code {exc.code}")
|
||||
log.error(f"Stdout: {stdout_str}")
|
||||
log.error(f"Stderr: {stderr_str}")
|
||||
log.debug("Stdout: %s", stdout_capture.getvalue())
|
||||
log.debug("Stderr: %s", stderr_capture.getvalue())
|
||||
raise APIError(APIErrorCode.DOWNLOAD_ERROR, f"Download failed with exit code {exc.code}")
|
||||
|
||||
except Exception as exc: # noqa: BLE001 - propagate to caller
|
||||
stdout_str = stdout_capture.getvalue()
|
||||
stderr_str = stderr_capture.getvalue()
|
||||
log.error(f"Download execution failed: {exc}")
|
||||
log.error(f"Stdout: {stdout_str}")
|
||||
log.error(f"Stderr: {stderr_str}")
|
||||
log.debug("Stdout: %s", stdout_capture.getvalue())
|
||||
log.debug("Stderr: %s", stderr_capture.getvalue())
|
||||
raise
|
||||
|
||||
# dl.result() catches a download-worker exception, reports it, but returns normally (exit 0).
|
||||
# It sets download_failed in that case, so the job isn't reported as completed with no output.
|
||||
if getattr(dl_instance, "download_failed", False):
|
||||
detail = (stdout_capture.getvalue() + stderr_capture.getvalue())[-200:].strip()
|
||||
raise APIError(APIErrorCode.WORKER_ERROR, "download worker failed: " + (detail or "see logs"))
|
||||
# It keeps the error in download_error, so the job isn't reported as completed with no output.
|
||||
if (error := getattr(dl_instance, "download_error", None)) is not None:
|
||||
# segment URLs carry signed CDN tokens (hmac, session IDs) that redact_text does not match
|
||||
detail = redact_all(dl_instance.failure_lines(error)[1].strip())
|
||||
raise APIError(APIErrorCode.WORKER_ERROR, f"download worker failed: {detail}")
|
||||
|
||||
if progress_callback:
|
||||
skipped_subs = getattr(dl_instance, "skipped_subtitles", None)
|
||||
|
||||
@@ -28,13 +28,13 @@ from unshackle.core.api.session_log import SessionLogBuffer, SessionLogMirror, c
|
||||
from unshackle.core.api.session_store import SessionStore
|
||||
from unshackle.core.cacher import Cacher
|
||||
from unshackle.core.cdm.detect import cdm_type_stub
|
||||
from unshackle.core.config import config
|
||||
from unshackle.core.config import config, load_yaml
|
||||
from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_CODEC_MAP
|
||||
from unshackle.core.drm import cbcs
|
||||
from unshackle.core.providers.anilist import parse_anilist_ref
|
||||
from unshackle.core.proxies.resolve import initialize_proxy_providers, resolve_proxy
|
||||
from unshackle.core.services import Services
|
||||
from unshackle.core.session import utf8_text
|
||||
from unshackle.core.session import RnetSession, utf8_text
|
||||
from unshackle.core.titles import Episode, Movie, Song, Title_T
|
||||
from unshackle.core.titles.music import SONG_FIELDS
|
||||
from unshackle.core.tracks import Audio, Subtitle, Tracks, Video
|
||||
@@ -190,13 +190,11 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]
|
||||
|
||||
def load_service_yaml(normalized_service: str) -> dict:
|
||||
"""Load a service's config.yaml and merge it with the global override block."""
|
||||
import yaml
|
||||
|
||||
from unshackle.core.utils.collections import merge_dict
|
||||
|
||||
service_config_path = Services.get_path(normalized_service) / config.filenames.config
|
||||
if service_config_path.exists():
|
||||
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8")) or {}
|
||||
service_config = load_yaml(service_config_path) or {}
|
||||
else:
|
||||
service_config = {}
|
||||
merge_dict(config.services.get(normalized_service), service_config)
|
||||
@@ -273,8 +271,7 @@ def instantiate_service(
|
||||
for k in cli_param_names & transport_keys & set(data):
|
||||
if k not in ("service", "title_id", "proxy", "no_proxy") and data.get(k) is not None:
|
||||
log.warning(
|
||||
f"Ignoring flat '{sanitize_log(k)}' as a service option (it names a transport field); "
|
||||
"update the client to send service options under 'service_params'"
|
||||
f"Ignoring flat '{sanitize_log(k)}' as a service option, send it under 'service_params'"
|
||||
)
|
||||
return parent_ctx.invoke(service_module.cli, title=title, **extras)
|
||||
|
||||
@@ -1057,8 +1054,7 @@ def choose_session_cdm(
|
||||
if data.get("cdm_type"):
|
||||
return False, cap
|
||||
raise capped_error(
|
||||
f"The server CDM licenses {service} up to {cap}p for this API key. "
|
||||
f"Set up a local CDM to license {wanted}p with your own device, or pass -q {cap}.",
|
||||
f"The server CDM licenses {service} up to {cap}p for this API key, {wanted}p needs a local CDM",
|
||||
cap,
|
||||
)
|
||||
|
||||
@@ -1208,8 +1204,7 @@ def next_server_profile(service: str, region: Optional[str]) -> Optional[str]:
|
||||
regions = ", ".join(covered.get("regions") or []) or "none"
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_INPUT,
|
||||
f"No server account for {service} in region '{region or 'unknown'}'. "
|
||||
f"Its accounts cover: {regions}. Pass --proxy with one of those country codes.",
|
||||
f"No server account for {service} in region '{region or 'unknown'}', accounts cover: {regions}",
|
||||
details={"service": service, "region": region, "regions": covered.get("regions") or []},
|
||||
)
|
||||
key = f"{service}:{','.join(p or '' for p in pool)}"
|
||||
@@ -2162,7 +2157,7 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
|
||||
"""
|
||||
for banned in ("postscript", "post_script", "post_scripts"):
|
||||
if data.get(banned):
|
||||
return f"'{banned}' is not accepted over the API. Define post_scripts in unshackle.yaml."
|
||||
return f"'{banned}' is not accepted over the API, only from unshackle.yaml"
|
||||
|
||||
# The worker reads cookies/{service}/{profile}.txt, so a profile must not be able to name a path.
|
||||
try:
|
||||
@@ -2269,10 +2264,7 @@ def validate_download_parameters(data: Dict[str, Any]) -> Optional[str]:
|
||||
|
||||
supplied_ids = [name for name in ("tmdb_id", "imdb_id", "tvdb_id") if data.get(name)]
|
||||
if len(supplied_ids) > 1:
|
||||
return (
|
||||
f"Cannot use multiple external IDs: {', '.join(supplied_ids)}. "
|
||||
"Give one ID and unshackle resolves the others from it."
|
||||
)
|
||||
return f"Cannot use multiple external IDs: {', '.join(supplied_ids)}; give one ID"
|
||||
|
||||
exclusive_flags = []
|
||||
if data.get("video_only"):
|
||||
@@ -2342,7 +2334,7 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request
|
||||
if not server_cdm_allowed(request, params.get("service")):
|
||||
raise APIError(
|
||||
APIErrorCode.FORBIDDEN,
|
||||
"Download jobs license with the server CDM, which is not enabled for this key on this service.",
|
||||
"Download jobs license with the server CDM, which is not enabled for this API key on this service",
|
||||
)
|
||||
cap = server_cdm_max_height(request, params.get("service"))
|
||||
if cap is not None:
|
||||
@@ -2352,27 +2344,16 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request
|
||||
"""The job's own value for this parameter, or the ``serve:`` dl default that the job falls back to."""
|
||||
return params[key] if key in params else serve_defaults.get(key)
|
||||
|
||||
capped = f"Download jobs license with the server CDM, up to {cap}p for this API key"
|
||||
wanted = requested_height(effective("quality"))
|
||||
if wanted is None or wanted > cap:
|
||||
raise capped_error(
|
||||
f"Download jobs license with the server CDM, which covers up to {cap}p for this API key. "
|
||||
f"Pass a quality of {cap} or lower.",
|
||||
cap,
|
||||
)
|
||||
raise capped_error(f"{capped}: pass a quality", cap)
|
||||
if effective("best_available"):
|
||||
raise capped_error(
|
||||
f"Download jobs license with the server CDM, which covers up to {cap}p for this API key. "
|
||||
"Turn off best_available, because it can fall back to a track above that height.",
|
||||
cap,
|
||||
)
|
||||
raise capped_error(f"{capped}: drop best_available", cap)
|
||||
ranges = effective("range") or []
|
||||
ranges = [ranges] if isinstance(ranges, str) else ranges
|
||||
if any(str(getattr(r, "value", r)).upper() == "HYBRID" for r in ranges):
|
||||
raise capped_error(
|
||||
f"Download jobs license with the server CDM, which covers up to {cap}p for this API key. "
|
||||
"Remove HYBRID from range, because it always keeps the lowest DV track, whatever the quality.",
|
||||
cap,
|
||||
)
|
||||
raise capped_error(f"{capped}: drop HYBRID", cap)
|
||||
|
||||
if params.get("proxy_download") is not None and not isinstance(params["proxy_download"], str):
|
||||
raise APIError(APIErrorCode.INVALID_INPUT, "proxy_download must be a string.")
|
||||
@@ -4070,8 +4051,8 @@ async def session_titles_handler(session_id: str, request: Optional[web.Request]
|
||||
)
|
||||
|
||||
|
||||
def client_session_auth(session: Any) -> tuple[Dict[str, str], Dict[str, str]]:
|
||||
"""The service session headers and cookies a remote client may receive.
|
||||
def client_session_auth(session: Any) -> tuple[Dict[str, str], list[Dict[str, Any]]]:
|
||||
"""The service session headers and cookies (name, value, domain, path, secure) a remote client may receive.
|
||||
|
||||
A client whose own login succeeded gets them back. For a server account the cookie jar
|
||||
stays on the server and the server drops the auth-bearing headers, so the client cannot
|
||||
@@ -4080,18 +4061,31 @@ def client_session_auth(session: Any) -> tuple[Dict[str, str], Dict[str, str]]:
|
||||
"""
|
||||
svc_session = session.service_instance.session
|
||||
headers = dict(svc_session.headers) if hasattr(svc_session, "headers") and svc_session.headers else {}
|
||||
cookies: Dict[str, str] = {}
|
||||
if hasattr(svc_session, "cookies"):
|
||||
for cookie in svc_session.cookies:
|
||||
if hasattr(cookie, "name") and hasattr(cookie, "value"):
|
||||
cookies[cookie.name] = cookie.value
|
||||
cookies: list[Dict[str, Any]] = []
|
||||
jar = svc_session.scoped_cookies() if isinstance(svc_session, RnetSession) else getattr(svc_session, "cookies", [])
|
||||
for cookie in jar:
|
||||
if hasattr(cookie, "name") and hasattr(cookie, "value"):
|
||||
cookies.append(
|
||||
{
|
||||
"name": cookie.name,
|
||||
"value": cookie.value,
|
||||
"domain": getattr(cookie, "domain", None) or "",
|
||||
"path": getattr(cookie, "path", None) or "/",
|
||||
"secure": bool(getattr(cookie, "secure", False)),
|
||||
}
|
||||
)
|
||||
if not client_owns_login(session):
|
||||
headers = {k: v for k, v in headers.items() if not CONFIG_SECRET_KEY_RE.search(str(k))}
|
||||
if not session.client_auth:
|
||||
cookies = {}
|
||||
cookies = []
|
||||
return headers, cookies
|
||||
|
||||
|
||||
def cookie_values(cookies: list[Dict[str, Any]]) -> Dict[str, str]:
|
||||
"""The ``{name: value}`` form of :func:`client_session_auth` cookies, for clients that predate the jar."""
|
||||
return {c["name"]: c["value"] for c in cookies}
|
||||
|
||||
|
||||
def client_owns_login(session: Any) -> bool:
|
||||
"""Whether the login state of ``session`` is the client's to take back.
|
||||
|
||||
@@ -4234,7 +4228,8 @@ async def session_tracks_handler(
|
||||
"manifests": manifests,
|
||||
"track_manifests": track_manifests,
|
||||
"session_headers": session_headers,
|
||||
"session_cookies": session_cookies,
|
||||
"session_cookies": cookie_values(session_cookies),
|
||||
"session_cookie_jar": session_cookies,
|
||||
"server_cdm": session.server_cdm,
|
||||
"server_cdm_type": server_cdm_type,
|
||||
"sample_decrypt_tracks": [
|
||||
@@ -4271,6 +4266,7 @@ async def session_segments_handler(
|
||||
|
||||
try:
|
||||
result: Dict[str, Any] = {}
|
||||
headers, cookies = client_session_auth(session)
|
||||
|
||||
for track_id in track_ids:
|
||||
track = session.tracks.get(track_id)
|
||||
@@ -4287,10 +4283,11 @@ async def session_segments_handler(
|
||||
"descriptor": descriptor_name,
|
||||
"url": str(track.url) if track.url else None,
|
||||
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
|
||||
"headers": headers,
|
||||
"cookies": cookie_values(cookies),
|
||||
"cookie_jar": cookies,
|
||||
}
|
||||
|
||||
track_info["headers"], track_info["cookies"] = client_session_auth(session)
|
||||
|
||||
# Include manifest-specific data for segment resolution. Round-trip through
|
||||
# JSON so any non-serializable value becomes its str() (default=str).
|
||||
if hasattr(track, "data") and track.data:
|
||||
@@ -4557,7 +4554,7 @@ def resolve_handler_proxy(
|
||||
):
|
||||
raise APIError(
|
||||
APIErrorCode.FORBIDDEN,
|
||||
"A server-account service does not accept a client proxy URI. Pass a country code or nothing.",
|
||||
"A server-account service takes a country code or no proxy, not a proxy URI",
|
||||
details={"service": normalized_service},
|
||||
)
|
||||
|
||||
@@ -4587,8 +4584,7 @@ def resolve_handler_proxy(
|
||||
)
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_PROXY,
|
||||
"This server does not resolve country codes into proxies. "
|
||||
"Pass --proxy with a full proxy URI (http://, https:// or socks5://).",
|
||||
"This server does not resolve country codes into proxies, pass a full proxy URI",
|
||||
details={"service": normalized_service},
|
||||
)
|
||||
|
||||
@@ -4614,8 +4610,7 @@ def resolve_handler_proxy(
|
||||
if client_region.lower() in {x.lower() for x in geoblock}:
|
||||
raise APIError(
|
||||
APIErrorCode.GEOFENCE,
|
||||
f"Service is not available in your region ({client_region.upper()}). "
|
||||
"Pass --proxy with a proxy outside the blocked regions.",
|
||||
f"Service is not available in your region ({client_region.upper()})",
|
||||
details={"service": normalized_service},
|
||||
)
|
||||
|
||||
@@ -4624,8 +4619,7 @@ def resolve_handler_proxy(
|
||||
if server_region and not in_client_region:
|
||||
raise APIError(
|
||||
APIErrorCode.INVALID_PROXY,
|
||||
"This server is in a different region from yours. Pass --proxy with a proxy "
|
||||
"in the region you want, or pass --no-proxy to use the server's own connection.",
|
||||
"This server is in a different region from yours, pass --proxy or --no-proxy",
|
||||
details={"service": normalized_service},
|
||||
)
|
||||
elif in_client_region:
|
||||
@@ -4914,11 +4908,12 @@ CACHED_PAIRS: set[tuple[str, str, str]] = set()
|
||||
|
||||
|
||||
def cache_to_vaults(keys: Dict[str, str], service_name: str) -> None:
|
||||
"""Cache newly obtained keys to server vaults.
|
||||
"""Cache the content keys of a CDM licence to the server vaults.
|
||||
|
||||
A PlayReady licence always runs, and a Widevine licence runs when one PSSH KID has no content
|
||||
key in a vault, so the same content keys come back on every request for a title. This process
|
||||
does not send a pair again after every vault accepted it.
|
||||
does not send a pair again after every vault accepted it. A pair a client reported as bad
|
||||
leaves that set, and is unflagged when the CDM returns it.
|
||||
"""
|
||||
from uuid import UUID
|
||||
|
||||
@@ -4931,6 +4926,8 @@ def cache_to_vaults(keys: Dict[str, str], service_name: str) -> None:
|
||||
return
|
||||
|
||||
key_map = {UUID(hex=kid): key for kid, key in new.items()}
|
||||
for kid in vaults.clear_licensed_flags(key_map):
|
||||
log.info(f"Unflagged {kid.hex}:{key_map[kid]}, the CDM returned it")
|
||||
cached = vaults.add_keys(key_map)
|
||||
if cached == len([vault for vault in vaults.vaults if not vault.no_push]):
|
||||
CACHED_PAIRS.update((service_name, kid, key) for kid, key in new.items())
|
||||
@@ -5053,7 +5050,7 @@ def handle_single_server_cdm(
|
||||
if sources is not None:
|
||||
sources.update(vault_hit[1])
|
||||
return vault_hit[0]
|
||||
raise refusal or APIError(APIErrorCode.NO_CONTENT, "The server vault holds no key for this track")
|
||||
raise refusal or APIError(APIErrorCode.NO_CONTENT, "The server vault holds no content key for this track")
|
||||
|
||||
api_key = request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous"
|
||||
user_config = serve_user_config(api_key)
|
||||
@@ -5333,7 +5330,7 @@ def license_response(
|
||||
):
|
||||
raise APIError(
|
||||
APIErrorCode.FORBIDDEN,
|
||||
"Server CDM licensing is not enabled for this key on this service. Use a local CDM (proxy mode).",
|
||||
"Server CDM licensing is not enabled for this API key on this service",
|
||||
)
|
||||
|
||||
if mode == "server_cdm" and track_ids:
|
||||
@@ -5603,7 +5600,8 @@ async def session_bad_key_handler(
|
||||
|
||||
The server never sees a segment, so the client is the only side that can test a content key.
|
||||
Only a pair this remote session served can be flagged, which keeps a client from
|
||||
poisoning the bad-key table for content keys it never received.
|
||||
poisoning the bad-key table for content keys it never received. A pair the CDM licensed is
|
||||
not flagged, because the CDM is the authority.
|
||||
"""
|
||||
from uuid import UUID
|
||||
|
||||
@@ -5616,8 +5614,12 @@ async def session_bad_key_handler(
|
||||
if not key or served_key.lower() != key:
|
||||
log.warning(f"Session {sanitize_log(session_id[:12])} reported a bad content key it was never served: {kid}")
|
||||
raise APIError(APIErrorCode.INVALID_INPUT, "This session was not served that KID:KEY pair")
|
||||
if source == "cdm":
|
||||
log.warning(f"Session {sanitize_log(session_id[:12])} reported a CDM key as bad, not flagged: {kid}")
|
||||
return web.json_response({"flagged": False})
|
||||
|
||||
vaults = load_server_vaults(session.service_instance.__class__.__name__)
|
||||
service_name = session.service_instance.__class__.__name__
|
||||
vaults = load_server_vaults(service_name)
|
||||
|
||||
def flag() -> None:
|
||||
for vault in vaults.vaults:
|
||||
@@ -5630,6 +5632,7 @@ async def session_bad_key_handler(
|
||||
|
||||
await asyncio.to_thread(flag)
|
||||
session.served_keys.pop(kid, None)
|
||||
CACHED_PAIRS.discard((service_name, kid, served_key))
|
||||
log.warning(
|
||||
f"Client proved {kid}:{served_key} from vault {sanitize_log(source)} wrong, flagged in the server vaults"
|
||||
)
|
||||
|
||||
@@ -202,10 +202,7 @@ class InputBridge:
|
||||
self._status = value
|
||||
|
||||
|
||||
NO_USER_MESSAGE = (
|
||||
"This login needs the user, and this request cannot reach the user. "
|
||||
"Start a download job or a remote session to complete the authentication"
|
||||
)
|
||||
NO_USER_MESSAGE = "This login needs the user, and only a download job or a remote session can reach the user"
|
||||
|
||||
|
||||
class NoUserBridge(InputBridge):
|
||||
|
||||
@@ -2421,7 +2421,7 @@ async def session_bad_key(request: web.Request) -> web.Response:
|
||||
description: Content key as hex
|
||||
responses:
|
||||
'200':
|
||||
description: The pair is flagged
|
||||
description: "flagged true when the pair is flagged; false when the CDM licensed it"
|
||||
'400':
|
||||
description: The remote session was not served that pair
|
||||
'404':
|
||||
|
||||
@@ -24,10 +24,7 @@ from unshackle.core.tracks import Track
|
||||
|
||||
log = logging.getLogger("api.session")
|
||||
|
||||
DRAIN_WARNING = (
|
||||
"This server restarts for an update when the open downloads finish. "
|
||||
"This download continues. Start new downloads after the restart."
|
||||
)
|
||||
DRAIN_WARNING = "This server restarts for an update after the open downloads finish; this download continues"
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -359,8 +356,7 @@ def refuse_while_draining() -> None:
|
||||
if get_session_store().draining:
|
||||
raise APIError(
|
||||
APIErrorCode.SERVER_DRAINING,
|
||||
"This server restarts for an update and does not start new sessions or downloads now. "
|
||||
"Try again in a few minutes.",
|
||||
"This server restarts for an update and starts no new sessions or downloads",
|
||||
retryable=True,
|
||||
)
|
||||
|
||||
|
||||
@@ -67,18 +67,14 @@ class Cacher:
|
||||
checksum = data["crc32"]
|
||||
calculated = zlib.crc32(jsonpickle.dumps(payload).encode("utf8"))
|
||||
if calculated != checksum:
|
||||
raise ValueError(
|
||||
f"The checksum of the Cache payload mismatched. Checksum: {checksum} !== Calculated: {calculated}"
|
||||
)
|
||||
raise ValueError(f"Cache payload checksum mismatch: {checksum} != calculated {calculated}")
|
||||
cache.data = data["data"]
|
||||
# The pickled `expiration` is local wall-clock and only right in the zone that wrote it.
|
||||
expires_at = data.get("expires_at")
|
||||
cache.expiration = datetime.fromtimestamp(expires_at) if expires_at is not None else data["expiration"]
|
||||
cache.version = data["version"]
|
||||
if cache.version != version:
|
||||
raise ValueError(
|
||||
f"The version of your {self.service_tag} {key} cache is outdated. Please delete: {cache.path}"
|
||||
)
|
||||
raise ValueError(f"The {self.service_tag} {key} cache is outdated, delete {cache.path}")
|
||||
return cache
|
||||
|
||||
def set(self, data: Any, expiration: Optional[EXP_T] = None) -> Any:
|
||||
|
||||
@@ -536,7 +536,7 @@ class DecryptLabsRemoteCDM:
|
||||
session["decrypt_labs_session_id"] = data["session_id"]
|
||||
return challenge
|
||||
|
||||
error_msg = f"Unexpected API response format. message_type={message_type}, available_fields={list(data.keys())}"
|
||||
error_msg = f"Unexpected API response format: message_type={message_type}, available_fields={list(data.keys())}"
|
||||
if data.get("message"):
|
||||
error_msg = f"API response: {data['message']} - {error_msg}"
|
||||
if "details" in data:
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
@@ -143,7 +144,7 @@ class Config:
|
||||
downloader_cfg = kwargs.get("downloader")
|
||||
if downloader_cfg and downloader_cfg != "requests":
|
||||
warnings.warn(
|
||||
f"downloader '{downloader_cfg}' is deprecated. The unified requests downloader is now used.",
|
||||
f"downloader '{downloader_cfg}' is deprecated, using the requests downloader",
|
||||
DeprecationWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
@@ -214,9 +215,7 @@ class Config:
|
||||
|
||||
if kwargs.get("scene_naming") is not None:
|
||||
raise SystemExit(
|
||||
"ERROR: The 'scene_naming' option has been removed.\n"
|
||||
"Please configure 'output_template' in your unshackle.yaml instead.\n"
|
||||
"See unshackle-example.yaml for examples."
|
||||
"ERROR: The 'scene_naming' option was removed, use 'output_template' in your unshackle.yaml"
|
||||
)
|
||||
|
||||
if self.output_template:
|
||||
@@ -374,7 +373,56 @@ class Config:
|
||||
raise FileNotFoundError(f"Config file path ({path}) was not found")
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(f"Config file path ({path}) is not to a file.")
|
||||
return cls(**restore_bool_languages(yaml.safe_load(path.read_text(encoding="utf8")) or {}))
|
||||
return cls(**restore_bool_languages(load_yaml(path) or {}))
|
||||
|
||||
|
||||
def duplicate_key_lines(root: Any) -> list[tuple[int, int]]:
|
||||
"""
|
||||
Return (first line, repeat line), counted from 1, for each repeated scalar key in a PyYAML or ruamel node tree.
|
||||
|
||||
Keys compare as written, so two spellings that load as the same key (yes and true, 1 and 0x1) are not found.
|
||||
"""
|
||||
pairs, stack, visited = [], [root], set()
|
||||
while stack:
|
||||
node = stack.pop()
|
||||
if node in visited:
|
||||
continue
|
||||
visited.add(node)
|
||||
if node.id == "mapping":
|
||||
first: dict[tuple[str, str], int] = {}
|
||||
for key_node, value_node in node.value:
|
||||
if key_node.id == "scalar" and key_node.tag != "tag:yaml.org,2002:merge":
|
||||
key = (key_node.tag, key_node.value)
|
||||
line = key_node.start_mark.line + 1
|
||||
if key in first:
|
||||
pairs.append((first[key], line))
|
||||
else:
|
||||
first[key] = line
|
||||
stack += [key_node, value_node]
|
||||
elif node.id == "sequence":
|
||||
stack += node.value
|
||||
return sorted(pairs)
|
||||
|
||||
|
||||
_warned: set[str] = set()
|
||||
|
||||
|
||||
def load_yaml(path: Path) -> Any:
|
||||
"""yaml.safe_load a file, and warn with the lines of each duplicate key, which PyYAML takes silently."""
|
||||
loader = yaml.SafeLoader(path.read_text(encoding="utf8"))
|
||||
try:
|
||||
node = loader.get_single_node()
|
||||
if node is None:
|
||||
return None
|
||||
if pairs := duplicate_key_lines(node):
|
||||
where = ", ".join(f"lines {first} and {repeat}" for first, repeat in pairs)
|
||||
message = f"{path}: duplicate key on {where}. Unshackle uses the later one."
|
||||
if message not in _warned:
|
||||
_warned.add(message)
|
||||
logging.getLogger("config").warning(message)
|
||||
return loader.construct_document(node)
|
||||
finally:
|
||||
loader.dispose()
|
||||
|
||||
|
||||
# noinspection PyProtectedMember
|
||||
|
||||
@@ -75,7 +75,7 @@ class Credential:
|
||||
credential = re.fullmatch(r"^([^:]+?):([^:]+?)(?::(.+))?$", text)
|
||||
if credential:
|
||||
return cls(*credential.groups())
|
||||
raise ValueError("No credentials found in text string. Expecting the format `username:password`")
|
||||
raise ValueError("No credentials in text string, expected `username:password`")
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: Path) -> Credential:
|
||||
|
||||
@@ -1,3 +1,17 @@
|
||||
from .requests import default_max_workers, format_speed, parse_speed_limit, requests, set_speed_limit
|
||||
from .requests import (
|
||||
default_max_workers,
|
||||
format_speed,
|
||||
parse_speed_limit,
|
||||
requests,
|
||||
retries_exhausted,
|
||||
set_speed_limit,
|
||||
)
|
||||
|
||||
__all__ = ("requests", "default_max_workers", "format_speed", "parse_speed_limit", "set_speed_limit")
|
||||
__all__ = (
|
||||
"requests",
|
||||
"default_max_workers",
|
||||
"format_speed",
|
||||
"parse_speed_limit",
|
||||
"retries_exhausted",
|
||||
"set_speed_limit",
|
||||
)
|
||||
|
||||
@@ -27,11 +27,12 @@ from requests.adapters import HTTPAdapter, Retry
|
||||
from rich import filesize
|
||||
|
||||
from unshackle.core.constants import DOWNLOAD_CANCELLED, DownloadCancelled
|
||||
from unshackle.core.utilities import get_debug_logger, get_extension
|
||||
from unshackle.core.utilities import get_debug_logger, get_extension, log_event
|
||||
from unshackle.core.utils import run_proxy
|
||||
|
||||
MAX_ATTEMPTS = 5
|
||||
RETRY_WAIT = 2
|
||||
GAVE_UP_NOTE = "gave up after"
|
||||
# a cut that still delivered bytes is resumable and refreshes the attempt budget; bound it
|
||||
# so a server that sends a little and closes, over and over, still terminates
|
||||
MAX_RESUMES = 50
|
||||
@@ -153,7 +154,7 @@ def parse_speed_limit(value: Union[str, int, float, None]) -> Optional[float]:
|
||||
return None
|
||||
match = re.fullmatch(r"(\d*\.?\d+)\s*([kmg]b?|b)?", text)
|
||||
if not match:
|
||||
raise ValueError(f"Invalid speed limit {value!r}: use e.g. 500k, 5M, 1.5G, plain bytes/sec, or 'off'.")
|
||||
raise ValueError(f"Invalid speed limit {value!r}: expected 500k, 5M, 1.5G, bytes per second or 'off'")
|
||||
return float(match.group(1)) * SPEED_UNITS[match.group(2) or ""]
|
||||
|
||||
|
||||
@@ -324,15 +325,12 @@ class AdaptiveWorkerController:
|
||||
def permanent_status(exc: BaseException) -> Optional[int]:
|
||||
"""HTTP status of ``exc`` when it is a client error that no retry can fix.
|
||||
|
||||
Both session types raise requests' HTTPError; RnetSession wraps it in MaxRetriesError
|
||||
as ``__cause__``, so unwrap one level like retry_sleep does. 408 and 429 do not count:
|
||||
they are transient, and a fan-out that trips a per-client rate limit can still succeed
|
||||
over one sequential connection.
|
||||
408 and 429 do not count: they are transient, and a fan-out that trips a per-client rate
|
||||
limit can still succeed over one sequential connection.
|
||||
"""
|
||||
response = getattr(exc, "response", None)
|
||||
if response is None:
|
||||
response = getattr(getattr(exc, "__cause__", None), "response", None)
|
||||
status = getattr(response, "status_code", None)
|
||||
from unshackle.core.session import failure_response
|
||||
|
||||
status = getattr(failure_response(exc), "status_code", None)
|
||||
if isinstance(status, int) and 400 <= status < 500 and status not in (408, 429):
|
||||
return status
|
||||
return None
|
||||
@@ -358,12 +356,9 @@ def retry_sleep(exc: Exception, attempts: int) -> float:
|
||||
else exponential growth from RETRY_WAIT with jitter so concurrently rate-limited
|
||||
segments do not retry in lockstep. Capped at ``session.MAX_BACKOFF`` either way.
|
||||
"""
|
||||
from unshackle.core.session import MAX_BACKOFF
|
||||
from unshackle.core.session import MAX_BACKOFF, failure_response
|
||||
|
||||
response = getattr(exc, "response", None)
|
||||
if response is None:
|
||||
# RnetSession wraps forcelist failures in MaxRetriesError with the HTTPError as __cause__
|
||||
response = getattr(getattr(exc, "__cause__", None), "response", None)
|
||||
response = failure_response(exc)
|
||||
if response is not None:
|
||||
try:
|
||||
retry_after = response.headers.get("Retry-After")
|
||||
@@ -387,6 +382,11 @@ def retry_sleep(exc: Exception, attempts: int) -> float:
|
||||
return min(backoff + random.uniform(0, backoff * 0.1), MAX_BACKOFF)
|
||||
|
||||
|
||||
def retries_exhausted(exc: BaseException) -> bool:
|
||||
"""Whether ``exc`` is the error download() re-raised after it spent every attempt."""
|
||||
return any(str(note).startswith(GAVE_UP_NOTE) for note in getattr(exc, "__notes__", ()))
|
||||
|
||||
|
||||
def is_requests_session(session: Any) -> bool:
|
||||
"""Whether the HTTP session is a standard requests.Session (it has ``resp.raw``)."""
|
||||
return isinstance(session, Session)
|
||||
@@ -814,6 +814,9 @@ def download(
|
||||
item_range = has_range_header(kwargs)
|
||||
|
||||
attempts = 1
|
||||
# attempts resets on progress, so the give-up note counts failures on their own
|
||||
failures = 0
|
||||
started = _time()
|
||||
written = 0
|
||||
resumes = 0
|
||||
request_limit: Optional[int] = None
|
||||
@@ -1141,11 +1144,13 @@ def download(
|
||||
# a superseded loser's error must not retry or kill the batch
|
||||
if claimed is not None and claimed():
|
||||
return
|
||||
failures += 1
|
||||
cancelled = DOWNLOAD_CANCELLED.is_set() or (abort is not None and abort.is_set())
|
||||
if cancelled or attempts == MAX_ATTEMPTS:
|
||||
# cancel/abort is not an error, but a genuine retry-exhaustion is, so surface it
|
||||
# so the caller fails fast instead of hitting a missing segment at merge time
|
||||
if not cancelled:
|
||||
exc.add_note(f"{GAVE_UP_NOTE} {failures} failed attempts over {_time() - started:.0f}s")
|
||||
raise
|
||||
return
|
||||
if not part_mode:
|
||||
@@ -1169,6 +1174,13 @@ def download(
|
||||
except Exception:
|
||||
pass
|
||||
delay = retry_sleep(exc, max(1, attempts))
|
||||
from unshackle.core.session import failure_reason
|
||||
|
||||
log_event(
|
||||
"segment_retry",
|
||||
message=f"{failure_reason(exc)}, retry {max(1, attempts)}/{MAX_ATTEMPTS - 1} in {delay:.1f}s",
|
||||
context={"file": save_path.name, "failures": failures, "resumes": resumes},
|
||||
)
|
||||
if abort is not None:
|
||||
# interruptible nap: backoff can reach MAX_BACKOFF, and teardown always
|
||||
# sets the batch abort, so a parked worker must wake and exit at once
|
||||
@@ -1315,8 +1327,13 @@ def mp_worker(queue: Any, kwargs: dict[str, Any]) -> None:
|
||||
for event in requests(**kwargs):
|
||||
queue.put(event)
|
||||
except Exception as exc:
|
||||
summary = " ".join(f"{type(exc).__name__}: {exc}".split())
|
||||
queue.put({"__mp_error__": f"{summary}\n{traceback.format_exc()}"})
|
||||
queue.put(
|
||||
{
|
||||
"__mp_error__": " ".join(f"{type(exc).__name__}: {exc}".split()),
|
||||
"__mp_traceback__": traceback.format_exc(),
|
||||
"__mp_notes__": [str(note) for note in getattr(exc, "__notes__", ())],
|
||||
}
|
||||
)
|
||||
finally:
|
||||
queue.put({"__mp_done__": True})
|
||||
|
||||
@@ -1432,7 +1449,11 @@ def download_multiprocess(
|
||||
if "__mp_error__" in event:
|
||||
for p in procs:
|
||||
p.terminate()
|
||||
raise RuntimeError(f"segment download child failed: {event['__mp_error__']}")
|
||||
log_event("segment_child_failed", level="ERROR", message=event.get("__mp_traceback__", ""))
|
||||
child_error = RuntimeError(f"segment download child failed: {event['__mp_error__']}")
|
||||
for note in event.get("__mp_notes__", ()):
|
||||
child_error.add_note(note)
|
||||
raise child_error
|
||||
if "total" in event:
|
||||
continue
|
||||
if "downloaded" in event:
|
||||
|
||||
@@ -15,6 +15,7 @@ import m3u8
|
||||
from construct import Container
|
||||
from pymp4.parser import Box
|
||||
from pyplayready.cdm import Cdm as PlayReadyCdm
|
||||
from pyplayready.misc.exceptions import InvalidPssh
|
||||
from pyplayready.system.pssh import PSSH
|
||||
from pyplayready.system.wrmheader import WRMHeader
|
||||
from requests import Session
|
||||
@@ -170,6 +171,26 @@ class PlayReady:
|
||||
|
||||
return []
|
||||
|
||||
@classmethod
|
||||
def from_pssh_b64(cls, pssh_b64: str, kid: Optional[UUID] = None) -> PlayReady:
|
||||
"""Load a base64 PSSH box or PlayReady Object, or a bare UTF-16LE WRMHEADER that some packagers emit.
|
||||
|
||||
It wraps a bare WRMHEADER in a PlayReady Object and keeps that as `pssh_b64`, so remote CDMs and
|
||||
the REST API can parse it again.
|
||||
"""
|
||||
try:
|
||||
pssh = PSSH(pssh_b64)
|
||||
except InvalidPssh:
|
||||
try:
|
||||
raw = base64.b64decode(pssh_b64)
|
||||
except ValueError:
|
||||
raw = b""
|
||||
if not raw.startswith(b"<\x00"):
|
||||
raise
|
||||
pssh_b64 = cls.pro_b64_for(WRMHeader(raw))
|
||||
pssh = PSSH(pssh_b64)
|
||||
return cls(pssh=pssh, kid=kid, pssh_b64=pssh_b64)
|
||||
|
||||
@classmethod
|
||||
def from_track(cls, track: AnyTrack, session: Optional[Session] = None) -> PlayReady:
|
||||
if not session:
|
||||
|
||||
@@ -9,7 +9,10 @@ DECRYPT_HOOK: Optional[Callable[..., None]] = None
|
||||
CONTAINERS = {b"moov", b"trak", b"mdia", b"minf", b"stbl", b"mvex", b"moof", b"traf", b"sinf", b"schi"}
|
||||
PIFF_TENC = bytes.fromhex("8974dbce7be74c5184f97148f9882554")
|
||||
PIFF_TFXD = bytes.fromhex("6d1d9b0542d544e680e2141daff757b2")
|
||||
PIFF_SENC = bytes.fromhex("a2394f525a9b4f14a2446c427c648df4")
|
||||
TENC = {b"tenc", b"uuid" + PIFF_TENC}
|
||||
ENCRYPTED_ENTRY = (b"enc", b"drm")
|
||||
SENC = {b"senc", b"uuid" + PIFF_SENC}
|
||||
# the fixed sample entry fields before the child boxes: visual; sound v0 (and ISO v1), QuickTime sound v1 and v2
|
||||
ENTRY_FIELDS = {b"encv": (78,), b"enca": (28, 44, 64)}
|
||||
|
||||
@@ -123,9 +126,34 @@ def entry_kid(data: bytes, kind: bytes, entry: int, end: int) -> Optional[UUID]:
|
||||
return None
|
||||
|
||||
|
||||
def sample_entries(data: bytes, body: int, end: int) -> list[Optional[UUID]]:
|
||||
"""Parse an ``stsd`` box. Returns the default KID of each sample entry, None for a clear one."""
|
||||
return [entry_kid(data, kind, entry, entry_end) for kind, entry, entry_end in children(data, body + 8, end)]
|
||||
def sample_entries(data: bytes, body: int, end: int) -> tuple[list[Optional[UUID]], bool]:
|
||||
"""Parse an ``stsd`` box. Returns the default KID of each sample entry (None for a clear one), and
|
||||
whether any entry is encrypted."""
|
||||
found = list(children(data, body + 8, end))
|
||||
kids = [entry_kid(data, kind, entry, entry_end) for kind, entry, entry_end in found]
|
||||
return kids, any(kind[:3] in ENCRYPTED_ENTRY for kind, _, _ in found)
|
||||
|
||||
|
||||
def has_encrypted_entry(moov: bytes) -> bool:
|
||||
"""True if a moov box holds a sample entry of an encrypted type (``enc*`` or ``drm*``)."""
|
||||
return any(
|
||||
kind == b"stsd" and any(entry[:3] in ENCRYPTED_ENTRY for entry, _, _ in children(moov, body + 8, end))
|
||||
for kind, body, end in walk(moov)
|
||||
)
|
||||
|
||||
|
||||
def senc_protects_samples(buf: bytes, body: int, end: int) -> bool:
|
||||
"""True if a senc/PIFF-uuid box describes protected samples.
|
||||
|
||||
An empty senc (sample_count 0) sits on a genuinely clear fragment, so treating it as
|
||||
a skipped one would raise on a healthy file. Field order follows
|
||||
read_per_sample_iv_size.
|
||||
"""
|
||||
flags = int.from_bytes(buf[body + 1 : body + 4], "big")
|
||||
pos = body + 4 + (20 if flags & 0x1 else 0) # PIFF override: AlgorithmID(3)+IV_size(1)+KID(16)
|
||||
if pos + 4 > end:
|
||||
return False
|
||||
return int.from_bytes(buf[pos : pos + 4], "big") > 0
|
||||
|
||||
|
||||
def read_top_boxes(f: BinaryIO) -> Iterator[tuple[bytes, Optional[bytes]]]:
|
||||
@@ -164,8 +192,10 @@ def kid_windows(path: Path) -> Optional[KidMap]:
|
||||
the result holds up to three (start, end) windows from ``pick_windows``, relative to
|
||||
the first fragment of the track. A fragment ends where the next fragment of the track
|
||||
starts. The last fragment ends after its sample durations, or after 3 seconds when
|
||||
they are not known. Returns None when the file has no encrypted fragment this reader
|
||||
can place in time, so the caller must use another check.
|
||||
they are not known. Returns an empty map for a clear MP4: no track with fragments (or,
|
||||
without fragments, no track) has an encrypted sample entry or a senc. Returns None when
|
||||
the file has no encrypted fragment this reader can place in time, so the caller must
|
||||
use another check; that includes an encrypted sample entry whose fragments are all clear.
|
||||
"""
|
||||
timescales: dict[int, int] = {}
|
||||
handlers: dict[int, bytes] = {}
|
||||
@@ -174,6 +204,7 @@ def kid_windows(path: Path) -> Optional[KidMap]:
|
||||
moov_groups: dict[int, list[Seig]] = {}
|
||||
fragments: list[Fragment] = []
|
||||
clock: dict[int, int] = {}
|
||||
protected: set[int] = set()
|
||||
try:
|
||||
with path.open("rb") as f:
|
||||
for kind, box in read_top_boxes(f):
|
||||
@@ -197,13 +228,15 @@ def kid_windows(path: Path) -> Optional[KidMap]:
|
||||
# the mdia handler comes first; a QuickTime minf can hold a data handler after it
|
||||
handlers.setdefault(track, box[body + 8 : body + 12])
|
||||
elif kind_ == b"stsd":
|
||||
entries[track] = sample_entries(box, body, box_end)
|
||||
entries[track], encrypted = sample_entries(box, body, box_end)
|
||||
if encrypted:
|
||||
protected.add(track)
|
||||
elif kind_ == b"sgpd":
|
||||
groups = seig_entries(box, body)
|
||||
if groups is not None:
|
||||
moov_groups[track] = groups
|
||||
elif kind == b"moof" and box:
|
||||
fragments.extend(read_fragment(box, entries, defaults, moov_groups, clock))
|
||||
fragments.extend(read_fragment(box, entries, defaults, moov_groups, clock, protected))
|
||||
except (OSError, struct.error, IndexError, ValueError, MemoryError, OverflowError, RecursionError):
|
||||
return None
|
||||
|
||||
@@ -230,7 +263,7 @@ def kid_windows(path: Path) -> Optional[KidMap]:
|
||||
for kid in kids:
|
||||
spans.setdefault(kid, []).append((start, end))
|
||||
if not spans:
|
||||
return None
|
||||
return KidMap({}, False) if entries and protected.isdisjoint(tracks or entries) else None
|
||||
return KidMap({kid: pick_windows(fragment_spans) for kid, fragment_spans in spans.items()}, video)
|
||||
|
||||
|
||||
@@ -284,11 +317,13 @@ def read_fragment(
|
||||
defaults: dict[int, tuple[int, int]],
|
||||
moov_groups: dict[int, list[Seig]],
|
||||
clock: dict[int, int],
|
||||
protected: set[int],
|
||||
) -> Iterator[Fragment]:
|
||||
"""Yield (track, decode time, duration, KIDs that encrypt samples) for each track fragment in a moof.
|
||||
|
||||
A fragment without a tfdt or tfxd starts where the track's previous fragment ended, by
|
||||
the running sum of sample durations in ``clock``.
|
||||
the running sum of sample durations in ``clock``. A track with a senc that protects
|
||||
samples goes into ``protected``.
|
||||
"""
|
||||
for kind, traf, traf_end in children(moof):
|
||||
if kind != b"traf":
|
||||
@@ -297,7 +332,7 @@ def read_fragment(
|
||||
groups: list[Seig] = []
|
||||
runs: Optional[list[tuple[int, int]]] = None
|
||||
truns: list[int] = []
|
||||
for kind_, body, _ in children(moof, traf, traf_end):
|
||||
for kind_, body, box_end in children(moof, traf, traf_end):
|
||||
if kind_ == b"tfhd":
|
||||
flags = int.from_bytes(moof[body + 1 : body + 4], "big")
|
||||
track = struct.unpack_from(">I", moof, body + 4)[0]
|
||||
@@ -317,6 +352,8 @@ def read_fragment(
|
||||
groups = seig_entries(moof, body) or groups
|
||||
elif kind_ == b"sbgp":
|
||||
runs = sbgp_runs(moof, body) or runs
|
||||
elif kind_ in SENC and senc_protects_samples(moof, body, box_end):
|
||||
protected.add(track)
|
||||
if track not in entries:
|
||||
continue
|
||||
default_entry, trex_duration = defaults.get(track, (1, 0))
|
||||
|
||||
@@ -17,7 +17,7 @@ from unshackle.core.config import config
|
||||
from unshackle.core.constants import AnyTrack
|
||||
from unshackle.core.credential import Credential
|
||||
from unshackle.core.drm import drm_from_dict, real_kids
|
||||
from unshackle.core.manifests import DASH, HLS, ISM
|
||||
from unshackle.core.manifests import DASH, HLS, ISM, MissingLanguageError
|
||||
from unshackle.core.remote_service import (
|
||||
RemoteService,
|
||||
build_title,
|
||||
@@ -123,17 +123,15 @@ def resolve_import_manifest_data(
|
||||
try:
|
||||
manifest = parser.from_url(url=url, session=session)
|
||||
parsed = manifest.to_tracks(language=fallback_lang)
|
||||
except MissingLanguageError:
|
||||
raise click.ClickException(
|
||||
f"No language for '{title}': the {manifest_type} manifest has none and Title.language is unset"
|
||||
)
|
||||
except ValueError as e:
|
||||
if "Language information could not be derived" in str(e):
|
||||
raise click.ClickException(
|
||||
f"No language for '{title}': the {manifest_type} manifest has none and the "
|
||||
f"service did not set Title.language."
|
||||
)
|
||||
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}'. ({e})")
|
||||
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}': {e}")
|
||||
except Exception as e:
|
||||
raise click.ClickException(
|
||||
f"Failed to fetch the {manifest_type} manifest for '{title}'. "
|
||||
f"The manifest URL may have expired since export. ({e})"
|
||||
f"Failed to fetch the {manifest_type} manifest for '{title}', the URL may have expired: {e}"
|
||||
)
|
||||
|
||||
local_tracks = [*parsed.videos, *parsed.audio, *parsed.subtitles]
|
||||
@@ -303,7 +301,7 @@ class ImportService:
|
||||
except Exception as e:
|
||||
if explicit:
|
||||
raise click.ClickException(f"Failed to resolve proxy '{proxy_query}': {e}")
|
||||
log.warning(f"Could not auto-select a proxy for export region '{region}': {e}. Continuing without proxy.")
|
||||
log.warning(f"No proxy for export region '{region}', continuing without one: {e}")
|
||||
proxy = None
|
||||
|
||||
if proxy:
|
||||
@@ -386,28 +384,21 @@ class ImportService:
|
||||
manifest = parser.from_url(url=manifest_url, session=self.session)
|
||||
except Exception as e:
|
||||
raise click.ClickException(
|
||||
f"Failed to fetch the {manifest_type} manifest for '{title}'. "
|
||||
f"The manifest URL may have expired since export. ({e})"
|
||||
f"Failed to fetch the {manifest_type} manifest for '{title}', the URL may have expired: {e}"
|
||||
)
|
||||
language = fallback_language(title.language)
|
||||
if language is None and isinstance(manifest, HLS):
|
||||
language = hls_audio_language(manifest)
|
||||
if language is None:
|
||||
raise click.ClickException(
|
||||
f"No language for '{title}': the export gives no title language and the HLS "
|
||||
"manifest gives no audio language."
|
||||
)
|
||||
raise click.ClickException(f"No language for '{title}': the export and the HLS manifest have none")
|
||||
try:
|
||||
parsed = manifest.to_tracks(language=language)
|
||||
except ValueError as e:
|
||||
if "Language information could not be derived" in str(e):
|
||||
raise click.ClickException(
|
||||
f"No language for '{title}': the {manifest_type} manifest has none and the "
|
||||
f"service did not set Title.language."
|
||||
)
|
||||
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}'. ({e})")
|
||||
except MissingLanguageError:
|
||||
raise click.ClickException(
|
||||
f"No language for '{title}': the {manifest_type} manifest has none and Title.language is unset"
|
||||
)
|
||||
except Exception as e:
|
||||
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}'. ({e})")
|
||||
raise click.ClickException(f"Failed to parse the {manifest_type} manifest for '{title}': {e}")
|
||||
for track in parsed:
|
||||
tracks.add(track)
|
||||
parsed_ids = {str(t.id) for t in tracks}
|
||||
@@ -787,10 +778,7 @@ class ImportService:
|
||||
}
|
||||
)
|
||||
subject = f"KID {', '.join(kids)}" if kids else f"track {track}"
|
||||
return click.ClickException(
|
||||
f"The export has no content key for {subject}, and no vault has it. An import does not "
|
||||
"license through a CDM: export the title again with the tracks you need selected."
|
||||
)
|
||||
return click.ClickException(f"The export has no content key to decrypt {subject}, and no vault has it")
|
||||
|
||||
def decrypts_samples(self, track: AnyTrack) -> bool:
|
||||
return False
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
from .dash import DASH
|
||||
from .dash import DASH, MissingLanguageError
|
||||
from .hls import HLS
|
||||
from .ism import ISM
|
||||
|
||||
__all__ = ("DASH", "HLS", "ISM")
|
||||
__all__ = ("DASH", "HLS", "ISM", "MissingLanguageError")
|
||||
|
||||
@@ -20,7 +20,6 @@ from zlib import crc32
|
||||
import requests
|
||||
from langcodes import Language, tag_is_valid
|
||||
from lxml.etree import Element, ElementTree, XMLSyntaxError
|
||||
from pyplayready.system.pssh import PSSH as PR_PSSH
|
||||
from pywidevine.cdm import Cdm as WidevineCdm
|
||||
from pywidevine.pssh import PSSH
|
||||
from requests import Session
|
||||
@@ -45,6 +44,19 @@ from unshackle.core.utils.redact import safe_display_url
|
||||
from unshackle.core.utils.xml import load_xml
|
||||
|
||||
|
||||
class MissingLanguageError(ValueError):
|
||||
"""A manifest track has no language, and the fallback language is missing or invalid."""
|
||||
|
||||
|
||||
def missing_language(where: str, fallback: Any) -> MissingLanguageError:
|
||||
"""The MissingLanguageError for a track at ``where``, naming why the fallback did not apply."""
|
||||
if fallback is None:
|
||||
return MissingLanguageError(f"No language for {where}, and no fallback language was given")
|
||||
if not tag_is_valid(str(fallback).strip()) or str(fallback).startswith("und"):
|
||||
return MissingLanguageError(f"No language for {where}, and the fallback language {fallback} is invalid")
|
||||
return MissingLanguageError(f"No language for {where}")
|
||||
|
||||
|
||||
def append_segment(dst: BinaryIO, segment_file: Path, is_text_subtitle: bool) -> None:
|
||||
"""Append one segment file to the output, with the text subtitle fixups when they apply."""
|
||||
if is_text_subtitle:
|
||||
@@ -287,12 +299,7 @@ class DASH:
|
||||
|
||||
track_lang = self.get_language(adaptation_set, rep, fallback=language)
|
||||
if not track_lang:
|
||||
msg = "Language information could not be derived from a Representation."
|
||||
if language is None:
|
||||
msg += " No fallback language was provided when calling DASH.to_tracks()."
|
||||
elif not tag_is_valid((str(language) or "").strip()) or str(language).startswith("und"):
|
||||
msg += f" The fallback language provided is also invalid: {language}"
|
||||
raise ValueError(msg)
|
||||
raise missing_language("a DASH Representation", language)
|
||||
|
||||
# for some reason it's incredibly common for services to not provide
|
||||
# a good and actually unique track ID, sometimes because of the lang
|
||||
@@ -1355,7 +1362,6 @@ class DASH:
|
||||
)
|
||||
if not pr_pssh_b64:
|
||||
continue
|
||||
pr_pssh = PR_PSSH(pr_pssh_b64)
|
||||
kid_b64 = protection.findtext("kid")
|
||||
kid = None
|
||||
if kid_b64:
|
||||
@@ -1364,7 +1370,10 @@ class DASH:
|
||||
except ValueError:
|
||||
kid = None
|
||||
|
||||
drm.append(PlayReady(pssh=pr_pssh, kid=kid, pssh_b64=pr_pssh_b64))
|
||||
try:
|
||||
drm.append(PlayReady.from_pssh_b64(pr_pssh_b64, kid=kid))
|
||||
except Exception as e:
|
||||
logging.getLogger("DASH").warning(f"Skipping unparseable PlayReady ContentProtection: {e!r}")
|
||||
|
||||
elif urn == ClearKeyCENC.urn:
|
||||
# W3C EME ClearKey (org.w3.clearkey): match the scheme UUID alone,
|
||||
|
||||
@@ -35,7 +35,7 @@ from unshackle.core.drm import DRM_T, ClearKey, MonaLisa, PlayReady, Widevine
|
||||
from unshackle.core.drm.segment_decrypt import SegmentDecrypter, can_use
|
||||
from unshackle.core.drm.verify import decrypt_track
|
||||
from unshackle.core.events import events
|
||||
from unshackle.core.manifests.dash import RollingMerge
|
||||
from unshackle.core.manifests.dash import RollingMerge, missing_language
|
||||
from unshackle.core.session import RnetResponse, RnetSession, utf8_text
|
||||
from unshackle.core.tracks import Audio, DownloadContext, Subtitle, Tracks, Video, resume
|
||||
from unshackle.core.tracks.track import DRM_PREFERENCE_TYPES, assert_fragments_decrypted
|
||||
@@ -305,12 +305,7 @@ class HLS:
|
||||
None,
|
||||
)
|
||||
if not track_lang:
|
||||
msg = "Language information could not be derived for a media."
|
||||
if language is None:
|
||||
msg += " No fallback language was provided when calling HLS.to_tracks()."
|
||||
elif not tag_is_valid((str(language) or "").strip()) or str(language).startswith("und"):
|
||||
msg += f" The fallback language provided is also invalid: {language}"
|
||||
raise ValueError(msg)
|
||||
raise missing_language("an HLS media", language)
|
||||
|
||||
tracks.add(
|
||||
track_type(
|
||||
@@ -664,8 +659,8 @@ class HLS:
|
||||
hls_data = track.data.setdefault("hls", {})
|
||||
if hls_data.get("unwanted_segments") and not unwanted_segments:
|
||||
log.warning(
|
||||
f"None of the {len(hls_data['unwanted_segments'])} segments that the export drops is in the "
|
||||
f"playlist of {track.id}. The download keeps every segment."
|
||||
f"None of the {len(hls_data['unwanted_segments'])} segments the export drops is in the "
|
||||
f"playlist of {track.id}; the download keeps every segment"
|
||||
)
|
||||
hls_data["unwanted_segments"] = [s.absolute_uri for s in unwanted_segments]
|
||||
|
||||
@@ -1578,10 +1573,7 @@ class HLS:
|
||||
**extra_params,
|
||||
)
|
||||
elif key.keyformat and key.keyformat.lower() in {f"urn:uuid:{PR_PSSH.SYSTEM_ID}", "com.microsoft.playready"}:
|
||||
drm = PlayReady(
|
||||
pssh=PR_PSSH(key.uri.split(",")[-1]),
|
||||
pssh_b64=key.uri.split(",")[-1],
|
||||
)
|
||||
drm = PlayReady.from_pssh_b64(key.uri.split(",")[-1])
|
||||
else:
|
||||
raise NotImplementedError(f"The key system is not supported: {key}")
|
||||
|
||||
@@ -1637,9 +1629,7 @@ class HLS:
|
||||
except NotImplementedError:
|
||||
unsupported_keys.append(key)
|
||||
except Widevine.Exceptions.KIDNotFound:
|
||||
logging.getLogger("HLS").warning(
|
||||
"Skipped an EXT-X-SESSION-KEY entry that carries no Key ID. The media playlist must supply the key."
|
||||
)
|
||||
logging.getLogger("HLS").warning("Skipped an EXT-X-SESSION-KEY entry with no Key ID")
|
||||
|
||||
if not drm_objects and unsupported_keys:
|
||||
logging.debug(
|
||||
|
||||
@@ -15,7 +15,6 @@ from typing import Any, BinaryIO, Optional, Union
|
||||
import requests
|
||||
from langcodes import Language, tag_is_valid
|
||||
from lxml.etree import Element
|
||||
from pyplayready.system.pssh import PSSH as PR_PSSH
|
||||
from pywidevine.pssh import PSSH
|
||||
from requests import Session
|
||||
|
||||
@@ -24,7 +23,7 @@ from unshackle.core.constants import DOWNLOAD_CANCELLED, DOWNLOAD_LICENCE_ONLY,
|
||||
from unshackle.core.drm import DRM_T, PlayReady, Widevine
|
||||
from unshackle.core.drm.verify import decrypt_track
|
||||
from unshackle.core.events import events
|
||||
from unshackle.core.manifests.dash import RollingMerge, append_segment
|
||||
from unshackle.core.manifests.dash import RollingMerge, append_segment, missing_language
|
||||
from unshackle.core.manifests.ism_init import (
|
||||
build_init_segment,
|
||||
parse_codec_private_data_vui,
|
||||
@@ -104,11 +103,9 @@ class ISM:
|
||||
drm.append(Widevine(pssh=pssh, kid=kid))
|
||||
elif system_id == "9a04f079-9840-4286-ab92-e65be0885f95":
|
||||
try:
|
||||
pr_pssh = PR_PSSH(data)
|
||||
drm.append(PlayReady.from_pssh_b64(data))
|
||||
except Exception as e:
|
||||
logging.getLogger("ISM").warning(f"Skipping unparseable PlayReady ProtectionHeader: {e!r}")
|
||||
continue
|
||||
drm.append(PlayReady(pssh=pr_pssh, pssh_b64=data))
|
||||
return drm
|
||||
|
||||
@staticmethod
|
||||
@@ -254,10 +251,7 @@ class ISM:
|
||||
track_lang = Language.get(lang)
|
||||
if not track_lang and not language:
|
||||
# Language is optional in MS-SSTR; video streams commonly omit it.
|
||||
raise ValueError(
|
||||
"Language information could not be derived from the manifest and no fallback "
|
||||
"language was provided when calling ISM.to_tracks()."
|
||||
)
|
||||
raise missing_language("an ISM stream", language)
|
||||
|
||||
track_urls: list[str] = []
|
||||
track_times: list[int] = []
|
||||
|
||||
@@ -191,10 +191,7 @@ class Gluetun(Proxy):
|
||||
verify_ip: Automatically make sure that the IP and region are correct after connection (default: True)
|
||||
"""
|
||||
if not binaries.Docker:
|
||||
raise RuntimeError(
|
||||
"Docker is not available. Please install Docker to use Gluetun proxy.\n"
|
||||
"Visit: https://docs.docker.com/engine/install/"
|
||||
)
|
||||
raise RuntimeError("Docker not found, the Gluetun proxy needs it")
|
||||
|
||||
self.providers = providers or {}
|
||||
self.base_port = base_port
|
||||
@@ -247,14 +244,14 @@ class Gluetun(Proxy):
|
||||
if len(parts) == 1:
|
||||
return None
|
||||
if len(parts) != 2:
|
||||
raise ValueError(f"Invalid query format: '{query}'. Expected 'provider:region' (e.g., 'windscribe:us')")
|
||||
raise ValueError(f"Invalid query '{query}': expected provider:region, such as windscribe:us")
|
||||
|
||||
provider_name = parts[0].lower()
|
||||
region = parts[1].lower()
|
||||
|
||||
if provider_name not in self.providers:
|
||||
available = ", ".join(self.providers.keys())
|
||||
raise ValueError(f"VPN provider '{provider_name}' not configured. Available providers: {available}")
|
||||
raise ValueError(f"VPN provider '{provider_name}' not configured, available: {available}")
|
||||
|
||||
query_key = f"{provider_name}:{region}"
|
||||
container_name = f"{self.container_prefix}-{provider_name}-{region}"
|
||||
@@ -320,15 +317,9 @@ class Gluetun(Proxy):
|
||||
if re.match(r"^[a-z]{2}$", region):
|
||||
country = get_country_name(region)
|
||||
if not country:
|
||||
raise ValueError(
|
||||
f"Country code '{region}' not recognized. "
|
||||
f"Configure it in server_countries or use a valid ISO 3166-1 alpha-2 code."
|
||||
)
|
||||
raise ValueError(f"Country code '{region}' not recognized")
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Region '{region}' not recognized for provider '{provider_name}'. "
|
||||
f"Configure it in server_countries or server_cities, or use a 2-letter country code."
|
||||
)
|
||||
raise ValueError(f"Region '{region}' not recognized for provider '{provider_name}'")
|
||||
|
||||
self.remove_stopped_container(container_name)
|
||||
|
||||
@@ -453,7 +444,7 @@ class Gluetun(Proxy):
|
||||
credentials = config.get("credentials", {})
|
||||
|
||||
if vpn_type not in ["wireguard", "openvpn"]:
|
||||
raise ValueError(f"Provider '{provider_name}': Invalid vpn_type '{vpn_type}'. Use 'wireguard' or 'openvpn'")
|
||||
raise ValueError(f"Provider '{provider_name}': vpn_type '{vpn_type}' is not wireguard or openvpn")
|
||||
|
||||
if vpn_type == "wireguard":
|
||||
# private_key is always required for WireGuard
|
||||
@@ -468,16 +459,11 @@ class Gluetun(Proxy):
|
||||
|
||||
# Windscribe requires preshared_key (can be empty string, but must be set)
|
||||
if provider_lower == "windscribe":
|
||||
if "preshared_key" not in credentials:
|
||||
raise ValueError(
|
||||
f"Provider '{provider_name}': Windscribe WireGuard requires 'preshared_key' in credentials "
|
||||
"(can be empty string, but must be set). Get it from windscribe.com/getconfig/wireguard"
|
||||
)
|
||||
if "addresses" not in credentials:
|
||||
raise ValueError(
|
||||
f"Provider '{provider_name}': Windscribe WireGuard requires 'addresses' in credentials. "
|
||||
"Get it from windscribe.com/getconfig/wireguard"
|
||||
)
|
||||
for key in ("preshared_key", "addresses"):
|
||||
if key not in credentials:
|
||||
raise ValueError(
|
||||
f"Provider '{provider_name}': Windscribe WireGuard needs '{key}' in credentials"
|
||||
)
|
||||
|
||||
# Providers that require addresses (but not preshared_key)
|
||||
elif provider_lower in ["surfshark", "mullvad", "ivpn"]:
|
||||
@@ -539,8 +525,8 @@ class Gluetun(Proxy):
|
||||
country_lower = country_code.lower()
|
||||
if provider_name == "surfshark":
|
||||
raise ValueError(
|
||||
"Surfshark has no numbered servers. Map the region to a server name such as "
|
||||
f"'{country_lower}-dal.prod.surfshark.com' in server_hostnames instead."
|
||||
f"Surfshark has no numbered servers: map the region in server_hostnames, "
|
||||
f"such as '{country_lower}-dal.prod.surfshark.com'"
|
||||
)
|
||||
hostname_formats = {
|
||||
"nordvpn": f"{country_lower}{server_num}.nordvpn.com",
|
||||
@@ -646,7 +632,7 @@ class Gluetun(Proxy):
|
||||
message=f"Failed to pull Docker image: {gluetun_image}",
|
||||
success=False,
|
||||
)
|
||||
raise RuntimeError(f"Failed to ensure Gluetun Docker image '{gluetun_image}' is available")
|
||||
raise RuntimeError(f"Failed to pull the Gluetun proxy Docker image '{gluetun_image}'")
|
||||
|
||||
vpn_type = provider_config.get("vpn_type", "wireguard").lower()
|
||||
credentials = provider_config.get("credentials", {})
|
||||
@@ -1234,7 +1220,7 @@ class Gluetun(Proxy):
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"Failed to verify VPN IP for {container['provider']}:{container['region']} "
|
||||
f"after {max_retries} attempts. Last error: {last_error}"
|
||||
f"after {max_retries} attempts: {last_error}"
|
||||
)
|
||||
|
||||
def remove_stopped_container(self, container_name: str) -> bool:
|
||||
|
||||
@@ -37,7 +37,7 @@ class Hola(Proxy):
|
||||
return None
|
||||
|
||||
ban = "Transaction error: temporary ban detected."
|
||||
banned = ConnectionError("Hola banned your IP temporarily from it's services. Try change your IP.")
|
||||
banned = ConnectionError("Hola temporarily banned your IP")
|
||||
try:
|
||||
# hola-proxy logs to stderr from another thread, so a merged stream splits the Login/Password lines.
|
||||
result = subprocess.run(
|
||||
|
||||
@@ -19,14 +19,11 @@ class NordVPN(Proxy):
|
||||
The Service Credentials can be found here: https://my.nordaccount.com/dashboard/nordvpn/
|
||||
"""
|
||||
if not username:
|
||||
raise ValueError("No Username was provided to the NordVPN Proxy Service.")
|
||||
raise ValueError("No username for the NordVPN proxy provider")
|
||||
if not password:
|
||||
raise ValueError("No Password was provided to the NordVPN Proxy Service.")
|
||||
raise ValueError("No password for the NordVPN proxy provider")
|
||||
if not re.match(r"^[a-z0-9]{48}$", username + password, re.IGNORECASE) or "@" in username:
|
||||
raise ValueError(
|
||||
"The Username and Password must be NordVPN Service Credentials, not your Login Credentials. "
|
||||
"The Service Credentials can be found here: https://my.nordaccount.com/dashboard/nordvpn/"
|
||||
)
|
||||
raise ValueError("NordVPN needs its Service Credentials, not your login credentials")
|
||||
|
||||
if server_map is not None and not isinstance(server_map, dict):
|
||||
raise TypeError(f"Expected server_map to be a dict mapping a region to a server ID, not '{server_map!r}'.")
|
||||
@@ -101,10 +98,7 @@ class NordVPN(Proxy):
|
||||
None,
|
||||
)
|
||||
if city_id is None:
|
||||
raise ValueError(
|
||||
f"No servers found in city '{city}' for country '{country['name']}'. "
|
||||
"Try a different city or check the city name spelling."
|
||||
)
|
||||
raise ValueError(f"No servers found in city '{city}' for country '{country['name']}'")
|
||||
if selector and is_ip(selector):
|
||||
recommended_servers = self.get_all_servers(country["id"])
|
||||
else:
|
||||
|
||||
@@ -226,8 +226,7 @@ class ProtonVPN(Proxy):
|
||||
self.load_session()
|
||||
if not self.access_token:
|
||||
log.error(
|
||||
"Proton: no session available; export account.proton.me cookies to %s "
|
||||
"or set proxy_providers.protonvpn.enable: true for the one-time TV login",
|
||||
"Proton: no access token, export account.proton.me cookies to %s or set proxy_providers.protonvpn.enable",
|
||||
self.cookie_path,
|
||||
)
|
||||
return None
|
||||
|
||||
@@ -54,7 +54,7 @@ def find_provider(proxy_providers: List[Any], name: str, hint: str = "") -> Any:
|
||||
|
||||
def load_proxy_providers(exclude: tuple[str, ...] = (), raise_errors: bool = False, quiet: bool = False) -> List[Any]:
|
||||
"""
|
||||
Build the proxy providers set in config, in the order a bare query tries them.
|
||||
Build the proxy providers set in config: Basic first, then config key order, Control D last.
|
||||
|
||||
Each proxy provider loads on its own. One that fails is logged and kept as Unavailable, so a bare
|
||||
query skips it and a query that names it reports why. *raise_errors* raises the failure instead.
|
||||
@@ -75,8 +75,15 @@ def load_proxy_providers(exclude: tuple[str, ...] = (), raise_errors: bool = Fal
|
||||
)
|
||||
|
||||
settings = config.proxy_providers
|
||||
# Control D is last: a bare region reaches it only when nothing else answers, since it changes the account.
|
||||
classes = (Basic, ExpressVPN, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN, Gluetun, Hola, ControlD)
|
||||
rank = {key: i for i, key in enumerate(settings)}
|
||||
classes = [
|
||||
Basic,
|
||||
*sorted(
|
||||
(ExpressVPN, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN, Gluetun, Hola),
|
||||
key=lambda c: rank.get(c.__name__.lower(), len(rank)),
|
||||
),
|
||||
ControlD,
|
||||
]
|
||||
|
||||
def build(cls: type) -> Any:
|
||||
key = cls.__name__.lower()
|
||||
@@ -120,7 +127,7 @@ def initialize_proxy_providers(raise_errors: bool = False, quiet: bool = False)
|
||||
"""
|
||||
proxy_providers = load_proxy_providers(("gluetun",), raise_errors, quiet)
|
||||
if not quiet and not proxy_providers:
|
||||
log.warning("No proxy providers were loaded. Check your proxy provider configuration in unshackle.yaml")
|
||||
log.warning("No proxy providers were loaded")
|
||||
return proxy_providers
|
||||
|
||||
|
||||
|
||||
@@ -91,14 +91,11 @@ class SurfsharkVPN(Proxy):
|
||||
The Service Credentials can be found here: https://my.surfshark.com/vpn/manual-setup/main/openvpn
|
||||
"""
|
||||
if not username:
|
||||
raise ValueError("No Username was provided to the SurfsharkVPN Proxy Service.")
|
||||
raise ValueError("No username for the SurfsharkVPN proxy provider")
|
||||
if not password:
|
||||
raise ValueError("No Password was provided to the SurfsharkVPN Proxy Service.")
|
||||
raise ValueError("No password for the SurfsharkVPN proxy provider")
|
||||
if not re.match(r"^[a-z0-9]{48}$", username + password, re.IGNORECASE) or "@" in username:
|
||||
raise ValueError(
|
||||
"The Username and Password must be SurfsharkVPN Service Credentials, not your Login Credentials. "
|
||||
"The Service Credentials can be found here: https://my.surfshark.com/vpn/manual-setup/main/openvpn"
|
||||
)
|
||||
raise ValueError("SurfsharkVPN needs its Service Credentials, not your login credentials")
|
||||
|
||||
if server_map is not None and not isinstance(server_map, dict):
|
||||
raise TypeError(
|
||||
@@ -113,8 +110,7 @@ class SurfsharkVPN(Proxy):
|
||||
if not server or server.isdigit():
|
||||
# A warning, not an error: one stale entry must not disable the whole proxy provider.
|
||||
log.warning(
|
||||
"SurfsharkVPN: ignoring server_map entry '%s: %s'. Use a server name such as 'us-dal'. "
|
||||
"SurfsharkVPN no longer uses numeric server IDs.",
|
||||
"SurfsharkVPN: ignoring server_map entry '%s: %s', use a server name such as 'us-dal'",
|
||||
region,
|
||||
server,
|
||||
)
|
||||
@@ -240,9 +236,7 @@ class SurfsharkVPN(Proxy):
|
||||
x for location, x in located if location.startswith(wanted)
|
||||
]
|
||||
if not servers:
|
||||
raise ValueError(
|
||||
f"No servers found in city '{city}'. Try a different city or check the city name spelling."
|
||||
)
|
||||
raise ValueError(f"No servers found in city '{city}'")
|
||||
|
||||
return servers
|
||||
|
||||
|
||||
@@ -21,9 +21,9 @@ class WindscribeVPN(Proxy):
|
||||
code with a city ("ca:toronto"), and unshackle uses a pinned hostname before any server lookup.
|
||||
"""
|
||||
if not username:
|
||||
raise ValueError("No Username was provided to the WindscribeVPN Proxy Service.")
|
||||
raise ValueError("No username for the WindscribeVPN proxy provider")
|
||||
if not password:
|
||||
raise ValueError("No Password was provided to the WindscribeVPN Proxy Service.")
|
||||
raise ValueError("No password for the WindscribeVPN proxy provider")
|
||||
|
||||
if server_map is not None and not isinstance(server_map, dict):
|
||||
raise TypeError(f"Expected server_map to be a dict mapping a region to a hostname, not '{server_map!r}'.")
|
||||
@@ -158,10 +158,7 @@ class WindscribeVPN(Proxy):
|
||||
if hostnames:
|
||||
return random.choice(hostnames)
|
||||
elif city:
|
||||
raise ValueError(
|
||||
f"No servers found in city '{city}' for country code '{country_code}'. "
|
||||
"Try a different city or check the city name spelling."
|
||||
)
|
||||
raise ValueError(f"No servers found in city '{city}' for country code '{country_code}'")
|
||||
|
||||
return None
|
||||
|
||||
|
||||
@@ -156,9 +156,7 @@ class RemoteClient:
|
||||
if isinstance(e.args[0] if e.args else None, urllib3.exceptions.ReadTimeoutError):
|
||||
raise self.timed_out(endpoint) from None
|
||||
server_url = safe_display_url(self.server_url)
|
||||
raise RemoteError(
|
||||
f"Could not connect to remote server at {server_url}. Is it running? (unshackle serve)", "CONNECT"
|
||||
) from None
|
||||
raise RemoteError(f"Could not connect to remote server at {server_url}", "CONNECT") from None
|
||||
except requests.Timeout:
|
||||
raise self.timed_out(endpoint) from None
|
||||
# servers differ on which header carries the key, so retry the rest before giving up
|
||||
@@ -196,8 +194,7 @@ class RemoteClient:
|
||||
return {}
|
||||
if self.tunnel and self.tunnel.lost:
|
||||
raise RemoteError(
|
||||
"The tunnel to the server closed, so the server can no longer send its requests through this "
|
||||
"machine. Run the command again.",
|
||||
"The tunnel to the server closed",
|
||||
"TUNNEL_CLOSED",
|
||||
)
|
||||
raise RemoteError(f"Server error [{error_code}]: {error_msg}", error_code)
|
||||
@@ -205,8 +202,7 @@ class RemoteClient:
|
||||
|
||||
def timed_out(self, endpoint: str) -> RemoteError:
|
||||
return RemoteError(
|
||||
f"Request to remote server timed out after {self.timeout}s: {endpoint}. "
|
||||
"Raise 'timeout' for this server in remote_services.",
|
||||
f"Request to remote server timed out after {self.timeout}s: {endpoint}",
|
||||
"TIMEOUT",
|
||||
)
|
||||
|
||||
@@ -426,6 +422,27 @@ def apply_service_track_data(tracks: Tracks, data: Dict[str, Any]) -> None:
|
||||
track.data.update(extra)
|
||||
|
||||
|
||||
def apply_session_cookies(session: requests.Session, result: dict) -> None:
|
||||
"""Copy the server's session cookies into ``session``, each scoped to its own domain.
|
||||
|
||||
An older server sends only ``session_cookies`` names and values, so those cookies go to every host.
|
||||
It skips malformed entries.
|
||||
"""
|
||||
jar = result.get("session_cookie_jar")
|
||||
if not isinstance(jar, list):
|
||||
old = result.get("session_cookies")
|
||||
jar = [{"name": k, "value": v} for k, v in old.items()] if isinstance(old, dict) else []
|
||||
for c in jar:
|
||||
if isinstance(c, dict) and isinstance(c.get("name"), str) and isinstance(c.get("value"), str):
|
||||
session.cookies.set(
|
||||
c["name"],
|
||||
c["value"],
|
||||
domain=str(c.get("domain") or ""),
|
||||
path=str(c.get("path") or "/"),
|
||||
secure=bool(c.get("secure")),
|
||||
)
|
||||
|
||||
|
||||
def resolve_track_manifests(tracks: Tracks, track_manifests: list) -> None:
|
||||
"""Fill track.data from the per-track MPD fragments the server sent.
|
||||
|
||||
@@ -706,8 +723,7 @@ def build_title(info: Dict[str, Any], service_tag: str, fallback_id: str) -> Tit
|
||||
)
|
||||
else:
|
||||
raise click.ClickException(
|
||||
f"The remote server sent a title of type {title_type!r}, which this client does not support. "
|
||||
"Update unshackle on both ends so they agree on the title types."
|
||||
f"The remote server sent an unknown title type {title_type!r}; unshackle versions differ"
|
||||
)
|
||||
apply_title_fields(title, info)
|
||||
return title
|
||||
@@ -723,10 +739,7 @@ def resolve_auth_headers(svc: dict, server_name: str) -> list[str]:
|
||||
or not auth_headers
|
||||
or not all(isinstance(h, str) and h.strip() for h in auth_headers)
|
||||
):
|
||||
raise click.ClickException(
|
||||
f"Remote service '{server_name}': 'auth_headers' must be a list of header names, e.g.\n\n"
|
||||
' auth_headers: ["Authorization"]'
|
||||
)
|
||||
raise click.ClickException(f"Remote service '{server_name}': 'auth_headers' must be a list of header names")
|
||||
resolved = [h.strip() for h in auth_headers]
|
||||
seen = {h.lower() for h in resolved}
|
||||
resolved.extend(h for h in DEFAULT_AUTH_HEADERS if h.lower() not in seen)
|
||||
@@ -743,19 +756,13 @@ def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]:
|
||||
"""
|
||||
remote_services = config.remote_services
|
||||
if not remote_services:
|
||||
raise click.ClickException(
|
||||
"No remote services configured. Add 'remote_services' to your unshackle.yaml:\n\n"
|
||||
" remote_services:\n"
|
||||
" my_server:\n"
|
||||
' url: "https://server:8080"\n'
|
||||
' api_key: "your-api-key"'
|
||||
)
|
||||
raise click.ClickException("No 'remote_services' in your unshackle.yaml")
|
||||
|
||||
if server_name:
|
||||
svc = remote_services.get(server_name)
|
||||
if not svc:
|
||||
available = ", ".join(remote_services.keys())
|
||||
raise click.ClickException(f"Remote service '{server_name}' not found. Available: {available}")
|
||||
raise click.ClickException(f"Remote service '{server_name}' not found, available: {available}")
|
||||
services = svc.get("services", {})
|
||||
services["_server_cdm"] = svc.get("server_cdm")
|
||||
services["_auth_headers"] = resolve_auth_headers(svc, server_name)
|
||||
@@ -774,7 +781,7 @@ def resolve_server(server_name: Optional[str]) -> tuple[str, str, dict]:
|
||||
return svc["url"], svc.get("api_key", ""), services
|
||||
|
||||
available = ", ".join(remote_services.keys())
|
||||
raise click.ClickException(f"Multiple remote services configured. Use --server to select one: {available}")
|
||||
raise click.ClickException(f"Multiple remote services configured, pick one with --server: {available}")
|
||||
|
||||
|
||||
REMOTE_LIST_TTL = 300.0
|
||||
@@ -937,7 +944,7 @@ def start_tunnel(client: RemoteClient) -> TunnelClient:
|
||||
try:
|
||||
tunnel.start()
|
||||
except ConnectionError as e:
|
||||
raise click.ClickException(f"--local-proxy needs a tunnel to the server. {e}")
|
||||
raise click.ClickException(f"--local-proxy could not open a tunnel to the server: {e}")
|
||||
client.tunnel = tunnel
|
||||
return tunnel
|
||||
|
||||
@@ -1024,9 +1031,11 @@ class ServerVault(Vault):
|
||||
``decrypt_verified`` needs a Vault to hold in ``Vaults.sources`` so a server-supplied
|
||||
content key gets the same decode check as a local vault key. The client never learns
|
||||
which server vault answered and can neither read nor write it, so every method returns
|
||||
nothing and a bad verdict goes to the server, which flags the row itself.
|
||||
nothing. A bad verdict goes to the server at once, so its next licence skips the pair.
|
||||
"""
|
||||
|
||||
report_now = True
|
||||
|
||||
def __init__(self, service: "RemoteService"):
|
||||
super().__init__("server_cdm", no_push=True)
|
||||
self.service = service
|
||||
@@ -1046,8 +1055,8 @@ class ServerVault(Vault):
|
||||
def get_services(self) -> Iterator[str]:
|
||||
return iter(())
|
||||
|
||||
def report_bad(self, kid: UUID, key: str) -> None:
|
||||
self.service.report_bad_key(kid, key)
|
||||
def flag_bad_key(self, service: str, kid: Union[UUID, str], key: str, source: str) -> None:
|
||||
self.service.report_bad_key(UUID(str(kid)), key)
|
||||
|
||||
|
||||
class RemoteService:
|
||||
@@ -1217,10 +1226,7 @@ class RemoteService:
|
||||
def require_reachable_proxy(self, proxy: Optional[str]) -> None:
|
||||
"""Refuse a proxy on this machine when the server is elsewhere: the server cannot reach it."""
|
||||
if proxy and is_loopback(proxy) and not is_loopback(self.client.server_url):
|
||||
raise click.ClickException(
|
||||
"That proxy is on your machine, so --remote cannot use it. "
|
||||
"To send the server's requests through this machine's connection, use --local-proxy without --proxy."
|
||||
)
|
||||
raise click.ClickException("--remote cannot use a proxy on this machine; use --local-proxy without --proxy")
|
||||
|
||||
def tunnel(self) -> str:
|
||||
"""The ``tunnel:<id>`` proxy to send the server for ``--local-proxy``, opening the tunnel the first time."""
|
||||
@@ -1274,9 +1280,8 @@ class RemoteService:
|
||||
self.log.info(f"Using a '{regions[0]}' proxy to match a server account ({', '.join(regions)})")
|
||||
except click.ClickException:
|
||||
self.log.warning(
|
||||
f"The server's accounts work in {', '.join(regions)} and you are in {client_region}. "
|
||||
"There is no local proxy for that region. Pass --proxy yourself if the server "
|
||||
"rejects the remote session"
|
||||
f"No local proxy for the server's account region ({', '.join(regions)}), "
|
||||
f"you are in {client_region}"
|
||||
)
|
||||
elif not no_proxy and proxy:
|
||||
resolved_proxy = resolve_remote_proxy_arg(server_proxy_arg(self.ctx.parent.params), providers)
|
||||
@@ -1651,8 +1656,7 @@ class RemoteService:
|
||||
for k, v in result.get("session_headers", {}).items():
|
||||
if k.lower() not in ("host", "content-length", "content-type"):
|
||||
self._session.headers[k] = v
|
||||
for k, v in result.get("session_cookies", {}).items():
|
||||
self._session.cookies.set(k, v)
|
||||
apply_session_cookies(self._session, result)
|
||||
|
||||
resolve_track_manifests(tracks, result.get("track_manifests", []))
|
||||
resolve_manifest_data(tracks, result.get("manifests", []))
|
||||
@@ -1808,25 +1812,23 @@ class RemoteService:
|
||||
else "no vault, neither the server's nor yours"
|
||||
)
|
||||
raise click.ClickException(
|
||||
f"The key for {track} is in {vaults}. This session runs on the server's device, which "
|
||||
"licenses nothing, so drop the service's server-identity option to license it with your own device."
|
||||
f"The content key for {track} is in {vaults}; drop the service's server-identity option to license it locally"
|
||||
)
|
||||
cap = refusal.get("max_height")
|
||||
if cap:
|
||||
why = f"The server CDM licenses up to {cap}p for this key"
|
||||
why = f"The server CDM licenses up to {cap}p for this API key"
|
||||
elif getattr(self, "_server_vault", False):
|
||||
why = "The server vault holds no key for this track"
|
||||
why = "The server vault holds no content key for this track"
|
||||
else:
|
||||
why = "The server CDM refused this track"
|
||||
cdm = self.local_cdm
|
||||
if cdm is None:
|
||||
raise click.ClickException(f"{why}, and {track} needs a local CDM to license it with your own device.")
|
||||
raise click.ClickException(f"{why}, and {track} needs a local CDM to license it")
|
||||
drm_class = "PlayReady" if is_playready_cdm(cdm) else "Widevine"
|
||||
if track.drm and not any(d.__class__.__name__ == drm_class for d in track.drm):
|
||||
height = getattr(track, "height", None)
|
||||
raise click.ClickException(
|
||||
f"{why}, and {track} carries no {drm_class} DRM for your local device to license. "
|
||||
f"Pass -q {height or 'with that height'} so the session starts on your own device."
|
||||
f"{why}, and {track} has no {drm_class} DRM for your local CDM; pass -q {height or 'with that height'}"
|
||||
)
|
||||
track.drm_preference = drm_class.lower()
|
||||
self.client_licensed.add(str(track.id))
|
||||
|
||||
+18
-49
@@ -25,7 +25,7 @@ from unshackle.core.constants import AnyTrack
|
||||
from unshackle.core.credential import Credential
|
||||
from unshackle.core.drm import DRM_T
|
||||
from unshackle.core.proxies.basic import Basic
|
||||
from unshackle.core.proxies.resolve import find_provider
|
||||
from unshackle.core.proxies.resolve import pick_proxy
|
||||
from unshackle.core.search_result import SearchResult
|
||||
from unshackle.core.session import (
|
||||
BACKOFF_FACTOR,
|
||||
@@ -229,40 +229,16 @@ class Service(metaclass=ABCMeta):
|
||||
self.log.info(
|
||||
f"Found service-specific proxy mapping: {full_proxy_key} -> {sanitize_proxy_for_log(mapped_value)}"
|
||||
)
|
||||
if proxy_provider_name:
|
||||
# Specific provider requested
|
||||
try:
|
||||
proxy_provider = find_provider(ctx.obj.proxy_providers, proxy_provider_name)
|
||||
except ValueError:
|
||||
proxy_provider = None
|
||||
if proxy_provider:
|
||||
mapped_proxy_uri = proxy_provider.get_proxy(mapped_value)
|
||||
if mapped_proxy_uri:
|
||||
proxy = mapped_proxy_uri
|
||||
self.log.info(
|
||||
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
|
||||
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
|
||||
)
|
||||
else:
|
||||
self.log.warning(
|
||||
f"Failed to get proxy for mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
|
||||
)
|
||||
else:
|
||||
self.log.warning(f"Proxy provider '{proxy_provider_name}' not found, using default proxy")
|
||||
else:
|
||||
for proxy_provider in ctx.obj.proxy_providers:
|
||||
mapped_proxy_uri = proxy_provider.get_proxy(mapped_value)
|
||||
if mapped_proxy_uri:
|
||||
proxy = mapped_proxy_uri
|
||||
self.log.info(
|
||||
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
|
||||
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
|
||||
)
|
||||
break
|
||||
else:
|
||||
self.log.warning(
|
||||
f"No provider could resolve mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
|
||||
)
|
||||
try:
|
||||
proxy_provider, proxy = pick_proxy(ctx.obj.proxy_providers, mapped_value, proxy_provider_name)
|
||||
self.log.info(
|
||||
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
|
||||
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
|
||||
)
|
||||
except ValueError:
|
||||
self.log.warning(
|
||||
f"No provider could resolve mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
|
||||
)
|
||||
|
||||
if not proxy:
|
||||
# don't override the explicit proxy set by the user, even if they may be geoblocked
|
||||
@@ -276,24 +252,18 @@ class Service(metaclass=ABCMeta):
|
||||
self.log.info("Service is not Geoblocked in your region")
|
||||
elif not (fence_targets := [x for x in self.GEOFENCE if self.is_region_allowed(x)]):
|
||||
raise click.ClickException(
|
||||
f"Service is not available in {whose} region ({current_region.upper()}). "
|
||||
"Pass --proxy with a proxy outside the blocked regions."
|
||||
f"Service is not available in {whose} region ({current_region.upper()})"
|
||||
)
|
||||
else:
|
||||
requested_proxy = fence_targets[0] # first is likely main region
|
||||
self.log.info(
|
||||
f"Service is Geoblocked in your region, getting a Proxy to {requested_proxy}"
|
||||
)
|
||||
for proxy_provider in ctx.obj.proxy_providers:
|
||||
proxy = proxy_provider.get_proxy(requested_proxy)
|
||||
if proxy:
|
||||
self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
|
||||
break
|
||||
if not proxy:
|
||||
self.log.warning(
|
||||
f"No proxy available for {requested_proxy}. "
|
||||
f"Pass --proxy with a proxy in {requested_proxy}, or the request can fail."
|
||||
)
|
||||
try:
|
||||
proxy_provider, proxy = pick_proxy(ctx.obj.proxy_providers, requested_proxy)
|
||||
self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
|
||||
except ValueError:
|
||||
self.log.warning(f"No proxy available for {requested_proxy}")
|
||||
except click.ClickException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -316,8 +286,7 @@ class Service(metaclass=ABCMeta):
|
||||
# Only GEOBLOCK applies here: an explicit --proxy outside GEOFENCE stays the user's call.
|
||||
if exit_region and any(x.lower() == exit_region.lower() for x in self.GEOBLOCK):
|
||||
raise click.ClickException(
|
||||
f"Service is not available in {whose} proxy's region ({exit_region.upper()}). "
|
||||
"Pass --proxy with a proxy outside the blocked regions."
|
||||
f"Service is not available in {whose} proxy's region ({exit_region.upper()})"
|
||||
)
|
||||
else:
|
||||
# No proxy, use cached IP info for title caching (non-critical)
|
||||
|
||||
@@ -332,10 +332,7 @@ def check_load_errors() -> None:
|
||||
"""
|
||||
if DIRTY_REPOS:
|
||||
joined = "\n".join(f" - {p}" for p in DIRTY_REPOS)
|
||||
raise click.ClickException(
|
||||
"Service repo has local changes - refusing to refresh so your edits are not lost.\n"
|
||||
"Commit and push them to the upstream repo (or revert them), then retry:\n" + joined
|
||||
)
|
||||
raise click.ClickException("Service repo has local changes, not refreshing:\n" + joined)
|
||||
global SUMMARY_LOGGED
|
||||
if not SUMMARY_LOGGED:
|
||||
SUMMARY_LOGGED = True
|
||||
@@ -422,7 +419,7 @@ class Services(click.Group):
|
||||
raise click.ClickException(
|
||||
f"There are no Services added yet, therefore the '{name}' Service could not be found."
|
||||
)
|
||||
raise click.ClickException(f"{e}. Available Services: {', '.join(available_services)}")
|
||||
raise click.ClickException(f"{e}, available: {', '.join(available_services)}")
|
||||
|
||||
if hasattr(service, "cli"):
|
||||
cli = service.cli
|
||||
@@ -518,8 +515,7 @@ class Services(click.Group):
|
||||
if services:
|
||||
available = ", ".join(sorted(svc["tag"] for svc in services))
|
||||
raise click.ClickException(
|
||||
f"The remote server does not offer a service named '{name}'. "
|
||||
f"It offers these services: {available}"
|
||||
f"The remote server does not offer a service named '{name}', available: {available}"
|
||||
)
|
||||
raise click.ClickException("The remote server offers no services to your API key.")
|
||||
tag = svc_info["tag"]
|
||||
|
||||
+67
-13
@@ -6,6 +6,7 @@ import http
|
||||
import logging
|
||||
import math
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator, MutableMapping
|
||||
@@ -17,6 +18,7 @@ from urllib.parse import urlencode, urlparse, urlunparse
|
||||
|
||||
import rnet
|
||||
from requests import HTTPError, Request, Response
|
||||
from requests.cookies import create_cookie
|
||||
from requests.structures import CaseInsensitiveDict
|
||||
|
||||
from unshackle.core.config import config
|
||||
@@ -50,11 +52,7 @@ def resolve_impersonate(browser: str) -> rnet.Impersonate:
|
||||
preset = getattr(rnet.Impersonate, browser, None)
|
||||
if preset is not None:
|
||||
return preset
|
||||
raise ValueError(
|
||||
f"Unknown impersonate preset: {browser!r}. "
|
||||
f"Use exact rnet preset names like 'Chrome131', 'OkHttp4_12', 'Edge101'. "
|
||||
f"See rnet.Impersonate for all available presets."
|
||||
)
|
||||
raise ValueError(f"Unknown impersonate preset {browser!r}, expected an rnet.Impersonate name such as 'Chrome131'")
|
||||
|
||||
|
||||
METHOD_MAP: dict[str, rnet.Method] = {
|
||||
@@ -530,6 +528,34 @@ class MaxRetriesError(Exception):
|
||||
self.__cause__ = cause
|
||||
|
||||
|
||||
# the innermost io error text in rnet's wreq::Error debug form, e.g. "stream closed because of a broken pipe"
|
||||
RNET_ERROR_TEXT_RE = re.compile(r'error: "([^"]+)"')
|
||||
|
||||
|
||||
def failure_response(exc: BaseException) -> Any:
|
||||
"""The HTTP response of a status failure, or None.
|
||||
|
||||
RnetSession wraps a forcelist HTTPError in MaxRetriesError as ``__cause__``, so this unwraps one level.
|
||||
"""
|
||||
response = getattr(exc, "response", None)
|
||||
return response if response is not None else getattr(exc.__cause__, "response", None)
|
||||
|
||||
|
||||
def failure_reason(exc: BaseException) -> str:
|
||||
"""Name a request failure without its URL, which can carry signed tokens.
|
||||
|
||||
Gives the HTTP status of a status failure, else the exception class and, for an rnet error,
|
||||
its innermost error text.
|
||||
"""
|
||||
inner = exc.__cause__ or exc
|
||||
response = failure_response(exc)
|
||||
status = getattr(response, "status_code", None)
|
||||
if isinstance(status, int):
|
||||
return f"{status} {getattr(response, 'reason', '') or ''}".strip()
|
||||
texts = RNET_ERROR_TEXT_RE.findall(str(inner))
|
||||
return f"{type(inner).__name__} ({texts[-1]})" if texts else type(inner).__name__
|
||||
|
||||
|
||||
class RnetSession:
|
||||
"""TLS-fingerprinted HTTP session powered by rnet (Rust/BoringSSL).
|
||||
|
||||
@@ -664,14 +690,19 @@ class RnetSession:
|
||||
self._client = client
|
||||
|
||||
@staticmethod
|
||||
def set_cookie_header(client: rnet.BlockingClient, origin: str, header: str) -> None:
|
||||
"""Set every cookie in a ``name=value; ...`` header string on one origin of *client*."""
|
||||
def parse_cookie_header(header: str) -> Iterator[tuple[str, str]]:
|
||||
"""The ``(name, value)`` pairs of a ``name=value; ...`` header string."""
|
||||
for pair in header.split(";"):
|
||||
name, separator, value = pair.partition("=")
|
||||
if not separator:
|
||||
continue
|
||||
if separator:
|
||||
yield name.strip(), value.strip()
|
||||
|
||||
@staticmethod
|
||||
def set_cookie_header(client: rnet.BlockingClient, origin: str, header: str) -> None:
|
||||
"""Set every cookie in a ``name=value; ...`` header string on one origin of *client*."""
|
||||
for name, value in RnetSession.parse_cookie_header(header):
|
||||
try:
|
||||
client.set_cookie(origin, rnet.Cookie(name.strip(), value.strip()))
|
||||
client.set_cookie(origin, rnet.Cookie(name, value))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -705,6 +736,26 @@ class RnetSession:
|
||||
exported[origin] = raw.decode("utf-8", errors="replace")
|
||||
return exported
|
||||
|
||||
def scoped_cookies(self) -> CookieJar:
|
||||
"""Every cookie this HTTP session sends, each scoped to its domain.
|
||||
|
||||
The live client's cookies of each requested origin come first, then the cookies set
|
||||
through :attr:`cookies` for any other host.
|
||||
"""
|
||||
jar = CookieJar()
|
||||
hosts: dict[str, set[str]] = {}
|
||||
for origin, header in self.export_origin_cookies().items():
|
||||
host = urlparse(origin).hostname or ""
|
||||
for name, value in self.parse_cookie_header(header):
|
||||
jar.set_cookie(create_cookie(name, value, domain=host))
|
||||
hosts.setdefault(name, set()).add(host)
|
||||
for domain, values in self.cookies.get_dict_by_domain().items():
|
||||
bare = domain.lstrip(".")
|
||||
for name, value in values.items():
|
||||
if not any(h == bare or h.endswith("." + bare) for h in hosts.get(name, ())):
|
||||
jar.set_cookie(create_cookie(name, value, domain=domain))
|
||||
return jar
|
||||
|
||||
def import_origin_cookies(self, exported: dict[str, str]) -> None:
|
||||
"""Load cookie headers from :meth:`export_origin_cookies` into this HTTP session's client."""
|
||||
if not exported:
|
||||
@@ -842,12 +893,12 @@ class RnetSession:
|
||||
if response.status_code not in self.status_forcelist:
|
||||
return response
|
||||
last_exception = HTTPError(f"Received status code: {response.status_code}", response=response)
|
||||
failure = f"{response.status_code} {response.reason}"
|
||||
failure = failure_reason(last_exception)
|
||||
|
||||
except self.catch_exceptions as e:
|
||||
last_exception = e
|
||||
response = None
|
||||
failure = e.__class__.__name__
|
||||
failure = failure_reason(e)
|
||||
|
||||
if attempt < max_retries:
|
||||
self.log.warning(f"{failure}({urlparse(url).path}). Retrying... ({attempt + 1}/{max_retries})")
|
||||
@@ -857,7 +908,10 @@ class RnetSession:
|
||||
else:
|
||||
break
|
||||
|
||||
raise MaxRetriesError(f"Max retries exceeded for {method} {url}", cause=last_exception)
|
||||
attempts = f" after {max_retries + 1} attempts" if max_retries else ""
|
||||
raise MaxRetriesError(
|
||||
f"{method_upper} {parsed_origin.hostname} failed{attempts}: {failure}", cause=last_exception
|
||||
)
|
||||
|
||||
def get(self, url: str, **kwargs: Any) -> RnetResponse:
|
||||
return self.request("GET", url, **kwargs)
|
||||
|
||||
@@ -386,10 +386,7 @@ class Hybrid:
|
||||
message="Video has no container duration; RPU keeps the active area of the DV track",
|
||||
context={"input": str(input_video)},
|
||||
)
|
||||
self.log.warning(
|
||||
"Video has no container duration, skipping crop detection. "
|
||||
"The RPU keeps the active area of the DV track."
|
||||
)
|
||||
self.log.warning("Video has no container duration, skipping crop detection")
|
||||
return None
|
||||
duration = float(duration_value)
|
||||
|
||||
@@ -665,7 +662,7 @@ class Hybrid:
|
||||
return
|
||||
|
||||
if not HDR10PlusTool:
|
||||
raise ValueError("HDR10Plus_tool not found. Please install it to use HDR10+ to DV conversion.")
|
||||
raise ValueError("HDR10Plus_tool not found, HDR10+ to DV conversion needs it")
|
||||
|
||||
try:
|
||||
run_step(
|
||||
@@ -817,10 +814,7 @@ class Hybrid:
|
||||
json.dump(generator_config, f)
|
||||
|
||||
if profile == "A":
|
||||
self.log.warning(
|
||||
"The HDR10+ metadata is Profile A, which is often less reliable than Profile B. "
|
||||
"Check the Dolby Vision result before you keep it."
|
||||
)
|
||||
self.log.warning("The HDR10+ metadata is Profile A, less reliable than Profile B")
|
||||
|
||||
try:
|
||||
dovi.generate(
|
||||
|
||||
@@ -1128,8 +1128,7 @@ class Subtitle(Track):
|
||||
except (IOError, OSError) as e:
|
||||
if "is not valid subtitle file" in str(e):
|
||||
log = logging.getLogger("Subtitle")
|
||||
log.warning(f"Failed to strip SDH from {self.path.name}: {e}")
|
||||
log.warning("Continuing without SDH stripping for this subtitle")
|
||||
log.warning(f"Failed to strip SDH from {self.path.name}, keeping it as is: {e}")
|
||||
else:
|
||||
raise
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ from unshackle.core.config import config
|
||||
from unshackle.core.constants import DOWNLOAD_ALL_DRM, DOWNLOAD_CANCELLED, DOWNLOAD_LICENCE_ONLY, DownloadCancelled
|
||||
from unshackle.core.downloaders import requests
|
||||
from unshackle.core.drm import DRM_T, ClearKeyCENC, PlayReady, Widevine
|
||||
from unshackle.core.drm.verify import decrypt_track
|
||||
from unshackle.core.drm.verify import decrypt_track, has_encrypted_entry, senc_protects_samples
|
||||
from unshackle.core.events import events
|
||||
from unshackle.core.session import MAX_BACKOFF, RnetSession, utf8_text
|
||||
from unshackle.core.temp import restore_segments, stash_segments
|
||||
@@ -244,42 +244,18 @@ def has_dts_uhd_sample_entry(path: Path) -> bool:
|
||||
|
||||
|
||||
def has_encrypted_sample_entry(path: Path) -> bool:
|
||||
"""True if the MP4's moov still carries an encrypted sample entry (encv/enca).
|
||||
"""True if the MP4's moov still carries an encrypted sample entry (``enc*`` or ``drm*``).
|
||||
|
||||
A faithful decrypt rewrites encv/enca back to the real codec 4CC through frma, so a
|
||||
survivor means the decrypt tool skipped/failed the track. This is a 4CC scan
|
||||
scoped to the moov box (where sample entries live), not a structural parse.
|
||||
The scoping avoids chance byte collisions in mdat. Any read error -> False.
|
||||
A faithful decrypt rewrites the entry back to the real codec 4CC through frma, so a
|
||||
survivor means the decrypt tool skipped/failed the track. Only the moov is read, and its
|
||||
stsd boxes are parsed, so byte runs in mdat or in metadata cannot match. Any read or
|
||||
parse error -> False.
|
||||
"""
|
||||
try:
|
||||
moov = read_top_level_box(path, b"moov")
|
||||
except OSError:
|
||||
return moov is not None and has_encrypted_entry(moov)
|
||||
except (OSError, struct.error, RecursionError):
|
||||
return False
|
||||
if not moov:
|
||||
return False
|
||||
for tok in (b"encv", b"enca"):
|
||||
i = moov.find(tok, 4)
|
||||
while i != -1:
|
||||
# require a plausible sample-entry box size right before the 4CC
|
||||
size = int.from_bytes(moov[i - 4 : i], "big")
|
||||
if 16 <= size <= len(moov) - (i - 4):
|
||||
return True
|
||||
i = moov.find(tok, i + 1)
|
||||
return False
|
||||
|
||||
|
||||
def senc_protects_samples(buf: bytes, body: int, end: int) -> bool:
|
||||
"""True if a senc/PIFF-uuid box describes protected samples.
|
||||
|
||||
An empty senc (sample_count 0) sits on a genuinely clear fragment, so treating it as
|
||||
a skipped one would raise on a healthy file. Field order follows
|
||||
read_per_sample_iv_size.
|
||||
"""
|
||||
flags = int.from_bytes(buf[body + 1 : body + 4], "big")
|
||||
pos = body + 4 + (20 if flags & 0x1 else 0) # PIFF override: AlgorithmID(3)+IV_size(1)+KID(16)
|
||||
if pos + 4 > end:
|
||||
return False
|
||||
return int.from_bytes(buf[pos : pos + 4], "big") > 0
|
||||
|
||||
|
||||
def moof_still_encrypted(moof: bytes) -> bool:
|
||||
@@ -370,9 +346,7 @@ def assert_fragments_decrypted(path: Path) -> None:
|
||||
first_offset=first_offset,
|
||||
)
|
||||
raise ValueError(
|
||||
f"{path.name}: {surviving}/{total} fragment(s) still encrypted after decryption (first at "
|
||||
f"byte {first_offset}). The decrypt tool skipped them silently, so check "
|
||||
f"tfhd.sample_description_index against the stsd entry count in the init segment."
|
||||
f"{path.name}: {surviving}/{total} fragment(s) still encrypted after decryption, first at byte {first_offset}"
|
||||
)
|
||||
|
||||
|
||||
@@ -553,9 +527,7 @@ class Track:
|
||||
return slice(None)
|
||||
low, high = (starts[0] + seconds * timescale for seconds in window)
|
||||
if low > starts[-1] and len(starts) > 1:
|
||||
raise click.ClickException(
|
||||
"The title ends before the sample window starts. Use --sample START-END with an earlier window."
|
||||
)
|
||||
raise click.ClickException("The title ends before the sample window starts, use an earlier --sample window")
|
||||
first = max(bisect_right(starts, low) - 1, 0)
|
||||
self.sample_start = (starts[first] - starts[0]) / timescale
|
||||
return slice(first, max(bisect_left(starts, high) + self.sample_tail, first + 1))
|
||||
@@ -1174,7 +1146,7 @@ class Track:
|
||||
drm_list.append(drm)
|
||||
elif fmt in {f"urn:uuid:{PR_PSSH.SYSTEM_ID}", "com.microsoft.playready"}:
|
||||
pssh_b64 = key.uri.split(",")[-1]
|
||||
drm = PlayReady(pssh=PR_PSSH(pssh_b64), pssh_b64=pssh_b64)
|
||||
drm = PlayReady.from_pssh_b64(pssh_b64)
|
||||
drm_list.append(drm)
|
||||
|
||||
if drm_list:
|
||||
|
||||
@@ -237,14 +237,7 @@ class Tracks:
|
||||
for track in flatten(tracks):
|
||||
if self.exists(by_id=track.id):
|
||||
if not warn_only:
|
||||
raise ValueError(
|
||||
"One or more of the provided Tracks is a duplicate. "
|
||||
"Track IDs must be unique but accurate using static values. The "
|
||||
"value should stay the same no matter when you request the same "
|
||||
"content. Use a value that has relation to the track content "
|
||||
"itself and is static or permanent and not random/RNG data that "
|
||||
"wont change each refresh or conflict in edge cases."
|
||||
)
|
||||
raise ValueError(f"Duplicate track ID {track.id}: track IDs must be unique and stable")
|
||||
duplicates += 1
|
||||
continue
|
||||
|
||||
@@ -573,11 +566,7 @@ class Tracks:
|
||||
"""
|
||||
mp4box = getattr(binaries, "MP4Box", None)
|
||||
if not mp4box:
|
||||
raise RuntimeError(
|
||||
"MP4Box (GPAC) is required to mux DTS-UHD (DTS:X Profile 2) audio but was not found. "
|
||||
"Install it with your package manager (e.g. `apt install gpac`), then check it with "
|
||||
"`unshackle env check`."
|
||||
)
|
||||
raise RuntimeError("MP4Box (GPAC) not found, muxing DTS-UHD (DTS:X Profile 2) audio needs it")
|
||||
|
||||
progress = progress or partial(lambda **kwargs: None)
|
||||
|
||||
|
||||
@@ -495,7 +495,7 @@ class SlowDelayRange(click.ParamType):
|
||||
|
||||
match = re.match(r"^(\d+)-(\d+)$", str(value))
|
||||
if not match:
|
||||
self.fail(f"'{value}' is not a valid range. Use format: MIN-MAX (e.g., 20-40)", param, ctx)
|
||||
self.fail(f"'{value}' is not a valid range, expected MIN-MAX such as 20-40", param, ctx)
|
||||
|
||||
low, high = int(match.group(1)), int(match.group(2))
|
||||
if low < 5:
|
||||
@@ -521,7 +521,7 @@ class SampleRange(click.ParamType):
|
||||
|
||||
match = re.match(r"^(\d+)-(\d+)$", str(value))
|
||||
if not match or int(match.group(1)) >= int(match.group(2)):
|
||||
self.fail(f"'{value}' is not a valid sample window. Use format: START-END (e.g., 60-120)", param, ctx)
|
||||
self.fail(f"'{value}' is not a valid sample window, expected START-END such as 60-120", param, ctx)
|
||||
|
||||
return (int(match.group(1)), int(match.group(2)))
|
||||
|
||||
|
||||
@@ -288,10 +288,7 @@ def verify_proxy_exit(session: requests.Session) -> dict:
|
||||
if len(errors) < len(build_providers()):
|
||||
log.warning("No geolocation API returned the proxy exit IP, so the exit IP check does not occur")
|
||||
return {}
|
||||
raise ConnectionError(
|
||||
f"Proxy check failed: no IP lookup got through the proxy ({type(errors[-1]).__name__}). "
|
||||
"The proxy is down or refused the connection."
|
||||
)
|
||||
raise ConnectionError(f"Proxy check failed: no IP lookup got through the proxy ({type(errors[-1]).__name__})")
|
||||
exit_ip = info.get("ip")
|
||||
server_ip = (get_ip_info(cached=True) or {}).get("ip")
|
||||
# The cached IP can be up to 24h old: a machine that was on a VPN when it cached its IP,
|
||||
|
||||
@@ -229,8 +229,7 @@ def dispatch(event: str, mode: str, context: dict[str, str], overrides: Sequence
|
||||
injected = _data_injected_flag(tokens, argv)
|
||||
if injected is not None:
|
||||
log.warning(
|
||||
"Skipping post-script: a substituted value expands to the option-like token %r; "
|
||||
"prefix the variable (--opt={var}) so service metadata cannot forge a flag",
|
||||
"Skipping post-script: a substituted value expands to the option-like token %r",
|
||||
redact_text(redact_path(injected)),
|
||||
)
|
||||
continue
|
||||
|
||||
@@ -131,6 +131,7 @@ def ffmpeg_decodes(path: Path, start: Optional[float] = None, seconds: float = 3
|
||||
return True
|
||||
if not keyframes and video is not False and any(m in err for m in FFMPEG_MISSING_REFERENCE):
|
||||
return decode(window, keyframes=True)
|
||||
log_tool_run("FFmpeg decode check", ffmpeg, ff.returncode, window=window, stderr=ff.stderr[-2000:])
|
||||
return False
|
||||
|
||||
tail = ["-sseof", f"-{FFMPEG_CHECK_TAIL}", "-t", str(FFMPEG_CHECK_TAIL)]
|
||||
|
||||
@@ -51,6 +51,7 @@ def vault_session(timeout: float) -> Session:
|
||||
class Vault(metaclass=ABCMeta):
|
||||
# a vault on this machine; Vaults.get_key tries these first, before the network ones
|
||||
local: bool = False
|
||||
report_now: bool = False
|
||||
|
||||
def __init__(self, name: str, no_push: bool = False):
|
||||
self.name = name
|
||||
|
||||
+34
-13
@@ -30,6 +30,7 @@ class Vaults:
|
||||
self.candidates: dict[Union[UUID, str], list[tuple[str, Vault]]] = {}
|
||||
self.warned: set[tuple[str, Union[UUID, str], str]] = set()
|
||||
self.unreachable: set[Vault] = set()
|
||||
self.unreported: dict[tuple[Union[UUID, str], str], Vault] = {}
|
||||
|
||||
def __iter__(self) -> Iterator[Vault]:
|
||||
return iter(self.vaults)
|
||||
@@ -153,22 +154,42 @@ class Vaults:
|
||||
return key if found else None
|
||||
|
||||
def flag_bad_key(self, kid: Union[UUID, str], key: str) -> None:
|
||||
"""Flag a KID:KEY that failed decryption, naming the Vault it came from.
|
||||
|
||||
Every local vault records it. A remote vault is told only when it served the pair,
|
||||
so it stops serving it too, and a vault that never held it is not bothered. A vault
|
||||
that cannot take the flag is skipped with a debug line: the local flag already
|
||||
protects this run, and a vault without the route is not a fault.
|
||||
"""
|
||||
"""Flag a KID:KEY in every local vault. A failed decode is no proof, so the remote vault that
|
||||
served it hears of it from ``report_replaced``, or at once when it sets ``report_now``."""
|
||||
self.flagged.add((kid, key))
|
||||
source = self.sources.pop(kid, None)
|
||||
name = source[1].name if source else "unknown"
|
||||
for vault in self.live():
|
||||
if not vault.local and (source is None or vault is not source[1]):
|
||||
continue
|
||||
try:
|
||||
vault.flag_bad_key(self.service, kid, key, source[1].name if source else "unknown")
|
||||
except Exception as e:
|
||||
log.debug(f"Could not flag {key} as bad in Vault '{vault.name}': {e!r}")
|
||||
if vault.local:
|
||||
self.tell(vault, kid, key, name)
|
||||
if source and not source[1].local:
|
||||
if source[1].report_now:
|
||||
self.tell(source[1], kid, key, name)
|
||||
else:
|
||||
self.unreported[kid, key] = source[1]
|
||||
|
||||
def report_replaced(self, content_keys: dict[Union[UUID, str], str]) -> None:
|
||||
"""Tell each remote vault about its flagged pair once ``content_keys`` holds another key for the KID."""
|
||||
live = self.live()
|
||||
for (kid, key), vault in list(self.unreported.items()):
|
||||
if content_keys.get(kid) not in (None, key):
|
||||
del self.unreported[kid, key]
|
||||
if vault in live:
|
||||
self.tell(vault, kid, key, vault.name)
|
||||
|
||||
def tell(self, vault: Vault, kid: Union[UUID, str], key: str, source: str) -> None:
|
||||
"""Flag one pair in one vault; a failure is only a debug line."""
|
||||
try:
|
||||
vault.flag_bad_key(self.service, kid, key, source)
|
||||
except Exception as e:
|
||||
log.debug(f"Could not flag {key} as bad in Vault '{vault.name}': {e!r}")
|
||||
|
||||
def clear_licensed_flags(self, content_keys: dict[UUID, str]) -> list[UUID]:
|
||||
"""Unflag each pair a licence returned, since the decode check was wrong. Returns their KIDs."""
|
||||
cleared = [kid for kid, key in content_keys.items() if self.is_flagged(kid, key)]
|
||||
for kid in cleared:
|
||||
self.unflag_bad_key(kid, content_keys[kid])
|
||||
return cleared
|
||||
|
||||
def unflag_bad_key(self, kid: Union[UUID, str], key: str) -> None:
|
||||
"""Remove a flag from every Vault."""
|
||||
|
||||
Reference in New Issue
Block a user