Compare commits

...
Author SHA1 Message Date
imSp4rky eeaf16cd83 fix(remote): keep each session cookie's domain, path and secure flag
The client set server cookies with no domain and sent all of them to every host, so a CDN refused segment requests with 400 header too large. rnet sessions now send cookies too.
2026-10-23 09:42:01 -06:00
imSp4rky af69e8aad4 feat(proxy): try bare --proxy providers in proxy_providers key order
Basic stays first and Control D last; the rest follow their config key order. Geofence and proxy_map lookups go through pick_proxy and skip a provider that fails.
2026-10-23 08:20:39 -06:00
imSp4rky 71f293161e fix(cfg): show only the duplicate-key warning, once 2026-10-22 20:43:40 -06:00
imSp4rky 3e9c26bd46 refactor(config): log the duplicate-key warning without the python warning prefix 2026-10-22 20:16:35 -06:00
imSp4rky 9dd97bcd25 feat(config): warn with the lines of a duplicate key in any config load 2026-10-22 20:12:59 -06:00
imSp4rky f83a594c91 fix(cfg): give the lines of each duplicate config key instead of crashing 2026-10-22 19:45:46 -06:00
20 changed files with 455 additions and 111 deletions
+4 -1
View File
@@ -1096,6 +1096,7 @@ Get tracks and chapters for a specific title within the remote session. Body: `{
"track_manifests": [ { "track_id": "s-1", "type": "dash", "url": "https://...", "data": "<base64 zlib xml>" } ],
"session_headers": { "User-Agent": "..." },
"session_cookies": { "sid": "..." },
"session_cookie_jar": [ { "name": "sid", "value": "...", "domain": ".example.com", "path": "/", "secure": true } ],
"server_cdm_type": "widevine"
}
```
@@ -1115,11 +1116,13 @@ Get the download URL, DRM info, headers, cookies, and raw track data for the sel
```json
{
"tracks": {
"v-1": { "descriptor": "DASH", "url": "https://...", "drm": [ ... ], "headers": { ... }, "cookies": { ... }, "data": { ... } }
"v-1": { "descriptor": "DASH", "url": "https://...", "drm": [ ... ], "headers": { ... }, "cookies": { ... }, "cookie_jar": [ ... ], "data": { ... } }
}
}
```
`cookies` gives names and values only. `cookie_jar` gives the same cookies with their domain, path and `secure` flag, in the `session_cookie_jar` form of the tracks response.
| Status | Error code | Meaning |
| --- | --- | --- |
| `200` | - | Segment info returned. |
+5 -3
View File
@@ -214,8 +214,10 @@ hood the client walks a remote session through its lifecycle.
downloading. A service can build an AdaptationSet of its own that the served
manifest does not contain. Each such track gets a one-AdaptationSet MPD in
`track_manifests`, and the client re-parses that instead. Any
`session_headers` / `session_cookies` the server used are merged into the
client's local HTTP session.
`session_headers` and cookies the server used are merged into the
client's local HTTP session. Each cookie in `session_cookie_jar` keeps its
domain, path and `secure` flag, so it goes only to its own hosts. `session_cookies` carries
the same cookies as names and values only, for older clients.
The tracks response repeats the title. A service can rewrite title fields inside
`get_tracks`, so the client takes this copy in place of the one from the title
@@ -653,7 +655,7 @@ remote session API and a good template for any client.
- **Flow.** `authenticate()` → `create` (+ poll `prompt` every 2s up to a 600s
deadline, answering `pending_input` prompts. With stdin closed the client logs an
error and raises `SystemExit(1)` instead of posting an empty answer) → `get_titles()` → `get_tracks()`
(merging returned `session_headers`/`session_cookies`, re-parsing `manifests`) →
(merging returned `session_headers`/`session_cookie_jar`, re-parsing `manifests`) →
get the licence in proxy mode or `server_cdm` mode → `close()` (`DELETE`, saving any returned
`cache`).
- **Server resolution.** `resolve_server()` reads `config.remote_services.<name>`
+1 -1
View File
@@ -339,7 +339,7 @@ unshackle cfg [KEY] [VALUE] [OPTIONS]
| `--list` | List all set configuration values. |
!!! warning "Comments are stripped"
Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error.
Writing a value through `cfg` rewrites the configuration file and **removes all comments** from it. If your config relies on comments, edit it by hand instead. Setting a value and using `--unset` together is an error. If a config key is set twice in the same block, every command warns with the line numbers of the two copies and uses the later copy. `cfg` stops without a change.
!!! example
```shell
+18 -21
View File
@@ -125,14 +125,18 @@ forwarder for it. See [Control D](#control-d).
### Provider load order
When you do a download without `--no-proxy`, unshackle instantiates every configured
proxy provider once, in this fixed order. It uses that same order for bare-query
resolution:
proxy provider once. It uses that same order for bare-query resolution:
**Basic → ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola → Control D**
1. Basic is always first, so your own proxies win.
2. The other proxy providers follow in the order of their keys under `proxy_providers` in
`unshackle.yaml`. Proxy providers that load without a config key, such as Hola, come
after the listed ones in this default order:
**ExpressVPN → NordVPN → Proton VPN → Surfshark → Windscribe → Gluetun → Hola**
3. Control D is always last.
So if you have both Basic and NordVPN configured and use `--proxy us`, a `us` entry in
your Basic config wins, because unshackle tries Basic first. To skip ahead to a specific
proxy provider, prefix the query (`--proxy nordvpn:us`).
So if you list `windscribevpn` above `nordvpn` and use `--proxy us`, a `us` entry in your
Basic config answers first, then Windscribe. To skip ahead to a specific proxy provider,
prefix the query (`--proxy nordvpn:us`).
If a proxy provider cannot load, for example because its server list cannot be fetched
or its config is not valid, unshackle logs a warning and continues with the other proxy
@@ -1025,21 +1029,14 @@ The returned proxy is HTTPS on **port 443**.
## Developer notes
!!! note "Two resolution paths (developers)"
unshackle has **two** proxy-resolution code paths that are intentionally not
identical:
- `unshackle/commands/dl.py` inlines its own logic and loads **all nine** providers,
including Windscribe and Gluetun.
- `unshackle/core/proxies/resolve.py::initialize_proxy_providers()` is the shared
resolver used by the REST API handlers and the remote-service client. It **omits**
Windscribe and Gluetun.
Both paths accept the same query grammar (a direct URI, a `provider:query`, a bare
country query tried against providers in order, or a `controld://` resolver), but the
set of available providers differs. Keep this in mind when you add a proxy provider:
an entry in `dl.py` does not make it available over the API, and the opposite is also
true.
!!! note "One resolution path (developers)"
`dl`, `search`, the REST API handlers and the remote-service client all load proxy
providers through `load_proxy_providers()` in `unshackle/core/proxies/resolve.py`.
The REST API and the remote-service client call it through
`initialize_proxy_providers()`, which leaves out Gluetun because Gluetun starts a
local Docker container. All paths accept the same query grammar: a direct URI, a
`provider:query`, a bare query tried against providers in order, or a `controld://`
resolver.
The remote-service client resolves `--proxy` with its own
`resolve_remote_proxy_arg()`. It changes a `controld:` query to a `controld://`
+5 -2
View File
@@ -61,8 +61,11 @@ headers:
- **Type:** `dict` &nbsp;·&nbsp; **Default:** `{}`
Proxy/VPN provider configuration. Each sub-key names a proxy provider, and unshackle passes its
block straight to that provider's constructor. See [Proxies & VPN](../../guide/proxies-and-vpn.md)
for the full proxy provider guide. Recognised proxy providers and their exit ports:
block straight to that provider's constructor. A bare `--proxy` query tries Basic first, then
the other proxy providers in the order of their sub-keys, and Control D last (see
[Provider load order](../../guide/proxies-and-vpn.md#provider-load-order)). See
[Proxies & VPN](../../guide/proxies-and-vpn.md) for the full proxy provider guide. Recognised proxy
providers and their exit ports:
| Provider | Config key | Credentials | Proxy scheme/port |
|----------|-----------|-------------|-------------------|
+60
View File
@@ -0,0 +1,60 @@
from __future__ import annotations
import logging
from pathlib import Path
import pytest
from click.testing import CliRunner
from unshackle.commands import cfg as cfg_module
@pytest.fixture
def config_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
path = tmp_path / "unshackle.yaml"
monkeypatch.setattr(cfg_module, "get_config_path", lambda: path)
return path
def test_a_duplicate_key_logs_its_lines_once_and_names_no_key_or_value(
config_file: Path, caplog: pytest.LogCaptureFixture
) -> None:
config_file.write_text(
"theme: rainbow\ncredentials:\n EXAMPLE: first-secret\n OTHER: other\n EXAMPLE: second-secret\n"
"serve:\n users:\n key-secret: {tier: a}\n key-secret: {tier: b}\n",
encoding="utf-8",
)
with caplog.at_level(logging.WARNING, logger="config"):
result = CliRunner().invoke(cfg_module.cfg, ["theme"])
CliRunner().invoke(cfg_module.cfg, ["theme"])
assert isinstance(result.exception, SystemExit), result.exception
assert result.exit_code == 1
assert caplog.messages == [
f"{config_file}: duplicate key on lines 3 and 5, lines 8 and 9. Unshackle uses the later one."
]
assert "Error" not in result.output and "lines" not in result.output
for output in (result.output, caplog.text):
assert "secret" not in output and "EXAMPLE" not in output
def test_a_duplicate_key_blocks_a_write(config_file: Path) -> None:
original = "theme: rainbow\ncredentials:\n EXAMPLE: a\n EXAMPLE: b\n"
config_file.write_text(original, encoding="utf-8")
result = CliRunner().invoke(cfg_module.cfg, ["theme", "plain"])
assert result.exit_code == 1
assert config_file.read_text(encoding="utf-8") == original
def test_a_valid_config_still_reads_a_key(config_file: Path) -> None:
config_file.write_text("theme: rainbow\n", encoding="utf-8")
result = CliRunner().invoke(cfg_module.cfg, ["theme"])
assert result.exit_code == 0, result.output
assert "rainbow" in result.output
def test_a_duplicate_only_ruamel_sees_still_gives_its_line(config_file: Path, caplog: pytest.LogCaptureFixture) -> None:
config_file.write_text("a: 1\n1: x\n0x1: y\n", encoding="utf-8")
with caplog.at_level(logging.WARNING):
result = CliRunner().invoke(cfg_module.cfg, ["a"])
assert result.exit_code == 1
assert caplog.messages == [f"{config_file}: duplicate key on line 3. Unshackle uses the later one."]
+53
View File
@@ -0,0 +1,53 @@
from __future__ import annotations
import logging
from pathlib import Path
import pytest
import yaml
from ruamel.yaml import YAML
from unshackle.core.config import Config, duplicate_key_lines, load_yaml
ALIASES = "base: &b\n x: 1\n x: 2\n self: *b\nother: *b\nlist:\n - y: 1\n y: 2\n"
@pytest.mark.parametrize("compose", [yaml.compose, YAML().compose], ids=["pyyaml", "ruamel"])
def test_duplicate_key_lines_follows_aliases_without_looping(compose) -> None:
assert duplicate_key_lines(compose(ALIASES)) == [(2, 3), (7, 8)]
def test_load_yaml_warns_once_with_lines_and_keeps_the_later_copy(
tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
path = tmp_path / "unshackle.yaml"
path.write_text("credentials:\n EXAMPLE: first-secret\n EXAMPLE: second-secret\n", encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="config"):
data = load_yaml(path)
load_yaml(path)
assert data == {"credentials": {"EXAMPLE": "second-secret"}}
assert caplog.messages == [f"{path}: duplicate key on lines 2 and 3. Unshackle uses the later one."]
assert "EXAMPLE" not in caplog.text and "secret" not in caplog.text
def test_config_from_yaml_warns_on_a_duplicate_key(tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
path = tmp_path / "unshackle.yaml"
path.write_text("tag: first\ntag: second\n", encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="config"):
assert Config.from_yaml(path).tag == "second"
assert "duplicate key on lines 1 and 2" in caplog.text
@pytest.mark.parametrize(
"text",
["a: 1\nb: [1, 2]\n", "", "a: &a\n x: 1\nb: &b\n y: 2\nc:\n <<: *a\n <<: *b\n"],
ids=["valid", "empty", "two-merges"],
)
def test_load_yaml_matches_safe_load_without_a_warning(
tmp_path: Path, text: str, caplog: pytest.LogCaptureFixture
) -> None:
path = tmp_path / "config.yaml"
path.write_text(text, encoding="utf-8")
with caplog.at_level(logging.WARNING, logger="config"):
assert load_yaml(path) == yaml.safe_load(text)
assert not caplog.records
+11
View File
@@ -63,6 +63,17 @@ def test_raise_errors_raises_the_load_failure() -> None:
load_proxy_providers(OTHERS, raise_errors=True)
def test_bare_query_order_follows_config_key_order_with_controld_last(monkeypatch: pytest.MonkeyPatch) -> None:
from unshackle.core.proxies import ControlD
order = {"controld": {"token": "t"}, "surfsharkvpn": CREDENTIALS, "nordvpn": CREDENTIALS, "basic": {"us": "x"}}
monkeypatch.setattr(config, "proxy_providers", order)
monkeypatch.setattr(ControlD, "get_locations", lambda self: [])
loaded = load_proxy_providers(("expressvpn", "protonvpn", "hola"), quiet=True)
names = [getattr(p, "name", p.__class__.__name__) for p in loaded]
assert names == ["Basic", "SurfsharkVPN", "NordVPN", "ControlD"]
def test_rest_path_loads_windscribe_but_not_gluetun(monkeypatch: pytest.MonkeyPatch) -> None:
from unshackle.core.proxies import Gluetun, WindscribeVPN
from unshackle.core.proxies.resolve import initialize_proxy_providers
+1 -1
View File
@@ -389,7 +389,7 @@ def test_a_lent_device_session_keeps_its_auth_headers_and_track_secrets() -> Non
headers, cookies = handlers.client_session_auth(session)
assert headers == {"User-Agent": "ua"}
assert cookies == {"account": "client"}
assert handlers.cookie_values(cookies) == {"account": "client"}
assert handlers.scrub_track_data({"token": "server", "id": 1}, session) == {"id": 1}
+120 -3
View File
@@ -1,13 +1,15 @@
from types import SimpleNamespace
import pytest
import requests
from unshackle.core.api.errors import APIError
from unshackle.core.api.handlers import client_session_auth, resolve_handler_proxy, scrub_track_data
from unshackle.core.remote_service import apply_session_cookies
def _session(client_auth):
cookie = SimpleNamespace(name="sid", value="secret")
cookie = SimpleNamespace(name="sid", value="secret", domain=".example.com", path="/")
svc = SimpleNamespace(headers={"Authorization": "Bearer x", "User-Agent": "ua"}, cookies=[cookie])
return SimpleNamespace(service_instance=SimpleNamespace(session=svc), client_auth=client_auth)
@@ -15,14 +17,129 @@ def _session(client_auth):
def test_client_login_keeps_auth():
headers, cookies = client_session_auth(_session(True))
assert headers == {"Authorization": "Bearer x", "User-Agent": "ua"}
assert cookies == {"sid": "secret"}
assert cookies == [{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/", "secure": False}]
assert scrub_track_data({"token": "t", "hls": {}}, _session(True)) == {"token": "t", "hls": {}}
def test_client_cookies_keep_their_domain():
session = requests.Session()
_, cookies = client_session_auth(_session(True))
apply_session_cookies(session, {"session_cookie_jar": cookies})
assert session.cookies.get("sid", domain=".example.com") == "secret"
cdn = session.prepare_request(requests.Request("GET", "https://cdn.other.net/seg.mp4"))
assert "Cookie" not in cdn.headers
own = session.prepare_request(requests.Request("GET", "https://www.example.com/api"))
assert own.headers["Cookie"] == "sid=secret"
def test_client_takes_old_server_cookies_without_domain():
session = requests.Session()
apply_session_cookies(session, {"session_cookies": {"sid": "secret"}})
assert session.cookies.get("sid") == "secret"
def test_client_keeps_secure_cookies_off_http():
session = requests.Session()
jar = [{"name": "sid", "value": "s", "domain": ".example.com", "path": "/", "secure": True}]
apply_session_cookies(session, {"session_cookie_jar": jar})
plain = session.prepare_request(requests.Request("GET", "http://cdn.example.com/seg.mp4"))
assert "Cookie" not in plain.headers
tls = session.prepare_request(requests.Request("GET", "https://cdn.example.com/seg.mp4"))
assert tls.headers["Cookie"] == "sid=s"
def test_client_skips_malformed_server_cookies():
session = requests.Session()
jar = [{"value": "v"}, "str", {"name": "n", "value": 1}, {"name": "ok", "value": "v", "domain": ".example.com"}]
apply_session_cookies(session, {"session_cookie_jar": jar})
assert [c.name for c in session.cookies] == ["ok"]
session = requests.Session()
apply_session_cookies(session, {"session_cookie_jar": 5, "session_cookies": ["x"]})
assert not session.cookies
def test_rnet_session_sends_client_and_set_cookies():
from unshackle.core.session import RnetSession
svc = RnetSession()
svc.cookies.set("sid", "stale", domain=".example.com")
svc.cookies.set("cdn", "c", domain="cdn.other.net")
svc.export_origin_cookies = lambda: {"https://www.example.com": "sid=live; cf=1"}
_, cookies = client_session_auth(SimpleNamespace(service_instance=SimpleNamespace(session=svc), client_auth=True))
assert {(c["name"], c["value"], c["domain"]) for c in cookies} == {
("sid", "live", "www.example.com"),
("cf", "1", "www.example.com"),
("cdn", "c", "cdn.other.net"),
}
def test_tracks_response_carries_the_cookie_jar():
import asyncio
import json
from unshackle.core.api import session_store as store_mod
from unshackle.core.api.handlers import session_tracks_handler
from unshackle.core.tracks import Tracks
cookie = SimpleNamespace(name="sid", value="secret", domain=".example.com", path="/", secure=True)
class StubService:
session = SimpleNamespace(headers={}, cookies=[cookie])
def get_tracks(self, title):
return Tracks()
def get_chapters(self, title):
raise NotImplementedError
async def run():
store = store_mod.get_session_store()
entry = await store.create(
service_tag="EXAMPLE", service_instance=StubService(), session_id="jar", owner_key=None
)
entry.title_map = {"t1": "Example Title"}
entry.client_auth = True
try:
return json.loads((await session_tracks_handler({"title_id": "t1"}, "jar")).body)
finally:
await store.delete("jar")
body = asyncio.run(run())
assert body["session_cookies"] == {"sid": "secret"}
assert body["session_cookie_jar"] == [
{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/", "secure": True}
]
def test_remote_get_tracks_scopes_server_cookies():
import logging
from unshackle.core.remote_service import RemoteService
jar = [{"name": "sid", "value": "secret", "domain": ".example.com", "path": "/"}]
service = object.__new__(RemoteService)
service.__dict__.update(
_tracks_by_title={},
_chapters_by_title={},
_sample_decrypt_tracks=set(),
_server_cdm=None,
_session=requests.Session(),
_session_id="s",
service_tag="EXAMPLE",
log=logging.getLogger("test"),
client=SimpleNamespace(post=lambda path, body: {"session_cookie_jar": jar}),
drain_server_logs=lambda: None,
)
service.get_tracks(SimpleNamespace(id="t1"))
cdn = service._session.prepare_request(requests.Request("GET", "https://cdn.other.net/seg.mp4"))
assert "Cookie" not in cdn.headers
assert service._session.cookies.get("sid", domain=".example.com") == "secret"
def test_server_login_strips_auth():
headers, cookies = client_session_auth(_session(False))
assert headers == {"User-Agent": "ua"}
assert cookies == {}
assert cookies == []
assert scrub_track_data({"token": "t", "hls": {}}, _session(False)) == {"hls": {}}
+12 -2
View File
@@ -4,8 +4,9 @@ import sys
import click
from ruamel.yaml import YAML
from ruamel.yaml.constructor import DuplicateKeyError
from unshackle.core.config import config, get_config_path
from unshackle.core.config import config, duplicate_key_lines, get_config_path, load_yaml
from unshackle.core.constants import context_settings
@@ -41,7 +42,16 @@ def cfg(ctx: click.Context, key: str, value: str, unset: bool, list_: bool) -> N
config_path = get_config_path() or config.directories.user_configs / config.filenames.root_config
if config_path.exists():
data = yaml.load(config_path)
try:
data = yaml.load(config_path)
except DuplicateKeyError as e:
if duplicate_key_lines(yaml.compose(config_path)):
load_yaml(config_path)
else:
log.warning(
f"{config_path}: duplicate key on line {e.problem_mark.line + 1}. Unshackle uses the later one."
)
sys.exit(1)
if not data:
log.warning("No config file was found or it has no data, yet")
+2 -3
View File
@@ -27,7 +27,6 @@ from uuid import UUID
import click
import mediaexport
import yaml
from click.core import ParameterSource
from langcodes import Language, tag_is_valid
from pymediainfo import MediaInfo
@@ -43,7 +42,7 @@ from rich.tree import Tree
from unshackle.core import __version__, binaries, providers
from unshackle.core.cdm import DecryptLabsRemoteCDM
from unshackle.core.cdm.detect import cdm_type_stub, is_playready_cdm, is_remote_cdm, is_widevine_cdm
from unshackle.core.config import cdm_entry_names_device, config, resolve_cdm_name, resolve_decryption
from unshackle.core.config import cdm_entry_names_device, config, load_yaml, resolve_cdm_name, resolve_decryption
from unshackle.core.console import GradientPulseBarColumn, SyncLive, console, listing_panel
from unshackle.core.constants import (
DEFAULT_SAMPLE,
@@ -1454,7 +1453,7 @@ class dl:
try:
service_config_path = Services.get_path(self.service) / config.filenames.config
if service_config_path.exists():
self.service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
self.service_config = load_yaml(service_config_path)
self.log.info("Service Config loaded")
# log key names only: the full config carries service certificates,
# device fingerprints and endpoints that bloat the log and may be sensitive
+2 -3
View File
@@ -6,13 +6,12 @@ import sys
from typing import Any, Optional
import click
import yaml
from rich.padding import Padding
from rich.rule import Rule
from rich.tree import Tree
from unshackle.commands.dl import dl
from unshackle.core.config import config
from unshackle.core.config import config, load_yaml
from unshackle.core.console import console
from unshackle.core.constants import context_settings
from unshackle.core.proxies.proxy import Proxy
@@ -91,7 +90,7 @@ def search(
with console.status("Loading Service Config...", spinner="dots"):
service_config_path = Services.get_path(service) / config.filenames.config
if service_config_path.exists():
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
service_config = load_yaml(service_config_path)
log.info("Service Config loaded")
merge_dict(ci_get(config.services, service), service_config)
+2 -3
View File
@@ -371,11 +371,10 @@ def perform_download(
# Import dl.py components lazily to avoid circular deps during module import
import click
import yaml
from unshackle.commands.dl import dl
from unshackle.core.api.errors import APIError, APIErrorCode
from unshackle.core.config import config
from unshackle.core.config import config, load_yaml
from unshackle.core.services import Services
from unshackle.core.tracks import Video
from unshackle.core.utils.click_types import ContextData
@@ -467,7 +466,7 @@ def perform_download(
remote = params.get("remote")
service_config_path = None if remote else Services.get_path(service) / config.filenames.config
if service_config_path and service_config_path.exists():
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8"))
service_config = load_yaml(service_config_path)
else:
service_config = {}
merge_dict(config.services.get(service), service_config)
+30 -16
View File
@@ -28,13 +28,13 @@ from unshackle.core.api.session_log import SessionLogBuffer, SessionLogMirror, c
from unshackle.core.api.session_store import SessionStore
from unshackle.core.cacher import Cacher
from unshackle.core.cdm.detect import cdm_type_stub
from unshackle.core.config import config
from unshackle.core.config import config, load_yaml
from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_CODEC_MAP
from unshackle.core.drm import cbcs
from unshackle.core.providers.anilist import parse_anilist_ref
from unshackle.core.proxies.resolve import initialize_proxy_providers, resolve_proxy
from unshackle.core.services import Services
from unshackle.core.session import utf8_text
from unshackle.core.session import RnetSession, utf8_text
from unshackle.core.titles import Episode, Movie, Song, Title_T
from unshackle.core.titles.music import SONG_FIELDS
from unshackle.core.tracks import Audio, Subtitle, Tracks, Video
@@ -190,13 +190,11 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]
def load_service_yaml(normalized_service: str) -> dict:
"""Load a service's config.yaml and merge it with the global override block."""
import yaml
from unshackle.core.utils.collections import merge_dict
service_config_path = Services.get_path(normalized_service) / config.filenames.config
if service_config_path.exists():
service_config = yaml.safe_load(service_config_path.read_text(encoding="utf8")) or {}
service_config = load_yaml(service_config_path) or {}
else:
service_config = {}
merge_dict(config.services.get(normalized_service), service_config)
@@ -4070,8 +4068,8 @@ async def session_titles_handler(session_id: str, request: Optional[web.Request]
)
def client_session_auth(session: Any) -> tuple[Dict[str, str], Dict[str, str]]:
"""The service session headers and cookies a remote client may receive.
def client_session_auth(session: Any) -> tuple[Dict[str, str], list[Dict[str, Any]]]:
"""The service session headers and cookies (name, value, domain, path, secure) a remote client may receive.
A client whose own login succeeded gets them back. For a server account the cookie jar
stays on the server and the server drops the auth-bearing headers, so the client cannot
@@ -4080,18 +4078,31 @@ def client_session_auth(session: Any) -> tuple[Dict[str, str], Dict[str, str]]:
"""
svc_session = session.service_instance.session
headers = dict(svc_session.headers) if hasattr(svc_session, "headers") and svc_session.headers else {}
cookies: Dict[str, str] = {}
if hasattr(svc_session, "cookies"):
for cookie in svc_session.cookies:
if hasattr(cookie, "name") and hasattr(cookie, "value"):
cookies[cookie.name] = cookie.value
cookies: list[Dict[str, Any]] = []
jar = svc_session.scoped_cookies() if isinstance(svc_session, RnetSession) else getattr(svc_session, "cookies", [])
for cookie in jar:
if hasattr(cookie, "name") and hasattr(cookie, "value"):
cookies.append(
{
"name": cookie.name,
"value": cookie.value,
"domain": getattr(cookie, "domain", None) or "",
"path": getattr(cookie, "path", None) or "/",
"secure": bool(getattr(cookie, "secure", False)),
}
)
if not client_owns_login(session):
headers = {k: v for k, v in headers.items() if not CONFIG_SECRET_KEY_RE.search(str(k))}
if not session.client_auth:
cookies = {}
cookies = []
return headers, cookies
def cookie_values(cookies: list[Dict[str, Any]]) -> Dict[str, str]:
"""The ``{name: value}`` form of :func:`client_session_auth` cookies, for clients that predate the jar."""
return {c["name"]: c["value"] for c in cookies}
def client_owns_login(session: Any) -> bool:
"""Whether the login state of ``session`` is the client's to take back.
@@ -4234,7 +4245,8 @@ async def session_tracks_handler(
"manifests": manifests,
"track_manifests": track_manifests,
"session_headers": session_headers,
"session_cookies": session_cookies,
"session_cookies": cookie_values(session_cookies),
"session_cookie_jar": session_cookies,
"server_cdm": session.server_cdm,
"server_cdm_type": server_cdm_type,
"sample_decrypt_tracks": [
@@ -4271,6 +4283,7 @@ async def session_segments_handler(
try:
result: Dict[str, Any] = {}
headers, cookies = client_session_auth(session)
for track_id in track_ids:
track = session.tracks.get(track_id)
@@ -4287,10 +4300,11 @@ async def session_segments_handler(
"descriptor": descriptor_name,
"url": str(track.url) if track.url else None,
"drm": serialize_drm(track.drm) if hasattr(track, "drm") and track.drm else None,
"headers": headers,
"cookies": cookie_values(cookies),
"cookie_jar": cookies,
}
track_info["headers"], track_info["cookies"] = client_session_auth(session)
# Include manifest-specific data for segment resolution. Round-trip through
# JSON so any non-serializable value becomes its str() (default=str).
if hasattr(track, "data") and track.data:
+51 -1
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import logging
import re
import warnings
from pathlib import Path
@@ -374,7 +375,56 @@ class Config:
raise FileNotFoundError(f"Config file path ({path}) was not found")
if not path.is_file():
raise FileNotFoundError(f"Config file path ({path}) is not to a file.")
return cls(**restore_bool_languages(yaml.safe_load(path.read_text(encoding="utf8")) or {}))
return cls(**restore_bool_languages(load_yaml(path) or {}))
def duplicate_key_lines(root: Any) -> list[tuple[int, int]]:
"""
Return (first line, repeat line), counted from 1, for each repeated scalar key in a PyYAML or ruamel node tree.
Keys compare as written, so two spellings that load as the same key (yes and true, 1 and 0x1) are not found.
"""
pairs, stack, visited = [], [root], set()
while stack:
node = stack.pop()
if node in visited:
continue
visited.add(node)
if node.id == "mapping":
first: dict[tuple[str, str], int] = {}
for key_node, value_node in node.value:
if key_node.id == "scalar" and key_node.tag != "tag:yaml.org,2002:merge":
key = (key_node.tag, key_node.value)
line = key_node.start_mark.line + 1
if key in first:
pairs.append((first[key], line))
else:
first[key] = line
stack += [key_node, value_node]
elif node.id == "sequence":
stack += node.value
return sorted(pairs)
_warned: set[str] = set()
def load_yaml(path: Path) -> Any:
"""yaml.safe_load a file, and warn with the lines of each duplicate key, which PyYAML takes silently."""
loader = yaml.SafeLoader(path.read_text(encoding="utf8"))
try:
node = loader.get_single_node()
if node is None:
return None
if pairs := duplicate_key_lines(node):
where = ", ".join(f"lines {first} and {repeat}" for first, repeat in pairs)
message = f"{path}: duplicate key on {where}. Unshackle uses the later one."
if message not in _warned:
_warned.add(message)
logging.getLogger("config").warning(message)
return loader.construct_document(node)
finally:
loader.dispose()
# noinspection PyProtectedMember
+10 -3
View File
@@ -54,7 +54,7 @@ def find_provider(proxy_providers: List[Any], name: str, hint: str = "") -> Any:
def load_proxy_providers(exclude: tuple[str, ...] = (), raise_errors: bool = False, quiet: bool = False) -> List[Any]:
"""
Build the proxy providers set in config, in the order a bare query tries them.
Build the proxy providers set in config: Basic first, then config key order, Control D last.
Each proxy provider loads on its own. One that fails is logged and kept as Unavailable, so a bare
query skips it and a query that names it reports why. *raise_errors* raises the failure instead.
@@ -75,8 +75,15 @@ def load_proxy_providers(exclude: tuple[str, ...] = (), raise_errors: bool = Fal
)
settings = config.proxy_providers
# Control D is last: a bare region reaches it only when nothing else answers, since it changes the account.
classes = (Basic, ExpressVPN, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN, Gluetun, Hola, ControlD)
rank = {key: i for i, key in enumerate(settings)}
classes = [
Basic,
*sorted(
(ExpressVPN, NordVPN, ProtonVPN, SurfsharkVPN, WindscribeVPN, Gluetun, Hola),
key=lambda c: rank.get(c.__name__.lower(), len(rank)),
),
ControlD,
]
def build(cls: type) -> Any:
key = cls.__name__.lower()
+22 -2
View File
@@ -426,6 +426,27 @@ def apply_service_track_data(tracks: Tracks, data: Dict[str, Any]) -> None:
track.data.update(extra)
def apply_session_cookies(session: requests.Session, result: dict) -> None:
"""Copy the server's session cookies into ``session``, each scoped to its own domain.
An older server sends only ``session_cookies`` names and values, so those cookies go to every host.
It skips malformed entries.
"""
jar = result.get("session_cookie_jar")
if not isinstance(jar, list):
old = result.get("session_cookies")
jar = [{"name": k, "value": v} for k, v in old.items()] if isinstance(old, dict) else []
for c in jar:
if isinstance(c, dict) and isinstance(c.get("name"), str) and isinstance(c.get("value"), str):
session.cookies.set(
c["name"],
c["value"],
domain=str(c.get("domain") or ""),
path=str(c.get("path") or "/"),
secure=bool(c.get("secure")),
)
def resolve_track_manifests(tracks: Tracks, track_manifests: list) -> None:
"""Fill track.data from the per-track MPD fragments the server sent.
@@ -1651,8 +1672,7 @@ class RemoteService:
for k, v in result.get("session_headers", {}).items():
if k.lower() not in ("host", "content-length", "content-type"):
self._session.headers[k] = v
for k, v in result.get("session_cookies", {}).items():
self._session.cookies.set(k, v)
apply_session_cookies(self._session, result)
resolve_track_manifests(tracks, result.get("track_manifests", []))
resolve_manifest_data(tracks, result.get("manifests", []))
+15 -41
View File
@@ -25,7 +25,7 @@ from unshackle.core.constants import AnyTrack
from unshackle.core.credential import Credential
from unshackle.core.drm import DRM_T
from unshackle.core.proxies.basic import Basic
from unshackle.core.proxies.resolve import find_provider
from unshackle.core.proxies.resolve import pick_proxy
from unshackle.core.search_result import SearchResult
from unshackle.core.session import (
BACKOFF_FACTOR,
@@ -229,40 +229,16 @@ class Service(metaclass=ABCMeta):
self.log.info(
f"Found service-specific proxy mapping: {full_proxy_key} -> {sanitize_proxy_for_log(mapped_value)}"
)
if proxy_provider_name:
# Specific provider requested
try:
proxy_provider = find_provider(ctx.obj.proxy_providers, proxy_provider_name)
except ValueError:
proxy_provider = None
if proxy_provider:
mapped_proxy_uri = proxy_provider.get_proxy(mapped_value)
if mapped_proxy_uri:
proxy = mapped_proxy_uri
self.log.info(
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
)
else:
self.log.warning(
f"Failed to get proxy for mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
)
else:
self.log.warning(f"Proxy provider '{proxy_provider_name}' not found, using default proxy")
else:
for proxy_provider in ctx.obj.proxy_providers:
mapped_proxy_uri = proxy_provider.get_proxy(mapped_value)
if mapped_proxy_uri:
proxy = mapped_proxy_uri
self.log.info(
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
)
break
else:
self.log.warning(
f"No provider could resolve mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
)
try:
proxy_provider, proxy = pick_proxy(ctx.obj.proxy_providers, mapped_value, proxy_provider_name)
self.log.info(
f"Using mapped proxy from {proxy_provider.__class__.__name__}: "
f"{sanitize_proxy_for_log(proxy, mask_host=isinstance(proxy_provider, Basic))}"
)
except ValueError:
self.log.warning(
f"No provider could resolve mapped value '{sanitize_proxy_for_log(mapped_value)}', using default"
)
if not proxy:
# don't override the explicit proxy set by the user, even if they may be geoblocked
@@ -284,12 +260,10 @@ class Service(metaclass=ABCMeta):
self.log.info(
f"Service is Geoblocked in your region, getting a Proxy to {requested_proxy}"
)
for proxy_provider in ctx.obj.proxy_providers:
proxy = proxy_provider.get_proxy(requested_proxy)
if proxy:
self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
break
if not proxy:
try:
proxy_provider, proxy = pick_proxy(ctx.obj.proxy_providers, requested_proxy)
self.log.info(f"Got Proxy from {proxy_provider.__class__.__name__}")
except ValueError:
self.log.warning(
f"No proxy available for {requested_proxy}. "
f"Pass --proxy with a proxy in {requested_proxy}, or the request can fail."
+31 -5
View File
@@ -17,6 +17,7 @@ from urllib.parse import urlencode, urlparse, urlunparse
import rnet
from requests import HTTPError, Request, Response
from requests.cookies import create_cookie
from requests.structures import CaseInsensitiveDict
from unshackle.core.config import config
@@ -664,14 +665,19 @@ class RnetSession:
self._client = client
@staticmethod
def set_cookie_header(client: rnet.BlockingClient, origin: str, header: str) -> None:
"""Set every cookie in a ``name=value; ...`` header string on one origin of *client*."""
def parse_cookie_header(header: str) -> Iterator[tuple[str, str]]:
"""The ``(name, value)`` pairs of a ``name=value; ...`` header string."""
for pair in header.split(";"):
name, separator, value = pair.partition("=")
if not separator:
continue
if separator:
yield name.strip(), value.strip()
@staticmethod
def set_cookie_header(client: rnet.BlockingClient, origin: str, header: str) -> None:
"""Set every cookie in a ``name=value; ...`` header string on one origin of *client*."""
for name, value in RnetSession.parse_cookie_header(header):
try:
client.set_cookie(origin, rnet.Cookie(name.strip(), value.strip()))
client.set_cookie(origin, rnet.Cookie(name, value))
except Exception:
pass
@@ -705,6 +711,26 @@ class RnetSession:
exported[origin] = raw.decode("utf-8", errors="replace")
return exported
def scoped_cookies(self) -> CookieJar:
"""Every cookie this HTTP session sends, each scoped to its domain.
The live client's cookies of each requested origin come first, then the cookies set
through :attr:`cookies` for any other host.
"""
jar = CookieJar()
hosts: dict[str, set[str]] = {}
for origin, header in self.export_origin_cookies().items():
host = urlparse(origin).hostname or ""
for name, value in self.parse_cookie_header(header):
jar.set_cookie(create_cookie(name, value, domain=host))
hosts.setdefault(name, set()).add(host)
for domain, values in self.cookies.get_dict_by_domain().items():
bare = domain.lstrip(".")
for name, value in values.items():
if not any(h == bare or h.endswith("." + bare) for h in hosts.get(name, ())):
jar.set_cookie(create_cookie(name, value, domain=domain))
return jar
def import_origin_cookies(self, exported: dict[str, str]) -> None:
"""Load cookie headers from :meth:`export_origin_cookies` into this HTTP session's client."""
if not exported: