fix(remote): forward dl track selection and profile to remote sessions

Remote sessions now carry -l/-vl/-al/-a/-fs under a nested dl_params, so services fetch the requested audio versions. Server-account tokens for non-default accounts move from default to the account name, so each account logs in once after upgrade.
This commit is contained in:
imSp4rky
2026-09-22 12:52:05 -06:00
parent b988951e17
commit db771a13c6
10 changed files with 805 additions and 77 deletions
+31 -1
View File
@@ -143,6 +143,7 @@ Find titles in a service by query string. The service must have a `search()` met
| `profile` | string | no | `null` | Credential/cookie profile to use. |
| `proxy` | string | no | `null` | Full proxy URI, or a country code when the API key has `server_proxy`. |
| `no_proxy` | boolean | no | `false` | Force-disable all proxy use. |
| `credentials`, `cookies`, `cache` | - | no | `null` | Your own login material, on a `--remote-only` server only. See [Client cache on a remote-only server](#client-cache-on-a-remote-only-server). |
=== "Request"
@@ -192,9 +193,34 @@ Get the list of titles behind a title ID, for example episodes or a movie, witho
| `proxy` | string | no | Full proxy URI, or a country code when the API key has `server_proxy`. |
| `no_proxy` | boolean | no | Force-disable proxy. |
| `cdm_type` | string | no | Preferred CDM type. |
| `credentials`, `cookies`, `cache` | - | no | Your own login material, on a `--remote-only` server only. See [Client cache on a remote-only server](#client-cache-on-a-remote-only-server). |
You can also pass service-specific CLI options as additional keys.
#### Client cache on a remote-only server
A `--remote-only` server keeps nothing that a client sends. When the request
does not use a server account, the server runs `search`, `list-titles` and
`list-tracks` on a new service cache directory for that request only, and
removes the directory when the request ends, on success and on error. A
`--remote-only` server exposes only `search` of these three endpoints, so the
rule applies to `list-titles` and `list-tracks` only if a later release exposes
them in that mode.
`credentials` (`{username, password, extra?}`) and `cookies` (base64 of a
zlib-compressed Netscape cookie file) are the same as on
[`POST /api/session/create`](#post-apisessioncreate). `cache` is a map of cache
files in the same form: each cache key is the file path relative to the service cache
directory, with `/` separators and no `.json` suffix, and each value is base64
of the zlib-compressed file. The server writes these files into the request
cache directory before the service authenticates.
When you send `credentials`, `cookies` or `cache`, a successful response also
has a `cache` field in the same form, with the updated files (for example a
refreshed token), but no `titles_` files. Save them to send with the next
request. A request that uses a server account, or a request to a full-mode
server, uses the server's own cache and returns no `cache` field.
=== "Request"
```bash
@@ -278,6 +304,8 @@ Show the video, audio, and subtitle tracks for a title. For series, you can scop
| `episode` | int/string | no | Episode number (combined with `season`). |
| `part` | int/string | no | Part index of a split episode (combined with `season` and `episode`). |
| `profile`, `proxy`, `no_proxy`, `cdm_type` | - | no | As on `list-titles`. |
| `credentials`, `cookies`, `cache` | - | no | As on `list-titles`. |
| `dl_params` | object | no | The `dl` track selection that the service reads from `ctx.parent.params`: `lang`, `v_lang`, `a_lang`, `acodec` and `forced_subs`, as on [`POST /api/session/create`](#post-apisessioncreate). Some services select the manifests they fetch from these values. |
When you give both `season` and `episode`, the server combines them into a `"{season}x{episode}"` selector. `part` narrows that to `"{season}x{episode}.{part}"`, one part of a [split episode](../creating-a-service.md#split-episodes). Without `part` you get every part of the episode. The server reads `part` only when you give both `season` and `episode`, and `wanted` takes precedence over all three. To scope a part by range, put it in `wanted` instead (`"S01E01.2"`).
@@ -857,7 +885,9 @@ DELETE /api/session/{id} → tear down, harvest updated cache
### `POST /api/session/create`
Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files, keyed by the file path relative to the service cache directory with `/` separators and no `.json` suffix, so a nested key such as `session_web/<sha1>` lands in a subdirectory), `client_region`, `proxy_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, `client`, and any service CLI options. `proxy_region` is the country code the client resolved `proxy` from; the server matches it against its own accounts. `client` is a freeform object the dashboard shows as sent (the CLI puts `version`, `code_hash`, `platform` and a redacted `argv` in it); the server ignores it above 4096 bytes of JSON.
Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files, keyed by the file path relative to the service cache directory with `/` separators and no `.json` suffix, so a nested key such as `session_web/<sha1>` lands in a subdirectory), `client_region`, `proxy_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, `dl_params`, `client`, and any service CLI options. `proxy_region` is the country code the client resolved `proxy` from; the server matches it against its own accounts. `client` is a freeform object the dashboard shows as sent (the CLI puts `version`, `code_hash`, `platform` and a redacted `argv` in it); the server ignores it above 4096 bytes of JSON.
`dl_params` is an object with the `dl` track selection: `lang`, `v_lang` and `a_lang` (arrays of language strings), `acodec` (an array of audio codec names or `dl -a` aliases such as `ddp`) and `forced_subs` (boolean). The service reads these values from `ctx.parent.params`. The server uses the `dl` default for each absent or malformed value, and drops an unknown codec name.
Service CLI options also travel in a nested `service_params` object, which wins over a flat key with the same name. `profile` at the top level always means the credentials profile, never a service's own `--profile` option.
+28
View File
@@ -124,6 +124,9 @@ hood the client walks a remote session through its lifecycle.
the client region itself
- Track-selection hints (`range_`, `vcodec`, `quality`, `best_available`) so
the server fetches the right manifests
- Your language and audio codec selection (`-l`, `-vl`, `-al`, `-a`, `-fs`)
in a `dl_params` object. Some services fetch one manifest
for each audio language, and pick the languages from these values
- Your local per-service **cache files** (e.g. refreshed tokens), including
files in subdirectories of the service cache directory, but only the files
for the active profile. The client withholds a file whose path, in any
@@ -207,6 +210,9 @@ hood the client walks a remote session through its lifecycle.
authentication. The login belongs to the client when the client sent
cookies, credentials, or cache files, or answered a login prompt (a device code
or an OTP) that led to a successful login. A server-account login never does.
The client saves only the returned files that pass the same profile check it
applies before it sends them, so a file for another profile cannot overwrite
that profile's local file.
!!! tip "Renaming remote titles locally"
You can rename titles for a remote service you do not have installed locally by
@@ -349,6 +355,28 @@ shows as sent), and the track-selection hints `range_`, `vcodec`,
`quality`, `best_available`, plus arbitrary service CLI options
(`additionalProperties: true`).
`dl_params` holds the `dl` track selection: `lang`, `v_lang` and `a_lang`
(arrays of language strings), `acodec` (an array of audio codec names, such as
`EC3`), and `forced_subs` (boolean). The server gives these values to the
service in `ctx.parent.params`, never as service options, so a service option
with the same name, such as an own `--lang`, does not get them. The server uses
the `dl` default for each absent or malformed value: `lang` is `["orig"]`,
`v_lang`, `a_lang` and `acodec` are `[]`, and `forced_subs` is `false`. The
server drops an unknown codec name. An empty array is a valid value and stays
empty.
The server does not forward the `dl` mode flags: `video_only`, `audio_only`,
`subs_only`, `chapters_only`, `list_`, `skip_dl`, `sub_format` and
`no_attachments`. A service that reads one of them to skip work fetches more on
the server, but the result is the same.
In a client-login session (not a server account), the server gives the service
the client's `profile` in `ctx.parent.params`, the same as a local run with that
profile. The profile name keys the service's token cache, so the cache files
the client sends and gets back belong to that profile. A service can also use
the name to select per-profile entries in the server's service config, such as
a per-profile ESN.
The response returns **before** authentication finishes:
```json
@@ -182,3 +182,45 @@ def test_worker_write_result_retries_denied_replace(tmp_path: Path, monkeypatch:
download_worker.write_result(target, {"progress": 42.0})
assert json.loads(target.read_text(encoding="utf-8")) == {"progress": 42.0}
assert len(attempts) == 2
class _CapturedCtx(Exception):
pass
@pytest.mark.parametrize(
("params", "expected"),
[
({}, (["orig"], [], [], [], False)),
(
{"lang": ["es-419"], "v_lang": ["en"], "a_lang": ["ja"], "acodec": "AAC,EC3", "forced_subs": True},
(["es-419"], ["en"], ["ja"], ["AAC", "EC3"], True),
),
],
)
def test_perform_download_puts_the_lang_selection_in_the_service_ctx(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, params: dict, expected: tuple
) -> None:
"""Services read the language selection from ctx.parent.params, not from dl.result()."""
import click
import unshackle.commands.dl as dl_module
from unshackle.core.api import download_manager, handlers
from unshackle.core.services import Services
class FakeDl:
cli = click.Command("dl")
def __init__(self, ctx: click.Context, **kwargs: object) -> None:
raise _CapturedCtx(ctx)
monkeypatch.setattr(dl_module, "dl", FakeDl)
monkeypatch.setattr(Services, "get_path", staticmethod(lambda s: tmp_path))
monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None)
with pytest.raises(_CapturedCtx) as exc:
download_manager.perform_download("job-1", "EXAMPLE", "t1", dict(params))
ctx = exc.value.args[0]
p = ctx.params
assert (p["lang"], p["v_lang"], p["a_lang"], [c.name for c in p["acodec"]], p["forced_subs"]) == expected
@@ -410,6 +410,27 @@ def test_save_returned_cache_writes_nested_keys_and_rejects_escapes(tmp_path, mo
]
def test_save_returned_cache_drops_other_profiles_files(tmp_path, monkeypatch) -> None:
"""A returned file for another profile must not overwrite that profile's local file."""
import base64
import logging
import zlib
from types import SimpleNamespace
from unshackle.core.config import config
from unshackle.core.remote_service import RemoteService
tag = "TESTSVC"
monkeypatch.setattr(config.directories, "cache", tmp_path)
monkeypatch.setattr(config, "credentials", {tag: {"default": "a@example.com:pw", "work": "b@example.com:pw"}})
stub = SimpleNamespace(service_tag=tag, log=logging.getLogger("test"))
blob = base64.b64encode(zlib.compress(b"x")).decode("ascii")
RemoteService.save_returned_cache(stub, {"tokens_work": blob, "tokens_default": blob, "tokens": blob}, "work")
assert sorted(p.name for p in (tmp_path / tag).iterdir()) == ["tokens.json", "tokens_work.json"]
def test_save_returned_cache_ignores_oversized_payload(tmp_path, monkeypatch) -> None:
import logging
from types import SimpleNamespace
@@ -536,3 +557,54 @@ def test_session_create_writes_nested_cache_and_skips_colliding_keys(tmp_path, m
written = {p.name: p.read_text() for p in tmp_path.rglob("*.json") if p.is_file()}
assert written == {"x.json": "flat", "keys.v2.json": "dotted"}
class _SentCreate(Exception):
pass
def _sent_create_data(parent_params: dict, service_params: dict) -> dict:
import logging
from types import SimpleNamespace
from unshackle.core.remote_service import RemoteService
def post(path: str, data: dict) -> dict:
raise _SentCreate(data)
svc = object.__new__(RemoteService)
svc.ctx = SimpleNamespace(parent=SimpleNamespace(params={"no_proxy": True, **parent_params}), obj=None)
svc.service_tag = "EXAMPLE"
svc.title_id = "t1"
svc._server_accounts = {"global": True}
svc._server_cdm = True
svc._service_params = service_params
svc.log = logging.getLogger("test")
svc.client = SimpleNamespace(post=post)
with pytest.raises(_SentCreate) as exc:
svc.authenticate()
return exc.value.args[0]
def test_dl_selection_travels_nested_and_a_service_option_cannot_override_it() -> None:
parent_params = {
"lang": ["orig", "es-419"],
"v_lang": [],
"a_lang": [],
"acodec": [Audio.Codec.EC3],
"forced_subs": True,
}
data = _sent_create_data(parent_params, {"lang": 3, "a_lang": "x"})
assert data["dl_params"] == {
"lang": ["orig", "es-419"],
"v_lang": [],
"a_lang": [],
"acodec": ["EC3"],
"forced_subs": True,
}
assert data["lang"] == 3
assert data["service_params"] == {"lang": 3, "a_lang": "x"}
def test_no_dl_params_when_the_ctx_has_no_selection() -> None:
assert "dl_params" not in _sent_create_data({}, {})
+173
View File
@@ -0,0 +1,173 @@
"""List and search requests on a --remote-only server run on a per-request cache directory.
The directory holds only what the client sent, goes back to the client when it logged in
itself, and is always removed when the request ends.
"""
import base64
import json
import zlib
import click
import pytest
from unshackle.core.api import handlers
from unshackle.core.api.stats import stats
from unshackle.core.cacher import Cacher
from unshackle.core.config import config
class _CacheService:
fail_titles = False
def __init__(self, ctx, title):
self.seen = _CacheService.seen
self.seen["parent_params"] = ctx.parent.params
self.cache = Cacher("EXAMPLE")
def authenticate(self, cookies, credential):
self.seen["seeded"] = self.cache.get("tokens").data
self.seen["cache_dir"] = self.cache.get("tokens").path.parent
self.cache.get("tokens").set("new")
self.cache.get("titles_abc").set("titles")
def get_titles(self):
if self.fail_titles:
raise RuntimeError("boom")
return []
def search(self):
return iter(())
@pytest.fixture
def seen(tmp_path, monkeypatch):
import types
_CacheService.seen = {}
_CacheService.fail_titles = False
@click.command()
@click.argument("title")
@click.pass_context
def cli(ctx, **kwargs):
return _CacheService(ctx, **kwargs)
module = types.SimpleNamespace(cli=cli)
monkeypatch.setattr(config.directories, "cache", tmp_path)
monkeypatch.setattr(stats, "mode", "remote_only")
monkeypatch.setattr(handlers, "validate_service", lambda tag, request=None: tag)
monkeypatch.setattr(handlers, "load_service_yaml", lambda service: {})
monkeypatch.setattr(handlers, "resolve_handler_proxy", lambda *args: (None, []))
monkeypatch.setattr(handlers, "resolve_server_account", lambda *args: None)
monkeypatch.setattr(handlers, "load_full_cdm", lambda *args: None)
monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda service: module))
monkeypatch.setattr("unshackle.commands.dl.dl.get_cookie_jar", staticmethod(lambda service, profile: None))
monkeypatch.setattr("unshackle.commands.dl.dl.get_credentials", staticmethod(lambda service, profile: None))
return _CacheService.seen
def client_cache(tmp_path, value):
Cacher("SRC").get("tokens").set(value)
text = (tmp_path / "SRC" / "tokens.json").read_bytes()
return {"tokens": base64.b64encode(zlib.compress(text)).decode("ascii")}
def request_dirs(tmp_path):
return list((tmp_path / "_requests").rglob("*")) if (tmp_path / "_requests").exists() else []
CREDENTIALS = {"credentials": {"username": "a@example.com", "password": "pw"}}
@pytest.mark.asyncio
async def test_seeded_cache_is_read_and_returned_then_removed(seen, tmp_path):
data = {"service": "EXAMPLE", "title_id": "t", "cache": client_cache(tmp_path, "old")}
response = await handlers.list_titles_handler(data)
body = json.loads(response.body)
assert seen["seeded"] == "old"
assert seen["cache_dir"].is_relative_to(tmp_path / "_requests")
assert set(body["cache"]) == {"tokens"}
returned = json.loads(zlib.decompress(base64.b64decode(body["cache"]["tokens"])))
assert returned["data"] == "new"
assert not seen["cache_dir"].exists()
assert not request_dirs(tmp_path)
@pytest.mark.asyncio
async def test_search_returns_cache_to_a_client_with_credentials(seen, tmp_path):
response = await handlers.search_handler({"service": "EXAMPLE", "query": "q", **CREDENTIALS})
body = json.loads(response.body)
assert body["count"] == 0
assert set(body["cache"]) == {"tokens"}
assert not request_dirs(tmp_path)
@pytest.mark.asyncio
async def test_no_cache_returned_without_client_login(seen, tmp_path):
response = await handlers.search_handler({"service": "EXAMPLE", "query": "q"})
assert "cache" not in json.loads(response.body)
assert seen["cache_dir"].is_relative_to(tmp_path / "_requests")
assert not request_dirs(tmp_path)
@pytest.mark.asyncio
async def test_cache_dir_removed_on_error(seen, tmp_path):
_CacheService.fail_titles = True
data = {"service": "EXAMPLE", "title_id": "t", **CREDENTIALS}
response = await handlers.list_tracks_handler(data)
assert response.status >= 400
assert "cache" not in json.loads(response.body)
assert seen["cache_dir"].is_relative_to(tmp_path / "_requests")
assert not request_dirs(tmp_path)
@pytest.mark.asyncio
async def test_invalid_client_cache_still_removes_the_dir(seen, tmp_path):
data = {"service": "EXAMPLE", "query": "q", "cache": {"tokens": 1}}
with pytest.raises(handlers.APIError):
await handlers.search_handler(data)
assert not request_dirs(tmp_path)
@pytest.mark.asyncio
async def test_server_account_uses_its_own_cache_and_returns_none(seen, tmp_path, monkeypatch):
monkeypatch.setattr(handlers, "resolve_server_account", lambda *args: "acct")
monkeypatch.setattr(handlers, "server_account_cookies", lambda *args: None)
data = {"service": "EXAMPLE", "query": "q", "cache": client_cache(tmp_path, "old"), **CREDENTIALS}
response = await handlers.search_handler(data)
assert "cache" not in json.loads(response.body)
assert not seen["seeded"]
assert seen["cache_dir"] == tmp_path / "_accounts" / "EXAMPLE" / "acct"
assert not (tmp_path / "_requests").exists()
@pytest.mark.asyncio
async def test_full_mode_keeps_the_server_cache(seen, tmp_path, monkeypatch):
monkeypatch.setattr(stats, "mode", "full")
data = {"service": "EXAMPLE", "query": "q", "cache": client_cache(tmp_path, "old"), **CREDENTIALS}
response = await handlers.search_handler(data)
assert "cache" not in json.loads(response.body)
assert not seen["seeded"]
assert seen["cache_dir"] == tmp_path / "EXAMPLE"
assert not (tmp_path / "_requests").exists()
@pytest.mark.asyncio
async def test_list_tracks_service_sees_dl_params(seen):
from unshackle.core.tracks import Audio
data = {"service": "EXAMPLE", "title_id": "t", "dl_params": {"a_lang": ["es-419"], "acodec": ["ddp"]}}
await handlers.list_tracks_handler(data)
assert seen["parent_params"]["a_lang"] == ["es-419"]
assert seen["parent_params"]["acodec"] == [Audio.Codec.EC3]
assert "dl_params" in handlers.LIST_HANDLER_TRANSPORT_KEYS
@pytest.mark.asyncio
async def test_search_service_gets_no_dl_params(seen):
await handlers.search_handler({"service": "EXAMPLE", "query": "q", "dl_params": {"a_lang": ["es-419"]}})
assert "a_lang" not in seen["parent_params"]
+87
View File
@@ -133,3 +133,90 @@ def test_server_cdm_ignores_client_cdm_type(monkeypatch):
handlers.create_service_instance("OTHER", "t", data, None, [], None, server_cdm=True)
handlers.create_service_instance("OTHER", "t", data, None, [], None)
assert seen == [None, "playready"]
class _LangOptionService:
def __init__(self, ctx, title, lang):
self.parent_params = ctx.parent.params
self.lang = lang
def _lang_option_module():
import click
@click.command()
@click.argument("title")
@click.option("--lang", type=int, default=None)
@click.pass_context
def cli(ctx, **kwargs):
return _LangOptionService(ctx, **kwargs)
return type("Module", (), {"cli": cli})
def _create_with_real_ctx(monkeypatch, data):
monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {})
monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None)
monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: _lang_option_module()))
svc, _, _ = handlers.create_service_instance("OTHER", "t", data, None, [], None)
return svc
def test_nested_dl_params_reach_the_parent_ctx(monkeypatch):
from unshackle.core.tracks import Audio
data = {
"dl_params": {
"lang": ["es-419", "en"],
"v_lang": ["ja"],
"a_lang": ["es-419"],
"acodec": ["EC3", "NOPE", "AAC"],
"forced_subs": True,
}
}
svc = _create_with_real_ctx(monkeypatch, data)
assert svc.parent_params["lang"] == ["es-419", "en"]
assert svc.parent_params["v_lang"] == ["ja"]
assert svc.parent_params["a_lang"] == ["es-419"]
assert svc.parent_params["acodec"] == [Audio.Codec.EC3, Audio.Codec.AAC]
assert svc.parent_params["forced_subs"] is True
def test_absent_dl_params_fall_back_to_the_dl_defaults(monkeypatch):
svc = _create_with_real_ctx(monkeypatch, {})
selection = {k: svc.parent_params[k] for k in ("lang", "v_lang", "a_lang", "acodec", "forced_subs")}
assert selection == {"lang": ["orig"], "v_lang": [], "a_lang": [], "acodec": [], "forced_subs": False}
def test_an_empty_lang_list_is_kept(monkeypatch):
svc = _create_with_real_ctx(monkeypatch, {"dl_params": {"lang": [], "a_lang": []}})
assert svc.parent_params["lang"] == []
def test_a_service_lang_option_never_gets_the_dl_lang(monkeypatch):
svc = _create_with_real_ctx(monkeypatch, {"dl_params": {"lang": ["en"]}})
assert svc.lang is None
assert svc.parent_params["lang"] == ["en"]
assert "dl_params" in handlers.SESSION_TRANSPORT_KEYS
@pytest.mark.parametrize(
"dl_params",
[
"en",
{"lang": "en", "v_lang": "ja", "a_lang": {"x": 1}, "acodec": "EC3", "forced_subs": "yes"},
{"lang": [1], "v_lang": [None], "a_lang": ["en", None], "acodec": [3], "forced_subs": 1},
],
)
def test_malformed_dl_params_are_ignored(monkeypatch, dl_params):
svc = _create_with_real_ctx(monkeypatch, {"dl_params": dl_params})
selection = {k: svc.parent_params[k] for k in ("lang", "v_lang", "a_lang", "acodec", "forced_subs")}
assert selection == {"lang": ["orig"], "v_lang": [], "a_lang": [], "acodec": [], "forced_subs": False}
def test_profile_reaches_the_parent_ctx_params(monkeypatch):
monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {})
monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None)
monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: _lang_option_module()))
svc, _, _ = handlers.create_service_instance("OTHER", "t", {}, None, [], "work")
assert svc.parent_params["profile"] == "work"
+15 -6
View File
@@ -18,7 +18,7 @@ from typing import Any, Callable, Dict, List, Optional
from unshackle.core.api.events import bus, publish_service_event
from unshackle.core.api.sanitize import sanitize_log
from unshackle.core.utils.click_types import VIDEO_CODEC_LIST
from unshackle.core.utils.click_types import AUDIO_CODEC_LIST, VIDEO_CODEC_LIST
from unshackle.core.utils.redact import REDACTED, URL_USERINFO_RE, redact_path, redact_text
log = logging.getLogger("download_manager")
@@ -392,6 +392,9 @@ def perform_download(
else:
params["vcodec"] = []
if params.get("acodec"):
params["acodec"] = AUDIO_CODEC_LIST.convert(params["acodec"])
range_raw = params.get("range")
if range_raw:
if isinstance(range_raw, str):
@@ -474,6 +477,12 @@ def perform_download(
"vcodec": params.get("vcodec", []),
"range_": params.get("range", [Video.Range.SDR]),
"best_available": params.get("best_available", False),
# services read these here, not from dl.result(), to pick the manifests they fetch
"acodec": params.get("acodec") or [],
"lang": params.get("lang", ["orig"]),
"v_lang": params.get("v_lang", []),
"a_lang": params.get("a_lang", []),
"forced_subs": params.get("forced_subs", False),
}
# Hand-built context: record parameter sources so service dl overrides
# apply to defaults but never clobber client-sent values.
@@ -567,7 +576,7 @@ def perform_download(
service=service_instance,
quality=params.get("quality", []),
vcodec=params.get("vcodec", []),
acodec=params.get("acodec"),
acodec=ctx.params["acodec"],
vbitrate=params.get("vbitrate"),
abitrate=params.get("abitrate"),
vbitrate_range=params.get("vbitrate_range"),
@@ -578,14 +587,14 @@ def perform_download(
select_titles=False,
wanted=params.get("wanted", []),
latest_episode=params.get("latest_episode", False),
lang=params.get("lang", ["orig"]),
v_lang=params.get("v_lang", []),
a_lang=params.get("a_lang", []),
lang=ctx.params["lang"],
v_lang=ctx.params["v_lang"],
a_lang=ctx.params["a_lang"],
s_lang=params.get("s_lang", ["all"]),
require_audio=params.get("require_audio", []),
require_video=params.get("require_video", []),
require_subs=params.get("require_subs", []),
forced_subs=params.get("forced_subs", False),
forced_subs=ctx.params["forced_subs"],
forced_s_lang=params.get("forced_s_lang", []),
exact_lang=params.get("exact_lang", False),
sub_format=params.get("sub_format"),
+180 -51
View File
@@ -13,7 +13,7 @@ from contextlib import suppress
from datetime import date as date_
from http.cookiejar import CookieJar, MozillaCookieJar
from pathlib import Path
from typing import Any, Callable, Dict, Iterable, List, Optional
from typing import Any, Callable, Dict, Iterable, List, Optional, cast
import click
from aiohttp import web
@@ -124,6 +124,8 @@ LIST_HANDLER_TRANSPORT_KEYS = {
"no_proxy",
"query",
"service_params",
"cache",
"dl_params",
}
@@ -286,12 +288,119 @@ def server_login_material(
return load_client_cookies(data.get("cookies")), credential
def api_key_namespace(request: Optional[web.Request]) -> str:
"""Name of the caller's cache directories, derived from its API key so no caller can guess another's."""
import hashlib
api_key = request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous"
return hashlib.pbkdf2_hmac("sha256", api_key.encode(), b"unshackle-session-ns", 100_000).hex()[:12]
def write_client_cache(cache_data: Any, cache_tag: str) -> None:
"""Write a client-sent ``cache`` map into the cache directory ``cache_tag``.
Each cache key is a file path relative to that directory, and each value is the file as
base64 of zlib-compressed bytes. The function skips an unsafe key or a file it cannot write,
and raises INVALID_INPUT for a map that is not an object, has too many entries, or holds a
value that is not a string.
"""
if not isinstance(cache_data, dict) or len(cache_data) > MAX_SESSION_CACHE_KEYS:
raise APIError(APIErrorCode.INVALID_INPUT, f"cache must hold at most {MAX_SESSION_CACHE_KEYS} entries")
cache_dir = config.directories.cache / cache_tag
cache_dir.mkdir(parents=True, exist_ok=True)
for key, content in cache_data.items():
safe_name = safe_cache_key(key)
if not safe_name:
log.warning(f"Rejecting unsafe session cache key: {sanitize_log(key)}")
continue
if not isinstance(content, str):
raise APIError(APIErrorCode.INVALID_INPUT, "cache values must be base64 strings")
decompressed = safe_inflate(base64.b64decode(content)).decode("utf-8")
target = cache_dir / f"{safe_name}.json"
try:
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(decompressed, encoding="utf-8")
except OSError as e:
log.warning(f"Skipping session cache key {sanitize_log(key)}: {e}")
def collect_cache_files(cache_tag: str) -> Dict[str, str]:
"""Read the cache directory ``cache_tag`` back as a ``cache`` map for the client.
The map has the same form that :func:`write_client_cache` reads. It leaves out the
``titles_`` files, because they hold no login state.
"""
cache_data: Dict[str, str] = {}
cache_dir = config.directories.cache / cache_tag
if cache_dir.is_dir():
for f in sorted(cache_dir.rglob("*.json")):
key = f.relative_to(cache_dir).as_posix()[: -len(".json")]
if f.stem.startswith("titles_") or not safe_cache_key(key):
continue
try:
cache_data[key] = base64.b64encode(zlib.compress(f.read_bytes())).decode("ascii")
except OSError:
pass
return cache_data
class RequestCache:
"""The service cache of one list or search request on a ``--remote-only`` server.
Such a server keeps nothing that a client sends, so the request runs on a cache directory
of its own that the handler removes when the request ends, on success and on error. Before
that, a client that logged in with its own cookies, credentials or cache gets the updated
files back, the same as when a remote session ends.
"""
def __init__(self) -> None:
self.tag: Optional[str] = None
self.client_auth = False
def attach(
self,
service_instance: Any,
data: Dict[str, Any],
normalized_service: str,
request: Optional[web.Request],
client_login: bool,
) -> None:
"""Give the service a new cache directory, seeded from the client's ``cache``.
Does nothing on a full-mode server, where the operator's own cache stays in use.
"""
import uuid
from unshackle.core.api.stats import stats
if stats.mode != "remote_only":
return
self.tag = f"_requests/{api_key_namespace(request)}/{uuid.uuid4()}/{normalized_service}"
service_instance.cache = Cacher(self.tag)
cache_data = data.get("cache")
if cache_data:
write_client_cache(cache_data, self.tag)
self.client_auth = client_login or bool(cache_data)
def json_response(self, payload: Dict[str, Any]) -> web.Response:
"""Answer with ``payload``, plus the updated ``cache`` for a client that logged in itself."""
if self.tag and self.client_auth:
cache_data = collect_cache_files(self.tag)
if cache_data:
payload["cache"] = cache_data
return web.json_response(payload)
def cleanup(self) -> None:
SessionStore.cleanup_cache_dir(self.tag)
def setup_list_service(
data: Dict[str, Any],
normalized_service: str,
profile: Optional[str],
title_id: str,
request: Optional[web.Request] = None,
request_cache: Optional[RequestCache] = None,
) -> Any:
"""Assemble and authenticate a service instance for list_titles / list_tracks.
@@ -318,13 +427,16 @@ def setup_list_service(
no_proxy,
proxy_providers,
service_config,
extra_params={"cookies_supplied": cookies is not None},
extra_params={"cookies_supplied": cookies is not None, **forwarded_dl_params(data)},
)
service_module = Services.load(normalized_service)
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
if account:
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}")
elif request_cache is not None:
client_login = cookies is not None or credential is not None
request_cache.attach(service_instance, data, normalized_service, request, client_login)
service_instance.authenticate(cookies, credential)
return service_instance
@@ -334,6 +446,7 @@ def run_service_search(
normalized_service: str,
query: str,
request: Optional[web.Request] = None,
request_cache: Optional[RequestCache] = None,
) -> List[Dict[str, Any]]:
"""Assemble and authenticate a service instance, then run its search.
@@ -381,6 +494,9 @@ def run_service_search(
if account:
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}")
elif request_cache is not None:
client_login = cookies is not None or credential is not None
request_cache.attach(service_instance, data, normalized_service, request, client_login)
service_instance.authenticate(cookies, credential)
results: List[Dict[str, Any]] = []
@@ -1297,9 +1413,12 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] =
details={"service": service_tag},
)
results = await asyncio.to_thread(run_service_search, data, normalized_service, query, request)
return web.json_response({"results": results, "count": len(results)})
request_cache = RequestCache()
try:
results = await asyncio.to_thread(run_service_search, data, normalized_service, query, request, request_cache)
return request_cache.json_response({"results": results, "count": len(results)})
finally:
request_cache.cleanup()
async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response:
@@ -1317,9 +1436,10 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
details={"service": service_tag},
)
request_cache = RequestCache()
try:
service_instance = await asyncio.to_thread(
setup_list_service, data, normalized_service, profile, title_id, request
setup_list_service, data, normalized_service, profile, title_id, request, request_cache
)
titles = await asyncio.to_thread(service_instance.get_titles)
@@ -1328,7 +1448,7 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
else:
title_list = [stamp_service_flags(serialize_title(titles), service_instance)]
return web.json_response({"titles": title_list})
return request_cache.json_response({"titles": title_list})
except APIError:
raise
@@ -1340,6 +1460,8 @@ async def list_titles_handler(data: Dict[str, Any], request: Optional[web.Reques
context={"operation": "list_titles", "service": normalized_service, "title_id": title_id},
debug_mode=debug_mode,
)
finally:
request_cache.cleanup()
async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Request] = None) -> web.Response:
@@ -1357,9 +1479,10 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
details={"service": service_tag},
)
request_cache = RequestCache()
try:
service_instance = await asyncio.to_thread(
setup_list_service, data, normalized_service, profile, title_id, request
setup_list_service, data, normalized_service, profile, title_id, request, request_cache
)
titles = await asyncio.to_thread(service_instance.get_titles)
@@ -1469,7 +1592,7 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
response = {"episodes": episodes_data}
if failed_episodes:
response["unavailable_episodes"] = failed_episodes
return web.json_response(response)
return request_cache.json_response(response)
else:
raise APIError(
APIErrorCode.NO_CONTENT,
@@ -1500,7 +1623,7 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
"subtitles": [serialize_subtitle_track(t) for t in tracks.subtitles],
}
return web.json_response(response)
return request_cache.json_response(response)
except APIError:
raise
@@ -1512,6 +1635,8 @@ async def list_tracks_handler(data: Dict[str, Any], request: Optional[web.Reques
context={"operation": "list_tracks", "service": normalized_service, "title_id": title_id},
debug_mode=debug_mode,
)
finally:
request_cache.cleanup()
VALID_VCODECS = [choice.upper() for choice in VIDEO_CODEC_LIST.choices]
@@ -2844,9 +2969,47 @@ SESSION_TRANSPORT_KEYS = {
"vcodec",
"quality",
"best_available",
"dl_params",
}
def forwarded_dl_params(data: Dict[str, Any]) -> Dict[str, Any]:
"""Read the dl track selection that a remote-dl client sends under ``dl_params``.
Services read these values from ``ctx.parent.params`` to pick the manifests they
fetch. They travel in their own object so that they never collide with a service option that
has the same name. An absent or malformed value gets the dl default, so an old client that
sends no ``dl_params`` gets the same result as a ``dl`` run with no selection flags.
``acodec`` holds codec names or aliases, which become ``Audio.Codec`` members the same way
``dl -a`` converts them; an unknown name is dropped.
"""
raw = data.get("dl_params")
if not isinstance(raw, dict):
raw = {}
def str_list(key: str) -> Optional[List[str]]:
value = raw.get(key)
if isinstance(value, list) and all(isinstance(v, str) for v in value):
return list(value)
if value is not None:
log.warning(f"Ignoring dl_params.{key}: it must be an array of strings")
return None
params: Dict[str, Any] = {}
for key in ("lang", "v_lang", "a_lang"):
langs = str_list(key)
params[key] = langs if langs is not None else list(cast(List[str], DEFAULT_DOWNLOAD_PARAMS[key]))
# the dl CLI default is [], and some services test membership in it
params["acodec"] = []
for name in str_list("acodec") or []:
try:
params["acodec"].extend(AUDIO_CODEC_LIST.convert(name))
except click.BadParameter:
log.warning(f"Ignoring unknown dl_params.acodec value: {sanitize_log(name)}")
params["forced_subs"] = raw.get("forced_subs") is True
return params
def create_service_instance(
normalized_service: str,
title_id: str,
@@ -2891,6 +3054,7 @@ def create_service_instance(
"vcodec": vcodec_values,
"quality": data.get("quality"),
"best_available": data.get("best_available", False),
**forwarded_dl_params(data),
}
if server_account:
@@ -2908,6 +3072,9 @@ def create_service_instance(
cookies = load_client_cookies(data.get("cookies"))
extra_params["cookies_supplied"] = cookies is not None
# Services key their token caches on this. Only sessions get it: they run on a cache
# directory of their own, where the list and search handlers share the server's cache.
extra_params["profile"] = profile
parent_ctx = build_parent_ctx(
profile,
@@ -2954,15 +3121,11 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
try:
proxy_param, proxy_providers = await asyncio.to_thread(resolve_handler_proxy, data, normalized_service, request)
import hashlib
import uuid as uuid_mod
from unshackle.core.config import config as app_config
session_id = str(uuid_mod.uuid4())
api_key = request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous"
api_key_hash = hashlib.pbkdf2_hmac("sha256", api_key.encode(), b"unshackle-session-ns", 100_000).hex()[:12]
session_cache_dir = f"_sessions/{api_key_hash}/{session_id}/{normalized_service}"
session_cache_dir = f"_sessions/{api_key_namespace(request)}/{session_id}/{normalized_service}"
session_cache_tag: Optional[str] = session_cache_dir
server_account = server_account_for(request, normalized_service)
@@ -3003,26 +3166,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
cache_data = data.get("cache", {})
if cache_data and session_cache_tag:
if not isinstance(cache_data, dict) or len(cache_data) > MAX_SESSION_CACHE_KEYS:
raise APIError(APIErrorCode.INVALID_INPUT, f"cache must hold at most {MAX_SESSION_CACHE_KEYS} entries")
import base64
cache_dir = app_config.directories.cache / session_cache_tag
cache_dir.mkdir(parents=True, exist_ok=True)
for key, content in cache_data.items():
safe_name = safe_cache_key(key)
if not safe_name:
log.warning(f"Rejecting unsafe session cache key: {sanitize_log(key)}")
continue
if not isinstance(content, str):
raise APIError(APIErrorCode.INVALID_INPUT, "cache values must be base64 strings")
decompressed = safe_inflate(base64.b64decode(content)).decode("utf-8")
target = cache_dir / f"{safe_name}.json"
try:
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(decompressed, encoding="utf-8")
except OSError as e:
log.warning(f"Skipping session cache key {sanitize_log(key)}: {e}")
write_client_cache(cache_data, session_cache_tag)
bridge = InputBridge()
service_instance._input_bridge = bridge
@@ -4510,11 +4654,7 @@ async def session_info_handler(session_id: str, request: Optional[web.Request] =
async def session_delete_handler(session_id: str, request: Optional[web.Request] = None) -> web.Response:
"""Delete a remote session, return updated cache files, and clean up server-side data."""
import base64
import zlib
from unshackle.core.api.session_store import get_session_store
from unshackle.core.config import config as app_config
session = await get_validated_session(session_id, request)
store = get_session_store()
@@ -4523,18 +4663,7 @@ async def session_delete_handler(session_id: str, request: Optional[web.Request]
session.input_bridge.cancel()
cache_tag = session.cache_tag
cache_data: Dict[str, str] = {}
if cache_tag and session.client_auth:
cache_dir = app_config.directories.cache / cache_tag
if cache_dir.is_dir():
for f in sorted(cache_dir.rglob("*.json")):
key = f.relative_to(cache_dir).as_posix()[: -len(".json")]
if f.stem.startswith("titles_") or not safe_cache_key(key):
continue
try:
cache_data[key] = base64.b64encode(zlib.compress(f.read_bytes())).decode("ascii")
except OSError:
pass
cache_data = collect_cache_files(cache_tag) if cache_tag and session.client_auth else {}
await store.delete(session_id)
+120
View File
@@ -408,6 +408,21 @@ async def search(request: web.Request) -> web.Response:
no_proxy:
type: boolean
description: Force disable all proxy use (default - false)
credentials:
type: object
additionalProperties: true
description: Your own login, on a --remote-only server only ({username, password, extra?})
cookies:
type: string
description: Your own cookies, on a --remote-only server only (base64 of a zlib-compressed Netscape cookie file)
cache:
type: object
additionalProperties:
type: string
description: |
Your own cache files, on a --remote-only server only: cache key to base64 of the
zlib-compressed file. The request runs on a cache directory of its own, which the
server removes when the request ends.
responses:
'200':
description: Search results
@@ -439,6 +454,14 @@ async def search(request: web.Request) -> web.Response:
count:
type: integer
description: Number of results returned
cache:
type: object
additionalProperties:
type: string
description: |
The updated cache files, in the same form as the request field. Only on a
--remote-only server, without a server account, when you sent credentials,
cookies or cache.
'400':
description: Invalid request
"""
@@ -485,9 +508,41 @@ async def list_titles(request: web.Request) -> web.Response:
title_id:
type: string
description: Title identifier
credentials:
type: object
additionalProperties: true
description: Your own login, on a --remote-only server only ({username, password, extra?})
cookies:
type: string
description: Your own cookies, on a --remote-only server only (base64 of a zlib-compressed Netscape cookie file)
cache:
type: object
additionalProperties:
type: string
description: |
Your own cache files, on a --remote-only server only: cache key to base64 of the
zlib-compressed file. The request runs on a cache directory of its own, which the
server removes when the request ends.
responses:
'200':
description: List of titles
content:
application/json:
schema:
type: object
properties:
titles:
type: array
items:
type: object
cache:
type: object
additionalProperties:
type: string
description: |
The updated cache files, in the same form as the request field. Only on a
--remote-only server, without a server account, when you sent credentials,
cookies or cache.
'400':
description: Invalid request (missing parameters, invalid service)
content:
@@ -614,9 +669,43 @@ async def list_tracks(request: web.Request) -> web.Response:
proxy:
type: string
description: Proxy configuration (optional)
dl_params:
type: object
description: |
The dl track selection (lang, v_lang, a_lang, acodec, forced_subs), in the same
form as on `/api/session/create`. The service reads it from `ctx.parent.params`.
credentials:
type: object
additionalProperties: true
description: Your own login, on a --remote-only server only ({username, password, extra?})
cookies:
type: string
description: Your own cookies, on a --remote-only server only (base64 of a zlib-compressed Netscape cookie file)
cache:
type: object
additionalProperties:
type: string
description: |
Your own cache files, on a --remote-only server only: cache key to base64 of the
zlib-compressed file. The request runs on a cache directory of its own, which the
server removes when the request ends.
responses:
'200':
description: Track information
content:
application/json:
schema:
type: object
additionalProperties: true
properties:
cache:
type: object
additionalProperties:
type: string
description: |
The updated cache files, in the same form as the request field. Only on a
--remote-only server, without a server account, when you sent credentials,
cookies or cache.
'400':
description: Invalid request
"""
@@ -1571,6 +1660,37 @@ async def session_create(request: web.Request) -> web.Response:
proxy_region:
type: string
description: Two-letter country the client resolved its proxy for; picks a server account
dl_params:
type: object
description: |
The dl track selection. The service reads it from ctx.parent.params, never as
service options. An absent or malformed value gets the dl default, and the
server drops an unknown codec name.
properties:
lang:
type: array
items:
type: string
default: ["orig"]
v_lang:
type: array
items:
type: string
default: []
acodec:
type: array
items:
type: string
default: []
description: Audio codec names, such as EC3
a_lang:
type: array
items:
type: string
default: []
forced_subs:
type: boolean
default: false
client:
type: object
additionalProperties: true
+57 -19
View File
@@ -731,6 +731,29 @@ def cache_stem_is_relevant(
return not matched_foreign
def cache_key_filter(service_tag: str, profile: Optional[str]) -> Callable[[str], bool]:
"""Return a check for the cache keys that belong to the active credential or profile.
The check runs :func:`cache_stem_is_relevant` on every path segment of a cache key,
because a service may embed the credential digest or profile name in a directory name
as well as in the basename. A profile with no credentials entry uses the default one.
"""
from unshackle.commands.dl import dl
credential = dl.get_credentials(service_tag, profile)
allowed = credential_cache_digests(credential) if credential else set()
active = profile or "default"
profiles = config.credentials.get(service_tag)
if isinstance(profiles, dict) and profile and profile not in profiles:
active = "default"
foreign = set(profiles) - {active} if isinstance(profiles, dict) else set()
def relevant(key: str) -> bool:
return all(cache_stem_is_relevant(part, allowed, active, foreign) for part in key.split("/"))
return relevant
class ServerVault(Vault):
"""The server's vaults as one source seen from the client: it can only be reported against.
@@ -944,6 +967,21 @@ class RemoteService:
create_data["quality"] = list(quality)
if self.ctx.parent.params.get("best_available"):
create_data["best_available"] = True
# Services pick the manifests to fetch from these. They go in their own
# object: flat, a service option with the same name would take them (or override them).
# An empty list is sent as-is, because only an absent key falls back to the dl default.
params = self.ctx.parent.params
dl_params: dict[str, Any] = {
key: [str(x) for x in params[key]]
for key in ("lang", "v_lang", "a_lang")
if params.get(key) is not None
}
if params.get("acodec") is not None:
dl_params["acodec"] = [c.name for c in params["acodec"]]
if "forced_subs" in params:
dl_params["forced_subs"] = bool(params["forced_subs"])
if dl_params:
create_data["dl_params"] = dl_params
if self._service_params:
create_data.update(self._service_params)
@@ -1400,18 +1438,21 @@ class RemoteService:
if session_id:
try:
result = self.client.delete(f"/api/session/{session_id}")
self.save_returned_cache(result.get("cache", {}))
profile = self.ctx.parent.params.get("profile") if self.ctx.parent else None
self.save_returned_cache(result.get("cache", {}), profile)
except SystemExit:
pass
except Exception as e:
self.log.warning(f"Failed to clean up remote session: {e}")
def save_returned_cache(self, cache_data: Dict[str, str]) -> None:
def save_returned_cache(self, cache_data: Dict[str, str], profile: Optional[str] = None) -> None:
"""Save cache files returned by the server to the local cache directory.
The server returns updated cache files (e.g. refreshed tokens) on
session close. Writing them locally means the next remote session
can forward them back, skipping interactive auth.
can forward them back, skipping interactive auth. The same filter that
:meth:`load_cache_files` applies drops a file that belongs to another
credential or profile, so it cannot overwrite that profile's local file.
"""
if not cache_data:
return
@@ -1425,59 +1466,56 @@ class RemoteService:
cache_dir = config.directories.cache / self.service_tag
cache_dir.mkdir(parents=True, exist_ok=True)
relevant = cache_key_filter(self.service_tag, profile)
saved = 0
for key, content in cache_data.items():
safe_name = safe_cache_key(key)
if not safe_name:
self.log.warning(f"Rejecting unsafe cache filename from server: {key!r}")
continue
if not relevant(safe_name):
self.log.debug(f"Ignoring cache file from the remote server for another profile: {safe_name}")
continue
try:
decompressed = safe_inflate(base64.b64decode(content))
# Not with_suffix: that would turn the key "keys.v2" into "keys.json".
target = cache_dir / f"{safe_name}.json"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(decompressed)
saved += 1
except Exception as e:
self.log.warning(f"Failed to save returned cache file '{safe_name}': {e}")
self.log.info(f"Saved {len(cache_data)} cache file(s) from server")
self.log.info(f"Saved {saved} cache file(s) from server")
def load_cache_files(self, profile: Optional[str] = None) -> Dict[str, str]:
"""Collect the cache files to forward, and withhold other profiles' files.
The client cannot rely on the server to filter, so it sends only the files
it can tie to the active credential or profile, plus service-global state.
At worst, a withheld file makes the server authenticate again.
it can tie to the active credential or profile (:func:`cache_key_filter`),
plus service-global state. At worst, a withheld file makes the server
authenticate again.
Each cache key is the file path relative to the service cache directory,
with posix separators and no ``.json`` suffix, so it equals the ``Cacher``
cache key the service reads it with (``session_web/<sha1>``). The relevance check
runs on every path segment, because a service may embed the credential
digest or profile name in a directory name as well as in the basename.
cache key the service reads it with (``session_web/<sha1>``).
"""
import zlib
from unshackle.commands.dl import dl
from unshackle.core.api.sanitize import safe_cache_key
cache_dir = config.directories.cache / self.service_tag
if not cache_dir.is_dir():
return {}
credential = dl.get_credentials(self.service_tag, profile)
allowed = credential_cache_digests(credential) if credential else set()
active = profile or "default"
profiles = config.credentials.get(self.service_tag)
if isinstance(profiles, dict) and profile and profile not in profiles:
active = "default"
foreign = set(profiles) - {active} if isinstance(profiles, dict) else set()
relevant = cache_key_filter(self.service_tag, profile)
files: Dict[str, str] = {}
for f in sorted(cache_dir.rglob("*.json")):
key = f.relative_to(cache_dir).as_posix()[: -len(".json")]
if f.stem.startswith("titles_") or not safe_cache_key(key):
continue
if not all(cache_stem_is_relevant(part, allowed, active, foreign) for part in key.split("/")):
if not relevant(key):
self.log.debug(f"Withholding cache file from the remote server: {key}")
continue
files[key] = base64.b64encode(zlib.compress(f.read_bytes())).decode("ascii")