feat(serve): per-api-key cdm and service config with server-side identity cache

serve.users.<key>.cdm and service_config give each API key its own device and service config. Service.IDENTITY_CACHE keys route to a per-key directory that clients never receive
This commit is contained in:
imSp4rky
2026-11-16 13:31:17 -07:00
parent 08481d3a33
commit f1e4081313
16 changed files with 508 additions and 63 deletions
+1 -1
View File
@@ -362,4 +362,4 @@ The REST API authenticates with `X-Secret-Key`. The middleware advertises `Autho
### Where device access is read
The map of accepted API keys lives at `request.app["config"]["users"]`. For DRM operations, an API key's permitted Widevine devices come from its `devices` list, and PlayReady devices from `playready_devices`. The server uses the first configured device when a request does not (or may not) override it. A server-wide `config.cdm[<service>]` mapping, if present, takes precedence over the API key's device list when the server chooses which device to use.
The map of accepted API keys lives at `request.app["config"]["users"]`. For DRM operations, an API key's permitted Widevine devices come from its `devices` list, and PlayReady devices from `playready_devices`. The server uses the first configured device when a request does not (or may not) override it. A `cdm` entry for the service takes precedence over the API key's device list when the server chooses which device to use: the API key's own `serve.users.<key>.cdm` entry, else the server-wide `config.cdm[<service>]` mapping.
+9 -2
View File
@@ -414,7 +414,12 @@ serve:
cannot read each other's cached tokens. The server deletes the directory when the
remote session ends. A remote session that uses a server account (`serve.server_accounts`)
gets a persistent per-account cache instead, so the account's tokens outlive the
remote session. The server and the client both refuse a forwarded cache file that is not
remote session. A service can name cache files that hold its device identity in
`Service.IDENTITY_CACHE`. A remote session on the server's device, or of an API key with its
own device config, writes those files to a directory of the API key, outside the remote
session's directory, so the server never sends them to the client and never takes them from
the client, and the next remote session of that
API key reads them again. The server and the client both refuse a forwarded cache file that is not
JSON or that names a type other than a date or time, and the service then authenticates again.
- **A tunnel belongs to its API key.** The server accepts `tunnel:<id>` only from the API key
that opened the tunnel. Its SOCKS5 proxy listens on loopback only, and it needs a username and
@@ -603,7 +608,9 @@ The handler writes forwarded `cache` files into that directory before authentica
On `DELETE`, when the login belongs to the client (`client_auth`) and the remote
session does not run on the server's device, the handler harvests updated cache files from the whole directory tree (compressing each
with zlib and base64-encoding, **excluding** `titles_*` files) and returns them
under a `cache` field, so the client can keep refreshed tokens. Each cache key
under a `cache` field, so the client can keep refreshed tokens. A remote session on the
server's device keeps the files of the service's `IDENTITY_CACHE` in
`_identity/<namespace>/<service>` instead, so they are never in that tree. Each cache key
is the file path relative to that directory, with `/` separators and no `.json`
suffix, which is the `Cacher` cache key the service reads it with:
+2
View File
@@ -19,6 +19,8 @@ a `widevine` or `playready` sub-entry, or a profile or `default` sub-entry. A pe
that selects only by quality, or only for a different profile, loads no device when the run
starts. When `server_cdm` is unset, unshackle then shows a warning, and the server decides who
licenses the remote session.
On a server, an API key can replace the entry for a service with its own (see
[`serve`](services.md#serve)).
With only the top-level `default` entry, the server CDM licenses when the API key has it. Set
`server_cdm: true` on the server entry to keep a per-service entry for local runs only. See
[`remote_services`](services.md#remote_services).
+36
View File
@@ -240,6 +240,42 @@ hold goes back to the client's device. When the service runs the remote session
own device (a service option, such as a server-identity flag), the client's own vaults, and the
server vault with this grant, are the only source, and a missing content key stops the download.
`cdm` gives the API key its own CDM device for a service. It maps a service tag to a device
name, or to `widevine`, `playready` and `default` sub-entries that name a device for each DRM
system. Per-profile sub-entries and a `default` service tag are not available here: name each
service tag. The entry replaces the whole top-level [`cdm`](drm.md#cdm) entry for that service,
both for the device the service sees when it starts and for the device that licenses its tracks.
A remote session keeps the API key's entry it started with until it ends, also after a reload. A
device name that names no device the server has, or any other sub-entry, stops the server at
startup or refuses the reload.
`service_config` gives the API key its own service config. It maps a service tag to config keys,
which the server merges over the service's `config.yaml` and the top-level `services` entry. A
section of the service config that sets one of the same plain (not mapping or list) keys, such as
a profile section with its own device identity, takes the API key's value too, so a client cannot
select another identity through its profile name. The values stay on the server: no response sends
them to a client. The cache that the service builds from them stays on the server only for a
service that lists its identity cache keys, and the server logs a warning for a service that lists
none. The server deletes that cache when a reload removes the API key.
```yaml
serve:
users:
"key-1":
server_cdm: [EXAMPLE]
cdm:
EXAMPLE: example_device_1
service_config:
EXAMPLE:
device_id: DEVICE-1
device_key: KEY-1
```
An API key with its own `cdm` or `service_config` entry for a service cannot submit
`/api/download` jobs for that service, because a job runs on the top-level config. On a server
account, it shares the account's login with the other API keys but keeps its own device identity
cache.
`admin` is a boolean that lets the API key run the maintenance endpoints (clear-cache, clear-temp, refresh-services). It is `false` unless the entry sets it. Keys that have no `users` entry, such as `api_secret`, keep that access.
`shared` is a boolean for an API key that a gateway in front of the server gives to many people.
+1 -1
View File
@@ -376,7 +376,7 @@ def test_a_lent_device_session_returns_no_cache_at_the_end(monkeypatch: pytest.M
monkeypatch.setattr(handlers, "get_validated_session", validated)
monkeypatch.setattr(session_store, "get_session_store", lambda: Store())
monkeypatch.setattr(handlers, "collect_cache_files", lambda tag: {"MSL/server-esn": "tokens"})
monkeypatch.setattr(handlers, "collect_cache_files", lambda tag, withhold=(): {"MSL/server-esn": "tokens"})
response = asyncio.run(handlers.session_delete_handler("sid", None))
+189
View File
@@ -0,0 +1,189 @@
"""An API key's own cdm and service_config entries, and the identity cache a server device keeps per key."""
import asyncio
import base64
import json
import zlib
from types import SimpleNamespace
import pytest
from unshackle.core.api import handlers
from unshackle.core.api.errors import APIError
from unshackle.core.api.input_bridge import AuthStatus
from unshackle.core.api.session_store import get_session_store
from unshackle.core.config import config
@pytest.fixture
def server(monkeypatch, tmp_path):
monkeypatch.setattr(config, "cdm", {"EXAMPLE": {"widevine": "shared_wv", "playready": "shared_pr"}, "OTHER": "x"})
monkeypatch.setattr(config.directories, "cache", tmp_path)
monkeypatch.setattr(handlers.Services, "get_tag", staticmethod(lambda tag: tag.upper()))
return tmp_path
def test_a_key_entry_replaces_the_whole_server_entry(server):
user = {"cdm": {"example": "own_pr"}}
assert handlers.key_cdm_config(user).cdm == {"OTHER": "x", "example": "own_pr"}
assert handlers.configured_cdm_name("EXAMPLE", "widevine", handlers.key_cdm_config(user)) == "own_pr"
assert handlers.key_cdm_config({}) is config
def test_licence_device_comes_from_the_key_entry(server):
assert handlers.resolve_device_name({"cdm": {"EXAMPLE": "own"}}, "playready", "EXAMPLE") == "own"
assert handlers.resolve_device_name({}, "playready", "EXAMPLE") == "shared_pr"
def test_a_session_keeps_the_user_config_it_was_created_with(server, monkeypatch):
session = SimpleNamespace(user_config={"cdm": {"EXAMPLE": "pinned"}})
monkeypatch.setattr(handlers, "serve_user_config", lambda key: {"cdm": {"EXAMPLE": "reloaded"}})
assert handlers.session_user_config(session, None)["cdm"]["EXAMPLE"] == "pinned"
assert handlers.session_user_config(SimpleNamespace(), None)["cdm"]["EXAMPLE"] == "reloaded"
def test_the_key_block_beats_the_server_config_and_profile_sections(server, monkeypatch):
base = {"device_id": "server", "device_key": "server", "p1": {"device_id": "p1", "other": 1}, "urls": {"a": "b"}}
monkeypatch.setattr(handlers, "load_service_yaml", lambda service: json.loads(json.dumps(base)))
user = {"service_config": {"example": {"device_id": "key", "device_key": "key"}}}
merged = handlers.key_service_yaml("EXAMPLE", user)
assert merged["device_id"] == merged["device_key"] == "key"
assert merged["p1"] == {"device_id": "key", "other": 1}
assert merged["urls"] == {"a": "b"}
assert handlers.key_service_yaml("EXAMPLE", {})["device_id"] == "server"
assert handlers.key_owns_identity(user, "EXAMPLE") and not handlers.key_owns_identity(user, "OTHER")
class _Service:
IDENTITY_CACHE = ("keys/",)
def test_identity_keys_live_in_the_api_key_directory_and_never_reach_the_client(server):
cache = handlers.keyed_cache("_s", _Service(), "EXAMPLE", "k", {}, temporary=True)
cache.get("keys/device1").set("keys")
cache.get("tokens").set("login")
assert set(handlers.collect_cache_files("_s")) == {"tokens"}
assert list((server / "_identity").rglob("device1.json"))
assert handlers.keyed_cache("_t", _Service(), "EXAMPLE", "k", {}, True).get("keys/device1").data == "keys"
text = base64.b64encode(zlib.compress(b'{"data": "planted"}')).decode("ascii")
handlers.write_client_cache({"keys/device1": text}, "_s")
assert cache.get("keys/device1").data == "keys"
def test_a_lasting_cache_keeps_identity_keys_apart_only_for_a_key_with_its_own_identity(server):
own = {"service_config": {"EXAMPLE": {"device_id": "key"}}}
assert handlers.keyed_cache("_accounts/EXAMPLE/p", _Service(), "EXAMPLE", "k", {}).route is None
assert handlers.keyed_cache("_accounts/EXAMPLE/p", _Service(), "EXAMPLE", "k", own).route is not None
assert handlers.keyed_cache("_s", object(), "EXAMPLE", "k", own, temporary=True).route is None
def test_a_job_is_refused_for_a_key_with_its_own_identity(server, monkeypatch):
monkeypatch.setattr(handlers, "server_cdm_allowed", lambda request, service: True)
monkeypatch.setattr(handlers, "server_cdm_max_height", lambda request, service: None)
monkeypatch.setattr(handlers, "serve_user_config", lambda key: {"cdm": {"EXAMPLE": "own"}})
with pytest.raises(APIError, match="its own"):
handlers.enforce_download_gates({"service": "EXAMPLE"})
@pytest.mark.parametrize(
"user_cfg, problem",
[
({"services": {"EXAMPLE": {}}}, "list of service tags"),
({"cdm": "dev"}, "mapping of service tag"),
({"cdm": {"default": "dev"}}, "a key has no default entry"),
({"service_config": {"DEFAULT": {"device_id": "x"}}}, "a key has no default entry"),
({"cdm": {"EXAMPLE": {"widevine": "dev", "tiers": "x"}}}, "device name or a widevine/playready/default"),
({"cdm": {"EXAMPLE": "missing"}}, "no CDM device named 'missing'"),
({"service_config": {"EXAMPLE": "device_id"}}, "mapping of config key"),
],
)
def test_bad_key_config_is_refused_at_load(server, monkeypatch, user_cfg, problem):
monkeypatch.setattr(handlers.Services, "get_tags", staticmethod(lambda: ["EXAMPLE"]))
assert problem in handlers.key_config_error("serve.users.u", user_cfg)
def test_a_known_device_passes_validation(server, monkeypatch):
monkeypatch.setattr(handlers.Services, "get_tags", staticmethod(lambda: ["EXAMPLE"]))
monkeypatch.setattr(handlers, "detect_cdm_type", lambda name, cfg: "playready")
assert handlers.key_config_error("u", {"cdm": {"EXAMPLE": {"playready": "dev"}}, "service_config": {}}) is None
assert handlers.key_config_error("u", {"cdm": {"EXAMPLE": {"default": "dev"}}, "service_config": None}) is None
assert handlers.key_config_error("u", {"cdm": {"EXAMPLE": {"widevine": "dev"}}}) is not None
@pytest.mark.parametrize(
"server_cdm, sent, own, returned",
[
(True, {}, True, {"tokens", "keys/planted"}),
(False, {}, True, {"tokens", "keys/planted"}),
(False, {"cdm_type": "playready", "cdm_relay": True}, True, {"tokens", "keys/planted"}),
(False, {"cdm_type": "playready", "cdm_relay": True}, False, {"tokens", "keys/device1", "keys/planted"}),
],
)
def test_a_session_keeps_identity_keys_from_the_client_unless_the_identity_is_the_clients(
server, monkeypatch, server_cdm, sent, own, returned
):
class Service(_Service):
_input_bridge = log = cache = None
def authenticate(self, cookies, credential):
self.cache.get("keys/device1").set("identity")
self.cache.get("tokens").set("login")
user = {"service_config": {"EXAMPLE": {"device_id": "key"}}} if own else {}
text = base64.b64encode(zlib.compress(b'{"data": "x"}')).decode("ascii")
monkeypatch.setattr(handlers, "validate_service", lambda service, request=None: service)
monkeypatch.setattr(handlers, "resolve_handler_proxy", lambda *args: (None, []))
monkeypatch.setattr(handlers, "server_account_for", lambda *args: False)
monkeypatch.setattr(handlers, "serve_user_config", lambda key: user)
monkeypatch.setattr(handlers, "choose_session_cdm", lambda *args: (server_cdm, None))
monkeypatch.setattr(handlers.Services, "load", lambda service: Service)
monkeypatch.setattr(handlers, "create_service_instance", lambda *args, **kwargs: (Service(), None, None))
async def run():
data = {"service": "EXAMPLE", "title_id": "t", "cookies": "x", "cache": {"keys/planted": text}, **sent}
response = await handlers.session_create_handler(data)
session = get_session_store().peek(json.loads(response.body)["session_id"])
try:
for _ in range(250):
if session.auth_status != AuthStatus.AUTHENTICATING:
break
await asyncio.sleep(0.02)
return session, set(handlers.collect_cache_files(session.cache_tag))
finally:
await get_session_store().delete(session.session_id)
session, sent_back = asyncio.run(run())
assert session.user_config is user
assert sent_back == returned
def test_a_reload_checks_key_devices_against_the_new_config(server, monkeypatch):
monkeypatch.setattr(handlers.Services, "get_tags", staticmethod(lambda: ["EXAMPLE"]))
fresh = SimpleNamespace(remote_cdm=[{"name": "new_remote", "device_type": "PLAYREADY"}])
user = {"cdm": {"EXAMPLE": "new_remote"}}
assert handlers.key_config_error("u", user, fresh) is None
assert handlers.key_config_error("u", user) is not None
def test_a_block_value_reaches_only_plain_values_of_other_sections(server, monkeypatch):
base = {"p1": {"device_id": "p1", "headers": {"a": "b"}}, "device_key": "server"}
monkeypatch.setattr(handlers, "load_service_yaml", lambda service: json.loads(json.dumps(base)))
block = {"device_id": "key", "headers": {"c": "d"}, "device_key": {"nested": 1}}
merged = handlers.key_service_yaml("EXAMPLE", {"service_config": {"EXAMPLE": block}})
assert merged["p1"] == {"device_id": "key", "headers": {"a": "b"}}
assert merged["device_key"] == {"nested": 1}
def test_a_removed_key_loses_its_identity_cache(server):
handlers.keyed_cache("_s", _Service(), "EXAMPLE", "gone", {}, True).get("keys/device1").set("keys")
handlers.keyed_cache("_s", _Service(), "EXAMPLE", "kept", {}, True).get("keys/device1").set("keys")
handlers.drop_identity_cache({"gone"})
assert not (server / "_identity" / handlers.key_namespace("gone")).exists()
assert (server / "_identity" / handlers.key_namespace("kept")).exists()
+1 -1
View File
@@ -126,7 +126,7 @@ def test_server_cdm_ignores_client_cdm_type(monkeypatch):
not swap it for a stub of the other DRM system."""
seen = []
monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {})
monkeypatch.setattr(handlers, "load_full_cdm", lambda s, p, t=None: seen.append(t))
monkeypatch.setattr(handlers, "load_full_cdm", lambda s, p, t=None, app_config=None: seen.append(t))
monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: object()))
monkeypatch.setattr(handlers, "instantiate_service", lambda *a: "svc")
data = {"cdm_type": "playready"}
@@ -132,6 +132,7 @@ async def test_batch_reports_vault_keys_from_the_init_segment_pssh(monkeypatch):
vault_only=False,
shared=None,
track_kid=None,
**_,
):
probed.append(track_kid)
if pssh_str == "init":
@@ -245,7 +245,9 @@ def test_the_server_device_follows_the_client_only_when_both_systems_are_mapped(
asked: list = []
monkeypatch.setattr(handlers.config, "cdm", mapping, raising=False)
monkeypatch.setattr(handlers, "load_service_yaml", lambda tag: {})
monkeypatch.setattr(handlers, "load_full_cdm", lambda service, profile, cdm_type: asked.append(cdm_type))
monkeypatch.setattr(
handlers, "load_full_cdm", lambda service, profile, cdm_type, app_config=None: asked.append(cdm_type)
)
monkeypatch.setattr(handlers, "build_parent_ctx", lambda *a, **k: None)
monkeypatch.setattr(handlers.Services, "load", lambda tag: object)
monkeypatch.setattr(handlers, "instantiate_service", lambda *a, **k: "service")
+11 -2
View File
@@ -5,6 +5,7 @@ import re
import signal
import subprocess
import sys
import threading
from contextlib import suppress
from datetime import datetime
from functools import partial
@@ -25,6 +26,8 @@ from unshackle.core.api.handlers import (
api_key_authentication,
dashboard_authentication,
dashboard_key,
drop_identity_cache,
key_config_error,
rate_limit_response,
request_secret_key,
server_account_regions,
@@ -130,9 +133,11 @@ RESTART_KEYS = ("remote_only", "services_refresh_interval")
log = logging.getLogger("serve")
def _prepare_serve(serve: dict, no_key: bool) -> Optional[str]:
def _prepare_serve(serve: dict, no_key: bool, app_config: Any = None) -> Optional[str]:
"""Glob the device directories into ``serve``, validate it in place and return the api_secret.
``app_config`` is the config ``serve`` belongs to, so a reload checks key devices against the new file.
Raises ValueError on a bad value. The api_secret is None under ``--no-key``.
"""
api_secret = None if no_key else serve.get("api_secret")
@@ -180,6 +185,8 @@ def _prepare_serve(serve: dict, no_key: bool) -> Optional[str]:
raise ValueError(problem)
if problem := server_cdm_max_height_error(f"serve.users.{username}", user_cfg.get("server_cdm_max_height")):
raise ValueError(problem)
if problem := key_config_error(f"serve.users.{username}", user_cfg, app_config):
raise ValueError(problem)
limit = key_rate_limit(user_key, serve)
if limit:
log.info(f"User '{username}' rate limited to {limit} requests/hour")
@@ -257,7 +264,7 @@ def reload_config(app: web.Application, *, no_key: bool, serve_widevine: bool, s
raise ValueError("no config file found")
fresh = Config.from_yaml(config_module.config_path)
serve = fresh.serve
api_secret = _prepare_serve(serve, no_key)
api_secret = _prepare_serve(serve, no_key, fresh)
except Exception as e:
log.error(f"Config reload rejected, keeping the running config: {e}")
return False
@@ -279,6 +286,8 @@ def reload_config(app: web.Application, *, no_key: bool, serve_widevine: bool, s
reset_remote_pool()
CATALOGUES.clear()
app["config"]["users"] = serve_config["users"]
if removed := old_users - new_users: # a PBKDF2 per key plus a tree delete: keep it off the event loop
threading.Thread(target=drop_identity_cache, args=(removed,), daemon=True).start()
if serve_widevine:
app["config"]["devices"] = _device_map(serve_config["devices"])
if playready_app := app.get("playready_app"):
+229 -51
View File
@@ -58,7 +58,7 @@ from unshackle.core.tunnel import (
)
from unshackle.core.utilities import declared_kwargs, kind_only_conflict, original_tag
from unshackle.core.utils.click_types import AUDIO_CODEC_LIST, QUALITY_LIST, VIDEO_CODEC_LIST
from unshackle.core.utils.collections import ci_get
from unshackle.core.utils.collections import ci_get, merge_dict
from unshackle.core.utils.hdr10plus import probed_tracks
from unshackle.core.utils.redact import REDACTED, URL_USERINFO_RE, redact_all, redact_secrets, redact_text
from unshackle.core.utils.service_params import FLAT_DL_KEYS, cast_service_options, service_options
@@ -192,18 +192,21 @@ LIST_HANDLER_TRANSPORT_KEYS = {
}
def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str] = None) -> Optional[Any]:
def load_full_cdm(
service: str, profile: Optional[str], cdm_type: Optional[str] = None, app_config: Any = None
) -> Optional[Any]:
"""Load a real CDM object for the given service.
Services often touch ``ctx.obj.cdm.security_level`` / ``.device_type`` / ``.system_id``
inside ``__init__``, so the lightweight ``resolve_server_cdm`` stub is not enough
for list_titles / list_tracks / search. Mirrors ``dl.get_cdm`` selection logic but
skips the quality-tier shortcuts (no track context yet) and falls back to the stub
if the configuration has no device, or if loading fails.
if the configuration has no device, or if loading fails. ``app_config`` is an API key's
view from :func:`key_cdm_config`; None reads the server's own ``cdm``.
"""
from unshackle.core.cdm import load_cdm
from unshackle.core.config import config as app_config
app_config = app_config or config
if not isinstance(cdm_type, str):
cdm_type = None
cdm_name = ci_get(app_config.cdm, service) or ci_get(app_config.cdm, "default")
@@ -221,7 +224,7 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]
cdm_name = cdm_name.get(profile) or cdm_name.get("default") or ci_get(app_config.cdm, "default")
if not cdm_name or not isinstance(cdm_name, str):
return resolve_server_cdm(service, profile, cdm_type)
return resolve_server_cdm(service, profile, cdm_type, app_config)
if cdm_type:
wanted = {"wv": "widevine", "widevine": "widevine", "pr": "playready", "playready": "playready"}.get(
@@ -236,7 +239,7 @@ def load_full_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]
log.warning(
f"load_cdm({sanitize_log(cdm_name)!r}) failed for {sanitize_log(service)}: {exc}; using lightweight stub"
)
return resolve_server_cdm(service, profile, cdm_type)
return resolve_server_cdm(service, profile, cdm_type, app_config)
SERVICE_YAML: Dict[Path, tuple[int, dict]] = {}
@@ -247,8 +250,6 @@ def load_service_yaml(normalized_service: str) -> dict:
The parsed file is kept until its mtime changes; each caller gets its own copy to change.
"""
from unshackle.core.utils.collections import merge_dict
service_config_path = Services.get_path(normalized_service) / config.filenames.config
try:
mtime = service_config_path.stat().st_mtime_ns
@@ -385,8 +386,8 @@ def write_client_cache(cache_data: Any, cache_tag: str) -> None:
Each cache key is a file path relative to that directory, and each value is the file as
base64 of zlib-compressed bytes. The function skips an unsafe key, a file that names a type
other than a datetime, or a file it cannot write, and raises INVALID_INPUT for a map that is
not an object, has too many entries, or holds a value that is not a string.
other than a datetime or a file it cannot write, and raises INVALID_INPUT for a map that is not an
object, has too many entries, or holds a value that is not a string.
"""
if not isinstance(cache_data, dict) or len(cache_data) > MAX_SESSION_CACHE_KEYS:
raise APIError(APIErrorCode.INVALID_INPUT, f"cache must hold at most {MAX_SESSION_CACHE_KEYS} entries")
@@ -451,19 +452,25 @@ class RequestCache:
normalized_service: str,
request: Optional[web.Request],
client_login: bool,
user_config: Dict[str, Any],
) -> None:
"""Give the service a new cache directory, seeded from the client's ``cache``.
Does nothing on a full-mode server, where the operator's own cache stays in use.
A full-mode server keeps the operator's own cache, with the key's own identity cache when it has one.
"""
import uuid
from unshackle.core.api.stats import stats
api_key = caller_key(request)
if stats.mode != "remote_only":
tag = service_instance.cache.service_tag
service_instance.cache = keyed_cache(tag, service_instance, normalized_service, api_key, user_config)
return
self.tag = f"_requests/{api_key_namespace(request)}/{uuid.uuid4()}/{normalized_service}"
service_instance.cache = Cacher(self.tag)
self.tag = f"_requests/{key_namespace(api_key)}/{uuid.uuid4()}/{normalized_service}"
service_instance.cache = keyed_cache(
self.tag, service_instance, normalized_service, api_key, user_config, temporary=True
)
cache_data = data.get("cache")
if cache_data:
write_client_cache(cache_data, self.tag)
@@ -849,7 +856,9 @@ def setup_list_service(
instantiate → authenticate. Raises APIError on proxy failure.
"""
service_config = load_service_yaml(normalized_service)
api_key = caller_key(request)
user_config = serve_user_config(api_key)
service_config = key_service_yaml(normalized_service, user_config)
no_proxy = data.get("no_proxy", False)
proxy_param, proxy_providers = resolve_handler_proxy(data, normalized_service, request)
@@ -858,7 +867,7 @@ def setup_list_service(
if account:
profile = None if account == "default" else account
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"), key_cdm_config(user_config))
cookies, credential = server_login_material(data, normalized_service, account, profile)
parent_ctx = build_parent_ctx(
@@ -874,10 +883,11 @@ def setup_list_service(
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
if account:
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}")
account_tag = f"_accounts/{normalized_service}/{account}"
service_instance.cache = keyed_cache(account_tag, service_instance, normalized_service, api_key, user_config)
elif request_cache is not None:
client_login = cookies is not None or credential is not None
request_cache.attach(service_instance, data, normalized_service, request, client_login)
request_cache.attach(service_instance, data, normalized_service, request, client_login, user_config)
service_instance._input_bridge = NoUserBridge()
service_instance.authenticate(cookies, credential)
return service_instance
@@ -902,14 +912,16 @@ def run_service_search(
profile = client_profile(data)
no_proxy = data.get("no_proxy", False)
service_config = load_service_yaml(normalized_service)
api_key = caller_key(request)
user_config = serve_user_config(api_key)
service_config = key_service_yaml(normalized_service, user_config)
proxy_param, proxy_providers = resolve_handler_proxy(data, normalized_service, request)
account = resolve_server_account(request, normalized_service, data.get("client_region"))
if account:
profile = None if account == "default" else account
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"), key_cdm_config(user_config))
service_module = Services.load(normalized_service)
anonymous = getattr(service_module, "ANONYMOUS_SEARCH", False)
cookies, credential = (
@@ -940,10 +952,11 @@ def run_service_search(
)
if account:
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}")
account_tag = f"_accounts/{normalized_service}/{account}"
service_instance.cache = keyed_cache(account_tag, service_instance, normalized_service, api_key, user_config)
elif request_cache is not None:
client_login = cookies is not None or credential is not None
request_cache.attach(service_instance, data, normalized_service, request, client_login)
request_cache.attach(service_instance, data, normalized_service, request, client_login, user_config)
service_instance._input_bridge = NoUserBridge()
if not anonymous:
service_instance.authenticate(cookies, credential)
@@ -1026,6 +1039,103 @@ def serve_user_config(api_key: str) -> Dict[str, Any]:
}
class KeyCdmConfig:
"""The app config as one API key sees it: ``cdm`` is the key's map, every other setting is the server's."""
def __init__(self, cdm: Dict[str, Any]) -> None:
self.cdm = cdm
def __getattr__(self, name: str) -> Any:
return getattr(config, name)
def own_cdm(user_config: Optional[Dict[str, Any]]) -> Dict[str, Any]:
"""The API key's own ``cdm`` map; empty when it sets none."""
own = (user_config or {}).get("cdm")
return own if isinstance(own, dict) else {}
def key_cdm_config(user_config: Optional[Dict[str, Any]]) -> Any:
"""The config the CDM lookups read for an API key: ``config`` itself unless the key maps a device of its own.
Each service entry of the key's own ``cdm`` replaces the server's whole entry for that service.
"""
own = own_cdm(user_config)
if not own:
return config
tags = {str(tag).lower() for tag in own}
return KeyCdmConfig({**{t: e for t, e in (config.cdm or {}).items() if str(t).lower() not in tags}, **own})
def key_service_config(user_config: Optional[Dict[str, Any]], service: str) -> Dict[str, Any]:
"""The API key's own ``service_config`` block for ``service``; empty when it sets none."""
blocks = (user_config or {}).get("service_config")
block = ci_get(blocks, Services.get_tag(service)) if isinstance(blocks, dict) else None
return block if isinstance(block, dict) else {}
def key_owns_identity(user_config: Optional[Dict[str, Any]], service: str) -> bool:
"""Whether the API key sets its own device or service config for ``service``."""
return bool(key_service_config(user_config, service)) or ci_get(own_cdm(user_config), service) is not None
def key_service_yaml(normalized_service: str, user_config: Optional[Dict[str, Any]]) -> dict:
""":func:`load_service_yaml` with the API key's own block merged over it.
A top-level section that sets one of the block's plain values (a profile section with its own identity) takes
the key's value too, so a client cannot pick the server's identity back through the profile it names.
"""
service_config = load_service_yaml(normalized_service)
block = key_service_config(user_config, normalized_service)
if block:
merge_dict(copy.deepcopy(block), service_config)
scalars = {key for key, value in block.items() if not isinstance(value, (dict, list))}
for name, section in service_config.items():
if isinstance(section, dict) and name not in block:
for key in scalars & section.keys():
section[key] = block[key]
return service_config
def keyed_cache(
tag: str,
service_instance: Any,
service: str,
api_key: str,
user_config: Dict[str, Any],
temporary: bool = False,
client_identity: bool = False,
) -> Cacher:
"""A cache on ``tag`` with the service's ``IDENTITY_CACHE`` keys in the API key's own directory.
Those cache keys move there when the API key has its own identity for the service, or when ``tag`` goes when the
request ends (``temporary``) and the identity is not the client's own (``client_identity``), so no client
receives an identity the server built and no other API key shares it.
"""
prefixes = tuple(getattr(service_instance, "IDENTITY_CACHE", ()))
owns = key_owns_identity(user_config, service)
if owns and not prefixes:
warn_no_identity_cache(service)
if prefixes and (owns or (temporary and not client_identity)):
return Cacher(tag, route=(prefixes, f"_identity/{key_namespace(api_key)}/{service}"))
return Cacher(tag)
@functools.lru_cache(maxsize=None)
def warn_no_identity_cache(service: str) -> None:
"""Warn once per service that an API key's own config runs on a service that marks no identity cache keys."""
log.warning(f"{service}: an API key sets its own config, but the service lists no IDENTITY_CACHE")
def drop_identity_cache(api_keys: Iterable[str]) -> None:
"""Delete the identity cache directories of API keys the server no longer has."""
from unshackle.core.api.session_store import SessionStore
if (config.directories.cache / "_identity").is_dir():
for api_key in api_keys:
SessionStore.cleanup_cache_dir(f"_identity/{key_namespace(api_key)}")
def server_cdm_allowed(request: Optional[web.Request] = None, service: Optional[str] = None) -> bool:
"""Whether the calling API key may have the server operate the CDM licensing for ``service``."""
return key_grant_allows(request, service, "server_cdm")
@@ -1081,6 +1191,46 @@ def server_cdm_max_height_error(where: str, value: Any) -> Optional[str]:
return None
def key_config_error(where: str, user_cfg: Dict[str, Any], app_config: Any = None) -> Optional[str]:
"""Why the ``services``, ``cdm`` or ``service_config`` of one serve.users entry cannot work; None when it can.
Devices are looked up in ``app_config`` (a reload passes the new file). A tag that names no service the server
has is logged, not refused: a later services refresh can add it.
"""
app_config = app_config or config
if isinstance(user_cfg.get("services"), dict):
return f"{where}.services must be a list of service tags; per-service settings go under service_config"
cdm, service_config = user_cfg.get("cdm") or {}, user_cfg.get("service_config") or {}
for name, entries in (("cdm", cdm), ("service_config", service_config)):
if not isinstance(entries, dict):
return f"{where}.{name} must be a mapping of service tag to settings"
if any(str(tag).lower() == "default" for tag in entries):
return f"{where}.{name}.default: name each service tag; a key has no default entry"
for tag, entry in cdm.items():
named = {"default": entry} if isinstance(entry, str) else entry if isinstance(entry, dict) else {}
named = {str(system).lower(): device for system, device in named.items()}
if (
not named
or not set(named) <= {"widevine", "playready", "default"}
or not all(isinstance(device, str) for device in named.values())
):
return f"{where}.cdm.{tag} must be a device name or a widevine/playready/default mapping of device names"
for system, device in named.items():
found = detect_cdm_type(device, app_config)
if found is None or system != "default" and found != system:
return f"{where}.cdm.{tag}: no {'CDM' if system == 'default' else system} device named '{device}'"
for tag, entry in service_config.items():
if not isinstance(entry, dict):
return f"{where}.service_config.{tag} must be a mapping of config key to value"
if not cdm and not service_config:
return None
known = {tag.lower() for tag in Services.get_tags()}
for tag in [*cdm, *service_config]:
if str(tag).lower() not in known:
log.warning(f"{where}: no service {tag} on this server")
return None
def server_cdm_max_height(request: Optional[web.Request], service: Optional[str]) -> Optional[int]:
"""The tallest video track the server CDM licenses live for this API key on ``service``; None for no cap.
@@ -2297,6 +2447,12 @@ def enforce_download_gates(params: Dict[str, Any], request: Optional[web.Request
APIErrorCode.FORBIDDEN,
"Download jobs license with the server CDM, which is not enabled for this API key on this service",
)
service = params.get("service")
if service and key_owns_identity(serve_user_config(caller_key(request)), service):
raise APIError(
APIErrorCode.FORBIDDEN,
"Download jobs run on the server's own device config, and this API key has its own for this service",
)
cap = server_cdm_max_height(request, params.get("service"))
if cap is not None:
serve_defaults = config.serve or {}
@@ -3694,9 +3850,12 @@ def create_service_instance(
server_cdm: bool = False,
client_device: bool = False,
input_bridge: Optional[InputBridge] = None,
user_config: Optional[Dict[str, Any]] = None,
) -> Any:
"""Make a service instance and resolve its credentials and cookies.
``user_config`` is the calling API key's serve.users entry: its own ``cdm`` and ``service_config`` apply.
With ``client_device`` the service sees a stand-in for the client's device, which relays CDM calls
through ``input_bridge`` when the client can answer them, and a config holding the client's own identity.
@@ -3708,8 +3867,9 @@ def create_service_instance(
from unshackle.commands.dl import dl
from unshackle.core.tracks import Video
server_config = service_config = load_service_yaml(normalized_service)
server_config = service_config = key_service_yaml(normalized_service, user_config)
service_module = Services.load(normalized_service)
key_config = key_cdm_config(user_config)
cdm: Any
client_identity = False
if client_device:
@@ -3724,12 +3884,12 @@ def create_service_instance(
if service_config is not server_config:
service_config.clear()
service_config.update(server_config)
return load_full_cdm(normalized_service, profile, None)
return load_full_cdm(normalized_service, profile, None, key_config)
cdm = client_device_cdm(data, input_bridge, lend_server_device)
else:
follows_client = not server_cdm or lists_both_systems(normalized_service, config)
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type") if follows_client else None)
follows_client = not server_cdm or lists_both_systems(normalized_service, key_config)
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type") if follows_client else None, key_config)
# Reconstruct enum track-selection params from client data so service code that reads
# ctx.parent.params (Service.__init__ proxy/range/vcodec/best_available block) sees enums.
@@ -3820,6 +3980,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
session_id = str(uuid_mod.uuid4())
api_key = caller_key(request)
user_config = serve_user_config(api_key)
session_cache_dir = f"_sessions/{api_key_namespace(request)}/{session_id}/{normalized_service}"
session_cache_tag: Optional[str] = session_cache_dir
@@ -3857,17 +4018,28 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
server_cdm=use_server_cdm,
client_device=client_device,
input_bridge=bridge,
user_config=user_config,
)
service_instance, cookies, credential = await asyncio.to_thread(build_service)
if log_buffer:
service_instance.log = SessionLogMirror(service_instance.log, log_buffer)
# the service asked for the server's device in __init__, so the session never licenses live
stand_in = getattr(getattr(getattr(service_instance, "ctx", None), "obj", None), "cdm", None)
server_device = getattr(stand_in, "lent", False) is True
cache_tag = f"_accounts/{normalized_service}/{profile or 'default'}" if server_account else session_cache_dir
if server_account:
session_cache_tag = None
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{profile or 'default'}")
else:
service_instance.cache = Cacher(session_cache_dir)
service_instance.cache = keyed_cache(
cache_tag,
service_instance,
normalized_service,
api_key,
user_config,
temporary=not server_account,
client_identity=client_device and not server_device,
)
cache_data = data.get("cache", {})
if cache_data and session_cache_tag:
@@ -3885,12 +4057,11 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
server_account=(profile or "default") if server_account else None,
)
session.cache_tag = session_cache_tag
session.user_config = user_config
session.server_cdm = use_server_cdm
session.server_cdm_max_height = max_height
session.server_vault = client_device and server_vault_allowed(request, normalized_service)
# the service asked for the server's device in __init__, so the session never licenses live
stand_in = getattr(getattr(getattr(service_instance, "ctx", None), "obj", None), "cdm", None)
session.server_device = getattr(stand_in, "lent", False) is True
session.server_device = server_device
session.client_auth = not server_account and (
cookies is not None or credential is not None or bool(cache_data and session_cache_tag)
)
@@ -4026,6 +4197,15 @@ def cookie_values(cookies: list[Dict[str, Any]]) -> Dict[str, str]:
return {c["name"]: c["value"] for c in cookies}
def session_user_config(session: Any, request: Optional[web.Request]) -> Dict[str, Any]:
"""The serve.users entry a remote session was created with, so a reload cannot change it for that session.
An API key with no ``cdm`` entry of its own for the service still follows the server's ``cdm`` after a reload.
"""
pinned = getattr(session, "user_config", None)
return pinned if isinstance(pinned, dict) else serve_user_config(caller_key(request))
def client_owns_login(session: Any) -> bool:
"""Whether the login state of ``session`` is the client's to take back.
@@ -4113,10 +4293,8 @@ async def session_tracks_handler(
session_headers, session_cookies = client_session_auth(session)
from unshackle.core.config import config as app_config
api_key = caller_key(request)
user_cfg = serve_user_config(api_key)
user_cfg = session_user_config(session, request)
app_config = key_cdm_config(user_cfg)
has_wv = bool(user_cfg.get("devices"))
has_pr = bool(user_cfg.get("playready_devices"))
@@ -4249,7 +4427,9 @@ async def session_segments_handler(
)
def resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[str]) -> Optional[Any]:
def resolve_server_cdm(
service: str, profile: Optional[str], cdm_type: Optional[str], app_config: Any = None
) -> Optional[Any]:
"""Get the CDM for the server context.
Checks the server's own CDM config (``config.cdm[service]``) to
@@ -4261,8 +4441,7 @@ def resolve_server_cdm(service: str, profile: Optional[str], cdm_type: Optional[
Falls back to a lightweight stub from *cdm_type* only if the configuration
has no server CDM for the service.
"""
from unshackle.core.config import config as app_config
app_config = app_config or config
cdm_name = ci_get(app_config.cdm, service)
if cdm_name:
if isinstance(cdm_name, dict):
@@ -4751,15 +4930,13 @@ def ensure_track_drm(track: Any, session: Any = None, init_data: Optional[bytes]
def resolve_device_name(user_config: dict, drm_type: str, service_tag: str = "") -> str:
"""Get the CDM device name, checking service-specific config.cdm first.
"""Get the CDM device name, checking the service's cdm mapping first.
Resolution order:
1. config.cdm[service_tag] (service-specific CDM mapping)
1. the key's own ``cdm`` entry for the service, else ``config.cdm[service_tag]``
2. serve.users.{key}.devices / playready_devices (user device list)
"""
from unshackle.core.config import config as app_config
cdm_name = configured_cdm_name(service_tag, drm_type, app_config)
cdm_name = configured_cdm_name(service_tag, drm_type, key_cdm_config(user_config))
if cdm_name:
return cdm_name
@@ -4794,8 +4971,7 @@ def server_drm_candidates(
system, so the route never plans a licence that fails on the wrong device type. The
track's own drm_preference moves its system to the front when the server has it.
"""
from unshackle.core.config import config as app_config
app_config = key_cdm_config(user_config)
server_type = detect_cdm_type_for_service(service_tag, app_config)
first = server_type or (client_drm_type if client_drm_type in ("widevine", "playready") else "widevine")
candidates: List[str] = []
@@ -5022,6 +5198,7 @@ def handle_single_server_cdm(
vault_only: bool = False,
shared: Optional[Dict[str, Any]] = None,
track_kid: Optional[uuid.UUID] = None,
user_config: Optional[Dict[str, Any]] = None,
) -> Dict[str, str]:
"""Do the single-track server_cdm licensing with the DRM class get_content_keys() flow.
@@ -5058,8 +5235,8 @@ def handle_single_server_cdm(
return vault_keys
raise refusal or APIError(APIErrorCode.NO_CONTENT, "The server vault holds no content key for this track")
api_key = caller_key(request)
user_config = serve_user_config(api_key)
if user_config is None:
user_config = serve_user_config(caller_key(request))
if drm_type == "playready":
from pyplayready.system.pssh import PSSH as PlayReadyPSSH
@@ -5338,8 +5515,7 @@ def license_response(
)
if mode == "server_cdm" and track_ids:
api_key = caller_key(request)
user_config = serve_user_config(api_key)
user_config = session_user_config(session, request)
service = session.service_instance
service_tag = session.service_tag
@@ -5411,6 +5587,7 @@ def license_response(
vault_only=vault_only,
shared=shared,
track_kid=track_kid,
user_config=user_config,
)
if keys:
keys_by_pssh[cache_key] = {**keys_by_pssh[cache_key], **keys}
@@ -5523,11 +5700,11 @@ def license_response(
if mode == "server_cdm":
# The server's config.cdm mapping decides the DRM system, as in the batch path.
# The client only knows its own local device, which the server never uses here.
api_key = caller_key(request)
user_config = session_user_config(session, request)
candidates = (
vault_candidates(drm_type)
if vault_only
else server_drm_candidates(session.service_tag, serve_user_config(api_key), drm_type, track)
else server_drm_candidates(session.service_tag, user_config, drm_type, track)
)
picked = pick_server_pssh(track, candidates)
if not picked:
@@ -5559,6 +5736,7 @@ def license_response(
live_licence_refusal(session, track),
vault_only=vault_only,
track_kid=asked_kid,
user_config=user_config,
)
log.info(f"Server CDM resolved {len(keys)} key(s) for track {track_log_label(track)}")
note_served_keys(session, keys, key_sources)
+2 -1
View File
@@ -1224,7 +1224,8 @@ async def download(request: web.Request) -> web.Response:
description: Invalid request
'403':
description: >-
A gated parameter is not permitted (FORBIDDEN), or the job goes above the
A gated parameter is not permitted, or the API key has its own cdm or
service_config entry for the service (FORBIDDEN), or the job goes above the
server_cdm_max_height of the API key (SERVER_CDM_CAPPED)
"""
data = await json_body(request)
+1
View File
@@ -53,6 +53,7 @@ class SessionEntry:
server_vault: bool = False # a session the client's device licenses may still take keys from the server vault
server_device: bool = False # its service runs on the server's lent device, so keys come only from a vault
client_auth: bool = False # the login is the client's: it succeeded without a server account
user_config: Optional[Dict[str, Any]] = None # the serve.users entry the session was created with
input_bridge: Optional[InputBridge] = None
log_buffer: Optional[Any] = None # SessionLogBuffer mirroring the service's self.log
auth_status: AuthStatus = AuthStatus.AUTHENTICATED
+16 -2
View File
@@ -1,5 +1,7 @@
from __future__ import annotations
import os
import threading
import time
import zlib
from datetime import datetime
@@ -28,8 +30,11 @@ class Cacher:
version: Optional[int] = 1,
data: Optional[Any] = None,
expiration: Optional[datetime] = None,
route: Optional[tuple[tuple[str, ...], str]] = None,
) -> None:
"""``route`` is ``(prefixes, tag)``: a cache key that starts with one of ``prefixes`` lives under ``tag``."""
self.service_tag = service_tag
self.route = route
self.key = key
self.version = version
self.data: Any = data or {}
@@ -63,7 +68,8 @@ class Cacher:
:param version: the config data version you expect to use.
:returns: Cache object containing the cached data or None if the file does not exist.
"""
cache = Cacher(self.service_tag, key, version)
tag = self.route[1] if self.route and key.startswith(self.route[0]) else self.service_tag
cache = Cacher(tag, key, version, route=self.route)
if cache.path.is_file():
data = jsonpickle.loads(cache.path.read_text(encoding="utf8"))
payload = data.copy()
@@ -108,7 +114,15 @@ class Cacher:
payload["crc32"] = zlib.crc32(jsonpickle.dumps(payload).encode("utf8"))
self.path.parent.mkdir(parents=True, exist_ok=True)
self.path.write_text(jsonpickle.dumps(payload))
# a reader must never see a half-written file; Windows refuses to replace a file a reader holds open
target = (
self.path
if os.name == "nt"
else self.path.with_name(f"{self.path.name}.{os.getpid()}.{threading.get_ident()}.tmp")
)
target.write_text(jsonpickle.dumps(payload))
if target != self.path:
os.replace(target, self.path)
return self.data
+3
View File
@@ -173,6 +173,9 @@ class Service(metaclass=ABCMeta):
# Every service option that makes the service call ``cdm.lend_server_device()``. A remote session with
# one of them set keeps the server CDM even when the remote client picked its own device.
SERVER_DEVICE_OPTIONS: tuple[str, ...] = ()
# Cache key prefixes that hold state built from the device identity (a device ID, device session keys), never
# login state. A remote session on the server's device keeps them on the server, per API key.
IDENTITY_CACHE: tuple[str, ...] = ()
def __init__(self, ctx: click.Context) -> None:
console.print(Padding(Rule(f"[rule.text]Service: {self.__class__.__name__}"), (1, 2)))
+3 -1
View File
@@ -57,7 +57,9 @@ def merge_dict(source: dict, destination: dict) -> None:
return
for key, value in source.items():
if isinstance(value, dict):
node = destination.setdefault(key, {})
node = destination.get(key)
if not isinstance(node, dict):
node = destination[key] = {}
merge_dict(value, node)
else:
destination[key] = value