diff --git a/docs/dev/rest-api/dashboard.md b/docs/dev/rest-api/dashboard.md index fd85cf38..8c608e36 100644 --- a/docs/dev/rest-api/dashboard.md +++ b/docs/dev/rest-api/dashboard.md @@ -44,7 +44,7 @@ All are `GET`. | Path | Returns | |------|---------| | `/api/dashboard/status` | Version, `code_hash`, bind host and port, `mode` (`full`, `api_only`, `remote_only`), uptime, `requests_total`, `requests_rejected` (401s), `requests_by_key` (label → count), loaded service count, session count, `max_sessions`, `session_ttl`, job counts by status. | -| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the first resolved title's display name), `auth_status`, `auth_error`, `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`. `client` is whatever the remote client sent as `client` in its session create request (the current CLI sends nothing, so it is `{}`). `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. | +| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the first resolved title's display name), `auth_status`, `auth_error`, `server_account` (the server profile lent to the remote session, else `null`), `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`. `client` is whatever the remote client sent as `client` in its session create request (the current CLI sends nothing, so it is `{}`). `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. | | `/api/dashboard/jobs` | Every download job with full detail, regardless of owner. Empty in `--remote-only` mode. | | `/api/dashboard/logs` | `{"seq": N, "records": [...]}`: the last 1000 log records. `?since=` returns only newer records; `?level=WARNING` sets the minimum level; `?logger=serve` keeps one logger and its children (`aiohttp.access` is the noisiest). `ts` is a Unix epoch in seconds. | | `/api/dashboard/events` | Server-Sent Events stream. With `?since=` it returns `{"seq", "stats", "events": [...]}` instead: every event after `seq` (the server keeps the last 20000, log records included, so a poll every few seconds is safe even under a busy access log) as one JSON burst, so a UI that polls every few seconds still sees a remote session that opened and closed in between. | diff --git a/docs/dev/rest-api/endpoints.md b/docs/dev/rest-api/endpoints.md index 4a0c45a5..37900215 100644 --- a/docs/dev/rest-api/endpoints.md +++ b/docs/dev/rest-api/endpoints.md @@ -120,6 +120,7 @@ Field notes: | `has_search` | Whether the service supports `POST /api/search`. | | `has_drm` | Whether the service overrides a Widevine or PlayReady license hook. | | `pending_update` | Set to `true` while a service repository refresh waits for this service's jobs to finish before it swaps in the new code. Absent otherwise. | +| `server_accounts` | Present when the server lends its own accounts for this service: `{regions: ["ca", "gb"], global: bool}`, the regions those accounts cover. A client sends no credentials, cookies, or cache for such a service. | | `auth_methods` | Declared or inferred auth methods, e.g. `["cookies"]`, `["credentials"]`. | | Status | Meaning | @@ -833,7 +834,7 @@ DELETE /api/session/{id} → tear down, harvest updated cache ### `POST /api/session/create` -Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files), `client_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, and any service CLI options. +Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files), `client_region`, `proxy_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, and any service CLI options. `proxy_region` is the country code the client resolved `proxy` from; the server matches it against its own accounts. The `proxy` value must be a full proxy URI, unless the operator gives your API key `server_proxy` in the `serve.users` config. Without it, the server does not resolve country codes with its own proxy providers. It rejects the request with `INVALID_PROXY` when no `proxy` is set and the reported `client_region` differs from the server's region. A request that reports no `client_region` is not blocked. Pass `proxy` with your own proxy, or `no_proxy` to accept the server's own connection. With `server_proxy`, the server resolves a country code and picks a proxy for your `client_region` itself. @@ -843,10 +844,13 @@ The `proxy` value must be a full proxy URI, unless the operator gives your API k { "session_id": "d4e5f6a7-...", "service": "EXAMPLE", - "status": "authenticating" + "status": "authenticating", + "server_account": false } ``` +`server_account` is `true` when the server authenticated with one of its own accounts (`serve.server_accounts`). Otherwise the server uses only what the client sent, and never falls back to its own credentials. + | Status | Error code | Meaning | | --- | --- | --- | | `200` | - | Session created; authentication in progress. | diff --git a/docs/dev/rest-api/remote-sessions.md b/docs/dev/rest-api/remote-sessions.md index 3d3b78f4..ce129ca5 100644 --- a/docs/dev/rest-api/remote-sessions.md +++ b/docs/dev/rest-api/remote-sessions.md @@ -126,6 +126,18 @@ hood the client walks a remote session through its lifecycle. the server fetches the right manifests - Your local per-service **cache files** (e.g. refreshed tokens) + The server uses its own accounts only when the operator lists the service in + `serve.server_accounts` and gives your API key `server_accounts` in `serve.users`. + For any other service, a client that sends no credentials or cookies authenticates + with nothing. For a listed service the client sends no credentials, cookies, or + cache at all: the server picks one of its own accounts that works in your region and + rotates through them. `--proxy ca` picks a `ca` account. + + `GET /api/services` advertises the regions those accounts cover. When your own + region is not one of them and you set no `--proxy`, the client resolves a proxy for + the first advertised region itself, so both sides sit in a region the account + works in. + The server responds immediately with a session ID and a status. Authentication runs in the background on the server. @@ -247,7 +259,9 @@ serve: - **Namespaced, isolated cache.** Each remote session gets its own cache directory, namespaced by a hash of the API key and the remote session ID, so remote sessions cannot read each other's cached tokens. The server deletes the directory when the - remote session ends. + remote session ends. A remote session that uses a server account (`serve.server_accounts`) + gets a persistent per-account cache instead, so the account's tokens outlive the + remote session. - **The server redacts secrets in logs.** It redacts remote session IDs, service tags, and other user values before it writes them to the log. diff --git a/docs/reference/configuration/services.md b/docs/reference/configuration/services.md index c39005e2..f4574ef2 100644 --- a/docs/reference/configuration/services.md +++ b/docs/reference/configuration/services.md @@ -128,6 +128,7 @@ is the [REST API](../../dev/rest-api/index.md) section. These are the config key | `global_speed_limit` | str | *(unlimited)* | Server-wide download speed cap, e.g. `10M`, `1.5G` or plain bytes/sec (same format as `speed_limit`). One shared budget across all concurrent jobs; per-job speed limits are ignored while it is set. | | `cdm_overrides` | list or bool | *(unset)* | Allowed per-request CDM overrides: a list of permitted device names, or `true` for any. Unset rejects every override. | | `allow_job_credentials` | bool | `false` | Permit clients to supply credentials per job. | +| `server_accounts` | dict | `{}` | Services whose remote sessions use the server's own `credentials` and cookie files instead of the client's (see below). | | `devices` | list | *(auto)* | Widevine devices offered; auto-filled from `directories.wvds`. | | `playready_devices` | list | *(auto)* | PlayReady devices; auto-filled from `directories.prds`. | @@ -145,6 +146,54 @@ client that asks anyway gets a `FORBIDDEN` error. Because a download job always an API key without `server_cdm` for that service also cannot submit or retry `/api/download` jobs. Keys that have no `users` entry, such as `api_secret`, keep server CDM access. +`server_accounts` lists, per service, the accounts the server lends to remote sessions. A remote +session for a listed service ignores the credentials, cookies, and cache the client sends and +uses one of the server's own profiles from `credentials`. For a service that is not listed the +server never uses its own accounts: a client that sends nothing authenticates with nothing. + +```yaml +credentials: + EXAMPLE1: + ca_main: user1@example.com:pw1 + eu_box: user2@example.com:pw2 + shared: user3@example.com:pw3 + +serve: + server_accounts: + EXAMPLE1: + ca_main: ca # one country + eu_box: [gb, fr, de] # a list of countries + shared: global # any region + EXAMPLE2: true # every profile (or the single credential), any region +``` + +Each entry under a service names a profile from `credentials`, or a cookie file +`cookies//.txt` for a service that has no credentials. Its value is the region or +regions that account works in: a two-letter country code, a list of them, or `global`. Quote +`no` (Norway), because YAML reads the bare word as `false`. + +The region of a remote session is the country the client asked for with `--proxy` (`ca`, `ca1`, +`provider:ca`), the server's own region with `--no-proxy`, or the client's own region otherwise. +A full proxy URI carries no country, so with one only a `global` account matches. The server +picks the profiles that cover that region (or are `global`) and rotates through them +round-robin, one account per remote session. With no matching account it rejects the remote +session. These services ignore the client's `-p/--profile`. + +Cookies come only from the profile's own file (`cookies/EXAMPLE1/ca_main.txt`; the server does +not read a bare `cookies/EXAMPLE1.txt`), and each account keeps its own persistent cache under +`cache/_accounts/EXAMPLE1//`, so refreshed tokens survive the remote session but never +cross accounts. `serve` refuses to start when a listed profile has neither an entry under +`credentials` nor a cookie file. +`/api/search`, `/api/list-titles` and `/api/list-tracks` use the same pool, matched on the +client's reported region, and return `FORBIDDEN` to an API key without the opt-in. Rotation is +per process and does not track which accounts are in use, so concurrent remote sessions can +share one account. + +A `users` entry opts in to those accounts with `server_accounts: true` (every listed service) +or a list of service tags. It is off unless set: an API key without it authenticates with the +credentials the client sends, as for any other service, and `/api/services` does not advertise +the server's accounts to it. API keys with no `users` entry, such as `api_secret`, keep access. + `server_proxy` decides whether that API key may use the proxy providers the server has configured. There is no implicit access: only the boolean `true` on the API key's `users` entry grants it, and an API key without a `users` entry has no access either. Without the opt-in the @@ -173,6 +222,7 @@ serve: c9d0e1f2: services: [EXAMPLE1, EXAMPLE2] server_cdm: [EXAMPLE1] # the server licenses only EXAMPLE1; EXAMPLE2 needs a local CDM + server_accounts: [EXAMPLE1] # may authenticate EXAMPLE1 with one of the server's own accounts server_proxy: true # this key may also use the server's proxy providers ``` diff --git a/tests/remote/unit/test_server_accounts.py b/tests/remote/unit/test_server_accounts.py new file mode 100644 index 00000000..c229a309 --- /dev/null +++ b/tests/remote/unit/test_server_accounts.py @@ -0,0 +1,121 @@ +import pytest + +from unshackle.core.api import handlers +from unshackle.core.api.errors import APIError + + +@pytest.fixture +def accounts(monkeypatch): + monkeypatch.setattr( + handlers.config, "credentials", {"EX": {"ca_main": "a:b", "eu_box": "c:d", "shared": "e:f"}, "SINGLE": "u:p"} + ) + monkeypatch.setattr( + handlers.config, + "serve", + { + "server_accounts": { + "EX": {"ca_main": "ca", "eu_box": ["gb", "fr", "de"], "shared": "global"}, + "SINGLE": True, + } + }, + ) + monkeypatch.setattr(handlers.Services, "get_tag", staticmethod(lambda s: s)) + handlers._account_counters.clear() + + +def test_pool_filters_by_region(accounts): + assert handlers.server_account_pool("EX", "gb") == ["eu_box", "shared"] + assert handlers.server_account_pool("EX", "CA") == ["ca_main", "shared"] + assert handlers.server_account_pool("EX", "jp") == ["shared"] + assert handlers.server_account_pool("SINGLE", None) == [None] + assert handlers.server_account_pool("OTHER", "gb") is None + + +def test_round_robin_and_empty_pool(accounts): + assert [handlers.next_server_profile("EX", "gb") for _ in range(3)] == ["eu_box", "shared", "eu_box"] + handlers.config.serve["server_accounts"]["EX"].pop("shared") + with pytest.raises(APIError): + handlers.next_server_profile("EX", "jp") + + +def test_advertised_regions(accounts): + assert handlers.server_account_regions("EX") == {"regions": ["ca", "de", "fr", "gb"], "global": True} + assert handlers.server_account_regions("SINGLE") == {"regions": [], "global": True} + assert handlers.server_account_regions("OTHER") is None + + +def test_falsy_or_malformed_spec_is_off(accounts): + handlers.config.serve["server_accounts"]["EX"] = False + assert handlers.server_account_spec("EX") is None + handlers.config.serve["server_accounts"]["EX"] = "ca" + assert handlers.server_account_spec("EX") is None + assert handlers.server_account_regions("EX") is None + with pytest.raises(ValueError): + handlers.validate_server_accounts() + + +def test_validate_rejects_unknown_profile(accounts, tmp_path, monkeypatch): + monkeypatch.setattr(handlers.config.directories, "cookies", tmp_path) + handlers.config.serve["server_accounts"]["EX"]["typo"] = "ca" + with pytest.raises(ValueError, match="typo"): + handlers.validate_server_accounts() + (tmp_path / "EX").mkdir() + (tmp_path / "EX" / "typo.txt").write_text("# Netscape HTTP Cookie File\n") + handlers.validate_server_accounts() + + +def test_single_account_always_same(accounts): + handlers.config.serve["server_accounts"]["EX"].pop("shared") + assert [handlers.next_server_profile("EX", "ca") for _ in range(3)] == ["ca_main"] * 3 + assert handlers.next_server_profile("SINGLE", None) is None + + +def test_client_managed_gets_nothing(monkeypatch, accounts): + monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {}) + monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None) + monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: object())) + monkeypatch.setattr(handlers, "instantiate_service", lambda *a: "svc") + svc, cookies, cred = handlers.create_service_instance("OTHER", "t", {}, None, [], None) + assert (svc, cookies, cred) == ("svc", None, None) + + +def test_per_key_gate(accounts, monkeypatch): + class Req: + def __init__(self, key): + self.headers = {"X-Secret-Key": key} + + monkeypatch.setattr(handlers, "request_secret_key", lambda r: r.headers["X-Secret-Key"]) + handlers.config.serve["users"] = {"k1": {}, "k2": {"server_accounts": True}, "k3": {"server_accounts": ["EX"]}} + assert handlers.server_accounts_allowed(Req("k1"), "EX") is False + assert handlers.server_accounts_allowed(Req("k2"), "EX") is True + assert handlers.server_accounts_allowed(Req("k3"), "EX") is True + assert handlers.server_accounts_allowed(Req("k3"), "SINGLE") is False + assert handlers.server_accounts_allowed(Req("admin"), "EX") is True + assert handlers.server_account_for(Req("k1"), "EX") is False + assert handlers.server_account_for(Req("k2"), "OTHER") is False + + +def test_rotation_per_region_pool(accounts): + picks = [handlers.next_server_profile("EX", r) for r in ["ca", "gb", "ca", "gb"]] + assert picks == ["ca_main", "eu_box", "shared", "shared"] + + +def test_server_account_ignores_client_credentials(monkeypatch, accounts): + monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {}) + monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None) + monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: object())) + monkeypatch.setattr(handlers, "instantiate_service", lambda *a: "svc") + monkeypatch.setattr(handlers, "server_account_cookies", lambda s, p: "server-jar") + data = {"credentials": {"username": "old", "password": "client"}, "cookies": "AAAA", "cache": {"x": "y"}} + svc, cookies, cred = handlers.create_service_instance("EX", "t", data, None, [], "ca_main", server_account=True) + assert cookies == "server-jar" + assert cred.username == "a" and cred.password == "b" + + +def test_validate_rejects_bad_region_values(accounts): + handlers.config.serve["server_accounts"]["EX"]["ca_main"] = ["gb", False] + with pytest.raises(ValueError, match="quote"): + handlers.validate_server_accounts() + handlers.config.serve["server_accounts"]["EX"]["ca_main"] = "canada" + with pytest.raises(ValueError): + handlers.validate_server_accounts() diff --git a/unshackle/commands/dl.py b/unshackle/commands/dl.py index 9c38660d..4e8d0784 100644 --- a/unshackle/commands/dl.py +++ b/unshackle/commands/dl.py @@ -4654,23 +4654,25 @@ class dl: cookie_file = dl.get_cookie_path(service, profile) if cookie_file: - cookie_jar = MozillaCookieJar(cookie_file) - cookie_data = html.unescape(cookie_file.read_text("utf8")).splitlines(keepends=False) - for i, line in enumerate(cookie_data): - if line and not line.startswith("#"): - line_data = line.lstrip().split("\t") - # Disable client-side expiry checks completely across everywhere - # Even though the cookies are loaded under ignore_expires=True, stuff - # like python-requests may not use them if they are expired - line_data[4] = "" - cookie_data[i] = "\t".join(line_data) - cookie_data = "\n".join(cookie_data) - cookie_file.write_text(cookie_data, "utf8") - cookie_jar.load(ignore_discard=True, ignore_expires=True) - return cookie_jar + return dl.load_cookie_file(cookie_file) return None + @staticmethod + def load_cookie_file(cookie_file: Path) -> MozillaCookieJar: + """Load a Netscape cookie file with every expiry blanked, so no layer drops an expired cookie.""" + cookie_jar = MozillaCookieJar(cookie_file) + cookie_data = html.unescape(cookie_file.read_text("utf8")).splitlines(keepends=False) + for i, line in enumerate(cookie_data): + if line and not line.startswith("#"): + line_data = line.lstrip().split("\t") + # Even under ignore_expires=True, requests may still skip an expired cookie + line_data[4] = "" + cookie_data[i] = "\t".join(line_data) + cookie_file.write_text("\n".join(cookie_data), "utf8") + cookie_jar.load(ignore_discard=True, ignore_expires=True) + return cookie_jar + @staticmethod def save_cookies(path: Path, cookies: CookieJar): if hasattr(cookies, "jar"): diff --git a/unshackle/commands/serve.py b/unshackle/commands/serve.py index 19805173..a3a2971b 100644 --- a/unshackle/commands/serve.py +++ b/unshackle/commands/serve.py @@ -17,6 +17,8 @@ from unshackle.core.api.handlers import ( dashboard_authentication, dashboard_key, request_secret_key, + server_account_regions, + validate_server_accounts, ) from unshackle.core.api.stats import ring, stats, stats_middleware from unshackle.core.config import config @@ -253,6 +255,14 @@ def serve( global_services = config.serve.get("services") if global_services: log.info(f"Global service allowlist: {', '.join(global_services)}") + try: + server_accounts = validate_server_accounts() + except ValueError as e: + raise click.ClickException(str(e)) + for tag in server_accounts: + regions = server_account_regions(tag) or {} + covered = list(regions.get("regions") or []) + (["global"] if regions.get("global") else []) + log.info(f"Server accounts for {tag}: {', '.join(covered)}") users = config.serve.get("users", {}) if isinstance(users, dict): # yaml keys can parse as int; hmac.compare_digest and allowlist lookups need str diff --git a/unshackle/core/api/handlers.py b/unshackle/core/api/handlers.py index 39a97b14..f3e5ef28 100644 --- a/unshackle/core/api/handlers.py +++ b/unshackle/core/api/handlers.py @@ -16,6 +16,7 @@ from unshackle.core.api.errors import APIError, APIErrorCode, handle_api_excepti from unshackle.core.api.input_bridge import AuthStatus, InputBridge from unshackle.core.api.sanitize import safe_cache_key, sanitize_log from unshackle.core.api.session_store import SessionStore +from unshackle.core.cacher import Cacher from unshackle.core.config import config from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_CODEC_MAP from unshackle.core.providers.anilist import parse_anilist_ref @@ -241,12 +242,20 @@ def setup_list_service( no_proxy = data.get("no_proxy", False) proxy_param, proxy_providers = resolve_handler_proxy(data, normalized_service, request) + account = resolve_server_account(request, normalized_service, data.get("client_region")) + if account: + profile = None if account == "default" else account + cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type")) parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config) service_module = Services.load(normalized_service) service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS) - cookies = dl.get_cookie_jar(normalized_service, profile) + if account: + service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}") + cookies = server_account_cookies(normalized_service, profile) + else: + cookies = dl.get_cookie_jar(normalized_service, profile) credential = dl.get_credentials(normalized_service, profile) service_instance.authenticate(cookies, credential) return service_instance @@ -307,6 +316,159 @@ def server_cdm_allowed(request: Optional[web.Request] = None, service: Optional[ return bool(allowed) +_account_counters: Dict[str, int] = {} + + +def validate_server_accounts() -> Dict[str, Any]: + """Check ``serve.server_accounts`` at startup; raise ValueError on a bad shape or unknown profile.""" + accounts = config.serve.get("server_accounts") or {} + if not isinstance(accounts, dict): + raise ValueError("serve.server_accounts must be a mapping of service tag to accounts") + for tag, spec in accounts.items(): + if spec is True: + continue + if not isinstance(spec, dict) or not spec: + raise ValueError(f"serve.server_accounts.{tag}: expected true or a mapping of profile to regions") + creds = config.credentials.get(Services.get_tag(str(tag))) + for profile, regions in spec.items(): + members = [regions] if isinstance(regions, str) else regions if isinstance(regions, list) else [] + if not members or not all( + isinstance(m, str) and (m.lower() == "global" or re.fullmatch(r"[A-Za-z]{2}", m)) for m in members + ): + raise ValueError( + f"serve.server_accounts.{tag}.{profile}: expected a two-letter country code, a list of them, " + "or global (quote codes like 'no')" + ) + cookie_file = config.directories.cookies / Services.get_tag(str(tag)) / f"{profile}.txt" + if not (isinstance(creds, dict) and profile in creds) and not cookie_file.exists(): + raise ValueError( + f"serve.server_accounts.{tag}.{profile}: no such profile under credentials.{tag} " + f"and no cookie file at {cookie_file}" + ) + return accounts + + +def server_account_spec(service: str) -> Any: + """The ``serve.server_accounts`` entry for ``service`` (``True`` or a profile->regions map), else None.""" + accounts = config.serve.get("server_accounts") or {} + tag = Services.get_tag(service).upper() + spec = next((v for k, v in accounts.items() if Services.get_tag(str(k)).upper() == tag), None) + return spec if spec is True or (isinstance(spec, dict) and spec) else None + + +def _regions(value: Any) -> set[str]: + values = [value] if isinstance(value, str) else list(value) if isinstance(value, (list, tuple)) else [] + return {str(v).lower() for v in values} + + +def server_account_cookies(service: str, profile: Optional[str]) -> Any: + """Cookies for a server account: the profile's own file only, so a bare ``cookies/{service}.txt`` cannot shadow it.""" + from unshackle.commands.dl import dl + + if profile is None: + return dl.get_cookie_jar(service, None) + path = config.directories.cookies / service / f"{profile}.txt" + return dl.load_cookie_file(path) if path.exists() else None + + +def server_account_regions(service: str) -> Optional[Dict[str, Any]]: + """What ``GET /api/services`` advertises: the union of regions the server's accounts cover.""" + spec = server_account_spec(service) + if not spec: + return None + if spec is True: + return {"regions": [], "global": True} + union: set[str] = set() + for value in spec.values(): + union |= _regions(value) + return {"regions": sorted(union - {"global"}), "global": "global" in union} + + +def server_region() -> Optional[str]: + """The server's own country code, from the cached IP lookup; None when unknown.""" + try: + from unshackle.core.utils.ip_info import get_ip_info + + info = get_ip_info(None, cached=True) + return (info.get("country") or "").lower() or None if info else None + except Exception: + return None + + +def server_account_pool(service: str, region: Optional[str]) -> Optional[list[Optional[str]]]: + """Profiles of the server's own accounts usable in ``region``; None when the service is client-managed.""" + spec = server_account_spec(service) + if not spec: + return None + creds = config.credentials.get(service) + if spec is True: + return sorted(creds) if isinstance(creds, dict) else [None] + region = (region or "").lower() + return sorted(p for p, v in spec.items() if {"global", region} & _regions(v)) + + +def next_server_profile(service: str, region: Optional[str]) -> Optional[str]: + """Round-robin the next server account profile for ``region``.""" + pool = server_account_pool(service, region) or [] + if not pool: + covered = server_account_regions(service) or {} + regions = ", ".join(covered.get("regions") or []) or "none" + raise APIError( + APIErrorCode.INVALID_INPUT, + f"No server account for {service} in region '{region or 'unknown'}'. " + f"Its accounts cover: {regions}. Pass --proxy with one of those country codes.", + details={"service": service, "region": region, "regions": covered.get("regions") or []}, + ) + key = f"{service}:{','.join(p or '' for p in pool)}" + n = _account_counters.get(key, 0) + _account_counters[key] = n + 1 + return pool[n % len(pool)] + + +def server_accounts_allowed(request: Optional[web.Request] = None, service: Optional[str] = None) -> bool: + """Whether the calling API key may authenticate ``service`` with one of the server's own accounts. + + Same shape as ``server_cdm``: ``server_accounts: true`` on the ``serve.users`` entry for every + service, or a list of service tags. Off unless set. The admin secret (no ``users`` entry) keeps + full access. Combine with ``server_account_spec``: the server treats an API key without the + opt-in as a client-managed remote session for that service. + """ + if not request: + return True + user_config = config.serve.get("users", {}).get(request_secret_key(request)) + if user_config is None: + return True + allowed = user_config.get("server_accounts", False) + if isinstance(allowed, str): + allowed = [allowed] + if isinstance(allowed, (list, tuple, set)): + return service is not None and Services.get_tag(service).upper() in { + Services.get_tag(s).upper() for s in allowed + } + return bool(allowed) + + +def resolve_server_account(request: Optional[web.Request], service: str, region: Any) -> Optional[str]: + """Profile of the server account a search or list call runs on; None when the service is client-managed. + + Raises FORBIDDEN for an API key without the ``server_accounts`` opt-in. + """ + if server_account_spec(service) is None: + return None + if not server_accounts_allowed(request, service): + raise APIError( + APIErrorCode.FORBIDDEN, + f"Your API key may not use the server's accounts for {service}.", + details={"service": service}, + ) + return next_server_profile(service, region if isinstance(region, str) else None) or "default" + + +def server_account_for(request: Optional[web.Request], service: str) -> bool: + """True when this request's remote session for ``service`` should run on a server account.""" + return server_account_spec(service) is not None and server_accounts_allowed(request, service) + + def server_proxy_allowed(request: Optional[web.Request] = None) -> bool: """Whether the calling API key may use the server's own configured proxy providers. @@ -831,6 +993,10 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = proxy_param, proxy_providers = await asyncio.to_thread(resolve_handler_proxy, data, normalized_service, request) + account = resolve_server_account(request, normalized_service, data.get("client_region")) + if account: + profile = None if account == "default" else account + cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type")) parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config) service_module = Services.load(normalized_service) @@ -844,7 +1010,11 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] = details={"service": normalized_service}, ) - cookies = dl.get_cookie_jar(normalized_service, profile) + if account: + service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}") + cookies = server_account_cookies(normalized_service, profile) + else: + cookies = dl.get_cookie_jar(normalized_service, profile) credential = dl.get_credentials(normalized_service, profile) service_instance.authenticate(cookies, credential) @@ -1846,6 +2016,12 @@ async def server_config_handler(request: Optional[web.Request] = None) -> web.Re "remote_only": bool(serve_cfg.get("remote_only", False)), "cdm_overrides": redact_config(serve_cfg.get("cdm_overrides")), "allow_job_credentials": bool(serve_cfg.get("allow_job_credentials", False)), + "server_accounts": { + tag: server_account_regions(tag) + for tag in (serve_cfg.get("server_accounts") or {}) + if server_accounts_allowed(request, tag) + } + or None, }, "directories": { "downloads": str(config.directories.downloads), @@ -2046,6 +2222,7 @@ SESSION_TRANSPORT_KEYS = { "cookies", "cache", "client_region", + "proxy_region", "cdm_type", "range_", "vcodec", @@ -2061,11 +2238,14 @@ def create_service_instance( proxy_param: Optional[str], proxy_providers: list, profile: Optional[str], + server_account: bool = False, ) -> Any: - """Make and authenticate a service instance. + """Make a service instance and resolve its credentials and cookies. - Supports client-sent credentials/cookies (for remote-dl) with fallback - to server-local config (for backward compatibility). + With ``server_account`` the service takes the server's own credential and cookie file for + ``profile`` and drops anything the client sent. Otherwise only client-sent data counts: a + client that sends nothing authenticates with nothing, because the server never lends its + own accounts. """ from unshackle.commands.dl import dl from unshackle.core.credential import Credential @@ -2118,7 +2298,14 @@ def create_service_instance( service_module = Services.load(normalized_service) service_instance = instantiate_service(parent_ctx, service_module, title_id, data, SESSION_TRANSPORT_KEYS) - # Resolve credentials: client-sent > server-local + if server_account: + return ( + service_instance, + server_account_cookies(normalized_service, profile), + dl.get_credentials(normalized_service, profile), + ) + + credential = None cred_data = data.get("credentials") if cred_data and isinstance(cred_data, dict): credential = Credential( @@ -2126,10 +2313,8 @@ def create_service_instance( password=cred_data["password"], extra=cred_data.get("extra"), ) - else: - credential = dl.get_credentials(normalized_service, profile) - # Resolve cookies: client-sent > server-local + cookies = None cookie_text = data.get("cookies") if cookie_text and isinstance(cookie_text, str): import base64 @@ -2149,8 +2334,6 @@ def create_service_instance( with suppress(OSError): os.unlink(tmp_path) - else: - cookies = dl.get_cookie_jar(normalized_service, profile) return service_instance, cookies, credential @@ -2187,13 +2370,23 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req import hashlib import uuid as uuid_mod - from unshackle.core.cacher import Cacher from unshackle.core.config import config as app_config session_id = str(uuid_mod.uuid4()) api_key = request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous" api_key_hash = hashlib.pbkdf2_hmac("sha256", api_key.encode(), b"unshackle-session-ns", 100_000).hex()[:12] - session_cache_tag = f"_sessions/{api_key_hash}/{session_id}/{normalized_service}" + session_cache_dir = f"_sessions/{api_key_hash}/{session_id}/{normalized_service}" + session_cache_tag: Optional[str] = session_cache_dir + + server_account = server_account_for(request, normalized_service) + if server_account: + region = data.get("proxy_region") or data.get("client_region") + if region is not None and not (isinstance(region, str) and re.fullmatch(r"[A-Za-z]{2}", region)): + raise APIError(APIErrorCode.INVALID_INPUT, "proxy_region must be a two-letter country code.") + if region is None and data.get("no_proxy"): + region = server_region() + profile = next_server_profile(normalized_service, region) + log.info(f"Using server account '{sanitize_log(profile or 'default')}' for {normalized_service}") service_instance, cookies, credential = create_service_instance( normalized_service, @@ -2202,12 +2395,17 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req proxy_param, proxy_providers, profile, + server_account=server_account, ) - service_instance.cache = Cacher(session_cache_tag) + if server_account: + session_cache_tag = None + service_instance.cache = Cacher(f"_accounts/{normalized_service}/{profile or 'default'}") + else: + service_instance.cache = Cacher(session_cache_dir) cache_data = data.get("cache", {}) - if cache_data: + if cache_data and session_cache_tag: import base64 cache_dir = app_config.directories.cache / session_cache_tag @@ -2230,6 +2428,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req session_id=session_id, owner_key=api_key, creator_ip=request.remote if request else None, + server_account=(profile or "default") if server_account else None, ) session.cache_tag = session_cache_tag # Echoed to every dashboard viewer, so cap what an arbitrary client can push into it. @@ -2256,6 +2455,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req "session_id": session.session_id, "service": normalized_service, "status": "authenticating", + "server_account": server_account, } ) diff --git a/unshackle/core/api/routes.py b/unshackle/core/api/routes.py index 065d43d0..5ed2d42e 100644 --- a/unshackle/core/api/routes.py +++ b/unshackle/core/api/routes.py @@ -40,6 +40,8 @@ from unshackle.core.api.handlers import ( refresh_services_handler, retry_download_job_handler, search_handler, + server_account_regions, + server_accounts_allowed, server_config_handler, session_create_handler, session_delete_handler, @@ -174,6 +176,16 @@ async def services(request: web.Request) -> web.Response: pending_update: type: boolean description: Present (true) while a repo refresh waits for this service's jobs to finish + server_accounts: + type: object + description: Present when the server lends its own accounts for this service (serve.server_accounts) + properties: + regions: + type: array + items: + type: string + global: + type: boolean aliases: type: array items: @@ -240,6 +252,8 @@ async def services(request: web.Request) -> web.Response: } if tag in services_module.PENDING: service_data["pending_update"] = True + if server_accounts_allowed(request, tag) and (server_accounts := server_account_regions(tag)) is not None: + service_data["server_accounts"] = server_accounts try: service_module = Services.load(tag) @@ -1532,9 +1546,29 @@ async def session_create(request: web.Request) -> web.Response: cache: type: object additionalProperties: true + client_region: + type: string + description: Two-letter country the client sits in + proxy_region: + type: string + description: Two-letter country the client resolved its proxy for; picks a server account responses: '200': - description: Remote session created with titles, tracks, and chapters + description: Remote session created; authentication continues in the background + content: + application/json: + schema: + type: object + properties: + session_id: + type: string + service: + type: string + status: + type: string + server_account: + type: boolean + description: True when the server authenticated with one of its own accounts '400': description: Invalid request '401': diff --git a/unshackle/core/api/session_store.py b/unshackle/core/api/session_store.py index 1962d34b..a9794fc6 100644 --- a/unshackle/core/api/session_store.py +++ b/unshackle/core/api/session_store.py @@ -40,6 +40,7 @@ class SessionEntry: creator_ip: Optional[str] = None owner_key: Optional[str] = None # X-Secret-Key that owns this session cache_tag: Optional[str] = None + server_account: Optional[str] = None # profile name when the server lent its own account input_bridge: Optional[InputBridge] = None auth_status: AuthStatus = AuthStatus.AUTHENTICATED auth_error: Optional[str] = None @@ -68,6 +69,7 @@ class SessionEntry: "tracks": len(self.tracks), "auth_status": self.auth_status.value, "auth_error": self.auth_error, + "server_account": self.server_account, "client": self.client, "actions": list(self.actions), "created_at": self.created_at.isoformat(), @@ -111,6 +113,7 @@ class SessionStore: session_id: Optional[str] = None, owner_key: Optional[str] = None, creator_ip: Optional[str] = None, + server_account: Optional[str] = None, ) -> SessionEntry: """Make a new remote session with an authenticated service instance.""" async with self._lock: @@ -126,6 +129,7 @@ class SessionStore: service_instance=service_instance, owner_key=owner_key, creator_ip=creator_ip, + server_account=server_account, ) self._sessions[session_id] = entry _publish("create", entry) diff --git a/unshackle/core/remote_service.py b/unshackle/core/remote_service.py index eae385d9..f10286cd 100644 --- a/unshackle/core/remote_service.py +++ b/unshackle/core/remote_service.py @@ -661,6 +661,7 @@ class RemoteService: svc_config = services_config.get(service_tag, {}) self._server_cdm: bool | None = services_config.get("_server_cdm") + self._server_accounts: Optional[dict] = services_config.get("_server_accounts") self.apply_service_config(svc_config) def apply_service_config(self, svc_config: dict) -> None: @@ -707,29 +708,57 @@ class RemoteService: create_data: Dict[str, Any] = {"service": self.service_tag, "title_id": self.title_id} - credentials = load_credentials_for_transport(self.service_tag, profile) - if credentials: - create_data["credentials"] = credentials + if self._server_accounts is None: + credentials = load_credentials_for_transport(self.service_tag, profile) + if credentials: + create_data["credentials"] = credentials - cookies_text = load_cookies_for_transport(self.service_tag, profile) - if cookies_text: - create_data["cookies"] = cookies_text + cookies_text = load_cookies_for_transport(self.service_tag, profile) + if cookies_text: + create_data["cookies"] = cookies_text + else: + self.credential = None + self.log.info( + "The server manages the account for this service" + (" and ignores --profile" if profile else "") + ) - if not no_proxy and proxy: - resolved_proxy = resolve_proxy_arg(proxy) - if resolved_proxy: - create_data["proxy"] = resolved_proxy - - if not no_proxy and not proxy: + client_region = None + if not no_proxy: try: from unshackle.core.utils.ip_info import get_ip_info ip_info = get_ip_info(self._session, cached=True) if ip_info and ip_info.get("country"): - create_data["client_region"] = ip_info["country"].lower() + client_region = ip_info["country"].lower() except Exception: pass + if not no_proxy and not proxy and self._server_accounts and not self._server_accounts.get("global"): + regions = [str(r).lower() for r in self._server_accounts.get("regions") or []] + if regions and client_region and client_region not in regions: + try: + create_data["proxy"] = resolve_proxy_arg(regions[0]) + create_data["proxy_region"] = regions[0] + self.log.info(f"Using a '{regions[0]}' proxy to match a server account ({', '.join(regions)})") + except click.ClickException: + self.log.warning( + f"The server's accounts work in {', '.join(regions)} and you are in {client_region}. " + "There is no local proxy for that region. Pass --proxy yourself if the server " + "rejects the remote session" + ) + elif not no_proxy and proxy: + resolved_proxy = resolve_proxy_arg(proxy) + if resolved_proxy: + create_data["proxy"] = resolved_proxy + query = (self.ctx.parent.params.get("proxy_query") if self.ctx.parent else None) or proxy + country = re.match(r"^(?:[a-z]+:)?([a-z]{2})\d*$", query, re.IGNORECASE) + if country: + create_data["proxy_region"] = country.group(1).lower() + if not no_proxy and not proxy and client_region: + create_data["client_region"] = client_region + elif not no_proxy and not proxy and self._server_accounts and not self._server_accounts.get("global"): + self.log.warning("Could not detect your region; pass --proxy so the server can pick an account") + if profile: create_data["profile"] = profile if no_proxy: @@ -757,7 +786,7 @@ class RemoteService: create_data["cdm_type"] = "playready" if is_playready_cdm(cdm) else "widevine" - cache_data = self.load_cache_files() + cache_data = self.load_cache_files() if self._server_accounts is None else None if cache_data: create_data["cache"] = cache_data diff --git a/unshackle/core/services.py b/unshackle/core/services.py index db459947..94b5665a 100644 --- a/unshackle/core/services.py +++ b/unshackle/core/services.py @@ -396,6 +396,7 @@ class Services(click.Group): server_name = ctx.parent.params.get("server") if ctx.parent else None server_url, api_key, services_config = resolve_server(server_name) + services_config["_server_accounts"] = svc_info.get("server_accounts") if svc_info else None service_params = {k: v for k, v in kwargs.items() if v is not None and v is not False} return RemoteService(ctx, tag, title, server_url, api_key, services_config, service_params=service_params)