feat(serve)!: add server-held accounts for remote sessions

serve.server_accounts lends region-tagged server credentials to remote sessions (round-robin per region, per-key opt-in, per-account cache). Remote sessions no longer fall back to the server's local credentials when the client sends none.
This commit is contained in:
imSp4rky
2026-08-28 16:36:39 -06:00
parent e3f4e52444
commit d9b21229c3
12 changed files with 517 additions and 48 deletions
+1 -1
View File
@@ -44,7 +44,7 @@ All are `GET`.
| Path | Returns |
|------|---------|
| `/api/dashboard/status` | Version, `code_hash`, bind host and port, `mode` (`full`, `api_only`, `remote_only`), uptime, `requests_total`, `requests_rejected` (401s), `requests_by_key` (label → count), loaded service count, session count, `max_sessions`, `session_ttl`, job counts by status. |
| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the first resolved title's display name), `auth_status`, `auth_error`, `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`. `client` is whatever the remote client sent as `client` in its session create request (the current CLI sends nothing, so it is `{}`). `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. |
| `/api/dashboard/sessions` | Every live remote session: `id`, `owner` (username or masked key), `creator_ip`, `service`, `title_id`, `titles`, `tracks`, `title` (the first resolved title's display name), `auth_status`, `auth_error`, `server_account` (the server profile lent to the remote session, else `null`), `client`, `actions`, `created_at` and `last_accessed` (ISO 8601) with `created_ts` and `last_accessed_ts` (Unix epoch, the same clock as log `ts`), `age_seconds`, `idle_seconds`. `client` is whatever the remote client sent as `client` in its session create request (the current CLI sends nothing, so it is `{}`). `actions` is the session's request log, newest last, capped at 500: `ts`, `method`, `action` (`titles`, `tracks`, `segments`, `license`, `prompt`, …), `query`, `status`, `ms`, `bytes_in`, `bytes_out`. |
| `/api/dashboard/jobs` | Every download job with full detail, regardless of owner. Empty in `--remote-only` mode. |
| `/api/dashboard/logs` | `{"seq": N, "records": [...]}`: the last 1000 log records. `?since=<seq>` returns only newer records; `?level=WARNING` sets the minimum level; `?logger=serve` keeps one logger and its children (`aiohttp.access` is the noisiest). `ts` is a Unix epoch in seconds. |
| `/api/dashboard/events` | Server-Sent Events stream. With `?since=<seq>` it returns `{"seq", "stats", "events": [...]}` instead: every event after `seq` (the server keeps the last 20000, log records included, so a poll every few seconds is safe even under a busy access log) as one JSON burst, so a UI that polls every few seconds still sees a remote session that opened and closed in between. |
+6 -2
View File
@@ -120,6 +120,7 @@ Field notes:
| `has_search` | Whether the service supports `POST /api/search`. |
| `has_drm` | Whether the service overrides a Widevine or PlayReady license hook. |
| `pending_update` | Set to `true` while a service repository refresh waits for this service's jobs to finish before it swaps in the new code. Absent otherwise. |
| `server_accounts` | Present when the server lends its own accounts for this service: `{regions: ["ca", "gb"], global: bool}`, the regions those accounts cover. A client sends no credentials, cookies, or cache for such a service. |
| `auth_methods` | Declared or inferred auth methods, e.g. `["cookies"]`, `["credentials"]`. |
| Status | Meaning |
@@ -833,7 +834,7 @@ DELETE /api/session/{id} → tear down, harvest updated cache
### `POST /api/session/create`
Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files), `client_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, and any service CLI options.
Make a remote session for a service and title. Authentication runs asynchronously in the background. This call returns immediately with `status: "authenticating"`, and you then poll the prompt endpoint. The body accepts `service` and `title_id` (both required). It also accepts a broad set of optional keys, because the body allows `additionalProperties`. These are `credentials` (`{username, password, extra?}`), `cookies` (base64 of zlib-compressed Netscape cookie file), `proxy`, `no_proxy`, `profile`, `cache` (map of forwarded cache files), `client_region`, `proxy_region`, `cdm_type`, `range_`, `vcodec`, `quality`, `best_available`, and any service CLI options. `proxy_region` is the country code the client resolved `proxy` from; the server matches it against its own accounts.
The `proxy` value must be a full proxy URI, unless the operator gives your API key `server_proxy` in the `serve.users` config. Without it, the server does not resolve country codes with its own proxy providers. It rejects the request with `INVALID_PROXY` when no `proxy` is set and the reported `client_region` differs from the server's region. A request that reports no `client_region` is not blocked. Pass `proxy` with your own proxy, or `no_proxy` to accept the server's own connection. With `server_proxy`, the server resolves a country code and picks a proxy for your `client_region` itself.
@@ -843,10 +844,13 @@ The `proxy` value must be a full proxy URI, unless the operator gives your API k
{
"session_id": "d4e5f6a7-...",
"service": "EXAMPLE",
"status": "authenticating"
"status": "authenticating",
"server_account": false
}
```
`server_account` is `true` when the server authenticated with one of its own accounts (`serve.server_accounts`). Otherwise the server uses only what the client sent, and never falls back to its own credentials.
| Status | Error code | Meaning |
| --- | --- | --- |
| `200` | - | Session created; authentication in progress. |
+15 -1
View File
@@ -126,6 +126,18 @@ hood the client walks a remote session through its lifecycle.
the server fetches the right manifests
- Your local per-service **cache files** (e.g. refreshed tokens)
The server uses its own accounts only when the operator lists the service in
`serve.server_accounts` and gives your API key `server_accounts` in `serve.users`.
For any other service, a client that sends no credentials or cookies authenticates
with nothing. For a listed service the client sends no credentials, cookies, or
cache at all: the server picks one of its own accounts that works in your region and
rotates through them. `--proxy ca` picks a `ca` account.
`GET /api/services` advertises the regions those accounts cover. When your own
region is not one of them and you set no `--proxy`, the client resolves a proxy for
the first advertised region itself, so both sides sit in a region the account
works in.
The server responds immediately with a session ID and a status. Authentication
runs in the background on the server.
@@ -247,7 +259,9 @@ serve:
- **Namespaced, isolated cache.** Each remote session gets its own cache directory,
namespaced by a hash of the API key and the remote session ID, so remote sessions
cannot read each other's cached tokens. The server deletes the directory when the
remote session ends.
remote session ends. A remote session that uses a server account (`serve.server_accounts`)
gets a persistent per-account cache instead, so the account's tokens outlive the
remote session.
- **The server redacts secrets in logs.** It redacts remote session IDs, service tags,
and other user values before it writes them to the log.
+50
View File
@@ -128,6 +128,7 @@ is the [REST API](../../dev/rest-api/index.md) section. These are the config key
| `global_speed_limit` | str | *(unlimited)* | Server-wide download speed cap, e.g. `10M`, `1.5G` or plain bytes/sec (same format as `speed_limit`). One shared budget across all concurrent jobs; per-job speed limits are ignored while it is set. |
| `cdm_overrides` | list or bool | *(unset)* | Allowed per-request CDM overrides: a list of permitted device names, or `true` for any. Unset rejects every override. |
| `allow_job_credentials` | bool | `false` | Permit clients to supply credentials per job. |
| `server_accounts` | dict | `{}` | Services whose remote sessions use the server's own `credentials` and cookie files instead of the client's (see below). |
| `devices` | list | *(auto)* | Widevine devices offered; auto-filled from `directories.wvds`. |
| `playready_devices` | list | *(auto)* | PlayReady devices; auto-filled from `directories.prds`. |
@@ -145,6 +146,54 @@ client that asks anyway gets a `FORBIDDEN` error. Because a download job always
an API key without `server_cdm` for that service also cannot submit or retry `/api/download`
jobs. Keys that have no `users` entry, such as `api_secret`, keep server CDM access.
`server_accounts` lists, per service, the accounts the server lends to remote sessions. A remote
session for a listed service ignores the credentials, cookies, and cache the client sends and
uses one of the server's own profiles from `credentials`. For a service that is not listed the
server never uses its own accounts: a client that sends nothing authenticates with nothing.
```yaml
credentials:
EXAMPLE1:
ca_main: user1@example.com:pw1
eu_box: user2@example.com:pw2
shared: user3@example.com:pw3
serve:
server_accounts:
EXAMPLE1:
ca_main: ca # one country
eu_box: [gb, fr, de] # a list of countries
shared: global # any region
EXAMPLE2: true # every profile (or the single credential), any region
```
Each entry under a service names a profile from `credentials`, or a cookie file
`cookies/<TAG>/<profile>.txt` for a service that has no credentials. Its value is the region or
regions that account works in: a two-letter country code, a list of them, or `global`. Quote
`no` (Norway), because YAML reads the bare word as `false`.
The region of a remote session is the country the client asked for with `--proxy` (`ca`, `ca1`,
`provider:ca`), the server's own region with `--no-proxy`, or the client's own region otherwise.
A full proxy URI carries no country, so with one only a `global` account matches. The server
picks the profiles that cover that region (or are `global`) and rotates through them
round-robin, one account per remote session. With no matching account it rejects the remote
session. These services ignore the client's `-p/--profile`.
Cookies come only from the profile's own file (`cookies/EXAMPLE1/ca_main.txt`; the server does
not read a bare `cookies/EXAMPLE1.txt`), and each account keeps its own persistent cache under
`cache/_accounts/EXAMPLE1/<profile>/`, so refreshed tokens survive the remote session but never
cross accounts. `serve` refuses to start when a listed profile has neither an entry under
`credentials` nor a cookie file.
`/api/search`, `/api/list-titles` and `/api/list-tracks` use the same pool, matched on the
client's reported region, and return `FORBIDDEN` to an API key without the opt-in. Rotation is
per process and does not track which accounts are in use, so concurrent remote sessions can
share one account.
A `users` entry opts in to those accounts with `server_accounts: true` (every listed service)
or a list of service tags. It is off unless set: an API key without it authenticates with the
credentials the client sends, as for any other service, and `/api/services` does not advertise
the server's accounts to it. API keys with no `users` entry, such as `api_secret`, keep access.
`server_proxy` decides whether that API key may use the proxy providers the server has
configured. There is no implicit access: only the boolean `true` on the API key's `users` entry
grants it, and an API key without a `users` entry has no access either. Without the opt-in the
@@ -173,6 +222,7 @@ serve:
c9d0e1f2:
services: [EXAMPLE1, EXAMPLE2]
server_cdm: [EXAMPLE1] # the server licenses only EXAMPLE1; EXAMPLE2 needs a local CDM
server_accounts: [EXAMPLE1] # may authenticate EXAMPLE1 with one of the server's own accounts
server_proxy: true # this key may also use the server's proxy providers
```
+121
View File
@@ -0,0 +1,121 @@
import pytest
from unshackle.core.api import handlers
from unshackle.core.api.errors import APIError
@pytest.fixture
def accounts(monkeypatch):
monkeypatch.setattr(
handlers.config, "credentials", {"EX": {"ca_main": "a:b", "eu_box": "c:d", "shared": "e:f"}, "SINGLE": "u:p"}
)
monkeypatch.setattr(
handlers.config,
"serve",
{
"server_accounts": {
"EX": {"ca_main": "ca", "eu_box": ["gb", "fr", "de"], "shared": "global"},
"SINGLE": True,
}
},
)
monkeypatch.setattr(handlers.Services, "get_tag", staticmethod(lambda s: s))
handlers._account_counters.clear()
def test_pool_filters_by_region(accounts):
assert handlers.server_account_pool("EX", "gb") == ["eu_box", "shared"]
assert handlers.server_account_pool("EX", "CA") == ["ca_main", "shared"]
assert handlers.server_account_pool("EX", "jp") == ["shared"]
assert handlers.server_account_pool("SINGLE", None) == [None]
assert handlers.server_account_pool("OTHER", "gb") is None
def test_round_robin_and_empty_pool(accounts):
assert [handlers.next_server_profile("EX", "gb") for _ in range(3)] == ["eu_box", "shared", "eu_box"]
handlers.config.serve["server_accounts"]["EX"].pop("shared")
with pytest.raises(APIError):
handlers.next_server_profile("EX", "jp")
def test_advertised_regions(accounts):
assert handlers.server_account_regions("EX") == {"regions": ["ca", "de", "fr", "gb"], "global": True}
assert handlers.server_account_regions("SINGLE") == {"regions": [], "global": True}
assert handlers.server_account_regions("OTHER") is None
def test_falsy_or_malformed_spec_is_off(accounts):
handlers.config.serve["server_accounts"]["EX"] = False
assert handlers.server_account_spec("EX") is None
handlers.config.serve["server_accounts"]["EX"] = "ca"
assert handlers.server_account_spec("EX") is None
assert handlers.server_account_regions("EX") is None
with pytest.raises(ValueError):
handlers.validate_server_accounts()
def test_validate_rejects_unknown_profile(accounts, tmp_path, monkeypatch):
monkeypatch.setattr(handlers.config.directories, "cookies", tmp_path)
handlers.config.serve["server_accounts"]["EX"]["typo"] = "ca"
with pytest.raises(ValueError, match="typo"):
handlers.validate_server_accounts()
(tmp_path / "EX").mkdir()
(tmp_path / "EX" / "typo.txt").write_text("# Netscape HTTP Cookie File\n")
handlers.validate_server_accounts()
def test_single_account_always_same(accounts):
handlers.config.serve["server_accounts"]["EX"].pop("shared")
assert [handlers.next_server_profile("EX", "ca") for _ in range(3)] == ["ca_main"] * 3
assert handlers.next_server_profile("SINGLE", None) is None
def test_client_managed_gets_nothing(monkeypatch, accounts):
monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {})
monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None)
monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: object()))
monkeypatch.setattr(handlers, "instantiate_service", lambda *a: "svc")
svc, cookies, cred = handlers.create_service_instance("OTHER", "t", {}, None, [], None)
assert (svc, cookies, cred) == ("svc", None, None)
def test_per_key_gate(accounts, monkeypatch):
class Req:
def __init__(self, key):
self.headers = {"X-Secret-Key": key}
monkeypatch.setattr(handlers, "request_secret_key", lambda r: r.headers["X-Secret-Key"])
handlers.config.serve["users"] = {"k1": {}, "k2": {"server_accounts": True}, "k3": {"server_accounts": ["EX"]}}
assert handlers.server_accounts_allowed(Req("k1"), "EX") is False
assert handlers.server_accounts_allowed(Req("k2"), "EX") is True
assert handlers.server_accounts_allowed(Req("k3"), "EX") is True
assert handlers.server_accounts_allowed(Req("k3"), "SINGLE") is False
assert handlers.server_accounts_allowed(Req("admin"), "EX") is True
assert handlers.server_account_for(Req("k1"), "EX") is False
assert handlers.server_account_for(Req("k2"), "OTHER") is False
def test_rotation_per_region_pool(accounts):
picks = [handlers.next_server_profile("EX", r) for r in ["ca", "gb", "ca", "gb"]]
assert picks == ["ca_main", "eu_box", "shared", "shared"]
def test_server_account_ignores_client_credentials(monkeypatch, accounts):
monkeypatch.setattr(handlers, "load_service_yaml", lambda s: {})
monkeypatch.setattr(handlers, "load_full_cdm", lambda *a: None)
monkeypatch.setattr(handlers.Services, "load", staticmethod(lambda s: object()))
monkeypatch.setattr(handlers, "instantiate_service", lambda *a: "svc")
monkeypatch.setattr(handlers, "server_account_cookies", lambda s, p: "server-jar")
data = {"credentials": {"username": "old", "password": "client"}, "cookies": "AAAA", "cache": {"x": "y"}}
svc, cookies, cred = handlers.create_service_instance("EX", "t", data, None, [], "ca_main", server_account=True)
assert cookies == "server-jar"
assert cred.username == "a" and cred.password == "b"
def test_validate_rejects_bad_region_values(accounts):
handlers.config.serve["server_accounts"]["EX"]["ca_main"] = ["gb", False]
with pytest.raises(ValueError, match="quote"):
handlers.validate_server_accounts()
handlers.config.serve["server_accounts"]["EX"]["ca_main"] = "canada"
with pytest.raises(ValueError):
handlers.validate_server_accounts()
+16 -14
View File
@@ -4654,23 +4654,25 @@ class dl:
cookie_file = dl.get_cookie_path(service, profile)
if cookie_file:
cookie_jar = MozillaCookieJar(cookie_file)
cookie_data = html.unescape(cookie_file.read_text("utf8")).splitlines(keepends=False)
for i, line in enumerate(cookie_data):
if line and not line.startswith("#"):
line_data = line.lstrip().split("\t")
# Disable client-side expiry checks completely across everywhere
# Even though the cookies are loaded under ignore_expires=True, stuff
# like python-requests may not use them if they are expired
line_data[4] = ""
cookie_data[i] = "\t".join(line_data)
cookie_data = "\n".join(cookie_data)
cookie_file.write_text(cookie_data, "utf8")
cookie_jar.load(ignore_discard=True, ignore_expires=True)
return cookie_jar
return dl.load_cookie_file(cookie_file)
return None
@staticmethod
def load_cookie_file(cookie_file: Path) -> MozillaCookieJar:
"""Load a Netscape cookie file with every expiry blanked, so no layer drops an expired cookie."""
cookie_jar = MozillaCookieJar(cookie_file)
cookie_data = html.unescape(cookie_file.read_text("utf8")).splitlines(keepends=False)
for i, line in enumerate(cookie_data):
if line and not line.startswith("#"):
line_data = line.lstrip().split("\t")
# Even under ignore_expires=True, requests may still skip an expired cookie
line_data[4] = ""
cookie_data[i] = "\t".join(line_data)
cookie_file.write_text("\n".join(cookie_data), "utf8")
cookie_jar.load(ignore_discard=True, ignore_expires=True)
return cookie_jar
@staticmethod
def save_cookies(path: Path, cookies: CookieJar):
if hasattr(cookies, "jar"):
+10
View File
@@ -17,6 +17,8 @@ from unshackle.core.api.handlers import (
dashboard_authentication,
dashboard_key,
request_secret_key,
server_account_regions,
validate_server_accounts,
)
from unshackle.core.api.stats import ring, stats, stats_middleware
from unshackle.core.config import config
@@ -253,6 +255,14 @@ def serve(
global_services = config.serve.get("services")
if global_services:
log.info(f"Global service allowlist: {', '.join(global_services)}")
try:
server_accounts = validate_server_accounts()
except ValueError as e:
raise click.ClickException(str(e))
for tag in server_accounts:
regions = server_account_regions(tag) or {}
covered = list(regions.get("regions") or []) + (["global"] if regions.get("global") else [])
log.info(f"Server accounts for {tag}: {', '.join(covered)}")
users = config.serve.get("users", {})
if isinstance(users, dict):
# yaml keys can parse as int; hmac.compare_digest and allowlist lookups need str
+215 -15
View File
@@ -16,6 +16,7 @@ from unshackle.core.api.errors import APIError, APIErrorCode, handle_api_excepti
from unshackle.core.api.input_bridge import AuthStatus, InputBridge
from unshackle.core.api.sanitize import safe_cache_key, sanitize_log
from unshackle.core.api.session_store import SessionStore
from unshackle.core.cacher import Cacher
from unshackle.core.config import config
from unshackle.core.constants import AUDIO_CODEC_MAP, DYNAMIC_RANGE_MAP, VIDEO_CODEC_MAP
from unshackle.core.providers.anilist import parse_anilist_ref
@@ -241,12 +242,20 @@ def setup_list_service(
no_proxy = data.get("no_proxy", False)
proxy_param, proxy_providers = resolve_handler_proxy(data, normalized_service, request)
account = resolve_server_account(request, normalized_service, data.get("client_region"))
if account:
profile = None if account == "default" else account
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
service_module = Services.load(normalized_service)
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, LIST_HANDLER_TRANSPORT_KEYS)
cookies = dl.get_cookie_jar(normalized_service, profile)
if account:
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}")
cookies = server_account_cookies(normalized_service, profile)
else:
cookies = dl.get_cookie_jar(normalized_service, profile)
credential = dl.get_credentials(normalized_service, profile)
service_instance.authenticate(cookies, credential)
return service_instance
@@ -307,6 +316,159 @@ def server_cdm_allowed(request: Optional[web.Request] = None, service: Optional[
return bool(allowed)
_account_counters: Dict[str, int] = {}
def validate_server_accounts() -> Dict[str, Any]:
"""Check ``serve.server_accounts`` at startup; raise ValueError on a bad shape or unknown profile."""
accounts = config.serve.get("server_accounts") or {}
if not isinstance(accounts, dict):
raise ValueError("serve.server_accounts must be a mapping of service tag to accounts")
for tag, spec in accounts.items():
if spec is True:
continue
if not isinstance(spec, dict) or not spec:
raise ValueError(f"serve.server_accounts.{tag}: expected true or a mapping of profile to regions")
creds = config.credentials.get(Services.get_tag(str(tag)))
for profile, regions in spec.items():
members = [regions] if isinstance(regions, str) else regions if isinstance(regions, list) else []
if not members or not all(
isinstance(m, str) and (m.lower() == "global" or re.fullmatch(r"[A-Za-z]{2}", m)) for m in members
):
raise ValueError(
f"serve.server_accounts.{tag}.{profile}: expected a two-letter country code, a list of them, "
"or global (quote codes like 'no')"
)
cookie_file = config.directories.cookies / Services.get_tag(str(tag)) / f"{profile}.txt"
if not (isinstance(creds, dict) and profile in creds) and not cookie_file.exists():
raise ValueError(
f"serve.server_accounts.{tag}.{profile}: no such profile under credentials.{tag} "
f"and no cookie file at {cookie_file}"
)
return accounts
def server_account_spec(service: str) -> Any:
"""The ``serve.server_accounts`` entry for ``service`` (``True`` or a profile->regions map), else None."""
accounts = config.serve.get("server_accounts") or {}
tag = Services.get_tag(service).upper()
spec = next((v for k, v in accounts.items() if Services.get_tag(str(k)).upper() == tag), None)
return spec if spec is True or (isinstance(spec, dict) and spec) else None
def _regions(value: Any) -> set[str]:
values = [value] if isinstance(value, str) else list(value) if isinstance(value, (list, tuple)) else []
return {str(v).lower() for v in values}
def server_account_cookies(service: str, profile: Optional[str]) -> Any:
"""Cookies for a server account: the profile's own file only, so a bare ``cookies/{service}.txt`` cannot shadow it."""
from unshackle.commands.dl import dl
if profile is None:
return dl.get_cookie_jar(service, None)
path = config.directories.cookies / service / f"{profile}.txt"
return dl.load_cookie_file(path) if path.exists() else None
def server_account_regions(service: str) -> Optional[Dict[str, Any]]:
"""What ``GET /api/services`` advertises: the union of regions the server's accounts cover."""
spec = server_account_spec(service)
if not spec:
return None
if spec is True:
return {"regions": [], "global": True}
union: set[str] = set()
for value in spec.values():
union |= _regions(value)
return {"regions": sorted(union - {"global"}), "global": "global" in union}
def server_region() -> Optional[str]:
"""The server's own country code, from the cached IP lookup; None when unknown."""
try:
from unshackle.core.utils.ip_info import get_ip_info
info = get_ip_info(None, cached=True)
return (info.get("country") or "").lower() or None if info else None
except Exception:
return None
def server_account_pool(service: str, region: Optional[str]) -> Optional[list[Optional[str]]]:
"""Profiles of the server's own accounts usable in ``region``; None when the service is client-managed."""
spec = server_account_spec(service)
if not spec:
return None
creds = config.credentials.get(service)
if spec is True:
return sorted(creds) if isinstance(creds, dict) else [None]
region = (region or "").lower()
return sorted(p for p, v in spec.items() if {"global", region} & _regions(v))
def next_server_profile(service: str, region: Optional[str]) -> Optional[str]:
"""Round-robin the next server account profile for ``region``."""
pool = server_account_pool(service, region) or []
if not pool:
covered = server_account_regions(service) or {}
regions = ", ".join(covered.get("regions") or []) or "none"
raise APIError(
APIErrorCode.INVALID_INPUT,
f"No server account for {service} in region '{region or 'unknown'}'. "
f"Its accounts cover: {regions}. Pass --proxy with one of those country codes.",
details={"service": service, "region": region, "regions": covered.get("regions") or []},
)
key = f"{service}:{','.join(p or '' for p in pool)}"
n = _account_counters.get(key, 0)
_account_counters[key] = n + 1
return pool[n % len(pool)]
def server_accounts_allowed(request: Optional[web.Request] = None, service: Optional[str] = None) -> bool:
"""Whether the calling API key may authenticate ``service`` with one of the server's own accounts.
Same shape as ``server_cdm``: ``server_accounts: true`` on the ``serve.users`` entry for every
service, or a list of service tags. Off unless set. The admin secret (no ``users`` entry) keeps
full access. Combine with ``server_account_spec``: the server treats an API key without the
opt-in as a client-managed remote session for that service.
"""
if not request:
return True
user_config = config.serve.get("users", {}).get(request_secret_key(request))
if user_config is None:
return True
allowed = user_config.get("server_accounts", False)
if isinstance(allowed, str):
allowed = [allowed]
if isinstance(allowed, (list, tuple, set)):
return service is not None and Services.get_tag(service).upper() in {
Services.get_tag(s).upper() for s in allowed
}
return bool(allowed)
def resolve_server_account(request: Optional[web.Request], service: str, region: Any) -> Optional[str]:
"""Profile of the server account a search or list call runs on; None when the service is client-managed.
Raises FORBIDDEN for an API key without the ``server_accounts`` opt-in.
"""
if server_account_spec(service) is None:
return None
if not server_accounts_allowed(request, service):
raise APIError(
APIErrorCode.FORBIDDEN,
f"Your API key may not use the server's accounts for {service}.",
details={"service": service},
)
return next_server_profile(service, region if isinstance(region, str) else None) or "default"
def server_account_for(request: Optional[web.Request], service: str) -> bool:
"""True when this request's remote session for ``service`` should run on a server account."""
return server_account_spec(service) is not None and server_accounts_allowed(request, service)
def server_proxy_allowed(request: Optional[web.Request] = None) -> bool:
"""Whether the calling API key may use the server's own configured proxy providers.
@@ -831,6 +993,10 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] =
proxy_param, proxy_providers = await asyncio.to_thread(resolve_handler_proxy, data, normalized_service, request)
account = resolve_server_account(request, normalized_service, data.get("client_region"))
if account:
profile = None if account == "default" else account
cdm = load_full_cdm(normalized_service, profile, data.get("cdm_type"))
parent_ctx = build_parent_ctx(profile, cdm, proxy_param, no_proxy, proxy_providers, service_config)
service_module = Services.load(normalized_service)
@@ -844,7 +1010,11 @@ async def search_handler(data: Dict[str, Any], request: Optional[web.Request] =
details={"service": normalized_service},
)
cookies = dl.get_cookie_jar(normalized_service, profile)
if account:
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{account}")
cookies = server_account_cookies(normalized_service, profile)
else:
cookies = dl.get_cookie_jar(normalized_service, profile)
credential = dl.get_credentials(normalized_service, profile)
service_instance.authenticate(cookies, credential)
@@ -1846,6 +2016,12 @@ async def server_config_handler(request: Optional[web.Request] = None) -> web.Re
"remote_only": bool(serve_cfg.get("remote_only", False)),
"cdm_overrides": redact_config(serve_cfg.get("cdm_overrides")),
"allow_job_credentials": bool(serve_cfg.get("allow_job_credentials", False)),
"server_accounts": {
tag: server_account_regions(tag)
for tag in (serve_cfg.get("server_accounts") or {})
if server_accounts_allowed(request, tag)
}
or None,
},
"directories": {
"downloads": str(config.directories.downloads),
@@ -2046,6 +2222,7 @@ SESSION_TRANSPORT_KEYS = {
"cookies",
"cache",
"client_region",
"proxy_region",
"cdm_type",
"range_",
"vcodec",
@@ -2061,11 +2238,14 @@ def create_service_instance(
proxy_param: Optional[str],
proxy_providers: list,
profile: Optional[str],
server_account: bool = False,
) -> Any:
"""Make and authenticate a service instance.
"""Make a service instance and resolve its credentials and cookies.
Supports client-sent credentials/cookies (for remote-dl) with fallback
to server-local config (for backward compatibility).
With ``server_account`` the service takes the server's own credential and cookie file for
``profile`` and drops anything the client sent. Otherwise only client-sent data counts: a
client that sends nothing authenticates with nothing, because the server never lends its
own accounts.
"""
from unshackle.commands.dl import dl
from unshackle.core.credential import Credential
@@ -2118,7 +2298,14 @@ def create_service_instance(
service_module = Services.load(normalized_service)
service_instance = instantiate_service(parent_ctx, service_module, title_id, data, SESSION_TRANSPORT_KEYS)
# Resolve credentials: client-sent > server-local
if server_account:
return (
service_instance,
server_account_cookies(normalized_service, profile),
dl.get_credentials(normalized_service, profile),
)
credential = None
cred_data = data.get("credentials")
if cred_data and isinstance(cred_data, dict):
credential = Credential(
@@ -2126,10 +2313,8 @@ def create_service_instance(
password=cred_data["password"],
extra=cred_data.get("extra"),
)
else:
credential = dl.get_credentials(normalized_service, profile)
# Resolve cookies: client-sent > server-local
cookies = None
cookie_text = data.get("cookies")
if cookie_text and isinstance(cookie_text, str):
import base64
@@ -2149,8 +2334,6 @@ def create_service_instance(
with suppress(OSError):
os.unlink(tmp_path)
else:
cookies = dl.get_cookie_jar(normalized_service, profile)
return service_instance, cookies, credential
@@ -2187,13 +2370,23 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
import hashlib
import uuid as uuid_mod
from unshackle.core.cacher import Cacher
from unshackle.core.config import config as app_config
session_id = str(uuid_mod.uuid4())
api_key = request.headers.get("X-Secret-Key", "anonymous") if request else "anonymous"
api_key_hash = hashlib.pbkdf2_hmac("sha256", api_key.encode(), b"unshackle-session-ns", 100_000).hex()[:12]
session_cache_tag = f"_sessions/{api_key_hash}/{session_id}/{normalized_service}"
session_cache_dir = f"_sessions/{api_key_hash}/{session_id}/{normalized_service}"
session_cache_tag: Optional[str] = session_cache_dir
server_account = server_account_for(request, normalized_service)
if server_account:
region = data.get("proxy_region") or data.get("client_region")
if region is not None and not (isinstance(region, str) and re.fullmatch(r"[A-Za-z]{2}", region)):
raise APIError(APIErrorCode.INVALID_INPUT, "proxy_region must be a two-letter country code.")
if region is None and data.get("no_proxy"):
region = server_region()
profile = next_server_profile(normalized_service, region)
log.info(f"Using server account '{sanitize_log(profile or 'default')}' for {normalized_service}")
service_instance, cookies, credential = create_service_instance(
normalized_service,
@@ -2202,12 +2395,17 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
proxy_param,
proxy_providers,
profile,
server_account=server_account,
)
service_instance.cache = Cacher(session_cache_tag)
if server_account:
session_cache_tag = None
service_instance.cache = Cacher(f"_accounts/{normalized_service}/{profile or 'default'}")
else:
service_instance.cache = Cacher(session_cache_dir)
cache_data = data.get("cache", {})
if cache_data:
if cache_data and session_cache_tag:
import base64
cache_dir = app_config.directories.cache / session_cache_tag
@@ -2230,6 +2428,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
session_id=session_id,
owner_key=api_key,
creator_ip=request.remote if request else None,
server_account=(profile or "default") if server_account else None,
)
session.cache_tag = session_cache_tag
# Echoed to every dashboard viewer, so cap what an arbitrary client can push into it.
@@ -2256,6 +2455,7 @@ async def session_create_handler(data: Dict[str, Any], request: Optional[web.Req
"session_id": session.session_id,
"service": normalized_service,
"status": "authenticating",
"server_account": server_account,
}
)
+35 -1
View File
@@ -40,6 +40,8 @@ from unshackle.core.api.handlers import (
refresh_services_handler,
retry_download_job_handler,
search_handler,
server_account_regions,
server_accounts_allowed,
server_config_handler,
session_create_handler,
session_delete_handler,
@@ -174,6 +176,16 @@ async def services(request: web.Request) -> web.Response:
pending_update:
type: boolean
description: Present (true) while a repo refresh waits for this service's jobs to finish
server_accounts:
type: object
description: Present when the server lends its own accounts for this service (serve.server_accounts)
properties:
regions:
type: array
items:
type: string
global:
type: boolean
aliases:
type: array
items:
@@ -240,6 +252,8 @@ async def services(request: web.Request) -> web.Response:
}
if tag in services_module.PENDING:
service_data["pending_update"] = True
if server_accounts_allowed(request, tag) and (server_accounts := server_account_regions(tag)) is not None:
service_data["server_accounts"] = server_accounts
try:
service_module = Services.load(tag)
@@ -1532,9 +1546,29 @@ async def session_create(request: web.Request) -> web.Response:
cache:
type: object
additionalProperties: true
client_region:
type: string
description: Two-letter country the client sits in
proxy_region:
type: string
description: Two-letter country the client resolved its proxy for; picks a server account
responses:
'200':
description: Remote session created with titles, tracks, and chapters
description: Remote session created; authentication continues in the background
content:
application/json:
schema:
type: object
properties:
session_id:
type: string
service:
type: string
status:
type: string
server_account:
type: boolean
description: True when the server authenticated with one of its own accounts
'400':
description: Invalid request
'401':
+4
View File
@@ -40,6 +40,7 @@ class SessionEntry:
creator_ip: Optional[str] = None
owner_key: Optional[str] = None # X-Secret-Key that owns this session
cache_tag: Optional[str] = None
server_account: Optional[str] = None # profile name when the server lent its own account
input_bridge: Optional[InputBridge] = None
auth_status: AuthStatus = AuthStatus.AUTHENTICATED
auth_error: Optional[str] = None
@@ -68,6 +69,7 @@ class SessionEntry:
"tracks": len(self.tracks),
"auth_status": self.auth_status.value,
"auth_error": self.auth_error,
"server_account": self.server_account,
"client": self.client,
"actions": list(self.actions),
"created_at": self.created_at.isoformat(),
@@ -111,6 +113,7 @@ class SessionStore:
session_id: Optional[str] = None,
owner_key: Optional[str] = None,
creator_ip: Optional[str] = None,
server_account: Optional[str] = None,
) -> SessionEntry:
"""Make a new remote session with an authenticated service instance."""
async with self._lock:
@@ -126,6 +129,7 @@ class SessionStore:
service_instance=service_instance,
owner_key=owner_key,
creator_ip=creator_ip,
server_account=server_account,
)
self._sessions[session_id] = entry
_publish("create", entry)
+43 -14
View File
@@ -661,6 +661,7 @@ class RemoteService:
svc_config = services_config.get(service_tag, {})
self._server_cdm: bool | None = services_config.get("_server_cdm")
self._server_accounts: Optional[dict] = services_config.get("_server_accounts")
self.apply_service_config(svc_config)
def apply_service_config(self, svc_config: dict) -> None:
@@ -707,29 +708,57 @@ class RemoteService:
create_data: Dict[str, Any] = {"service": self.service_tag, "title_id": self.title_id}
credentials = load_credentials_for_transport(self.service_tag, profile)
if credentials:
create_data["credentials"] = credentials
if self._server_accounts is None:
credentials = load_credentials_for_transport(self.service_tag, profile)
if credentials:
create_data["credentials"] = credentials
cookies_text = load_cookies_for_transport(self.service_tag, profile)
if cookies_text:
create_data["cookies"] = cookies_text
cookies_text = load_cookies_for_transport(self.service_tag, profile)
if cookies_text:
create_data["cookies"] = cookies_text
else:
self.credential = None
self.log.info(
"The server manages the account for this service" + (" and ignores --profile" if profile else "")
)
if not no_proxy and proxy:
resolved_proxy = resolve_proxy_arg(proxy)
if resolved_proxy:
create_data["proxy"] = resolved_proxy
if not no_proxy and not proxy:
client_region = None
if not no_proxy:
try:
from unshackle.core.utils.ip_info import get_ip_info
ip_info = get_ip_info(self._session, cached=True)
if ip_info and ip_info.get("country"):
create_data["client_region"] = ip_info["country"].lower()
client_region = ip_info["country"].lower()
except Exception:
pass
if not no_proxy and not proxy and self._server_accounts and not self._server_accounts.get("global"):
regions = [str(r).lower() for r in self._server_accounts.get("regions") or []]
if regions and client_region and client_region not in regions:
try:
create_data["proxy"] = resolve_proxy_arg(regions[0])
create_data["proxy_region"] = regions[0]
self.log.info(f"Using a '{regions[0]}' proxy to match a server account ({', '.join(regions)})")
except click.ClickException:
self.log.warning(
f"The server's accounts work in {', '.join(regions)} and you are in {client_region}. "
"There is no local proxy for that region. Pass --proxy yourself if the server "
"rejects the remote session"
)
elif not no_proxy and proxy:
resolved_proxy = resolve_proxy_arg(proxy)
if resolved_proxy:
create_data["proxy"] = resolved_proxy
query = (self.ctx.parent.params.get("proxy_query") if self.ctx.parent else None) or proxy
country = re.match(r"^(?:[a-z]+:)?([a-z]{2})\d*$", query, re.IGNORECASE)
if country:
create_data["proxy_region"] = country.group(1).lower()
if not no_proxy and not proxy and client_region:
create_data["client_region"] = client_region
elif not no_proxy and not proxy and self._server_accounts and not self._server_accounts.get("global"):
self.log.warning("Could not detect your region; pass --proxy <country> so the server can pick an account")
if profile:
create_data["profile"] = profile
if no_proxy:
@@ -757,7 +786,7 @@ class RemoteService:
create_data["cdm_type"] = "playready" if is_playready_cdm(cdm) else "widevine"
cache_data = self.load_cache_files()
cache_data = self.load_cache_files() if self._server_accounts is None else None
if cache_data:
create_data["cache"] = cache_data
+1
View File
@@ -396,6 +396,7 @@ class Services(click.Group):
server_name = ctx.parent.params.get("server") if ctx.parent else None
server_url, api_key, services_config = resolve_server(server_name)
services_config["_server_accounts"] = svc_info.get("server_accounts") if svc_info else None
service_params = {k: v for k, v in kwargs.items() if v is not None and v is not False}
return RemoteService(ctx, tag, title, server_url, api_key, services_config, service_params=service_params)